AI crawler log analysis: the recipe to see which pages ChatGPT reads on your site
Partner · Strategy, advertising and measurement
To find out which pages of your site ChatGPT, Claude or Perplexity actually read, you need your server logs: every visit from an AI crawler leaves a line with its name, the page it read and the date. The recipe takes six steps. You collect thirty days of logs, keep the AI crawler lines, weed out impostors, count pages in a spreadsheet, read three lists, then fix what they reveal.
- GA4 does not see AI crawlers: they do not run its code. Only server logs show what they read.
- A visit from
ChatGPT-User,Claude-UserorPerplexity-Usermeans a person just asked a question and the AI fetched your page to answer it. - A spreadsheet is enough for most sites. Paid tools become useful on large catalogues.
- Missing URLs (404 errors) that AI crawlers keep requesting can be recovered with a redirect.
- A visit is not a citation: logs need to be paired with tests run directly in the AI assistants.
On this page
AI crawler log analysis
AI crawler log analysis means reading your server logs to spot visits from artificial intelligence crawlers: which crawler came, which page it read, when, and with what result. SEO has used log analysis for years to follow Googlebot. The AI version applies the same method to the crawlers of OpenAI, Anthropic, Perplexity and others, separating those that train a model from those that read your page because a person just asked a question.
Why GA4 is not enough
Google Analytics 4 counts visits through a small piece of code that the visitor's browser runs. AI crawlers do not run it, so their visits never appear in your reports. GA4 only sees the next step, when a person clicks a link cited in an answer (AI-referred traffic). Even then, the source has to be passed along: in October 2026, a bug caused visits from ChatGPT to be classified as direct traffic, as explained in our note on that bug.
The search engines' own tools fill in another part of the picture. The AI Performance report in Bing Webmaster Tools shows your pages' citations in Copilot, with the queries. The generative AI report in Search Console shows your impressions in Google's AI surfaces, without the queries. Neither sees ChatGPT, Claude or Perplexity. What each source measures is detailed in our comparison of AI visibility measurement tools.
Server logs therefore remain the only source you own that shows what these AI assistants come to read on your site.
What you need
| Ingredient | Why | Where to find it |
|---|---|---|
| Thirty days of access logs | Less than thirty days shows no trend on a small business site | Your host (step 1) |
| The list of AI crawlers | To keep the right lines and sort them | The table in step 2 |
| A spreadsheet | To count pages by crawler | Google Sheets or Excel |
Step 1: get your logs
The access log (often named access.log) is a text file where the server writes one line per request. Each line holds the visitor's IP address, the date, the requested page, the response code (200 when all is well, 404 when the page cannot be found) and the user agent, the name the visitor gives itself. Where to find it depends on your hosting.
| Hosting | Where to find the logs |
|---|---|
| cPanel (most shared hosts) | Metrics section, “Raw Access” tool: today's log and the archives download as compressed files (.gz). |
| Kinsta | MyKinsta, Sites, your site, Logs tab. The file downloads through the file manager or SFTP. Kinsta keeps it for up to four days. |
| WP Engine | User Portal, Sites, the environment, then Advanced, Logs and the Access tab. Edge logs cover up to 20 million requests from the last 72 hours. |
| Cloudflare in front of your site | AI Crawl Control, available on all plans (Free included), shows requests per crawler, the most read pages and grouped response codes. No file needed: you can skip straight to step 5. |
| Dedicated or virtual server | Usually in /var/log/apache2/ or /var/log/nginx/. Your system administrator knows where to find them. |
If you do not know where to look, write to your host: “Can you send me the raw access logs for the last thirty days?” Many keep them only a few days or weeks. If yours deletes them quickly (Kinsta keeps four days), download them twice a week until you have your month.
Shopify does not give access to its server logs and Squarespace documents no such access. Wix shows reports drawn from its logs in its dashboard, but its official help describes them for search engine crawlers. On these platforms, rely on Bing's AI Performance report, referred traffic in GA4 and tests run directly in the AI assistants.
Step 2: recognize AI crawlers
Each crawler names itself in the user agent of its request, so you only need to search for these names in the file. The classification that matters separates three families, from most to least useful for your content strategy.
| Family | What the visit means | Names to look for |
|---|---|---|
| User-initiated fetch | A person just asked a question and the AI reads your page to answer it | ChatGPT-User (OpenAI), Claude-User (Anthropic), Perplexity-User, MistralAI-User (Mistral) |
| Search | The AI indexes your site so it can cite it in its answers | OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot, MistralAI-Index (Mistral) |
| Training | The AI collects pages to train a model, unrelated to any specific question | GPTBot (OpenAI), ClaudeBot (Anthropic), MistralAI-Training (Mistral), CCBot (Common Crawl), Meta-ExternalAgent (Meta), Bytespider (ByteDance) |
The first family is the one that matters most to you. These retrieval crawlers only come when a real person asks a question your page can answer. Names change from time to time: each vendor keeps its list up to date in its documentation (see the sources at the bottom of this page). Deciding whether to allow or block each family is a separate topic, covered in our article on which AI crawlers to allow or block.
Step 3: weed out impostors
Any program can call itself GPTBot. Content scrapers do exactly that to get past filters. Before drawing conclusions, check that the IP addresses of the lines you kept really belong to the vendor.
OpenAI, Anthropic, Perplexity and Mistral publish the IP addresses of their crawlers in small files linked in this article's sources. Compare the addresses in your lines with these lists: a line that carries a crawler's name without coming from one of its vendor's addresses is an impostor. If your site runs through Cloudflare, this check is already done: Cloudflare keeps its own list of verified bots.
On a small business site, impostors are rarely numerous, but a single address reading thousands of pages in an hour under an AI crawler's name can skew your whole ranking. Leave it out.
Step 4: count the pages
The goal is simple: for each page, how many reads by crawler family. Three ways to get there, depending on your comfort level.
In a spreadsheet
Import the file into Google Sheets or Excel with a space as the column separator. Add a “Family” column based on the table in step 2, then build a pivot table: pages as rows, families as columns, number of reads as values. Google Sheets accepts up to 20 million cells per file since September 2026. A very large log can still exceed that limit: in that case, keep only the crawler lines with the first command in method 03.
With a log file analyser
Screaming Frog's Log File Analyser reads the file, recognizes crawlers and verifies GPTBot and Perplexity's crawlers. You can add the other names from the table in step 2. The free version is limited to 1,000 lines, enough for a trial. The licence costs £99 a year. On a very large catalogue, platforms such as Botify or Oncrawl run log analysis at scale.
With one command line
On a Mac or Linux, the Terminal does the count, even on a file of several gigabytes. This command ranks the pages most read by user-initiated fetches:
grep -E "ChatGPT-User|Claude-User|Perplexity-User|MistralAI-User" access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -50And this one lists the missing URLs requested by AI crawlers:
grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User" access.log \
| awk '$9 == 404 {print $7}' | sort | uniq -c | sort -rnThey assume the most common log format (where the page is the seventh field and the response code the ninth). If the results look empty or odd, open one line of the file to see the order in which the information is written.
You can also hand the count to Claude or ChatGPT. Filter the file first to keep only the crawler lines: that way you send none of your visitors' IP addresses, which can be personal information under Quebec's Law 25. Then ask for a ranking of pages by crawler family and check a few numbers by hand before relying on it.
Step 5: read three lists
The pivot table gives you hundreds of rows. Three lists are enough to decide what to do.
The pages most read by user-initiated fetches
These are the topics real people ask AI assistants about, and for which your page was judged useful. You do not see the question, but the page that was read gives a good idea of it. Keep these pages up to date first, with the answer in the first paragraph and dated figures.
The important pages AI crawlers never read
Your service pages, your prices, your category pages: if no AI crawler visits them in thirty days, check your robots.txt file and the settings of your firewall or Cloudflare, which can block these crawlers without your knowledge. Also check that the page's text displays without JavaScript: most AI crawlers do not run it and only see an empty page. Finally, a page that nothing links to is hard to find, for AI assistants as for Google.
The missing URLs AI crawlers request
An AI that requests a page returning 404 is often looking for an old address it remembered, or an address it pieced together from your domain name. Every address that comes back several times deserves a 301 redirect to the closest page. It is one of the quickest wins of the exercise.
Step 6: fix, then repeat every month
Each list leads to specific actions. Enrich and date the pages on the first list, then check in GA4 whether they receive visits from AI assistants. Unblock or rewrite the pages on the second. Redirect the addresses on the third. For how to write a page an AI can reuse, the method is in our article on the answer-first structure.
Then run the analysis once a month on the same date and compare with the previous month. On a small business site, user-initiated fetches are often counted in dozens a month rather than thousands: the trend over several months is what counts, not one week's number.
On falia.co, we no longer download any file. The site is served by Cloudflare, and a small function placed in front of it recognizes each AI crawler read as it happens, then writes it to a Google spreadsheet: the date, the crawler, its family and the page. A daily report sums it all up. The recipe is the same as in this article: we simply automated steps 1 to 4.
What logs do not tell you
A read is not a citation. A crawler can read your page and then cite another provider's. Conversely, an AI can cite you without reading your page that day: it often relies on its own index, on a search engine's index or on a recent copy. ChatGPT, for example, often answers from the index built by OAI-SearchBot or from a partner search engine's results, as explained in our article on ChatGPT's search engine. Logs therefore show only part of your presence in answers.
Nor do they tell you who asked the question or in what words. To know what AI assistants actually say about your business, you need to ask them your customers' questions and note who gets cited. The two measures complement each other: logs tell you what AI assistants come to read, tests tell you what they do with it. That is the combination we use in our AI visibility work.
Knowing what AI assistants read on your site is the starting point of the goal Strengthen your visibility in AI answers.
To better understand where ChatGPT's answers come from, also read what search engine ChatGPT uses.
Frequently asked questions about AI crawler log analysis
Can I see the questions people ask ChatGPT about my business?
No. Logs show the page the AI came to read, the date and the crawler, never the question or the person who asked it. The page that was read gives a good idea of the question's topic. To know the answers themselves, ask AI assistants your customers' questions and note who gets cited.
My site runs on Shopify: how can I tell what AI assistants read?
Shopify does not give access to its server logs. You can still follow Bing Webmaster Tools' AI Performance report, which shows your citations in Copilot, visits from AI assistants in GA4 and the results of tests run directly in ChatGPT, Claude or Perplexity.
How much data do I need before drawing conclusions?
Plan on at least thirty days for a first reading, then compare month over month. On a small business site, user-initiated fetches can number in the dozens a month: a single week shows no trend.
If I block training crawlers, does the analysis change?
User-initiated fetches remain visible. Blocking GPTBot or ClaudeBot in robots.txt does not affect ChatGPT-User or Perplexity-User: OpenAI states that robots.txt rules may not apply to actions initiated by a user, and Perplexity says its user-initiated crawler generally ignores them.
Do I need a paid tool for this analysis?
No, not for most sites. A spreadsheet or two command lines are enough. A log file analyser such as Screaming Frog's becomes useful when the file is large or when you want crawlers verified automatically. Large-scale analysis platforms make sense for big catalogues.
- OpenAI, Overview of OpenAI Crawlers (GPTBot, OAI-SearchBot, ChatGPT-User), accessed October 2026. IP address lists: gptbot.json, searchbot.json and chatgpt-user.json.
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? (ClaudeBot, Claude-User, Claude-SearchBot), accessed October 2026. IP address list: bots.json.
- Perplexity, Perplexity Crawlers, accessed October 2026. IP address lists: perplexitybot.json and perplexity-user.json.
- Mistral AI, Mistral crawlers, accessed October 2026. IP address lists: mistralai-user-ips.json and mistralai-index-ips.json.
- Cloudflare, AI Crawl Control and Analyze AI traffic, accessed October 2026.
- Screaming Frog, SEO Log File Analyser and its configuration guide, accessed October 2026.
- cPanel, Raw Access; Kinsta, Server Logs; WP Engine, Access Logs and Common Errors, accessed October 2026.
- Wix, Using Your Site’s Log Data in Wix SEO Reports, accessed October 2026.
- Google Workspace Updates, Doubled cell limits in Google Sheets now generally available, September 10, 2026.
Gabriel almost always takes your first call and carries out your audit. He builds the strategy starting from your growth goal: where to put your budget, which market to test and how to connect each lead to a real sale in your CRM. He mainly leads engagements for three goals: Optimize the profitability of your digital campaigns, Develop a new market, and Generate demand and growth. With Geneviève, he also works on organic search (SEO), AI visibility (GEO) and conversion rate optimization (CRO). The sales a Google Ads or Meta Ads campaign brings in depend on the page that receives the click. He writes mainly about marketing strategy, paid advertising and measurement.
About Falia →
SEO
Paid advertising
Conversion
AI visibility
Web design