AI Crawler Log Analysis

TL;DR — AI crawlerAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. log analysis means looking at your server’s raw access logs to see which AI botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — like OpenAI’s GPTBot or Anthropic’s ClaudeBot — actually visited your site, how often, and what they got. Your logs are the only place that records this. Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. doesn’t show AI bots, and Google Analytics only sees people who click through, not the bots themselves. The catch: lots of traffic claims to be a famous AI bot but isn’t, so you can’t trust the name alone.

What a log file is

Server access logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. record requests that reached the server, including user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., time, path, and response details when configured. Evidence for this claim Server access logs record requests and can include fields such as request path, status, referrer, and user agent. Scope: Apache HTTP Server logging; available fields depend on server configuration. Confidence: high · Verified: 2026-07-14 Apache: Log files Vendor crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. documentation identifies user agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., but a string alone does not prove the requester is genuine. Evidence for this claim OpenAI publishes user-agent tokens and controls for several of its crawlers. Scope: OpenAI's documented crawlers; a user-agent string is not proof of downstream training, retrieval, citation, or use. Confidence: high · Verified: 2026-07-14 OpenAI: Crawlers

Every time anything — a person’s browser, GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., GPTBot — asks your server for a page, your server writes a line to a log file. Each line records the time, the visitor’s IP address, what they asked for, the “user-agent” (a label the visitor uses to identify itself, like GPTBot), and the response code your server sent back (200 for success, 404 for not foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore., and so on).

AI crawler log analysisAI crawler log analysis is the practice of pulling raw server or CDN access logs and examining them for requests from AI bots — training crawlers, AI-search indexers, and user-triggered fetchers — to verify with first-party data which bots actually hit your site, whether they're real or spoofed, and what they got. is just reading those lines and filtering them down to the AI bots. It answers questions you literally can’t answer any other way:

• Did an AI bot actually visit my site at all?
• Which pages did it fetch, and which important pages did it ignore?
• Was it the real bot, or something pretending to be it?
• What did it get — my actual content, or an error page?

If you want the full cast of AI bots and what each one is for (training bots vs. search bots vs. the “fetch this page right now” bots), that’s a separate topic — see AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls.. This page is about the how: pulling the logs and reading them.

Why your dashboards don’t cover this

• Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. has a Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root)., but it’s only about GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.. It shows nothing about GPTBot, ClaudeBot, or PerplexityBot.
• Google Analytics (GA4) only records visitors who click a link to your site and arrive with a “referrer.” A crawler grabbing your page in the background never shows up there at all.

So the logs are the ground truth. Nothing else sees the bot.

The one thing beginners miss

A GPTBot label in your logs does not prove it was really OpenAI. Anyone can put any user-agent string they want on a request — it’s like writing a return address on an envelope. Lots of scrapers slap a famous bot’s name on their traffic to look legitimate. The only way to be sure is to check the visitor’s IP address against the list of real addresses the AI company publishes. That verification step is the heart of doing this properly, and the Advanced tab walks through it.

TL;DR — Pull your Apache/Nginx or CDN access logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened., then do four things: verify (match the user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. and confirm the IP against each operator’s published list — spoof rates run from HUMAN Security’s 5.7% to Duane Forrester’s self-tested 81.8%); choose a tool by scale (grep → Screaming Frog Log File Analyser → ELK/Splunk → BigQuery/Cloudflare); measure crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. by botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., which pages get hit, crawl-vs-render, and status codes; and interpret honestly — retrieval is necessary but not sufficient, so “more crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. = more citations” is unproven. This is the method sibling to AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. (which owns what the bots are), AI traffic attributionAI traffic attribution is the practice of correctly identifying and measuring website visits that come from AI tools — ChatGPT, Perplexity, Gemini, Claude, AI Overviews, and AI browsers. It's hard because many of those tools strip the referrer header, so the visits land in your analytics as Direct traffic with no source. (the click side), and LLM visibilityLLM visibility (or AI visibility) is the aggregate measure of how often and how prominently a brand or page shows up in AI-generated answers — across AI Overviews, ChatGPT, Perplexity, Copilot, and Gemini. It's the AI-search analog of organic visibility, but it's driven by different signals. (the retrieved → mentioned → cited framework this observes the first stage of).

What this article owns — and what it doesn’t

Logs demonstrate requests, not whether content was used for training, retrieval, or an answer. Evidence for this claim Server access logs record requests and can include fields such as request path, status, referrer, and user agent. Scope: Apache HTTP Server logging; available fields depend on server configuration. Confidence: high · Verified: 2026-07-14 Apache: Log files Verify bots using provider-published mechanisms where available and treat attribution as bounded evidence. Evidence for this claim OpenAI publishes user-agent tokens and controls for several of its crawlers. Scope: OpenAI's documented crawlers; a user-agent string is not proof of downstream training, retrieval, citation, or use. Confidence: high · Verified: 2026-07-14 OpenAI: Crawlers

The sibling AI crawlers articleAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. already owns the bot-by-bot table, the three-category taxonomy (training / AI-search / user-triggered fetch), the Google-Extended-is-a-token-not-a-bot distinction, and the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. recipes. I’m not re-deriving any of that here. AI traffic attributionAI traffic attribution is the practice of correctly identifying and measuring website visits that come from AI tools — ChatGPT, Perplexity, Gemini, Claude, AI Overviews, and AI browsers. It's hard because many of those tools strip the referrer header, so the visits land in your analytics as Direct traffic with no source. owns the GA4/referrer side — what the bot sent back. This article is the opposite side of the funnel: what the bot took. And LLM visibility owns the retrieved → mentioned → cited framework; log analysis is how you observe the retrieved stage specifically, and nothing past it.

This is the AI-era sequel to classic SEO log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened.. When I reviewed Ahrefs’ How to Do an SEO Log File Analysis, the dimensions were crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial., URLs crawled, status codes, and bot verification against Google’s published IPs. Same skeleton here — but pointed at a bot population that mostly doesn’t render JavaScript, gets impersonated constantly, and crawls in erratic bursts rather than a steady stream.

Step 1 — Get your logs

Your logs live in one of two places, or both:

• Origin server logs. Apache (access.log), Nginx (access.log), or your app server. Each line has, at minimum: timestamp, client IP, request method + path, status code, bytes, referrer, and user-agent.
• CDN / edge logs. If you’re behind Cloudflare, Fastly, Akamai, etc., a lot of bot traffic is answered at the edge and may never reach your origin — so the edge log is the fuller record. Cloudflare exposes this via Logpush (and a GraphQL API); Fastly via real-time log streaming.

The fields you actually need for AI-bot analysis: timestamp, client IP, user-agent, request path, status code, and ideally bytes and referrer.

The practical problem is retention. Many hosts keep only a few days of logs. Lauren Busby’s Search Engine Land piece names the fix directly — a scheduled pull turns a short window into something analyzable over time: a scheduled SFTP job, built in a workflow tool like n8n or scripted, is enough to turn a short retention window into something you can actually analyze over time (Busby, SEL). Set that up before you need the data.

One honest limit to keep in mind from the start: as Busby puts it, log files show you what reached your site, but they don’t always show you what tried to (SEL) — requests blocked or answered upstream may not appear at all.

Step 2 — Verify before you trust the user-agent

This is the step that separates AI-bot log analysis from the classic version, and the one most competing how-tos skip. User-agent strings are trivially spoofed. Two independent data points on how bad it gets:

• HUMAN Security analyzed two weeks of traffic claiming to be one of 16 well-known AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. and found 5.7% of it was spoofed — roughly 1 in 18 requests (reported via SEJ).
• Duane Forrester ran the same check on his own logs and found far worse. Of 33 requests carrying a live-fetch name, six came from an IP the vendor publishes and twenty-seven did not — an 81.8% spoof rate among the requests he could check (SEJ). His GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. number was worse still: of 799 requests carrying the Googlebot name, only 107 came from a verified Google address — the other roughly 87% were not Google (SEJ).

Treat those as directional from different samples and methods, not one universal number. More importantly, do not generalize one provider’s verification method to every bot. Some operators publish address ranges; others document user-agent tokens without a current public range or DNS-verification contract (SEJ).

The verification method:

1. Match the user-agent string. GPTBot identifies itself as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot; OAI-SearchBot and ChatGPT-User carry their own tokens (OpenAI bots docs).
2. Use the provider’s current official verification method when one exists. OpenAI ships openai.com/gptbot.json, searchbot.json, and chatgpt-user.json. Anthropic’s current crawler documentation says an address in its published list indicates a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. is coming from Anthropic. Use that current provider-specific method; do not generalize it into a universal bot-verification rule.
3. Do not invent a fallback. Reverse-plus-forward DNS is valid for a provider only when that provider publishes the hostname pattern and verification procedure. A generic PTR match proves control of DNS, not the claimed bot identity. Screaming Frog’s Log File Analyser can apply publicly confirmed lists where available; its verify-on-import feature performs a lookup against publicly confirmed IP lists to confirm bots are genuine (Screaming Frog).

Verification should inform precise policy and incident response. Prefer the provider’s documented robots token for a crawl policy; reserve network controls for abuse or security cases and label them separately from robots compliance.

TIP Verify crawler claims before trusting the totals

The tool can flag a claimed crawler outside its operator's published ranges; treat that as a spoofed claim, then choose a response based on behavior.

Upload a representative access-log sample and inspect identity, response, and crawl-pattern findings with my free Log File Analyzer Free

1. Include timestamp, IP, user-agent, path, and status fields so identity and response findings have evidence.
2. Review range mismatches before aggregating requests under a famous crawler name.
3. Use the exact paths and status groups to separate a spoofing problem from wasted crawl, errors, redirects, or missing important URLs.

Tap to enlarge

The completed Log File Analyzer result reports that 30 requests spoofed a verified crawler because one IP used a search or AI-crawler user-agent outside that operator's published ranges. It recommends verifying the owning network before blocking or rate-limiting. A separate finding says AI-training crawlers account for 32.1 percent of crawler traffic, or 93 hits, and frames allow versus disallow as a business decision. Another finding says Googlebot spent 49.6 percent of 129 requests on non-200 responses, including 404 and 410 URLs, redirects, and server errors. These are distinct findings with distinct remedies.

Step 3 — Choose your tool

Scale it to the size of the job, roughly free → paid, ground-truth → managed:

• grep / PowerShell — a quick one-off count and a verification-aware filter. The AI crawlers articleAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. has the basic bot-count snippet; the Scripts tab here extends it into IP verification, a status-code breakdown, and crawl-vs-render detection.
• Screaming Frog Log File Analyser — a desktop importer with built-in AI-bot presets (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, CCBot) and a “Verify Bots When Importing” toggle. Its Response Codes tab breaks 2XX/3XX/4XX/5XX per URL, the User AgentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. tab shows requests and error rates per bot, the URLs tab sorts by Num Events (most-fetched pages), and the IPs tab lets you investigate suspicious sources (tutorial).
• ELK Stack (Elasticsearch / Logstash / Kibana) or Splunk — for continuous, larger-scale ingestion with dashboards and alerting, once a single-import desktop tool gets too slow or you want ongoing monitoring rather than periodic exports.
• BigQuery — for long-term retention and SQL querying at scale, commonly fed by Cloudflare Logpush or a scheduled GraphQL pull from Cloudflare’s httpRequestsAdaptiveGroups dataset, so you can query bot activity by page, date, and status code without re-importing flat files.
• Cloudflare AI Crawl Control (for sites behind Cloudflare) — a managed dashboard for crawler activity and request patterns, bot verification, and directive-compliance tracking, without building your own pipeline (docs).

Step 4 — What to measure and how to read it

Crawl frequency by bot. Hits per day/week per bot name. AI bots crawl in bursts, not steady streams. In WISLR’s 48-day CDN-log case study, GPTBot was absent for weeks, then hit 187 requests in a single week — 152 of them in a three-minute burst, peaking at 114 requests/minute (WISLR). Read frequency as pattern, not just a total.

Which pages get hit — and which important ones don’t. Sort by request count. Busby notes AI crawlers commonly stay shallow — it’s common to see them limited to top-level pages: the homepage, primary navigation, and a small number of high-level URLs (SEL) — dropping off sharply for deep pages even when those deep pages matter most for citation. A deep page that never appears in the logs can’t be retrieved.

Raw-HTML dependency — measure, don’t universalize. Provider documentation does not give one shared JavaScript-renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. contract for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and other agents. One observed signature is still useful: in WISLR’s sample, ChatGPT-User fetched HTML only — zero requests for images, CSS, or JS files — while Googlebot and OAI-SearchBot also pulled images (WISLR). If your logs show AI bots hammering pages whose raw HTML is a near-empty JS shell, that’s a checkable dependency risk, not proof that every provider always behaves this way. Compare delivered HTML with outcomes or provider-specific fetch tests. (For the rendering fundamentals, see JavaScript SEOMaking sure search engines can crawl, render, and index content that depends on JavaScript..)

Status-code / blocked breakdown. Watch: 200 (success), 304 (not modified — good, efficient re-crawl), 404 (broken links the bot followed), and 403/429 (blocked / rate-limited). Busby specifically calls out that log files surface where crawlers encounter issues, including 403 responses (blocked requests) and 429 responses (rate limiting) (SEL). Check whether that blocking is intentional or an accident.

robots.txt and llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. requests as their own signal. Cross-reference which bots request /robots.txt at all before crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. — in WISLR’s sample, GPTBot and Meta-WebIndexer never checked it across 48 days — and whether disallowed paths get hit anyway. WISLR also logged zero requests to /llms.txt from any AI bot over those 48 days (WISLR), consistent with the roughly 97%-unread finding covered in the AI crawlers article. Don’t expect llms.txt requests in your logs as proof it “works.”

Does more crawling mean more citations? Be honest.

No established data supports “crawl more, get cited more.” Retrieval is a necessary precondition for citation but nowhere near sufficient — pages get crawled constantly and never cited, whether because of client-side rendering, paywalls, thin or duplicate contentThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling., or simply losing to a better source in the model’s retrieval/ranking step. The load-bearing framing lives in the LLM visibility article: retrieved → mentioned → cited, and log analysis observes only that first stage.

The way to close the loop honestly is to pair the crawl-input side (your logs) with the citation-output side. Bing’s AI Performance reportThe Google Search Console report that shows how your site actually performed in Google Search, built from real impressions and clicks. It reports four metrics — clicks, impressions, average CTR, and average position — and keeps the most recent 16 months of data. (public preview, Feb 2026) is the first official tool exposing citation data alongside its groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. queries — the key phrases the AI used when retrieving content that was referenced in AI-generated answers (Bing Webmaster Blog). Logs tell you what got taken; grounding queries and citation counts tell you what came of it. Neither alone is the whole picture — see the measurement and reportingAI search measurement and reporting is the practice of tracking and communicating how AI search features — AI Overviews, AI Mode, ChatGPT, Perplexity, Gemini, Copilot — affect a brand's visibility, traffic, and revenue. It exists because most AI-referred traffic and exposure is invisible in standard analytics. hub for how these layers stack.

The stealth-crawling wrinkle

Verification isn’t a one-time audit. Per Seer Interactive’s Clint Spaulding, once blocked, stealth crawlers can reappear under generic browser headers and unrelated IPs — these sessions look human in logs, which means session counts get inflated, bot traffic gets undercounted, and GEOGenerative Engine Optimization — visibility inside AI answer engines. segmentation becomes less trustworthy (Seer). His blunt summary: if you can’t see these stealth crawlers, you can’t measure their impact. This is exactly why log analysis needs periodic re-verification and re-baselining, not a single pass.

AI summary

A condensed take on the Advanced version:

• AI crawlerAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. log analysis = reading raw server/CDN logs for AI-botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. requests to verify, with first-party data, which bots hit you, whether they’re real, and what they got. GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. (GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.-only) and GA4 (referrer-only) don’t see it.
• Scope: this is the method. What the bots are lives in AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls.; the click side in AI traffic attributionAI traffic attribution is the practice of correctly identifying and measuring website visits that come from AI tools — ChatGPT, Perplexity, Gemini, Claude, AI Overviews, and AI browsers. It's hard because many of those tools strip the referrer header, so the visits land in your analytics as Direct traffic with no source.; the retrieved → mentioned → cited framework in LLM visibilityLLM visibility (or AI visibility) is the aggregate measure of how often and how prominently a brand or page shows up in AI-generated answers — across AI Overviews, ChatGPT, Perplexity, Copilot, and Gemini. It's the AI-search analog of organic visibility, but it's driven by different signals. (logs observe only retrieved).
• Separate verified identity from a claimed user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target.. User-agents are spoofed — HUMAN Security measured 5.7% across 16 bots; Duane Forrester self-tested 81.8% fake among his live-fetch requests (87% for GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.). Match the UA and check the source IP against an operator’s current official list when one exists (for example, openai.com/gptbot.json or Anthropic’s claude.com/crawling/bots.json). Do not invent a generic reverse-DNS fallback.
• Tools by scale: grep → Screaming Frog LFA (AI presets + verify-on-import) → ELK/Splunk → BigQuery / Cloudflare AI Crawl Control.
• Measure: crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. by bot (bursty, e.g. WISLR’s 152 GPTBot requests in 3 minutes), pages hit, raw-HTML dependency (WISLR’s ChatGPT-User sample fetched zero images/CSS/JS), status codes (403/429), and robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere./llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. requests (WISLR logged zero llms.txt hits in 48 days).
• Crawl ≠ citation. Retrieval is necessary, not sufficient — “more crawls = more citations” is unproven. Pair logs with Bing’s groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. queries to close the loop.
• Re-verify periodically — blocked stealth crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. resurface looking human.

Official documentation

Primary sources — the botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. operators’ own verification files and the platform docs.

OpenAI

• OpenAI bots / crawlers docs — user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. strings and behavior for GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot.
• Published IP lists for verification: gptbot.json, searchbot.json, chatgpt-user.json, adsbot.json.

Anthropic

• Does Anthropic crawl data from the web? — current bot names, purposes, robots controls, and the statement that an address on its published list indicates an Anthropic crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index..
• Anthropic crawler IP list — current shared verification source for ClaudeBot, Claude-SearchBot, and Claude-User.

Google

• Crawl Stats report — Google’s own crawl-activity view (GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. only; it does not cover third-party AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls., which is why raw logs are needed for them).

Cloudflare

• AI Crawl Control — managed dashboard for AI-crawler activity, bot verification, and directive compliance.

Bing / Microsoft

• Introducing AI Performance in Bing Webmaster Tools (Public Preview) — the citation-outcome counterpart to log analysis: Total Citations, Average Cited Pages, GroundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. Queries, and page-level citation activity.

Quotes from the source

On-the-record statements from operators and named practitioners.

Anthropic — current scope

• Anthropic’s current support page distinguishes ClaudeBot, Claude-SearchBot, and Claude-User, documents how its robots controls apply, and says an address in its published list indicates an Anthropic crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. Source

Lauren Busby, Co-Founder, Trebletree — Search Engine Land

• “Log filesLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. are the closest thing to that missing layer. They don’t summarize or interpret activity. They record it — every request, every URL, every crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index..”
• “Tools like Screaming Frog Log File Analyzer make it possible to process that data quickly.”
• On AI-crawler depth: “It’s common to see them limited to top-level pages – the homepage, primary navigation, and a small number of high-level URLs.”
• On what surfaces: “Log files also surface where crawlers encounter issues. This includes: 403 responses (blocked requests). 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. responses (rate limiting).”
• On retention: “A scheduled SFTP job – whether built in a workflow tool like n8n, or scripted – is enough to turn a short retention window into something you can actually analyze over time.”
• The honest limit: “Log files show you what reached your site. They don’t always show you what tried to.” Source

Duane Forrester — Search Engine Journal

• “Of 33 requests carrying one of those live-fetch names. Six came from an IP the vendor publishes. Twenty-seven did not. That is an 81.8% spoof rate among the requests I could check.”
• “The real check is not complicated. The major operators publish the actual IP addresses their bots use, as plain files you can open right now, and a request is legitimate only if the name matches and the address sits inside the published list.”
• On GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., for scale: “Of 799 requests carrying the GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. name, only 107 came from a verified Google address. The other 692, roughly 87%, were not Google.”
• His call to action: “Do not take my numbers; take the method… Pull a date range, match the names, verify the IPs against the published lists, and find your real fraction.” Source

Clint Spaulding, Sr. Manager Technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own., Seer Interactive

• “Once blocked, stealth crawlers can reappear under generic browser headers and unrelated IPs.”
• “These sessions look human in logs. That means session counts get inflated, bot traffic gets undercounted, and GEOGenerative Engine Optimization (GEO) is the practice of optimizing content and brand presence so AI-powered search engines and assistants — Google AI Overviews, ChatGPT, Perplexity — cite, recommend, or mention you when generating answers. Google's position is that it's still SEO. segmentation becomes less trustworthy.”
• “If you can’t see these stealth crawlers, you can’t measure their impact.” Source

Screaming Frog — Log File Analyser tutorial (product doc)

• On verify-on-import: “Search engine bots are often spoofed, and this performs a lookup against publicly confirmed IP lists to confirm they are genuine.”
• “If you see high request volumes from IPs that don’t verify, you’re likely dealing with fake bot traffic that should be blocked at server level.” Source

Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. — AI Performance reportThe Google Search Console report that shows how your site actually performed in Google Search, built from real impressions and clicks. It reports four metrics — clicks, impressions, average CTR, and average position — and keeps the most recent 16 months of data. (Feb 2026 public preview)

• GroundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. Queries: “Shows the key phrases the AI used when retrieving content that was referenced in AI-generated answers.”
• Total Citations: “Shows the total number of citations that are displayed as sources in AI-generated answers during the selected time frame.” Source

”Is this AI-bot request real, and what do I do about it?”

Walk a single suspicious log line — or a whole botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.’s traffic — through this. It’s the verification logic from Step 2, turned into a flow.

AI-crawler log-analysis checklist

A repeatable pass, from pulling logs to reading them:

• Logs are being captured with the fields you need: timestamp, client IP, user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., request path, status code (bytes + referrer help).
• You’re pulling from the right layer — CDN/edge logs if you’re behind Cloudflare/Fastly (a lot of botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. traffic never reaches origin).
• A scheduled pull (SFTP/n8n/script) beats the retention window so you have history, not just the last few days.
• Every AI-bot request is verified: user-agent matched and source IP checked against the operator’s published list (reverse-DNS as backup).
• Spoofed/unverified requests are excluded from your AI-crawl metrics, not counted as the named bot.
• Crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. baselined per bot (watch for bursty vs. steady patterns).
• Most-fetched pages identified — and important deep pages confirmed to be getting crawled at all.
• Crawl-vs-render checked: are AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. pulling only HTML on pages whose content is JS-rendered? (Record this as a raw-HTML dependency test; provider behavior is agent-specific and may be undocumented.)
• Status codes reviewed: 200/304 healthy; 404 broken links; 403/429 confirmed intentional or fixed.
• robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. request presence checked (do bots fetch it before crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.?); no surprise hits on disallowed paths.
• Paired with citation-side data (Bing groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. queries / GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. AI features) before drawing any “is it working” conclusion.
• Re-verification scheduled — this is not a one-time audit (stealth crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. resurface looking human).

AI-crawler log-analysis cheat sheet

Verification files (bookmark these)

Status codes to watch

Verify, don’t trust

• User-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. alone = unverified. Match the UA and provider-published verification data where available; otherwise preserve the uncertainty.
• Spoof rates seen in the wild: 5.7% (HUMAN, 16 bots) → 81.8% (Forrester’s live-fetch logs); 87% fake GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. in the same audit.

Raw-HTML dependency tell

• A bot fetching only HTML with zero .js/.css/image requests is an observed fetch pattern. On a JS-dependent page it is a risk signal, not a universal capability proof.

Fast facts

• Crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. ≠ citation likelihood. Retrieval is necessary, not sufficient.
• Use documented robots tokens for crawl policy; keep network blocking as a separate abuse/security control.

Scripts for AI-crawler log analysis

The AI crawlers articleAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. has the basic “count the botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. hits” snippet. These go further: extraction, verification, status-code breakdown, and crawl-vs-render detection.

1. Extract every AI-bot line (grep + regex)

macOS / Linux — one alternation over the common AI user-agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target.:

# Pull all AI-bot requests from a combined-format access log
grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Applebot|Amazonbot|Bytespider|CCBot|Meta-ExternalAgent' \
  access.log > ai-bots.log

# Count requests per bot (which token, how many hits)
grep -Eoi 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider' \
  access.log | sort | uniq -c | sort -rn

Windows PowerShell:

Select-String -Path .\access.log -Pattern 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot|CCBot|Bytespider' |
  ForEach-Object { ($_ -match '(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot|CCBot|Bytespider)') | Out-Null; $Matches[1] } |
  Group-Object | Sort-Object Count -Descending | Select-Object Count, Name

2. Verify a claimed GPTBot IP against OpenAI’s published list

User-agent alone proves nothing — check the IP. This pulls OpenAI’s list and tests whether an IP from your logs falls inside any published CIDR:

# Requires jq and (for CIDR math) grepcidr — brew/apt install both
IP="203.0.113.45"   # the IP from your log line
curl -s https://openai.com/gptbot.json \
  | jq -r '.prefixes[].ipv4Prefix // .prefixes[].ipv6Prefix' \
  | while read -r cidr; do
      echo "$IP" | grepcidr "$cidr" >/dev/null 2>&1 && echo "VERIFIED in $cidr"
    done
# No output = the IP is NOT in OpenAI's published range → treat as spoofed.

For Anthropic, fetch the current https://claude.com/crawling/bots.json list and adjust the jq path to its documented shape. If another operator does not publish a current verification feed, retain the result as a claimed user-agent.

3. Reverse + forward DNS fallback (off-list IPs)

Use this only when the named provider publishes an expected hostname suffix and the reverse-plus-forward verification procedure. It is not a generic proof of bot identity:

IP="203.0.113.45"
HOST=$(host "$IP" | awk '/pointer/ {print $NF}' | sed 's/\.$//')
echo "PTR: $HOST"
host "$HOST" | grep -q "$IP" && echo "FORWARD-CONFIRMED" || echo "MISMATCH → suspect"

4. Status-code breakdown per bot

Spot 403/429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. blocks and 404s the bots are hitting:

# For GPTBot: tally status codes (combined log format; $9 is the status)
grep -i 'GPTBot' access.log | awk '{print $9}' | sort | uniq -c | sort -rn
# → e.g. "812 200 / 47 404 / 15 403" — the 403s are worth investigating

5. Raw-HTML dependency signal — what file types is the bot fetching?

If a claimed bot only ever pulls .html// and never .js/.css/images, and your content is JS-rendered, that is a dependency signal to investigate:

# What extensions is ChatGPT-User actually requesting?
grep -i 'ChatGPT-User' access.log \
  | awk '{print $7}' \
  | grep -oE '\.(html?|js|css|png|jpe?g|webp|svg|woff2?)(\?|$)' \
  | sort | uniq -c | sort -rn
# All HTML, zero subresources = compare the raw body with provider-specific outcomes.

6. Chrome DevTools Console — spot AI fetchers on a live page

Not a log query, but a quick check of what a page ships in raw HTML vs. after JS (what an AI botAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. would and wouldn’t see). Paste into the Console:

// Compare rendered text length to what's in the initial HTML source.
// A big gap means most content depends on JS; provider behavior must be verified separately.
(async () => {
  const raw = await (await fetch(location.href, { cache: "no-store" })).text();
  const rawText = new DOMParser().parseFromString(raw, "text/html").body.innerText.trim().length;
  const renderedText = document.body.innerText.trim().length;
  console.log({ rawText, renderedText, jsDependentRatio: +(1 - rawText / renderedText).toFixed(2) });
})();
// jsDependentRatio near 1 = almost all content is JS-injected; flag dependency, not invisibility.

7. Bookmarklet — open every bot’s IP list at once

Drag this as a bookmark to open the verification files in tabs for a quick lookup:

javascript:(function(){["https://openai.com/gptbot.json","https://openai.com/searchbot.json","https://openai.com/chatgpt-user.json","https://claude.com/crawling/bots.json"].forEach(u=>window.open(u,"_blank"));})();

SOP: monthly AI-crawler log review

A recurring procedure to run on a cadence (monthly for most sites; weekly if AI traffic is material to you). It produces a comparable baseline each time.

Prep

1. Pull the log window since your last review from the right layer (CDN/edge logs if you’re behind a CDN, origin logs otherwise). Confirm the scheduled SFTP/n8n pull actually ran — gaps here silently break trend lines.
2. Load into your tool of record (Screaming Frog LFA for periodic imports; BigQuery/ ELK if you’re on a pipeline).

Verify (never skip) 3. Refresh each current provider-published verification source (gptbot.json, searchbot.json, chatgpt-user.json, adsbot.json, and Anthropic’s claude.com/crawling/bots.json) — these change. 4. Enable verify-on-import (Screaming Frog) or run the IP-in-CIDR check (Scripts tab) against every AI-botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. request. 5. Split traffic into verified and unverified. Report the unverified fraction — that’s your spoof rate this period. A jump is a finding on its own.

Measure (verified traffic only) 6. Crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. per bot vs. last period — note new bots, disappeared bots, and bursts. 7. Top-fetched pages; confirm priority/deep pages are being crawled at all. 8. Raw-HTML dependency spot check on 2–3 JS-heavy templates (are bots pulling only HTML?). 9. Status-code breakdown per bot; investigate any new 403/429/404 clusters. 10. robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. / llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. request presence and any disallowed-path hits.

Report & act 11. Log the period’s numbers to a running sheet (verified hits by bot, spoof %, top pages, error clusters) so you have trend, not snapshots. 12. Pair with citation-side data (Bing groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. queries / GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. AI features) before concluding anything about visibility. 13. File concrete fixes: intentional-vs-accidental blocks, JS-render gaps, broken links, precise user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. rules for confirmed impersonators.

Cadence note: re-run verification every period — don’t cache a “known-good” IP list. Stealth crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. resurface under new IPs and generic browser UAs.

Playbook: “GPTBot (or ClaudeBot) is getting 403s / 429s and I don’t know if that’s on purpose”

A linear runbook for the common incident — a verified AI botAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. is hitting errors in your logs. Work it in order.

Step 1 — Confirm it’s the real botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. first. Before anything else, use the named provider’s current official verification method. If none exists, the identity remains unverified; do not silently promote it to genuine or demote it to spoofed. The 403 may still be your WAF doing its job.

Step 2 — Identify the exact response and where it’s coming from. Pull the status-code breakdown for that bot (Scripts tab). Is it 403 (blocked), 429 (rate-limited), or 503? Note whether the block is at your CDN/WAF, your server config, or robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.-adjacent rules.

Step 3 — Decide: is the block intentional?

• You meant to block this bot (e.g. a training crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. you’ve opted out of) → the 403 is working as designed. Keep the documented robots policy separate from any network-level abuse/security rule. Done.
• You did NOT mean to block it (e.g. it’s OAI-SearchBot / PerplexityBot and you want AI-search visibility) → continue.

Step 4 — Find the accidental rule. Common culprits: an over-broad WAF “bot” rule, a rate-limit threshold too low for a bursty crawler (recall WISLR’s 114 requests/minute GPTBot spike — a per-minute cap can trip on legitimate bursts), a robots.txt disallow you forgot, or a geoGenerative Engine Optimization (GEO) is the practice of optimizing content and brand presence so AI-powered search engines and assistants — Google AI Overviews, ChatGPT, Perplexity — cite, recommend, or mention you when generating answers. Google's position is that it's still SEO./ASN block catching the operator’s ranges.

Step 5 — Fix precisely. Whitelist the verified bot by user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. + published IP range, or raise the rate-limit ceiling for that verified bot specifically. Don’t loosen protection for everyone.

Step 6 — Verify the fix in the logs. After deploying, re-pull the logs for that bot. You want the 403/429 cluster to turn into 200/304 on the pages you care about.

Step 7 — Re-baseline and watch. Note the change in your running sheet. Recheck next period — and stay alert for the bot resurfacing under a different UA if you did intend to block it (stealth crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.).

Log-analysis mistakes that produce wrong conclusions

Each is a belief from the wild, why it’s wrong, and what to do instead.

Myth: “If my robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. blocks a botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., it won’t be in my logs / it’s not a problem.” Why it’s wrong: robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. is a request, not enforcement. User-triggered fetchers explicitly carve themselves out, and some crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. have been reported bypassing blocks (Perplexity stealth crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., Cloudflare Aug 2025). The block might be ignored entirely. Do instead: Use the logs to check whether the rule is honored — look for disallowed-path hits and whether the bot even requests /robots.txt.

Myth: “A GPTBot / ClaudeBot user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. in my logs means it’s really OpenAI / Anthropic.” Why it’s wrong: User-agents are trivially spoofed — HUMAN Security measured 5.7% fake across 16 bots, and Duane Forrester foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. 81.8% fake among his own live-fetch requests. Do instead: Match the user-agent and use the operator’s current official verification method where one exists; otherwise label the identity unverified.

Myth: “More AI-crawler hits = more likely to get cited.” Why it’s wrong: No established causal/correlational data supports it. Retrieval is necessary but not sufficient — heavily crawled pages go uncited constantly. Do instead: Treat logs as visibility into the retrieved stage only, and pair them with citation-side data (Bing groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. queries, GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. AI features).

Myth: “Every AI crawlerAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. has the same renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. behavior.” Why it’s wrong: providers do not publish one shared JavaScript/subresource contract, and observed behavior differs by agent and sample. Do instead: Compare the raw response with the rendered page, inspect subresource requests by named bot, and label the result observed rather than universal.

Myth: “AI bots are at least checking for my llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it..” Why it’s wrong: WISLR’s 48-day sample recorded zero /llms.txt requests from any AI bot, consistent with the ~97%-unread finding in the AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. article. Do instead: Don’t treat llms.txt requests as expected validation; measure what bots actually fetch.

Myth: “Network blocking and robots policy are the same control.” Why it’s wrong: robots communicates crawl preferences; a firewall enforces network access and can affect unrelated traffic. Do instead: Use the documented user-agent rule for policy, and reserve network blocking for separately justified abuse/security response.

Real cases

Duane Forrester — self-tested spoof rates on his own logs. Forrester ran this exact method on his site and published the numbers. Among 33 live-fetch requests, only 6 came from a vendor-published IP — an 81.8% spoof rate; among 799 GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.-named requests, only 107 verified — roughly 87% fake (SEJ). Before: trusting the user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., his “AI assistant” traffic looked real. After: matching names to published IP lists, most of it was impersonation. Takeaway: the method matters more than his specific numbers — as he says, pull your own date range and find your real fraction.

WISLR — 48 days of CDN logs, the crawl-vs-render signature. Tony Castillo analyzed 288,566 CDN log lines (12,099 AI/botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. requests) over 48 days (WISLR). Concrete findings: GPTBot was absent for weeks then burst 152 requests in three minutes (114 req/min peak); ChatGPT-User fetched zero images, CSS, or JS — pure HTML extraction; and there were zero /llms.txt requests across the whole window. Before: you’d assume steady crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and JS-aware bots. After: the logs show bursty, HTML-only behavior — n=1, one site’s data, but a vivid picture of what a real analysis surfaces. Takeaway: the zero-JS fetch pattern is the direct evidence behind the crawl-vs-render check.

Cloudflare — the crawl-to-referral ratio, at network scale. Cloudflare’s aggregate data shows how little crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. translates to traffic: for every visitor Anthropic refers back to a website, its crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. have already visited tens of thousands of pages (Cloudflare). Before: the intuition that heavy crawling means engagement. After: at network scale the ratio is lopsided — training now drives the majority of AI-bot activity, and training bots aren’t meant to send traffic back at all (Cloudflare). Takeaway: a page crawled heavily with zero referral is normal, not a sign of success — which is exactly why crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. isn’t a citation predictor.

Ready-to-use AI prompts

Copy-paste prompts for using an LLMA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). to speed up AI-crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. log analysis. Always sanity-check the output against the raw logs — LLMs hallucinate, and a verification pipeline that trusts a hallucinated IP match is worse than none.

Draft a verification-aware log parser

Write a Python script that parses combined-format Nginx access logs and, for each
request whose user-agent matches a known AI bot (GPTBot, OAI-SearchBot,
ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, CCBot,
Bytespider), does the following:

1. Extract timestamp, client IP, request path, status code, user-agent.
2. Fetch and cache OpenAI's current IP lists (gptbot.json, searchbot.json,
   chatgpt-user.json, adsbot.json) and Anthropic's current bots.json list.
3. Mark an OpenAI or Anthropic request VERIFIED only if the user-agent matches AND
   the client IP falls inside the matching provider-published ranges; mark providers
   without an official method UNVERIFIED, not spoofed.
4. Output two CSVs: verified requests and unverified ("spoofed") requests.
5. Print a summary: verified vs. unverified count per bot, and the top 20 fetched
   paths (verified only).

Do NOT count unverified requests in any per-bot metric. Add clear comments.

Summarize a status-code + crawl-vs-render report

I'll paste a table of AI-bot log data (columns: bot, path, status_code,
file_extension). Produce:
- A per-bot status-code breakdown, flagging any 403/429/404 clusters.
- A raw-HTML dependency read: for each bot, the ratio of HTML requests to
  JS/CSS/image requests, and a note on whether the bot appears to fetch only HTML
  in this sample. Treat HTML-only on a JS-rendered page as a dependency risk, not
  proof of a universal no-JavaScript capability.
Keep every conclusion tied to a number from the data — do not infer beyond it.

DATA:
[paste]

Triage a suspicious IP

A request in my logs claims to be [BOT NAME] from IP [IP ADDRESS]. Walk me through
verifying it: which operator IP-list file to check, how to test whether the IP is
in range, and the reverse+forward DNS fallback if it's not on a published list.
Tell me explicitly what result means "verified" vs. "treat as spoofed." Do not
guess whether this specific IP is legitimate — give me the steps to check.

Tools for AI-crawler log analysis

Roughly free/ground-truth → paid/managed:

• grep / PowerShell — fastest way to a botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-hit count and a verification-aware filter over a raw access log. Zero setup; see the Scripts tab.
• Screaming Frog Log File Analyser — desktop importer with built-in AI-bot presets and a “Verify Bots When Importing” toggle that checks against publicly confirmed IP lists. Response Codes, User Agents, URLs (sort by Num Events), and IPs tabs. Best for periodic imports.
• ELK Stack (Elasticsearch / Logstash / Kibana) or Splunk — continuous ingestion, dashboards, and alerting once a desktop import is too slow or you want ongoing monitoring.
• BigQuery — long-term retention and SQL at scale, commonly fed by Cloudflare Logpush or a scheduled GraphQL pull from httpRequestsAdaptiveGroups.
• Cloudflare AI Crawl Control — for sites behind Cloudflare: managed crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-activity, bot-verification, and directive-compliance views without your own pipeline.
• Google Search Console — Crawl Stats — not for AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. (GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. only), but the model for the by-crawler + by-response- code breakdown you’re rebuilding for AI bots from raw logs.
• Bing Webmaster Tools — AI Performance — the citation-outcome counterpart: pair its groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. queries and citation counts with your crawl-input logs to close the loop.

Claimed AI-bot traffic spikes overnight

Symptom: requests with a famous crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. jump suddenly. Likely cause: spoofing, monitoring traffic, or a real crawl change. Fix: verify source IPs against the operator’s current published method before attributing the traffic, then segment by ASN, path, status, and time.

Logs show crawling but the content is never cited

Symptom: verified bots fetch pages without visible citation gains. Likely cause: crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is only eligibility evidence; retrieval and answer selection are separate. Fix: confirm the bot received substantive HTML, then evaluate indexability, passage quality, query fit, and off-site corroboration without treating crawl count as rank.

Every request appears to return 200

Symptom: missing URLs and blocked content are recorded as success. Likely cause: an app shell, CDN rule, or custom error page returns a soft 404A soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing.. Fix: sample response bodies and final headers, then repair status handling rather than trusting status alone.

Verified bots receive an empty shell

Symptom: a browser renders content, but log-matched fetches receive little useful HTML. Likely cause: the page depends on client-side JavaScript the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. does not execute. Fix: compare raw and rendered output and serve critical content and links in HTML through SSR, static renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., or another reliable delivery strategy.

Metrics for AI-crawler logs

Resources worth your time

My related writing

• How to Do an SEO Log File Analysis (Ahrefs, reviewed by Patrick Stox & Michal Pecánek) — the pre-AI-era template this article is the AI-specific sequel to: what to measure, tools, and botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. verification.
• What is Log File Analysis? (Ahrefs glossary) — the definitional companion.
• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — my Cloudflare Radar analysis of AI-bot crawl share.
• The AI Bots That ~140 Million Websites Block the Most — robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. block-rate data across the open web.
• 80% of Our AI Search Traffic Goes to Our Homepage, Product Pages, and Free Tools — page-type-level analysis of AI activity, the analog to “which pages get crawled.”

My speaking

• How Search Works (SlideShare) — my walkthrough of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and ranking, useful background for reading what bots do in your logs. (Standing disclaimer applies: this is my understanding of the systems, not a guarantee it’s 100% complete or accurate.)

From around the industry

• Why log file analysis matters for AI crawlers and search visibility — Lauren Busby (Trebletree), Search Engine Land: the “logs are the missing layer,” retention-window, and 403/429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. framing.
• 81.8% Of My “AI Assistant” Traffic Was Fake. The Googlebot Number Was Worse — Duane Forrester, Search Engine Journal: the standout first-party verification case study and method.
• Perplexity, Stealth AI Crawling, and the Impacts on GEO and Log File Analysis — Clint Spaulding, Seer Interactive: why blocked bots resurface looking human.
• How to Monitor AI Bots in the Log File Analyser — Screaming Frog: the practical tool walkthrough with verify-on-import.
• The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals — Cloudflare: network-level crawl-to-referral ratios and the training-vs-search split.
• Does Anthropic crawl data from the web? — Anthropic’s current bot-purpose and robots-control documentation.
• OpenAI bots / crawlers docs — user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. strings and the published IP files for verification.

Stats worth citing

• Spoof rate — 5.7% across 16 AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls.. HUMAN Security’s two-week analysis of traffic claiming to be one of 16 well-known AI crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. roughly 1 in 18 requests was spoofed (vendor-sourced, relayed via SEJ).
• Spoof rate — 81.8% in one practitioner’s own logs. Duane Forrester’s self-run audit found 27 of 33 live-fetch requests came from IPs the vendors don’t publish; his GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. number was ~87% fake (SEJ).
• Crawl-to-referral, ClaudeBot vs. OpenAI. Cloudflare’s data (week of May 25– June 1, 2026) put ClaudeBot at ~11,122 pages crawled per referral and OpenAI at ~857:1, versus GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. around 5:1 (Cloudflare).
• Training drives most AI-bot activity. Per Cloudflare, training now drives nearly 80% of AI-bot activity, up from 72% a year earlier — context for why heavy crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. with zero referral is normal (Cloudflare).
• One real log window: 288,566 lines, 12,099 bot requests, 48 days. WISLR’s case study — with GPTBot’s 152-requests-in-3-minutes burst and ChatGPT-User’s zero image/CSS/JS fetches (WISLR).

Test yourself: AI Crawler Log Analysis

Five quick questions on pulling and reading AI-botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. logs. Pick an answer for each, then check.

AI Crawler Log Analysis

AI crawler log analysis is the practice of pulling raw server or CDN access logs and examining them for requests from AI bots — training crawlers, AI-search indexers, and user-triggered fetchers — to verify with first-party data which bots actually hit your site, whether they're real or spoofed, and what they got.

Related: Log file analysis, AI Crawlers

AI Crawler Log Analysis

AI crawlerAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. log analysis is the AI-search-era update to classic SEO log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened.. Instead of auditing GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. for crawl budgetThe number of URLs an engine will crawl in a timeframe., you pull your raw server (Apache/Nginx) or CDN access logs and examine them specifically for requests from AI botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — training crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. (GPTBot, ClaudeBot), AI-search indexers (OAI-SearchBot, PerplexityBot, Applebot), and user-triggered fetchers (ChatGPT-User, Perplexity-User) — to answer questions robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. files and analytics dashboards can’t.

Logs are the only first-party ground truth for four things: which of your pages an AI bot actually fetched, how often, whether the requester is the real bot or a spoof, and what it got — your rendered content or a JavaScript-empty shell — plus the response codes it hit. Neither Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. (GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.-only) nor GA4 (which only sees clicks that carry a referrer) shows any of this.

Two things make it different from ordinary log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened.. First, verification matters more: AI-bot user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. strings are spoofed at a documented, non-trivial rate, so a GPTBot string alone proves nothing — you match the user-agent and confirm the source IP against the operator’s published range. Second, you have to check crawl vs. render: unlike Googlebot, the major AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. don’t execute JavaScript, so a page can be crawled constantly and still deliver nothing if its content is client-side rendered.

Log analysis observes only the retrieved stage of the retrieved → mentioned → cited funnel. Being crawled is necessary for a citation but nowhere near sufficient, so treat crawl-frequency data as visibility into one input, not a citation predictor.

Related: Log file analysis, AI Crawlers