AI Crawler Log Analysis

TL;DR — AI crawler log analysis means looking at your server’s raw access logs to see which AI bots — like OpenAI’s GPTBot or Anthropic’s ClaudeBot — actually visited your site, how often, and what they got. Your logs are the only place that records this. Google Search Console doesn’t show AI bots, and Google Analytics only sees people who click through, not the bots themselves. The catch: lots of traffic claims to be a famous AI bot but isn’t, so you can’t trust the name alone.

What a log file is

Server access logs record requests that reached the server, including user agent, time, path, and response details when configured. Evidence for this claim Server access logs record requests and can include fields such as request path, status, referrer, and user agent. Scope: Apache HTTP Server logging; available fields depend on server configuration. Confidence: high · Verified: 2026-07-14 Apache: Log files Vendor crawler documentation identifies user agents, but a string alone does not prove the requester is genuine. Evidence for this claim OpenAI publishes user-agent tokens and controls for several of its crawlers. Scope: OpenAI's documented crawlers; a user-agent string is not proof of downstream training, retrieval, citation, or use. Confidence: high · Verified: 2026-07-14 OpenAI: Crawlers

Every time anything — a person’s browser, Googlebot, GPTBot — asks your server for a page, your server writes a line to a log file. Each line records the time, the visitor’s IP address, what they asked for, the “user-agent” (a label the visitor uses to identify itself, like GPTBot), and the response code your server sent back (200 for success, 404 for not found, and so on).

AI crawler log analysis is just reading those lines and filtering them down to the AI bots. It answers questions you literally can’t answer any other way:

• Did an AI bot actually visit my site at all?
• Which pages did it fetch, and which important pages did it ignore?
• Was it the real bot, or something pretending to be it?
• What did it get — my actual content, or an error page?

If you want the full cast of AI bots and what each one is for (training bots vs. search bots vs. the “fetch this page right now” bots), that’s a separate topic — see AI crawlers. This page is about the how: pulling the logs and reading them.

Why your dashboards don’t cover this

• Google Search Console has a Crawl Stats report, but it’s only about Googlebot. It shows nothing about GPTBot, ClaudeBot, or PerplexityBot.
• Google Analytics (GA4) only records visitors who click a link to your site and arrive with a “referrer.” A crawler grabbing your page in the background never shows up there at all.

So the logs are the ground truth. Nothing else sees the bot.

The one thing beginners miss

A GPTBot label in your logs does not prove it was really OpenAI. Anyone can put any user-agent string they want on a request — it’s like writing a return address on an envelope. Lots of scrapers slap a famous bot’s name on their traffic to look legitimate. The only way to be sure is to check the visitor’s IP address against the list of real addresses the AI company publishes. That verification step is the heart of doing this properly, and the Advanced tab walks through it.

TL;DR — Pull your Apache/Nginx or CDN access logs, then do four things: verify (match the user-agent and confirm the IP against each operator’s published list — spoof rates run from HUMAN Security’s 5.7% to Duane Forrester’s self-tested 81.8%); choose a tool by scale (grep → Screaming Frog Log File Analyser → ELK/Splunk → BigQuery/Cloudflare); measure crawl frequency by bot, which pages get hit, crawl-vs-render, and status codes; and interpret honestly — retrieval is necessary but not sufficient, so “more crawling = more citations” is unproven. This is the method sibling to AI crawlers (which owns what the bots are), AI traffic attribution (the click side), and LLM visibility (the retrieved → mentioned → cited framework this observes the first stage of).

What this article owns — and what it doesn’t

Logs demonstrate requests, not whether content was used for training, retrieval, or an answer. Evidence for this claim Server access logs record requests and can include fields such as request path, status, referrer, and user agent. Scope: Apache HTTP Server logging; available fields depend on server configuration. Confidence: high · Verified: 2026-07-14 Apache: Log files Verify bots using provider-published mechanisms where available and treat attribution as bounded evidence. Evidence for this claim OpenAI publishes user-agent tokens and controls for several of its crawlers. Scope: OpenAI's documented crawlers; a user-agent string is not proof of downstream training, retrieval, citation, or use. Confidence: high · Verified: 2026-07-14 OpenAI: Crawlers

The sibling AI crawlers article already owns the bot-by-bot table, the three-category taxonomy (training / AI-search / user-triggered fetch), the Google-Extended-is-a-token-not-a-bot distinction, and the robots.txt recipes. I’m not re-deriving any of that here. AI traffic attribution owns the GA4/referrer side — what the bot sent back. This article is the opposite side of the funnel: what the bot took. And LLM visibility owns the retrieved → mentioned → cited framework; log analysis is how you observe the retrieved stage specifically, and nothing past it.

This is the AI-era sequel to classic SEO log file analysis. When I reviewed Ahrefs’ How to Do an SEO Log File Analysis, the dimensions were crawl frequency, URLs crawled, status codes, and bot verification against Google’s published IPs. Same skeleton here — but pointed at a bot population that mostly doesn’t render JavaScript, gets impersonated constantly, and crawls in erratic bursts rather than a steady stream.

Step 1 — Get your logs

Your logs live in one of two places, or both:

• Origin server logs. Apache (access.log), Nginx (access.log), or your app server. Each line has, at minimum: timestamp, client IP, request method + path, status code, bytes, referrer, and user-agent.
• CDN / edge logs. If you’re behind Cloudflare, Fastly, Akamai, etc., a lot of bot traffic is answered at the edge and may never reach your origin — so the edge log is the fuller record. Cloudflare exposes this via Logpush (and a GraphQL API); Fastly via real-time log streaming.

The fields you actually need for AI-bot analysis: timestamp, client IP, user-agent, request path, status code, and ideally bytes and referrer.

The practical problem is retention. Many hosts keep only a few days of logs. Lauren Busby’s Search Engine Land piece names the fix directly — a scheduled pull turns a short window into something analyzable over time: a scheduled SFTP job, built in a workflow tool like n8n or scripted, is enough to turn a short retention window into something you can actually analyze over time (Busby, SEL). Set that up before you need the data.

One honest limit to keep in mind from the start: as Busby puts it, log files show you what reached your site, but they don’t always show you what tried to (SEL) — requests blocked or answered upstream may not appear at all.

Step 2 — Verify before you trust the user-agent

This is the step that separates AI-bot log analysis from the classic version, and the one most competing how-tos skip. User-agent strings are trivially spoofed. Two independent data points on how bad it gets:

• HUMAN Security analyzed two weeks of traffic claiming to be one of 16 well-known AI crawlers and found 5.7% of it was spoofed — roughly 1 in 18 requests (reported via SEJ).
• Duane Forrester ran the same check on his own logs and found far worse. Of 33 requests carrying a live-fetch name, six came from an IP the vendor publishes and twenty-seven did not — an 81.8% spoof rate among the requests he could check (SEJ). His Googlebot number was worse still: of 799 requests carrying the Googlebot name, only 107 came from a verified Google address — the other roughly 87% were not Google (SEJ).

Treat those as directional from different samples and methods, not one universal number. More importantly, do not generalize one provider’s verification method to every bot. Some operators publish address ranges; others document user-agent tokens without a current public range or DNS-verification contract (SEJ).

The verification method:

1. Match the user-agent string. GPTBot identifies itself as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot; OAI-SearchBot and ChatGPT-User carry their own tokens (OpenAI bots docs).
2. Use the provider’s current official verification method when one exists. OpenAI ships openai.com/gptbot.json, searchbot.json, and chatgpt-user.json. Anthropic’s current crawler documentation says an address in its published list indicates a crawler is coming from Anthropic. Use that current provider-specific method; do not generalize it into a universal bot-verification rule.
3. Do not invent a fallback. Reverse-plus-forward DNS is valid for a provider only when that provider publishes the hostname pattern and verification procedure. A generic PTR match proves control of DNS, not the claimed bot identity. Screaming Frog’s Log File Analyser can apply publicly confirmed lists where available; its verify-on-import feature performs a lookup against publicly confirmed IP lists to confirm bots are genuine (Screaming Frog).

Verification should inform precise policy and incident response. Prefer the provider’s documented robots token for a crawl policy; reserve network controls for abuse or security cases and label them separately from robots compliance.

Step 3 — Choose your tool

Scale it to the size of the job, roughly free → paid, ground-truth → managed:

• grep / PowerShell — a quick one-off count and a verification-aware filter. The AI crawlers article has the basic bot-count snippet; the Scripts tab here extends it into IP verification, a status-code breakdown, and crawl-vs-render detection.
• Screaming Frog Log File Analyser — a desktop importer with built-in AI-bot presets (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, CCBot) and a “Verify Bots When Importing” toggle. Its Response Codes tab breaks 2XX/3XX/4XX/5XX per URL, the User Agents tab shows requests and error rates per bot, the URLs tab sorts by Num Events (most-fetched pages), and the IPs tab lets you investigate suspicious sources (tutorial).
• ELK Stack (Elasticsearch / Logstash / Kibana) or Splunk — for continuous, larger-scale ingestion with dashboards and alerting, once a single-import desktop tool gets too slow or you want ongoing monitoring rather than periodic exports.
• BigQuery — for long-term retention and SQL querying at scale, commonly fed by Cloudflare Logpush or a scheduled GraphQL pull from Cloudflare’s httpRequestsAdaptiveGroups dataset, so you can query bot activity by page, date, and status code without re-importing flat files.
• Cloudflare AI Crawl Control (for sites behind Cloudflare) — a managed dashboard for crawler activity and request patterns, bot verification, and directive-compliance tracking, without building your own pipeline (docs).

Step 4 — What to measure and how to read it

Crawl frequency by bot. Hits per day/week per bot name. AI bots crawl in bursts, not steady streams. In WISLR’s 48-day CDN-log case study, GPTBot was absent for weeks, then hit 187 requests in a single week — 152 of them in a three-minute burst, peaking at 114 requests/minute (WISLR). Read frequency as pattern, not just a total.

Which pages get hit — and which important ones don’t. Sort by request count. Busby notes AI crawlers commonly stay shallow — it’s common to see them limited to top-level pages: the homepage, primary navigation, and a small number of high-level URLs (SEL) — dropping off sharply for deep pages even when those deep pages matter most for citation. A deep page that never appears in the logs can’t be retrieved.

Raw-HTML dependency — measure, don’t universalize. Provider documentation does not give one shared JavaScript-rendering contract for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and other agents. One observed signature is still useful: in WISLR’s sample, ChatGPT-User fetched HTML only — zero requests for images, CSS, or JS files — while Googlebot and OAI-SearchBot also pulled images (WISLR). If your logs show AI bots hammering pages whose raw HTML is a near-empty JS shell, that’s a checkable dependency risk, not proof that every provider always behaves this way. Compare delivered HTML with outcomes or provider-specific fetch tests. (For the rendering fundamentals, see JavaScript SEO.)

Status-code / blocked breakdown. Watch: 200 (success), 304 (not modified — good, efficient re-crawl), 404 (broken links the bot followed), and 403/429 (blocked / rate-limited). Busby specifically calls out that log files surface where crawlers encounter issues, including 403 responses (blocked requests) and 429 responses (rate limiting) (SEL). Check whether that blocking is intentional or an accident.

robots.txt and llms.txt requests as their own signal. Cross-reference which bots request /robots.txt at all before crawling — in WISLR’s sample, GPTBot and Meta-WebIndexer never checked it across 48 days — and whether disallowed paths get hit anyway. WISLR also logged zero requests to /llms.txt from any AI bot over those 48 days (WISLR), consistent with the roughly 97%-unread finding covered in the AI crawlers article. Don’t expect llms.txt requests in your logs as proof it “works.”

Does more crawling mean more citations? Be honest.

No established data supports “crawl more, get cited more.” Retrieval is a necessary precondition for citation but nowhere near sufficient — pages get crawled constantly and never cited, whether because of client-side rendering, paywalls, thin or duplicate content, or simply losing to a better source in the model’s retrieval/ranking step. The load-bearing framing lives in the LLM visibility article: retrieved → mentioned → cited, and log analysis observes only that first stage.

The way to close the loop honestly is to pair the crawl-input side (your logs) with the citation-output side. Bing’s AI Performance report (public preview, Feb 2026) is the first official tool exposing citation data alongside its grounding queries — the key phrases the AI used when retrieving content that was referenced in AI-generated answers (Bing Webmaster Blog). Logs tell you what got taken; grounding queries and citation counts tell you what came of it. Neither alone is the whole picture — see the measurement and reporting hub for how these layers stack.

The stealth-crawling wrinkle

Verification isn’t a one-time audit. Per Seer Interactive’s Clint Spaulding, once blocked, stealth crawlers can reappear under generic browser headers and unrelated IPs — these sessions look human in logs, which means session counts get inflated, bot traffic gets undercounted, and GEO segmentation becomes less trustworthy (Seer). His blunt summary: if you can’t see these stealth crawlers, you can’t measure their impact. This is exactly why log analysis needs periodic re-verification and re-baselining, not a single pass.

AI summary

A condensed take on the Advanced version:

• AI crawler log analysis = reading raw server/CDN logs for AI-bot requests to verify, with first-party data, which bots hit you, whether they’re real, and what they got. GSC (Googlebot-only) and GA4 (referrer-only) don’t see it.
• Scope: this is the method. What the bots are lives in AI crawlers; the click side in AI traffic attribution; the retrieved → mentioned → cited framework in LLM visibility (logs observe only retrieved).
• Separate verified identity from a claimed user-agent. User-agents are spoofed — HUMAN Security measured 5.7% across 16 bots; Duane Forrester self-tested 81.8% fake among his live-fetch requests (87% for Googlebot). Match the UA and check the source IP against an operator’s current official list when one exists (for example, openai.com/gptbot.json or Anthropic’s claude.com/crawling/bots.json). Do not invent a generic reverse-DNS fallback.
• Tools by scale: grep → Screaming Frog LFA (AI presets + verify-on-import) → ELK/Splunk → BigQuery / Cloudflare AI Crawl Control.
• Measure: crawl frequency by bot (bursty, e.g. WISLR’s 152 GPTBot requests in 3 minutes), pages hit, raw-HTML dependency (WISLR’s ChatGPT-User sample fetched zero images/CSS/JS), status codes (403/429), and robots.txt/llms.txt requests (WISLR logged zero llms.txt hits in 48 days).
• Crawl ≠ citation. Retrieval is necessary, not sufficient — “more crawls = more citations” is unproven. Pair logs with Bing’s grounding queries to close the loop.
• Re-verify periodically — blocked stealth crawlers resurface looking human.

Official documentation

Primary sources — the bot operators’ own verification files and the platform docs.

OpenAI

• OpenAI bots / crawlers docs — user-agent strings and behavior for GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot.
• Published IP lists for verification: gptbot.json, searchbot.json, chatgpt-user.json, adsbot.json.

Anthropic

• Does Anthropic crawl data from the web? — current bot names, purposes, robots controls, and the statement that an address on its published list indicates an Anthropic crawler.
• Anthropic crawler IP list — current shared verification source for ClaudeBot, Claude-SearchBot, and Claude-User.

Google

• Crawl Stats report — Google’s own crawl-activity view (Googlebot only; it does not cover third-party AI bots, which is why raw logs are needed for them).

Cloudflare

• AI Crawl Control — managed dashboard for AI-crawler activity, bot verification, and directive compliance.

Bing / Microsoft

• Introducing AI Performance in Bing Webmaster Tools (Public Preview) — the citation-outcome counterpart to log analysis: Total Citations, Average Cited Pages, Grounding Queries, and page-level citation activity.

Quotes from the source

On-the-record statements from operators and named practitioners.

Anthropic — current scope

• Anthropic’s current support page distinguishes ClaudeBot, Claude-SearchBot, and Claude-User, documents how its robots controls apply, and says an address in its published list indicates an Anthropic crawler. Source

Lauren Busby, Co-Founder, Trebletree — Search Engine Land

• “Log files are the closest thing to that missing layer. They don’t summarize or interpret activity. They record it — every request, every URL, every crawler.”
• “Tools like Screaming Frog Log File Analyzer make it possible to process that data quickly.”
• On AI-crawler depth: “It’s common to see them limited to top-level pages – the homepage, primary navigation, and a small number of high-level URLs.”
• On what surfaces: “Log files also surface where crawlers encounter issues. This includes: 403 responses (blocked requests). 429 responses (rate limiting).”
• On retention: “A scheduled SFTP job – whether built in a workflow tool like n8n, or scripted – is enough to turn a short retention window into something you can actually analyze over time.”
• The honest limit: “Log files show you what reached your site. They don’t always show you what tried to.” Source

Duane Forrester — Search Engine Journal

• “Of 33 requests carrying one of those live-fetch names. Six came from an IP the vendor publishes. Twenty-seven did not. That is an 81.8% spoof rate among the requests I could check.”
• “The real check is not complicated. The major operators publish the actual IP addresses their bots use, as plain files you can open right now, and a request is legitimate only if the name matches and the address sits inside the published list.”
• On Googlebot, for scale: “Of 799 requests carrying the Googlebot name, only 107 came from a verified Google address. The other 692, roughly 87%, were not Google.”
• His call to action: “Do not take my numbers; take the method… Pull a date range, match the names, verify the IPs against the published lists, and find your real fraction.” Source

Clint Spaulding, Sr. Manager Technical SEO, Seer Interactive

• “Once blocked, stealth crawlers can reappear under generic browser headers and unrelated IPs.”
• “These sessions look human in logs. That means session counts get inflated, bot traffic gets undercounted, and GEO segmentation becomes less trustworthy.”
• “If you can’t see these stealth crawlers, you can’t measure their impact.” Source

Screaming Frog — Log File Analyser tutorial (product doc)

• On verify-on-import: “Search engine bots are often spoofed, and this performs a lookup against publicly confirmed IP lists to confirm they are genuine.”
• “If you see high request volumes from IPs that don’t verify, you’re likely dealing with fake bot traffic that should be blocked at server level.” Source

Bing Webmaster Tools — AI Performance report (Feb 2026 public preview)

• Grounding Queries: “Shows the key phrases the AI used when retrieving content that was referenced in AI-generated answers.”
• Total Citations: “Shows the total number of citations that are displayed as sources in AI-generated answers during the selected time frame.” Source

”Is this AI-bot request real, and what do I do about it?”

Walk a single suspicious log line — or a whole bot’s traffic — through this. It’s the verification logic from Step 2, turned into a flow.

AI-crawler log-analysis checklist

A repeatable pass, from pulling logs to reading them:

• Logs are being captured with the fields you need: timestamp, client IP, user-agent, request path, status code (bytes + referrer help).
• You’re pulling from the right layer — CDN/edge logs if you’re behind Cloudflare/Fastly (a lot of bot traffic never reaches origin).
• A scheduled pull (SFTP/n8n/script) beats the retention window so you have history, not just the last few days.
• Every AI-bot request is verified: user-agent matched and source IP checked against the operator’s published list (reverse-DNS as backup).
• Spoofed/unverified requests are excluded from your AI-crawl metrics, not counted as the named bot.
• Crawl frequency baselined per bot (watch for bursty vs. steady patterns).
• Most-fetched pages identified — and important deep pages confirmed to be getting crawled at all.
• Crawl-vs-render checked: are AI bots pulling only HTML on pages whose content is JS-rendered? (Record this as a raw-HTML dependency test; provider behavior is agent-specific and may be undocumented.)
• Status codes reviewed: 200/304 healthy; 404 broken links; 403/429 confirmed intentional or fixed.
• robots.txt request presence checked (do bots fetch it before crawling?); no surprise hits on disallowed paths.
• Paired with citation-side data (Bing grounding queries / GSC AI features) before drawing any “is it working” conclusion.
• Re-verification scheduled — this is not a one-time audit (stealth crawlers resurface looking human).

AI-crawler log-analysis cheat sheet

Verification files (bookmark these)

Status codes to watch

Verify, don’t trust

• User-agent alone = unverified. Match the UA and provider-published verification data where available; otherwise preserve the uncertainty.
• Spoof rates seen in the wild: 5.7% (HUMAN, 16 bots) → 81.8% (Forrester’s live-fetch logs); 87% fake Googlebot in the same audit.

Raw-HTML dependency tell

• A bot fetching only HTML with zero .js/.css/image requests is an observed fetch pattern. On a JS-dependent page it is a risk signal, not a universal capability proof.

Fast facts

• Crawl frequency ≠ citation likelihood. Retrieval is necessary, not sufficient.
• Use documented robots tokens for crawl policy; keep network blocking as a separate abuse/security control.

Scripts for AI-crawler log analysis

The AI crawlers article has the basic “count the bot hits” snippet. These go further: extraction, verification, status-code breakdown, and crawl-vs-render detection.

1. Extract every AI-bot line (grep + regex)

macOS / Linux — one alternation over the common AI user-agents:

# Pull all AI-bot requests from a combined-format access log
grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Applebot|Amazonbot|Bytespider|CCBot|Meta-ExternalAgent' \
  access.log > ai-bots.log

# Count requests per bot (which token, how many hits)
grep -Eoi 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider' \
  access.log | sort | uniq -c | sort -rn

Windows PowerShell:

Select-String -Path .\access.log -Pattern 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot|CCBot|Bytespider' |
  ForEach-Object { ($_ -match '(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot|CCBot|Bytespider)') | Out-Null; $Matches[1] } |
  Group-Object | Sort-Object Count -Descending | Select-Object Count, Name

2. Verify a claimed GPTBot IP against OpenAI’s published list

User-agent alone proves nothing — check the IP. This pulls OpenAI’s list and tests whether an IP from your logs falls inside any published CIDR:

# Requires jq and (for CIDR math) grepcidr — brew/apt install both
IP="203.0.113.45"   # the IP from your log line
curl -s https://openai.com/gptbot.json \
  | jq -r '.prefixes[].ipv4Prefix // .prefixes[].ipv6Prefix' \
  | while read -r cidr; do
      echo "$IP" | grepcidr "$cidr" >/dev/null 2>&1 && echo "VERIFIED in $cidr"
    done
# No output = the IP is NOT in OpenAI's published range → treat as spoofed.

For Anthropic, fetch the current https://claude.com/crawling/bots.json list and adjust the jq path to its documented shape. If another operator does not publish a current verification feed, retain the result as a claimed user-agent.

3. Reverse + forward DNS fallback (off-list IPs)

Use this only when the named provider publishes an expected hostname suffix and the reverse-plus-forward verification procedure. It is not a generic proof of bot identity:

IP="203.0.113.45"
HOST=$(host "$IP" | awk '/pointer/ {print $NF}' | sed 's/\.$//')
echo "PTR: $HOST"
host "$HOST" | grep -q "$IP" && echo "FORWARD-CONFIRMED" || echo "MISMATCH → suspect"

4. Status-code breakdown per bot

Spot 403/429 blocks and 404s the bots are hitting:

# For GPTBot: tally status codes (combined log format; $9 is the status)
grep -i 'GPTBot' access.log | awk '{print $9}' | sort | uniq -c | sort -rn
# → e.g. "812 200 / 47 404 / 15 403" — the 403s are worth investigating

5. Raw-HTML dependency signal — what file types is the bot fetching?

If a claimed bot only ever pulls .html// and never .js/.css/images, and your content is JS-rendered, that is a dependency signal to investigate:

# What extensions is ChatGPT-User actually requesting?
grep -i 'ChatGPT-User' access.log \
  | awk '{print $7}' \
  | grep -oE '\.(html?|js|css|png|jpe?g|webp|svg|woff2?)(\?|$)' \
  | sort | uniq -c | sort -rn
# All HTML, zero subresources = compare the raw body with provider-specific outcomes.

6. Chrome DevTools Console — spot AI fetchers on a live page

Not a log query, but a quick check of what a page ships in raw HTML vs. after JS (what an AI bot would and wouldn’t see). Paste into the Console:

// Compare rendered text length to what's in the initial HTML source.
// A big gap means most content depends on JS; provider behavior must be verified separately.
(async () => {
  const raw = await (await fetch(location.href, { cache: "no-store" })).text();
  const rawText = new DOMParser().parseFromString(raw, "text/html").body.innerText.trim().length;
  const renderedText = document.body.innerText.trim().length;
  console.log({ rawText, renderedText, jsDependentRatio: +(1 - rawText / renderedText).toFixed(2) });
})();
// jsDependentRatio near 1 = almost all content is JS-injected; flag dependency, not invisibility.

7. Bookmarklet — open every bot’s IP list at once

Drag this as a bookmark to open the verification files in tabs for a quick lookup:

javascript:(function(){["https://openai.com/gptbot.json","https://openai.com/searchbot.json","https://openai.com/chatgpt-user.json","https://claude.com/crawling/bots.json"].forEach(u=>window.open(u,"_blank"));})();

SOP: monthly AI-crawler log review

A recurring procedure to run on a cadence (monthly for most sites; weekly if AI traffic is material to you). It produces a comparable baseline each time.

Prep

1. Pull the log window since your last review from the right layer (CDN/edge logs if you’re behind a CDN, origin logs otherwise). Confirm the scheduled SFTP/n8n pull actually ran — gaps here silently break trend lines.
2. Load into your tool of record (Screaming Frog LFA for periodic imports; BigQuery/ ELK if you’re on a pipeline).

Verify (never skip) 3. Refresh each current provider-published verification source (gptbot.json, searchbot.json, chatgpt-user.json, adsbot.json, and Anthropic’s claude.com/crawling/bots.json) — these change. 4. Enable verify-on-import (Screaming Frog) or run the IP-in-CIDR check (Scripts tab) against every AI-bot request. 5. Split traffic into verified and unverified. Report the unverified fraction — that’s your spoof rate this period. A jump is a finding on its own.

Measure (verified traffic only) 6. Crawl frequency per bot vs. last period — note new bots, disappeared bots, and bursts. 7. Top-fetched pages; confirm priority/deep pages are being crawled at all. 8. Raw-HTML dependency spot check on 2–3 JS-heavy templates (are bots pulling only HTML?). 9. Status-code breakdown per bot; investigate any new 403/429/404 clusters. 10. robots.txt / llms.txt request presence and any disallowed-path hits.

Report & act 11. Log the period’s numbers to a running sheet (verified hits by bot, spoof %, top pages, error clusters) so you have trend, not snapshots. 12. Pair with citation-side data (Bing grounding queries / GSC AI features) before concluding anything about visibility. 13. File concrete fixes: intentional-vs-accidental blocks, JS-render gaps, broken links, precise user-agent rules for confirmed impersonators.

Cadence note: re-run verification every period — don’t cache a “known-good” IP list. Stealth crawlers resurface under new IPs and generic browser UAs.

Playbook: “GPTBot (or ClaudeBot) is getting 403s / 429s and I don’t know if that’s on purpose”

A linear runbook for the common incident — a verified AI bot is hitting errors in your logs. Work it in order.

Step 1 — Confirm it’s the real bot first. Before anything else, use the named provider’s current official verification method. If none exists, the identity remains unverified; do not silently promote it to genuine or demote it to spoofed. The 403 may still be your WAF doing its job.

Step 2 — Identify the exact response and where it’s coming from. Pull the status-code breakdown for that bot (Scripts tab). Is it 403 (blocked), 429 (rate-limited), or 503? Note whether the block is at your CDN/WAF, your server config, or robots.txt-adjacent rules.

Step 3 — Decide: is the block intentional?

• You meant to block this bot (e.g. a training crawler you’ve opted out of) → the 403 is working as designed. Keep the documented robots policy separate from any network-level abuse/security rule. Done.
• You did NOT mean to block it (e.g. it’s OAI-SearchBot / PerplexityBot and you want AI-search visibility) → continue.

Step 4 — Find the accidental rule. Common culprits: an over-broad WAF “bot” rule, a rate-limit threshold too low for a bursty crawler (recall WISLR’s 114 requests/minute GPTBot spike — a per-minute cap can trip on legitimate bursts), a robots.txt disallow you forgot, or a geo/ASN block catching the operator’s ranges.

Step 5 — Fix precisely. Whitelist the verified bot by user-agent + published IP range, or raise the rate-limit ceiling for that verified bot specifically. Don’t loosen protection for everyone.

Step 6 — Verify the fix in the logs. After deploying, re-pull the logs for that bot. You want the 403/429 cluster to turn into 200/304 on the pages you care about.

Step 7 — Re-baseline and watch. Note the change in your running sheet. Recheck next period — and stay alert for the bot resurfacing under a different UA if you did intend to block it (stealth crawling).

Log-analysis mistakes that produce wrong conclusions

Each is a belief from the wild, why it’s wrong, and what to do instead.

Myth: “If my robots.txt blocks a bot, it won’t be in my logs / it’s not a problem.” Why it’s wrong: robots.txt is a request, not enforcement. User-triggered fetchers explicitly carve themselves out, and some crawlers have been reported bypassing blocks (Perplexity stealth crawling, Cloudflare Aug 2025). The block might be ignored entirely. Do instead: Use the logs to check whether the rule is honored — look for disallowed-path hits and whether the bot even requests /robots.txt.

Myth: “A GPTBot / ClaudeBot user-agent in my logs means it’s really OpenAI / Anthropic.” Why it’s wrong: User-agents are trivially spoofed — HUMAN Security measured 5.7% fake across 16 bots, and Duane Forrester found 81.8% fake among his own live-fetch requests. Do instead: Match the user-agent and use the operator’s current official verification method where one exists; otherwise label the identity unverified.

Myth: “More AI-crawler hits = more likely to get cited.” Why it’s wrong: No established causal/correlational data supports it. Retrieval is necessary but not sufficient — heavily crawled pages go uncited constantly. Do instead: Treat logs as visibility into the retrieved stage only, and pair them with citation-side data (Bing grounding queries, GSC AI features).

Myth: “Every AI crawler has the same rendering behavior.” Why it’s wrong: providers do not publish one shared JavaScript/subresource contract, and observed behavior differs by agent and sample. Do instead: Compare the raw response with the rendered page, inspect subresource requests by named bot, and label the result observed rather than universal.

Myth: “AI bots are at least checking for my llms.txt.” Why it’s wrong: WISLR’s 48-day sample recorded zero /llms.txt requests from any AI bot, consistent with the ~97%-unread finding in the AI crawlers article. Do instead: Don’t treat llms.txt requests as expected validation; measure what bots actually fetch.

Myth: “Network blocking and robots policy are the same control.” Why it’s wrong: robots communicates crawl preferences; a firewall enforces network access and can affect unrelated traffic. Do instead: Use the documented user-agent rule for policy, and reserve network blocking for separately justified abuse/security response.

Real cases

Duane Forrester — self-tested spoof rates on his own logs. Forrester ran this exact method on his site and published the numbers. Among 33 live-fetch requests, only 6 came from a vendor-published IP — an 81.8% spoof rate; among 799 Googlebot-named requests, only 107 verified — roughly 87% fake (SEJ). Before: trusting the user-agent, his “AI assistant” traffic looked real. After: matching names to published IP lists, most of it was impersonation. Takeaway: the method matters more than his specific numbers — as he says, pull your own date range and find your real fraction.

WISLR — 48 days of CDN logs, the crawl-vs-render signature. Tony Castillo analyzed 288,566 CDN log lines (12,099 AI/bot requests) over 48 days (WISLR). Concrete findings: GPTBot was absent for weeks then burst 152 requests in three minutes (114 req/min peak); ChatGPT-User fetched zero images, CSS, or JS — pure HTML extraction; and there were zero /llms.txt requests across the whole window. Before: you’d assume steady crawling and JS-aware bots. After: the logs show bursty, HTML-only behavior — n=1, one site’s data, but a vivid picture of what a real analysis surfaces. Takeaway: the zero-JS fetch pattern is the direct evidence behind the crawl-vs-render check.

Cloudflare — the crawl-to-referral ratio, at network scale. Cloudflare’s aggregate data shows how little crawling translates to traffic: for every visitor Anthropic refers back to a website, its crawlers have already visited tens of thousands of pages (Cloudflare). Before: the intuition that heavy crawling means engagement. After: at network scale the ratio is lopsided — training now drives the majority of AI-bot activity, and training bots aren’t meant to send traffic back at all (Cloudflare). Takeaway: a page crawled heavily with zero referral is normal, not a sign of success — which is exactly why crawl frequency isn’t a citation predictor.

Ready-to-use AI prompts

Copy-paste prompts for using an LLM to speed up AI-crawler log analysis. Always sanity-check the output against the raw logs — LLMs hallucinate, and a verification pipeline that trusts a hallucinated IP match is worse than none.

Draft a verification-aware log parser

Write a Python script that parses combined-format Nginx access logs and, for each
request whose user-agent matches a known AI bot (GPTBot, OAI-SearchBot,
ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, CCBot,
Bytespider), does the following:

1. Extract timestamp, client IP, request path, status code, user-agent.
2. Fetch and cache OpenAI's current IP lists (gptbot.json, searchbot.json,
   chatgpt-user.json, adsbot.json) and Anthropic's current bots.json list.
3. Mark an OpenAI or Anthropic request VERIFIED only if the user-agent matches AND
   the client IP falls inside the matching provider-published ranges; mark providers
   without an official method UNVERIFIED, not spoofed.
4. Output two CSVs: verified requests and unverified ("spoofed") requests.
5. Print a summary: verified vs. unverified count per bot, and the top 20 fetched
   paths (verified only).

Do NOT count unverified requests in any per-bot metric. Add clear comments.

Summarize a status-code + crawl-vs-render report

I'll paste a table of AI-bot log data (columns: bot, path, status_code,
file_extension). Produce:
- A per-bot status-code breakdown, flagging any 403/429/404 clusters.
- A raw-HTML dependency read: for each bot, the ratio of HTML requests to
  JS/CSS/image requests, and a note on whether the bot appears to fetch only HTML
  in this sample. Treat HTML-only on a JS-rendered page as a dependency risk, not
  proof of a universal no-JavaScript capability.
Keep every conclusion tied to a number from the data — do not infer beyond it.

DATA:
[paste]

Triage a suspicious IP

A request in my logs claims to be [BOT NAME] from IP [IP ADDRESS]. Walk me through
verifying it: which operator IP-list file to check, how to test whether the IP is
in range, and the reverse+forward DNS fallback if it's not on a published list.
Tell me explicitly what result means "verified" vs. "treat as spoofed." Do not
guess whether this specific IP is legitimate — give me the steps to check.

Tools for AI-crawler log analysis

Roughly free/ground-truth → paid/managed:

• grep / PowerShell — fastest way to a bot-hit count and a verification-aware filter over a raw access log. Zero setup; see the Scripts tab.
• Screaming Frog Log File Analyser — desktop importer with built-in AI-bot presets and a “Verify Bots When Importing” toggle that checks against publicly confirmed IP lists. Response Codes, User Agents, URLs (sort by Num Events), and IPs tabs. Best for periodic imports.
• ELK Stack (Elasticsearch / Logstash / Kibana) or Splunk — continuous ingestion, dashboards, and alerting once a desktop import is too slow or you want ongoing monitoring.
• BigQuery — long-term retention and SQL at scale, commonly fed by Cloudflare Logpush or a scheduled GraphQL pull from httpRequestsAdaptiveGroups.
• Cloudflare AI Crawl Control — for sites behind Cloudflare: managed crawler-activity, bot-verification, and directive-compliance views without your own pipeline.
• Google Search Console — Crawl Stats — not for AI bots (Googlebot only), but the model for the by-crawler + by-response- code breakdown you’re rebuilding for AI bots from raw logs.
• Bing Webmaster Tools — AI Performance — the citation-outcome counterpart: pair its grounding queries and citation counts with your crawl-input logs to close the loop.

Claimed AI-bot traffic spikes overnight

Symptom: requests with a famous crawler user-agent jump suddenly. Likely cause: spoofing, monitoring traffic, or a real crawl change. Fix: verify source IPs against the operator’s current published method before attributing the traffic, then segment by ASN, path, status, and time.

Logs show crawling but the content is never cited

Symptom: verified bots fetch pages without visible citation gains. Likely cause: crawling is only eligibility evidence; retrieval and answer selection are separate. Fix: confirm the bot received substantive HTML, then evaluate indexability, passage quality, query fit, and off-site corroboration without treating crawl count as rank.

Every request appears to return 200

Symptom: missing URLs and blocked content are recorded as success. Likely cause: an app shell, CDN rule, or custom error page returns a soft 404. Fix: sample response bodies and final headers, then repair status handling rather than trusting status alone.

Verified bots receive an empty shell

Symptom: a browser renders content, but log-matched fetches receive little useful HTML. Likely cause: the page depends on client-side JavaScript the crawler does not execute. Fix: compare raw and rendered output and serve critical content and links in HTML through SSR, static rendering, or another reliable delivery strategy.

Metrics for AI-crawler logs

Resources worth your time

My related writing

• How to Do an SEO Log File Analysis (Ahrefs, reviewed by Patrick Stox & Michal Pecánek) — the pre-AI-era template this article is the AI-specific sequel to: what to measure, tools, and bot verification.
• What is Log File Analysis? (Ahrefs glossary) — the definitional companion.
• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — my Cloudflare Radar analysis of AI-bot crawl share.
• The AI Bots That ~140 Million Websites Block the Most — robots.txt block-rate data across the open web.
• 80% of Our AI Search Traffic Goes to Our Homepage, Product Pages, and Free Tools — page-type-level analysis of AI activity, the analog to “which pages get crawled.”

My speaking

• How Search Works (SlideShare) — my walkthrough of crawling, rendering, indexing, and ranking, useful background for reading what bots do in your logs. (Standing disclaimer applies: this is my understanding of the systems, not a guarantee it’s 100% complete or accurate.)

From around the industry

• Why log file analysis matters for AI crawlers and search visibility — Lauren Busby (Trebletree), Search Engine Land: the “logs are the missing layer,” retention-window, and 403/429 framing.
• 81.8% Of My “AI Assistant” Traffic Was Fake. The Googlebot Number Was Worse — Duane Forrester, Search Engine Journal: the standout first-party verification case study and method.
• Perplexity, Stealth AI Crawling, and the Impacts on GEO and Log File Analysis — Clint Spaulding, Seer Interactive: why blocked bots resurface looking human.
• How to Monitor AI Bots in the Log File Analyser — Screaming Frog: the practical tool walkthrough with verify-on-import.
• The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals — Cloudflare: network-level crawl-to-referral ratios and the training-vs-search split.
• Does Anthropic crawl data from the web? — Anthropic’s current bot-purpose and robots-control documentation.
• OpenAI bots / crawlers docs — user-agent strings and the published IP files for verification.

Stats worth citing

• Spoof rate — 5.7% across 16 AI crawlers. HUMAN Security’s two-week analysis of traffic claiming to be one of 16 well-known AI crawlers found roughly 1 in 18 requests was spoofed (vendor-sourced, relayed via SEJ).
• Spoof rate — 81.8% in one practitioner’s own logs. Duane Forrester’s self-run audit found 27 of 33 live-fetch requests came from IPs the vendors don’t publish; his Googlebot number was ~87% fake (SEJ).
• Crawl-to-referral, ClaudeBot vs. OpenAI. Cloudflare’s data (week of May 25– June 1, 2026) put ClaudeBot at ~11,122 pages crawled per referral and OpenAI at ~857:1, versus Googlebot around 5:1 (Cloudflare).
• Training drives most AI-bot activity. Per Cloudflare, training now drives nearly 80% of AI-bot activity, up from 72% a year earlier — context for why heavy crawling with zero referral is normal (Cloudflare).
• One real log window: 288,566 lines, 12,099 bot requests, 48 days. WISLR’s case study — with GPTBot’s 152-requests-in-3-minutes burst and ChatGPT-User’s zero image/CSS/JS fetches (WISLR).

Test yourself: AI Crawler Log Analysis

Five quick questions on pulling and reading AI-bot logs. Pick an answer for each, then check.

AI Crawler Log Analysis

AI crawler log analysis is the practice of pulling raw server or CDN access logs and examining them for requests from AI bots — training crawlers, AI-search indexers, and user-triggered fetchers — to verify with first-party data which bots actually hit your site, whether they're real or spoofed, and what they got.

Related: Log file analysis, AI Crawlers

AI Crawler Log Analysis

AI crawler log analysis is the AI-search-era update to classic SEO log file analysis. Instead of auditing Googlebot for crawl budget, you pull your raw server (Apache/Nginx) or CDN access logs and examine them specifically for requests from AI bots — training crawlers (GPTBot, ClaudeBot), AI-search indexers (OAI-SearchBot, PerplexityBot, Applebot), and user-triggered fetchers (ChatGPT-User, Perplexity-User) — to answer questions robots.txt files and analytics dashboards can’t.

Logs are the only first-party ground truth for four things: which of your pages an AI bot actually fetched, how often, whether the requester is the real bot or a spoof, and what it got — your rendered content or a JavaScript-empty shell — plus the response codes it hit. Neither Google Search Console (Googlebot-only) nor GA4 (which only sees clicks that carry a referrer) shows any of this.

Two things make it different from ordinary log file analysis. First, verification matters more: AI-bot user-agent strings are spoofed at a documented, non-trivial rate, so a GPTBot string alone proves nothing — you match the user-agent and confirm the source IP against the operator’s published range. Second, you have to check crawl vs. render: unlike Googlebot, the major AI crawlers don’t execute JavaScript, so a page can be crawled constantly and still deliver nothing if its content is client-side rendered.

Log analysis observes only the retrieved stage of the retrieved → mentioned → cited funnel. Being crawled is necessary for a citation but nowhere near sufficient, so treat crawl-frequency data as visibility into one input, not a citation predictor.

Related: Log file analysis, AI Crawlers