Log File Analysis

TL;DR — Your web server keeps a log of every request it gets, including visits from search engine bots like Googlebot. Log file analysis means reading that log to see exactly which of your pages the bots actually fetched, how often, and whether they hit errors. It’s the only place that shows what really happened — but most small sites don’t need it.

Tap to expand

The four-week chart compares automated and human requests to HTML content. Bot share is higher in the captured worldwide Cloudflare traffic period.

What a log file is

Every time anyone — a person, Googlebot, or some random bot — requests a page from your site, your server writes a line about it in a file. That file is your access log. Each line records roughly the same things:

• who asked (an IP address and a “user agent” name like Googlebot),
• what they asked for (the URL),
• when (a timestamp), and
• what they got back (an HTTP status code — 200 for OK, 404 for not found, and so on).

Stack up weeks of those lines and you have a complete, honest record of how search engines crawl your site. Not an estimate. The actual requests.

Why bother reading it

Other SEO tools either guess at how bots crawl you (a crawl tool pretends to be a search engine and walks your site) or summarize it (Google Search Console shows you a sampled, rounded-off view). Your logs show the real thing, request by request. That lets you answer questions like:

• Which pages does Googlebot actually visit — and which does it ignore?
• Is the bot wasting time on junk URLs instead of your important pages?
• Are bots hitting broken pages (404) or server errors (5xx)?
• Are there pages bots have never reached at all?

The one rule you can’t skip

Anyone can pretend to be Googlebot. The user-agent name in a log line is just text — a scraper can put Googlebot in there to sneak past your defenses. So before you trust a single “Googlebot” line, you have to verify it’s really Google (there’s a simple DNS check for this in the Advanced and Scripts tabs). Skip verification and you’ll draw conclusions from fake traffic.

Do you even need this?

Honestly? Probably not, if you run a small site. Log file analysis pays off for large sites — tens of thousands of URLs, ecommerce with lots of filtered pages, sites going through a migration, or sites where Google says it “discovered” pages but never indexed them. If your site has a few hundred pages and they all get crawled fine, your time is better spent elsewhere. (Same logic as crawl budget — most sites don’t need to worry about it.)

Want the real workflow — verifying bots properly, finding crawl waste, spotting orphan pages? Switch to the Advanced tab.

TL;DR — Logs are the unsampled ground truth for crawling: every request, every bot, every status code. The non-negotiable first step is verifying Googlebot/Bingbot via reverse + forward DNS (or Google’s published IP-range JSON) — user agents are spoofed constantly, and you do all crawl math on the verified set only. Then read the logs for most/least-crawled URLs and sections, crawl frequency over time, status codes prioritized by frequency, crawl waste (params, facets, internal search, infinite pagination), orphan and uncrawled pages (cross-referenced against a crawl), and the mobile-vs-desktop Googlebot split. Bing publishes no official IP ranges, so DNS to *.search.msn.com is the method there. Logs complement GSC Crawl Stats — they don’t replace it. And in 2026, AI bots are now a huge share of what shows up.

Why logs are the ground truth

There are three ways to “see” how search engines crawl you, and they are not equal:

• A crawl tool (Screaming Frog SEO Spider, Ahrefs Site Audit) simulates a crawl. It tells you what a bot could find, not what Google did fetch.
• GSC Crawl Stats summarizes the real thing, but it’s sampled, aggregated, and capped (roughly 1,000 rows, ~90 days, no per-URL export).
• Server logs record the real thing — every request, for every bot, with the exact URL, timestamp, and status code.

The Ahrefs log-file guide I reviewed puts it plainly: server logs are “the most trustworthy source of information to understand the URLs that search engines have crawled.” That’s the whole reason this technique exists. When I want to know what Googlebot actually did — not what it might do, not a rounded summary — I go to the logs.

A typical log line carries the IP address, user agent, URL path, timestamp, request method (GET/POST), and HTTP status code. Everything below is just slicing those fields intelligently.

When you actually need it (and when you don’t)

Be honest with yourself here. Log file analysis is a large-site tool. It earns its keep on sites with tens of thousands of URLs, ecommerce and faceted navigation, sites mid-migration, and sites stuck in Discovered – currently not indexed. As I wrote in my crawl budget guide, “Most sites don’t need to worry about crawl budget, but there are few cases where you may want to take a look.” Daniel Waisberg of Google has made a similar point about Crawl Stats — per Search Engine Journal’s coverage, the report isn’t much of a concern for sites under ~1,000 pages.

If your few-hundred-page site is being crawled fine, skip this and go fix something with more leverage.

How to get your logs (the hardest part is access)

Logs live wherever the request actually terminated:

• Apache & Nginx → the Apache “combined” log format (the most common).
• Microsoft IIS → W3C format.
• AWS ELB/ALB → ELB format.
• CDNs (Cloudflare, Fastly, Akamai) → their own log exports. This matters: at a CDN-fronted site, an origin-only log misses edge-cached hits, so pull logs at the layer the bot actually reached.

Aim for 30 days minimum, 90 ideal, to capture crawl-frequency variation. And plan for friction — getting access to server logs is often the genuinely hard part (DevOps gatekeeping). Even Googlers, in a migration episode of Search Off the Record, flagged how hard log files can be to obtain in practice. Budget time for the request.

Logs aren’t just bot traffic — they capture every request, including real visitors, and can carry query-string values, session identifiers, or other sensitive data alongside the URL path. OWASP’s logging guidance is blunt about this: authentication credentials, access tokens, and personally identifiable information generally shouldn’t land directly in a log; they should be removed, masked, or hashed first. Build that into your access controls and export process before you hand a log file to anyone for analysis, not after.

Step 1 — Verify the bots are real (do this before anything else)

This is the step most guides wave at in one line. Do not. Many bots pretend to be Googlebot to get past firewalls (Ahrefs). The user agent is unauthenticated text; treat every “Googlebot” line as a claim to be proven.

Googlebot — two valid methods:

1. Reverse + forward DNS (the bidirectional check). Google’s own steps: run a reverse DNS lookup on the IP from your logs with the host command; verify the domain is googlebot.com, google.com, or googleusercontent.com; then run a forward DNS lookup on that hostname and verify it resolves back to the original IP. The forward step is what makes this trustworthy — a spoofer can point reverse DNS at a *.googlebot.com name, but only the round-trip back to the same IP proves it. (Commands for macOS/Linux and Windows are in the Scripts tab.)
2. Match against Google’s published IP ranges. Google publishes JSON files of its crawler IPs in CIDR format — common-crawlers.json for Googlebot and friends, plus special-crawlers.json, the user-triggered-fetcher files, and an all-Google goog.json. As I noted in my Googlebot guide, Google “provided a list of public IPs you can use to verify the requests are from Google… You can compare this to the data in your server logs.”

Bingbot — DNS only. This is the key contrast: Bing does not officially publish IP ranges. Bing’s own wording is that “…like other search engines, Bing does not publish a list of IP addresses or ranges from which we crawl the Internet,” because “the IP addresses or ranges we use can change any time.” So for Bingbot you do reverse + forward DNS to a hostname ending in *.search.msn.com (e.g. msnbot-157-55-33-18.search.msn.com), or use the Verify Bingbot tool. (Microsoft has since released a bingbot IP JSON, but its official verification guidance still centers on DNS precisely because IPs change.)

Then throw out the fakes. Do all of your crawl math on the verified set only. Unverified “Googlebot” is almost always a scraper or a spoofed bot and belongs in a security review, not your crawl-waste analysis.

Step 2 — What to look for

Once you’re working with verified hits, here’s the read:

• Most & least crawled URLs and sections. Rank requests by URL and by directory. This is where your crawl budget is actually going — and it’s usually surprising.
• Crawl frequency over time. Trend crawls by URL/section to catch drops (a migration broke something) or spikes (a new section, or a spider trap spinning up infinite URLs).
• Status codes bots hit, prioritized by frequency. Quantify 200 vs. 301/302 (and chains), 404, and 5xx. A 404 hit 5,000×/week is a different problem than a 404 hit once — fix by crawl frequency, not by mere existence.
• Crawl waste. Faceted nav, URL parameters, internal search results, and infinite calendars/pagination can eat a large share of crawl budget on bad offenders. Logs show exactly which junk patterns the bots are burning time on.
• Orphan & uncrawled pages. This needs both datasets. Cross-reference logs against a site crawl: URLs in the logs but not in the crawl = orphans, old redirects, or externally-linked pages; URLs in the crawl but not in the logs = pages Google has never fetched.
• Mobile vs. desktop Googlebot. Split by user agent. Post mobile-first, it should be majority Googlebot Smartphone — a desktop-heavy split is worth a look.
• Response time & crawl health. Rising average response time correlates with reduced crawling. Per SEJ’s write-up of Waisberg’s guidance: “Watch out for a consistent increase in average response time. Google says it might not affect crawl rate immediately, but it’s a good indicator that your servers might not be handling all the load.”

What logs do NOT tell you

Keep these straight or you’ll over-read the data:

• Crawl ≠ index. A URL Googlebot fetches daily can stay unindexed indefinitely. Logs prove fetching, not index status — pair them with GSC’s Page Indexing / URL Inspection to learn the index side.
• Crawl ≠ rank, and more crawling doesn’t help. As I’ve said repeatedly, “The rate of crawling isn’t going to impact your rankings.” Don’t chase crawl volume as if it were a ranking lever.
• noindex doesn’t reduce crawling. noindex controls indexation, not crawling — to actually stop the crawl you use robots.txt or a status code.
• Crawl ≠ model training or citation. A verified hit from GPTBot, ClaudeBot, or PerplexityBot proves that request happened — a fetch at that layer. It doesn’t prove the page was used to train a model, retained anywhere downstream, or cited in a chat answer. Those are separate, unobserved outcomes; don’t stretch a verified log line further than it goes.

The 2026 wrinkle: AI bots are all over your logs now

The cast of characters in a modern log file has changed. In my analysis of Cloudflare Radar data (Meet the New Web Crawlers), search-engine bots still crawl the most — but AI bots are firmly in second place and on track to overtake them within a couple of years. GPTBot, ClaudeBot, PerplexityBot, and friends now show up heavily. When you segment your verified hits by user agent, don’t be surprised to find AI crawlers rivaling the search engines for share of requests. (Screaming Frog’s Log File Analyser has added a dedicated AI-bot tutorial for exactly this.)

How this fits with the rest of crawling

Logs are the diagnostic layer under the whole crawling cluster. They’re how you actually measure the crawl budget spend that engines describe in the abstract (Gary Illyes defines it as “the number of URLs Googlebot can and is willing or is instructed to crawl”). They’re also how you catch spider traps red-handed — an infinite URL space from a calendar or facet shows up as a flood of near-identical requests — and how you confirm whether your crawl frequency work (accurate lastmod, internal links to important pages) actually changed bot behavior. And remember they complement, not replace, GSC Crawl Stats: Crawl Stats is the sampled on-ramp; logs are the unsampled, multi-bot, per-URL detail.

AI summary

A condensed take on the Advanced version:

• Logs = ground truth. Crawl tools simulate, GSC samples; server access logs record every request, every bot, with URL, timestamp, and status code.
• Verify before you analyze. User agents are spoofed constantly. Confirm Googlebot via reverse + forward DNS or Google’s published IP-range JSON; confirm Bingbot via reverse DNS to *.search.msn.com (Bing publishes no official IP ranges). Do all crawl math on the verified set only.
• What to read: most/least-crawled URLs and sections; crawl frequency over time; status codes prioritized by frequency (a 404 hit 5,000×/week ≠ once); crawl waste (params, facets, internal search, infinite pagination); orphan and uncrawled pages (cross-reference a crawl); mobile-vs-desktop Googlebot split; rising response time as a crawl-health warning.
• Don’t over-read it: crawl ≠ index ≠ rank, more crawling doesn’t help rankings, noindex doesn’t reduce crawling, and a verified AI-bot hit proves a fetch — not that the page was used for model training or cited in an answer.
• Scope: a large-site / ecommerce / migration tool. Most small sites don’t need it.
• 2026: AI bots (GPTBot, ClaudeBot, PerplexityBot) are now a major and growing share of log traffic.
• Complements GSC Crawl Stats — use both.

Official documentation

Primary-source documentation for verifying crawlers and reading crawl data.

Google

• Verify Requests from Google Crawlers and Fetchers — the two official methods: manual reverse + forward DNS, and matching against the published IP ranges.
• How to verify Googlebot (Search Central Blog) — the older companion post, still cited.
• common-crawlers.json — Googlebot and common crawler IPs in CIDR format (note: “the IP addresses in the JSON files are represented in CIDR format”). Companions: special-crawlers.json, user-triggered-fetchers.json, and the all-Google goog.json.
• Overview of Google crawlers and fetchers — every Google user agent you’ll see in logs.
• Crawl Stats report (Help) and the launch blog — the sampled, official view that logs complement.

Bing / Microsoft

• How to Verify Bingbot (Webmaster help) — the official verification page.
• How to Verify that Bingbot is Bingbot (blog) — the canonical reverse + forward DNS method, and the statement that Bing doesn’t publish IP ranges.
• Verify Bingbot tool — paste an IP to check it.

Quotes from the source

On-the-record statements. Where supported, each link is a deep link that jumps to the quoted passage on the source page.

Google — verifying crawlers

• “Run a reverse DNS lookup on the accessing IP address from your logs, using the host command.” — Google Search Central docs. Jump to quote
• “Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.” Jump to quote
• “Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name.” Jump to quote
• “Verify that it’s the same as the original accessing IP address from your logs.” Jump to quote

Bing — verification and no published IP ranges

• “Perform a reverse DNS lookup using the IP address from the logs to verify that it resolves to a name that end with search.msn.com.” — Bing Webmaster Blog. Jump to quote
• “…like other search engines, Bing does not publish a list of IP addresses or ranges from which we crawl the Internet.” — and the reason: “the IP addresses or ranges we use can change any time, so responding to requests differently based on a hardcoded list is not a recommended approach.” Jump to quote

Patrick Stox — on what logs are for (from my work at Ahrefs)

• Server logs are “the most trustworthy source of information to understand the URLs that search engines have crawled.” Jump to quote
• “Many bots pretend to be Googlebot to get past firewalls.” Jump to quote
• “If you want to see hits from all bots and users, you’ll need access to your log files.” Jump to quote
• “The rate of crawling isn’t going to impact your rankings.” Jump to quote

Gary Illyes, Google — crawl budget (what logs let you measure)

• Crawl budget is “the number of URLs Googlebot can and is willing or is instructed to crawl.” Read the coverage

Daniel Waisberg, Google — response time as a crawl-health signal (SEJ’s framing of his Crawl Stats guidance)

• “Watch out for a consistent increase in average response time. Google says it might not affect crawl rate immediately, but it’s a good indicator that your servers might not be handling all the load.” Jump to quote

Verify a bot is really Googlebot (reverse + forward DNS)

The user agent in a log line is just text — scrapers spoof Googlebot to get past firewalls. The only trustworthy check is bidirectional DNS: reverse-lookup the IP, confirm the hostname is a Google domain, then forward-lookup that hostname and confirm it resolves back to the same IP.

macOS / Linux (uses host)

# 1) Reverse DNS the IP from your logs — it must end in googlebot.com,
#    google.com, or googleusercontent.com
host 66.249.66.1
# → 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com

# 2) Forward DNS that hostname back — it must resolve to the same IP
host crawl-66-249-66-1.googlebot.com
# → crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Windows (uses nslookup)

:: 1) Reverse DNS the IP — confirm it ends in a Google domain
nslookup 66.249.66.1

:: 2) Forward DNS the returned hostname — confirm it matches the original IP
nslookup crawl-66-249-66-1.googlebot.com

If the reverse lookup doesn’t land on a Google domain, or the forward lookup doesn’t return the original IP, it is not Googlebot. (For Bingbot, run the exact same two steps but expect a hostname ending in search.msn.com.) You can also skip DNS and match the IP against Google’s published ranges (common-crawlers.json, in CIDR format).

Extract and count crawler hits from a raw log

Quick one-liners for a combined-format Apache/Nginx access log. (These select on the user-agent string — remember to verify the IPs before trusting the counts.)

# Pull only the lines claiming to be Googlebot
grep -i "googlebot" access.log > googlebot-hits.log

# Count Googlebot hits per URL, most-crawled first
# (combined format: $7 is the request path)
grep -i "googlebot" access.log \
  | awk '{print $7}' \
  | sort | uniq -c | sort -rn | head -50

# Count Googlebot hits per status code (combined format: $9 is the status)
grep -i "googlebot" access.log \
  | awk '{print $9}' \
  | sort | uniq -c | sort -rn

# List the URLs Googlebot hit that returned a 404, by frequency
grep -i "googlebot" access.log \
  | awk '$9 == 404 {print $7}' \
  | sort | uniq -c | sort -rn

# Get the unique IPs claiming Googlebot — the list you then verify by DNS
grep -i "googlebot" access.log | awk '{print $1}' | sort -u

Adjust the $7/$9 field positions if your log format differs (IIS/W3C and ELB order fields differently).

Log file analysis workflow

• Get the logs — server access logs, or CDN/load-balancer exports. At a CDN-fronted site, pull edge logs too (origin misses cache hits).
• Grab enough window — 30 days minimum, 90 ideal.
• Verify the bots first — reverse + forward DNS, or Google’s published IP ranges. Bingbot → DNS to *.search.msn.com.
• Drop the fakes — do all crawl math on the verified set; route spoofed “Googlebot” to a security review.
• Rank most/least-crawled URLs and sections — see where budget goes.
• Trend crawl frequency over time — catch drops (migrations) and spikes (new sections, spider traps).
• Tally status codes — 200 / 301-302 chains / 404 / 5xx, and prioritize fixes by crawl frequency, not mere existence.
• Hunt crawl waste — parameters, faceted nav, internal search, infinite pagination/calendars.
• Find orphans & uncrawled pages — cross-reference logs against a site crawl (in logs not crawl = orphan; in crawl not logs = never fetched).
• Check the mobile vs. desktop Googlebot split — should be Smartphone-majority.
• Watch average response time — a rising trend can throttle crawling.
• Segment AI bots — GPTBot, ClaudeBot, PerplexityBot, etc. are now a real share of traffic.
• Confirm with GSC — pair crawl findings with Crawl Stats and URL Inspection (crawl ≠ index).

What to build in your analysis sheet

Whether you use Screaming Frog’s Log File Analyser, BigQuery, or the Ahrefs log-file template, you’re building the same handful of views. After verifying bots, parse each line into columns and pivot.

Columns to parse out of each log line

Pivots to build

• Crawls per URL (and per directory), descending — most & least crawled.
• Crawls per status code, then status code × URL (so a high-frequency 404 jumps out).
• Crawls per day/week, segmented by section — the frequency trend.
• Crawls per bot/user agent — the mobile/desktop split, plus AI-bot share.
• A logs vs. crawl join (VLOOKUP/merge against a Screaming Frog or Ahrefs crawl export) to surface orphans and never-fetched pages.

The Ahrefs guide ships a downloadable template that sets these up; the log-file-analysis guide links it.

How to read a log file: what to look for

A repeatable lens for any log analysis. Verify first; then run these passes.

1. Verify, then analyze. The data is only as good as the bot identity behind it. Reverse + forward DNS (or IP ranges); analyze the verified set only. Spoofed bots → security, not SEO.

2. Where is budget going? (most vs. least crawled.) Rank by URL and by section. The goal is to find attention spent on the wrong places — and important pages getting too little.

3. What are bots hitting? (status codes, by frequency.) 200 is healthy; 3xx chains, 404, and 5xx are leaks. Triage by how often the bot hits each, not by whether the error merely exists.

4. What’s being wasted? (crawl waste.) Parameters, faceted nav, internal search, infinite pagination/calendars — the classic budget sinks. Logs name the exact offending patterns.

5. What’s missing? (orphans & uncrawled.) Cross-reference logs with a crawl. In logs but not crawl = orphan/old redirect/external link. In crawl but not logs = Google never fetched it.

6. Who’s crawling? (bot segmentation.) Mobile vs. desktop Googlebot (should be Smartphone-majority), plus the AI-bot share that now rivals search engines.

7. Is the server healthy? (response time.) Rising average response time is an early warning that crawling may get throttled.

Tools for log file analysis

• Screaming Frog SEO Log File Analyser — the workhorse desktop tool. It auto-verifies search bots and flags spoofed IPs, and has User-Agent + Verification-Status filters for granular bot analysis. Import a SEO Spider crawl and use the “Not In URL Data” filter to find orphans — “URLs which were discovered in your logs, but are not present in the crawl data imported.” It also now has a dedicated AI-bot monitoring tutorial. Tool
• BigQuery (and Splunk / ELK Stack / Logflare / logz.io) — for raw, large-scale storage and querying when log volume is too big for a desktop tool.
• Ahrefs — cross-reference logs against an Ahrefs Site Audit crawl to find orphans and confirm what the bots reached. (My home tool.)
• Semrush Log File Analyzer, OnCrawl, Botify, JetOctopus — other crawl+log platforms, the latter three aimed at enterprise.
• GSC Crawl Stats — the official, free, sampled on-ramp. Complementary to raw logs, not a replacement: it’s aggregated, capped, and has no per-URL export.
• Verify Bingbot tool — paste an IP to confirm it’s really Bingbot: bing.com/toolbox/verify-bingbot.

Mistakes to avoid

• Trusting the “Googlebot” string without verifying it. Why it’s wrong: the user agent is unauthenticated text — anyone can put Googlebot in a request header to slip past defenses or pollute your analysis with fake traffic. Do instead: run reverse + forward DNS (or match against Google’s published IP ranges) before a single line counts toward your crawl math.
• Treating GSC Crawl Stats as the full picture. Why it’s wrong: it’s sampled, rounded, capped at roughly 1,000 rows and ~90 days, with no per-URL export — it summarizes, it doesn’t record. Do instead: use logs as the ground truth and Crawl Stats as a complementary, faster on-ramp.
• Pulling origin-only logs on a CDN-fronted site. Why it’s wrong: an origin log misses every request the CDN served from edge cache, so you’re analyzing an incomplete picture of what bots actually got. Do instead: pull logs at the layer the bot actually reached — the CDN/edge export, not just the origin server.
• Fixing 404s and 5xx errors by existence, not frequency. Why it’s wrong: a 404 hit once a month and a 404 hit 5,000 times a week are not the same problem, but treating every error line as equally urgent wastes fix effort. Do instead: rank status-code issues by how often bots actually hit them.
• Chasing crawl volume as if it were a ranking lever. Why it’s wrong: more crawling doesn’t move rankings — it’s a diagnostic signal, not a growth metric. Do instead: use crawl frequency to catch problems (drops, spider traps), not as a KPI to maximize.
• Using noindex to try to stop crawling. Why it’s wrong: noindex controls indexation, not crawling — Google still has to fetch the page to see the tag. Do instead: block the crawl itself with robots.txt or a status code if that’s the actual goal.
• Calling a URL “orphaned” from logs alone. Why it’s wrong: a URL that shows up in logs but not in a fresh crawl could be an old redirect, an external link, or a genuine orphan — logs alone can’t tell you which. Do instead: cross-reference logs against an actual site crawl before drawing conclusions either way.

Standing KPIs for log file analysis

Verified-bot share

• Metric: Verified Googlebot/Bingbot hits ÷ all hits claiming to be Googlebot/Bingbot by user agent.
• What it tells you: How much of your “bot traffic” is actually spoofed scrapers rather than real search engines.
• How to pull it: Run every claimed-bot IP through reverse + forward DNS (or the IP-range JSON) and count the pass rate.
• Benchmark / realistic range: No universal number — depends on how aggressively your site is scraped. A low or dropping verified share is the signal to act on, not a fixed threshold.
• Cadence: Every time you pull a new log window.

Crawl waste share

• Metric: % of verified bot requests hitting parameters, facets, internal search, or infinite pagination/calendars.
• What it tells you: How much of your crawl budget is going to junk URLs instead of pages that matter.
• How to pull it: Segment verified hits by URL pattern (query strings, known facet/search paths).
• Benchmark / realistic range: No universal figure — genuinely situational, depending on your site’s URL structure and faceting. Establish your own baseline on the first pull, then track the trend.
• Cadence: 30–90 day log window; re-check after any cleanup (robots.txt rules, parameter handling, pagination fixes).

Status-code mix, weighted by frequency

• Metric: 200 / 3xx / 404 / 5xx as a share of verified bot requests.
• What it tells you: Where bots are burning fetches on errors instead of live content, and whether that’s getting worse.
• How to pull it: Tally the status-code field from verified log lines.
• Benchmark / realistic range: No universal target — depends on site age and redirect history. Watch the trend, not a single snapshot; a rising 404/5xx share is the actionable signal.
• Cadence: 30–90 days, or immediately after a migration.

Mobile vs. desktop Googlebot split

• Metric: Share of verified Googlebot hits from the Smartphone user agent vs. Desktop.
• What it tells you: Whether Google is actually crawling you mobile-first, as expected post mobile-first indexing.
• How to pull it: Segment verified hits by the Googlebot UA string (Smartphone vs. Desktop).
• Benchmark / realistic range: Should be Smartphone-majority for most sites; a desktop-heavy split is worth investigating, not a hard failure on its own.
• Cadence: Each log pull.

Orphan / uncrawled page count

• Metric: URLs in logs but not in a fresh crawl (orphans/old links), and URLs in a fresh crawl but not in logs (never fetched).
• What it tells you: Pages bots can’t easily reach, and pages you’re linking to that Google has never bothered to fetch.
• How to pull it: Join the verified-hit URL list against a site crawl export (Screaming Frog, Ahrefs).
• Benchmark / realistic range: Entirely situational — depends on site size and how recently you migrated or restructured. Track the count over time rather than comparing to an outside number.
• Cadence: Quarterly for large sites, or immediately after a migration.

Ready-to-copy prompts for log analysis

These are for interpreting log data you’ve already pulled and verified — not for generating log data (never let an AI invent log lines or stats).

Summarize crawl waste from a URL sample

Here is a list of URL paths that verified Googlebot hits landed on, one per
line, from my server logs. Group them into patterns (query parameters,
faceted navigation, internal search, pagination/calendars, or "looks like a
real page"), and tell me which pattern has the most URLs. Don't invent URLs
that aren't in the list — only group what I've pasted.

[paste your URL list here]

What to expect back: a small set of named clusters with counts, plus a flag on which cluster looks like the biggest crawl-waste offender — treat it as a starting point to verify against the actual paths, not a final answer.

Prioritize a status-code breakdown

I have this table of HTTP status codes and how many times verified Googlebot
hit each one over the last 30 days. Rank them by which I should fix first,
weighting frequency over severity — a 404 hit 5,000 times matters more than a
500 hit twice. Explain the reasoning in one line per row.

status_code, hit_count
[paste your table here]

What to expect back: the same rows re-ordered by fix priority with a one-line reason each — sanity-check the reasoning against your own site context before acting.

Draft a log-access request to DevOps

Write a short, plain-English email to my DevOps/hosting team asking for
30-90 days of raw web server access logs (Apache/Nginx combined format, or
our CDN's edge logs if we're behind one) for [site name]. Explain in one
sentence why I need it (verifying real Googlebot/Bingbot crawl activity vs
GSC's sampled report) and ask what export format and delivery method works
for them.

What to expect back: a short draft email you can edit with your actual site name and send — review it yourself before sending, this tool won’t send messages on your behalf.

Explain a DNS verification result

I ran a reverse DNS lookup on an IP from my server logs and then a forward
DNS lookup on the hostname it returned. Here's the raw output from the
`host` command. Tell me plainly whether this confirms the request came from
real Googlebot or Bingbot, and point to exactly which line proves or
disproves it.

[paste your host/nslookup output here]

What to expect back: a plain-language verdict tied to the specific line in your output — treat it as a second opinion, not a replacement for knowing the actual rule (hostname ends in a Google/Bing domain, and the forward lookup returns the original IP).

Resources worth your time

My related writing

• How to Do an SEO Log File Analysis [Template Included] — the Ahrefs cornerstone guide I reviewed; the framing and template this article leans on.
• When Should You Worry About Crawl Budget? — when log analysis is (and isn’t) worth your time.
• What Is Googlebot & How Does It Work? — the crawlers and IP-verification background.
• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — why your logs look different in 2026.
• The Beginner’s Guide to Technical SEO — where crawling and logs fit in the bigger picture.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawling: Googlebot as 1,000+ systems with many specialized crawlers (Desktop, Mobile, Image, News, Video, Ads) sharing one crawl-budget pool, and requests mostly originating out of Mountain View — a handy log sanity-check alongside, never instead of, proper verification. (Standing disclaimer: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• Screaming Frog Log File Analyser — user guide and AI-bot tutorial.
• Search Engine Land — Log file analysis guide (Kody Wirth) — practical walkthrough covering formats, tools, and what patterns to look for.
• Search Engine Journal — How to Use Google’s Crawl Stats Report — Daniel Waisberg’s (Google) guidance on reading Crawl Stats, including the response-time warning; the best companion to log analysis.
• Search Engine Journal — Indexing & Crawl Budget (Illyes + Splitt) — on-the-record definition of crawl budget and how quality drives crawl demand.
• Search Engine Roundtable — Bingbot IP addresses released — covers the nuance that Microsoft eventually published a bingbot IP JSON even though official guidance still centers on DNS.
• Conductor — Log File Analysis for SEO — accessible explainer of what logs reveal and how to act on them.
• r/TechSEO — the community for crawl/index debugging.

Videos

• Google Search Central (YouTube) — Martin Splitt’s crawling/rendering explainers and the How Google Search Works series; useful background on the bots whose hits you’re verifying in your logs. Channel

Test yourself

Five questions on verifying bots and reading what your logs actually show.

Log file analysis

Log file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened.

Related: Crawling, Crawl Budget

Log file analysis

Log file analysis (for SEO) is the practice of reading a web server’s raw access logs — the server’s own line-by-line record of every request it received — to see exactly which URLs search engine crawlers (Googlebot, Bingbot, and now AI bots) actually requested, when, how often, and what HTTP status code the server returned. Where a crawl tool simulates crawler behavior and Google Search Console samples and summarizes it, the logs are the ground-truth record of what really happened on the server.

A typical log line contains an IP address, user agent, URL path, timestamp, request method (GET/POST), and HTTP status code. Logs come in a handful of formats depending on the server — Apache (the combined format, also used by Nginx), W3C (Microsoft IIS), ELB (Amazon), or Custom — and at CDN-fronted sites you often need the edge logs too, since origin logs miss cache-served hits.

The single most important step is verification. User agents are trivially spoofed, so before you analyze anything you confirm each “Googlebot” or “Bingbot” line is genuine — via reverse + forward DNS, or (for Google) by matching the request IP against Google’s published IP-range JSON files. Bing notably does not officially publish fixed IP ranges, so for Bingbot you rely on reverse DNS to a *.search.msn.com hostname. All crawl math should be done on the verified set only.

Once verified, log analysis answers questions no other tool can with the same fidelity: which URLs and sections get the most (and least) crawl attention, how crawl frequency changes over time, which status codes bots are hitting and how often, where crawl budget is being wasted (parameters, faceted nav, internal search, infinite pagination), which pages bots never reach (orphans), and how the mobile-vs-desktop Googlebot split looks. Logs complement — they don’t replace — the GSC Crawl Stats report.

Related: Crawling, Crawl Budget

Changelog