Log File Analysis

TL;DR — Your web server keeps a log of every request it gets, including visits from search engine botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. like GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.. Log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. means reading that log to see exactly which of your pages the bots actually fetched, how often, and whether they hit errors. It’s the only place that shows what really happened — but most small sites don’t need it.

What a log file is

Every time anyone — a person, GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., or some random bot — requests a page from your site, your server writes a line about it in a file. That file is your access log. Each line records roughly the same things:

• who asked (an IP address and a “user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target.” name like Googlebot),
• what they asked for (the URL),
• when (a timestamp), and
• what they got back (an HTTP status codeAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. — 200 for OK, 404 for not foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore., and so on).

Stack up weeks of those lines and you have a complete, honest record of how search engines crawl your site. Not an estimate. The actual requests.

Why bother reading it

Other SEO tools either guess at how bots crawl you (a crawl tool pretends to be a search engine and walks your site) or summarize it (Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. shows you a sampled, rounded-off view). Your logs show the real thing, request by request. That lets you answer questions like:

• Which pages does Googlebot actually visit — and which does it ignore?
• Is the bot wasting time on junk URLs instead of your important pages?
• Are bots hitting broken pages (404) or server errors (5xx)?
• Are there pages bots have never reached at all?

The one rule you can’t skip

Anyone can pretend to be Googlebot. The user-agent name in a log line is just text — a scraper can put Googlebot in there to sneak past your defenses. So before you trust a single “Googlebot” line, you have to verify it’s really Google (there’s a simple DNS check for this in the Advanced and Scripts tabs). Skip verification and you’ll draw conclusions from fake traffic.

Do you even need this?

Honestly? Probably not, if you run a small site. Log file analysis pays off for large sites — tens of thousands of URLs, ecommerce with lots of filtered pages, sites going through a migration, or sites where Google says it “discovered” pages but never indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. them. If your site has a few hundred pages and they all get crawled fine, your time is better spent elsewhere. (Same logic as crawl budget — most sites don’t need to worry about it.)

Want the real workflow — verifying bots properly, finding crawl waste, spotting orphan pagesAn orphan page is a page on your site that no other page links to internally. Because crawlers discover pages by following links, an orphan page is effectively invisible to search engines unless it's in an XML sitemap or linked from an external site.? Switch to the Advanced tab.

TL;DR — Logs are the unsampled ground truth for crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.: every request, every botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., every status code. The non-negotiable first step is verifying GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer./BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. via reverse + forward DNS (or Google’s published IP-range JSON) — user agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. are spoofed constantly, and you do all crawl math on the verified set only. Then read the logs for most/least-crawled URLs and sections, crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. over time, status codes prioritized by frequency, crawl waste (params, facets, internal search, infinite paginationPagination splits a large set of content — product listings, blog archives, search results — across multiple sequentially numbered URLs. For SEO, each paginated page should be crawlable, indexable, and self-canonical; Google no longer uses rel=prev/next, but Bing still does.), orphan and uncrawled pages (cross-referenced against a crawl), and the mobile-vs-desktop GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. split. Bing publishes no official IP ranges, so DNS to *.search.msn.com is the method there. Logs complement GSC Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). — they don’t replace it. And in 2026, AI bots are now a huge share of what shows up.

Why logs are the ground truth

There are three ways to “see” how search engines crawl you, and they are not equal:

• A crawl tool (Screaming Frog SEO Spider, Ahrefs Site Audit) simulates a crawl. It tells you what a bot could find, not what Google did fetch.
• GSC Crawl Stats summarizes the real thing, but it’s sampled, aggregated, and capped (roughly 1,000 rows, ~90 days, no per-URL export).
• Server logs record the real thing — every request, for every bot, with the exact URL, timestamp, and status code.

The Ahrefs log-file guide I reviewed puts it plainly: server logs are “the most trustworthy source of information to understand the URLs that search engines have crawled.” That’s the whole reason this technique exists. When I want to know what Googlebot actually did — not what it might do, not a rounded summary — I go to the logs.

A typical log line carries the IP address, user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., URL path, timestamp, request method (GET/POST), and HTTP status codeAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index.. Everything below is just slicing those fields intelligently.

When you actually need it (and when you don’t)

Be honest with yourself here. Log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. is a large-site tool. It earns its keep on sites with tens of thousands of URLs, ecommerce and faceted navigationFaceted navigation (faceted search, product filtering) lets visitors refine a list of products or content by attribute — price, color, size, brand, rating. The SEO problem: each filter combination can spawn a distinct crawlable URL, turning a small catalog into millions of near-duplicate pages that waste crawl budget and dilute ranking signals., sites mid-migration, and sites stuck in Discovered – currently not indexed. As I wrote in my crawl budget guide, “Most sites don’t need to worry about crawl budget, but there are few cases where you may want to take a look.” Daniel Waisberg of Google has made a similar point about Crawl Stats — per Search Engine Journal’s coverage, the report isn’t much of a concern for sites under ~1,000 pages.

If your few-hundred-page site is being crawled fine, skip this and go fix something with more leverage.

How to get your logs (the hardest part is access)

Logs live wherever the request actually terminated:

• Apache & Nginx → the Apache “combined” log format (the most common).
• Microsoft IIS → W3C format.
• AWS ELB/ALB → ELB format.
• CDNs (Cloudflare, Fastly, Akamai) → their own log exports. This matters: at a CDN-fronted site, an origin-only log misses edge-cached hits, so pull logs at the layer the bot actually reached.

Aim for 30 days minimum, 90 ideal, to capture crawl-frequency variation. And plan for friction — getting access to server logs is often the genuinely hard part (DevOps gatekeeping). Even Googlers, in a migration episode of Search Off the Record, flagged how hard log files can be to obtain in practice. Budget time for the request.

Logs aren’t just bot traffic — they capture every request, including real visitors, and can carry query-string values, session identifiers, or other sensitive data alongside the URL path. OWASP’s logging guidance is blunt about this: authentication credentials, access tokens, and personally identifiable information generally shouldn’t land directly in a log; they should be removed, masked, or hashed first. Build that into your access controls and export process before you hand a log file to anyone for analysis, not after.

Step 1 — Verify the bots are real (do this before anything else)

This is the step most guides wave at in one line. Do not. Many bots pretend to be Googlebot to get past firewalls (Ahrefs). The user agent is unauthenticated text; treat every “Googlebot” line as a claim to be proven.

Googlebot — two valid methods:

1. Reverse + forward DNS (the bidirectional check). Google’s own steps: run a reverse DNS lookup on the IP from your logs with the host command; verify the domain is googlebot.com, google.com, or googleusercontent.com; then run a forward DNS lookup on that hostname and verify it resolves back to the original IP. The forward step is what makes this trustworthy — a spoofer can point reverse DNS at a *.googlebot.com name, but only the round-trip back to the same IP proves it. (Commands for macOS/Linux and Windows are in the Scripts tab.)
2. Match against Google’s published IP ranges. Google publishes JSON files of its crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. IPs in CIDR format — common-crawlers.json for Googlebot and friends, plus special-crawlers.json, the user-triggered-fetcher files, and an all-Google goog.json. As I noted in my Googlebot guide, Google “provided a list of public IPs you can use to verify the requests are from Google… You can compare this to the data in your server logs.”

BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. — DNS only. This is the key contrast: Bing does not officially publish IP ranges. Bing’s own wording is that “…like other search engines, Bing does not publish a list of IP addresses or ranges from which we crawl the Internet,” because “the IP addresses or ranges we use can change any time.” So for Bingbot you do reverse + forward DNS to a hostname ending in *.search.msn.com (e.g. msnbot-157-55-33-18.search.msn.com), or use the Verify Bingbot tool. (Microsoft has since released a bingbot IP JSON, but its official verification guidance still centers on DNS precisely because IPs change.)

Then throw out the fakes. Do all of your crawl math on the verified set only. Unverified “Googlebot” is almost always a scraper or a spoofed bot and belongs in a security review, not your crawl-waste analysis.

TIP Verify identities before calculating crawl totals

This result keeps identity, crawler purpose, and response waste as separate findings instead of one blended bot score.

Upload a representative access-log sample and inspect the exact requests behind each finding with my free Log File Analyzer Free

1. Include IP, user-agent, path, status, and timestamp fields.
2. Resolve crawler range mismatches before aggregating traffic under a famous bot name.
3. Use the status and path tables for crawl decisions; logs do not prove indexing or citations.

Tap to enlarge

The sample reports 30 requests claiming a known crawler from an IP outside the operator's published ranges, a separate training-crawler traffic share, and a separate non-200 response finding. Each result has its own evidence and remedy.

Step 2 — What to look for

Once you’re working with verified hits, here’s the read:

• Most & least crawled URLs and sections. Rank requests by URL and by directory. This is where your crawl budgetThe number of URLs an engine will crawl in a timeframe. is actually going — and it’s usually surprising.
• Crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. over time. Trend crawls by URL/section to catch drops (a migration broke something) or spikes (a new section, or a spider trapA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content. spinning up infinite URLs).
• Status codes bots hit, prioritized by frequency. Quantify 200 vs. 301/302 (and chains), 404, and 5xx. A 404 hit 5,000×/week is a different problem than a 404 hit once — fix by crawl frequency, not by mere existence.
• Crawl waste. Faceted nav, URL parametersThe `?key=value` data tacked onto the end of a URL after a question mark — used for tracking, sessions, filtering, sorting, and search — and one of the biggest sources of duplicate URLs and wasted crawling in SEO., internal search results, and infinite calendars/paginationPagination splits a large set of content — product listings, blog archives, search results — across multiple sequentially numbered URLs. For SEO, each paginated page should be crawlable, indexable, and self-canonical; Google no longer uses rel=prev/next, but Bing still does. can eat a large share of crawl budget on bad offenders. Logs show exactly which junk patterns the bots are burning time on.
• Orphan & uncrawled pages. This needs both datasets. Cross-reference logs against a site crawl: URLs in the logs but not in the crawl = orphans, old redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., or externally-linked pages; URLs in the crawl but not in the logs = pages Google has never fetched.
• Mobile vs. desktop Googlebot. Split by user agent. Post mobile-first, it should be majority Googlebot Smartphone — a desktop-heavy split is worth a look.
• Response time & crawl health. Rising average response time correlates with reduced crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. Per SEJ’s write-up of Waisberg’s guidance: “Watch out for a consistent increase in average response time. Google says it might not affect crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. immediately, but it’s a good indicator that your servers might not be handling all the load.”

What logs do NOT tell you

Keep these straight or you’ll over-read the data:

• Crawl ≠ index. A URL Googlebot fetches daily can stay unindexed indefinitely. Logs prove fetching, not index status — pair them with GSC’s Page IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. / URL Inspection to learn the index side.
• Crawl ≠ rank, and more crawling doesn’t help. As I’ve said repeatedly, “The rate of crawling isn’t going to impact your rankings.” Don’t chase crawl volume as if it were a ranking lever.
• noindex doesn’t reduce crawling. noindex controls indexation, not crawling — to actually stop the crawl you use robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. or a status code.
• Crawl ≠ model training or citation. A verified hit from GPTBot, ClaudeBot, or PerplexityBot proves that request happened — a fetch at that layer. It doesn’t prove the page was used to train a model, retained anywhere downstream, or cited in a chat answer. Those are separate, unobserved outcomes; don’t stretch a verified log line further than it goes.

The 2026 wrinkle: AI bots are all over your logs now

The cast of characters in a modern log file has changed. In my analysis of Cloudflare Radar data (Meet the New Web Crawlers), search-engine bots still crawl the most — but AI bots are firmly in second place and on track to overtake them within a couple of years. GPTBot, ClaudeBot, PerplexityBot, and friends now show up heavily. When you segment your verified hits by user agent, don’t be surprised to find AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. rivaling the search engines for share of requests. (Screaming Frog’s Log File Analyser has added a dedicated AI-bot tutorial for exactly this.)

How this fits with the rest of crawling

Logs are the diagnostic layer under the whole crawling cluster. They’re how you actually measure the crawl budget spend that engines describe in the abstract (Gary Illyes defines it as “the number of URLs Googlebot can and is willing or is instructed to crawl”). They’re also how you catch spider traps red-handed — an infinite URL space from a calendar or facet shows up as a flood of near-identical requests — and how you confirm whether your crawl frequency work (accurate lastmod, internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. to important pages) actually changed bot behavior. And remember they complement, not replace, GSC Crawl Stats: Crawl Stats is the sampled on-ramp; logs are the unsampled, multi-bot, per-URL detail.

AI summary

A condensed take on the Advanced version:

• Logs = ground truth. Crawl tools simulate, GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. samples; server access logs record every request, every botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., with URL, timestamp, and status code.
• Verify before you analyze. User agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. are spoofed constantly. Confirm GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. via reverse + forward DNS or Google’s published IP-range JSON; confirm BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. via reverse DNS to *.search.msn.com (Bing publishes no official IP ranges). Do all crawl math on the verified set only.
• What to read: most/least-crawled URLs and sections; crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. over time; status codes prioritized by frequency (a 404 hit 5,000×/week ≠ once); crawl waste (params, facets, internal search, infinite paginationPagination splits a large set of content — product listings, blog archives, search results — across multiple sequentially numbered URLs. For SEO, each paginated page should be crawlable, indexable, and self-canonical; Google no longer uses rel=prev/next, but Bing still does.); orphan and uncrawled pages (cross-reference a crawl); mobile-vs-desktop GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. split; rising response time as a crawl-health warning.
• Don’t over-read it: crawl ≠ indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. ≠ rank, more crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. doesn’t help rankings, noindex doesn’t reduce crawling, and a verified AI-bot hit proves a fetch — not that the page was used for model training or cited in an answer.
• Scope: a large-site / ecommerce / migration tool. Most small sites don’t need it.
• 2026: AI bots (GPTBot, ClaudeBot, PerplexityBot) are now a major and growing share of log traffic.
• Complements GSC Crawl Stats — use both.

Official documentation

Primary-source documentation for verifying crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. and reading crawl data.

Google

• Verify Requests from Google Crawlers and Fetchers — the two official methods: manual reverse + forward DNS, and matching against the published IP ranges.
• How to verify Googlebot (Search Central Blog) — the older companion post, still cited.
• common-crawlers.json — GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. and common crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. IPs in CIDR format (note: “the IP addresses in the JSON files are represented in CIDR format”). Companions: special-crawlers.json, user-triggered-fetchers.json, and the all-Google goog.json.
• Overview of Google crawlers and fetchers — every Google user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. you’ll see in logs.
• Crawl Stats report (Help) and the launch blog — the sampled, official view that logs complement.

Bing / Microsoft

• How to Verify Bingbot (Webmaster help) — the official verification page.
• How to Verify that Bingbot is Bingbot (blog) — the canonical reverse + forward DNS method, and the statement that Bing doesn’t publish IP ranges.
• Verify Bingbot tool — paste an IP to check it.

Quotes from the source

On-the-record statements. Where supported, each link is a deep link that jumps to the quoted passage on the source page.

Google — verifying crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.

• “Run a reverse DNS lookup on the accessing IP address from your logs, using the host command.” — Google Search Central docs. Jump to quote
• “Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.” Jump to quote
• “Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name.” Jump to quote
• “Verify that it’s the same as the original accessing IP address from your logs.” Jump to quote

Bing — verification and no published IP ranges

• “Perform a reverse DNS lookup using the IP address from the logs to verify that it resolves to a name that end with search.msn.com.” — Bing WebmasterMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. Blog. Jump to quote
• “…like other search engines, Bing does not publish a list of IP addresses or ranges from which we crawl the Internet.” — and the reason: “the IP addresses or ranges we use can change any time, so responding to requests differently based on a hardcoded list is not a recommended approach.” Jump to quote

Patrick Stox — on what logs are for (from my work at Ahrefs)

• Server logs are “the most trustworthy source of information to understand the URLs that search engines have crawled.” Jump to quote
• “Many bots pretend to be GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. to get past firewalls.” Jump to quote
• “If you want to see hits from all bots and users, you’ll need access to your log files.” Jump to quote
• “The rate of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. isn’t going to impact your rankings.” Jump to quote

Gary Illyes, Google — crawl budgetThe number of URLs an engine will crawl in a timeframe. (what logs let you measure)

• Crawl budgetThe number of URLs an engine will crawl in a timeframe. is “the number of URLs GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. can and is willing or is instructed to crawl.” Read the coverage

Daniel Waisberg, Google — response time as a crawl-health signal (SEJ’s framing of his Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). guidance)

• “Watch out for a consistent increase in average response time. Google says it might not affect crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. immediately, but it’s a good indicator that your servers might not be handling all the load.” Jump to quote

Verify a bot is really Googlebot (reverse + forward DNS)

The user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. in a log line is just text — scrapers spoof Googlebot to get past firewalls. The only trustworthy check is bidirectional DNS: reverse-lookup the IP, confirm the hostname is a Google domain, then forward-lookup that hostname and confirm it resolves back to the same IP.

macOS / Linux (uses host)

# 1) Reverse DNS the IP from your logs — it must end in googlebot.com,
#    google.com, or googleusercontent.com
host 66.249.66.1
# → 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com

# 2) Forward DNS that hostname back — it must resolve to the same IP
host crawl-66-249-66-1.googlebot.com
# → crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Windows (uses nslookup)

:: 1) Reverse DNS the IP — confirm it ends in a Google domain
nslookup 66.249.66.1

:: 2) Forward DNS the returned hostname — confirm it matches the original IP
nslookup crawl-66-249-66-1.googlebot.com

If the reverse lookup doesn’t land on a Google domain, or the forward lookup doesn’t return the original IP, it is not GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.. (For BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share., run the exact same two steps but expect a hostname ending in search.msn.com.) You can also skip DNS and match the IP against Google’s published ranges (common-crawlers.json, in CIDR format).

Extract and count crawler hits from a raw log

Quick one-liners for a combined-format Apache/Nginx access log. (These select on the user-agent string — remember to verify the IPs before trusting the counts.)

# Pull only the lines claiming to be Googlebot
grep -i "googlebot" access.log > googlebot-hits.log

# Count Googlebot hits per URL, most-crawled first
# (combined format: $7 is the request path)
grep -i "googlebot" access.log \
  | awk '{print $7}' \
  | sort | uniq -c | sort -rn | head -50

# Count Googlebot hits per status code (combined format: $9 is the status)
grep -i "googlebot" access.log \
  | awk '{print $9}' \
  | sort | uniq -c | sort -rn

# List the URLs Googlebot hit that returned a 404, by frequency
grep -i "googlebot" access.log \
  | awk '$9 == 404 {print $7}' \
  | sort | uniq -c | sort -rn

# Get the unique IPs claiming Googlebot — the list you then verify by DNS
grep -i "googlebot" access.log | awk '{print $1}' | sort -u

Adjust the $7/$9 field positions if your log format differs (IIS/W3C and ELB order fields differently).

Log file analysis workflow

• Get the logs — server access logs, or CDN/load-balancer exports. At a CDN-fronted site, pull edge logs too (origin misses cache hits).
• Grab enough window — 30 days minimum, 90 ideal.
• Verify the botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. first — reverse + forward DNS, or Google’s published IP ranges. BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. → DNS to *.search.msn.com.
• Drop the fakes — do all crawl math on the verified set; route spoofed “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.” to a security review.
• Rank most/least-crawled URLs and sections — see where budget goes.
• Trend crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. over time — catch drops (migrations) and spikes (new sections, spider trapsA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content.).
• Tally status codes — 200 / 301-302 chains / 404 / 5xx, and prioritize fixes by crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial., not mere existence.
• Hunt crawl waste — parameters, faceted nav, internal search, infinite paginationPagination splits a large set of content — product listings, blog archives, search results — across multiple sequentially numbered URLs. For SEO, each paginated page should be crawlable, indexable, and self-canonical; Google no longer uses rel=prev/next, but Bing still does./calendars.
• Find orphans & uncrawled pages — cross-reference logs against a site crawl (in logs not crawl = orphan; in crawl not logs = never fetched).
• Check the mobile vs. desktop GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. split — should be Smartphone-majority.
• Watch average response time — a rising trend can throttle crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..
• Segment AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. — GPTBot, ClaudeBot, PerplexityBot, etc. are now a real share of traffic.
• Confirm with GSC — pair crawl findings with Crawl Stats and URL Inspection (crawl ≠ index).

What to build in your analysis sheet

Whether you use Screaming Frog’s Log File Analyser, BigQuery, or the Ahrefs log-file template, you’re building the same handful of views. After verifying botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., parse each line into columns and pivot.

Columns to parse out of each log line

Pivots to build

• Crawls per URL (and per directory), descending — most & least crawled.
• Crawls per status code, then status code × URL (so a high-frequency 404 jumps out).
• Crawls per day/week, segmented by section — the frequency trend.
• Crawls per bot/user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. — the mobile/desktop split, plus AI-bot share.
• A logs vs. crawl join (VLOOKUP/merge against a Screaming Frog or Ahrefs crawl export) to surface orphans and never-fetched pages.

The Ahrefs guide ships a downloadable template that sets these up; the log-file-analysis guide links it.

How to read a log file: what to look for

A repeatable lens for any log analysis. Verify first; then run these passes.

1. Verify, then analyze. The data is only as good as the botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. identity behind it. Reverse + forward DNS (or IP ranges); analyze the verified set only. Spoofed bots → security, not SEO.

2. Where is budget going? (most vs. least crawled.) Rank by URL and by section. The goal is to find attention spent on the wrong places — and important pages getting too little.

3. What are bots hitting? (status codes, by frequency.) 200 is healthy; 3xx chains, 404, and 5xx are leaks. Triage by how often the bot hits each, not by whether the error merely exists.

4. What’s being wasted? (crawl waste.) Parameters, faceted nav, internal search, infinite paginationPagination splits a large set of content — product listings, blog archives, search results — across multiple sequentially numbered URLs. For SEO, each paginated page should be crawlable, indexable, and self-canonical; Google no longer uses rel=prev/next, but Bing still does./calendars — the classic budget sinks. Logs name the exact offending patterns.

5. What’s missing? (orphans & uncrawled.) Cross-reference logs with a crawl. In logs but not crawl = orphan/old redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't./external link. In crawl but not logs = Google never fetched it.

6. Who’s crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.? (bot segmentation.) Mobile vs. desktop GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. (should be Smartphone-majority), plus the AI-bot share that now rivals search engines.

7. Is the server healthy? (response time.) Rising average response time is an early warning that crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. may get throttled.

Tools for log file analysis

• Screaming Frog SEO Log File Analyser — the workhorse desktop tool. It auto-verifies search botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. and flags spoofed IPs, and has User-AgentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. + Verification-Status filters for granular bot analysis. Import a SEO Spider crawl and use the “Not In URL Data” filter to find orphans — “URLs which were discovered in your logs, but are not present in the crawl data imported.” It also now has a dedicated AI-bot monitoring tutorial. Tool
• BigQuery (and Splunk / ELK Stack / Logflare / logz.io) — for raw, large-scale storage and querying when log volume is too big for a desktop tool.
• Ahrefs — cross-reference logs against an Ahrefs Site Audit crawl to find orphans and confirm what the bots reached. (My home tool.)
• Semrush Log File Analyzer, OnCrawl, Botify, JetOctopus — other crawl+log platforms, the latter three aimed at enterprise.
• GSC Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). — the official, free, sampled on-ramp. Complementary to raw logs, not a replacement: it’s aggregated, capped, and has no per-URL export.
• Verify BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. tool — paste an IP to confirm it’s really BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.: bing.com/toolbox/verify-bingbot.

Mistakes to avoid

• Trusting the “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.” string without verifying it. Why it’s wrong: the user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. is unauthenticated text — anyone can put Googlebot in a request header to slip past defenses or pollute your analysis with fake traffic. Do instead: run reverse + forward DNS (or match against Google’s published IP ranges) before a single line counts toward your crawl math.
• Treating GSC Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). as the full picture. Why it’s wrong: it’s sampled, rounded, capped at roughly 1,000 rows and ~90 days, with no per-URL export — it summarizes, it doesn’t record. Do instead: use logs as the ground truth and Crawl Stats as a complementary, faster on-ramp.
• Pulling origin-only logs on a CDN-fronted site. Why it’s wrong: an origin log misses every request the CDN served from edge cache, so you’re analyzing an incomplete picture of what botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. actually got. Do instead: pull logs at the layer the bot actually reached — the CDN/edge export, not just the origin server.
• Fixing 404s and 5xx errorsThe 4xx (client error) and 5xx (server error) HTTP status codes a server returns instead of a successful 2xx — and how Google treats each class very differently. by existence, not frequency. Why it’s wrong: a 404 hit once a month and a 404 hit 5,000 times a week are not the same problem, but treating every error line as equally urgent wastes fix effort. Do instead: rank status-code issues by how often bots actually hit them.
• Chasing crawl volume as if it were a ranking lever. Why it’s wrong: more crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. doesn’t move rankings — it’s a diagnostic signal, not a growth metric. Do instead: use crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. to catch problems (drops, spider traps), not as a KPI to maximize.
• Using noindex to try to stop crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. Why it’s wrong: noindex controls indexation, not crawling — Google still has to fetch the page to see the tag. Do instead: block the crawl itself with robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. or a status code if that’s the actual goal.
• Calling a URL “orphaned” from logs alone. Why it’s wrong: a URL that shows up in logs but not in a fresh crawl could be an old redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., an external link, or a genuine orphan — logs alone can’t tell you which. Do instead: cross-reference logs against an actual site crawl before drawing conclusions either way.

Standing KPIs for log file analysis

Verified-bot share

• Metric: Verified GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer./BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. hits ÷ all hits claiming to be GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer./BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. by user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target..
• What it tells you: How much of your “botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. traffic” is actually spoofed scrapers rather than real search engines.
• How to pull it: Run every claimed-bot IP through reverse + forward DNS (or the IP-range JSON) and count the pass rate.
• Benchmark / realistic range: No universal number — depends on how aggressively your site is scraped. A low or dropping verified share is the signal to act on, not a fixed threshold.
• Cadence: Every time you pull a new log window.

Crawl waste share

• Metric: % of verified bot requests hitting parameters, facets, internal search, or infinite paginationPagination splits a large set of content — product listings, blog archives, search results — across multiple sequentially numbered URLs. For SEO, each paginated page should be crawlable, indexable, and self-canonical; Google no longer uses rel=prev/next, but Bing still does./calendars.
• What it tells you: How much of your crawl budgetThe number of URLs an engine will crawl in a timeframe. is going to junk URLs instead of pages that matter.
• How to pull it: Segment verified hits by URL pattern (query stringsThe `?key=value` data tacked onto the end of a URL after a question mark — used for tracking, sessions, filtering, sorting, and search — and one of the biggest sources of duplicate URLs and wasted crawling in SEO., known facet/search paths).
• Benchmark / realistic range: No universal figure — genuinely situational, depending on your site’s URL structureURL structure is how the parts of a web address — scheme, domain, path, query string, and fragment — are organized and formatted. It mostly affects crawling, usability, and how engines understand a page, not rankings directly. and faceting. Establish your own baseline on the first pull, then track the trend.
• Cadence: 30–90 day log window; re-check after any cleanup (robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. rules, parameter handling, paginationPagination splits a large set of content — product listings, blog archives, search results — across multiple sequentially numbered URLs. For SEO, each paginated page should be crawlable, indexable, and self-canonical; Google no longer uses rel=prev/next, but Bing still does. fixes).

Status-code mix, weighted by frequency

• Metric: 200 / 3xx / 404 / 5xx as a share of verified bot requests.
• What it tells you: Where bots are burning fetches on errors instead of live content, and whether that’s getting worse.
• How to pull it: Tally the status-code field from verified log lines.
• Benchmark / realistic range: No universal target — depends on site age and redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. history. Watch the trend, not a single snapshot; a rising 404/5xx share is the actionable signal.
• Cadence: 30–90 days, or immediately after a migration.

Mobile vs. desktop Googlebot split

• Metric: Share of verified Googlebot hits from the Smartphone user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. vs. Desktop.
• What it tells you: Whether Google is actually crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. you mobile-first, as expected post mobile-first indexingGoogle's practice of using the mobile version of a page's content — crawled by Googlebot smartphone — for indexing and ranking. It is not a separate index and not a ranking boost..
• How to pull it: Segment verified hits by the Googlebot UA string (Smartphone vs. Desktop).
• Benchmark / realistic range: Should be Smartphone-majority for most sites; a desktop-heavy split is worth investigating, not a hard failure on its own.
• Cadence: Each log pull.

Orphan / uncrawled page count

• Metric: URLs in logs but not in a fresh crawl (orphans/old links), and URLs in a fresh crawl but not in logs (never fetched).
• What it tells you: Pages bots can’t easily reach, and pages you’re linking to that Google has never bothered to fetch.
• How to pull it: Join the verified-hit URL list against a site crawl export (Screaming Frog, Ahrefs).
• Benchmark / realistic range: Entirely situational — depends on site size and how recently you migrated or restructured. Track the count over time rather than comparing to an outside number.
• Cadence: Quarterly for large sites, or immediately after a migration.

Ready-to-copy prompts for log analysis

These are for interpreting log data you’ve already pulled and verified — not for generating log data (never let an AI invent log lines or stats).

Summarize crawl waste from a URL sample

Here is a list of URL paths that verified Googlebot hits landed on, one per
line, from my server logs. Group them into patterns (query parameters,
faceted navigation, internal search, pagination/calendars, or "looks like a
real page"), and tell me which pattern has the most URLs. Don't invent URLs
that aren't in the list — only group what I've pasted.

[paste your URL list here]

What to expect back: a small set of named clusters with counts, plus a flag on which cluster looks like the biggest crawl-waste offender — treat it as a starting point to verify against the actual paths, not a final answer.

Prioritize a status-code breakdown

I have this table of HTTP status codes and how many times verified Googlebot
hit each one over the last 30 days. Rank them by which I should fix first,
weighting frequency over severity — a 404 hit 5,000 times matters more than a
500 hit twice. Explain the reasoning in one line per row.

status_code, hit_count
[paste your table here]

What to expect back: the same rows re-ordered by fix priority with a one-line reason each — sanity-check the reasoning against your own site context before acting.

Draft a log-access request to DevOps

Write a short, plain-English email to my DevOps/hosting team asking for
30-90 days of raw web server access logs (Apache/Nginx combined format, or
our CDN's edge logs if we're behind one) for [site name]. Explain in one
sentence why I need it (verifying real Googlebot/Bingbot crawl activity vs
GSC's sampled report) and ask what export format and delivery method works
for them.

What to expect back: a short draft email you can edit with your actual site name and send — review it yourself before sending, this tool won’t send messages on your behalf.

Explain a DNS verification result

I ran a reverse DNS lookup on an IP from my server logs and then a forward
DNS lookup on the hostname it returned. Here's the raw output from the
`host` command. Tell me plainly whether this confirms the request came from
real Googlebot or Bingbot, and point to exactly which line proves or
disproves it.

[paste your host/nslookup output here]

What to expect back: a plain-language verdict tied to the specific line in your output — treat it as a second opinion, not a replacement for knowing the actual rule (hostname ends in a Google/Bing domain, and the forward lookup returns the original IP).

Resources worth your time

My related writing

• How to Do an SEO Log File Analysis [Template Included] — the Ahrefs cornerstone guide I reviewed; the framing and template this article leans on.
• When Should You Worry About Crawl Budget? — when log analysis is (and isn’t) worth your time.
• What Is Googlebot & How Does It Work? — the crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. and IP-verification background.
• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — why your logs look different in 2026.
• The Beginner’s Guide to Technical SEO — where crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and logs fit in the bigger picture.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.: GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. as 1,000+ systems with many specialized crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. (Desktop, Mobile, Image, News, Video, Ads) sharing one crawl-budget pool, and requests mostly originating out of Mountain View — a handy log sanity-check alongside, never instead of, proper verification. (Standing disclaimer: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• Screaming Frog Log File Analyser — user guide and AI-bot tutorial.
• Search Engine Land — Log file analysis guide (Kody Wirth) — practical walkthrough covering formats, tools, and what patterns to look for.
• Search Engine Journal — How to Use Google’s Crawl Stats Report — Daniel Waisberg’s (Google) guidance on reading Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root)., including the response-time warning; the best companion to log analysis.
• Search Engine Journal — Indexing & Crawl Budget (Illyes + Splitt) — on-the-record definition of crawl budgetThe number of URLs an engine will crawl in a timeframe. and how quality drives crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side..
• Search Engine Roundtable — Bingbot IP addresses released — covers the nuance that Microsoft eventually published a bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. IP JSON even though official guidance still centers on DNS.
• Conductor — Log File Analysis for SEO — accessible explainer of what logs reveal and how to act on them.
• r/TechSEO — the community for crawl/indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. debugging.

Videos

• Google Search Central (YouTube) — Martin Splitt’s crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor./renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. explainers and the How Google Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank. series; useful background on the botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. whose hits you’re verifying in your logs. Channel

Test yourself

Five questions on verifying botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. and reading what your logs actually show.

Log file analysis

Log file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened.

Related: Crawling, Crawl Budget

Log file analysis

Log file analysis (for SEO) is the practice of reading a web server’s raw access logs — the server’s own line-by-line record of every request it received — to see exactly which URLs search engine crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. (GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share., and now AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls.) actually requested, when, how often, and what HTTP status codeAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. the server returned. Where a crawl tool simulates crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. behavior and Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. samples and summarizes it, the logs are the ground-truth record of what really happened on the server.

A typical log line contains an IP address, user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., URL path, timestamp, request method (GET/POST), and HTTP status codeAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index.. Logs come in a handful of formats depending on the server — Apache (the combined format, also used by Nginx), W3C (Microsoft IIS), ELB (Amazon), or Custom — and at CDN-fronted sites you often need the edge logs too, since origin logs miss cache-served hits.

The single most important step is verification. User agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. are trivially spoofed, so before you analyze anything you confirm each “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.” or “BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.” line is genuine — via reverse + forward DNS, or (for Google) by matching the request IP against Google’s published IP-range JSON files. Bing notably does not officially publish fixed IP ranges, so for Bingbot you rely on reverse DNS to a *.search.msn.com hostname. All crawl math should be done on the verified set only.

Once verified, log analysis answers questions no other tool can with the same fidelity: which URLs and sections get the most (and least) crawl attention, how crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. changes over time, which status codes bots are hitting and how often, where crawl budgetThe number of URLs an engine will crawl in a timeframe. is being wasted (parameters, faceted nav, internal search, infinite paginationPagination splits a large set of content — product listings, blog archives, search results — across multiple sequentially numbered URLs. For SEO, each paginated page should be crawlable, indexable, and self-canonical; Google no longer uses rel=prev/next, but Bing still does.), which pages bots never reach (orphans), and how the mobile-vs-desktop Googlebot split looks. Logs complement — they don’t replace — the GSC Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root)..

Related: Crawling, Crawl Budget

Changelog