Crawl Stats Report

TL;DR — The Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). shows how Google has been crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. your site over the last 90 days — how many times it asked for pages, how much it downloaded, and how fast your server responded. You find it under Settings in Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results., and it is available for a root-level property rather than a path-scoped property. For most sites it’s a health check, not something you need to optimize.

What the Crawl Stats report is

When Google’s crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. — visits your site, every request it makes gets logged on Google’s side. The Crawl Stats report is where Google shows you a summary of that activity: how often it came by, what it downloaded, and whether it ran into problems.

You’ll find it in Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. under Settings → Crawl stats (it’s not in the main sidebar with the other reports). It covers roughly the last 90 days.

Where to find it (and why you might not see it)

The most common reason people can’t find the report is simple: it only shows up for root-level properties. That covers a domain property (just the domain, no https:// prefix) and a URL-prefix property verified at the site’s root (like https://example.com). If you added a URL-prefix property pointing at a subfolder or subpath, you won’t get the report.

The fix is to verify a root-level property — domain or root URL-prefix. Same site, different property type — and the report appears.

The three numbers up top

• Total crawl requests — how many times GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. asked for a URL on your site, whether it succeeded or not.
• Total download size — how many bytes Google downloaded while crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..
• Average response time — how long, on average, your server took to respond.

These are trends to watch, not targets to hit. There’s no “right” number. What matters is the shape of the line over time.

Should you even worry about this?

Probably not, honestly. There’s a lot of anxiety around “crawl budgetThe number of URLs an engine will crawl in a timeframe.,” but most sites never need to think about it. The report is far more useful as an early-warning system for server problems than as a dashboard to optimize.

The thing to watch for is your server struggling. If your average response time climbs or you start seeing server errors, Google notices and crawls you less to avoid making things worse. So if pages suddenly aren’t getting picked up, this report is one of the first places I’d look.

There’s also a Host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. summary here that flags whether Google had trouble reaching your site at all (fetching robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., DNS, connecting to the server). That’s the most important block — if it’s red, fix that first.

Want the full metric-by-metric breakdown and what each pattern means? Switch to the Advanced tab.

TL;DR — Crawl Stats (Settings → Crawl stats, root-level properties only — Domain or root URL-prefix) reports total crawl requests, total download size, and average response time over ~90 days, plus breakdowns by response, file type, GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. type, and purpose (Discovery vs Refresh). Read it as a server-health monitor, not a crawl-budget vanity dashboard: rising response time or 5xx makes Google reduce crawl capacityThe number of URLs an engine will crawl in a timeframe., and a sustained robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. failure can halt crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. It’s the closest GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. gets to log data, but it covers Google’s crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. only and isn’t guaranteed to be request-complete — for all bots, users, and a complete record you still need server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened.. Google scopes its own crawl-budget guidance to large, frequently-changing sites; for most smaller sites, in my experience, it almost never matters.

What it is and where it lives

Google’s own framing is direct: “The Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). shows you statistics about Google’s crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. history on your website.” You get to it under Settings → Crawl stats — not in the main report sidebar, which trips a lot of people up.

The single biggest reason people think the report is “missing” or “broken”: Google only shows it for root-level properties. In their words, “This report is available only for root-level properties.” That covers both a Domain property and a URL-prefix property verified at the site’s root (for example, https:// example.com) — it isn’t limited to protocol-less Domain properties. If you verified a URL-prefix property scoped to a subfolder or subpath, re-verify at the root level and it appears. There’s nothing wrong with your site.

Should you care? The crawl-budget reframe

Most sites do not need to optimize crawl budgetThe number of URLs an engine will crawl in a timeframe., and the report’s real value is health monitoring rather than budget tuning. Google scopes its own crawl-budget guidance to large sites (1M+ unique pages) and medium-or-larger sites (10k+ pages with daily change). Below that, this is a diagnostic, not a project.

Google also frames the report itself as built for advanced users, and says sites under roughly 1,000 pages generally don’t need to dig into it at this granularity — a smaller threshold than the crawl-budget guidance above. My own rule of thumb is looser: for most sites under about 10,000 pages, I still wouldn’t spend much time on crawl budget. Treat that as practitioner judgment, not an official cutoff.

Crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. is governed by two things working together — crawl capacity limit and crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side.. Capacity is what your server can handle. Google defines the capacity limit as “The maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches.” Demand is how much Google wants to crawl, driven by factors like popularity and staleness. The report is where the consequences of both show up. (More on the concept in crawl budget.)

A few scope details worth knowing before you read the numbers: the report counts the actual URL GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. requested rather than the canonical it resolves to, and it counts duplicate requests and each server-side redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. hop separately (client-side redirects aren’t counted as redirect responses here). It’s scoped to the property or host you’re viewing — resources on other domains, and sometimes sibling or child domains, can be under- or over-represented depending on property type. And per Google’s own documentation, the report reflects most crawl requests but can omit some, so a small mismatch against your own server logs is expected, not a bug.

This is the closest thing Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. gives you to log-file data — but it’s Google’s crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. only, aggregated, and not guaranteed to be request-complete. If you want hits from all bots and users, or a guaranteed complete record, you need your server logs. Treat Crawl Stats as the free first look before you invest in log analysis.

The three top-line metrics

Read these as trends, not targets:

• Total crawl requests — “The total number of crawl requests issued for URLs on your site, whether successful or not.” Every request counts, including the ones that errored.
• Total download size — “Total number of bytes downloaded from your site during crawling, for the specified time period.”
• Average response time — “Average response time for all resources fetched from your site during the specified time period.” This is effectively a server-health proxy. A sustained spike is a performance alarm, and Google may respond by dropping your crawl capacity limit — so the SEO harm is indirect (slower, less crawling), separate from any page-speed ranking factor.

Breakdown: By response

This table, in Google’s words, shows “the responses that Google received when crawling your site, grouped by response type, as a percentage of all crawl responses.” The response types include OK (200), Moved permanently (301), Moved temporarily (302), Not found (404), Server error (5XX), and robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. not available.

How I read each pattern:

• 200 should be the large majority. Google doesn’t publish a specific target percentage, so judge this against your own baseline and watch for sustained shifts rather than chasing an exact number.
• Lots of 404s isn’t fatal but signals wasted crawl — Google spending requests on URLs that don’t exist.
• Lots of 301s points at redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency. or internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. still aimed at old URLs. Fix the links to point at the final destination.
• A 5xx spike is the most damaging pattern in the report. Your server is erroring under crawl load, which drops the capacity limit, which means less crawling. Sustained 5xx can effectively throttle Google to a crawl.
• robots.txt not available shouldn’t show up in any sustained, meaningful way. Google doesn’t publish a fixed target percentage, but a persistent share means Google periodically couldn’t fetch your robots.txt. That matters because if your robots.txt stays unavailable, Google will conservatively pause crawling for a while rather than guess. (For what robots.txt does and doesn’t control, see robots.txt.)

Breakdown: By file type

Google groups crawling by file type — HTML, image, JavaScript, CSS, JSON, PDF, and others. This is a diagnostic lead, not proof of waste on its own — how much JS, CSS, image, or JSON crawling is normal depends on how your site is built (a JS-heavy app legitimately needs more resource crawling than a static HTML site). A JS/CSS/image share that’s high relative to your own baseline, or that jumps suddenly, is worth investigating as possible render-resource overhead. Unexpected file types showing up can reveal a crawl trap — an infinite or near-infinite URL space generating junk.

Breakdown: By Googlebot type

This splits crawling by which Googlebot did the fetching — smartphone, desktop, image, video, page resource load, and AdsBot. Google doesn’t document a required mix, but in a mobile-first crawl, Smartphone typically leads for most sites. A desktop-heavy profile, or unexpected volume from Image or AdsBot, is a lead worth investigating rather than a violation by itself — it can mean Google isn’t treating your site as mobile-first, that ad/image crawling is eating into budget you’d rather spend on content, or just that your site has unusual resource needs.

Breakdown: By purpose — Discovery vs Refresh

Two categories, and Google defines them only by prior crawl history — the labels alone don’t tell you whether the mix is healthy:

• Discovery — “The URL requested was never crawled by Google before.”
• Refresh — “A recrawl of a known page.”

A rise in Discovery is expected during a launch or migration (Google is finding your new URLs); the same rise is a problem if it’s coming from infinite-space or parameter URLs — that’s crawl budget going to garbage. A Refresh-heavy mix is normal for an established site that isn’t publishing much new content. Judge either against your own baseline and what you know changed, not against a fixed “healthy” ratio.

Host status — the most actionable block

Host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. summarizes whether Google hit availability problems trying to crawl you, grouped into three sub-checks: robots.txt fetch, DNS resolution, and server connectivity. A red host status is one of the most actionable signals in the report — it usually means Google was blocked or throttled at the infrastructure level, before it ever got to your content, during that window. It’s an aggregate alert, though, not proof that every URL was affected or that the problem is still happening now — check the timeline and confirm current status (a live fetch, URL Inspection, or your logs) before you consider it fixed. Still, treat a red host status as the first thing to investigate. (Covered in its own right under Host status.)

Reading spikes and drops

The mechanism to internalize: your server’s behavior feeds back into how much Google crawls. If the site responds quickly and reliably, Google’s capacity limit can rise; if it slows down or starts returning server errors, the limit drops. So:

• Response-time spike → capacity limit can drop → less crawling.
• 5xx surge → capacity limit drops → less crawling; sustained, it can throttle you hard.
• robots.txt unavailable → if it stays unavailable, Google pauses crawling conservatively.

None of this is a ranking penalty. The damage is indirect: if pages aren’t crawled and indexed, they can’t rank at all — but extra crawling doesn’t lift rankings either. (See the crawl budget concept for the full capacity-plus-demand model.)

Crawl Stats vs server logs

Crawl Stats is the closest GSC gets to logs, but the gaps matter:

If you outgrow Crawl Stats — you need per-URL data, non-Google bots, or longer history — that’s when you graduate to log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened.. Crawl Stats tells you something’s off; logs tell you exactly where.

TIP Move from Crawl Stats trends to the exact wasted URLs

Search Console shows Google-level trends; server logs identify the request paths and responses behind them. This sample also separates spoofed claims and AI-crawler policy from crawl waste.

Inspect a log sample with my free Log File Analyzer Free

1. Export a representative access-log window that overlaps the Search Console anomaly.
2. Verify crawler identity and group requests by status, path, and crawler purpose.
3. Fix broken links, redirect chains, and server errors, then watch both logs and Crawl Stats for recovery.

Tap to enlarge

The log sample reports 30 requests spoofing a verified crawler, AI-training crawlers at 32.1 percent of crawler traffic, and Googlebot spending 49.6 percent of 129 requests on non-200 responses, including 404 and 410 URLs, redirects, and server errors.

How this relates to the rest of crawling

The Crawl Stats report is your view into crawling after the fact. For the indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. side of what happens next, the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. is the companion — Crawl Stats shows what Google fetched; Page Indexing shows what made it into the index. And Host status, surfaced inside this report, is the infrastructure-level signal worth leading with.

AI summary

A condensed take on the Advanced version:

• Where it is: Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. → Settings → Crawl stats. Covers ~90 days. Only available for root-level properties — a Domain property or a URL-prefix property verified at the site’s root — that’s why it’s “missing” for URL-prefix subfolders.
• Three top-line metrics: total crawl requests, total download size, average response time. Read them as trends, not targets.
• Four breakdowns: By response (200/301/404/5xx/robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. unavailable), By file type (HTML/JS/CSS/image/PDF…), By GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. type (smartphone/desktop/image/video/AdsBot), By purpose (Discovery vs Refresh).
• It’s a server-health monitor, not a budget dashboard. Rising response time or 5xx makes Google drop its crawl capacity limitCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. and crawl less; sustained robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. unavailability can pause crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..
• Host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. (robots.txt fetch, DNS, server connectivity) is the most actionable block — fix a red host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. first.
• Crawl ≠ ranking: more crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. doesn’t lift rankings, but uncrawled pages can’t rank at all.
• Closest GSC gets to logs, but it’s Google-only, aggregated, and not guaranteed request-complete — for all botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index./users and per-URL detail, use server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened..
• Google scopes official crawl-budget guidance to large, frequently-changing sites (and even flags the report itself as unnecessary detail for sites under ~1,000 pages). Practitioner rule of thumb: most sites well under 10k pages don’t need to optimize crawl budgetThe number of URLs an engine will crawl in a timeframe. — but that’s judgment, not an official cutoff.

Official documentation

Primary-source documentation from Google.

Google

• Crawl Stats report — the Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. Help page for this report: metrics, breakdowns, host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity., and the root-level-property requirement.
• Optimize your crawl budget — crawl capacity limitCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. + crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side., and who actually needs to manage budget.
• Crawling and Indexing — the broader hub for robots, sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., and crawl controls.

Quotes from the source

On-the-record statements from Google. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what the report shows

• “The Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). shows you statistics about Google’s crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. history on your website.” — Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. Help. Jump to quote
• “This report is available only for root-level properties.” Jump to quote

Google — the three metrics

• “The total number of crawl requests issued for URLs on your site, whether successful or not.” Jump to quote
• “Total number of bytes downloaded from your site during crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., for the specified time period.” Jump to quote
• “Average response time for all resources fetched from your site during the specified time period.” Jump to quote

Google — the breakdowns

• “This table shows the responses that Google received when crawling your site, grouped by response type, as a percentage of all crawl responses.” Jump to quote
• “Discovery: The URL requested was never crawled by Google before.” Jump to quote
• “Refresh: A recrawlCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. of a known page.” Jump to quote

Google — crawl capacityThe number of URLs an engine will crawl in a timeframe. (the budget mechanism behind the numbers)

• Crawl capacity limitCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.: “The maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches.” — Google Search Central, large-site crawl budgetThe number of URLs an engine will crawl in a timeframe. guide. Jump to quote

Reading the Crawl Stats report — checklist

A quick pass to get the signal out of the report:

• You’re on a domain (root-level) property — otherwise the report won’t show.
• Host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. is green (robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. fetch, DNS, server connectivity all OK). Fix any red here first.
• Average response time is stable, not trending up.
• By response is a large majority 200; 5xx and robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.-unavailable show no sustained, meaningful share (Google publishes no fixed target — watch your own trend).
• No surprise volume of 404s (wasted crawl) or 301s (redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency. / stale internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them.).
• By GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. type generally leans Smartphone in a mobile-first crawl — there’s no official required mix, but a sudden shift is worth a look.
• By file type isn’t dominated by JS/CSS/image relative to your own baseline, and shows no surprise file types (possible crawl trapA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content.).
• By purpose: judge Discovery vs Refresh against your own baseline and what changed (launch, migration, sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. update) — an unexplained jump either way is worth checking for parameter/infinite-URL waste or stalled recrawling.
• If you need per-URL detail, non-Google botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., or longer history, graduate to server log analysis.

The mental models

1. It’s a health monitor, not a scoreboard. The numbers are trends to watch, not targets to maximize. You’re looking for the shape of the curve — sudden spikes and drops — not a “good” absolute value.

2. The server feedback loop. Fast, stable server → capacity limit can rise → more crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. Slow server or 5xx → capacity limit drops → less crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. Response time and error rate are the levers that quietly govern everything else in the report.

3. Host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. first, then responses, then the rest. Triage in order of leverage: a red host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. blocks everything → fix it. Then by response (5xx and robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.-unavailable are the dangerous ones). Only then worry about file-type/GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.-type/purpose nuance.

4. Crawl Stats vs logs — the graduation rule. Crawl Stats tells you something is off (Google-only, aggregated, not guaranteed complete, 90 days). Server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. tell you exactly what (all botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index./users, every request, per-URL). Start with Crawl Stats; move to logs when you need precision.

5. The “should I care?” gate. Google frames the report itself for advanced users and flags it as unnecessary for sites under ~1,000 pages; official crawl-budget guidance targets much larger, frequently-changing sites. My own practitioner rule of thumb is looser — under ~10k pages, crawl budgetThe number of URLs an engine will crawl in a timeframe. basically doesn’t apply — but treat that as judgment, not an official cutoff. Either way, use the report for server-health early warning, and stop optimizing a number that doesn’t move rankings.

Crawl Stats report — cheat sheet

Where & scope

• Settings → Crawl stats in Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.. ~90 days. Root-level properties only — Domain or root URL-prefix.

Three top-line metrics

• Total crawl requests — all requests, success or not.
• Total download size — bytes downloaded.
• Average response time — server-health proxy; sustained spike → less crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..

By response — what each pattern means

By file type — HTML, Image, JavaScript, CSS, JSON, PDF, others. A high share relative to your own baseline is a diagnostic lead, not proof of waste; watch for resource overhead and surprise types (crawl trapsA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content.).

By GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. type — Smartphone (typically leads, no official required mix), Desktop, Image, Video, Page resource load, AdsBot.

By purpose — Discovery (URL never crawled before) vs Refresh (recrawl of a known page).

Host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. — robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. fetch · DNS resolution · server connectivity. Red = fix first.

Fast facts

• Report “missing”? You’re on a URL-prefix property scoped to a subfolder — re-verify at the root level (Domain property or root URL-prefix property).
• More crawling ≠ better rankings; but uncrawled = unrankable.
• Closest GSC gets to logs, but Google’s crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. only, aggregated, not guaranteed request-complete, 90 days.

Tools for seeing how Google crawls you

• Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. — Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). — the subject of this page: Settings → Crawl stats, for root-level properties (Domain or root URL-prefix).
• Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. — URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. — check how a single URL was crawled, rendered, and indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..
• Google Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. — Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. — the companion view: what made it from crawl into the index.
• Server log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. — the ground truth, covering all botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. and users. Tools: Screaming Frog Log File Analyser, or pipe logs into BigQuery / a log platform.
• CrawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. / site audits — Ahrefs Site Audit and Screaming Frog SEO Spider simulate a crawl and surface redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency., blocked URLs, and trap-like patterns.
• Ahrefs Webmaster ToolsThe free, first-party consoles search engines give site owners — Google Search Console and Bing Webmaster Tools — to see how their pages are crawled, indexed, and ranked, and to fix problems. The legacy name is 'webmaster tools.' — free crawl + audit for sites you verify.

Test yourself: Crawl Stats

SOP: Weekly crawl-health review

Use this as a short recurring check—not a reason to optimize crawl volume for its own sake.

1. Open the correct property. Go to the root-level Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. property, then Settings → Crawl stats.
2. Set context. Note deployments, migrations, outages, traffic events, and sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. changes that overlap the chart.
3. Check Host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. first. Triage robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. fetch, DNS, or server-connectivity warnings before interpreting the rest of the report.
4. Compare the three trends. Review requests, download size, and response time against the property’s own prior weeks—not a generic benchmark.
5. Open “By response.” Investigate sustained 5xx, robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. unavailability, redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. growth, and newly rising 404 patterns.
6. Review the mix. Look at file type, GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. type, and Discovery versus Refresh for unexplained shifts or crawl trapsA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content..
7. Correlate downstream. Check URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. and Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. for representative URLs. More requests do not matter if important pages remain uncrawled or unindexed.
8. Escalate when aggregates are insufficient. Pull server logs for per-URL evidence, exact timestamps, user agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., and non-Google traffic.
9. Record the finding. Save the date, anomaly, likely cause, owner, action, and the metric that should normalize. Close the issue only after the trend recovers.

Crawl Stats troubleshooting

The report is missing

Likely cause: you’re viewing a path-level URL-prefix property. Fix: open or verify the root-level property. The report’s absence from a narrower property says nothing about whether Google is crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. the site.

Average response time rises while crawl requests fall

Likely cause: Google is backing off as the origin, CDN, or application slows under load. Fix: correlate the timestamp with monitoring and logs, identify slow routes or resource types, restore reliable capacity, then watch whether response time falls before expecting crawl volume to recover.

5xx responses spike

Likely cause: an outage, rate limiter, overloaded origin, or botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-specific security rule is failing GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. requests. Fix: inspect the exact log entries, validate GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. when needed, remove accidental blocking, and test affected URLs from outside the application session.

“robots.txt not available” appears

Likely cause: DNS, TLS, CDN, redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., or server failures prevented Google from fetching the file—not necessarily a bad directive inside it. Fix: request /robots.txt directly across affected hosts, confirm a stable successful response, and check Host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. plus logs for the failure window.

Discovery climbs unexpectedly

Likely cause: a new launch or migration is working as intended, or Google foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. an unbounded URL space such as parameters, filters, calendars, or internal-search pages. Fix: sample logs and internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. to identify the new URL pattern. Preserve intentional discoveries; constrain crawl trapsA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content. at their source.

The report looks healthy, but important pages are not indexed

Likely cause: aggregate host health is fine while individual URLs have discovery, canonical, quality, or indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. problems. Fix: use URL Inspection and Page IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. for those pages. Crawl Stats cannot prove that a particular URL was selected for the index.

Measure crawl health as trends, not quotas

There is no universal “good” number of crawl requests. Build a baseline for this site and annotate it with deployments and incidents.

Define success against the incident you are solving. After a server problem, success is stable availability and normalized response time followed by recovered crawling. After closing a crawl trap, success is fewer requests to the unwanted pattern without reducing discovery or refresh of important URLs. Extra requests by themselves are not an SEO outcome, and Crawl Stats alone cannot measure ranking or index quality.

Resources worth your time

My related writing

• When Should You Worry About Crawl Budget? — my take on crawl budgetThe number of URLs an engine will crawl in a timeframe., where Crawl Stats fits, and why most sites don’t need to obsess over it.
• What Is Googlebot & How Does It Work? — the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. behind every number in this report.
• The Beginner’s Guide to Technical SEO — where crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. fit in the bigger picture.

Official

• Crawl Stats report — Search Console Help — the primary source for every metric and breakdown.
• Optimize your crawl budget — capacity + demand, and the size thresholds.

From around the industry

• Googlebot Crawl Budget Explained by Google’s Gary Illyes — Search Engine Journal’s coverage of Gary Illyes’ crawl-budget explainer; useful context for interpreting what drives the numbers in the report.
• Crawl Budget for SEO: Everything You Need to Know — Search Engine Journal deep dive on crawl budgetThe number of URLs an engine will crawl in a timeframe. concepts, capacity, and demand.
• Crawling and Indexing — Google Search Central hub for all official crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. documentation, robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., and sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing..
• Search Off the Record: How Googlebot Crawls the Web — Gary Illyes and Martin Splitt on crawling infrastructure and how the scheduler reacts to your server.
• r/TechSEO — the community for crawl/indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. debugging.

Podcasts

• Search Off the Record (Google Search Relations) — How GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. crawls the web. Gary Illyes and Martin Splitt on crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. infrastructure, conditional requests, and how the crawl scheduler reacts to your server — useful background for what drives the numbers in this report. Listen

Videos

• Google Search Central (YouTube) — Martin Splitt’s crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor./renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. explainers and the How Google Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank. series. Good groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. for what GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. is doing before it ever reaches the Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root).. Channel

Crawl Stats report

A Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root).

Related: Google Search Console, Host status, Crawl Budget

Crawl Stats report

The Crawl Stats report lives under Settings in Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. and shows statistics about Google’s crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. history on your site over the trailing 90 days. The three top-line numbers are total crawl requests, total download size in bytes, and average response time.

Below those, the report breaks crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. down four ways: by response (200, 301, 404, 5xx, robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. unavailable, and more), by file type (HTML, image, JavaScript, CSS, JSON, PDF, and others), by GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. type (smartphone, desktop, image, video, page resource load, AdsBot), and by purpose (Discovery vs Refresh). It also surfaces a Host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. summary flagging whether Google hit availability problems fetching robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., resolving DNS, or connecting to your server.

The report is only available for root-level properties — domain properties or root-level URL-prefix properties (like https://example.com). If you verified your site as a URL-prefix property scoped to a subfolder, you won’t see it. It’s the closest thing Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. offers to server log data on GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. activity, but it covers Google’s own crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. only, is aggregated, and per Google’s own documentation isn’t guaranteed to report every request — so it isn’t a substitute for full log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened..

Related: Google Search Console, Host status, Crawl Budget

Changelog