Crawl Stats Report

TL;DR — The Crawl Stats report shows how Google has been crawling your site over the last 90 days — how many times it asked for pages, how much it downloaded, and how fast your server responded. You find it under Settings in Google Search Console, and it is available for a root-level property rather than a path-scoped property. For most sites it’s a health check, not something you need to optimize.

What the Crawl Stats report is

When Google’s crawler — Googlebot — visits your site, every request it makes gets logged on Google’s side. The Crawl Stats report is where Google shows you a summary of that activity: how often it came by, what it downloaded, and whether it ran into problems.

You’ll find it in Google Search Console under Settings → Crawl stats (it’s not in the main sidebar with the other reports). It covers roughly the last 90 days.

Where to find it (and why you might not see it)

The most common reason people can’t find the report is simple: it only shows up for root-level properties. That covers a domain property (just the domain, no https:// prefix) and a URL-prefix property verified at the site’s root (like https://example.com). If you added a URL-prefix property pointing at a subfolder or subpath, you won’t get the report.

The fix is to verify a root-level property — domain or root URL-prefix. Same site, different property type — and the report appears.

The three numbers up top

• Total crawl requests — how many times Googlebot asked for a URL on your site, whether it succeeded or not.
• Total download size — how many bytes Google downloaded while crawling.
• Average response time — how long, on average, your server took to respond.

These are trends to watch, not targets to hit. There’s no “right” number. What matters is the shape of the line over time.

Should you even worry about this?

Probably not, honestly. There’s a lot of anxiety around “crawl budget,” but most sites never need to think about it. The report is far more useful as an early-warning system for server problems than as a dashboard to optimize.

The thing to watch for is your server struggling. If your average response time climbs or you start seeing server errors, Google notices and crawls you less to avoid making things worse. So if pages suddenly aren’t getting picked up, this report is one of the first places I’d look.

There’s also a Host status summary here that flags whether Google had trouble reaching your site at all (fetching robots.txt, DNS, connecting to the server). That’s the most important block — if it’s red, fix that first.

Want the full metric-by-metric breakdown and what each pattern means? Switch to the Advanced tab.

TL;DR — Crawl Stats (Settings → Crawl stats, root-level properties only — Domain or root URL-prefix) reports total crawl requests, total download size, and average response time over ~90 days, plus breakdowns by response, file type, Googlebot type, and purpose (Discovery vs Refresh). Read it as a server-health monitor, not a crawl-budget vanity dashboard: rising response time or 5xx makes Google reduce crawl capacity, and a sustained robots.txt failure can halt crawling. It’s the closest GSC gets to log data, but it covers Google’s crawlers only and isn’t guaranteed to be request-complete — for all bots, users, and a complete record you still need server logs. Google scopes its own crawl-budget guidance to large, frequently-changing sites; for most smaller sites, in my experience, it almost never matters.

What it is and where it lives

Google’s own framing is direct: “The Crawl Stats report shows you statistics about Google’s crawling history on your website.” You get to it under Settings → Crawl stats — not in the main report sidebar, which trips a lot of people up.

The single biggest reason people think the report is “missing” or “broken”: Google only shows it for root-level properties. In their words, “This report is available only for root-level properties.” That covers both a Domain property and a URL-prefix property verified at the site’s root (for example, https:// example.com) — it isn’t limited to protocol-less Domain properties. If you verified a URL-prefix property scoped to a subfolder or subpath, re-verify at the root level and it appears. There’s nothing wrong with your site.

Should you care? The crawl-budget reframe

Most sites do not need to optimize crawl budget, and the report’s real value is health monitoring rather than budget tuning. Google scopes its own crawl-budget guidance to large sites (1M+ unique pages) and medium-or-larger sites (10k+ pages with daily change). Below that, this is a diagnostic, not a project.

Google also frames the report itself as built for advanced users, and says sites under roughly 1,000 pages generally don’t need to dig into it at this granularity — a smaller threshold than the crawl-budget guidance above. My own rule of thumb is looser: for most sites under about 10,000 pages, I still wouldn’t spend much time on crawl budget. Treat that as practitioner judgment, not an official cutoff.

Crawl rate is governed by two things working together — crawl capacity limit and crawl demand. Capacity is what your server can handle. Google defines the capacity limit as “The maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches.” Demand is how much Google wants to crawl, driven by factors like popularity and staleness. The report is where the consequences of both show up. (More on the concept in crawl budget.)

A few scope details worth knowing before you read the numbers: the report counts the actual URL Googlebot requested rather than the canonical it resolves to, and it counts duplicate requests and each server-side redirect hop separately (client-side redirects aren’t counted as redirect responses here). It’s scoped to the property or host you’re viewing — resources on other domains, and sometimes sibling or child domains, can be under- or over-represented depending on property type. And per Google’s own documentation, the report reflects most crawl requests but can omit some, so a small mismatch against your own server logs is expected, not a bug.

This is the closest thing Search Console gives you to log-file data — but it’s Google’s crawlers only, aggregated, and not guaranteed to be request-complete. If you want hits from all bots and users, or a guaranteed complete record, you need your server logs. Treat Crawl Stats as the free first look before you invest in log analysis.

The three top-line metrics

Read these as trends, not targets:

• Total crawl requests — “The total number of crawl requests issued for URLs on your site, whether successful or not.” Every request counts, including the ones that errored.
• Total download size — “Total number of bytes downloaded from your site during crawling, for the specified time period.”
• Average response time — “Average response time for all resources fetched from your site during the specified time period.” This is effectively a server-health proxy. A sustained spike is a performance alarm, and Google may respond by dropping your crawl capacity limit — so the SEO harm is indirect (slower, less crawling), separate from any page-speed ranking factor.

Breakdown: By response

This table, in Google’s words, shows “the responses that Google received when crawling your site, grouped by response type, as a percentage of all crawl responses.” The response types include OK (200), Moved permanently (301), Moved temporarily (302), Not found (404), Server error (5XX), and robots.txt not available.

How I read each pattern:

• 200 should be the large majority. Google doesn’t publish a specific target percentage, so judge this against your own baseline and watch for sustained shifts rather than chasing an exact number.
• Lots of 404s isn’t fatal but signals wasted crawl — Google spending requests on URLs that don’t exist.
• Lots of 301s points at redirect chains or internal links still aimed at old URLs. Fix the links to point at the final destination.
• A 5xx spike is the most damaging pattern in the report. Your server is erroring under crawl load, which drops the capacity limit, which means less crawling. Sustained 5xx can effectively throttle Google to a crawl.
• robots.txt not available shouldn’t show up in any sustained, meaningful way. Google doesn’t publish a fixed target percentage, but a persistent share means Google periodically couldn’t fetch your robots.txt. That matters because if your robots.txt stays unavailable, Google will conservatively pause crawling for a while rather than guess. (For what robots.txt does and doesn’t control, see robots.txt.)

Breakdown: By file type

Google groups crawling by file type — HTML, image, JavaScript, CSS, JSON, PDF, and others. This is a diagnostic lead, not proof of waste on its own — how much JS, CSS, image, or JSON crawling is normal depends on how your site is built (a JS-heavy app legitimately needs more resource crawling than a static HTML site). A JS/CSS/image share that’s high relative to your own baseline, or that jumps suddenly, is worth investigating as possible render-resource overhead. Unexpected file types showing up can reveal a crawl trap — an infinite or near-infinite URL space generating junk.

Breakdown: By Googlebot type

This splits crawling by which Googlebot did the fetching — smartphone, desktop, image, video, page resource load, and AdsBot. Google doesn’t document a required mix, but in a mobile-first crawl, Smartphone typically leads for most sites. A desktop-heavy profile, or unexpected volume from Image or AdsBot, is a lead worth investigating rather than a violation by itself — it can mean Google isn’t treating your site as mobile-first, that ad/image crawling is eating into budget you’d rather spend on content, or just that your site has unusual resource needs.

Breakdown: By purpose — Discovery vs Refresh

Two categories, and Google defines them only by prior crawl history — the labels alone don’t tell you whether the mix is healthy:

• Discovery — “The URL requested was never crawled by Google before.”
• Refresh — “A recrawl of a known page.”

A rise in Discovery is expected during a launch or migration (Google is finding your new URLs); the same rise is a problem if it’s coming from infinite-space or parameter URLs — that’s crawl budget going to garbage. A Refresh-heavy mix is normal for an established site that isn’t publishing much new content. Judge either against your own baseline and what you know changed, not against a fixed “healthy” ratio.

Host status — the most actionable block

Host status summarizes whether Google hit availability problems trying to crawl you, grouped into three sub-checks: robots.txt fetch, DNS resolution, and server connectivity. A red host status is one of the most actionable signals in the report — it usually means Google was blocked or throttled at the infrastructure level, before it ever got to your content, during that window. It’s an aggregate alert, though, not proof that every URL was affected or that the problem is still happening now — check the timeline and confirm current status (a live fetch, URL Inspection, or your logs) before you consider it fixed. Still, treat a red host status as the first thing to investigate. (Covered in its own right under Host status.)

Reading spikes and drops

The mechanism to internalize: your server’s behavior feeds back into how much Google crawls. If the site responds quickly and reliably, Google’s capacity limit can rise; if it slows down or starts returning server errors, the limit drops. So:

• Response-time spike → capacity limit can drop → less crawling.
• 5xx surge → capacity limit drops → less crawling; sustained, it can throttle you hard.
• robots.txt unavailable → if it stays unavailable, Google pauses crawling conservatively.

None of this is a ranking penalty. The damage is indirect: if pages aren’t crawled and indexed, they can’t rank at all — but extra crawling doesn’t lift rankings either. (See the crawl budget concept for the full capacity-plus-demand model.)

Crawl Stats vs server logs

Crawl Stats is the closest GSC gets to logs, but the gaps matter:

If you outgrow Crawl Stats — you need per-URL data, non-Google bots, or longer history — that’s when you graduate to log file analysis. Crawl Stats tells you something’s off; logs tell you exactly where.

How this relates to the rest of crawling

The Crawl Stats report is your view into crawling after the fact. For the indexing side of what happens next, the Page Indexing report is the companion — Crawl Stats shows what Google fetched; Page Indexing shows what made it into the index. And Host status, surfaced inside this report, is the infrastructure-level signal worth leading with.

AI summary

A condensed take on the Advanced version:

• Where it is: Search Console → Settings → Crawl stats. Covers ~90 days. Only available for root-level properties — a Domain property or a URL-prefix property verified at the site’s root — that’s why it’s “missing” for URL-prefix subfolders.
• Three top-line metrics: total crawl requests, total download size, average response time. Read them as trends, not targets.
• Four breakdowns: By response (200/301/404/5xx/robots.txt unavailable), By file type (HTML/JS/CSS/image/PDF…), By Googlebot type (smartphone/desktop/image/video/AdsBot), By purpose (Discovery vs Refresh).
• It’s a server-health monitor, not a budget dashboard. Rising response time or 5xx makes Google drop its crawl capacity limit and crawl less; sustained robots.txt unavailability can pause crawling.
• Host status (robots.txt fetch, DNS, server connectivity) is the most actionable block — fix a red host status first.
• Crawl ≠ ranking: more crawling doesn’t lift rankings, but uncrawled pages can’t rank at all.
• Closest GSC gets to logs, but it’s Google-only, aggregated, and not guaranteed request-complete — for all bots/users and per-URL detail, use server logs.
• Google scopes official crawl-budget guidance to large, frequently-changing sites (and even flags the report itself as unnecessary detail for sites under ~1,000 pages). Practitioner rule of thumb: most sites well under 10k pages don’t need to optimize crawl budget — but that’s judgment, not an official cutoff.

Official documentation

Primary-source documentation from Google.

Google

• Crawl Stats report — the Search Console Help page for this report: metrics, breakdowns, host status, and the root-level-property requirement.
• Optimize your crawl budget — crawl capacity limit + crawl demand, and who actually needs to manage budget.
• Crawling and Indexing — the broader hub for robots, sitemaps, and crawl controls.

Quotes from the source

On-the-record statements from Google. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what the report shows

• “The Crawl Stats report shows you statistics about Google’s crawling history on your website.” — Google Search Console Help. Jump to quote
• “This report is available only for root-level properties.” Jump to quote

Google — the three metrics

• “The total number of crawl requests issued for URLs on your site, whether successful or not.” Jump to quote
• “Total number of bytes downloaded from your site during crawling, for the specified time period.” Jump to quote
• “Average response time for all resources fetched from your site during the specified time period.” Jump to quote

Google — the breakdowns

• “This table shows the responses that Google received when crawling your site, grouped by response type, as a percentage of all crawl responses.” Jump to quote
• “Discovery: The URL requested was never crawled by Google before.” Jump to quote
• “Refresh: A recrawl of a known page.” Jump to quote

Google — crawl capacity (the budget mechanism behind the numbers)

• Crawl capacity limit: “The maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches.” — Google Search Central, large-site crawl budget guide. Jump to quote

Reading the Crawl Stats report — checklist

A quick pass to get the signal out of the report:

• You’re on a domain (root-level) property — otherwise the report won’t show.
• Host status is green (robots.txt fetch, DNS, server connectivity all OK). Fix any red here first.
• Average response time is stable, not trending up.
• By response is a large majority 200; 5xx and robots.txt-unavailable show no sustained, meaningful share (Google publishes no fixed target — watch your own trend).
• No surprise volume of 404s (wasted crawl) or 301s (redirect chains / stale internal links).
• By Googlebot type generally leans Smartphone in a mobile-first crawl — there’s no official required mix, but a sudden shift is worth a look.
• By file type isn’t dominated by JS/CSS/image relative to your own baseline, and shows no surprise file types (possible crawl trap).
• By purpose: judge Discovery vs Refresh against your own baseline and what changed (launch, migration, sitemap update) — an unexplained jump either way is worth checking for parameter/infinite-URL waste or stalled recrawling.
• If you need per-URL detail, non-Google bots, or longer history, graduate to server log analysis.

The mental models

1. It’s a health monitor, not a scoreboard. The numbers are trends to watch, not targets to maximize. You’re looking for the shape of the curve — sudden spikes and drops — not a “good” absolute value.

2. The server feedback loop. Fast, stable server → capacity limit can rise → more crawling. Slow server or 5xx → capacity limit drops → less crawling. Response time and error rate are the levers that quietly govern everything else in the report.

3. Host status first, then responses, then the rest. Triage in order of leverage: a red host status blocks everything → fix it. Then by response (5xx and robots.txt-unavailable are the dangerous ones). Only then worry about file-type/Googlebot-type/purpose nuance.

4. Crawl Stats vs logs — the graduation rule. Crawl Stats tells you something is off (Google-only, aggregated, not guaranteed complete, 90 days). Server logs tell you exactly what (all bots/users, every request, per-URL). Start with Crawl Stats; move to logs when you need precision.

5. The “should I care?” gate. Google frames the report itself for advanced users and flags it as unnecessary for sites under ~1,000 pages; official crawl-budget guidance targets much larger, frequently-changing sites. My own practitioner rule of thumb is looser — under ~10k pages, crawl budget basically doesn’t apply — but treat that as judgment, not an official cutoff. Either way, use the report for server-health early warning, and stop optimizing a number that doesn’t move rankings.

Crawl Stats report — cheat sheet

Where & scope

• Settings → Crawl stats in Search Console. ~90 days. Root-level properties only — Domain or root URL-prefix.

Three top-line metrics

• Total crawl requests — all requests, success or not.
• Total download size — bytes downloaded.
• Average response time — server-health proxy; sustained spike → less crawling.

By response — what each pattern means

By file type — HTML, Image, JavaScript, CSS, JSON, PDF, others. A high share relative to your own baseline is a diagnostic lead, not proof of waste; watch for resource overhead and surprise types (crawl traps).

By Googlebot type — Smartphone (typically leads, no official required mix), Desktop, Image, Video, Page resource load, AdsBot.

By purpose — Discovery (URL never crawled before) vs Refresh (recrawl of a known page).

Host status — robots.txt fetch · DNS resolution · server connectivity. Red = fix first.

Fast facts

• Report “missing”? You’re on a URL-prefix property scoped to a subfolder — re-verify at the root level (Domain property or root URL-prefix property).
• More crawling ≠ better rankings; but uncrawled = unrankable.
• Closest GSC gets to logs, but Google’s crawlers only, aggregated, not guaranteed request-complete, 90 days.

Tools for seeing how Google crawls you

• Google Search Console — Crawl Stats report — the subject of this page: Settings → Crawl stats, for root-level properties (Domain or root URL-prefix).
• Google Search Console — URL Inspection — check how a single URL was crawled, rendered, and indexed.
• Google Search Console — Page Indexing report — the companion view: what made it from crawl into the index.
• Server log file analysis — the ground truth, covering all bots and users. Tools: Screaming Frog Log File Analyser, or pipe logs into BigQuery / a log platform.
• Crawlers / site audits — Ahrefs Site Audit and Screaming Frog SEO Spider simulate a crawl and surface redirect chains, blocked URLs, and trap-like patterns.
• Ahrefs Webmaster Tools — free crawl + audit for sites you verify.

Test yourself: Crawl Stats

SOP: Weekly crawl-health review

Use this as a short recurring check—not a reason to optimize crawl volume for its own sake.

1. Open the correct property. Go to the root-level Search Console property, then Settings → Crawl stats.
2. Set context. Note deployments, migrations, outages, traffic events, and sitemap changes that overlap the chart.
3. Check Host status first. Triage robots.txt fetch, DNS, or server-connectivity warnings before interpreting the rest of the report.
4. Compare the three trends. Review requests, download size, and response time against the property’s own prior weeks—not a generic benchmark.
5. Open “By response.” Investigate sustained 5xx, robots.txt unavailability, redirect growth, and newly rising 404 patterns.
6. Review the mix. Look at file type, Googlebot type, and Discovery versus Refresh for unexplained shifts or crawl traps.
7. Correlate downstream. Check URL Inspection and Page Indexing for representative URLs. More requests do not matter if important pages remain uncrawled or unindexed.
8. Escalate when aggregates are insufficient. Pull server logs for per-URL evidence, exact timestamps, user agents, and non-Google traffic.
9. Record the finding. Save the date, anomaly, likely cause, owner, action, and the metric that should normalize. Close the issue only after the trend recovers.

Crawl Stats troubleshooting

The report is missing

Likely cause: you’re viewing a path-level URL-prefix property. Fix: open or verify the root-level property. The report’s absence from a narrower property says nothing about whether Google is crawling the site.

Average response time rises while crawl requests fall

Likely cause: Google is backing off as the origin, CDN, or application slows under load. Fix: correlate the timestamp with monitoring and logs, identify slow routes or resource types, restore reliable capacity, then watch whether response time falls before expecting crawl volume to recover.

5xx responses spike

Likely cause: an outage, rate limiter, overloaded origin, or bot-specific security rule is failing Googlebot requests. Fix: inspect the exact log entries, validate Googlebot when needed, remove accidental blocking, and test affected URLs from outside the application session.

“robots.txt not available” appears

Likely cause: DNS, TLS, CDN, redirect, or server failures prevented Google from fetching the file—not necessarily a bad directive inside it. Fix: request /robots.txt directly across affected hosts, confirm a stable successful response, and check Host status plus logs for the failure window.

Discovery climbs unexpectedly

Likely cause: a new launch or migration is working as intended, or Google found an unbounded URL space such as parameters, filters, calendars, or internal-search pages. Fix: sample logs and internal links to identify the new URL pattern. Preserve intentional discoveries; constrain crawl traps at their source.

The report looks healthy, but important pages are not indexed

Likely cause: aggregate host health is fine while individual URLs have discovery, canonical, quality, or indexing problems. Fix: use URL Inspection and Page Indexing for those pages. Crawl Stats cannot prove that a particular URL was selected for the index.

Measure crawl health as trends, not quotas

There is no universal “good” number of crawl requests. Build a baseline for this site and annotate it with deployments and incidents.

Define success against the incident you are solving. After a server problem, success is stable availability and normalized response time followed by recovered crawling. After closing a crawl trap, success is fewer requests to the unwanted pattern without reducing discovery or refresh of important URLs. Extra requests by themselves are not an SEO outcome, and Crawl Stats alone cannot measure ranking or index quality.

Resources worth your time

My related writing

• When Should You Worry About Crawl Budget? — my take on crawl budget, where Crawl Stats fits, and why most sites don’t need to obsess over it.
• What Is Googlebot & How Does It Work? — the crawler behind every number in this report.
• The Beginner’s Guide to Technical SEO — where crawling and Search Console fit in the bigger picture.

Official

• Crawl Stats report — Search Console Help — the primary source for every metric and breakdown.
• Optimize your crawl budget — capacity + demand, and the size thresholds.

From around the industry

• Googlebot Crawl Budget Explained by Google’s Gary Illyes — Search Engine Journal’s coverage of Gary Illyes’ crawl-budget explainer; useful context for interpreting what drives the numbers in the report.
• Crawl Budget for SEO: Everything You Need to Know — Search Engine Journal deep dive on crawl budget concepts, capacity, and demand.
• Crawling and Indexing — Google Search Central hub for all official crawling documentation, robots.txt, and sitemaps.
• Search Off the Record: How Googlebot Crawls the Web — Gary Illyes and Martin Splitt on crawling infrastructure and how the scheduler reacts to your server.
• r/TechSEO — the community for crawl/index debugging.

Podcasts

• Search Off the Record (Google Search Relations) — How Googlebot crawls the web. Gary Illyes and Martin Splitt on crawling infrastructure, conditional requests, and how the crawl scheduler reacts to your server — useful background for what drives the numbers in this report. Listen

Videos

• Google Search Central (YouTube) — Martin Splitt’s crawling/rendering explainers and the How Google Search Works series. Good grounding for what Googlebot is doing before it ever reaches the Crawl Stats report. Channel

Crawl Stats report

A Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root).

Related: Google Search Console, Host status, Crawl Budget

Crawl Stats report

The Crawl Stats report lives under Settings in Google Search Console and shows statistics about Google’s crawling history on your site over the trailing 90 days. The three top-line numbers are total crawl requests, total download size in bytes, and average response time.

Below those, the report breaks crawling down four ways: by response (200, 301, 404, 5xx, robots.txt unavailable, and more), by file type (HTML, image, JavaScript, CSS, JSON, PDF, and others), by Googlebot type (smartphone, desktop, image, video, page resource load, AdsBot), and by purpose (Discovery vs Refresh). It also surfaces a Host status summary flagging whether Google hit availability problems fetching robots.txt, resolving DNS, or connecting to your server.

The report is only available for root-level properties — domain properties or root-level URL-prefix properties (like https://example.com). If you verified your site as a URL-prefix property scoped to a subfolder, you won’t see it. It’s the closest thing Search Console offers to server log data on Googlebot activity, but it covers Google’s own crawlers only, is aggregated, and per Google’s own documentation isn’t guaranteed to report every request — so it isn’t a substitute for full log file analysis.

Related: Google Search Console, Host status, Crawl Budget

Changelog