Crawl Budget

TL;DR — Crawl budget is how much a search engine is willing to crawl your site. It’s two things multiplied together: how much your server can handle and how much Google wants to crawl. Crawling more doesn’t make you rank higher — and for most sites, this is a non-issue. If your pages get crawled the same day you publish them, you don’t have a crawl budget problem.

What crawl budget is

When a search engine crawls your site, it doesn’t crawl forever. It crawls some number of your URLs in a given window of time, then moves on. That amount is what SEOs call your crawl budget.

It comes down to two questions the search engine is constantly answering:

• How much can I crawl? Your server can only take so much before it slows down or starts throwing errors. Google watches for that and backs off. This is the crawl capacity (older docs called it the crawl rate limit).
• How much do I want to crawl? Popular pages and pages that change often get crawled more. Pages nobody links to, or that never change, get crawled rarely. This is crawl demand.

Multiply the two and you get your crawl budget: roughly, the number of URLs Google can and wants to crawl. Evidence for this claim Google defines crawl budget using crawl capacity limit and crawl demand. Scope: Google Search crawling for larger sites. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide

One distinction worth making early: crawl budget is about whether Google fetches a URL at all. Whether it then indexes what it finds is a separate, later decision. A page can be crawled and still not get indexed — crawling doesn’t guarantee indexing, it’s just the gate you have to get through first.

The thing to understand first: you probably don’t need to worry about this

This is the part most articles bury. Most sites do not have a crawl budget problem. Google says so directly — if your pages get crawled the same day they’re published, or your site isn’t huge and fast-changing, you can skip the whole topic. Evidence for this claim Google says sites without many rapidly changing pages, or whose pages are crawled the day they publish, generally do not need crawl-budget guidance. Scope: Google's rough applicability guidance, not a guarantee for every site. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide

A simple gut check: search for a few of your URLs in Google, or use the URL Inspection tool in Search Console. If your new pages show up within a day or two, your crawl budget is fine. Spend your time on content and links instead.

Crawl budget starts to matter when a site has hundreds of thousands or millions of pages, especially if a lot of them change constantly (think a big ecommerce store or a large news site).

More crawling does not mean better rankings

Here’s a myth worth killing early: crawling a page more often doesn’t push it up the results. Crawling is just how the engine finds and downloads your page. It’s a gate you have to get through to rank at all — but once you’re through it, how often you get re-crawled isn’t a ranking factor.

So the goal isn’t “get crawled more.” The goal is to make sure the pages you care about get crawled, and that the engine isn’t wasting its time on junk URLs.

What wastes crawl budget

If you do have a big site, the budget gets drained by:

• Filter and sort URLs (faceted navigation) — every filter combination on an ecommerce store can spawn thousands of near-identical URLs.
• Duplicate pages — the same content reachable at multiple URLs (with and without www, with parameters tacked on, etc.).
• Soft 404s — “not found” pages that return a 200 OK status, so the engine keeps crawling them.
• Broken redirect chains and slow pages — these make every crawl cost more.

The fix is almost always removing waste so the budget lands on real pages — not trying to convince Google to crawl “more.”

Want the full model — crawl capacity vs. demand, the exact size thresholds, what to do about faceted navigation, how Bing handles this differently, and how to measure it — switch to the Advanced tab.

TL;DR — Crawl budget = crawl capacity limit (what your server can take) × crawl demand (popularity + staleness + perceived inventory). It’s an efficiency concern, not a ranking signal. Internally it’s scheduling by importance gated by host load, not a flat per-site quota. Most sites can ignore it — Mueller pegs 100k URLs as “usually not enough,” and Google tells same-day-crawled sites to skip the guide. It bites at ~1M+ pages (weekly change), 10k+ pages (daily change), or when “Discovered – currently not indexed” balloons. The highest-leverage move is cutting waste — faceted nav, duplicates, soft 404s, infinite spaces — so the budget consolidates on URLs that matter.

The two-factor model

Google defines it cleanly: “The amount of time and resources that Google devotes to crawling a site is commonly called the site’s crawl budget and it’s determined by two main elements: crawl capacity limit and crawl demand.” The 2017 framing from Gary Illyes is the one-liner I still reach for: crawl budget is “the number of URLs Googlebot can and wants to crawl.” Evidence for this claim Google defines crawl budget using crawl capacity limit and crawl demand. Scope: Google Search crawling for larger sites. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide Keep this scoped: crawl budget governs fetching, not indexing. A crawled URL still goes through a separate indexing decision — folding the two together overstates what crawl budget controls.

Crawl capacity limit (the supply side). This is “the maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches.” It moves with your server’s health. Respond fast and clean and the limit rises; serve slow responses, 5xx errors, or 429s and Googlebot backs off. In my How Search Works deck I list the same rate-limit triggers: server stability, slow responses, 5xx server errors, and 429 (too many requests). This is the “can.”

Crawl demand (the demand side). Driven by popularity (how linked-to / important a URL is) and staleness (how long since it was last crawled, how often it changes). That same deck breaks demand into PageRank, how frequently the page changes, time since last crawl, and major site changes. Critically, Google flags perceived inventory as the lever you control most: “Without guidance from you, Google tries to crawl all or most of the URLs that it knows about on your site. If many of these URLs are duplicates, or you don’t want them crawled for some other reason… this wastes a lot of Google crawling time on your site. This is the factor that you can positively control the most.”

A couple of structural facts that catch people out:

• Budget is per hostname. “https://www.example.com/ and https://code.example.com/ are two different hostnames, and therefore have separate crawl budgets.” Subdomains don’t share.
• The different Googlebot types likely draw from one pool. In my own reporting, image, news, video, ads, and the rest appear to pull from the same per-site budget — I don’t have a current primary Google source pinning this down exactly, so treat it as a practitioner observation rather than documented policy. Either way, it’s worth checking the Crawl Stats report’s by-crawler-type breakdown if you suspect one type is elbowing out the rest.

What it really is internally: scheduling by importance

“Crawl budget” is an SEO-coined umbrella term. Internally it’s closer to scheduling gated by host load. As Illyes has described it, Google’s scheduler “sets a bucket of URLs in importance order and GoogleBot will crawl in that order based on the schedule the host load decided. If Google thinks your server can handle it, it will crawl the whole bucket, if not, it will stop.”

That reframes the whole topic. It isn’t a flat “you get N pages a day” quota — it’s a prioritized queue, and crawling tracks search demand. Illyes again: “If search demand goes down, then that also correlates to the crawl limit going down,” and “if you want to increase how much we crawl, then you somehow have to convince search that your stuff is worth fetching, which is basically what the scheduler is listening to.” The Search Relations team has explicitly called the “fixed daily page quota” idea a misconception.

Does your site actually have a crawl budget problem?

This is the most valuable section, so I’ll be blunt: most sites don’t need to worry about crawl budget. Google’s own guide opens with the de-escalation: “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide. For Google Search specifically, merely keeping your sitemap up to date and checking your index coverage regularly is adequate.” Evidence for this claim Google says sites without many rapidly changing pages, or whose pages are crawled the day they publish, generally do not need crawl-budget guidance. Scope: Google's rough applicability guidance, not a guarantee for every site. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide

John Mueller gave the concrete number: “100k URLs is usually not enough to affect crawl budget (it’s <1/minute over 3 months).” If you’re under six figures of URLs and pages get crawled promptly, move on.

When it does matter, Google’s rough thresholds are:

• Large sites — 1 million+ unique pages with content that changes moderately often (about weekly).
• Medium or larger sites — 10,000+ unique pages with very rapidly changing content (daily).
• Sites with a large portion of URLs classified as “Discovered – currently not indexed” in Search Console — that’s the warning light that Google knows about URLs it isn’t getting to.

Google adds the disclaimer that “the numbers given here are a rough estimate… not exact thresholds.” And even on big sites, the nuance from my own work holds: it’s usually new, poorly-linked, or static pages that lag — not your popular ones.

Does crawl budget affect rankings? No.

Crawling is necessary to rank, but it is not a ranking signal. Google in 2017: “An increased crawl rate will not necessarily lead to better positions in Search results. Google uses hundreds of signals to rank the results, and while crawling is necessary for being in the results, it’s not a ranking signal.” I put it the same way in my Ahrefs guide: “More crawling doesn’t mean you’ll rank better, but if your pages aren’t crawled and indexed they aren’t going to rank at all.” Treat crawl budget as an efficiency problem, full stop.

What wastes crawl budget

Illyes published the canonical list of low-value-add URLs “in order of significance”:

1. Faceted navigation and session identifiers — the #1 culprit, especially ecommerce filter/sort combinations that multiply URLs combinatorially.
2. On-site duplicate content — the classic technical variants: HTTP vs HTTPS, non-www vs www, trailing slash vs not, uppercase vs lowercase, default/index pages, and URL parameters. (Roughly 60% of the web is duplicate content, by Google’s own internal estimate.)
3. Soft error pages — soft 404s that return 200 keep getting crawled.
4. Hacked pages.
5. Infinite spaces and proxies — calendars, infinite-scroll pagination that duplicates, faceted combinations; the classic spider-trap patterns.
6. Low-quality and spam content.

The cost is concrete: “Wasting server resources on pages like these will drain crawl activity from pages that do actually have value, which may cause a significant delay in discovering great content on a site.” On top of the list, long redirect chains “have a negative effect on crawling,” and slow, heavy pages make every fetch more expensive.

How to optimize it

The whole game is consolidating the budget onto URLs that matter:

• Consolidate duplicates. Google: “Consolidate duplicate content to focus crawling on unique content rather than unique URLs.” Pick one host, one protocol, one trailing-slash convention; canonicalize; handle parameters.
• Block truly worthless paths with robots.txt — but only paths you never want crawled. For faceted navigation, the usual options are blocking the parameter paths in robots.txt or using a # instead of a ? so the URLs aren’t crawlable in the first place.
• Don’t use noindex to save budget. Google: “Don’t use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time.” The request still costs you. Block at robots.txt if you never want it fetched.
• Don’t expect robots.txt to reallocate budget. “Google won’t shift this newly available crawl budget to other pages unless Google is already hitting your site’s serving limit.” Blocking junk is good hygiene, but it doesn’t hand the freed-up crawls to your good pages unless you were capacity-bound.
• Fix soft 404s; return real 404/410 for gone pages. “A 404 status code is a strong signal not to crawl that URL again.”
• Shorten redirect chains, keep sitemaps current (honest lastmod), and improve server speed.
• Strengthen internal links to important and new pages — easier than anything else because you fully control it.

And the two — only two — ways Google says you can actually increase budget: “Add more server resources… [and] optimize your content’s quality.” Note the trap there: a faster server lifts the capacity ceiling, but if demand is low Google still crawls less. You need both.

How to measure it

• GSC > Settings > Crawl Stats report — total crawl requests over time, average response time, host status, and breakdowns by response code, file type, and Googlebot type. This is Google’s own view of how it crawls you.
• Server log file analysis — the ground truth. Real Googlebot hits by URL pattern, so you can see crawl waste and uncrawled important pages. Verify the bot is genuinely Googlebot via reverse + forward DNS or Google’s published IP ranges (plenty of fake bots spoof the user-agent).
• “Discovered – currently not indexed” in GSC — treat a growing pile here as a crawl-budget warning light: Google knows about the URLs but isn’t getting to them.

Bing and other engines: “crawl efficiency”

Bing reframes the topic as crawl efficiency rather than budget. Fabrice Canel’s definition: “The crawl efficiency is how often we crawl and discover new and fresh content per page crawled.” The goal is to “crawl an URL only when the content has been added (URL not crawled before), updated (fresh on-page context or useful outbound links).” Bing’s blunt philosophy: “Less is more for SEO. Never forget that. Less URLs to crawl, better for SEO.”

Bing’s preferred fix is IndexNow — push changed URLs so bingbot doesn’t need exploratory crawls — and Crawl Control in Bing Webmaster Tools, which lets you schedule when bingbot crawls by hour to protect server load. This is a real Google/Bing divergence worth noting: Google deprecated its old crawl-rate limiter in Search Console, while Bing still lets you actively shape the crawl schedule.

Crawl budget myths, corrected

• “Every site should optimize crawl budget.” No — most shouldn’t. Same-day crawling and sub-100k URLs mean you’re fine.
• “More crawling = better rankings.” No. Crawling is necessary but not a ranking signal.
• “It’s a fixed daily page quota.” No — it’s importance-driven scheduling gated by host load.
• “Use noindex to save budget.” No — Google still requests the page first.
• “Block pages in robots.txt to give other pages more budget.” Generally no, unless you’re already at your serving limit.
• “A faster server alone raises your budget.” It lifts the capacity ceiling only; low demand still means fewer crawls.

For the broader pipeline this sits inside — discovery, the crawl scheduler, rendering, and how crawling differs from indexing — see the crawling hub. The sibling topics (crawl rate, crawl frequency, spider traps, and log file analysis) each go deeper on one piece of this.

AI summary

A condensed take on the Advanced version:

• Crawl budget = crawl capacity limit × crawl demand. Capacity is what your server can take (rises on fast/clean responses, drops on 5xx/429); demand is popularity + staleness + perceived inventory. Google’s one-liner: “the number of URLs Googlebot can and wants to crawl.”
• It’s not a ranking factor. Crawling is necessary to rank, but more crawling doesn’t lift positions — it’s purely an efficiency concern.
• Internally it’s scheduling by importance, gated by host load — a prioritized queue, not a flat daily page quota. Crawling tracks search demand.
• Most sites don’t need to manage it. Mueller: 100k URLs usually isn’t enough to matter. Google: if pages crawl same-day, skip the guide. It bites at ~1M+ pages (weekly change), 10k+ (daily change), or a big “Discovered – currently not indexed” pile.
• What wastes it (Google’s order): faceted nav / session IDs, duplicate content, soft 404s, hacked pages, infinite spaces, low-quality content — plus redirect chains and slow pages.
• Optimize by removing waste: consolidate duplicates, block worthless paths in robots.txt (not noindex — Google still requests first), fix soft 404s, shorten redirects, keep sitemaps clean, strengthen internal links. Only two ways to truly increase budget: more server capacity and higher content quality.
• Measure with GSC Crawl Stats, server log analysis, and the “Discovered – currently not indexed” report.
• Bing calls it crawl efficiency (“less is more”), and pushes IndexNow + Crawl Control instead of Google’s hands-off approach.

Official documentation

Primary-source documentation from the search engines.

Google

• Optimize your crawl budget — the canonical doc (moved from the old “large site owner’s guide” URL in Google’s crawling-infrastructure docs, last updated December 2025): crawl capacity + demand, who needs it, the waste list, and how to optimize. Start here.
• What Crawl Budget Means for Googlebot (Gary Illyes, 2017) — the original framing: crawl rate limit + crawl demand, and the low-value-add URL categories.
• Crawling and Indexing — the hub for robots, sitemaps, canonicalization, and crawl controls.
• Crawling December series (2024) — Googlebot, HTTP caching, faceted navigation, and CDNs.

Bing / Microsoft

• bingbot Series: Maximizing Crawl Efficiency — Bing’s “crawl efficiency” reframing and its north star.
• bingbot Series: Optimizing Crawl Frequency — how Bing decides re-crawl cadence.
• Bing Webmaster Tools — Crawl Control — schedule when bingbot crawls, by hour.
• IndexNow / indexnow.org — push changed URLs so engines don’t need exploratory crawls.

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — the definition

• “The amount of time and resources that Google devotes to crawling a site is commonly called the site’s crawl budget and it’s determined by two main elements: crawl capacity limit and crawl demand.” — Google Search Central docs. Jump to quote
• “Google’s crawlers calculate a crawl capacity limit, which is the maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches.” Jump to quote
• “Taking crawl rate and crawl demand together we define crawl budget as the number of URLs Googlebot can and wants to crawl.” — Gary Illyes, Google (2017). Jump to quote

Google — when you don’t need to care

• “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.” Jump to quote
• “100k URLs is usually not enough to affect crawl budget (it’s <1/minute over 3 months).” — John Mueller, Google (2021). Read the coverage

Google — what you control, and crawling vs ranking

• “If many of these URLs are duplicates, or you don’t want them crawled for some other reason (removed, unimportant, and so on), this wastes a lot of Google crawling time on your site. This is the factor that you can positively control the most.” Jump to quote
• “An increased crawl rate will not necessarily lead to better positions in Search results. Google uses hundreds of signals to rank the results, and while crawling is necessary for being in the results, it’s not a ranking signal.” — Gary Illyes, Google (2017). Jump to quote

Google — the waste list and what not to do

• “Faceted navigation and session identifiers / On-site duplicate content / Soft error pages / Hacked pages / Infinite spaces and proxies / Low quality and spam content” — the low-value-add categories, in order of significance. — Gary Illyes, Google (2017). Jump to quote
• “Consolidate duplicate content to focus crawling on unique content rather than unique URLs.” Jump to quote
• “Don’t use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time.” Jump to quote
• “Google won’t shift this newly available crawl budget to other pages unless Google is already hitting your site’s serving limit.” Jump to quote

Gary Illyes, Google — scheduling and host load (via Search Engine Roundtable’s coverage of a Stone Temple Q&A)

• “Host load kind of sets a bucket of URLs in importance order and GoogleBot will crawl in that order based on the schedule the host load decided. If Google thinks your server can handle it, it will crawl the whole bucket, if not, it will stop.” Jump to quote

Fabrice Canel, Microsoft Bing

• “The crawl efficiency is how often we crawl and discover new and fresh content per page crawled.” Jump to quote
• “Less is more for SEO. Never forget that. Less URLs to crawl, better for SEO.” Jump to quote

Crawl-budget optimization checklist

Only worth working through if you’ve confirmed you actually have a budget problem (big site, slow same-day crawling, or a swelling “Discovered – currently not indexed” pile). Then:

• Confirmed the problem is real — checked GSC Crawl Stats and “Discovered – currently not indexed” before optimizing anything.
• Picked one canonical convention (host, protocol, trailing slash, case) and consolidated duplicates to it.
• Handled URL parameters — canonicalized, or blocked the ones that only create duplicate/low-value variants.
• Tamed faceted navigation — blocked worthless filter/sort combinations in robots.txt, or moved them behind a # so they aren’t crawlable.
• Fixed soft 404s; returning real 404/410 for permanently removed pages.
• Removed or shortened redirect chains.
• Sitemaps list only canonical, indexable URLs with an accurate lastmod.
• Did not use noindex to “save” budget (Google requests it anyway) and did not expect robots.txt blocks to reallocate budget to other pages.
• Strengthened internal links to important and newly published pages.
• Improved server response time and page load efficiency.
• (Bing) Adopted IndexNow to push changes; used Crawl Control if server load is the constraint.

The mental models

1. Budget = capacity × demand. Crawl capacity is what your server can take; crawl demand is popularity + staleness + perceived inventory. Both have to be there: a fast server with low demand still gets crawled less, and high demand throttles against a struggling server. To raise effective budget you almost always remove waste, not “ask for more.”

2. The “do I even have a problem?” gate. Run this before touching anything:

• Pages crawled the same day they publish? → No problem. Stop here.
• Under ~100k URLs (Mueller’s rule of thumb)? → Almost certainly no problem.
• 1M+ pages changing weekly, or 10k+ changing daily, or a big “Discovered – currently not indexed” pile? → Now it’s worth your time.

3. Remove waste before asking Google to crawl more. The single highest-leverage idea on this page. Google tries to crawl everything it knows about; if half of that is duplicates, facets, and soft 404s, it’s spending your budget on junk. Cut the junk and the budget consolidates onto pages that matter — that’s “the factor you can positively control the most.”

4. Crawling ≠ ranking. Crawling is a gate, not a scoreboard. More crawling won’t lift positions. So the objective is coverage of the right URLs, never raw crawl volume.

5. It’s a priority queue, not a quota. Internally it’s scheduling by importance gated by host load — a prioritized bucket of URLs, not a flat “N pages per day.” You move up the queue by being more important (links, demand) and by not wasting fetches.

Crawl budget — cheat sheet

What wastes it (Google’s order of significance)

Plus: long redirect chains and slow/heavy pages make every fetch cost more.

Do you even need to care?

• Same-day crawling → no.
• < ~100k URLs (Mueller) → almost certainly no.
• 1M+ pages weekly change, or 10k+ daily change, or big “Discovered – currently not indexed” → yes.

Fixes that work

• Consolidate duplicates / canonicalize.
• Block worthless paths in robots.txt (facets, params) — never use noindex to “save” budget.
• Fix soft 404s; return real 404/410.
• Shorten redirect chains; speed up the server.
• Clean sitemaps; strengthen internal links.

Only two ways to increase budget (Google): more server capacity and higher content quality. One without the other won’t do it.

Bing: it’s “crawl efficiency” — push with IndexNow, schedule with Crawl Control.

Tools for measuring and managing crawl budget

• Google Search Console — Crawl Stats report (Settings → Crawl Stats) — Google’s own view: total requests over time, average response time, host status, and breakdowns by response code, file type, and Googlebot type. Start here.
• GSC “Discovered – currently not indexed” (Pages report) — the warning light for URLs Google knows about but isn’t crawling.
• Server log file analysis — the ground truth for which URLs bots actually hit and how often. Tools: Screaming Frog Log File Analyser, or pipe logs into BigQuery / a log platform. (See log file analysis.)
• Ahrefs Site Audit / Screaming Frog SEO Spider — simulate a crawl to surface duplicate URLs, redirect chains, soft 404s, blocked URLs, and trap-like faceted spaces.
• Ahrefs Webmaster Tools — free crawl + audit for sites you verify.
• URL Inspection (GSC) — quick same-day-crawl gut check for a single URL.
• Bing Webmaster Tools — Crawl Control — schedule bingbot’s crawl by hour if server load is the constraint.

Do you have a crawl-budget problem?

Monthly crawl-budget health check

1. Export search-crawler requests from server logs for the same reporting window each month.
2. Split requests by status code, directory, page type, and whether the URL is indexable.
3. Review high-volume parameter patterns, duplicate paths, redirects, soft 404s, and server errors.
4. Compare crawl share for valuable templates with the previous period and annotate launches or migrations.
5. Assign each new waste pattern an owner and a control: links, parameters, redirects, canonicalization, removal, or server repair.
6. Recheck the affected pattern in the next log window; the SOP is done when valuable URLs retain access and waste declines from the site’s own baseline.

Crawl-budget mistakes

• Chasing a bigger request count. More crawling does not improve rankings. Track whether important URLs are crawled when needed instead.
• Blocking waste only in robots.txt. That may stop fetching without removing discovered URLs or fixing the links that create them. Remove the crawl paths and consolidate inventory where appropriate.
• Updating every sitemap lastmod on every build. False freshness teaches crawlers not to trust the signal. Change it only for meaningful page updates.
• Ignoring server errors while tuning URL patterns. A capacity problem caps useful crawling. Repair timeouts and 5xx responses first.
• Treating every excluded URL as waste. Some non-indexed resources support rendering or discovery. Classify purpose before blocking anything.

Prompt: classify crawl-log waste

Paste a log-derived CSV with URL, template, status, bot, hits, bytes, and indexability. Remove query values or user data first.

Act as a technical SEO analyst. Classify each URL pattern as valuable crawling, necessary support crawling, redirect/error waste, duplicate/parameter waste, or unclear. Do not infer intent from the URL alone: list the evidence needed for every unclear row. Rank patterns by crawler requests and bytes, propose the safest control, and state what could break if that control is wrong. Return a table plus a short validation plan.

Prompt: challenge a crawl-budget diagnosis

Review the crawl-budget diagnosis below. Separate evidence of capacity, demand, discovery, and URL-inventory problems. Flag claims that confuse crawling with indexing or rankings. Then give the three smallest tests that would confirm or reject the diagnosis. Do not invent thresholds; use changes against the site's own baseline.

[PASTE DIAGNOSIS AND OBSERVATIONS]

Summarize crawler status codes from an access log

Run this against an nginx/Apache-style log after adjusting the bot pattern to your verified crawler traffic:

awk 'BEGIN{IGNORECASE=1} /Googlebot|bingbot/ {print $9}' access.log | sort | uniq -c | sort -nr

On PowerShell:

Select-String -Path .\access.log -Pattern 'Googlebot|bingbot' | ForEach-Object { if ($_.Line -match '"\s(\d{3})\s') { $Matches[1] } } | Group-Object | Sort-Object Count -Descending

Extract query-parameter families

Use this regex in a crawler export or text editor to capture the first query-parameter name:

\?([^=&]+)(?:=[^&]*)?

Group 1 is the parameter name. High request counts identify patterns to investigate, not URLs to block automatically.

Test yourself: Crawl budget

Resources worth your time

My related writing

• When Should You Worry About Crawl Budget? — my full Ahrefs guide on this topic, with the optimization detail and faceted-nav decisions.
• How to Fix “Discovered – currently not indexed” — the GSC signal that doubles as a crawl-budget warning light.
• The Beginner’s Guide to Technical SEO — where crawl budget fits in the bigger picture.
• Enterprise SEO Strategies — for the large-site minority that actually needs this.

My speaking

• How Search Works (SlideShare) — my walkthrough of the crawl-demand vs crawl-rate-limit model. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From around the industry

• Google’s Crawling December series (official, but the best concentrated set of crawl explainers, including faceted navigation).
• Google explains what “crawl budget” means for webmasters (Barry Schwartz, Search Engine Land, 2017) — plain-English summary of Google’s original crawl-budget post; good for the “most sites don’t need to worry” context.
• Gary Illyes Explains The Difference Between Crawl Budget, Scheduling & Host Load (Search Engine Roundtable) — the source for the “importance-order bucket” / host-load framing that shows it’s a priority queue, not a flat quota.
• Google: 100,000 URLs Usually Won’t Impact Crawl Budget (Search Engine Roundtable, 2021) — Mueller’s concrete 100k anchor and the <1/minute-over-3-months context.
• Google’s Crawling Priorities: Insights From Analyst Gary Illyes (Search Engine Journal) — covers the “fixed daily page quota” debunk and the “convince search your stuff is worth fetching” angle.
• The five infrastructure gates behind crawl, render, and index (Search Engine Land) — source for Fabrice Canel’s “Less is more for SEO” quote and Bing’s crawl-efficiency philosophy.
• bingbot Series: Optimizing Crawl Frequency (Bing Webmaster Blog, Fabrice Canel) — how Bing decides re-crawl cadence; the companion to the Maximizing Crawl Efficiency post.
• r/TechSEO — the community for crawl/index debugging.

Stats worth citing

• 100k URLs usually isn’t enough to matter. John Mueller’s concrete anchor: “100k URLs is usually not enough to affect crawl budget (it’s <1/minute over 3 months).” The single most useful number for de-escalating crawl-budget worry. Source
• The thresholds where it starts to matter: 1 million+ pages changing roughly weekly, or 10,000+ pages changing daily (Google’s rough estimates, not exact thresholds). Source
• ~60% of the web is duplicate content — Google’s internal estimate, which is why duplicate URLs are such a reliable crawl-budget drain. Source

The standing KPI for crawl efficiency

Crawl budget isn’t a number Google hands you — you infer it from where Googlebot actually spends its requests. The KPI is the split: how much crawl lands on pages you want indexed versus pages that are pure waste. (First, the honesty check: this only matters at scale — Google says crawl budget is a concern mainly for sites around 1M+ pages, or mid-size sites generating lots of auto-created URLs. A small static site can skip this metric entirely.)

Crawl allocation — valuable vs. wasted requests

• Metric — The share of verified-Googlebot requests hitting important, indexable URLs vs. wasted ones (duplicates, parameter/faceted URLs, non-canonical variants, redirect hops, 4xx/5xx).
• What it tells you — Whether your crawl capacity is being spent on pages that can rank. A rising wasted share is a leading indicator of discovery/indexing lag before it ever shows up as a ranking problem.
• How to pull it — Run server logs through the Log File Analyzer and segment verified-Googlebot hits by URL class; corroborate with the GSC Crawl Stats report (total requests, by response, by file purpose, host status).
• Benchmark / realistic range — Situational — there’s no honest universal “waste %,” because it depends on your URL architecture. Establish your own baseline and drive the wasted share down over time; the trend is the signal, not an invented absolute number.
• Cadence — Monthly, and more often right after a structural change (new facets, a migration) or when Crawl Stats shows a sudden spike in requests or in 4xx/5xx responses. Wasted-crawl is a leading metric; indexed-page counts lag it.

Crawl Budget

The number of URLs an engine will crawl in a timeframe.

Related: Log file analysis

Crawl Budget

The number of URLs a search engine will crawl on your site in a given timeframe. It mostly matters for very large sites; for most, it is a distraction from real problems.

Related: Log file analysis

Changelog