Crawl Budget

TL;DR — Crawl budgetThe number of URLs an engine will crawl in a timeframe. is how much a search engine is willing to crawl your site. It’s two things multiplied together: how much your server can handle and how much Google wants to crawl. CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. more doesn’t make you rank higher — and for most sites, this is a non-issue. If your pages get crawled the same day you publish them, you don’t have a crawl budget problem.

What crawl budget is

When a search engine crawls your site, it doesn’t crawl forever. It crawls some number of your URLs in a given window of time, then moves on. That amount is what SEOs call your crawl budget.

It comes down to two questions the search engine is constantly answering:

• How much can I crawl? Your server can only take so much before it slows down or starts throwing errors. Google watches for that and backs off. This is the crawl capacity (older docs called it the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. limit).
• How much do I want to crawl? Popular pages and pages that change often get crawled more. Pages nobody links to, or that never change, get crawled rarely. This is crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side..

Multiply the two and you get your crawl budget: roughly, the number of URLs Google can and wants to crawl. Evidence for this claim Google defines crawl budget using crawl capacity limit and crawl demand. Scope: Google Search crawling for larger sites. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide

One distinction worth making early: crawl budget is about whether Google fetches a URL at all. Whether it then indexes what it finds is a separate, later decision. A page can be crawled and still not get indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. doesn’t guarantee indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., it’s just the gate you have to get through first.

The thing to understand first: you probably don’t need to worry about this

This is the part most articles bury. Most sites do not have a crawl budget problem. Google says so directly — if your pages get crawled the same day they’re published, or your site isn’t huge and fast-changing, you can skip the whole topic. Evidence for this claim Google says sites without many rapidly changing pages, or whose pages are crawled the day they publish, generally do not need crawl-budget guidance. Scope: Google's rough applicability guidance, not a guarantee for every site. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide

A simple gut check: search for a few of your URLs in Google, or use the URL Inspection tool in Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.. If your new pages show up within a day or two, your crawl budget is fine. Spend your time on content and links instead.

Crawl budget starts to matter when a site has hundreds of thousands or millions of pages, especially if a lot of them change constantly (think a big ecommerce store or a large news site).

More crawling does not mean better rankings

Here’s a myth worth killing early: crawling a page more often doesn’t push it up the results. Crawling is just how the engine finds and downloads your page. It’s a gate you have to get through to rank at all — but once you’re through it, how often you get re-crawled isn’t a ranking factor.

So the goal isn’t “get crawled more.” The goal is to make sure the pages you care about get crawled, and that the engine isn’t wasting its time on junk URLs.

What wastes crawl budget

If you do have a big site, the budget gets drained by:

• Filter and sort URLs (faceted navigationFaceted navigation (faceted search, product filtering) lets visitors refine a list of products or content by attribute — price, color, size, brand, rating. The SEO problem: each filter combination can spawn a distinct crawlable URL, turning a small catalog into millions of near-duplicate pages that waste crawl budget and dilute ranking signals.) — every filter combination on an ecommerce store can spawn thousands of near-identical URLs.
• Duplicate pagesThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling. — the same content reachable at multiple URLs (with and without www, with parameters tacked on, etc.).
• Soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing. — “not foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore.” pages that return a 200 OK status, so the engine keeps crawling them.
• Broken redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency. and slow pages — these make every crawl cost more.

The fix is almost always removing waste so the budget lands on real pages — not trying to convince Google to crawl “more.”

Want the full model — crawl capacity vs. demand, the exact size thresholds, what to do about faceted navigationFaceted navigation (faceted search, product filtering) lets visitors refine a list of products or content by attribute — price, color, size, brand, rating. The SEO problem: each filter combination can spawn a distinct crawlable URL, turning a small catalog into millions of near-duplicate pages that waste crawl budget and dilute ranking signals., how Bing handles this differently, and how to measure it — switch to the Advanced tab.

TL;DR — Crawl budgetThe number of URLs an engine will crawl in a timeframe. = crawl capacity limitCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. (what your server can take) × crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. (popularity + staleness + perceived inventoryCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side.). It’s an efficiency concern, not a ranking signal. Internally it’s scheduling by importance gated by host load, not a flat per-site quota. Most sites can ignore it — Mueller pegs 100k URLs as “usually not enough,” and Google tells same-day-crawled sites to skip the guide. It bites at ~1M+ pages (weekly change), 10k+ pages (daily change), or when “Discovered – currently not indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.” balloons. The highest-leverage move is cutting waste — faceted nav, duplicates, soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing., infinite spaces — so the budget consolidates on URLs that matter.

The two-factor model

Google defines it cleanly: “The amount of time and resources that Google devotes to crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. a site is commonly called the site’s crawl budget and it’s determined by two main elements: crawl capacity limit and crawl demand.” The 2017 framing from Gary Illyes is the one-liner I still reach for: crawl budget is “the number of URLs GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. can and wants to crawl.” Evidence for this claim Google defines crawl budget using crawl capacity limit and crawl demand. Scope: Google Search crawling for larger sites. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide Keep this scoped: crawl budget governs fetching, not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. A crawled URL still goes through a separate indexing decision — folding the two together overstates what crawl budget controls.

Crawl capacity limit (the supply side). This is “the maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches.” It moves with your server’s health. Respond fast and clean and the limit rises; serve slow responses, 5xx errors, or 429s and GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. backs off. In my How Search Works deck I list the same rate-limit triggers: server stability, slow responses, 5xx server errors, and 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. (too many requests). This is the “can.”

Crawl demand (the demand side). Driven by popularity (how linked-to / important a URL is) and staleness (how long since it was last crawled, how often it changes). That same deck breaks demand into PageRankPageRank is Google's original recursive link-graph algorithm: a page's score depends on the scores of the pages linking to it, and in the published model each page's score is split across its outbound links (the simplified version: links are weighted votes). Google says it's evolved since launch but still part of its core ranking systems., how frequently the page changes, time since last crawl, and major site changes. Critically, Google flags perceived inventory as the lever you control most: “Without guidance from you, Google tries to crawl all or most of the URLs that it knows about on your site. If many of these URLs are duplicates, or you don’t want them crawled for some other reason… this wastes a lot of Google crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. time on your site. This is the factor that you can positively control the most.”

A couple of structural facts that catch people out:

• Budget is per hostname. “https://www.example.com/ and https://code.example.com/ are two different hostnames, and therefore have separate crawl budgets.” Subdomains don’t share.
• The different Googlebot types likely draw from one pool. In my own reporting, image, news, video, ads, and the rest appear to pull from the same per-site budget — I don’t have a current primary Google source pinning this down exactly, so treat it as a practitioner observation rather than documented policy. Either way, it’s worth checking the Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root).’s by-crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-type breakdown if you suspect one type is elbowing out the rest.

What it really is internally: scheduling by importance

“Crawl budget” is an SEO-coined umbrella term. Internally it’s closer to scheduling gated by host load. As Illyes has described it, Google’s scheduler “sets a bucket of URLs in importance order and GoogleBot will crawl in that order based on the schedule the host load decided. If Google thinks your server can handle it, it will crawl the whole bucket, if not, it will stop.”

That reframes the whole topic. It isn’t a flat “you get N pages a day” quota — it’s a prioritized queue, and crawling tracks search demand. Illyes again: “If search demand goes down, then that also correlates to the crawl limit going down,” and “if you want to increase how much we crawl, then you somehow have to convince search that your stuff is worth fetching, which is basically what the scheduler is listening to.” The Search Relations team has explicitly called the “fixed daily page quota” idea a misconception.

Does your site actually have a crawl budget problem?

This is the most valuable section, so I’ll be blunt: most sites don’t need to worry about crawl budget. Google’s own guide opens with the de-escalation: “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide. For Google Search specifically, merely keeping your sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. up to date and checking your index coverage regularly is adequate.” Evidence for this claim Google says sites without many rapidly changing pages, or whose pages are crawled the day they publish, generally do not need crawl-budget guidance. Scope: Google's rough applicability guidance, not a guarantee for every site. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide

John Mueller gave the concrete number: “100k URLs is usually not enough to affect crawl budget (it’s <1/minute over 3 months).” If you’re under six figures of URLs and pages get crawled promptly, move on.

When it does matter, Google’s rough thresholds are:

• Large sites — 1 million+ unique pages with content that changes moderately often (about weekly).
• Medium or larger sites — 10,000+ unique pages with very rapidly changing content (daily).
• Sites with a large portion of URLs classified as “Discovered – currently not indexed” in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. — that’s the warning light that Google knows about URLs it isn’t getting to.

Google adds the disclaimer that “the numbers given here are a rough estimate… not exact thresholds.” And even on big sites, the nuance from my own work holds: it’s usually new, poorly-linked, or static pages that lag — not your popular ones.

Does crawl budget affect rankings? No.

Crawling is necessary to rank, but it is not a ranking signal. Google in 2017: “An increased crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. will not necessarily lead to better positions in Search results. Google uses hundreds of signals to rank the results, and while crawling is necessary for being in the results, it’s not a ranking signal.” I put it the same way in my Ahrefs guide: “More crawling doesn’t mean you’ll rank better, but if your pages aren’t crawled and indexed they aren’t going to rank at all.” Treat crawl budget as an efficiency problem, full stop.

What wastes crawl budget

Illyes published the canonical list of low-value-add URLs “in order of significance”:

1. Faceted navigationFaceted navigation (faceted search, product filtering) lets visitors refine a list of products or content by attribute — price, color, size, brand, rating. The SEO problem: each filter combination can spawn a distinct crawlable URL, turning a small catalog into millions of near-duplicate pages that waste crawl budget and dilute ranking signals. and session identifiers — the #1 culprit, especially ecommerce filter/sort combinations that multiply URLs combinatorially.
2. On-site duplicate contentThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling. — the classic technical variants: HTTP vs HTTPSHTTPS is the encrypted version of HTTP — it uses TLS to authenticate the server and protect data in transit between a browser and a website. Google announced it as a lightweight ranking signal in 2014 and today conditionally prefers HTTPS pages as canonical; Chrome marks plain HTTP pages 'Not Secure.', non-www vs www, trailing slashA trailing slash is the forward slash (/) at the end of a URL — example.com/page/ versus example.com/page. Except at the bare root domain, the two versions are different URLs to search engines, so you pick one format and enforce it. vs not, uppercase vs lowercase, default/index pages, and URL parametersThe `?key=value` data tacked onto the end of a URL after a question mark — used for tracking, sessions, filtering, sorting, and search — and one of the biggest sources of duplicate URLs and wasted crawling in SEO.. (Roughly 60% of the web is duplicate content, by Google’s own internal estimate.)
3. Soft error pages — soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing. that return 200 keep getting crawled.
4. Hacked pages.
5. Infinite spaces and proxies — calendars, infinite-scroll paginationPagination splits a large set of content — product listings, blog archives, search results — across multiple sequentially numbered URLs. For SEO, each paginated page should be crawlable, indexable, and self-canonical; Google no longer uses rel=prev/next, but Bing still does. that duplicates, faceted combinations; the classic spider-trap patterns.
6. Low-quality and spam content.

The cost is concrete: “Wasting server resources on pages like these will drain crawl activity from pages that do actually have value, which may cause a significant delay in discovering great content on a site.” On top of the list, long redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency. “have a negative effect on crawling,” and slow, heavy pages make every fetch more expensive.

How to optimize it

The whole game is consolidating the budget onto URLs that matter:

• Consolidate duplicates. Google: “Consolidate duplicate content to focus crawling on unique content rather than unique URLs.” Pick one host, one protocol, one trailing-slash convention; canonicalize; handle parameters.
• Block truly worthless paths with robots.txt — but only paths you never want crawled. For faceted navigation, the usual options are blocking the parameter paths in robots.txt or using a # instead of a ? so the URLs aren’t crawlable in the first place.
• Don’t use noindex to save budget. Google: “Don’t use noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed., as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time.” The request still costs you. Block at robots.txt if you never want it fetched.
• Don’t expect robots.txt to reallocate budget. “Google won’t shift this newly available crawl budget to other pages unless Google is already hitting your site’s serving limit.” Blocking junk is good hygiene, but it doesn’t hand the freed-up crawls to your good pages unless you were capacity-bound.
• Fix soft 404s; return real 404/410 for gone pages. “A 404 status code is a strong signal not to crawl that URL again.”
• Shorten redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. chains, keep sitemaps current (honest lastmod), and improve server speed.
• Strengthen internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. to important and new pages — easier than anything else because you fully control it.

And the two — only two — ways Google says you can actually increase budget: “Add more server resources… [and] optimize your content’s quality.” Note the trap there: a faster server lifts the capacity ceiling, but if demand is low Google still crawls less. You need both.

How to measure it

• GSC > Settings > Crawl Stats report — total crawl requests over time, average response time, host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity., and breakdowns by response code, file type, and Googlebot type. This is Google’s own view of how it crawls you.
• Server log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. — the ground truth. Real Googlebot hits by URL pattern, so you can see crawl waste and uncrawled important pages. Verify the bot is genuinely Googlebot via reverse + forward DNS or Google’s published IP ranges (plenty of fake bots spoof the user-agent).
• “Discovered – currently not indexedA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal.” in GSC — treat a growing pile here as a crawl-budget warning light: Google knows about the URLs but isn’t getting to them.

TIP Find where crawler requests are being spent

A sitewide crawl total is not enough; separate live pages from redirects, gone URLs, and server errors before deciding what to fix.

Upload a representative access-log sample and inspect the exact paths behind wasted status groups with my free Log File Analyzer Free

1. Include status, path, user-agent, and timestamp fields so the result can separate crawler activity by outcome.
2. Verify crawler identity before counting requests under a Googlebot name.
3. Fix the exact broken paths, redirect chains, and server errors instead of optimizing the aggregate crawl count.

Tap to enlarge

The finding reports that 64 of 129 Googlebot requests, or 49.6 percent, went to non-200 responses. It breaks that total into 40 requests to 404 or 410 URLs, 12 redirects, and 4 server errors, then directs the reader to the exact paths in the wasted-URL table.

Bing and other engines: “crawl efficiency”

Bing reframes the topic as crawl efficiency rather than budget. Fabrice Canel’s definition: “The crawl efficiency is how often we crawl and discover new and fresh content per page crawled.” The goal is to “crawl an URL only when the content has been added (URL not crawled before), updated (fresh on-page context or useful outbound links).” Bing’s blunt philosophy: “Less is more for SEO. Never forget that. Less URLs to crawl, better for SEO.”

Bing’s preferred fix is IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. — push changed URLs so bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. doesn’t need exploratory crawls — and Crawl Control in Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility., which lets you schedule when bingbot crawls by hour to protect server load. This is a real Google/Bing divergence worth noting: Google deprecated its old crawl-rate limiter in Search Console, while Bing still lets you actively shape the crawl schedule.

Crawl budget myths, corrected

• “Every site should optimize crawl budget.” No — most shouldn’t. Same-day crawling and sub-100k URLs mean you’re fine.
• “More crawling = better rankings.” No. Crawling is necessary but not a ranking signal.
• “It’s a fixed daily page quota.” No — it’s importance-driven scheduling gated by host load.
• “Use noindex to save budget.” No — Google still requests the page first.
• “Block pages in robots.txt to give other pages more budget.” Generally no, unless you’re already at your serving limit.
• “A faster server alone raises your budget.” It lifts the capacity ceiling only; low demand still means fewer crawls.

For the broader pipeline this sits inside — discovery, the crawl scheduler, renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., and how crawling differs from indexing — see the crawling hub. The sibling topics (crawl rate, crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial., spider trapsA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content., and log file analysis) each go deeper on one piece of this.

AI summary

A condensed take on the Advanced version:

• Crawl budgetThe number of URLs an engine will crawl in a timeframe. = crawl capacity limitCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. × crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side.. Capacity is what your server can take (rises on fast/clean responses, drops on 5xx/429); demand is popularity + staleness + perceived inventoryCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side.. Google’s one-liner: “the number of URLs GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. can and wants to crawl.”
• It’s not a ranking factor. CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is necessary to rank, but more crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. doesn’t lift positions — it’s purely an efficiency concern.
• Internally it’s scheduling by importance, gated by host load — a prioritized queue, not a flat daily page quota. Crawling tracks search demand.
• Most sites don’t need to manage it. Mueller: 100k URLs usually isn’t enough to matter. Google: if pages crawl same-day, skip the guide. It bites at ~1M+ pages (weekly change), 10k+ (daily change), or a big “Discovered – currently not indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.” pile.
• What wastes it (Google’s order): faceted nav / session IDs, duplicate content, soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing., hacked pages, infinite spaces, low-quality contentThin content is web content that provides little or no value to users. Google's spam policies name it 'thin content with little or no added value' — and it's about value per page, not word count. — plus redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency. and slow pages.
• Optimize by removing waste: consolidate duplicates, block worthless paths in robots.txt (not noindex — Google still requests first), fix soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing., shorten redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., keep sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. clean, strengthen internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them.. Only two ways to truly increase budget: more server capacity and higher content quality.
• Measure with GSC Crawl Stats, server log analysis, and the “Discovered – currently not indexed” report.
• Bing calls it crawl efficiency (“less is more”), and pushes IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. + Crawl Control instead of Google’s hands-off approach.

Official documentation

Primary-source documentation from the search engines.

Google

• Optimize your crawl budget — the canonical doc (moved from the old “large site owner’s guide” URL in Google’s crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.-infrastructure docs, last updated December 2025): crawl capacity + demand, who needs it, the waste list, and how to optimize. Start here.
• What Crawl Budget Means for Googlebot (Gary Illyes, 2017) — the original framing: crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. limit + crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side., and the low-value-add URL categories.
• Crawling and Indexing — the hub for robots, sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it., and crawl controls.
• Crawling December series (2024) — GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., HTTP cachingCaching stores a copy of a page or resource — in a browser, a CDN edge node, or a search crawler's own cache — so it can be served again without regenerating or re-downloading it. It isn't a direct ranking factor, but it feeds page speed and crawl efficiency., faceted navigationFaceted navigation (faceted search, product filtering) lets visitors refine a list of products or content by attribute — price, color, size, brand, rating. The SEO problem: each filter combination can spawn a distinct crawlable URL, turning a small catalog into millions of near-duplicate pages that waste crawl budget and dilute ranking signals., and CDNs.

Bing / Microsoft

• bingbot Series: Maximizing Crawl Efficiency — Bing’s “crawl efficiency” reframing and its north star.
• bingbot Series: Optimizing Crawl Frequency — how Bing decides re-crawl cadence.
• Bing Webmaster Tools — Crawl Control — schedule when bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. crawls, by hour.
• IndexNow / indexnow.org — push changed URLs so engines don’t need exploratory crawls.

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — the definition

• “The amount of time and resources that Google devotes to crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. a site is commonly called the site’s crawl budgetThe number of URLs an engine will crawl in a timeframe. and it’s determined by two main elements: crawl capacity limitCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. and crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side..” — Google Search Central docs. Jump to quote
• “Google’s crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. calculate a crawl capacity limit, which is the maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches.” Jump to quote
• “Taking crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. and crawl demand together we define crawl budget as the number of URLs GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. can and wants to crawl.” — Gary Illyes, Google (2017). Jump to quote

Google — when you don’t need to care

• “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.” Jump to quote
• “100k URLs is usually not enough to affect crawl budget (it’s <1/minute over 3 months).” — John Mueller, Google (2021). Read the coverage

Google — what you control, and crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. vs ranking

• “If many of these URLs are duplicates, or you don’t want them crawled for some other reason (removed, unimportant, and so on), this wastes a lot of Google crawling time on your site. This is the factor that you can positively control the most.” Jump to quote
• “An increased crawl rate will not necessarily lead to better positions in Search results. Google uses hundreds of signals to rank the results, and while crawling is necessary for being in the results, it’s not a ranking signal.” — Gary Illyes, Google (2017). Jump to quote

Google — the waste list and what not to do

• “Faceted navigationFaceted navigation (faceted search, product filtering) lets visitors refine a list of products or content by attribute — price, color, size, brand, rating. The SEO problem: each filter combination can spawn a distinct crawlable URL, turning a small catalog into millions of near-duplicate pages that waste crawl budget and dilute ranking signals. and session identifiers / On-site duplicate contentThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling. / Soft error pages / Hacked pages / Infinite spaces and proxies / Low quality and spam content” — the low-value-add categories, in order of significance. — Gary Illyes, Google (2017). Jump to quote
• “Consolidate duplicate contentThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling. to focus crawling on unique content rather than unique URLs.” Jump to quote
• “Don’t use noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed., as Google will still request, but then drop the page when it sees a noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. meta tag or header in the HTTP response, wasting crawling time.” Jump to quote
• “Google won’t shift this newly available crawl budget to other pages unless Google is already hitting your site’s serving limit.” Jump to quote

Gary Illyes, Google — scheduling and host load (via Search Engine Roundtable’s coverage of a Stone Temple Q&A)

• “Host load kind of sets a bucket of URLs in importance order and GoogleBotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. will crawl in that order based on the schedule the host load decided. If Google thinks your server can handle it, it will crawl the whole bucket, if not, it will stop.” Jump to quote

Fabrice Canel, Microsoft Bing

• “The crawl efficiency is how often we crawl and discover new and fresh contentContent freshness is how recent or up-to-date a page is — by its original publish date, its last substantive revision, or the currency of the facts inside it. It only helps rankings when the query itself benefits from recent results (Query Deserves Freshness), and cosmetic date changes with no real update don't count. per page crawled.” Jump to quote
• “Less is more for SEO. Never forget that. Less URLs to crawl, better for SEO.” Jump to quote

Crawl-budget optimization checklist

Only worth working through if you’ve confirmed you actually have a budget problem (big site, slow same-day crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., or a swelling “Discovered – currently not indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.” pile). Then:

• Confirmed the problem is real — checked GSC Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). and “Discovered – currently not indexed” before optimizing anything.
• Picked one canonical convention (host, protocol, trailing slashA trailing slash is the forward slash (/) at the end of a URL — example.com/page/ versus example.com/page. Except at the bare root domain, the two versions are different URLs to search engines, so you pick one format and enforce it., case) and consolidated duplicates to it.
• Handled URL parametersThe `?key=value` data tacked onto the end of a URL after a question mark — used for tracking, sessions, filtering, sorting, and search — and one of the biggest sources of duplicate URLs and wasted crawling in SEO. — canonicalized, or blocked the ones that only create duplicate/low-value variants.
• Tamed faceted navigationFaceted navigation (faceted search, product filtering) lets visitors refine a list of products or content by attribute — price, color, size, brand, rating. The SEO problem: each filter combination can spawn a distinct crawlable URL, turning a small catalog into millions of near-duplicate pages that waste crawl budget and dilute ranking signals. — blocked worthless filter/sort combinations in robots.txt, or moved them behind a # so they aren’t crawlable.
• Fixed soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing.; returning real 404/410 for permanently removed pages.
• Removed or shortened redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency..
• SitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. list only canonical, indexable URLs with an accurate lastmod.
• Did not use noindex to “save” budget (Google requests it anyway) and did not expect robots.txt blocks to reallocate budget to other pages.
• Strengthened internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. to important and newly published pages.
• Improved server response time and page load efficiency.
• (Bing) Adopted IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. to push changes; used Crawl Control if server load is the constraint.

The mental models

1. Budget = capacity × demand. Crawl capacity is what your server can take; crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. is popularity + staleness + perceived inventoryCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side.. Both have to be there: a fast server with low demand still gets crawled less, and high demand throttles against a struggling server. To raise effective budget you almost always remove waste, not “ask for more.”

2. The “do I even have a problem?” gate. Run this before touching anything:

• Pages crawled the same day they publish? → No problem. Stop here.
• Under ~100k URLs (Mueller’s rule of thumb)? → Almost certainly no problem.
• 1M+ pages changing weekly, or 10k+ changing daily, or a big “Discovered – currently not indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.” pile? → Now it’s worth your time.

3. Remove waste before asking Google to crawl more. The single highest-leverage idea on this page. Google tries to crawl everything it knows about; if half of that is duplicates, facets, and soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing., it’s spending your budget on junk. Cut the junk and the budget consolidates onto pages that matter — that’s “the factor you can positively control the most.”

4. CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. ≠ ranking. CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is a gate, not a scoreboard. More crawling won’t lift positions. So the objective is coverage of the right URLs, never raw crawl volume.

5. It’s a priority queue, not a quota. Internally it’s scheduling by importance gated by host load — a prioritized bucket of URLs, not a flat “N pages per day.” You move up the queue by being more important (links, demand) and by not wasting fetches.

Crawl budget — cheat sheet

What wastes it (Google’s order of significance)

Plus: long redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency. and slow/heavy pages make every fetch cost more.

Do you even need to care?

• Same-day crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. → no.
• < ~100k URLs (Mueller) → almost certainly no.
• 1M+ pages weekly change, or 10k+ daily change, or big “Discovered – currently not indexed” → yes.

Fixes that work

• Consolidate duplicates / canonicalize.
• Block worthless paths in robots.txt (facets, params) — never use noindex to “save” budget.
• Fix soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing.; return real 404/410.
• Shorten redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency.; speed up the server.
• Clean sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing.; strengthen internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them..

Only two ways to increase budget (Google): more server capacity and higher content quality. One without the other won’t do it.

Bing: it’s “crawl efficiency” — push with IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it., schedule with Crawl Control.

Tools for measuring and managing crawl budget

• Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. — Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). (Settings → Crawl Stats) — Google’s own view: total requests over time, average response time, host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity., and breakdowns by response code, file type, and GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. type. Start here.
• GSC “Discovered – currently not indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.” (Pages report) — the warning light for URLs Google knows about but isn’t crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..
• Server log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. — the ground truth for which URLs botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. actually hit and how often. Tools: Screaming Frog Log File Analyser, or pipe logs into BigQuery / a log platform. (See log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened..)
• Ahrefs Site Audit / Screaming Frog SEO Spider — simulate a crawl to surface duplicate URLs, redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency., soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing., blocked URLs, and trap-like faceted spaces.
• Ahrefs Webmaster Tools — free crawl + audit for sites you verify.
• URL Inspection (GSC) — quick same-day-crawl gut check for a single URL.
• Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. — Crawl Control — schedule bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.’s crawl by hour if server load is the constraint.

Do you have a crawl-budget problem?

Monthly crawl-budget health check

1. Export search-crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. requests from server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. for the same reporting window each month.
2. Split requests by status code, directory, page type, and whether the URL is indexable.
3. Review high-volume parameter patterns, duplicate paths, redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing., and server errors.
4. Compare crawl share for valuable templates with the previous period and annotate launches or migrations.
5. Assign each new waste pattern an owner and a control: links, parameters, redirects, canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it., removal, or server repair.
6. Recheck the affected pattern in the next log window; the SOP is done when valuable URLs retain access and waste declines from the site’s own baseline.

Crawl-budget mistakes

• Chasing a bigger request count. More crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. does not improve rankings. Track whether important URLs are crawled when needed instead.
• Blocking waste only in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.. That may stop fetching without removing discovered URLs or fixing the links that create them. Remove the crawl paths and consolidate inventory where appropriate.
• Updating every sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. lastmod on every build. False freshness teaches crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. not to trust the signal. Change it only for meaningful page updates.
• Ignoring server errors while tuning URL patterns. A capacity problem caps useful crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. Repair timeouts and 5xx responses first.
• Treating every excluded URL as waste. Some non-indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. resources support renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. or discovery. Classify purpose before blocking anything.

Prompt: classify crawl-log waste

Paste a log-derived CSV with URL, template, status, botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., hits, bytes, and indexability. Remove query values or user data first.

Act as a technical SEO analyst. Classify each URL pattern as valuable crawling, necessary support crawling, redirect/error waste, duplicate/parameter waste, or unclear. Do not infer intent from the URL alone: list the evidence needed for every unclear row. Rank patterns by crawler requests and bytes, propose the safest control, and state what could break if that control is wrong. Return a table plus a short validation plan.

Prompt: challenge a crawl-budget diagnosis

Review the crawl-budget diagnosis below. Separate evidence of capacity, demand, discovery, and URL-inventory problems. Flag claims that confuse crawling with indexing or rankings. Then give the three smallest tests that would confirm or reject the diagnosis. Do not invent thresholds; use changes against the site's own baseline.

[PASTE DIAGNOSIS AND OBSERVATIONS]

Summarize crawler status codes from an access log

Run this against an nginx/Apache-style log after adjusting the botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. pattern to your verified crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. traffic:

awk 'BEGIN{IGNORECASE=1} /Googlebot|bingbot/ {print $9}' access.log | sort | uniq -c | sort -nr

On PowerShell:

Select-String -Path .\access.log -Pattern 'Googlebot|bingbot' | ForEach-Object { if ($_.Line -match '"\s(\d{3})\s') { $Matches[1] } } | Group-Object | Sort-Object Count -Descending

Extract query-parameter families

Use this regex in a crawler export or text editor to capture the first query-parameter name:

\?([^=&]+)(?:=[^&]*)?

Group 1 is the parameter name. High request counts identify patterns to investigate, not URLs to block automatically.

Test yourself: Crawl budget

Resources worth your time

My related writing

• When Should You Worry About Crawl Budget? — my full Ahrefs guide on this topic, with the optimization detail and faceted-nav decisions.
• How to Fix “Discovered – currently not indexed” — the GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. signal that doubles as a crawl-budget warning light.
• The Beginner’s Guide to Technical SEO — where crawl budgetThe number of URLs an engine will crawl in a timeframe. fits in the bigger picture.
• Enterprise SEO Strategies — for the large-site minority that actually needs this.

My speaking

• How Search Works (SlideShare) — my walkthrough of the crawl-demand vs crawl-rate-limit model. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From around the industry

• Google’s Crawling December series (official, but the best concentrated set of crawl explainers, including faceted navigationFaceted navigation (faceted search, product filtering) lets visitors refine a list of products or content by attribute — price, color, size, brand, rating. The SEO problem: each filter combination can spawn a distinct crawlable URL, turning a small catalog into millions of near-duplicate pages that waste crawl budget and dilute ranking signals.).
• Google explains what “crawl budget” means for webmasters (Barry Schwartz, Search Engine Land, 2017) — plain-English summary of Google’s original crawl-budget post; good for the “most sites don’t need to worry” context.
• Gary Illyes Explains The Difference Between Crawl Budget, Scheduling & Host Load (Search Engine Roundtable) — the source for the “importance-order bucket” / host-load framing that shows it’s a priority queue, not a flat quota.
• Google: 100,000 URLs Usually Won’t Impact Crawl Budget (Search Engine Roundtable, 2021) — Mueller’s concrete 100k anchor and the <1/minute-over-3-months context.
• Google’s Crawling Priorities: Insights From Analyst Gary Illyes (Search Engine Journal) — covers the “fixed daily page quota” debunk and the “convince search your stuff is worth fetching” angle.
• The five infrastructure gates behind crawl, render, and index (Search Engine Land) — source for Fabrice Canel’s “Less is more for SEO” quote and Bing’s crawl-efficiency philosophy.
• bingbot Series: Optimizing Crawl Frequency (Bing WebmasterMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. Blog, Fabrice Canel) — how Bing decides re-crawl cadence; the companion to the Maximizing Crawl Efficiency post.
• r/TechSEO — the community for crawl/indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. debugging.

Stats worth citing

• 100k URLs usually isn’t enough to matter. John Mueller’s concrete anchor: “100k URLs is usually not enough to affect crawl budgetThe number of URLs an engine will crawl in a timeframe. (it’s <1/minute over 3 months).” The single most useful number for de-escalating crawl-budget worry. Source
• The thresholds where it starts to matter: 1 million+ pages changing roughly weekly, or 10,000+ pages changing daily (Google’s rough estimates, not exact thresholds). Source
• ~60% of the web is duplicate contentThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling. — Google’s internal estimate, which is why duplicate URLs are such a reliable crawl-budget drain. Source

The standing KPI for crawl efficiency

Crawl budgetThe number of URLs an engine will crawl in a timeframe. isn’t a number Google hands you — you infer it from where GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. actually spends its requests. The KPI is the split: how much crawl lands on pages you want indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. versus pages that are pure waste. (First, the honesty check: this only matters at scale — Google says crawl budget is a concern mainly for sites around 1M+ pages, or mid-size sites generating lots of auto-created URLs. A small static site can skip this metric entirely.)

Crawl allocation — valuable vs. wasted requests

• Metric — The share of verified-GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. requests hitting important, indexable URLs vs. wasted ones (duplicates, parameter/faceted URLs, non-canonical variants, redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. hops, 4xx/5xx).
• What it tells you — Whether your crawl capacity is being spent on pages that can rank. A rising wasted share is a leading indicator of discovery/indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. lag before it ever shows up as a ranking problem.
• How to pull it — Run server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. through the Log File Analyzer and segment verified-Googlebot hits by URL class; corroborate with the GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). report (total requests, by response, by file purpose, host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity.).
• Benchmark / realistic range — Situational — there’s no honest universal “waste %,” because it depends on your URL architecture. Establish your own baseline and drive the wasted share down over time; the trend is the signal, not an invented absolute number.
• Cadence — Monthly, and more often right after a structural change (new facets, a migration) or when Crawl Stats shows a sudden spike in requests or in 4xx/5xx responses. Wasted-crawl is a leading metric; indexed-page counts lag it.

Crawl Budget

The number of URLs an engine will crawl in a timeframe.

Related: Log file analysis

Crawl Budget

The number of URLs a search engine will crawl on your site in a given timeframe. It mostly matters for very large sites; for most, it is a distraction from real problems.

Related: Log file analysis

Changelog