Crawl Rate

TL;DR — Crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. is how fast a search engine fetches pages from your site. Google sets it automatically based on how healthy your server is — there’s no longer a button to turn it up or down. If your server is getting hammered, the fix is to make it faster (or, for a day or two only, have it return a “slow down” error). CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. faster doesn’t help you rank.

What crawl rate is

When a search engine crawls your site, it doesn’t grab every page at once — it paces itself. Crawl rate is that pace: how many pages a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. fetches per moment, and how long it waits between fetches. GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. for Google, BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. for Bing. Evidence for this claim Google defines crawl capacity using simultaneous connections and the delay between fetches, adjusted according to site responses. Scope: Google crawler capacity, not a ranking factor. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide

The whole point of the pacing is politeness. A crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. could easily overwhelm a small server if it asked for hundreds of pages a second, so it watches how your site responds and backs off when the server starts to struggle. Answer quickly and it’ll fetch a bit faster; slow down or throw errors and it crawls less.

The big thing that changed

For years the advice was “go into Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. and turn down the crawl rate slider.” That slider is gone — Google removed it on January 8, 2024. So if you find a tutorial telling you to adjust it, that tutorial is out of date. Evidence for this claim Google deprecated the Search Console crawl-rate limiter and removed it on January 8, 2024. Scope: Google Search Console's legacy crawl-rate limiter. Confidence: high · Verified: 2026-07-14 Google: Crawl rate limiter deprecation

Today the rate is automatic. You don’t dial it; Google reads your server’s responses and decides.

How to slow Googlebot down

If your server really is being overloaded, in order of preference:

1. Make the server faster or give it more resources. This is the real fix. GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. will naturally crawl a healthy, fast server more comfortably.
2. For a true emergency only: have your server return a 500, 503, or 429 error instead of the normal page. Google reads that as “slow down” almost immediately. But do this for a day or two at most — leave it on longer and Google may start dropping your pages from search.

What not to do: don’t block Googlebot with 403/404 errors (it doesn’t slow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and risks losing pages), and don’t rely on crawl-delay in your robots.txt — Google ignores it (Bing does honor it).

Can you make Google crawl faster?

Not on demand — there’s no “crawl more” button, and you can’t request an increase. You can encourage it indirectly: a faster server, a clean sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., good internal links, and getting rid of duplicate or junk URLs all help. But more crawling is not a goal in itself — it won’t make you rank higher.

Want the status-code details, the deprecation history, and how Bing differs? Switch to the Advanced tab.

TL;DR — Crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. is the supply side of crawl budgetThe number of URLs an engine will crawl in a timeframe. — how fast a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. fetches, set by what Google calls the crawl capacity limit (parallel connections

• delay between fetches). It’s automatic and reacts to server health; the manual GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. slider was removed Jan 8, 2024. To slow GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. now you signal via HTTP: 500/503/429 for 1–2 days max, never 401/403/404, and crawl-delay is ignored by Google (honored by Bing). You can’t request an increase — improve it indirectly. Bing still has a manual Crawl Control grid. Crawl rate is not a ranking factor, and most sites never need to touch it.

What crawl rate actually is

Crawl rate is how fast a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. fetches pages from your server — the number of simultaneous requests it makes and the delay between them. Google’s name for it is the crawl capacity limit: “the maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches.” Evidence for this claim Google defines crawl capacity using simultaneous connections and the delay between fetches, adjusted according to site responses. Scope: Google crawler capacity, not a ranking factor. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide

It’s one half of crawl budgetThe number of URLs an engine will crawl in a timeframe.. As I put it in my Ahrefs crawl-budget guide, crawl budget breaks into “crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. which is how many pages a search engine wants to crawl on your site and crawl rate which is how fast they can crawl.” Crawl rate is the supply side (how fast you’ll let them); crawl demand is the demand side (how much they want to). Crawl budget is where the two meet — and rankings sit outside that loop entirely.

In my How Search Works deck I frame the crawl rate limit as simply what your site can support — it’s driven by your server’s stability and crawl health, slow responses, 5xx (server) errors, and 429 (too many requests429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content.) responses. Google backs off when your server starts to struggle because it doesn’t want to crash your site. One detail people miss: all the GooglebotsGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. share a single crawl pool — the bots for search, images, ads, and so on all draw from the same rate. So a runaway crawl of one resource type eats into the crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. of everything else.

What sets the crawl rate

The capacity limit is automatic and reacts to your server in real time. Google: “If the site responds quickly for a while, the limit goes up, meaning more connections can be used to crawl. If the site slows down or responds with server errors, the limit goes down and Google crawls less.”

There’s a second lever you don’t control: Google’s own resources. “Google has a lot of machines, but not infinite machines. We still need to make choices with the resources that we have.” So your server health sets the ceiling Google is willing to use, but Google’s own capacity and your site’s crawl demand decide how much of that ceiling actually gets used.

The myths doc confirms the server-health link works both ways: “A speedy site is a sign of healthy servers, so it can get more content over the same number of connections,” while “a significant number of 5xx HTTP response status codes (server errors) or connection timeouts signal the opposite, and crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. slows down.”

Does crawl rate affect rankings? No.

Bust this first, because it drives a lot of misguided effort. Crawling is required to appear in search, but it is not a ranking signal. Google is explicit: “Improving your crawl rate won’t necessarily lead to better positions in Google Search results.” Faster or more crawling buys you fresher discovery and indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., not higher positions. Crawl rate is an efficiency and server-health concern — full stop.

How to reduce Googlebot’s crawl rate

In order, from the real fix to the emergency lever:

1. Fix the server (the real fix). Speed it up or add resources. Because the capacity limit tracks response time and errors, a healthier server is the durable way to keep crawling in a comfortable range — and it’s the one that doesn’t risk your indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..

2. The emergency lever — 500/503/429. Google: “return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests.” Its crawlers “treat the 429 status code as a signal that the server is overloaded,” and “5xx and 429 server errors prompt Google’s crawlers to temporarily slow down with crawling.” Gary Illyes put the speed of this in concrete terms: “if the server persistently returns HTTP 500A 500 Internal Server Error is a generic HTTP status code — RFC 9110 defines it as an unexpected condition that stopped the server from fulfilling the request, and nothing more. The request may have been fine — something broke server-side while generating the response. For SEO, an isolated 500 is typically retried, but persistent, site-wide 500s get Google's documented response: slower crawling and, if the errors don't clear, eventual removal from the index. status codes for a range of URLs, Googlebot will automatically, and almost immediately slow down crawling.” Prefer 429 when you can — it explicitly means “too many requests” and can carry a Retry-After header.

The catch is that this is strictly temporary. Google: “We don’t recommend that you do this for a long period of time (meaning, longer than 1-2 days).” Leave it on and the cost is real — if those codes persist on the same URL for multiple days, “the URL may be dropped from Google’s index” and, for Google Ads, “your campaigns may be cancelled or paused, and your ads may not serve.”

3. What NOT to do. Don’t use 4xx to throttle. “The 4xx status codes, except 429, have no effect on crawl rate,” and Google says plainly “Don’t use 401 and 403 status codes for limiting the crawl rate.” In a dedicated 2023 post about people doing exactly this, Google wrote: “Over the last few months we noticed an uptick in website owners and some content delivery networks (CDNs) attempting to use 404 and other 4xx client errors (but not 429) to attempt to reduce Googlebot’s crawl rate. The short version of this blog post is: please don’t do that…” And don’t reach for robots.txt crawl-delay: “The non-standard ‘crawl-delay’ robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. rule is not processed by Google’s crawlers.”

4. The non-emergency request. For an ongoing problem that isn’t a fire, you can “file a special request to report a problem with unusually high crawl rate, mentioning the optimal rate for your site.” It’s slow, and it only goes one direction (see below).

Can you increase crawl rate? No — not directly.

There is no manual increase. Google: “You cannot request an increase in crawl rate, and it may take several days for the request to be evaluated and fulfilled.” What you can do is improve it indirectly. From my crawl-budget guide, the levers that actually move crawl efficiency: speed up your server / add resources; keep important pages in clean sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing.; kill duplicate contentThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling.; earn more links (external and internal); fix redirected links; use GET instead of POST where possible; and use the Indexing API where you’re eligible. Note that “a speedy site… can get more content over the same number of connections” — so server health is the lever that shows up on both the reduce and the increase side.

How to control Bingbot’s crawl rate

Here’s the sharp cross-engine contrast. Where Google retired manual control, Bing kept it. Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. has Crawl Control (under Configuration): an hour-by-hour grid where each hour shows bricks representing crawl speed — more bricks means faster crawling, fewer means slower. You can pick a preset built around your peak business hours, or choose “Custom” and draw your own pattern across the day. Bing also still honors crawl-delay in robots.txt — the exact directive Google ignores. So the two-engine rule of thumb: for Google you signal through server responses; for Bing you have an actual dial.

What happened to the Search Console crawl rate tool?

The timeline, because so much advice still references a tool that no longer exists:

• Dec 2008 — Google introduces user crawl-rate control in Webmaster Tools.
• Feb 2023 — Google publishes “don’t use 403s or 404s for rate limiting.”
• Nov 24, 2023 — Google announces the deprecation of the Crawl Rate Limiter Tool. Illyes’ reasoning: “with the improvements we’ve made to our crawling logic and other tools available to publishers, its usefulness has dissipated.” The old tool was slow and barely used — it had “a much slower effect” and “would have taken over a day for the new limits to be applied on crawling,” and was used “rarely,” with those who did “in many cases set the crawling speed to the bare minimum.”
• Jan 8, 2024 — the tool is removed. Evidence for this claim Google deprecated the Search Console crawl-rate limiter and removed it on January 8, 2024. Scope: Google Search Console's legacy crawl-rate limiter. Confidence: high · Verified: 2026-07-14 Google: Crawl rate limiter deprecation Google also lowered the floor: “With the deprecation of the crawl limiter tool, we’re also setting the minimum crawling speed to a lower rate, comparable to the old crawl rate limits.”

The practical upshot: the old manual slider had a >24-hour lag anyway; today’s server-signal approach (5xx/429) slows Googlebot almost immediately, which is strictly better for an actual emergency.

How to monitor crawl rate

The GSC Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). is your window into what Google is actually doing: total crawl requests over time, total download size, average response time, and a host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. view of your site’s availability to Google over the last ~90 days, plus a breakdown by response code, file type, crawl purpose, and Googlebot type. Watch average response time and the host-status signals — a climbing response time or a rash of 5xx is exactly what makes Googlebot throttle you, so this report is where you’ll see a self-inflicted slowdown before you go hunting for one. On Bing, Crawl Control and the crawl information in Bing Webmaster Tools are the equivalents.

Crawl rate vs crawl budget vs crawl frequency

Keep these straight:

• Crawl rate = how fast (supply / capacity).
• Crawl demand = how much they want to (popularity + staleness).
• Crawl budget = the interaction of the two — “the amount of time and resources a search engine allows for crawling a website.”
• Crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. = how often a given page gets re-crawled, which is mostly a crawl-demand question (popularity and how fresh/stale the page is).

And the reassuring part, which is my standing take: most sites don’t need to worry about any of this. “Most sites don’t need to worry about crawl budget, but there are few cases where you may want to take a look” — newer sites with a lot of pages, very large or rapidly changing sites, and sites with a pile of “Discovered – currently not indexed” URLs in GSC. If that’s not you, leave crawl rate alone and let Google’s automation do its job.

AI summary

A condensed take on the Advanced version:

• Crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. = how fast a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. fetches pages — the “crawl capacityThe number of URLs an engine will crawl in a timeframe. limit” (parallel connections + delay between fetches). It’s the supply side of crawl budget; crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. is the demand side.
• It’s automatic and reacts to server health: fast/healthy server → higher limit; slow responses or 5xx/429 → Google crawls less. Google’s own capacity is a second ceiling you don’t control. All GooglebotsGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. share one crawl pool.
• The GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. manual rate slider was removed Jan 8, 2024. Advice telling you to “adjust the slider” is outdated.
• To slow GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. today: fix the server (best), or return 500/503/429 for 1–2 days max (longer risks de-indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. + Ads pausing). Prefer 429 (carries Retry-After). Never use 401/403/404, and crawl-delay is ignored by Google.
• You can’t request an increase. Improve it indirectly: faster server, clean sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., fewer duplicate/junk URLs, more internal + external links, Indexing APIThe Google Indexing API lets site owners notify Google directly when a URL is added, updated, or removed. Google officially supports it only for pages with JobPosting or BroadcastEvent (livestream) structured data — not for general content. where eligible.
• Bing kept manual control: Crawl Control hourly grid + crawl-delay honored.
• Crawl rate is not a ranking factor — “improving your crawl rate won’t necessarily lead to better positions.” Most sites never need to manage it.

Official documentation

Primary-source documentation from the search engines.

Google

• Reduce the Googlebot crawl rate — the headline how-to: 500/503/429, the 1–2 day limit, and the request form (you can lower, not raise).
• Crawl Budget Management — defines the crawl capacityThe number of URLs an engine will crawl in a timeframe. limit and how it reacts to server health.
• How HTTP Status Codes Affect Google’s Crawlers — exactly which codes throttle crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. (5xx/429) and which don’t (4xx except 429).
• Myths and facts about crawling — crawl-delay not processed; crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. ≠ rankings; server health affects crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..
• Upcoming deprecation of Crawl Rate Limiter Tool (Nov 2023) — the deprecation announcement.
• Don’t use 403s or 404s for rate limiting (Feb 2023) — why 4xx is the wrong lever.
• New and improved crawl stats for your site (Nov 2020) — how to read the Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root)..
• Optimize your crawl budget — capacity + demand, and who actually needs it.

Bing / Microsoft

• Bing Webmaster Tools — Crawl Control — the manual hour-by-hour crawl-rate grid Google retired.
• Bingbot guidance — current Bing WebmasterMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. guidance documents crawl-delay values from 1–20 seconds.
• bingbot Series: Optimizing Crawl Frequency (Oct 2018) — Bing’s take on when to re-fetch.

Quotes from the source

On-the-record statements from Google. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. is and what sets it

• “If the site responds quickly for a while, the limit goes up, meaning more connections can be used to crawl. If the site slows down or responds with server errors, the limit goes down and Google crawls less.” — Crawl BudgetThe number of URLs an engine will crawl in a timeframe. Management. Jump to quote
• “Google has a lot of machines, but not infinite machines. We still need to make choices with the resources that we have.” Jump to quote

Google — how to reduce the crawl rate

• “return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests.” — Reduce the GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. crawl rate. Jump to quote
• “We don’t recommend that you do this for a long period of time (meaning, longer than 1-2 days).” Jump to quote
• “You cannot request an increase in crawl rate, and it may take several days for the request to be evaluated and fulfilled.” Jump to quote

Google — which status codes throttle crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.

• “Google’s crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. treat the 429 status code as a signal that the server is overloaded, and it’s considered a server error.” — How HTTP Status CodesAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. Affect Google’s CrawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. Jump to quote
• “The 4xx status codes, except 429, have no effect on crawl rate.” / “Don’t use 401 and 403 status codes for limiting the crawl rate.” Jump to quote

Google — crawl-delayCrawl-delay is a non-standard robots.txt directive that asks bots to wait between successive page fetches to ease server load. Google has ignored it since September 1, 2019 and Yandex dropped it on February 22, 2018; Bing and many other crawlers still honor it, but Bing interprets the value differently than the plain reading. and rankings

• “The non-standard ‘crawl-delay’ robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. rule is not processed by Google’s crawlers.” — Myths and facts about crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. Jump to quote
• “Improving your crawl rate won’t necessarily lead to better positions in Google Search results.” Jump to quote

Gary Illyes, Google (on the crawl-rate tool deprecation)

• “if the server persistently returns HTTP 500A 500 Internal Server Error is a generic HTTP status code — RFC 9110 defines it as an unexpected condition that stopped the server from fulfilling the request, and nothing more. The request may have been fine — something broke server-side while generating the response. For SEO, an isolated 500 is typically retried, but persistent, site-wide 500s get Google's documented response: slower crawling and, if the errors don't clear, eventual removal from the index. status codes for a range of URLs, GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. will automatically, and almost immediately slow down crawling.” Read the coverage
• “with the improvements we’ve made to our crawling logic and other tools available to publishers, its usefulness has dissipated.” Read the coverage

Crawl-rate checklists

Safely reduce GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.’s crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. (in order)

• First, confirm the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. is the real problem — check GSC Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). (average response time, host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity.) and your server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened., not just a hunch.
• Fix the root cause: speed up the server or add resources. This is the durable fix and the only one that doesn’t risk indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..
• If it’s a genuine emergency, return 500/503/429 to crawl requests (prefer 429 with a Retry-After). GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. slows almost immediately.
• Keep the emergency response on for 1–2 days at most — longer risks pages being dropped from the index and Ads being paused.
• For an ongoing (non-fire) issue, file Google’s special crawl-rate request, naming the optimal rate for your site.
• For Bing, set the speed in Crawl Control and/or add crawl-delay to robots.txt.

Do NOT do these

• Don’t use 401/403/404 to throttle — no effect on rate, and you risk losing pages.
• Don’t rely on robots.txt crawl-delay for Google — it’s ignored (Bing honors it).
• Don’t look for the old GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. crawl-rate slider — it was removed Jan 8, 2024.

When you want more crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. (you can’t force it — improve it indirectly)

• Speed up / add resources to the server.
• Keep canonical, indexable URLs in clean sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. with accurate lastmod.
• Remove duplicate and low-value URLs that waste crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..
• Strengthen internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. and earn more external links.
• Use GET instead of POST where possible; use the IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. API where eligible.

Crawl rate — cheat sheet

Status codes and what they do to Google’s crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.

Fast facts

• Crawl rate = how fast (supply); crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. = how much they want (demand); crawl budgetThe number of URLs an engine will crawl in a timeframe. = the two together.
• Google’s term: crawl capacity limit — parallel connections + delay between fetches. Automatic, tracks server health.
• Emergency 5xx/429 window: 1–2 days max — longer risks de-indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. + Ads pausing.
• No manual increase — you can only ask Google to lower, never raise.
• Manual GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. crawl-rate slider: removed Jan 8, 2024 (announced Nov 24, 2023).
• All GooglebotsGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. share one crawl pool (search, images, ads, etc.).
• Crawl rate is not a ranking factor.
• Bing equivalent: Crawl Control grid + crawl-delay honored.

Temporarily tell Googlebot to slow down

This is the emergency lever only — return a 503 (or 429) with a Retry-After header to the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. instead of a 200. Google reads it as “slow down” almost immediately. Keep it on for a day or two at most: leave it longer and the affected URLs may be dropped from the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. (and any Google Ads pointing at them may be paused). The right long-term fix is a faster, healthier server, not a permanent error response.

Apache (.htaccess) — return 503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. with Retry-After

# Emergency only — remove within 1–2 days.
# Sends Googlebot a "slow down / try later" signal.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Googlebot|bingbot) [NC]
RewriteRule ^ - [R=503,L]
Header always set Retry-After "3600"
ErrorDocument 503 "Server temporarily overloaded — please retry later."

Nginx — return 503 with Retry-After for crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.

# Emergency only — remove within 1–2 days.
if ($http_user_agent ~* (Googlebot|bingbot)) {
    return 503;
}
# Send a Retry-After hint with the 503 response.
add_header Retry-After 3600 always;

Express / Node.js — 429 Too Many Requests429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. with Retry-After

// Emergency only — remove within 1–2 days.
// 429 explicitly means "too many requests" and carries a Retry-After.
app.use((req, res, next) => {
  const ua = req.get("user-agent") || "";
  if (/Googlebot|bingbot/i.test(ua)) {
    res.set("Retry-After", "3600"); // seconds
    return res.status(429).send("Too many requests — please retry later.");
  }
  next();
});

Reminder: this only slows crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. temporarily. It is not a way to pause or block indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and it is not a substitute for fixing server capacity.

Tools for seeing and controlling crawl rate

• Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. — Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). — Google’s own view: total crawl requests over time, total download size, average response time, host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. (site availability to Google over ~90 days), and breakdowns by response code, file type, crawl purpose, and GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. type. This is where you’ll see throttling.
• Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. — Crawl Control — the manual hour-by-hour grid (presets or custom-drawn) for setting BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.’s speed by time of day. The manual lever Google retired.
• robots.txt crawl-delay — honored by Bing (and some other engines), ignored by Google. Useful for Bing, useless for Google.
• Server log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. — the ground truth for how fast botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. are actually hitting you and what status codes they’re getting. (See log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened..)
• Site audits / crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — Ahrefs Site Audit and Screaming Frog SEO Spider to find the duplicate, parameterized, and trap-like URLs that waste crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..

What should you do about crawl rate?

Incident playbook: crawler traffic is overloading the origin

1. Verify the botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. Confirm the source IP belongs to the claimed crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. If not, block or rate-limit the impersonator through normal security controls.
2. Measure impact. Correlate crawler requests with latency, saturation, timeouts, and 5xx responses. If they do not align, investigate the actual load source.
3. Protect availability. Shed only the necessary traffic; for a temporary GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. emergency, use 429 or 503, not 403 or 404.
4. Find the hot patterns. Group requests by directory, parameters, response code, and bytes. If a runaway URL space dominates, fix its links or generation rules.
5. Repair the cause. Increase capacity, cache safe responses, remove crawler trapsA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content., or use the specific crawler’s supported controls.
6. Restore and verify. Remove temporary throttling, then confirm user latency and crawler error rates return to the site’s baseline.

Crawl-rate mistakes

• Using crawl-delay for GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.. Google ignores it. Use temporary 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content./503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. responses only for an emergency and fix the underlying load.
• Returning 403 or 404 to slow GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.. Those statuses communicate access or absence, not temporary overload. Use the correct temporary signal.
• Trying to force a permanent increase. The retired Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. slider is gone, and more crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. does not improve rankings. Improve server health and demand signals.
• Trusting the user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string. Spoofers can claim to be Googlebot. Verify the IP before changing site behavior.
• Leaving emergency throttling in place. Long-running errors can harm crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Set an owner and removal condition before deployment.

The verify → protect → repair framework

• Verify: prove the traffic is a real crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. and that it correlates with server harm.
• Protect: use the narrowest temporary control that preserves user availability and communicates the right HTTP meaning.
• Repair: remove the capacity bottleneck or runaway URL space, then withdraw the temporary control.

Keep crawl rate separate from crawl demand: a healthy server can raise the capacity ceiling, but it cannot force a search engine to want more URLs.

Prove a crawl-rate intervention worked

Temporary throttle response

Test to run: request a throttled test URL with curl -I. Expected result: the planned 429 or 503 appears only during the incident and normal URLs remain available. Failure interpretation: the rule is mis-scoped or the wrong status is being returned. Monitoring window: immediate. Rollback trigger: users or unrelated botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. receive the throttle unexpectedly.

Recovery after removing the throttle

Test to run: repeat header checks and monitor server plus access logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened.. Expected result: normal 200 responses return, crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. errors decline, and user latency stays at baseline. Failure interpretation: the temporary rule remains active or the capacity problem persists. Monitoring window: immediate for HTTP behavior; continue through the next normal crawl window. Rollback trigger: renewed saturation or 5xx errorsThe 4xx (client error) and 5xx (server error) HTTP status codes a server returns instead of a successful 2xx — and how Google treats each class very differently. require returning to the incident plan.

URL-space repair

Test to run: crawl and log-check the parameter or path pattern that caused excess requests. Expected result: new trap URLs are no longer generated or linked, while valuable URLs remain reachable. Failure interpretation: another discovery path still exposes the pattern. Monitoring window: compare equivalent log windows. Rollback trigger: valuable pages or required resources become inaccessible.

Metrics for crawl-rate health

Verified crawler request rate

Metric: requests per time unit from verified crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. IPs. What it tells you: actual crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. pace. How to pull it: access logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. after bot verification. Benchmark / realistic range: establish a baseline by crawler and traffic period; no universal safe rate exists. Cadence: daily during incidents, monthly otherwise.

Crawler-correlated error and latency rate

Metric: 5xx/timeouts and origin latency during crawler activity. What it tells you: whether rate exceeds capacity. How to pull it: align server telemetry with crawler log timestamps. Benchmark / realistic range: use the site’s normal non-incident baseline and capacity objectives. Cadence: continuous alerting for critical sites.

Useful-request share

Metric: verified crawler requests to valuable 200 pages versus redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., errors, and known trap URLs. What it tells you: whether capacity is spent productively. How to pull it: classify log URLs and statuses. Benchmark / realistic range: trend against the site’s own inventory; avoid a universal target. Cadence: monthly.

Test yourself: Crawl rate

Resources worth your time

My related writing

• When Should You Worry About Crawl Budget? — crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. as one half of crawl budgetThe number of URLs an engine will crawl in a timeframe., plus the reduce/increase levers.
• What Is Googlebot & How Does It Work? — how GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. decides how fast and what to crawl (and why “change crawl rate” is now deprecated).
• The Beginner’s Guide to Technical SEO — where crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and crawl rate fit in the bigger picture.

My speaking

• How Search Works (SlideShare) — where I frame the crawl rate limit as “what your site can support” and note all GooglebotsGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. share one crawl pool. (Standing disclaimer: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• Search Engine Land — Googlebot crawl rate tool is going away and the tool is now gone — the deprecation, with Illyes’ reasoning.
• Google’s Crawling December series — the best concentrated set of official crawl explainers.
• Search Engine Journal — Google Removing Crawl Rate Limiter Tool From Search Console — Roger Montti’s relay of the Nov-2023 deprecation with the new auto-set minimum crawl speed detail.
• Search Engine Journal — Don’t Use 403/400 Error Responses For Rate Limiting Googlebot — covers Google’s Feb-2023 blog post warning against using 4xx to throttle crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..
• Bing WebmasterMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. Blog — bingbot Series: Optimizing Crawl Frequency — Bing’s own explanation of when and how BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. decides to re-fetch; useful counterpoint to Google’s automatic approach.
• Bing Webmaster Blog — Getting most of Bingbot via Bing Webmaster Tools — how to use Crawl Control and other BWT settings to manage BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.’s crawl rate.

Crawl rate

Crawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.

Related: Crawl Budget, Crawling

Crawl rate

Crawl rate is how fast a search engine crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. requests pages from your server — the number of simultaneous parallel connections it opens and the time it waits between fetches. Google calls its version the crawl capacityThe number of URLs an engine will crawl in a timeframe. limit, and it’s one of the two halves of crawl budgetThe number of URLs an engine will crawl in a timeframe.; the other half is crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. (how much an engine actually wants to crawl your pages).

You don’t set crawl rate by hand anymore. GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. raises or lowers it automatically based on how your server responds: if the site answers quickly, the limit goes up; if it slows down or returns server errors, Google crawls less. Google retired its manual crawl-rate slider in Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. (removed January 8, 2024), so today the only reliable way to signal “slow down” is your server’s HTTP responses — returning 500, 503, or 429 for short emergencies. You can’t request a permanent increase. Bing still offers a manual Crawl Control grid in Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility..

Crawl rate is distinct from both crawl budget and rankings. A higher crawl rate doesn’t lift your positions — crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is required to appear in search, but it isn’t a ranking signal. For most sites, crawl rate is a server-health and efficiency question, not something to manage.

Related: Crawl Budget, Crawling