Crawl Delay

TL;DR — Crawl-delay is a line you can put in robots.txt asking botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. to slow down — wait a few seconds between page fetches. The big thing to know: Google ignores it (and has since 2019). Bing still honors it. If you’re trying to stop GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. from hammering your server, crawl-delay won’t help — use a different approach (below).

What crawl-delay is

Crawl-delay is a line you add to your robots.txt file that asks a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. to wait a certain number of seconds between requests, so it doesn’t fetch your pages so fast that it slows your server down. It looks like this:

User-agent: *
Crawl-delay: 10

That’s saying, roughly, “every bot, please pause between page fetches.”

The most important thing: Google ignores it

This is the part that trips people up. Google does not honor crawl-delay. It stopped processing the directive on September 1, 2019, and even before that it never officially supported it. Evidence for this claim Google does not support or process the non-standard crawl-delay robots.txt field. Scope: Google crawlers; other crawlers may support the field. Confidence: high · Verified: 2026-07-14 Google: robots.txt specifications So if you’ve put crawl-delay in your robots.txt hoping to slow GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. down, it’s doing nothing.

Bing is different — Bing does honor crawl-delay. So do some other crawlers (more on that in the Advanced tab).

If you’re trying to slow Google down

Don’t reach for crawl-delay. Instead:

• Best long-term: fix or upgrade your server so it can handle the crawl.
• Emergency only: have your server return a 503 or 429 status code to Googlebot. Google reads those as “slow down” almost immediately. Only do this for a day or two — longer than that and Google can start dropping pages. Evidence for this claim Google recommends temporarily returning 500, 503, or 429 to reduce crawl rate and warns against doing so for longer than one or two days. Scope: Temporary Googlebot overload response, not routine crawl management. Confidence: high · Verified: 2026-07-14 Google: Reduce Googlebot crawl rate
• Never use 403 or 404 to try to throttle — those don’t affect crawl speed.

If you’re trying to slow Bing down

Bing does respect crawl-delay, but Bing’s own preferred tool is Crawl Control inside Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. — a grid where you set crawl speed by time of day. Use that first; it’s more precise.

Want the history, why Yandex dropped it in 2018, and which AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. respect it? Switch to the Advanced tab.

TL;DR — Crawl-delay was an unofficial robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. directive for throttling botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.; it was never in the formal standard (RFC 9309). Google retired it September 1, 2019 and explicitly doesn’t process it — slow GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. with 429/503 (1-2 days max) or by fixing the server, not crawl-delay. Bing’s current guidance documents values from 1–20 seconds. Yandex dropped support for it on February 22, 2018 — it’s not honored there either; Yandex now uses a crawl-rate setting inside Yandex WebmasterYandex Webmaster (webmaster.yandex.com) is Yandex's free search console — the Yandex equivalent of Google Search Console — for verifying a site, monitoring indexing and queries, requesting reindexing, running diagnostics, and setting a site region. It's essential for any site targeting Russia and the wider CIS, where Yandex, not Google, is the dominant search engine.. Many SEO crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. (AhrefsBot, Semrush) and some AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. (ClaudeBot) respect it too — which is the one real use case left: controlling non-Google, non-Bing, non-Yandex botsYandexBot is the main web crawler operated by Yandex, Russia's dominant search engine. It discovers and fetches pages for Yandex Search, is one of a family of Yandex robots, and supports a Yandex-only robots.txt directive (Clean-param) that Google and Bing don't have..

Why crawl-delay exists

Crawl-delay is a throttle. The idea is straightforward: an aggressive crawler fetching pages back-to-back can put real load on a server, especially a small or slow one. Crawl-delay was the courtesy lever — a way to ask a bot to pause a few seconds between requests so your server could keep up. It lives in robots.txt inside a User-agent group, the same place your disallow and allow lines go. (For the full file, see robots-txt.)

The catch is that it was never standardized. The Robots Exclusion Protocol was finally formalized as RFC 9309 in 2022, and crawl-delay isn’t in it. It was always an unofficial extension that different crawlers chose to implement — or not — and interpreted in their own way. That inconsistency is exactly why Google walked away from it.

Google retired it on September 1, 2019

On July 2, 2019, Gary Illyes published A note on unsupported rules in robots.txt on Google Search Central. Google was open-sourcing its robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. parser and, as part of that, retiring all code that handled rules that were never part of the internet draft — specifically noindex, nofollow, and crawl-delay. The retirement took effect September 1, 2019.

Google’s reasoning: these rules were undocumented, unofficial, and interpreted inconsistently across crawlers, which created ambiguity. The current robots.txt documentation is now explicit — Google supports four fields (user-agent, allow, disallow, and sitemap), and the docs say “other fields such as crawl-delay aren’t supported.” Google’s Myths and facts about crawling page repeats it: “The non-standard ‘crawl-delay’ robots.txt rule is not processed by Google’s crawlers.” Evidence for this claim Google does not support or process the non-standard crawl-delay robots.txt field. Scope: Google crawlers; other crawlers may support the field. Confidence: high · Verified: 2026-07-14 Google: robots.txt specifications

Worth being clear about the scope: it’s ignored no matter which user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. group you put it in. A User-agent: Googlebot block with a Crawl-delay line is just as ignored as one under User-agent: *.

What to use instead for Google

Google’s crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. is fully automatic now, tuned to your server’s health — and the old manual crawl-rate slider in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. was removed on January 8, 2024 (announced the prior November). So when you genuinely need GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. to back off, you have three levers:

1. Fix the server. The real fix. If your server can take the load, you don’t need to throttle anything.
2. Return 429, 500, or 503 for emergencies. Google’s Reduce the Googlebot crawl rate docs say to “return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests.” Googlebot reads these as “slow down” almost immediately. The hard rule: “We don’t recommend that you do this for a long period of time (meaning, longer than 1-2 days).” Sustained 5xx risks pages getting dropped (and Google Ads pausing). Evidence for this claim Google recommends temporarily returning 500, 503, or 429 to reduce crawl rate and warns against doing so for longer than one or two days. Scope: Temporary Googlebot overload response, not routine crawl management. Confidence: high · Verified: 2026-07-14 Google: Reduce Googlebot crawl rate
3. File a crawl-rate reduction request through Search Console for a persistent problem. It’s slow, and it only reduces, never raises.

One trap to avoid: don’t use 401, 403, or 404 to throttle. Per Google’s HTTP status codes guide, “the 4xx status codes, except 429, have no effect on crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.,” and you should not “use 401 and 403 status codes for limiting the crawl rate.”

This is also why, in my crawl-rate guideCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor., I treat crawl-delay as the thing not to use for Google — see that article for the full speed-up / slow-down playbook.

Bing documents a 1–20 second range

Current Bing Webmaster guidance documents crawl-delay values from 1–20 seconds. That does not make the field a portable standard: Google ignores it, and every other crawler needs its own documented confirmation.

Yandex dropped it in 2018

Yandex used to honor crawl-delay as a literal minimum number of seconds between requests — that’s the version of the fact you’ll still see repeated across most SEO blogs. It’s stale. Yandex’s own current documentation is explicit: “From February 22, 2018, Yandex doesn’t take into account the Crawl-delay directive.” Evidence for this claim Yandex stopped honoring the Crawl-delay directive on February 22, 2018, and now recommends setting crawl rate inside Yandex Webmaster instead. Scope: Yandex's crawler only; does not apply to Google, Bing, or other crawlers. Confidence: high · Verified: 2026-07-18 Yandex Webmaster: The Crawl-delay directive To set how fast Yandex’s bots crawl your site now, you use the crawl rate setting inside Yandex Webmaster directly — not robots.txt. So as of today, the honest scorecard is: Bing honors crawl-delay within its documented range, Google never did, and Yandex used to but stopped in 2018.

Other crawlers: SEO tools and AI bots

Plenty of well-behaved non-search-engine crawlers honor crawl-delay as a courtesy:

• AhrefsBot and Semrush’s bot both respect it.
• AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. are a newer audience. Anthropic’s ClaudeBot is documented as supporting the non-standard crawl-delay directive. Others (GPTBot, Perplexity) should be checked against their own published docs rather than assumed.

The pattern: smaller, compliance-focused, or courtesy-minded crawlers tend to honor crawl-delay; the major search engines have largely moved to internal signals (Google) or richer manual tools (Bing). And none of this binds actual bad actors — scrapers that ignore robots.txt entirely will ignore your crawl-delay too.

The use case that’s left

So is crawl-delay dead? Not quite. It’s useless for Google, redundant-at-best for Bing (use Crawl Control), but it’s genuinely the simplest way to ask the broad middle — SEO tools, AI bots, and other minor crawlers that respect it — to ease off. If a specific tool’s bot is overrunning your server, a targeted User-agent: group with a crawl-delay is a reasonable first move:

User-agent: SomeBot
Crawl-delay: 10

Just remember it’s a request, not a guarantee, and it scopes per host and per user-agent group like everything else in robots.txt. For the broader efficiency picture — what bots actually cost you and who needs to care — see crawl-budget.

AI summary

A condensed take on the Advanced version:

• What it is: Crawl-delay is a non-standard robots.txt directive asking botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. to pause a set number of seconds between fetches to reduce server load. It was never in the formal standard (RFC 9309, 2022).
• Google ignores it. Retired September 1, 2019 (Gary Illyes’ 2019 post); current docs say “other fields such as crawl-delay aren’t supported.” Ignored regardless of which user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. group it’s in.
• Slow Google instead with: fixing the server (best), or returning 429/500/503 for emergencies only (1-2 days max — longer risks de-indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.). Never use 401/403/404 to throttle. The GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. manual crawl-rate slider was removed January 8, 2024.
• Bing documents 1–20 second values. Treat that as Bing-specific guidance, not a portable robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. standard.
• Yandex dropped support for it on February 22, 2018. Older advice that it reads the value as literal seconds is stale — Yandex now uses a crawl-rate setting inside Yandex WebmasterYandex Webmaster (webmaster.yandex.com) is Yandex's free search console — the Yandex equivalent of Google Search Console — for verifying a site, monitoring indexing and queries, requesting reindexing, running diagnostics, and setting a site region. It's essential for any site targeting Russia and the wider CIS, where Yandex, not Google, is the dominant search engine. instead.
• Other crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.: AhrefsBot and Semrush respect it; Anthropic’s ClaudeBot is documented as supporting it. Bad-actor scrapers ignore robots.txt entirely.
• Remaining use case: controlling non-Google, non-Bing, non-Yandex crawlersYandexBot is the main web crawler operated by Yandex, Russia's dominant search engine. It discovers and fetches pages for Yandex Search, is one of a family of Yandex robots, and supports a Yandex-only robots.txt directive (Clean-param) that Google and Bing don't have. (SEO tools, AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls.) via a targeted User-agent: group.

Official documentation

Primary-source documentation on crawl-delay and its alternatives.

Google

• robots.txt documentation — lists the four supported fields and states crawl-delay isn’t supported.
• A note on unsupported rules in robots.txt (July 2, 2019) — the retirement announcement (effective September 1, 2019) covering noindex, nofollow, and crawl-delay.
• Myths and facts about crawling — “The non-standard ‘crawl-delay’ robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. rule is not processed by Google’s crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index..”
• Reduce the Googlebot crawl rate — what to use instead: 429/500/503, and the 1-2 day limit.
• How HTTP Status Codes Affect Google’s Crawlers — why 4xx (except 429) don’t throttle.
• Upcoming deprecation of the Crawl Rate Limiter Tool (Nov 2023) — the GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. slider’s removal.

Bing / Microsoft

• Bingbot guidance — current Bing WebmasterMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. guidance documents values from 1–20 seconds.
• Bing Webmaster Tools — Crawl Control — the preferred alternative to crawl-delay for Bing.

Yandex

• Yandex Webmaster — The Crawl-delay directive — states Yandex has not honored the directive since February 22, 2018 and points to the Yandex WebmasterYandex Webmaster (webmaster.yandex.com) is Yandex's free search console — the Yandex equivalent of Google Search Console — for verifying a site, monitoring indexing and queries, requesting reindexing, running diagnostics, and setting a site region. It's essential for any site targeting Russia and the wider CIS, where Yandex, not Google, is the dominant search engine. crawl-rate setting instead.

The standard

• RFC 9309: Robots Exclusion Protocol — the formalized standard, which does not include crawl-delay.

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page where supported.

Google — crawl-delay is not processed

• “The non-standard ‘crawl-delay’ robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. rule is not processed by Google’s crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index..” — Google Search Central docs. Jump to quote
• “Google supports the following fields … (other fields such as crawl-delay aren’t supported).” — robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. documentation. Jump to quote

Google — what to use instead

• “To quickly reduce the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor., you can return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests.” — Reduce the GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.. Jump to quote
• “We don’t recommend that you do this for a long period of time (meaning, longer than 1-2 days).” — Reduce the GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. crawl rate. Jump to quote

Gary Illyes, Google (on the GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. crawl-rate tool deprecation, November 2023)

• “if the server persistently returns HTTP 500A 500 Internal Server Error is a generic HTTP status code — RFC 9110 defines it as an unexpected condition that stopped the server from fulfilling the request, and nothing more. The request may have been fine — something broke server-side while generating the response. For SEO, an isolated 500 is typically retried, but persistent, site-wide 500s get Google's documented response: slower crawling and, if the errors don't clear, eventual removal from the index. status codes for a range of URLs, Googlebot will automatically, and almost immediately slow down crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..” He added that the tool’s “usefulness has dissipated.” Relayed via Search Engine Land’s coverage of the deprecation post — read the coverage; confirm against Google’s original post before treating as final.

Bing / Microsoft

• Current Bingbot guidance documents crawl-delay values from 1–20 seconds. It is Bing-specific guidance, not a general REP rule.

Yandex (current Webmaster documentation)

• “From February 22, 2018, Yandex doesn’t take into account the Crawl-delay directive. To set the rate at which the indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. bots should access your site’s pages, use the site crawl rate setting in Yandex WebmasterYandex Webmaster (webmaster.yandex.com) is Yandex's free search console — the Yandex equivalent of Google Search Console — for verifying a site, monitoring indexing and queries, requesting reindexing, running diagnostics, and setting a site region. It's essential for any site targeting Russia and the wider CIS, where Yandex, not Google, is the dominant search engine..” — Yandex Webmaster: The Crawl-delay directive

Crawl-delay decision checklist

Work down the list based on which botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. you’re trying to slow:

• Audit your robots.txt. Is there a crawl-delay line already? Note which User-agent group(s) it’s in and which bots you actually mean to throttle.
• If it’s there for Google: remove it (or document that it’s deliberately targeting other bots). It does nothing for GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer..
• Trying to slow Google? Fix the server first. For a genuine emergency, return 429/503 — for no more than 1-2 days. Never use 401/403/404 to throttle. For a persistent problem, file a crawl-rate request in Search Console.
• Trying to slow Bing? Use Crawl Control in Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. first. If you do use crawl-delay, keep the value at 10 or below, and remember it applies from the * group even with a bingbot section present.
• Trying to slow another crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. (SEO tool, AI botAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls., etc.)? Put a crawl-delay in a targeted User-agent: group for that bot — it helps if that crawler honors the directive (many do).
• Confirm the value’s meaning for your target bot: Bing reads it as a relative throttle factor. Yandex used to read it as literal seconds but stopped honoring the directive entirely on February 22, 2018 — use the Yandex WebmasterYandex Webmaster (webmaster.yandex.com) is Yandex's free search console — the Yandex equivalent of Google Search Console — for verifying a site, monitoring indexing and queries, requesting reindexing, running diagnostics, and setting a site region. It's essential for any site targeting Russia and the wider CIS, where Yandex, not Google, is the dominant search engine. crawl-rate setting for Yandex instead.
• Don’t expect scrapers or bad actors to comply — they ignore robots.txt entirely. crawl-delay is for well-behaved bots only.

Crawl-delay cheat sheet

Who honors crawl-delay?

Bing throttle scale

What to use instead, by engine

Don’t

• Don’t use crawl-delay expecting it to affect Google or Yandex.
• Don’t use 401/403/404 to throttle GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. (no effect; only 429 among 4xx does).
• Don’t set Bing values above 10.

How should you slow a crawler?

Tools for crawl-delay work

• robots.txt Tester checks which user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. group and rule apply; it helps catch scoping mistakes even though support for crawl-delay remains crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-specific.
• robots.txt Generator helps build separate bot groups instead of applying a broad delay to every crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index..
• Log File Analyzer shows the request rate actually produced after a change and separates search crawlers from spoofed user agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target..
• Googlebot Verifier verifies whether the IP creating load really belongs to the crawler it claims to be.

Resources worth your time

My related writing

• Robots.txt and SEO: Everything You Need to Know — the full robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. picture, including why crawl-delay isn’t a Google directive.
• The Beginner’s Guide to Technical SEO — where crawl controls sit in the bigger picture.
• Indexed, though blocked by robots.txt — a reminder that robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..

My speaking

• How Search Works (SlideShare) — my walkthrough of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and ranking. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From around the industry

• A note on unsupported rules in robots.txt (Google Search Central) — the original 2019 retirement announcement.
• Bingbot guidance — current Bing WebmasterMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. guidance documents 1–20 second values.
• Google To Drop Support For crawl-delay, nofollow, and noindex in robots.txt (Search Engine Roundtable) — coverage of the 2019 change.
• Google removes support for crawl-delay, nofollow, and noindex rules in robots.txt (Coywolf) — additional context on the retirement.
• Googlebot crawl rate tool in Search Console is going away (Search Engine Land) — Gary Illyes’ quotes on the alternatives being better.
• Crawl-delay: 10 in robots.txt: what does it mean? (Conductor) — a widely-cited explainer on the directive.

Test yourself: Crawl Delay

Five quick questions on the crawl-delay directive. Pick an answer for each, then check.

Crawl Delay

Crawl-delay is a non-standard robots.txt directive that asks bots to wait between successive page fetches to ease server load. Google has ignored it since September 1, 2019 and Yandex dropped it on February 22, 2018; Bing and many other crawlers still honor it, but Bing interprets the value differently than the plain reading.

Related: Crawl rate, Crawl Budget

Crawl Delay

Crawl-delay is an unofficial robots.txt directive that tells a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. to wait a set number of seconds between requests so it doesn’t overwhelm your server. The syntax is simple — a Crawl-delay: line with a positive integer inside a User-agent group:

User-agent: *
Crawl-delay: 10

It was never part of the formal Robots Exclusion ProtocolA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. (RFC 9309), and Google has not processed it since September 1, 2019 — Google’s docs state plainly that “other fields such as crawl-delay aren’t supported.” To slow GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. you return 429/503 responses (for a day or two) or fix server capacity, not crawl-delay.

Bing does honor it, but not as literal seconds — it treats the value as a relative throttle factor and recommends keeping it at 10 or below. Yandex used to treat it as a literal minimum number of seconds between requests, but stopped honoring the directive entirely on February 22, 2018; Yandex’s crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. is now set inside Yandex WebmasterYandex Webmaster (webmaster.yandex.com) is Yandex's free search console — the Yandex equivalent of Google Search Console — for verifying a site, monitoring indexing and queries, requesting reindexing, running diagnostics, and setting a site region. It's essential for any site targeting Russia and the wider CIS, where Yandex, not Google, is the dominant search engine., not robots.txt. Several SEO tools (AhrefsBot, Semrush) and AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. (Anthropic’s ClaudeBot) still respect it. That makes crawl-delay still useful for controlling non-Google, non-Yandex crawlersYandexBot is the main web crawler operated by Yandex, Russia's dominant search engine. It discovers and fetches pages for Yandex Search, is one of a family of Yandex robots, and supports a Yandex-only robots.txt directive (Clean-param) that Google and Bing don't have., even though it does nothing for Google or Yandex.

Related: Crawl rate, Crawl Budget

Changelog