Crawl Delay

TL;DR — Crawl-delay is a line you can put in robots.txt asking bots to slow down — wait a few seconds between page fetches. The big thing to know: Google ignores it (and has since 2019). Bing still honors it. If you’re trying to stop Googlebot from hammering your server, crawl-delay won’t help — use a different approach (below).

What crawl-delay is

Crawl-delay is a line you add to your robots.txt file that asks a crawler to wait a certain number of seconds between requests, so it doesn’t fetch your pages so fast that it slows your server down. It looks like this:

User-agent: *
Crawl-delay: 10

That’s saying, roughly, “every bot, please pause between page fetches.”

The most important thing: Google ignores it

This is the part that trips people up. Google does not honor crawl-delay. It stopped processing the directive on September 1, 2019, and even before that it never officially supported it. Evidence for this claim Google does not support or process the non-standard crawl-delay robots.txt field. Scope: Google crawlers; other crawlers may support the field. Confidence: high · Verified: 2026-07-14 Google: robots.txt specifications So if you’ve put crawl-delay in your robots.txt hoping to slow Googlebot down, it’s doing nothing.

Bing is different — Bing does honor crawl-delay. So do some other crawlers (more on that in the Advanced tab).

If you’re trying to slow Google down

Don’t reach for crawl-delay. Instead:

• Best long-term: fix or upgrade your server so it can handle the crawl.
• Emergency only: have your server return a 503 or 429 status code to Googlebot. Google reads those as “slow down” almost immediately. Only do this for a day or two — longer than that and Google can start dropping pages. Evidence for this claim Google recommends temporarily returning 500, 503, or 429 to reduce crawl rate and warns against doing so for longer than one or two days. Scope: Temporary Googlebot overload response, not routine crawl management. Confidence: high · Verified: 2026-07-14 Google: Reduce Googlebot crawl rate
• Never use 403 or 404 to try to throttle — those don’t affect crawl speed.

If you’re trying to slow Bing down

Bing does respect crawl-delay, but Bing’s own preferred tool is Crawl Control inside Bing Webmaster Tools — a grid where you set crawl speed by time of day. Use that first; it’s more precise.

Want the history, why Yandex dropped it in 2018, and which AI bots respect it? Switch to the Advanced tab.

TL;DR — Crawl-delay was an unofficial robots.txt directive for throttling bots; it was never in the formal standard (RFC 9309). Google retired it September 1, 2019 and explicitly doesn’t process it — slow Googlebot with 429/503 (1-2 days max) or by fixing the server, not crawl-delay. Bing’s current guidance documents values from 1–20 seconds. Yandex dropped support for it on February 22, 2018 — it’s not honored there either; Yandex now uses a crawl-rate setting inside Yandex Webmaster. Many SEO crawlers (AhrefsBot, Semrush) and some AI bots (ClaudeBot) respect it too — which is the one real use case left: controlling non-Google, non-Bing, non-Yandex bots.

Why crawl-delay exists

Crawl-delay is a throttle. The idea is straightforward: an aggressive crawler fetching pages back-to-back can put real load on a server, especially a small or slow one. Crawl-delay was the courtesy lever — a way to ask a bot to pause a few seconds between requests so your server could keep up. It lives in robots.txt inside a User-agent group, the same place your disallow and allow lines go. (For the full file, see robots-txt.)

The catch is that it was never standardized. The Robots Exclusion Protocol was finally formalized as RFC 9309 in 2022, and crawl-delay isn’t in it. It was always an unofficial extension that different crawlers chose to implement — or not — and interpreted in their own way. That inconsistency is exactly why Google walked away from it.

Google retired it on September 1, 2019

On July 2, 2019, Gary Illyes published A note on unsupported rules in robots.txt on Google Search Central. Google was open-sourcing its robots.txt parser and, as part of that, retiring all code that handled rules that were never part of the internet draft — specifically noindex, nofollow, and crawl-delay. The retirement took effect September 1, 2019.

Google’s reasoning: these rules were undocumented, unofficial, and interpreted inconsistently across crawlers, which created ambiguity. The current robots.txt documentation is now explicit — Google supports four fields (user-agent, allow, disallow, and sitemap), and the docs say “other fields such as crawl-delay aren’t supported.” Google’s Myths and facts about crawling page repeats it: “The non-standard ‘crawl-delay’ robots.txt rule is not processed by Google’s crawlers.” Evidence for this claim Google does not support or process the non-standard crawl-delay robots.txt field. Scope: Google crawlers; other crawlers may support the field. Confidence: high · Verified: 2026-07-14 Google: robots.txt specifications

Worth being clear about the scope: it’s ignored no matter which user-agent group you put it in. A User-agent: Googlebot block with a Crawl-delay line is just as ignored as one under User-agent: *.

What to use instead for Google

Google’s crawl rate is fully automatic now, tuned to your server’s health — and the old manual crawl-rate slider in Search Console was removed on January 8, 2024 (announced the prior November). So when you genuinely need Googlebot to back off, you have three levers:

1. Fix the server. The real fix. If your server can take the load, you don’t need to throttle anything.
2. Return 429, 500, or 503 for emergencies. Google’s Reduce the Googlebot crawl rate docs say to “return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests.” Googlebot reads these as “slow down” almost immediately. The hard rule: “We don’t recommend that you do this for a long period of time (meaning, longer than 1-2 days).” Sustained 5xx risks pages getting dropped (and Google Ads pausing). Evidence for this claim Google recommends temporarily returning 500, 503, or 429 to reduce crawl rate and warns against doing so for longer than one or two days. Scope: Temporary Googlebot overload response, not routine crawl management. Confidence: high · Verified: 2026-07-14 Google: Reduce Googlebot crawl rate
3. File a crawl-rate reduction request through Search Console for a persistent problem. It’s slow, and it only reduces, never raises.

One trap to avoid: don’t use 401, 403, or 404 to throttle. Per Google’s HTTP status codes guide, “the 4xx status codes, except 429, have no effect on crawl rate,” and you should not “use 401 and 403 status codes for limiting the crawl rate.”

This is also why, in my crawl-rate guide, I treat crawl-delay as the thing not to use for Google — see that article for the full speed-up / slow-down playbook.

Bing documents a 1–20 second range

Current Bing Webmaster guidance documents crawl-delay values from 1–20 seconds. That does not make the field a portable standard: Google ignores it, and every other crawler needs its own documented confirmation.

Yandex dropped it in 2018

Yandex used to honor crawl-delay as a literal minimum number of seconds between requests — that’s the version of the fact you’ll still see repeated across most SEO blogs. It’s stale. Yandex’s own current documentation is explicit: “From February 22, 2018, Yandex doesn’t take into account the Crawl-delay directive.” Evidence for this claim Yandex stopped honoring the Crawl-delay directive on February 22, 2018, and now recommends setting crawl rate inside Yandex Webmaster instead. Scope: Yandex's crawler only; does not apply to Google, Bing, or other crawlers. Confidence: high · Verified: 2026-07-18 Yandex Webmaster: The Crawl-delay directive To set how fast Yandex’s bots crawl your site now, you use the crawl rate setting inside Yandex Webmaster directly — not robots.txt. So as of today, the honest scorecard is: Bing honors crawl-delay within its documented range, Google never did, and Yandex used to but stopped in 2018.

Other crawlers: SEO tools and AI bots

Plenty of well-behaved non-search-engine crawlers honor crawl-delay as a courtesy:

• AhrefsBot and Semrush’s bot both respect it.
• AI crawlers are a newer audience. Anthropic’s ClaudeBot is documented as supporting the non-standard crawl-delay directive. Others (GPTBot, Perplexity) should be checked against their own published docs rather than assumed.

The pattern: smaller, compliance-focused, or courtesy-minded crawlers tend to honor crawl-delay; the major search engines have largely moved to internal signals (Google) or richer manual tools (Bing). And none of this binds actual bad actors — scrapers that ignore robots.txt entirely will ignore your crawl-delay too.

The use case that’s left

So is crawl-delay dead? Not quite. It’s useless for Google, redundant-at-best for Bing (use Crawl Control), but it’s genuinely the simplest way to ask the broad middle — SEO tools, AI bots, and other minor crawlers that respect it — to ease off. If a specific tool’s bot is overrunning your server, a targeted User-agent: group with a crawl-delay is a reasonable first move:

User-agent: SomeBot
Crawl-delay: 10

Just remember it’s a request, not a guarantee, and it scopes per host and per user-agent group like everything else in robots.txt. For the broader efficiency picture — what bots actually cost you and who needs to care — see crawl-budget.

AI summary

A condensed take on the Advanced version:

• What it is: Crawl-delay is a non-standard robots.txt directive asking bots to pause a set number of seconds between fetches to reduce server load. It was never in the formal standard (RFC 9309, 2022).
• Google ignores it. Retired September 1, 2019 (Gary Illyes’ 2019 post); current docs say “other fields such as crawl-delay aren’t supported.” Ignored regardless of which user-agent group it’s in.
• Slow Google instead with: fixing the server (best), or returning 429/500/503 for emergencies only (1-2 days max — longer risks de-indexing). Never use 401/403/404 to throttle. The GSC manual crawl-rate slider was removed January 8, 2024.
• Bing documents 1–20 second values. Treat that as Bing-specific guidance, not a portable robots.txt standard.
• Yandex dropped support for it on February 22, 2018. Older advice that it reads the value as literal seconds is stale — Yandex now uses a crawl-rate setting inside Yandex Webmaster instead.
• Other crawlers: AhrefsBot and Semrush respect it; Anthropic’s ClaudeBot is documented as supporting it. Bad-actor scrapers ignore robots.txt entirely.
• Remaining use case: controlling non-Google, non-Bing, non-Yandex crawlers (SEO tools, AI bots) via a targeted User-agent: group.

Official documentation

Primary-source documentation on crawl-delay and its alternatives.

Google

• robots.txt documentation — lists the four supported fields and states crawl-delay isn’t supported.
• A note on unsupported rules in robots.txt (July 2, 2019) — the retirement announcement (effective September 1, 2019) covering noindex, nofollow, and crawl-delay.
• Myths and facts about crawling — “The non-standard ‘crawl-delay’ robots.txt rule is not processed by Google’s crawlers.”
• Reduce the Googlebot crawl rate — what to use instead: 429/500/503, and the 1-2 day limit.
• How HTTP Status Codes Affect Google’s Crawlers — why 4xx (except 429) don’t throttle.
• Upcoming deprecation of the Crawl Rate Limiter Tool (Nov 2023) — the GSC slider’s removal.

Bing / Microsoft

• Bingbot guidance — current Bing Webmaster guidance documents values from 1–20 seconds.
• Bing Webmaster Tools — Crawl Control — the preferred alternative to crawl-delay for Bing.

Yandex

• Yandex Webmaster — The Crawl-delay directive — states Yandex has not honored the directive since February 22, 2018 and points to the Yandex Webmaster crawl-rate setting instead.

The standard

• RFC 9309: Robots Exclusion Protocol — the formalized standard, which does not include crawl-delay.

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page where supported.

Google — crawl-delay is not processed

• “The non-standard ‘crawl-delay’ robots.txt rule is not processed by Google’s crawlers.” — Google Search Central docs. Jump to quote
• “Google supports the following fields … (other fields such as crawl-delay aren’t supported).” — robots.txt documentation. Jump to quote

Google — what to use instead

• “To quickly reduce the crawl rate, you can return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests.” — Reduce the Googlebot crawl rate. Jump to quote
• “We don’t recommend that you do this for a long period of time (meaning, longer than 1-2 days).” — Reduce the Googlebot crawl rate. Jump to quote

Gary Illyes, Google (on the GSC crawl-rate tool deprecation, November 2023)

• “if the server persistently returns HTTP 500 status codes for a range of URLs, Googlebot will automatically, and almost immediately slow down crawling.” He added that the tool’s “usefulness has dissipated.” Relayed via Search Engine Land’s coverage of the deprecation post — read the coverage; confirm against Google’s original post before treating as final.

Bing / Microsoft

• Current Bingbot guidance documents crawl-delay values from 1–20 seconds. It is Bing-specific guidance, not a general REP rule.

Yandex (current Webmaster documentation)

• “From February 22, 2018, Yandex doesn’t take into account the Crawl-delay directive. To set the rate at which the indexing bots should access your site’s pages, use the site crawl rate setting in Yandex Webmaster.” — Yandex Webmaster: The Crawl-delay directive

Crawl-delay decision checklist

Work down the list based on which bot you’re trying to slow:

• Audit your robots.txt. Is there a crawl-delay line already? Note which User-agent group(s) it’s in and which bots you actually mean to throttle.
• If it’s there for Google: remove it (or document that it’s deliberately targeting other bots). It does nothing for Googlebot.
• Trying to slow Google? Fix the server first. For a genuine emergency, return 429/503 — for no more than 1-2 days. Never use 401/403/404 to throttle. For a persistent problem, file a crawl-rate request in Search Console.
• Trying to slow Bing? Use Crawl Control in Bing Webmaster Tools first. If you do use crawl-delay, keep the value at 10 or below, and remember it applies from the * group even with a bingbot section present.
• Trying to slow another crawler (SEO tool, AI bot, etc.)? Put a crawl-delay in a targeted User-agent: group for that bot — it helps if that crawler honors the directive (many do).
• Confirm the value’s meaning for your target bot: Bing reads it as a relative throttle factor. Yandex used to read it as literal seconds but stopped honoring the directive entirely on February 22, 2018 — use the Yandex Webmaster crawl-rate setting for Yandex instead.
• Don’t expect scrapers or bad actors to comply — they ignore robots.txt entirely. crawl-delay is for well-behaved bots only.

Crawl-delay cheat sheet

Who honors crawl-delay?

Bing throttle scale

What to use instead, by engine

Don’t

• Don’t use crawl-delay expecting it to affect Google or Yandex.
• Don’t use 401/403/404 to throttle Googlebot (no effect; only 429 among 4xx does).
• Don’t set Bing values above 10.

How should you slow a crawler?

Tools for crawl-delay work

• robots.txt Tester checks which user-agent group and rule apply; it helps catch scoping mistakes even though support for crawl-delay remains crawler-specific.
• robots.txt Generator helps build separate bot groups instead of applying a broad delay to every crawler.
• Log File Analyzer shows the request rate actually produced after a change and separates search crawlers from spoofed user agents.
• Googlebot Verifier verifies whether the IP creating load really belongs to the crawler it claims to be.

Resources worth your time

My related writing

• Robots.txt and SEO: Everything You Need to Know — the full robots.txt picture, including why crawl-delay isn’t a Google directive.
• The Beginner’s Guide to Technical SEO — where crawl controls sit in the bigger picture.
• Indexed, though blocked by robots.txt — a reminder that robots.txt controls crawling, not indexing.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawling, rendering, indexing, and ranking. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From around the industry

• A note on unsupported rules in robots.txt (Google Search Central) — the original 2019 retirement announcement.
• Bingbot guidance — current Bing Webmaster guidance documents 1–20 second values.
• Google To Drop Support For crawl-delay, nofollow, and noindex in robots.txt (Search Engine Roundtable) — coverage of the 2019 change.
• Google removes support for crawl-delay, nofollow, and noindex rules in robots.txt (Coywolf) — additional context on the retirement.
• Googlebot crawl rate tool in Search Console is going away (Search Engine Land) — Gary Illyes’ quotes on the alternatives being better.
• Crawl-delay: 10 in robots.txt: what does it mean? (Conductor) — a widely-cited explainer on the directive.

Test yourself: Crawl Delay

Five quick questions on the crawl-delay directive. Pick an answer for each, then check.

Crawl Delay

Crawl-delay is a non-standard robots.txt directive that asks bots to wait between successive page fetches to ease server load. Google has ignored it since September 1, 2019 and Yandex dropped it on February 22, 2018; Bing and many other crawlers still honor it, but Bing interprets the value differently than the plain reading.

Related: Crawl rate, Crawl Budget

Crawl Delay

Crawl-delay is an unofficial robots.txt directive that tells a crawler to wait a set number of seconds between requests so it doesn’t overwhelm your server. The syntax is simple — a Crawl-delay: line with a positive integer inside a User-agent group:

User-agent: *
Crawl-delay: 10

It was never part of the formal Robots Exclusion Protocol (RFC 9309), and Google has not processed it since September 1, 2019 — Google’s docs state plainly that “other fields such as crawl-delay aren’t supported.” To slow Googlebot you return 429/503 responses (for a day or two) or fix server capacity, not crawl-delay.

Bing does honor it, but not as literal seconds — it treats the value as a relative throttle factor and recommends keeping it at 10 or below. Yandex used to treat it as a literal minimum number of seconds between requests, but stopped honoring the directive entirely on February 22, 2018; Yandex’s crawl rate is now set inside Yandex Webmaster, not robots.txt. Several SEO tools (AhrefsBot, Semrush) and AI crawlers (Anthropic’s ClaudeBot) still respect it. That makes crawl-delay still useful for controlling non-Google, non-Yandex crawlers, even though it does nothing for Google or Yandex.

Related: Crawl rate, Crawl Budget

Changelog