429 Too Many Requests

TL;DR — A 429 Too Many Requests means your server told a visitor or bot “you’re asking for too many pages too fast — slow down.” That’s called rate limiting. The good news for SEO: 429 is the one error in its family that Google handles gently. Instead of dropping your pages, Googlebot just eases off and crawls you more slowly for a while. It only becomes a problem if your server keeps sending 429s for days on end.

What a 429 actually says

Every time a browser, script, or search-engine crawler asks your server for a page, the server answers with a status code. 200 means “here’s the page.” A 429 means “you’ve made too many requests in a short window, so I’m not answering this one — come back later.”

Servers use 429 on purpose to protect themselves. If one visitor (or one bot) is hammering the site fast enough to slow it down for everyone else, returning 429 is how the server says knock it off for a bit. A well-behaved 429 response also includes a Retry-After header — a note telling the client how many seconds to wait before trying again. Evidence for this claim A 429 response means the user sent too many requests in a given time, and the response may include Retry-After. Scope: RFC 9110 defines the status and optional Retry-After field; it does not define a universal rate threshold. Confidence: high · Verified: 2026-07-14 IETF: RFC 9110 §15.5.20 — 429 Too Many Requests

Why 429 is the “nice” error for SEO

Here’s the part that surprises people. 429 lives in the “4xx client error” group, right next to codes like 403 Forbidden and 404 Not Found. Those two are bad news if Google keeps seeing them — Google eventually drops those pages from search.

429 is the exception. Google reads a 429 as the server is busy, slow down — the same way it reads a 503 or a 500 server error. So instead of removing your pages, Googlebot just crawls your site more slowly for a while. Google actually recommends 429 as the correct way to slow a crawler down, and specifically warns against using 403 or 404 to do it. Evidence for this claim Google treats 429 as a server-overload signal that reduces crawl rate and recommends 429, 500, or 503 for temporary crawl reduction instead of other 4xx codes. Scope: Google's guidance covers Google crawler behavior and short-term overload; it does not promise indexing preservation during prolonged unavailability. Confidence: high · Verified: 2026-07-14 Google: Reduce Google crawl rate

When 429 becomes a problem

A 429 here and there is completely normal and self-healing — once your server stops sending them, Google speeds its crawling back up on its own. You don’t have to ask.

The danger is when 429s stick around. If Googlebot keeps hitting 429 on the same pages for more than a day or two, Google may start dropping those pages from its index — because from Google’s side it looks like your site has been broken and unavailable for days. So 429 is a short-term tool, not a permanent setting.

What you’d do about it

• If you didn’t mean to send 429s (they show up in Google Search Console or your crawl tool by surprise), something is rate-limiting too aggressively — often a security plugin, a firewall (WAF), or your hosting/CDN. Find it and loosen the limit so it stops blocking real search engines.
• If you did mean to send them — say your server is under heavy load — that’s fine, just time-box it and add a Retry-After header. Turn it off within a day or two.

Want the server-config how-to, Google’s exact wording, and the myths people get wrong? Switch to the Advanced tab.

TL;DR — 429 is the single 4xx code Google treats like a 5xx: once it encounters a significant number of 500/503/429 responses, it reads that as a server-overload signal and throttles Googlebot’s crawl rate hostname-wide instead of removing content. It’s the one code Google endorses for slowing a crawler down — never 403 or 404. Google’s docs give emergency-use guidance of “a couple of hours, or 1–2 days” — not a guaranteed safe window; sustained 429s on the same URLs risk those URLs eventually being dropped from the index, and the throttle starts increasing again (not necessarily instantly or fully) once error volume falls. RFC 6585 says a 429 SHOULD explain the condition and MAY include Retry-After — sending it is recommended practice, not a compliance requirement. Scope the limit to the right traffic and verify crawler identity before writing exceptions.

What 429 is at the protocol level

Straight from the spec, as MDN puts it: “The HTTP 429 Too Many Requests client error response status code indicates the client has sent too many requests in a given amount of time. This mechanism of asking the client to slow down the rate of requests is commonly called ‘rate limiting.’” Evidence for this claim A 429 response means the user sent too many requests in a given time, and the response may include Retry-After. Scope: RFC 9110 defines the status and optional Retry-After field; it does not define a universal rate threshold. Confidence: high · Verified: 2026-07-14 IETF: RFC 9110 §15.5.20 — 429 Too Many Requests

RFC 6585 §4 is more precise than most summaries of it get. A 429 representation SHOULD explain the condition and MAY include a Retry-After header giving the client a concrete number of seconds (RFC 9110 also allows an HTTP date) to wait before retrying — Retry-After is a recommended practice, not a compliance requirement. The spec also doesn’t define how a client is identified or how requests are counted; that’s left entirely to whatever issued the response (per IP, per session, per API key, per resource — implementation policy, not protocol). One more rule that’s easy to miss: RFC 6585 says a 429 response must not be stored by a cache. If you see a 429 that looks cached or replayed on a healthy origin, that’s an intermediary (CDN, proxy) misbehaving, not the origin re-deciding to limit you.

I’ve always framed it plainly in my HTTP Status Codes & Their SEO Impact guide: 429 is “a form of rate-limiting to protect the server because the client sent too many requests to the server too fast.” Nominally it’s a client error — the client did something wrong by asking too much. But that framing is exactly where the SEO story diverges from the spec.

The one exception among 4xx codes

The single most important fact on this page: Google does not treat 429 like the rest of the 4xx family. Gary Illyes wrote a whole Google Search Central blog post about this in February 2023, because enough sites and CDNs were misusing 404s to throttle Googlebot that Google had to tell them to stop.

His rule: “The one exception is 429, which stands for ‘too many requests’. This error is a clear signal to any well-behaved robot, including our beloved Googlebot, that it needs to slow down because it’s overloading the server.” And the flip side, in Google’s status-code reference: “Don’t use 401 and 403 status codes for limiting the crawl rate. The 4xx status codes, except 429, have no effect on crawl rate.”

So in the same breath where 403 and 404 will get your content removed from Search, 429 gets you a temporary slowdown. Google literally buckets it with server errors: “Google’s crawlers treat the 429 status code as a signal that the server is overloaded, and it’s considered a server error.” Evidence for this claim Google treats 429 as a server-overload signal that reduces crawl rate and recommends 429, 500, or 503 for temporary crawl reduction instead of other 4xx codes. Scope: Google's guidance covers Google crawler behavior and short-term overload; it does not promise indexing preservation during prolonged unavailability. Confidence: high · Verified: 2026-07-14 Google: Reduce Google crawl rate

This is what makes 429 the useful sibling of 503 Service Unavailable (the traditional maintenance/temporary-unavailable signal) — and the exact opposite of 403 Forbidden, which is the wrong tool for rate limiting despite how many firewalls default to it.

Crawl-rate impact is hostname-wide — with a threshold

Two different scopes get conflated here, and it’s worth separating them. How you count and key a rate limit — per IP, per session, per API key, per resource, per server — is your own policy; the HTTP spec doesn’t define it. What Google does with the errors it observes is a separate, documented behavior, and it’s conditioned on volume, not a single response: “Google’s crawling infrastructure reduces your site’s crawling rate when it encounters a significant number of URLs with 500, 503, or 429 HTTP response status codes.” Once that threshold is met, “the reduced crawl rate affects the whole hostname of your site (for example, subdomain.example.com), both the crawling of the URLs that return errors, as well as the URLs that return content.”

In other words, if you 429 a subset of pages (say, a heavy API path) at real volume, Googlebot slows its crawling of the entire hostname — including the pages still returning 200. That’s usually the intended effect when your goal is to reduce total load. But one isolated 429 on one path, on its own, doesn’t establish that hostname-wide effect — Google’s own wording is scoped to “a significant number” of error responses, not any single one.

The 1–2 day guidance — when 429 turns risky

429 is a short-term signal, and Google gives concrete emergency-use guidance for it — not a guaranteed safe window or a hard cliff. From the “reduce crawl rate” docs: “If you need to urgently reduce the crawl rate for short period of time (for example, a couple of hours, or 1-2 days), then return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests.”

Push past that and you’re in riskier territory, though Google frames it as a possibility, not a promise: “We don’t recommend that you do this for a long period of time (meaning, longer than 1-2 days) as it may have a negative effect on how your site appears in Google products… if Googlebot observes these status codes on the same URL for multiple days, the URL may be dropped from Google’s index.” The status-code reference says the same thing for 5xx and 429 together: “already indexed URLs are preserved in the index, but eventually dropped.”

The risk model, stated honestly: short-term 429 fits the window Google recommends for emergency use; sustained 429 on the same URLs over multiple days is where Google’s own language turns to “may” and “eventually dropped” — a documented risk, not a guaranteed outcome either way. It’s the same dynamic as a prolonged 503.

Crawl rate recovers automatically

The reassuring flip side: there’s no penalty flag that follows your site around. Once the errors subside, Google says “the crawl rate will automatically start increasing again.” You don’t file anything, you don’t re-request anything. Note the exact wording, though — Google says “starts increasing,” not “instantly returns to your prior rate.” Treat recovery as a documented direction with no fixed timeline or guaranteed endpoint, not an SLA.

(Contrast that with the opposite problem — wanting Google to permanently crawl you less. If serving errors isn’t feasible, Google says to “file a special request to report a problem with unusually high crawl rate” — a manual path that can take days and isn’t guaranteed. There’s no such friction for recovery.)

How Bing handles 429

Bingbot is widely reported to behave similarly — 429/500/503 signaling overload and Bingbot backing off — though I couldn’t independently confirm current wording to that effect on Bing’s own help pages in this research pass (they’re JS-rendered SPAs that didn’t yield fetchable static text). Treat the parity claim as industry-reported, not something I’ve verified against current Bing documentation. What Bing does confirm, in its own historical guidance, are two proactive controls Google doesn’t offer in the same form:

• Crawl Control in Bing Webmaster Tools, a requests-per-second grid where you set Bingbot’s speed by hour of day.
• The crawl-delay robots.txt directive. Current Bing Webmaster guidance documents values from 1–20 seconds. This is Bing-specific; it does not throttle Googlebot.

So on Bing you can throttle proactively with crawl-delay or Crawl Control instead of reactively with status codes. (Google retired its own manual crawl-rate slider in 2024 and now leans entirely on your server’s responses.)

When you’d deliberately send a 429

Legitimate reasons to return 429 on purpose:

• Emergency server load — a traffic spike, a botched migration, or an outage where you need Googlebot to ease off right now for a few hours.
• Protecting APIs and non-HTML endpoints from crawler/bot abuse — search-engine crawlers, third-party SEO crawlers (Ahrefs, Screaming Frog), and scrapers all hit rate limits meant to stop abuse.

What 429 is not for: permanently blocking bots you don’t want at all. If you never want something crawled, that’s a robots.txt disallow job, not a 429. And if you want to keep a page but out of the index, that’s noindex. 429 only means “later,” not “never.”

Unintentional 429s — the usual culprits

When 429s show up in GSC’s Page Indexing or Crawl Stats report and you didn’t put them there, the source is typically one of these — I’m not aware of good evidence for a universal ranking of which is most common, so treat this as a list of candidates to rule in or out, not a diagnosis:

• WAF / firewall rules misfiring on legitimate crawler IP ranges.
• Shared hosting or CDN default rate limits that are too tight for a real crawl.
• Bot-management tools misclassifying Googlebot or Bingbot as abusive traffic.
• Aggressive rate-limiting middleware meant for API abuse catching your own crawlers.

Before you touch a threshold or write a rule that exempts real search bots, establish provenance first — don’t guess which layer owns the response. Pull the raw response headers, the exact request log lines (not a dashboard summary), the rule or rate-limit-zone identifier that fired, the client key it counted against (IP, session, API key), the route, the CDN POP or edge location, and the time window. That combination tells you which layer actually issued the 429 and what it was counting — only then does it make sense to loosen a limit or add an exception. Verify crawler identity with reverse-plus-forward DNS (Google’s official method), not the user-agent string alone — spoofed “Googlebot” user-agents are common. The Scripts tab has the exact commands.

The short version of the playbook

1. Use 429 or 503 with Retry-After to slow a crawler — never 403 or 404.
2. Keep it to hours, or 1–2 days — beyond that, URLs can be dropped.
3. Remember the throttle is hostname-wide, and auto-recovers when errors stop.
4. Scope the limit to the right traffic and verify crawler identity before exempting bots.

AI summary

A condensed take on the Advanced version:

• 429 Too Many Requests = the server telling a client (browser, script, or crawler) it sent too many requests too fast — “rate limiting.” Nominally a 4xx client error.
• Protocol level: RFC 6585 says a 429 response SHOULD explain the condition and MAY include Retry-After — recommended, not required. It also MUST NOT be stored by a cache; a cached/replayed 429 is an intermediary bug, not the origin. How you count/key a limit (IP, session, API key, resource) is your own policy — the spec doesn’t define it.
• The one 4xx exception: Google treats 429 like a 5xx server error, not like 403/404. It reads it as “server overloaded, slow down” and throttles the crawl rate instead of removing content.
• Google endorses 429 (and 500/503) for slowing crawlers — never 403 or 404. Gary Illyes wrote a 2023 post specifically telling sites/CDNs to stop using 404s to throttle Googlebot.
• Throttling is hostname-wide, above a threshold — Google’s own wording conditions the whole-hostname effect on encountering “a significant number” of 500/503/429 responses, not any single 429. Above that threshold, pages still serving 200 get crawled less too.
• 1–2 day guidance, not a hard cliff: Google’s docs give emergency-use guidance of “a couple of hours, or 1–2 days.” Sustained 429s on the same URLs over multiple days risk those URLs eventually being dropped from the index — “may” and “eventually,” not a guarantee — similar to a prolonged 503.
• Recovery is directional, not instant: stop sending 429s and Google says the crawl rate “starts increasing again” — no re-request, no lingering penalty, but also no promise of an immediate or fully restored rate on a fixed timeline.
• Client-side retries: honor a valid Retry-After when present; when absent, the RFC prescribes no formula — use a bounded backoff-with-jitter policy, cap attempts, and check idempotency before repeating non-idempotent requests.
• Scope the limit to the right traffic and verify crawler identity (reverse+forward DNS) before exempting bots.
• Bing is widely reported to back off on 429 similarly, though that parity claim wasn’t independently confirmed against current Bing documentation in this pass; Bing’s historical guidance does confirm crawl-delay and the Crawl Control grid as proactive alternatives.
• 429 ≠ block: it means “later,” not “never.” Use robots.txt to keep bots out and noindex to remove a page.

Official documentation

Primary-source documentation from the search engines and the HTTP spec.

Google

• Don’t use 403s or 404s for rate limiting — Gary Illyes’ Feb 2023 post; the canonical statement that 429 is the exception and 403/404 are the wrong tools.
• Reduce the Google crawl rate — the “return 500, 503, or 429” guidance, the 1–2 day window, hostname-wide throttling, and auto-recovery.
• HTTP status codes, network, and DNS errors — how Google handles each code; 429 grouped with server errors (also served at the older search/docs/crawling-indexing/http-network-errors path).
• Reduce the Google crawl rate — special request path — the manual “report a problem with unusually high crawl rate” path when serving errors isn’t feasible.

Bing / Microsoft

• Crawl Control — Bingbot’s requests-per-second scheduler in Bing Webmaster Tools.
• Bingbot guidance — current Bing Webmaster guidance for crawl-delay (1–20 seconds).

HTTP spec

• MDN — 429 Too Many Requests — the protocol definition, Retry-After, and rate-limiting basics.

Quotes from the source

On-the-record statements from Google, Bing, and the HTTP spec. Each link is a deep link that jumps to the quoted passage on the source page.

Google — Gary Illyes, “Don’t use 403s or 404s for rate limiting” (Feb 2023)

• “Over the last few months we noticed an uptick in website owners and some content delivery networks (CDNs) attempting to use 404 and other 4xx client errors (but not 429) to attempt to reduce Googlebot’s crawl rate.” Jump to quote
• “The one exception is 429, which stands for ‘too many requests’. This error is a clear signal to any well-behaved robot, including our beloved Googlebot, that it needs to slow down because it’s overloading the server.” Jump to quote
• “All 4xx HTTP status codes (again, except 429) will cause your content to be removed from Google Search.” Jump to quote
• “Use Search Console to temporarily reduce crawl rate. Return a 500, 503, or 429 HTTP status code to Googlebot when it’s crawling too fast.” Jump to quote

Google — Reduce the crawl rate / status-code reference

• “If you need to urgently reduce the crawl rate for short period of time (for example, a couple of hours, or 1-2 days), then return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests.” Jump to quote
• “The reduced crawl rate affects the whole hostname of your site… Once the number of these errors is reduced, the crawl rate will automatically start increasing again.” Jump to quote
• “We don’t recommend that you do this for a long period of time (meaning, longer than 1-2 days)… the URL may be dropped from Google’s index.” Jump to quote
• “Google’s crawlers treat the 429 status code as a signal that the server is overloaded, and it’s considered a server error.” Jump to quote
• “Don’t use 401 and 403 status codes for limiting the crawl rate. The 4xx status codes, except 429, have no effect on crawl rate.” Jump to quote

MDN — the protocol definition

• “The HTTP 429 Too Many Requests client error response status code indicates the client has sent too many requests in a given amount of time. This mechanism of asking the client to slow down the rate of requests is commonly called ‘rate limiting.’” Jump to quote

Bing — current Webmaster guidance

• Bingbot guidance documents crawl-delay values from 1–20 seconds. Treat it as Bing-specific guidance rather than a generic crawler throttle.

Industry corroboration of Google’s statement

• The Feb 2023 Illyes post was covered verbatim by Search Engine Land, Search Engine Roundtable, and Search Engine Journal — all three framing 429 as “the one exception.” These are trade-press write-ups of the same Google post; the primary source is the Illyes blog post quoted above.

Send a proper 429 — with Retry-After

The single most important piece of a well-behaved 429 is the Retry-After header. It tells any compliant client — including Googlebot — how long to wait. It takes either a number of seconds or an HTTP date:

HTTP/1.1 429 Too Many Requests
Retry-After: 3600
Content-Type: text/html

<html><body>Too many requests. Please retry later.</body></html>

HTTP/1.1 429 Too Many Requests
Retry-After: Wed, 01 Jul 2026 12:00:00 GMT

Omitting Retry-After doesn’t break compliance, but including it is best practice and gives crawlers a concrete back-off signal.

nginx — rate limiting that returns 429

By default nginx’s limit_req returns 503. For crawler-friendly semantics, override it to 429 and add a Retry-After header. This example allows 10 requests/second per IP with a small burst:

# In the http {} block: define a shared memory zone keyed by client IP
limit_req_zone $binary_remote_addr zone=crawl_limit:10m rate=10r/s;

server {
    location / {
        limit_req zone=crawl_limit burst=20 nodelay;

        # Return 429 (not the default 503) when the limit is exceeded
        limit_req_status 429;
    }

    # Attach a Retry-After header to 429 responses
    error_page 429 = @too_many;
    location @too_many {
        add_header Retry-After 3600 always;
        return 429;
    }
}

Apache — rate limiting with mod_ratelimit / mod_evasive

Apache’s core mod_ratelimit throttles bandwidth, not request count, so for request-rate limiting you typically reach for mod_evasive (or a WAF). To make a throttled response return 429 with a Retry-After, set it explicitly:

# Send a 429 with Retry-After for a chosen condition (e.g. a rate-limit env var)
<If "%{ENV:RATE_LIMITED} == '1'">
    Header always set Retry-After "3600"
    Redirect 429 /
</If>

# mod_evasive: throttle abusive request bursts (returns 429 in recent versions;
# older builds default to 403 — override where possible)
<IfModule mod_evasive20.c>
    DOSPageCount        5
    DOSSiteCount        50
    DOSPageInterval     1
    DOSBlockingPeriod   60
</IfModule>

Note the caveat: older mod_evasive builds default the block response to 403 — which is exactly the code Google says not to use for rate limiting. Confirm your build returns 429 (or front it with a CDN/WAF that does).

Cloudflare / CDN — rate limit to 429

At the CDN layer, set your rate-limiting rule’s action response to 429 (many default to 403 or a challenge). In Cloudflare’s Rate Limiting rules the response status for exceeded limits is configurable — choose 429 and, where supported, attach a Retry-After. Same principle on Fastly, Akamai, or an API gateway: the action on limit-exceeded should be 429 Too Many Requests, not 403 Forbidden.

Client-side: retrying after a 429 without a formula

If you’re writing the client, the RFC gives you exactly one hard rule and no fallback algorithm: honor a valid Retry-After when it’s present. When it’s absent, RFC 6585 doesn’t prescribe a retry interval, a backoff formula, a jitter distribution, an attempt count, or a “success” condition — those are your policy to set, not a spec requirement. Don’t present any one formula (including the one below) as HTTP law; it’s a reasonable bounded default, not the only correct one.

on 429 response:
    if Retry-After header present and valid:
        wait = parse(Retry-After)          # seconds or HTTP-date
    else:
        wait = min(cap, base * 2^attempt) + random_jitter(0, jitter_window)

    if attempt >= max_attempts or wait > cap:
        stop and surface the failure        # don't retry forever
    if request is not idempotent (e.g. a POST that isn't safe to repeat):
        confirm idempotency (idempotency key, or a safe no-op check) before retrying
    sleep(wait)
    attempt += 1
    retry request

The pieces that matter regardless of the exact numbers you pick: a cap on total wait and attempt count (don’t retry forever), jitter (so a fleet of clients doesn’t retry in lockstep and re-trigger the limit), and an idempotency check before retrying anything that isn’t safe to repeat (a POST that isn’t idempotent needs a dedup key, not a blind retry).

Verify a bot is really Googlebot before exempting it

If you’re writing rate-limit exceptions for search engines, confirm identity with a reverse + forward DNS check — spoofed “Googlebot” user-agents are common and the UA string alone proves nothing.

macOS / Linux

# 1) Reverse DNS the IP from your logs — it should end in googlebot.com or google.com
host 66.249.66.1
# → ... domain name pointer crawl-66-249-66-1.googlebot.com

# 2) Forward DNS that hostname back — it must resolve to the same IP
host crawl-66-249-66-1.googlebot.com
# → crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Windows

nslookup 66.249.66.1
nslookup crawl-66-249-66-1.googlebot.com

If the reverse lookup doesn’t end in a Google domain, or the forward lookup doesn’t match the original IP, it isn’t Googlebot. You can also match against Google’s published IP ranges (googlebot.json).

Common myths about 429 and SEO

Myth: “Any 429 will hurt my SEO / get me deindexed.” No. Google designed 429 to be a safe, expected throttling signal. Short-term or intermittent 429s are normal and self-correcting. Deindexing risk only appears with sustained 429s on the same URLs for multiple days — Google’s own 1–2 day guidance is the line.

Myth: “429 and 503 are basically interchangeable for SEO.” For crawl-rate throttling, Google treats them similarly. But they mean different things: 503 Service Unavailable is the traditional “temporarily down / maintenance” signal, while 429 specifically communicates rate/volume overload. Using the semantically correct one matters for your own monitoring, tooling, and anything downstream that reads status codes — even if Googlebot backs off either way.

Myth: “You can use 403 or 404 to slow down Googlebot just like 429.” Directly refuted by Illyes’ Feb 2023 post — this was common enough that Google wrote a dedicated article telling people to stop. 403/404 have no effect on crawl rate and actively remove content from the index. 429 is the only 4xx code that throttles.

Myth: “Rate-limiting search bots causes a permanent crawl-budget cut.” The reduction is temporary and auto-recovers once error volume subsides. There’s no lingering flag on your site after the errors stop — Google’s docs say the crawl rate “will automatically start increasing again.”

Myth: “If Googlebot gets a 429, it gives up on that URL for good.” Google retries later. The concern is sustained 429s over multiple days per URL — not a one-off or occasional rate limit, which Googlebot simply respects and comes back from.

Frequently asked questions

Does a 429 error hurt my SEO? Not by itself. Short-term or intermittent 429s just slow Googlebot down temporarily and self-heal. The risk only appears when the same URLs return 429 for multiple days.

How long can I return a 429 before Google deindexes my pages? Google’s docs say limit it to “a couple of hours, or 1–2 days.” Beyond that, “the URL may be dropped from Google’s index.” Treat 1–2 days as a hard ceiling, not a target.

What’s the difference between 429 and 503 for SEO? Google throttles crawling on both similarly. But 503 means “temporarily unavailable / maintenance,” while 429 specifically means “you’re sending too many requests.” Use the semantically correct one so your own monitoring and tooling read it right.

Should I block Googlebot with a 429 if I want it to crawl less permanently? No — 429 is a short-term signal, not a permanent setting. For a lasting reduction, Google says to file a special request about high crawl rate. To keep bots out of a space entirely, use robots.txt; to remove a page from the index, use noindex.

Does Bingbot respect 429 the same way Googlebot does? Bingbot also backs off on overload signals like 429/500/503. Bing additionally offers proactive controls Google doesn’t — the crawl-delay robots.txt directive and the Crawl Control requests-per-second grid in Bing Webmaster Tools.

What is the Retry-After header and do I need it? Retry-After tells a client how long to wait before retrying (a number of seconds or an HTTP date). It’s not strictly required, but it’s best practice and gives crawlers a concrete back-off signal.

Will Google resume normal crawling after I stop returning 429s? Yes, automatically. Once error volume drops, “the crawl rate will automatically start increasing again.” No re-request needed, and no lasting penalty.

Can WAF or CDN tools accidentally 429 Googlebot? Very commonly. Aggressive firewall rules, CDN/hosting default limits, and bot-management tools can misclassify Googlebot or Bingbot. Verify crawler identity (reverse + forward DNS) before writing exceptions, and loosen limits that catch real search engines.

Is 429 a client error or a server error? Technically it’s a 4xx client error in the HTTP spec. But Google treats it as a server error for crawling purposes — it’s the one 4xx code grouped with 5xx.

What should my client do if it gets a 429 with no Retry-After header? There’s no HTTP-mandated formula for this — the spec leaves it to policy. A reasonable bounded default: exponential backoff with jitter, a hard cap on total wait time and attempt count so you don’t retry forever, and an idempotency check before repeating any request that isn’t safe to send twice. See the Scripts tab for a worked pseudocode version.

What should I do about a 429?

Common 429 problems

Googlebot gets 429 but normal visitors do not

Symptom: crawler reports show 429 while browser checks return 200. Likely cause: bot management, a UA rule, or IP-based rate limiting. Fix: correlate WAF events with verified crawler IPs, then narrow or correct the responsible rule; do not allowlist a user-agent string alone.

The whole hostname’s crawl rate falls

Symptom: crawling slows beyond the URLs that returned 429. Likely cause: Google applies the overload signal hostname-wide. Fix: stop unintended 429s, restore stable success responses, and let crawl rate recover automatically.

429 responses continue after the incident

Symptom: the server is healthy but URLs still return 429. Likely cause: a CDN cache, edge rule, or limiter state outlived the event. Fix: disable or expire the temporary rule, purge an incorrectly cached response, and verify across representative paths and regions.

Retry-After is absent or unusable

Symptom: clients know they were limited but not when to retry. Likely cause: the response was generated by a generic security rule. Fix: make the issuing layer send a valid delay or HTTP date and test the raw header.

Prompt: audit a rate-limit rule

Review this 429 rate-limit configuration for search-crawler safety. Identify its key,
scope, threshold source, Retry-After behavior, hostname-wide SEO impact, and any
user-agent-only exceptions. Separate deliberate short-term overload protection from
permanent crawl control. Return a minimal safe revision, test matrix, monitoring
signals, and rollback conditions. Do not invent provider syntax.

[PASTE CONFIGURATION AND SANITIZED SAMPLE LOGS]

Prompt: diagnose unexplained 429s

Use these headers, access-log rows, WAF events, and timestamps to determine which
layer generated the 429 and which traffic dimension triggered it. Give competing
hypotheses ranked by evidence, the next exact check for each, and a fix that does not
trust claimed crawler user agents. Do not infer missing logs.

[PASTE EVIDENCE]

Tools for investigating 429s

• Bulk HTTP Status Code Checker: test a representative URL set and export which paths currently return 429.
• HTTP Header Checker: inspect Retry-After, CDN fingerprints, and redirect hops on the limited response.
• Googlebot Verifier: validate IP evidence before creating a crawler exception.
• Log File Analyzer: segment status-code waste by bot and section while keeping uploaded logs in the browser.
• Search Console Crawl Stats: compare the timing of 429s with changes in crawl requests and host response behavior.

Validate a 429 configuration

Response-contract test

Test to run: trigger the limit safely in a controlled environment and inspect the raw response. Expected result: 429 with a valid Retry-After and no accidental redirect or success status. Failure interpretation: the wrong layer or error template owns the response. Monitoring window: immediate. Rollback trigger: normal requests are limited or the test destabilizes the service.

Scope test

Test to run: exercise an intended limited client plus normal users and verified crawler traffic across representative URL classes. Expected result: only the defined traffic exceeds the limit. Failure interpretation: the rate-limit key or rule scope is too broad. Monitoring window: through the controlled test and edge propagation. Rollback trigger: unrelated routes, users, or hosts receive 429.

Recovery test

Test to run: stop the trigger, wait through the configured retry interval, and repeat the requests. Expected result: stable normal responses resume without a manual per-URL workaround. Failure interpretation: cached 429s or limiter state persist. Monitoring window: the configured interval plus deployment propagation. Rollback trigger: the hostname remains limited after the underlying load is gone.

Cache-storage test

Test to run: put a cache or CDN edge in front of the limited route, trigger a 429, then request the same URL again after the underlying condition clears. Expected result: the second request is evaluated fresh — no cached or replayed 429 served from the edge. Failure interpretation: an intermediary is storing a response RFC 6585 says must not be cached; check cache-control headers and edge rule configuration, not the origin. Monitoring window: immediate, across a representative set of edge nodes/POPs. Rollback trigger: any cached 429 is served after the origin has recovered.

Client-retry-policy test

Test to run: send a client through a 429 both with and without a valid Retry-After header present. Expected result: with the header, the client waits the specified interval; without it, the client applies a bounded backoff-with-jitter policy, respects a maximum attempt count, and checks idempotency before repeating a non-idempotent request. Failure interpretation: a client that retries immediately, retries unboundedly, or blindly repeats a non-idempotent request has a broken retry policy, not an HTTP-compliance issue. Monitoring window: through the full bounded retry sequence. Rollback trigger: the client re-triggers the same rate limit through immediate or unbounded retries.

Resources worth your time

My related writing

• HTTP Status Codes & Their SEO Impact — my full guide to how every status code affects SEO, including where 429 fits.
• The Beginner’s Guide to Technical SEO — where crawl controls and status codes sit in the bigger picture.
• The Story of Blocking 2 High-Ranking Pages With Robots.txt — my first-party experiment on what actually happens when you cut off crawlers.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawling, rendering, indexing, and ranking, including how servers signal crawlers to slow down. (My standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From around the industry

• Don’t use 403s or 404s for rate limiting (Google Search Central, Gary Illyes) — the canonical statement that 429 is the exception.
• Reduce the Google crawl rate (Google) — the 1–2 day window, hostname-wide throttling, and auto-recovery.
• Google warns against using 403 or 404 status codes for Googlebot crawl rate limiting (Search Engine Land) — trade-press coverage of the Illyes post.
• Google Says Stop Using 403s or 404s To Reduce Googlebot Crawl Rates (Search Engine Roundtable) — Barry Schwartz’s summary framing 429 as “the one exception.”
• Google: Don’t Use 403/404 Error Responses For Rate Limiting Googlebot (Search Engine Journal) — a third independent write-up of the same guidance.
• MDN — 429 Too Many Requests — the HTTP spec definition and Retry-After reference.
• Bingbot guidance — current Bing guidance for the crawl-delay directive.

Test yourself: 429 Too Many Requests

Five quick questions on how 429 works and how Google treats it. Pick an answer for each, then check.

429 Too Many Requests

429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content.

Related: HTTP Status Code, Crawl Budget

429 Too Many Requests

429 Too Many Requests is an HTTP status code that a server returns to tell a client — a browser, a script, or a search-engine crawler like Googlebot — that it has sent too many requests in too short a time. That mechanism is commonly called rate limiting. The response often carries a Retry-After header saying how long to wait before trying again.

Although 429 lives in the 4xx “client error” range, Google treats it very differently from its neighbors. Codes like 403 and 404 either have no effect on crawl rate or actively cause pages to drop from the index. 429 is the one exception: Google groups it with 5xx server errors, reads it as “the server is overloaded, slow down,” and — once it encounters a significant number of these errors — throttles Googlebot’s crawl rate across the whole hostname rather than removing content.

That makes 429 the one 4xx code Google actually endorses for slowing crawlers down. The catch is that it’s a short-term signal: Google’s own docs give emergency-use guidance of a couple of hours or 1–2 days, not a guaranteed safe window. Sustained 429s on the same URLs for multiple days can eventually get those URLs dropped from the index — the same dynamic as a prolonged 503. Once the errors stop, Google says the crawl rate starts increasing again, though not necessarily instantly or on a fixed timeline.

Related: HTTP Status Code, Crawl Budget

Changelog