HTTP Error Codes

TL;DR — HTTP error codesThe 4xx (client error) and 5xx (server error) HTTP status codes a server returns instead of a successful 2xx — and how Google treats each class very differently. are the “something went wrong” responses your server sends instead of a normal “200 OKHTTP 200 OK is the standard 2xx success status code, meaning the server received, understood, and fulfilled the request and is returning the resource. It's the code every page you want indexed should return — but a 200 alone doesn't guarantee Google will index the page..” They come in two families: 4xx (the page is missing or blocked — a client problem) and 5xx (the server itself is struggling). Google treats them very differently: a 4xx just drops the page from search; a 5xx makes Google slow down crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. your whole site until the server recovers. And despite the panic they cause, 404 errors404 Not Found is the HTTP client-error status code a server returns when it can't find the requested URL — RFC 9110 defines it as no current representation, or unwillingness to disclose one. A \"hard 404\" actually returns the 404 status; a \"soft 404\" returns a success code (like 200) for a page that's really gone. 404s are normal and expected: the fact that some URLs 404 doesn't affect your site's other, successful pages, and Google de-indexes 404'd URLs over time (probably retrying for some period, less and less often). are not a penalty — you only need to fix the ones that matter.

What HTTP error codes are

HTTP status codesAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. are standardized response metadata grouped into client-error 4xx and server-error 5xx classes. Evidence for this claim Primary standard or official documentation supporting the adjacent article claim. Scope: Protocol semantics and Search behavior are kept separate; no indexing, ranking, or migration-timing guarantee is inferred. Confidence: high · Verified: 2026-07-14 RFC 9110: Status codes Google documents distinct crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. handling for those response families. Evidence for this claim Primary standard or official documentation supporting the adjacent article claim. Scope: Protocol semantics and Search behavior are kept separate; no indexing, ranking, or migration-timing guarantee is inferred. Confidence: high · Verified: 2026-07-14 Google: HTTP and network errors

Every time a browser or a search engine botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. requests a page, your server answers with a three-digit status code. 200 means “here’s the page, all good.” The error codes are the ones in the 400s and 500s — they mean the request didn’t succeed.

There are two families, and the difference between them is the single most important thing on this page:

• 4xx — client errors. The problem is with the request: the page is gone (404), it’s been permanently removed (410), or you’re not allowed in (401, 403). To a search engine, a 4xx basically says “there’s no content here.”
• 5xx — server errors. The problem is on your end: the server crashed (500), a system it depends on failed (502, 504), or it’s overloaded or down for maintenance (503). To a search engine, a 5xx says “the server itself is broken right now.”

How Google reacts to each family

This is where the two families split:

• A 4xx page just gets dropped from search. Google decides the content doesn’t exist and removes the URL from its index over time. Importantly, a bunch of 404s does not slow down how Google crawls the rest of your site.
• A 5xx makes Google crawl your whole site more slowly. When the server is erroring, Google eases off to avoid making things worse. Your already-indexed pages stay in the index at first — Google only starts dropping them if the errors keep going. Once your server is healthy again, crawling speeds back up on its own.

The thing most people get wrong

A pile of 404 errors is not a penalty. Google’s John Mueller has said flat out that 404s are “not a quality signal, it’s not a SEO signal.” Broken and missing pages are a normal part of the web. So when you open Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. and see thousands of “Not foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore.” errors, don’t panic — you only need to fix the ones that actually matter: pages people still link to, pages in your sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., pages that were getting traffic. The genuinely dead ones can just stay 404.

The one pattern that does have a real, documented downside is a server that keeps throwing 5xx errors — that’s the case where Google throttles crawling and eventually drops pages. Even then it’s reversible: fix the server and it recovers.

A quick tour of the codes

Each of these has its own deep-dive page — here’s the one-line version:

• 401 / 403 — the page is blocked (you need to log in, or you’re forbidden).
• 404 — page not found. 410 — page gone for good.
• 429 — “too many requests” (rate limiting). Looks like a client error but Google treats it like a server error.
• 451 — blocked for legal reasons (a takedown, a country-level block).
• 500 / 502 / 503 / 504 — the server-error family (crash, bad gateway, unavailable, gateway timeout).
• Soft 404A soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing. — the sneaky one: a page that says “not found” but returns a 200 code, so Google keeps checking it.

Want the mechanics — exactly how Google’s crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. reacts, when pages get deindexed, how to take a site down the right way, and how to monitor all of this in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance.? Switch to the Advanced tab.

TL;DR — HTTP error codesThe 4xx (client error) and 5xx (server error) HTTP status codes a server returns instead of a successful 2xx — and how Google treats each class very differently. are the 4xx (client) and 5xx (server) status classes. Google draws a hard behavioral line between them: 4xx (except 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content.) means “the content doesn’t exist” — the URL is dropped from the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., with “no effect on crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.”; 5xx (and 429) means “the server is failing” — Google throttles crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. proportionately, preserves indexed URLs at first, and only deindexes if the errors persist, then ramps crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. back up gradually after recovery. 429 is numerically a 4xx but Google calls it “a server error.” Content from any error response is ignored. 404s are not a quality signal (Mueller) — triage by links/traffic, don’t fix everything, and remember an isolated error’s real impact depends on the URL (robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. has special error handling that an ordinary page doesn’t). Soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing. (a 200 that reads as “not foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore.”) are flagged by Google as wasting crawl budgetThe number of URLs an engine will crawl in a timeframe. — mainly a large-site concern, not a guaranteed effect on every site. For planned downtime use 503 + Retry-After, kept to “a few days at most,” and never 503 the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.. Monitor the family through GSC’s Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.. This hub maps and routes to every individual code.

The one distinction that runs the whole topic

The protocol code describes the HTTP outcome; a Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. label describes how Google classified an observed fetch. Evidence for this claim Primary standard or official documentation supporting the adjacent article claim. Scope: Protocol semantics and Search behavior are kept separate; no indexing, ranking, or migration-timing guarantee is inferred. Confidence: high · Verified: 2026-07-14 RFC 9110: Status codes Do not infer a single root cause or exact removal time from the family alone. Evidence for this claim Primary standard or official documentation supporting the adjacent article claim. Scope: Protocol semantics and Search behavior are kept separate; no indexing, ranking, or migration-timing guarantee is inferred. Confidence: high · Verified: 2026-07-14 Google: HTTP and network errors

Four layers get flattened into “it’s a 404” whenever people talk about errors, and keeping them apart is what makes the rest of this page make sense:

1. Protocol semantics — what the status code means per the HTTP spec (a 404 means “not found,” full stop).
2. The observed fetch — what GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. actually got back on a specific request at a specific time, which can differ from what a browser sees.
3. Search processing — how Google’s indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. pipeline classifies and acts on that observed fetch (drops the URL, throttles the crawl, ignores the body).
4. Root cause — the actual reason on your server (a bad deploy, an overloaded database, a WAF rule), which the status code alone never tells you.

Knowing the family (4xx vs 5xx) tells you which of the first three layers you’re looking at. It never substitutes for layer 4 — you still have to go find out why.

Google’s own docs frame the split cleanly, and it’s worth internalizing before anything else: 4xx errors say “the content doesn’t exist”; 5xx errors say “the server itself is failing.” Those are two completely different problems, and Google’s indexing pipeline responds to them in two completely different ways.

A useful anchor first: even a success code isn’t a promise. Google says plainly that “for Google Search, an HTTP 2xx (success) status code doesn’t guarantee indexing.” Error codes are actually the more deterministic half of the picture — they tell Google, unambiguously, either “gone” or “broken.”

How Google treats 4xx errors

For client errors, Google’s behavior is uniform and blunt: “Google doesn’t use the content from URLs that return 4xx status codes,” and “Google crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. inform the next processing system that the content doesn’t exist.” Even if your 403 or 404 page renders a wall of real text, none of it gets indexed — the body of an error response is ignored.

Two consequences follow:

• Previously indexed URLs get dropped. Once a page reliably returns a 4xx, Google removes it from the index over time.
• There’s no crawl-rate penalty. This is the myth-buster: “The 4xx status codes, except 429, have no effect on crawl rate.” A mountain of 404s does not slow Google’s crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. of the rest of your site.

That last point kills a bad instinct: don’t try to throttle Googlebot with a 401 or 403. Google explicitly warns “don’t use 401 and 403 status codes for limiting the crawl rate.” An auth wall doesn’t slow the crawl — it just makes the content invisible.

How Google treats 5xx errors (and 429)

Server errors trigger the “the server is struggling” branch, and here Google is deliberately protective of your site:

• Crawl rate throttles first, proportionate to volume. “5xx and 429 server errors prompt Google’s crawlers to temporarily slow down with crawling,” and “the decrease in crawl rate is proportionate to the number of individual URLs that are returning a server error.” A handful of 500s barely registers; a site-wide outage cuts crawling hard.
• Indexed URLs are preserved… until the errors persist. “Already indexed URLs are preserved in the index, but eventually dropped,” and Google “removes from the index URLs that persistently return a server error.” A short blip doesn’t deindex you; a sustained failure does.
• Content from 5xx is ignored too. “Any content Google receives from URLs that return a 5xx status code is ignored.”
• Recovery is automatic but gradual. “Once the server starts responding with a 2xx status code, Google gradually increases the crawl rate for the site.” Google is fast to back off and cautious to ramp back up — there’s no manual “unlock,” you just fix the root cause and wait.

Why 429 lives in the server-error bucket

This is the single most-missed nuance in most guides. 429 Too Many Requests is numerically a 4xx, but Google treats it as a server signal: “Google’s crawlers treat the 429 status code as a signal that the server is overloaded, and it’s considered a server error.” So a WAF or rate limiter that starts firing 429s at Googlebot will throttle your crawl the same way a wave of 500s would — not drop individual pages the way a 404 does. When I break the codes down in my HTTP status codes guide at Ahrefs, I put 429 with the server errors for exactly this reason: it’s “a form of rate-limiting to protect the server,” and it makes Google slow down.

TIP Verify the live status before choosing the error-code fix

This example is a 503 server error. Other 4xx and 5xx statuses need their own diagnosis, but the first step is always to observe the actual response at scale.

Check and export an affected URL list with my free HTTP Status Checker Free

1. Test the affected URLs and group them by observed status family.
2. Separate intended 404 or 410 responses from accidental 5xx and rate-limit failures.
3. Fix the responsible layer, then rerun the same list and monitor Search Console recovery.

Tap to enlarge

The result contains one tested URL, zero successful responses, and one 5xx response. The URL returns HTTP 503 and is labeled 5xx server error.

The error-code family map

Here’s the fast triage view of the whole family. Each code below is its own deep dive nested under this hub (they’re in the sidebar too):

Blocked / access errors

• 401 Unauthorized401 Unauthorized is a client-error HTTP status code meaning the request lacks valid authentication credentials — the server wants you to log in before it serves the page. For SEO it means the content is gated: Googlebot never sends credentials, so a 401'd page can't be seen and won't be indexed. — the client hasn’t identified or verified itself when needed. Blocked to Googlebot behind an auth request.
• 403 ForbiddenAn HTTP 403 client-error status code meaning the server understood the request but refuses to fulfill it — access is denied, whether or not credentials are involved. For SEO, a 403 served to Googlebot on a page meant to be public and indexable usually points to a misconfigured rule; a persistent 403 keeps a page out of the index either way. — the client is known but doesn’t have access rights.

Not-found errors

• 404 Not Found404 Not Found is the HTTP client-error status code a server returns when it can't find the requested URL — RFC 9110 defines it as no current representation, or unwillingness to disclose one. A \"hard 404\" actually returns the 404 status; a \"soft 404\" returns a success code (like 200) for a page that's really gone. 404s are normal and expected: the fact that some URLs 404 doesn't affect your site's other, successful pages, and Google de-indexes 404'd URLs over time (probably retrying for some period, less and less often). — the requested resource isn’t found.
• 404 vs 410 — the practical difference between “not found” and “gone” (it’s smaller than people think).
• 410 GoneA 410 Gone status code tells search engines a page was intentionally and permanently removed, prompting slightly faster de-indexing than a standard 404. — like a 404, but it also says the resource won’t be back. Drops pages slightly faster.

The dual-identity code

• 429 Too Many Requests429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. — rate limiting; numerically 4xx but Google treats it as a server error for crawl-rate purposes.

Legal

• 451 Unavailable For Legal ReasonsAn HTTP 451 client-error status code that tells a browser or crawler a resource can't be served because of a legal demand — a court order, government censorship, DMCA takedown, or regulatory requirement — rather than a technical failure (404) or an access-permissions problem (403). It's the only status code built specifically for legal transparency, and RFC 7725 asks servers to explain who is demanding the block and under what authority. — blocked for a legal reason: country-level blocks, DMCA takedowns.

Server errors (5xx)

• 500 Internal Server ErrorA 500 Internal Server Error is a generic HTTP status code — RFC 9110 defines it as an unexpected condition that stopped the server from fulfilling the request, and nothing more. The request may have been fine — something broke server-side while generating the response. For SEO, an isolated 500 is typically retried, but persistent, site-wide 500s get Google's documented response: slower crawling and, if the errors don't clear, eventual removal from the index. — the server hit an issue it can’t handle.
• 502 Bad GatewayA 502 Bad Gateway is an HTTP server error a gateway or reverse proxy (a CDN, load balancer, or proxy like Nginx) returns when it gets an invalid or no response from the upstream origin server it was trying to reach. It signals a communication failure *between* servers, not necessarily that the origin itself is down. For SEO, Google treats 502 exactly like 500 and 503: crawling slows, and persistent errors eventually get pages dropped from the index. — a bad response from an upstream server.
• 503 Service Unavailable503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. — the server is overloaded or down for maintenance (the correct code for planned downtime).
• 504 Gateway TimeoutA 504 Gateway Timeout is an HTTP 5xx server error a gateway or proxy (a CDN, load balancer, or reverse proxy) returns when it doesn't get a timely response from the upstream server behind it. Unlike a 502 (a bad/garbled response) or a 503 (server explicitly unavailable), a 504 means no response arrived in time. For SEO, isolated 504s are retried and tolerated, but sustained 504s and timeouts make Googlebot slow its crawl and eventually drop pages from the index. — no timely response from an upstream server.

The trap case

• Soft 404A soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing. — a page that returns 200 OK but reads as “not found.” The worst of both worlds — more on this below.

Broken redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. are error-adjacent but categorized separately (Google surfaces them as “Redirect errorA Google Search Console Page Indexing status meaning Googlebot couldn't follow a redirect — a chain too long, a loop, a URL over the max length, or a bad/empty URL in the chain. It's a broken redirect to fix, not the normal \"Page with redirect.\"” in Search Console, distinct from a normal, working “Page with redirectA Google Search Console Page Indexing status for a URL that redirects elsewhere. It's not indexed by design because it's a redirect — the destination is a separate question, and Google says the target may or may not end up indexed — usually expected, not an error (unlike the separate \"Redirect error\").”). By default Google’s crawlers “follow up to 10 redirect hops” before giving up; a chain that’s too long, loops, or contains a bad URL turns into that Redirect-error status.

Does an HTTP error hurt your SEO?

Lead with the verdict: an isolated error on an ordinary page is almost never a problem, and 404s specifically are not a ranking or quality signal. Mueller has been explicit and repeated about this. The reflex — “I have 50,000 404s, my site must be penalized” — is the number-one myth to defuse. Errors are a normal part of the web; having pages 404 or 410 is the technically correct way to handle URLs that don’t exist.

“Isolated” doesn’t mean “always harmless,” though — the real impact depends on what’s returning the error. An ordinary page 404ing is a non-event; a special resource is different. Google gives robots.txt its own error-handling rules distinct from regular URLs, so a server error on robots.txt can affect crawling in a way an ordinary page’s 404 never would. Judge an error by what URL it’s on and what depends on it, not by the status code alone.

The one pattern with a real, documented mechanism is persistent, mass 5xx: crawl-rate throttle → eventual deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist.. And even that is proportionate to how many URLs are erroring and generally reversible once the server recovers — Google describes the ramp-back-up as gradual, not instant, so treat the exact timing, retry cadence, and recovery speed as evidence-dependent (what Google’s docs describe happening) rather than a fixed guarantee. The distinction Mueller draws is the one to remember: crawl rate reacts to server errors (429/500/503/ timeouts), not to 404s.

Crawl waste vs deindexing — two different harms

It helps to separate the two ways errors can cost you:

• Deindexing — persistent 4xx (page dropped as “gone”) or persistent 5xx (pages dropped after sustained server failure). This is about pages leaving the index.
• Crawl-budget waste — mostly a large-site concern. Google’s crawl-budget guide notes “if the site slows down or responds with server errors, the limit goes down and Google crawls less.” But the real budget-waster is the soft 404: “soft 404 pages will continue to be crawled, and waste your budget.” Because a soft 404 looks alive (it returns 200), Google keeps re-checking a page that isn’t really there. That’s why the soft 404 is the “worst of both” case — it doesn’t cleanly drop like a true 404, it lingers and burns fetches.

The soft-404 trap has two common causes, both worth naming: a custom “page not found” template that returns 200 instead of a real 404, and a blanket redirect of every dead URL to the homepage — Google recognizes that pattern as a soft 404 too. A friendly 404 page is good for UX and fully recommended — as long as it still returns the actual HTTP 404 status code.

Doing planned downtime the right way

When you intentionally take a site (or a section) offline, the correct code is 503 Service Unavailable with a Retry-After header — never a 4xx. Google’s “Pause your online business” guidance is specific:

• “If you need to urgently disable the site for 1-2 days, then return an informational error page with a 503 HTTP response status code.”
• “This is an extreme measure that should only be taken for a very short period of time (a few days at most),” because “completely closing a site even for just a few weeks can have negative consequences on Google’s indexing of your site.”
• “Don’t block the website by returning 403, 404, 410 HTTP status codesAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index.” during downtime — a 4xx says “permanently gone,” which is exactly the wrong signal for a temporary outage.
• The trap almost everyone misses: “Don’t return a 503 HTTP response status code for the robots.txt file because this blocks all crawling.” 503 your pages, not your robots.txt.

How to monitor the whole family in Search Console

Day to day, you’ll meet these errors in Search Console’s Page Indexing report, where each maps to a distinct status:

• Not found (404) — “this page returned a 404 error when requested.”
• Server error (5xx) — “your server returned a 500-level error when the page was requested.”
• Blocked due to unauthorized request (401) — “the page was blocked to Googlebot by a request for authorization.”
• Blocked due to access forbidden (403) — a 403 where credentials were provided but access wasn’t granted.
• Blocked due to other 4xx issueA Google Search Console Page Indexing status: Googlebot got a 4xx client error not covered by another issue type (e.g. 400, 405, 410, 429, 451) when crawling the URL, so the page can't be indexed. — a 4xx not covered by another status; use URL Inspection to debug.
• Soft 404 — a “user-friendly ‘not found’ message but not a 404 HTTP response code.”
• Redirect error — chain too long, a loop, an over-long URL, or a bad URL in the chain.

Each status points at a different root cause and fix path. Once you’ve resolved one, use Validate Fix to prompt a recrawl — but set realistic expectations on timing; recrawl isn’t instant. And remember Google’s own framing: “it’s fine for a URL not to be indexed for the right reasons — for example… a 404 for a page that you’ve removed and have no replacement for.” Not every error is a to-do.

Search Console alone isn’t enough to act on — it’s one vantage point, sampled and delayed. When you log an error for triage, record at minimum: the URL, when you observed it, the vantage point (Search Console vs. a live check vs. server/CDN logs), the user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. the request came in on, the HTTP method, the final status code and response path (including any redirect chainA → B → C instead of A → C. Each hop loses link equity and adds latency.), whether it’s a one-off or recurring, and a post-fix verification step once you’ve redeployed. Triangulating GSC against a live status check and your own logs is what turns a stale report row into a confirmed, fixable problem.

How to fix and prioritize errors

• Triage 404s by value. Fix the ones with inbound links, internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. presence, or lingering traffic — 301-redirect those to a relevant page to recover the link equity. Let genuinely dead URLs 404 or 410. As I put it in my Ahrefs guide, the practical fix for most of these is that “you just need to 301 redirectA 301 redirect is the HTTP status code for a permanent move: it tells browsers and search engines a URL has moved for good, and it's the strongest signal for consolidating a page's ranking signals onto the new URL. Google says permanent redirects don't cause a loss in PageRank. each of these pages to a relevant page” — but only where a relevant target exists. Don’t blanket-redirect everything to the homepage (that’s a soft 404).
• 410 vs 404 is marginal. A 410 drops a page slightly faster than a 404; the practical SEO difference is minor. Use 410 when you want to be explicit that something is gone for good, but don’t expect it to be dramatically better.
• Root-cause 5xx. The fixes are on the server side: capacity and timeouts for 500s, upstream/CDN health for 502/504, and WAF or rate-limiting rules misfiring on Googlebot for 403/429. Confirm the real bot with a reverse/forward DNS check before you go rate-limiting it.
• Common root causes worth naming: app and database errors and overloaded hosts (5xx), upstream/CDN failures (502/504), rate limiting or bot-blocking WAFs (403/429), broken migrations and stale internal links (404), and misconfigured “friendly error pages” (soft 404).

Bing: a similar pattern, not independently verified code-for-code

Bing’s public statements point in the same direction as Google’s approach — 400-range codes are treated as missing or forbidden, and 500-range codes signal server trouble that works against crawl efficiency — but I haven’t independently verified full, current, code-by-code parity against Bing’s own documentation, so treat this as directional rather than a confirmed one-to-one match. Fabrice Canel frames Bing’s goal as a “crawl efficiency north star … to crawl a URL only when the content has been added … updated,” and persistent errors work directly against that — Bing spends crawl footprint on URLs that aren’t yielding fresh, indexable content. Bing also recommends a 503 with Retry-After for planned downtime rather than serving error pages as 200. You’ll find Bing’s crawl-error surfaces in Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. (URL Inspection, Crawl Control, Site Scan).

Where to go next

This page is the conceptual hub for the error-code family. It sits inside the broader HTTP Status CodesAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. cluster (the full 1xx–5xx picture, plus redirects and success codes); this sub-hub is the map for the error half of that. Each code below is its own deep dive:

Blocked / access

• 401 Unauthorized — what triggers the “Blocked due to unauthorized request” status, and why auth walls don’t throttle Googlebot.
• 403 Forbidden — credentials-provided-but-denied, and the WAF/bot-blocking patterns that cause false 403s to Googlebot.

Not found

• 404 Not Found — how Google handles missing pages, why it’s not a penalty, and which 404s to actually fix.
• 404 vs 410 — the real, small difference, and when to reach for each.
• 410 Gone — the “permanently gone” signal and its slightly faster drop.

Rate limiting

• 429 Too Many Requests — the 4xx that behaves like a 5xx, and how to keep rate limits from throttling your crawl.

Legal

• 451 Unavailable For Legal Reasons — takedowns, country blocks, and how legal removals show up.

Server errors

• 500 Internal Server Error — the generic server failure and how to root-cause it.
• 502 Bad Gateway — upstream/proxy failures.
• 503 Service Unavailable — the correct code for maintenance and planned downtime (with the robots.txt trap).
• 504 Gateway Timeout — upstream timeouts.

The trap case

• Soft 404 — the 200-that-reads-as-gone, why it wastes crawl budgetThe number of URLs an engine will crawl in a timeframe., and how to turn it into a real 404.

Broken redirects are handled separately as the Redirect error status — related but categorized on its own in Search Console.

AI summary

A condensed take on the Advanced version:

• Two families, two behaviors. 4xx = “content doesn’t exist”; 5xx = “the server is failing.” Google’s indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. pipeline responds to each very differently.
• 4xx (except 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content.): content is ignored, the URL drops from the index, and there’s “no effect on crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor..” Don’t use 401/403 to throttle GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. — Google explicitly says not to.
• 5xx (and 429): Google throttles crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. first (proportionate to how many URLs error), preserves indexed URLs at first, and deindexes only if errors persist. Recovery is automatic but the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. ramps back up gradually.
• 429 is the odd one out: numerically 4xx, but Google treats it as “a server error” for crawl-rate purposes — it behaves like a 5xx.
• 404s are not a quality/ranking signal (Mueller). Don’t panic over an error count; triage by links, sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. presence, and traffic — 301 the ones that matter, let dead pages 404/410.
• An isolated error’s impact depends on the URL, not just the status code — an ordinary page 404ing is a non-event, but special resources like robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. have their own error handling and can affect crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. differently.
• Soft 404A soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing. is the trap: a 200 that reads as “not foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore.,” which Google flags as wasting crawl budgetThe number of URLs an engine will crawl in a timeframe. because it keeps re-checking a page that looks alive — mainly a large-site concern, not a guaranteed effect on every site.
• 410 vs 404 is marginal — 410 drops slightly faster; the practical difference is minor.
• Planned downtime: use 503 + Retry-After, “a few days at most,” never 403/404/410, and never 503 your robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. (it blocks all crawling).
• Monitor via GSC’s Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. — seven statuses (404, 5xx, 401, 403, other 4xx, soft 404, redirect errorA Google Search Console Page Indexing status meaning Googlebot couldn't follow a redirect — a chain too long, a loop, a URL over the max length, or a bad/empty URL in the chain. It's a broken redirect to fix, not the normal \"Page with redirect.\") — plus your own live checks and server logs, and use Validate Fix.
• Bing’s public statements point the same direction as Google’s — but full code-by-code parity isn’t independently verified; Bing also recommends 503 + Retry-After for downtime.

Official documentation

Primary-source documentation from the search engines.

Google

• HTTP status codes, network errors, and DNS errors — the definitive doc on how Google handles 4xx, 5xx, 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content., redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., and network/DNS errors.
• Page Indexing report (Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. Help) — every error status you’ll see day to day (404, 5xx, 401, 403, other 4xx, soft 404A soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing., redirect errorA Google Search Console Page Indexing status meaning Googlebot couldn't follow a redirect — a chain too long, a loop, a URL over the max length, or a bad/empty URL in the chain. It's a broken redirect to fix, not the normal \"Page with redirect.\") and Validate Fix.
• Optimize your crawl budget — how server errors reduce the crawl limit and why soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing. waste budget.
• Pause your online business in Google Search — the 503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. + Retry-After playbook for planned downtime, and what not to do.
• Soft 404 errors — what a soft 404 is and how to fix it.
• Make your 404 pages more useful — older but still-standing guidance: a friendly 404 page404 Not Found is the HTTP client-error status code a server returns when it can't find the requested URL — RFC 9110 defines it as no current representation, or unwillingness to disclose one. A \"hard 404\" actually returns the 404 status; a \"soft 404\" returns a success code (like 200) for a page that's really gone. 404s are normal and expected: the fact that some URLs 404 doesn't affect your site's other, successful pages, and Google de-indexes 404'd URLs over time (probably retrying for some period, less and less often). is fine as long as it returns a real 404 code.

Bing / Microsoft

• Crawl error alerts (Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. Help) — where Bing surfaces crawl errors.
• bingbot Series: Maximizing Crawl Efficiency — Bing’s “crawl efficiency north star,” which persistent errors work against.

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — the 4xx vs 5xx split

• “For Google Search, an HTTP 2xx (success) status code doesn’t guarantee indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..” — Google Search Central docs. Jump to quote
• “Google doesn’t use the content from URLs that return 4xx status codes.” Jump to quote
• “The 4xx status codes, except 429, have no effect on crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor..” Jump to quote

Google — 5xx and 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content.

• “5xx and 429 server errors prompt Google’s crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. to temporarily slow down with crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..” Jump to quote
• “already indexed URLs are preserved in the index, but eventually dropped.” Jump to quote
• “Any content Google receives from URLs that return a 5xx status code is ignored.” Jump to quote
• “Google’s crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. treat the 429 status code as a signal that the server is overloaded, and it’s considered a server error.” Jump to quote

Google — crawl budgetThe number of URLs an engine will crawl in a timeframe. and soft 404A soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing.

• “If the site slows down or responds with server errors, the limit goes down and Google crawls less.” Jump to quote
• “Soft 404 pages404 Not Found is the HTTP client-error status code a server returns when it can't find the requested URL — RFC 9110 defines it as no current representation, or unwillingness to disclose one. A \"hard 404\" actually returns the 404 status; a \"soft 404\" returns a success code (like 200) for a page that's really gone. 404s are normal and expected: the fact that some URLs 404 doesn't affect your site's other, successful pages, and Google de-indexes 404'd URLs over time (probably retrying for some period, less and less often). will continue to be crawled, and waste your budget.” Jump to quote

Google — planned downtime

• “If you need to urgently disable the site for 1-2 days, then return an informational error page with a 503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. HTTP response status code.” Jump to quote
• “Don’t return a 503 HTTP response status code for the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file because this blocks all crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..” Jump to quote

Google — it’s fine not to be indexed

• “It’s fine for a URL not to be indexed for the right reasons — for example, an expected robots.txt rule on your site, a noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. tag on the page, a duplicate URL, or a 404 for a page that you’ve removed and have no replacement for.” — Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. Help. Jump to quote

John Mueller, Google — 404s are not a quality signal

• “404s are not a quality signal, it’s not a SEO signal.” Read the coverage
• On crawl-rate reactions: “I’d only expect the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. to react that quickly if they were returning 429 / 500 / 503 / timeouts… (404s are generally fine & once discovered, GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. will retry them anyway)… Once things settle down on the server, the crawl rate will return to normal automatically.” Read the coverage

Fabrice Canel, Microsoft Bing

• “Our crawl efficiency north star is to crawl a URL only when the content has been added (URL not crawled before), updated (fresh on-page context or useful outbound links).” Jump to quote

Which error do I deal with first?

When a batch of errors shows up, work the triage in this order — server health before missing pages, because that’s where the real, site-wide damage lives.

1. Are you seeing 5xx or 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. (Server error, or a spike of timeouts)?

• Yes → Treat it as urgent. This is the branch that throttles crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. site-wide and can deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. pages if it persists. Root-cause it now: server capacity/timeouts (500), upstream/CDN health (502/504A 504 Gateway Timeout is an HTTP 5xx server error a gateway or proxy (a CDN, load balancer, or reverse proxy) returns when it doesn't get a timely response from the upstream server behind it. Unlike a 502 (a bad/garbled response) or a 503 (server explicitly unavailable), a 504 means no response arrived in time. For SEO, isolated 504s are retried and tolerated, but sustained 504s and timeouts make Googlebot slow its crawl and eventually drop pages from the index.), or a WAF / rate limiter misfiring on GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. (403/429). Recovery is automatic once you return 2xx, but crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. ramps back up gradually.
• No → Move on; 4xx has no crawl-rate effect.

2. Is it a soft 404A soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing. (a 200 that reads as “not foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore.”)?

• Yes → Fix the response code. Either return a real 404/410, or if the page moved, 301 it to a relevant page. Stop blanket-redirecting dead URLs to the homepage. Soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing. waste crawl budgetThe number of URLs an engine will crawl in a timeframe. because Google keeps re-checking them.
• No → Continue.

3. Is it a 404 / 410 (Not found / Gone)?

• Does the URL have inbound links, internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. presence, or lingering traffic?
  • Yes → 301-redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. it to a relevant page to recover the equity.
  • No → Leave it. A genuinely dead page should just 404/410 — it’s not a penalty, and you don’t need to fix it.

4. Is it a 401 / 403 (Blocked)?

• Should this page be public? Yes → Fix the auth/permission config or the WAF rule that’s blocking GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.. No → It’s fine to be blocked — but don’t expect the 401/403 to throttle Googlebot; it only makes the content invisible.

5. Is it a 451 (Legal)?

• That’s a legal takedown or geoGenerative Engine Optimization — visibility inside AI answer engines.-block, not a technical bug — handle it through the legal/compliance process, not the SEO one.

6. Is it a Redirect errorA Google Search Console Page Indexing status meaning Googlebot couldn't follow a redirect — a chain too long, a loop, a URL over the max length, or a bad/empty URL in the chain. It's a broken redirect to fix, not the normal \"Page with redirect.\"?

• Shorten the chain (Google follows up to ~10 hops), break the loop, or fix the bad/over-long URL in the chain so it resolves to a single final 200.

Deindexing or crawl waste — which harm am I actually facing?

• Pages leaving the index? → Look for persistent 4xx (dropped as “gone”) or persistent 5xx (dropped after sustained failure). Isolated errors don’t do this.
• Crawl budget being burned (large sites)? → Look for soft 404s and mass errors — the pages that keep getting re-crawled because they look alive.

Error-code → behavior → fix cheat sheet

How Google treats each code

Fast facts

• 4xx (except 429): content ignored, URL dropped, zero crawl-rate effect.
• 5xx + 429: throttle first, deindex only if persistent, recovery is gradual.
• 404s are not a quality/ranking signal — triage by links/traffic.
• Soft 404 = the trap: 200 that reads as gone → wastes crawl budgetThe number of URLs an engine will crawl in a timeframe..
• Planned downtime = 503 + Retry-After, “a few days at most,” never 403/404/410 — and never 503 the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. (blocks all crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.).
• RedirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't.: Google follows up to ~10 hops; over-long/looping chains = “Redirect errorA Google Search Console Page Indexing status meaning Googlebot couldn't follow a redirect — a chain too long, a loop, a URL over the max length, or a bad/empty URL in the chain. It's a broken redirect to fix, not the normal \"Page with redirect.\"” in GSC.

GSC Page IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. statuses to watch Not found (404) · Server error (5xx) · Blocked due to unauthorized request (401) · Blocked due to access forbidden (403) · Blocked due to other 4xx issueA Google Search Console Page Indexing status: Googlebot got a 4xx client error not covered by another issue type (e.g. 400, 405, 410, 429, 451) when crawling the URL, so the page can't be indexed. · Soft 404 · Redirect error — then Validate Fix once resolved.

Resources worth your time

My related writing

• HTTP Status Codes & Their SEO Impact — my full code-by-code breakdown: 4xx, 5xx, 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content., and the practical fixes, across the whole status-code range.
• 11 Types Of Redirects & Their SEO Impact — the redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. side, which is a leading cause of both redirect errorsA Google Search Console Page Indexing status meaning Googlebot couldn't follow a redirect — a chain too long, a loop, a URL over the max length, or a bad/empty URL in the chain. It's a broken redirect to fix, not the normal \"Page with redirect.\" and soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing..
• The Beginner’s Guide to Technical SEO — where status codes fit in the bigger picture.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and serving, including how status codes gate the pipeline. (My standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From around the industry

• Googlebot Crawl Slump? Mueller Points To Server Errors (Search Engine Journal) — the write-up anchoring the “crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. reacts to 429/500/503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime./timeouts, not 404s” distinction, in Mueller’s own words.
• The ultimate guide to HTTP status codes for SEO (Search Engine Land) — a broad reference across every status class.
• HTTP Status Codes Explained (Semrush) — implementation-focused overview of the code families.
• An SEO Guide to HTTP Status Codes (Search Engine Journal) — a code-by-code reference with SEO context.
• r/TechSEO — the community for crawl-error and indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. debugging.

The two-harm model: disappearance vs. crawl throttling

Do not prioritize an HTTP error from the first digit alone. Ask which of two harms it creates.

Harm 1: the URL says the content does not exist

Most 4xx responses tell Google the requested content is unavailable. Google does not indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. the returned body, and a persistently affected URL can leave the index. The scope is usually the URL itself; ordinary 4xx responses do not tell Google that the whole server is unhealthy. The exception is 429, which Google handles like a server availability problem.

Decision question: Is this URL meant to exist, receive traffic, or preserve links? If yes, fix the erroneous status or send users to a genuinely equivalent destination. If no, a 404 or 410 may be the correct outcome rather than an SEO issue.

Harm 2: the server says it cannot reliably respond

5xx responses and 429 signal availability or capacity trouble. Google can reduce crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. to avoid adding load; only sustained failures threaten indexed URLs. The harm can therefore spread beyond one failing URL into slower discovery and refresh across the site.

Decision question: Is the failure persistent or broad enough to change crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. behavior? If yes, treat it as an infrastructure incident and restore reliable responses. Do not “fix” it inside Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results..

Priority overlay: value and intent

After choosing the harm, rank affected URLs by whether they should be live, have internal or external links, appear in a sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., receive search traffic, or represent a critical template. An error count without this context is not a priority list.

Evidence/action matrix: what to record per error

The two-harm model tells you which problem you’re facing. This matrix is what to actually record for each affected URL before you decide what to do with it — skipping any of these columns is how a triage list turns into a guess:

Prompt: turn a GSC error export into a triage queue

Export the affected URL list from Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. and enrich it where possible with status code, sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. presence, organic clicks/impressions, internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., external links, and intended URL state. Paste only non-sensitive data.

You are triaging an HTTP-error URL export for technical SEO. Use only the supplied
columns. Do not assume that every 404 is a problem or that every 5xx is permanent.

For each URL:
1. Classify the response as 4xx disappearance risk, 5xx/429 crawl-throttling risk,
   soft-404 mismatch, or insufficient evidence.
2. Decide whether the URL appears intended to exist, based only on sitemap presence,
   links, traffic, template, and the supplied intended-state field.
3. Assign priority: incident-now, high, medium, low, or expected/no fix.
4. Explain the priority using the exact supplied signals.
5. Recommend the next verification, not a guessed root cause.

Prioritize broad or persistent 5xx/429 incidents first, then intended URLs with
traffic or links, then sitemap/internal-link hygiene. Keep intentional 404/410 URLs
out of the fix queue. Return a table plus a short list of missing data that would
change the ranking.

DATA:
[PASTE SANITIZED EXPORT HERE]

Verify the output against live status checks and the site’s intended URL map before assigning work.

Tools for error triage

• Bulk HTTP Status Code Checker — paste the exported URLs to collect live status codes, redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency., destinations, and latency, then filter and export the result. This converts a stale report row into current HTTP evidence.
• Website Down Checker — check whether a suspected 5xx incident is externally reproducible before changing application or crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. settings.
• Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. — identify Google’s reason groups and affected examples; use URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. for a representative live check.
• Server or CDN access logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. — measure crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-facing error rates and distinguish a single URL problem from a site-wide availability pattern.

Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. reports what Google observed; it does not replace a live request or the server evidence needed to find the cause.

Intended URLs returning errors

Metric: Count and share of URLs that are supposed to be live but return 4xx, 5xx, or 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content., segmented by response family and template.

What it tells you: Separates real availability/indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. defects from intentional 404/410 cleanup.

How to pull it: Join a current crawl or Bulk HTTP Status CodeAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. Checker export with your sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., internal-link graph, and intended URL inventory.

Benchmark / realistic range: The target is zero unintended errors on URLs that must be live. Expected retired URLs should be tracked separately, not forced into a made-up site-wide error-rate target.

Cadence: Daily for critical templates and during releases; weekly for the broader inventory.

Crawler request error rate

Metric: Search-engine requests returning 4xx, 429, or 5xx as a share of verified crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. requests, split by status, hostname, and path group.

What it tells you: Shows whether crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. waste is concentrated in retired URLs or whether server failures may be suppressing crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side..

How to pull it: Aggregate verified crawler requests from CDN or server access logs; do not trust the user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string alone when verification is available.

Benchmark / realistic range: Establish the normal mix for your site. Intentional 404s may be legitimate; unexplained 5xx/429 responses on intended URLs should not be treated as normal baseline noise.

Cadence: Monitor continuously for 5xx/429 alerts; review segmented trends weekly.

Search Console error inventory trend

Metric: Affected URL counts by Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. reason, paired with representative URL state and report update date.

What it tells you: Whether Google’s observed inventory is shrinking after fixes or expanding into additional templates.

How to pull it: Export the Page IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. reason tables and retain dated snapshots; spot-check examples with URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. and live requests.

Benchmark / realistic range: Compare with the site’s intended state and its own prior snapshots. Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. is sampled and delayed, so do not invent a real-time pass threshold.

Cadence: Weekly and after Google has had time to recrawl a deployed fix; use logs and live checks for immediate incident monitoring.

Test yourself: HTTP Error Codes

Five quick questions on how Google handles 4xx and 5xx errors. Pick an answer for each, then check.

HTTP Error Code

The 4xx (client error) and 5xx (server error) HTTP status codes a server returns instead of a successful 2xx — and how Google treats each class very differently.

Related: 404 Not Found, 410 Gone, 429 Too Many Requests, 500 Internal Server Error, 503 Service Unavailable, Soft 404, HTTP Status Code

HTTP Error Code

An HTTP error code is any response in the 4xx (client error) or 5xx (server error) class that a server returns instead of a successful 2xx response. In SEO terms they matter because GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.’s reaction to an error code decides whether a URL gets indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., stays indexed, or gets dropped — and, for 5xx and 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content., how fast Google keeps crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. the whole site.

The fundamental split: 4xx means “the content doesn’t exist” and 5xx means “the server itself is failing.” Google treats those two situations very differently. 4xx codes (except 429) tell Google the content is gone, so the URL drops from the index — but they have no effect on crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.. 5xx codes (and 429) tell Google the server is struggling, so Google throttles crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. site-wide first and only drops the pages if the errors persist.

429 (Too Many Requests) is the odd one out: numerically it’s a 4xx, but Google explicitly treats it as a server error for crawl-rate purposes, so it behaves like a 5xx. A soft 404A soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing. is the trap case — a page that returns 200 OK but reads as “not foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore.,” which wastes crawl budgetThe number of URLs an engine will crawl in a timeframe. instead of cleanly dropping the URL.

You monitor the whole family through Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason., where each error surfaces as its own status: Not found (404), Server error (5xx), Blocked due to unauthorized request (401), Blocked due to access forbidden (403), Blocked due to other 4xx issueA Google Search Console Page Indexing status: Googlebot got a 4xx client error not covered by another issue type (e.g. 400, 405, 410, 429, 451) when crawling the URL, so the page can't be indexed., Soft 404, and Redirect errorA Google Search Console Page Indexing status meaning Googlebot couldn't follow a redirect — a chain too long, a loop, a URL over the max length, or a bad/empty URL in the chain. It's a broken redirect to fix, not the normal \"Page with redirect.\".

Related: 404 Not Found, 410 Gone, 429 Too Many Requests, 500 Internal Server Error, 503 Service Unavailable, Soft 404, HTTP Status Code

Changelog