Server Error (5xx)

TL;DR — “Server error (5xx)” means Google tried to load your page and your server answered with an error (a 500-level code) instead of showing the page. Google can’t indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. an error, so the URL won’t appear in search. A quick blip is fine — Google just backs off and retries — but if the errors keep happening, your pages can eventually drop out of Google. Fix the server, and crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. comes back on its own.

What this status means

This label means Google’s request received a server response in the 5xx range. Evidence for this claim Google reports Server error 5xx when the server returned a 500-level response for the requested page. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report Google slows crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. in response to 5xx errorsThe 4xx (client error) and 5xx (server error) HTTP status codes a server returns instead of a successful 2xx — and how Google treats each class very differently. and can eventually remove persistently failing URLs. Evidence for this claim Google slows crawling for 5xx responses and may eventually remove persistently failing URLs from the index. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: HTTP status codes

When GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. asks your server for a page, the server answers with a status code. A 200 means “here’s the page.” A 5xx is a whole family of error codes — 500, 502, 503, 504 — that all mean “something went wrong on my end.” When Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. shows Server error (5xx) under Not indexed, it’s telling you Google asked for that URL and got one of those errors back instead of the page.

Google can’t index an error page. There’s no real content to read, so the URL stays out of search until the server starts responding normally again.

Is this an emergency?

It depends on whether it’s a one-off or a pattern:

• A brief hiccup — your server was busy for a minute, a deploy restarted it — is normal. Google notices the error, slows down, and tries again later. No lasting harm.
• Errors that keep happening are the problem. The longer your server keeps returning 5xx, the more Google crawls you, until eventually pages that were in Google get dropped.

Google doesn’t publish a “safe” number of flagged URLs — the crawl-rate hit scales with however many URLs are erroring, not a fixed threshold. So prioritize by what actually matters: is it an important page, is it still failing when you check again, and is it a handful of URLs or a lot of them? A single important URL that keeps failing deserves attention now; a one-time blip on a low-value page usually resolves itself. A lot of URLs flagged, or the same ones flagged over and over, means something on your site is actually broken and needs attention.

How to start fixing it

1. Check if your site is actually up. Visit a flagged page yourself. If it errors for you too, that’s your answer — the server has a problem.
2. Ask your host or developer. 5xx errors almost always come from the server, the app, or the database — not from anything in your page’s content or SEO settings.
3. Look at Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. (Settings → Crawl stats). It shows whether Google has been hitting errors across your whole site recently.
4. Once it’s fixed, tell Google to re-check. In the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason., open the “Server error (5xx)” issue and click Validate Fix. Google re-crawls the affected URLs and clears them as they come back clean.

The one thing people get wrong

If you ever need to take your site down on purpose — for maintenance, say — don’t just show an error page or a “we’ll be back soon” page that returns a normal 200. And don’t make it a 404. The right move is a 503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. status (it literally means “service unavailable, temporarily”). That tells Google “I’m down for a bit, come back later” instead of “this page is broken” or “this page is gone.” There’s a right way to do that, covered in the Advanced tab.

Want the full version — exactly how Google reacts to 5xx, the 500-vs-502-vs-503 differences, how to diagnose the cause, and the proper maintenance-mode setup? Switch to the Advanced tab.

TL;DR — “Server error (5xx)” means the server returned a 500-level code when GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. requested the URL, so the page can’t be indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Two distinct harms: Google slows the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. (proportionate to how many URLs are erroring) and ignores any content a 5xx returns; if the errors persist, already-indexed URLs are preserved at first, then dropped. Recovery is automatic once you return 2xx, but the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. ramps back up gradually. Google buckets 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. (rate limiting / “server overloaded”) with 5xx. Causes: overloaded/misconfigured host, app/DB errors, upstream or CDN failures, a CDN/WAF blocking GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. (verify with reverse DNS/IP ranges before changing security rules — a Googlebot user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. is spoofable). Diagnose with Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). host availability → server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. → URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. live test. Google’s default recommendation for planned closures is to stay online with limited functionality; if you must fully disable, the correct response is 503 + Retry-After for a day or two (“a few days at most” — weeks of it harms indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. with no fixed recovery time) — and robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. must keep returning 200, because a 503 on robots.txt can pause crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. site-wide. Then run Validate Fix (optional — it just tracks the fix).

What the status actually reports

The label reports the response Google observed, not the specific origin, proxy, database, or CDN failure that caused it. Evidence for this claim Google reports Server error 5xx when the server returned a 500-level response for the requested page. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report Google’s HTTP guidance defines the separate crawl and indexing effects. Evidence for this claim Google slows crawling for 5xx responses and may eventually remove persistently failing URLs from the index. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: HTTP status codes

Google’s own definition is one line: your server returned a 500-level error when the page was requested. That’s it. Googlebot made a request, and instead of a 200 with content it got a 500, 502, 503, 504, or similar. The URL lands under Not indexed because there was no usable content to index.

Keep one thing straight: this is about the response code, not the page’s content, markup, or SEO setup. A 5xx is a server-and-infrastructure problem. No amount of editing the page’s HTML fixes a 502 coming from an overloaded backend.

How Google treats 5xx — the crawl economics

This is the part most explainers skip, and it’s the whole reason 5xx matters more than, say, a 404. Google’s HTTP-errors documentation spells out the behavior:

• Crawling slows down. 5xx (and 429) errors prompt Google’s crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. to temporarily slow down. The decrease in crawl rate is proportionate to the number of individual URLs returning a server error — a handful of erroring URLs is a nudge; a site-wide 5xx is a hard brake.
• The content is ignored. Anything Google receives from a URL returning a 5xx is thrown away. There’s no “partial credit” for an error body — Google does not index a “site down” message you served with a 500.
• Indexed URLs are preserved, then eventually dropped. Already-indexed URLs stay in the index at first. But Google’s indexing pipeline removes URLs that persistently return a server error. So a short outage costs you nothing in the index; a prolonged one costs you the pages.
• Recovery is automatic — but gradual. Once the server starts responding with 2xx again, Google gradually increases the crawl rate back up. You don’t file a ticket; you fix the server and the crawl ramps back cautiously on its own.

That sequence — slow down → ignore content → preserve → eventually drop → recover on 2xx — is the accuracy spine of this whole topic.

429 counts as a server error. Worth flagging because most competitor content misses it: Google treats a 429 Too Many Requests as a signal that the server is overloaded, and buckets it with 5xx. If your rate limiting or bot protection is firing 429s at Googlebot, you’re getting the same crawl slowdown as a 500.

The common 5xx codes, by likely root cause

Knowing which 5xx you’re getting is a starting point for where to look — not proof of what’s broken. RFC 9110 (the HTTP specification) defines each code by what the responding component was doing, and any of these can be emitted by an origin server, an application server, a load balancer, a CDN, or a proxy sitting in front of the real origin. Treat the code as the first clue, then correlate it against edge, origin, application, and database/dependency logs for the same request before you conclude which layer actually failed:

• 500 Internal Server ErrorA 500 Internal Server Error is a generic HTTP status code — RFC 9110 defines it as an unexpected condition that stopped the server from fulfilling the request, and nothing more. The request may have been fine — something broke server-side while generating the response. For SEO, an isolated 500 is typically retried, but persistent, site-wide 500s get Google's documented response: slower crawling and, if the errors don't clear, eventual removal from the index. — per the spec, “the server encountered an unexpected condition that prevented it from fulfilling the request.” That’s deliberately generic: it’s often application code or a runtime/config error, but the status code alone doesn’t prove that — check app logs to confirm.
• 502 Bad GatewayA 502 Bad Gateway is an HTTP server error a gateway or reverse proxy (a CDN, load balancer, or proxy like Nginx) returns when it gets an invalid or no response from the upstream origin server it was trying to reach. It signals a communication failure *between* servers, not necessarily that the origin itself is down. For SEO, Google treats 502 exactly like 500 and 503: crawling slows, and persistent errors eventually get pages dropped from the index. — “the server, while acting as a gateway or proxy, received an invalid response from an inbound server.” This points at an upstream interaction, not a single vendor: check the chain of load balancer, reverse proxy, CDN, and the origin behind them.
• 503 Service Unavailable503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. — “the server is currently unable to handle the request due to a temporary overload or scheduled maintenance.” Look at capacity, traffic spikes, and whether something is in a maintenance state. (This is also the code you want to send on purpose during planned downtime — see below.) Note the spec’s own caveat: an overloaded server isn’t required to return 503 at all — “some servers might simply refuse the connection,” which can show up as a timeout or connection error instead of a clean status code.
• 504 Gateway TimeoutA 504 Gateway Timeout is an HTTP 5xx server error a gateway or proxy (a CDN, load balancer, or reverse proxy) returns when it doesn't get a timely response from the upstream server behind it. Unlike a 502 (a bad/garbled response) or a 503 (server explicitly unavailable), a 504 means no response arrived in time. For SEO, isolated 504s are retried and tolerated, but sustained 504s and timeouts make Googlebot slow its crawl and eventually drop pages from the index. — “the server, while acting as a gateway or proxy, did not receive a timely response from an upstream server.” It marks a timeout boundary, not which upstream component was slow: check long database queries, slow third-party calls, and an origin struggling under load.

A practical thread runs through these: 5xx often traces back to an overloaded or slow backend, so server performance work — faster queries, lighter server-side renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., more capacity — is frequently part of the fix, not a side quest. But because the same status code can come from any layer in the chain, the code narrows your search; the logs tell you where the failure actually happened.

Common causes

• Overloaded host. Traffic spikes (including aggressive crawling) outrun your server’s capacity and it starts shedding requests as 5xx.
• Application or database errors. Unhandled exceptions, a downed DB, a bad deploy, exhausted connection pools.
• Misconfiguration. A broken config after a change, an expired dependency, a full disk.
• Upstream / CDN failures. Your origin is fine but a proxy, load balancer, or CDN in front of it is returning 502/504 — or the CDN itself had an outage.
• A CDN/WAF or rate limiter blocking Googlebot. Bot protection, security rules, or rate limiting that mistakes Googlebot for an attacker can return 5xx or 429 only to Googlebot while real users see the site fine. This one is under-covered and a frequent real-world cause — which is why you have to reproduce as Googlebot, not just check the page in your browser.

How to diagnose it

Work from the whole-site view down to the single URL:

1. Crawl Stats → Host availability. In Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance., Settings → Crawl stats. The host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. section and the by-response-code breakdown show whether 5xx is a persistent, large-scale issue or an isolated blip, and roughly when it started.
2. Server and access logs. The ground truth. Filter your logs to Googlebot (verified — see the Scripts tab) and look at the status codes it actually received and when. Logs will show you a CDN/WAF blocking Googlebot that a browser test never would.
3. URL Inspection → Live Test. Run a flagged URL through URL Inspection and use Test Live URL. This confirms whether Google-InspectionTool can access the page right now — useful, but it’s a current-state check, not proof of what happened at the earlier time Google logged the 5xx. A page that loads fine for you (in a browser, or via Live Test) minutes or hours later doesn’t rule out a real error at the moment Googlebot actually hit it — time, IP, geoGenerative Engine Optimization — visibility inside AI answer engines., cache state, and bot-detection rules can all differ between the two requests.
4. Reproduce as Googlebot. Request the URL with Googlebot’s user-agent (and, if you can, from outside your network) to catch WAF/rate-limit rules that only fire for the bot. Treat this as a differential test, not verified proof of what Googlebot itself saw — a Googlebot user-agent string is trivial to spoof in either direction. Before you loosen a WAF, CDN, or rate-limit rule because “it’s only blocking Googlebot,” confirm the traffic in your logs is real Googlebot via reverse DNS or Google’s published IP ranges (see the Scripts tab) — don’t change a security control based on the user-agent header alone.

TIP Separate an isolated 5xx from a crawl-wide failure pattern

A spot check tells you what a URL returns now; access logs show which responses verified crawlers actually received over time. Use both before calling the incident fixed.

Check the affected URLs with my HTTP Status Checker, then quantify the historical 5xx requests in my Log File Analyzer. Log File Analyzer Free

1. Test the exact URLs with the browser and Googlebot user-agent options.
2. Filter access logs to verified Googlebot and inspect the 5xx count, paths, and timing.
3. Retest after the server or edge fix and confirm the 5xx share falls.

Tap to enlarge

How to fix it

Once you know the cause, the fixes follow Google’s own “fixing server errors” guidance:

• Confirm the scale in Crawl Stats before you change anything — is this persistent and large, or a one-off?
• Reduce excessive page loading for dynamic requests. Cache expensive pages, optimize slow queries, and stop generating heavy responses on every hit.
• Make sure the host isn’t down, overloaded, or misconfigured. Add capacity, fix the config, restart the broken service, check the database.
• Make sure you’re not inadvertently blocking Google. Audit CDN/WAF/bot rules and rate limits for anything firing 5xx or 429 at Googlebot.
• Control crawling wisely. If aggressive crawling is overloading you, the answer is a temporary 503/429 to ease the bot off — not a permanent block.

The right way to take a site down on purpose — 503 + Retry-After

Sometimes you want the site unavailable: a migration, scheduled maintenance, pausing an online business. Doing it wrong turns a planned event into a deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. event. Google’s Pause your online business in Google Search doc is explicit about the correct approach.

Google’s default recommendation is to keep the site up, just limited. If the closure is temporary and you plan to reopen, Google’s own preference is that you “keep your site online and limit the functionality” rather than take it fully offline. Full disable-and-503 is the urgent option, not the default one.

When you do need the whole site down, use 503 (Service Unavailable) with a Retry-After header. Google: “If you need to urgently disable the site for 1-2 days, then return an informational error page with a 503 HTTP response status code.” And pair it with the header: “Use the retry-after HTTP header with a best effort date or duration.” The 503 says “temporarily down,” and Retry-After tells the bot roughly when to come back — but treat that header as advisory, not a guarantee: it’s a best-effort signal, not a promise Google will recrawl at that exact time.

It’s a short-term measure only. Google calls it “an extreme measure that should only be taken for a very short period of time (a few days at most).” And the warning that competitors leave out: “Completely closing a site even for just a few weeks can have negative consequences on Google’s indexing of your site.” Google is explicit that there’s no way to shortcut your way back from that either — “there’s no fixed time for a recovery from a complete removal, and there’s no mechanism to speed that up.” In other words, a correct 503 protects you for a day or two — but no status code, and no amount of validating in Search Console, saves you from the indexing damage of being down for weeks, or gives you a guaranteed comeback date. If you need a long outage, that’s a different conversation (and probably a redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. or a real plan).

Why not a 200 or a 404

• Don’t serve a 200 “under maintenance” page. Google will treat that page’s content as the real page and may index your “we’ll be back soon” message. The content from a proper 503, by contrast, is ignored — which is what you want.
• Don’t return a 404 (or 403/410). Google’s guidance is explicit: don’t block the site by returning 403, 404, or 410 during downtime. A 404 signals gone, not temporarily down; you want the page preserved, and a 503 does that.

The robots.txt trap — keep it crawlable

This is the single most-missed detail, and it can pause crawling for your entire site. During a 503 maintenance window, your robots.txt file must keep returning 200 and stay crawlable. Google is blunt: “Don’t return a 503 HTTP response status code for the robots.txt file because this blocks all crawling.” Carve robots.txt out of the maintenance handler so it always answers 200 — don’t rely on what happens if you don’t.

For context, Google’s robots.txt spec doc lays out what actually happens if robots.txt itself starts erroring, and it’s staged rather than an instant, permanent lockout: for the first 12 hours Google stops crawling the site while still retrying robots.txt; for the next 30 days it falls back to the last known good version of robots.txt while still trying to fetch a fresh one; and after 30 days, if the site is otherwise reachable, Google drops the cached version and behaves as if there’s no robots.txt file at all (i.e., no crawl restrictions from it) rather than continuing to block. If there’s no cached version to fall back on in the first place, Google likewise assumes there’s no crawl restriction. None of that is a reason to risk it on purpose — a multi-day crawl pause at the start of the window is still real damage — it just means “robots.txt 503’d” isn’t a permanent, unrecoverable state if it happens by accident and gets fixed.

Validating the fix in GSC

After the server is healthy again:

1. Open the Server error (5xx) issue in the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason..
2. Click Validate Fix. Google re-crawls the affected URLs in batches.
3. Watch the validation state. As URLs come back 200, they clear; if some still error, the validation flags them and you diagnose those specifically.

You don’t have to wait for validation to re-crawl naturally — but Validate Fix prioritizes the affected set and gives you a status to track. Remember the crawl rate itself ramps back gradually once you’re returning 2xx, so don’t expect an instant snap-back to your old crawl volume.

Where this sits

A 5xx is a crawl-and-serve problem, so it touches the neighbors: persistent 5xx hammers your crawl rate (the lever Googlebot pulls when your server struggles), and the host-level view of it lives in the Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). and its host status. It’s distinct from a 404 (not found), which signals gone rather than broken and is handled far more gently. For the bigger picture of how discovery and fetching work, see the crawling hub; for the rest of the Page Indexing statuses, see the GSC Page Indexing hub.

AI summary

A condensed take on the Advanced version:

• What it is: “Server error (5xx)” in GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. means GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. requested a URL and the server returned a 500-level code (500/502/503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime./504A 504 Gateway Timeout is an HTTP 5xx server error a gateway or proxy (a CDN, load balancer, or reverse proxy) returns when it doesn't get a timely response from the upstream server behind it. Unlike a 502 (a bad/garbled response) or a 503 (server explicitly unavailable), a 504 means no response arrived in time. For SEO, isolated 504s are retried and tolerated, but sustained 504s and timeouts make Googlebot slow its crawl and eventually drop pages from the index.) instead of a 200. The page lands under Not indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — there’s no usable content to index. It’s a server/infrastructure problem, not a content one.
• How Google reacts: crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. slows down (proportionate to how many URLs are erroring), content from a 5xx is ignored, already-indexed URLs are preserved then eventually dropped if errors persist, and the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. recovers gradually once you return 2xx.
• 429 counts: Google buckets 429 Too Many Requests (“server overloaded”) with 5xx — rate limiting GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. triggers the same slowdown.
• Codes as a starting point, not proof: 500 = an unexpected server-side condition; 502 = a gateway/proxy got a bad response from upstream; 503 = overloaded or in maintenance; 504 = a gateway/proxy timed out waiting on upstream. Any of these can originate at the origin, app, database, load balancer, or CDN — correlate logs across layers before assuming which one.
• Common causes: overloaded/misconfigured host, app/DB errors, upstream/CDN failures, and a CDN/WAF or rate limiter blocking Googlebot (returns 5xx/429 to the bot while users see the site fine) — confirm it’s really Googlebot via reverse DNS/IP ranges before changing a security rule; the user-agent alone is spoofable.
• Diagnose: Crawl Stats host availability → Googlebot-filtered server logs → URL Inspection live test → reproduce as Googlebot. A page working now shows the current state only, not what Googlebot saw at the time it logged the error.
• Planned downtime — the right way: Google’s default preference is to stay online with limited functionality. If you must fully disable, use 503 + Retry-After (Retry-After is best-effort, not a guarantee), for a day or two (“a few days at most”); weeks of downtime harms indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. with no fixed recovery time afterward. Not a 200 “maintenance” page (Google indexes it) and not a 404 (signals gone).
• The robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. trap: keep robots.txt returning 200 during a 503 window — a 503 on robots.txt can pause crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. site-wide (Google’s spec doc: 12 hours of no crawling, then up to 30 days on the cached robots.txt, before it falls back to assuming no restrictions).
• Validate: fix the server, then Validate Fix in the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. (optional — Google updates the issue count on any recrawl either way); the crawl rate ramps back gradually.

Official documentation

Primary-source documentation from the search engines.

Google

• Page Indexing report — the report itself, including the “Server error (5xx)” status definition and the “fixing server errors” guidance.
• How HTTP status codes affect Google’s crawlers — exactly how Google handles 5xx and 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content.: crawl slowdown, ignored content, preservation then drop, and recovery on 2xx.
• Pause your online business in Google Search — the canonical “how to take a site down correctly” doc: 503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. + Retry-After, “a few days at most,” and the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.-must-stay-crawlable rule.
• How to deal with planned site downtime (Search Central blog, 2011) — the older but still-cited write-up of the same 503 advice.
• How Google interprets the robots.txt specification — the staged behavior when robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. itself returns a 5xx: a 12-hour pause, then up to 30 days on the cached version, before Google falls back to assuming no restrictions.
• Optimize your crawl budget — how server responses (including 5xx) factor into crawl capacityThe number of URLs an engine will crawl in a timeframe..

Bing / Microsoft

• Bing Webmaster Guidelines — Bing surfaces server errors in its crawl reporting; like Google, it recommends a 503 with a Retry-After header for temporary downtime so BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. comes back rather than dropping pages.

Quotes from the source

On-the-record statements from Google. Each link is a deep link that jumps to the quoted passage on the source page.

Google — the Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. status definition

• “Your server returned a 500-level error when the page was requested.” — Google, Page IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. report help doc. Jump to quote

Google — how 5xx is handled (HTTP status codesAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. doc)

• “5xx and 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. server errors prompt Google’s crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. to temporarily slow down with crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. For Google Search, already indexed URLs are preserved in the index, but eventually dropped.” Jump to quote
• “Any content Google receives from URLs that return a 5xx status code is ignored.” Jump to quote
• “Google decreases the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. for the site. The decrease in crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. is proportionate to the number of individual URLs that are returning a server error.” Jump to quote
• “Once the server starts responding with a 2xx status code, Google gradually increases the crawl rate for the site.” Jump to quote
• “Google’s crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. treat the 429 status code as a signal that the server is overloaded, and it’s considered a server error.” Jump to quote

Google — taking a site down correctly (Pause your online business)

• “If you need to urgently disable the site for 1-2 days, then return an informational error page with a 503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. HTTP response status code.” Jump to quote
• “This is an extreme measure that should only be taken for a very short period of time (a few days at most).” Jump to quote
• “Completely closing a site even for just a few weeks can have negative consequences on Google’s indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. of your site.” Jump to quote
• “Use the retry-after HTTP header with a best effort date or duration.” Jump to quote
• “Don’t return a 503 HTTP response status code for the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file because this blocks all crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..” Jump to quote

Fixing a “Server error (5xx)” — checklist

Work top to bottom; the early steps tell you whether this is an emergency.

• Confirm the scale. GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. → Settings → Crawl statsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). → host availability and the by-response-code chart. Persistent/large-scale, or an isolated blip?
• Reproduce it. Load a flagged URL yourself, then run URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. → Test Live URL to see the error from Google’s side right now.
• Check it’s not botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-specific. Request the URL with GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.’s user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. (and from outside your network) — catch CDN/WAF/rate-limit rules that only fire for the bot (5xx or 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content.).
• Read the logs. Filter server/access logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. to verified GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.; confirm which status codes it received and when the 5xx started.
• Identify the code. 500 (app/DB) · 502 (bad upstream) · 503 (overload / maintenance) · 504 (upstream timeout) — it points you at the layer to fix.
• Reduce load on dynamic requests. Cache expensive pages, optimize slow queries, fix the heavy server-side work behind 503/504.
• Fix host health. Capacity, config, restarts, database, dependencies.
• Audit Google-blocking. CDN/WAF/bot/rate-limit rules returning 5xx/429 to Googlebot.
• Verify robots.txt returns 200 — especially if a maintenance handler is involved (a 503 on robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. can pause crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. site-wide).
• Run Validate Fix once the server is healthy; watch the validation state.

Planned-downtime (“503 done right”) checklist

Before you put the site into maintenance mode:

• Serve 503 (Service Unavailable), not 200, 403, 404, or 410.
• Add a Retry-After header (a best-effort date or duration).
• Keep it short — 1–2 days, a few days at most. Weeks of downtime harms indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. regardless of the status code.
• Exclude robots.txt from the maintenance rule so it keeps returning 200 and stays crawlable.
• Serve a useful, human-readable maintenance page in the 503 body (the content is ignored by Google, but real users will see it).
• When you’re back, confirm pages return 200 and the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. recovers (it ramps up gradually).

5xx cheat sheets

The common 5xx codes — what they mean and where to look

How Google reacts to a 5xx

Which status code for which situation

The mental models

1. A 5xx is “broken,” a 404 is “gone.” Google handles them very differently. A 404 is fine — once discovered, GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. just retries it and it drops gently. A 5xx says something is wrong with the server, so Google slows down, ignores the content, and — if it persists — drops your pages. When you’re choosing a status code on purpose, choose the one whose meaning matches reality.

2. Slow down → ignore → preserve → drop → recover. That’s the lifecycle of a 5xx, in order. The takeaways fall out of it: short outages are tolerated (you’re still in “preserve”), prolonged outages cost you the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. (you’ve reached “drop”), and the cure is simply returning 2xx (which moves you to “recover” — gradually).

3. The proportionality rule. The crawl slowdown is proportionate to how many URLs are erroring. One flaky URL is a nudge; a site-wide 5xx is a hard brake. This tells you where to triage: a small flagged set is low-urgency; a host-level error is an emergency.

4. 503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. is a feature, not just an error. For planned downtime, 503 + Retry-After is the correct, SEO-safe response — because it’s temporary by design. But it’s a short-term tool: a few days at most. No status code protects you from the indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. damage of being down for weeks.

5. robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. is load-bearing. Treat your robots.txt as the one URL that must never 5xx. A 503 on robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. doesn’t just hide robots.txt — it pauses crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. for the whole site. Whatever maintenance rule you write, carve robots.txt out of it.

6. Reproduce as the botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., not as yourself. The nastiest 5xx causes (WAF rules, rate limiting429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content.) fire only for GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.. If the page loads fine in your browser but GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. says 5xx, you haven’t reproduced the problem yet — test with Googlebot’s user-agent and read the logs.

Check what status code a URL actually returns

The first move is confirming the response code from the command line — and, ideally, as GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., since some 5xx/429s429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. only fire for the botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index..

macOS / Linux

# Status code as a normal client
curl -s -o /dev/null -w "%{http_code}\n" https://www.example.com/page/

# Status code as Googlebot (catches WAF / rate-limit rules that target the bot)
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://www.example.com/page/

# Full response headers — confirm a 503 carries a Retry-After during maintenance
curl -sI https://www.example.com/page/

Windows (PowerShell)

# Status code (note: 5xx throws, so capture the response in the catch block)
try {
  (Invoke-WebRequest -Uri "https://www.example.com/page/" -UseBasicParsing).StatusCode
} catch {
  $_.Exception.Response.StatusCode.value__
}

# As Googlebot
try {
  (Invoke-WebRequest -Uri "https://www.example.com/page/" -UseBasicParsing `
    -UserAgent "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)").StatusCode
} catch { $_.Exception.Response.StatusCode.value__ }

Confirm the traffic was really Googlebot (before trusting your logs)

A “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.” 5xx in your logs might be a spoofed user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target.. Verify with a reverse + forward DNS check — Google publishes no shortcut.

macOS / Linux

# 1) Reverse DNS the IP from your logs — it should end in googlebot.com or google.com
host 66.249.66.1
# 2) Forward DNS that hostname back — it must resolve to the same IP
host crawl-66-249-66-1.googlebot.com

Windows

nslookup 66.249.66.1
nslookup crawl-66-249-66-1.googlebot.com

Maintenance mode done right — 503 + Retry-After, robots.txt stays 200

The critical bit: serve 503 for the site, but never for robots.txt.

Apache (.htaccess)

# Let robots.txt through untouched so crawling isn't halted
RewriteEngine On
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteCond %{REQUEST_URI} !^/maintenance\.html$
RewriteRule ^ /maintenance.html [R=503,L]

# Attach the headers to the 503 response
<If "%{REQUEST_URI} != '/robots.txt'">
  Header always set Retry-After "86400"
</If>
ErrorDocument 503 /maintenance.html

Nginx

location / {
    # robots.txt is matched by a more specific block below, so it's exempt
    return 503;
}

location = /robots.txt {
    # Always serve the real robots.txt with a 200
    try_files $uri =404;
}

error_page 503 /maintenance.html;
location = /maintenance.html { internal; }

# Add Retry-After to 503 responses (a day, in seconds)
add_header Retry-After 86400 always;

Confirm it with curl -sI from the section above: the site URLs should return 503 with a Retry-After header, and https://www.example.com/robots.txt should still return 200.

Tools for finding and confirming a 5xx

Start with the tools that can check status directly against your flagged URLs, then narrow to logs if the answer isn’t obvious from a single request.

• Bulk HTTP Status Code Checker — paste every URL Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. flagged under “Server error (5xx)” (up to 500 at once) and see the actual status code, redirect chainA → B → C instead of A → C. Each hop loses link equity and adds latency., and latency for each one right now. This is the fastest way to confirm whether a fix really took before you click Validate Fix in GSC, or whether some URLs are still erroring.
• Website Down Checker — a single-URL “is it actually down right now” check from a live vantage point, with response timing and redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't.. Good for the first step in this article’s diagnosis flow: “visit a flagged page yourself” — before you go digging in logs.
• HTTP Header Checker — inspect the full response headers across every redirect hop. Use it to confirm a 503 actually carries a Retry-After header during planned maintenance, and to check that robots.txt is still answering 200 with no maintenance rule accidentally catching it.
• Log File Analyzer — drop in your server access log and see status-code waste and crawl activity by botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., plus a spoofer report that separates verified GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. from user-agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. faking it. This is the tool for the trickiest cause in this article: a CDN/WAF or rate limiter that returns 5xx or 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. only to GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. while real visitors see the site fine — something a browser test alone won’t catch.

From Google and third-party crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.:

• Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. — Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). (Settings → Crawl stats) is the whole-site view of host availability and the by-response-code breakdown covered in the Advanced tab.
• Screaming Frog SEO Spider / Ahrefs Site Audit can crawl your site the way Googlebot would and surface which URLs are erroring at scale, beyond the handful Search Console has already sampled.

Mistakes that turn a fixable 5xx into an indexing problem

• Serving a 200 “we’ll be back soon” page during maintenance instead of a 503. Google treats whatever comes back with a 200 as the real page and may indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. your maintenance message in place of your actual content. Serve a 503 instead — its content is ignored, which is what you want.
• Returning a 404, 410, or 403 during planned downtime. These say “gone” or “forbidden,” not “temporarily unavailable.” Google’s guidance is explicit: don’t block a paused site this way. A 503 preserves the page’s standing in the index; a 404/410 doesn’t.
• Letting maintenance mode503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. return 503 for robots.txt too. A blanket rule that 503s everything, robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. included, pauses crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. for the entire site — not just the pages you meant to take down. Carve robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. out of the maintenance handler so it always answers 200.
• Assuming a page that loads fine right now rules out an earlier GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.-only 5xx. A CDN/WAF or rate limiter can return 5xx or 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. to GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. specifically while every human visitor sees a normal 200 — and the fact it works for you this minute doesn’t tell you what it returned when Google actually hit it. Test with Googlebot’s user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., read the logs, and confirm the traffic is really Googlebot (reverse DNS/IP ranges) before you trust the user-agent header alone.
• Leaving a “temporary” maintenance window up for weeks. A 503 is only correct for a short outage — Google’s own wording is “a few days at most.” Even a technically-correct 503 response causes indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. harm if the site stays down for weeks, and Google is explicit that there’s no fixed recovery time or way to speed one up after that.

Confirming a 5xx fix actually took

Each test below checks one specific claim after you believe the server error is resolved. Run them in order — the first two catch most false “it’s fixed” calls.

Test 1 — The flagged URLs return 200 right now

• Test to run: Run the Bulk HTTP Status Code Checker against every URL Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. flagged, or curl -sI each one.
• Expected result: A clean 200 on every URL, consistently across repeat requests, not intermittent.
• Failure interpretation: Any URL still returning 5xx means the underlying server/host issue isn’t resolved; a mix of 200s and 5xxs across repeats points at an overloaded or unstable upstream, not a one-time bug.
• Monitoring window: Immediate — a single clean pass is enough to move on to clicking Validate Fix.
• Rollback trigger: Any 5xx recurrence before you click Validate Fix — hold off and keep diagnosing instead of validating a fix that hasn’t stuck.

Test 2 — Google’s own re-crawl clears it

• Test to run: Open the Server error (5xx) issue in the Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. report and click Validate Fix.
• Expected result: The validation state moves to “Validation passed” as Google re-crawls the affected URLs and confirms 2xx.
• Failure interpretation: A “Validation failed” result on specific URLs — even after your own checks looked clean — usually means those URLs are still erroring for GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. specifically, which points back at a botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-targeted WAF/rate-limit rule.
• Monitoring window: Google re-crawls the flagged set in batches on its own schedule; there’s no published fixed cadence for this, so check back periodically rather than expecting a specific turnaround date. Validating isn’t even required — Google updates the issue count whenever it recrawlsCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. a page with known issues, validation or not — but it does prioritize the affected set and gives you a status to track.
• Rollback trigger: Any URL that gets re-flagged under Server error (5xx) after previously passing validation.

Test 3 — Crawl Stats host availability recovers

• Test to run: Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. → Settings → Crawl statsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). → host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity..
• Expected result: The host availability graph returns to normal, with no new red flags in the by-response-code breakdown.
• Failure interpretation: A graph that’s still showing red after you believe the fix landed means Google’s crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. are still hitting errors — the fix hasn’t reached whatever GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. is actually requesting.
• Monitoring window: Allow a few days after the fix for Crawl Stats to reflect it; it’s a rolling report, not real-time.
• Rollback trigger: Host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. flipping back to red after a period of green.

Test 4 — robots.txt still returns 200

• Test to run: curl -sI on your robots.txt URL, or run it through the HTTP Header Checker.
• Expected result: 200, exactly as it did before the incident or maintenance window.
• Failure interpretation: A 503 (or any non-200) on robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. means you’re blocking crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. for the whole site, not just the URLs you meant to affect.
• Monitoring window: Immediate.
• Rollback trigger: Any non-200 response on robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. — treat this as the single highest-priority thing to fix first.

Test 5 — Googlebot gets the same response a browser does

• Test to run: Request the URL with Googlebot’s user-agent (see the Scripts tab) and cross-check against the Log File Analyzer’s verified-Googlebot vs. spoofer report.
• Expected result: Googlebot receives the same 200 a normal browser request gets.
• Failure interpretation: A status mismatch between the Googlebot-UA request and a normal browser request means a WAF, rate limiter, or bot-detection rule is targeting the bot specifically.
• Monitoring window: Immediate.
• Rollback trigger: Any mismatch between what Googlebot receives and what real traffic receives.

The standing KPIs for server health

These aren’t a one-time fix check (that’s the Validation Tests tab) — they’re what to keep an eye on quarter over quarter so a 5xx pattern doesn’t build up unnoticed.

5xx rate in your server/access logs

• What it tells you: The ground-truth share of requests actually failing at the server, independent of what Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. has gotten around to sampling and reporting.
• How to pull it: The Log File Analyzer — drop in your access log and see status-code waste, filterable to GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer..
• Benchmark / realistic range: There’s no universal healthy percentage here — establish your own baseline from a known-good period and watch for a rise against it, rather than chasing an invented target number.
• Cadence: Check after any deploy or traffic spike; a weekly glance is enough otherwise for most sites.

GSC Crawl Stats — host status by response code

• What it tells you: Google’s own recent read on how often its crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. hit errors on your host, independent of your own logs.
• How to pull it: Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. → Settings → Crawl statsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root)..
• Benchmark / realistic range: Same honesty rule applies — track your own trend line rather than a fixed percentage; treat any new red flag in the host status section as worth investigating, whatever the number.
• Cadence: A weekly glance, or immediately after any known outage or maintenance window.

Uptime / origin availability

• What it tells you: Whether the origin server itself is reachable at all — upstream of whatever status code GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. ends up seeing.
• How to pull it: An uptime monitor for continuous coverage, or an ad hoc check with the Website Down Checker.
• Benchmark / realistic range: Depends entirely on your hosting tier and any SLA you’re on — there’s no universal “good” uptime figure to borrow; use your own host’s SLA, or your own historical baseline, as the reference point.
• Cadence: Continuous if you can, or at minimum a check after any known incident.

Server error (5xx) count in the Page Indexing report

• What it tells you: How many URLs Google currently has flagged as erroring — the lagging, official record, as opposed to the real-time server/log view.
• How to pull it: Search Console → Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. → Not indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → Server error (5xx).
• Benchmark / realistic range: Zero is the honest target for any URL you want indexed. Anything above zero is worth running through this article’s diagnosis steps, whatever the count.
• Cadence: Whenever you check Search Console generally, and always after a known incident or deploy.

AI prompts for 5xx diagnosis and maintenance-mode review

Two prompts built around the specific tasks this article walks through — paste your own data into either one.

Prompt: spot patterns in raw log lines around a 5xx incident

Here are raw server/access log lines from around the time my site started
returning 5xx errors:

<paste log lines>

From only what's in these lines, tell me:
1. Which HTTP status codes appear, and in what proportion.
2. Whether requests from Googlebot's IP ranges or user-agent get a different
   status code than other traffic (this could indicate a WAF or rate limiter
   targeting the bot specifically).
3. Any timing pattern — recurring at a fixed interval, clustered around a
   traffic spike, or starting right after a specific timestamp.

Don't guess at a root cause you can't see in the data — only report what the
log lines actually show.

Prompt: review a maintenance-mode config for the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. trap

Here is my maintenance-mode configuration (nginx/Apache/other):

<paste config>

Check specifically whether this configuration:
1. Returns a 503 status code for regular site pages during maintenance.
2. Excludes /robots.txt from the 503 rule, so robots.txt keeps returning 200.
3. Sets a Retry-After header on the 503 responses.

Flag anything that would cause robots.txt to return a non-200 status, and
anything missing a Retry-After header.

Test yourself: Server error (5xx)

Five questions on what “Server error (5xx)” means, how Google reacts to it, and the right way to take a site down on purpose. Pick an answer for each, then check.

Server error (5xx)

A Page Indexing status in Google Search Console meaning Googlebot requested a URL and the server returned a 500-level HTTP error (500, 502, 503, 504) instead of a 200, so the page can't be indexed.

Related: Crawl Stats report, Crawl rate, Crawling

Server error (5xx)

Server error (5xx) is a “Not indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.” status in Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.. It means GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. requested the URL and the server replied with a 500-level HTTP status codeAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. — most often 500 (internal server errorA 500 Internal Server Error is a generic HTTP status code — RFC 9110 defines it as an unexpected condition that stopped the server from fulfilling the request, and nothing more. The request may have been fine — something broke server-side while generating the response. For SEO, an isolated 500 is typically retried, but persistent, site-wide 500s get Google's documented response: slower crawling and, if the errors don't clear, eventual removal from the index.), 502 (bad gateway), 503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. (service unavailable), or 504A 504 Gateway Timeout is an HTTP 5xx server error a gateway or proxy (a CDN, load balancer, or reverse proxy) returns when it doesn't get a timely response from the upstream server behind it. Unlike a 502 (a bad/garbled response) or a 503 (server explicitly unavailable), a 504 means no response arrived in time. For SEO, isolated 504s are retried and tolerated, but sustained 504s and timeouts make Googlebot slow its crawl and eventually drop pages from the index. (gateway timeout) — instead of a successful 200. Google couldn’t retrieve usable content, so the page isn’t in the index, or won’t stay there.

It matters because a 5xx is a “something broke on the server” signal, and Google reacts to it. Repeated 5xx errors make GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. slow its crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor., any content returned with a 5xx is ignored, and pages that were already indexed are preserved at first but eventually dropped if the errors persist. CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. recovers on its own once the server starts returning 2xx again — gradually, not instantly.

The one nuance worth knowing: 503 (Service Unavailable) is the correct response for short, planned downtime, because it tells crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. “temporary, come back later” rather than “gone.” Google’s default preference is to keep a site online with limited functionality rather than fully disable it; full disable is framed as an urgent, few-days-at-most measure with no fixed recovery time if you overstay it. Your robots.txt file must keep returning 200 throughout — a 503 on robots.txt can pause crawling for the whole site.

Related: Crawl Stats report, Crawl rate, Crawling

Changelog