Server Error (5xx)

TL;DR — “Server error (5xx)” means Google tried to load your page and your server answered with an error (a 500-level code) instead of showing the page. Google can’t index an error, so the URL won’t appear in search. A quick blip is fine — Google just backs off and retries — but if the errors keep happening, your pages can eventually drop out of Google. Fix the server, and crawling comes back on its own.

What this status means

This label means Google’s request received a server response in the 5xx range. Evidence for this claim Google reports Server error 5xx when the server returned a 500-level response for the requested page. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report Google slows crawling in response to 5xx errors and can eventually remove persistently failing URLs. Evidence for this claim Google slows crawling for 5xx responses and may eventually remove persistently failing URLs from the index. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: HTTP status codes

When Googlebot asks your server for a page, the server answers with a status code. A 200 means “here’s the page.” A 5xx is a whole family of error codes — 500, 502, 503, 504 — that all mean “something went wrong on my end.” When Search Console shows Server error (5xx) under Not indexed, it’s telling you Google asked for that URL and got one of those errors back instead of the page.

Google can’t index an error page. There’s no real content to read, so the URL stays out of search until the server starts responding normally again.

Is this an emergency?

It depends on whether it’s a one-off or a pattern:

• A brief hiccup — your server was busy for a minute, a deploy restarted it — is normal. Google notices the error, slows down, and tries again later. No lasting harm.
• Errors that keep happening are the problem. The longer your server keeps returning 5xx, the more Google crawls you, until eventually pages that were in Google get dropped.

Google doesn’t publish a “safe” number of flagged URLs — the crawl-rate hit scales with however many URLs are erroring, not a fixed threshold. So prioritize by what actually matters: is it an important page, is it still failing when you check again, and is it a handful of URLs or a lot of them? A single important URL that keeps failing deserves attention now; a one-time blip on a low-value page usually resolves itself. A lot of URLs flagged, or the same ones flagged over and over, means something on your site is actually broken and needs attention.

How to start fixing it

1. Check if your site is actually up. Visit a flagged page yourself. If it errors for you too, that’s your answer — the server has a problem.
2. Ask your host or developer. 5xx errors almost always come from the server, the app, or the database — not from anything in your page’s content or SEO settings.
3. Look at Crawl Stats in Search Console (Settings → Crawl stats). It shows whether Google has been hitting errors across your whole site recently.
4. Once it’s fixed, tell Google to re-check. In the Page Indexing report, open the “Server error (5xx)” issue and click Validate Fix. Google re-crawls the affected URLs and clears them as they come back clean.

The one thing people get wrong

If you ever need to take your site down on purpose — for maintenance, say — don’t just show an error page or a “we’ll be back soon” page that returns a normal 200. And don’t make it a 404. The right move is a 503 status (it literally means “service unavailable, temporarily”). That tells Google “I’m down for a bit, come back later” instead of “this page is broken” or “this page is gone.” There’s a right way to do that, covered in the Advanced tab.

Want the full version — exactly how Google reacts to 5xx, the 500-vs-502-vs-503 differences, how to diagnose the cause, and the proper maintenance-mode setup? Switch to the Advanced tab.

TL;DR — “Server error (5xx)” means the server returned a 500-level code when Googlebot requested the URL, so the page can’t be indexed. Two distinct harms: Google slows the crawl rate (proportionate to how many URLs are erroring) and ignores any content a 5xx returns; if the errors persist, already-indexed URLs are preserved at first, then dropped. Recovery is automatic once you return 2xx, but the crawl rate ramps back up gradually. Google buckets 429 (rate limiting / “server overloaded”) with 5xx. Causes: overloaded/misconfigured host, app/DB errors, upstream or CDN failures, a CDN/WAF blocking Googlebot (verify with reverse DNS/IP ranges before changing security rules — a Googlebot user-agent is spoofable). Diagnose with Crawl Stats host availability → server logs → URL Inspection live test. Google’s default recommendation for planned closures is to stay online with limited functionality; if you must fully disable, the correct response is 503 + Retry-After for a day or two (“a few days at most” — weeks of it harms indexing with no fixed recovery time) — and robots.txt must keep returning 200, because a 503 on robots.txt can pause crawling site-wide. Then run Validate Fix (optional — it just tracks the fix).

What the status actually reports

The label reports the response Google observed, not the specific origin, proxy, database, or CDN failure that caused it. Evidence for this claim Google reports Server error 5xx when the server returned a 500-level response for the requested page. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report Google’s HTTP guidance defines the separate crawl and indexing effects. Evidence for this claim Google slows crawling for 5xx responses and may eventually remove persistently failing URLs from the index. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: HTTP status codes

Google’s own definition is one line: your server returned a 500-level error when the page was requested. That’s it. Googlebot made a request, and instead of a 200 with content it got a 500, 502, 503, 504, or similar. The URL lands under Not indexed because there was no usable content to index.

Keep one thing straight: this is about the response code, not the page’s content, markup, or SEO setup. A 5xx is a server-and-infrastructure problem. No amount of editing the page’s HTML fixes a 502 coming from an overloaded backend.

How Google treats 5xx — the crawl economics

This is the part most explainers skip, and it’s the whole reason 5xx matters more than, say, a 404. Google’s HTTP-errors documentation spells out the behavior:

• Crawling slows down. 5xx (and 429) errors prompt Google’s crawlers to temporarily slow down. The decrease in crawl rate is proportionate to the number of individual URLs returning a server error — a handful of erroring URLs is a nudge; a site-wide 5xx is a hard brake.
• The content is ignored. Anything Google receives from a URL returning a 5xx is thrown away. There’s no “partial credit” for an error body — Google does not index a “site down” message you served with a 500.
• Indexed URLs are preserved, then eventually dropped. Already-indexed URLs stay in the index at first. But Google’s indexing pipeline removes URLs that persistently return a server error. So a short outage costs you nothing in the index; a prolonged one costs you the pages.
• Recovery is automatic — but gradual. Once the server starts responding with 2xx again, Google gradually increases the crawl rate back up. You don’t file a ticket; you fix the server and the crawl ramps back cautiously on its own.

That sequence — slow down → ignore content → preserve → eventually drop → recover on 2xx — is the accuracy spine of this whole topic.

429 counts as a server error. Worth flagging because most competitor content misses it: Google treats a 429 Too Many Requests as a signal that the server is overloaded, and buckets it with 5xx. If your rate limiting or bot protection is firing 429s at Googlebot, you’re getting the same crawl slowdown as a 500.

The common 5xx codes, by likely root cause

Knowing which 5xx you’re getting is a starting point for where to look — not proof of what’s broken. RFC 9110 (the HTTP specification) defines each code by what the responding component was doing, and any of these can be emitted by an origin server, an application server, a load balancer, a CDN, or a proxy sitting in front of the real origin. Treat the code as the first clue, then correlate it against edge, origin, application, and database/dependency logs for the same request before you conclude which layer actually failed:

• 500 Internal Server Error — per the spec, “the server encountered an unexpected condition that prevented it from fulfilling the request.” That’s deliberately generic: it’s often application code or a runtime/config error, but the status code alone doesn’t prove that — check app logs to confirm.
• 502 Bad Gateway — “the server, while acting as a gateway or proxy, received an invalid response from an inbound server.” This points at an upstream interaction, not a single vendor: check the chain of load balancer, reverse proxy, CDN, and the origin behind them.
• 503 Service Unavailable — “the server is currently unable to handle the request due to a temporary overload or scheduled maintenance.” Look at capacity, traffic spikes, and whether something is in a maintenance state. (This is also the code you want to send on purpose during planned downtime — see below.) Note the spec’s own caveat: an overloaded server isn’t required to return 503 at all — “some servers might simply refuse the connection,” which can show up as a timeout or connection error instead of a clean status code.
• 504 Gateway Timeout — “the server, while acting as a gateway or proxy, did not receive a timely response from an upstream server.” It marks a timeout boundary, not which upstream component was slow: check long database queries, slow third-party calls, and an origin struggling under load.

A practical thread runs through these: 5xx often traces back to an overloaded or slow backend, so server performance work — faster queries, lighter server-side rendering, more capacity — is frequently part of the fix, not a side quest. But because the same status code can come from any layer in the chain, the code narrows your search; the logs tell you where the failure actually happened.

Common causes

• Overloaded host. Traffic spikes (including aggressive crawling) outrun your server’s capacity and it starts shedding requests as 5xx.
• Application or database errors. Unhandled exceptions, a downed DB, a bad deploy, exhausted connection pools.
• Misconfiguration. A broken config after a change, an expired dependency, a full disk.
• Upstream / CDN failures. Your origin is fine but a proxy, load balancer, or CDN in front of it is returning 502/504 — or the CDN itself had an outage.
• A CDN/WAF or rate limiter blocking Googlebot. Bot protection, security rules, or rate limiting that mistakes Googlebot for an attacker can return 5xx or 429 only to Googlebot while real users see the site fine. This one is under-covered and a frequent real-world cause — which is why you have to reproduce as Googlebot, not just check the page in your browser.

How to diagnose it

Work from the whole-site view down to the single URL:

1. Crawl Stats → Host availability. In Search Console, Settings → Crawl stats. The host status section and the by-response-code breakdown show whether 5xx is a persistent, large-scale issue or an isolated blip, and roughly when it started.
2. Server and access logs. The ground truth. Filter your logs to Googlebot (verified — see the Scripts tab) and look at the status codes it actually received and when. Logs will show you a CDN/WAF blocking Googlebot that a browser test never would.
3. URL Inspection → Live Test. Run a flagged URL through URL Inspection and use Test Live URL. This confirms whether Google-InspectionTool can access the page right now — useful, but it’s a current-state check, not proof of what happened at the earlier time Google logged the 5xx. A page that loads fine for you (in a browser, or via Live Test) minutes or hours later doesn’t rule out a real error at the moment Googlebot actually hit it — time, IP, geo, cache state, and bot-detection rules can all differ between the two requests.
4. Reproduce as Googlebot. Request the URL with Googlebot’s user-agent (and, if you can, from outside your network) to catch WAF/rate-limit rules that only fire for the bot. Treat this as a differential test, not verified proof of what Googlebot itself saw — a Googlebot user-agent string is trivial to spoof in either direction. Before you loosen a WAF, CDN, or rate-limit rule because “it’s only blocking Googlebot,” confirm the traffic in your logs is real Googlebot via reverse DNS or Google’s published IP ranges (see the Scripts tab) — don’t change a security control based on the user-agent header alone.

How to fix it

Once you know the cause, the fixes follow Google’s own “fixing server errors” guidance:

• Confirm the scale in Crawl Stats before you change anything — is this persistent and large, or a one-off?
• Reduce excessive page loading for dynamic requests. Cache expensive pages, optimize slow queries, and stop generating heavy responses on every hit.
• Make sure the host isn’t down, overloaded, or misconfigured. Add capacity, fix the config, restart the broken service, check the database.
• Make sure you’re not inadvertently blocking Google. Audit CDN/WAF/bot rules and rate limits for anything firing 5xx or 429 at Googlebot.
• Control crawling wisely. If aggressive crawling is overloading you, the answer is a temporary 503/429 to ease the bot off — not a permanent block.

The right way to take a site down on purpose — 503 + Retry-After

Sometimes you want the site unavailable: a migration, scheduled maintenance, pausing an online business. Doing it wrong turns a planned event into a deindexing event. Google’s Pause your online business in Google Search doc is explicit about the correct approach.

Google’s default recommendation is to keep the site up, just limited. If the closure is temporary and you plan to reopen, Google’s own preference is that you “keep your site online and limit the functionality” rather than take it fully offline. Full disable-and-503 is the urgent option, not the default one.

When you do need the whole site down, use 503 (Service Unavailable) with a Retry-After header. Google: “If you need to urgently disable the site for 1-2 days, then return an informational error page with a 503 HTTP response status code.” And pair it with the header: “Use the retry-after HTTP header with a best effort date or duration.” The 503 says “temporarily down,” and Retry-After tells the bot roughly when to come back — but treat that header as advisory, not a guarantee: it’s a best-effort signal, not a promise Google will recrawl at that exact time.

It’s a short-term measure only. Google calls it “an extreme measure that should only be taken for a very short period of time (a few days at most).” And the warning that competitors leave out: “Completely closing a site even for just a few weeks can have negative consequences on Google’s indexing of your site.” Google is explicit that there’s no way to shortcut your way back from that either — “there’s no fixed time for a recovery from a complete removal, and there’s no mechanism to speed that up.” In other words, a correct 503 protects you for a day or two — but no status code, and no amount of validating in Search Console, saves you from the indexing damage of being down for weeks, or gives you a guaranteed comeback date. If you need a long outage, that’s a different conversation (and probably a redirect or a real plan).

Why not a 200 or a 404

• Don’t serve a 200 “under maintenance” page. Google will treat that page’s content as the real page and may index your “we’ll be back soon” message. The content from a proper 503, by contrast, is ignored — which is what you want.
• Don’t return a 404 (or 403/410). Google’s guidance is explicit: don’t block the site by returning 403, 404, or 410 during downtime. A 404 signals gone, not temporarily down; you want the page preserved, and a 503 does that.

The robots.txt trap — keep it crawlable

This is the single most-missed detail, and it can pause crawling for your entire site. During a 503 maintenance window, your robots.txt file must keep returning 200 and stay crawlable. Google is blunt: “Don’t return a 503 HTTP response status code for the robots.txt file because this blocks all crawling.” Carve robots.txt out of the maintenance handler so it always answers 200 — don’t rely on what happens if you don’t.

For context, Google’s robots.txt spec doc lays out what actually happens if robots.txt itself starts erroring, and it’s staged rather than an instant, permanent lockout: for the first 12 hours Google stops crawling the site while still retrying robots.txt; for the next 30 days it falls back to the last known good version of robots.txt while still trying to fetch a fresh one; and after 30 days, if the site is otherwise reachable, Google drops the cached version and behaves as if there’s no robots.txt file at all (i.e., no crawl restrictions from it) rather than continuing to block. If there’s no cached version to fall back on in the first place, Google likewise assumes there’s no crawl restriction. None of that is a reason to risk it on purpose — a multi-day crawl pause at the start of the window is still real damage — it just means “robots.txt 503’d” isn’t a permanent, unrecoverable state if it happens by accident and gets fixed.

Validating the fix in GSC

After the server is healthy again:

1. Open the Server error (5xx) issue in the Page Indexing report.
2. Click Validate Fix. Google re-crawls the affected URLs in batches.
3. Watch the validation state. As URLs come back 200, they clear; if some still error, the validation flags them and you diagnose those specifically.

You don’t have to wait for validation to re-crawl naturally — but Validate Fix prioritizes the affected set and gives you a status to track. Remember the crawl rate itself ramps back gradually once you’re returning 2xx, so don’t expect an instant snap-back to your old crawl volume.

Where this sits

A 5xx is a crawl-and-serve problem, so it touches the neighbors: persistent 5xx hammers your crawl rate (the lever Googlebot pulls when your server struggles), and the host-level view of it lives in the Crawl Stats report and its host status. It’s distinct from a 404 (not found), which signals gone rather than broken and is handled far more gently. For the bigger picture of how discovery and fetching work, see the crawling hub; for the rest of the Page Indexing statuses, see the GSC Page Indexing hub.

AI summary

A condensed take on the Advanced version:

• What it is: “Server error (5xx)” in GSC Page Indexing means Googlebot requested a URL and the server returned a 500-level code (500/502/503/504) instead of a 200. The page lands under Not indexed — there’s no usable content to index. It’s a server/infrastructure problem, not a content one.
• How Google reacts: crawling slows down (proportionate to how many URLs are erroring), content from a 5xx is ignored, already-indexed URLs are preserved then eventually dropped if errors persist, and the crawl rate recovers gradually once you return 2xx.
• 429 counts: Google buckets 429 Too Many Requests (“server overloaded”) with 5xx — rate limiting Googlebot triggers the same slowdown.
• Codes as a starting point, not proof: 500 = an unexpected server-side condition; 502 = a gateway/proxy got a bad response from upstream; 503 = overloaded or in maintenance; 504 = a gateway/proxy timed out waiting on upstream. Any of these can originate at the origin, app, database, load balancer, or CDN — correlate logs across layers before assuming which one.
• Common causes: overloaded/misconfigured host, app/DB errors, upstream/CDN failures, and a CDN/WAF or rate limiter blocking Googlebot (returns 5xx/429 to the bot while users see the site fine) — confirm it’s really Googlebot via reverse DNS/IP ranges before changing a security rule; the user-agent alone is spoofable.
• Diagnose: Crawl Stats host availability → Googlebot-filtered server logs → URL Inspection live test → reproduce as Googlebot. A page working now shows the current state only, not what Googlebot saw at the time it logged the error.
• Planned downtime — the right way: Google’s default preference is to stay online with limited functionality. If you must fully disable, use 503 + Retry-After (Retry-After is best-effort, not a guarantee), for a day or two (“a few days at most”); weeks of downtime harms indexing with no fixed recovery time afterward. Not a 200 “maintenance” page (Google indexes it) and not a 404 (signals gone).
• The robots.txt trap: keep robots.txt returning 200 during a 503 window — a 503 on robots.txt can pause crawling site-wide (Google’s spec doc: 12 hours of no crawling, then up to 30 days on the cached robots.txt, before it falls back to assuming no restrictions).
• Validate: fix the server, then Validate Fix in the Page Indexing report (optional — Google updates the issue count on any recrawl either way); the crawl rate ramps back gradually.

Official documentation

Primary-source documentation from the search engines.

Google

• Page Indexing report — the report itself, including the “Server error (5xx)” status definition and the “fixing server errors” guidance.
• How HTTP status codes affect Google’s crawlers — exactly how Google handles 5xx and 429: crawl slowdown, ignored content, preservation then drop, and recovery on 2xx.
• Pause your online business in Google Search — the canonical “how to take a site down correctly” doc: 503 + Retry-After, “a few days at most,” and the robots.txt-must-stay-crawlable rule.
• How to deal with planned site downtime (Search Central blog, 2011) — the older but still-cited write-up of the same 503 advice.
• How Google interprets the robots.txt specification — the staged behavior when robots.txt itself returns a 5xx: a 12-hour pause, then up to 30 days on the cached version, before Google falls back to assuming no restrictions.
• Optimize your crawl budget — how server responses (including 5xx) factor into crawl capacity.

Bing / Microsoft

• Bing Webmaster Guidelines — Bing surfaces server errors in its crawl reporting; like Google, it recommends a 503 with a Retry-After header for temporary downtime so Bingbot comes back rather than dropping pages.

Quotes from the source

On-the-record statements from Google. Each link is a deep link that jumps to the quoted passage on the source page.

Google — the Page Indexing status definition

• “Your server returned a 500-level error when the page was requested.” — Google, Page Indexing report help doc. Jump to quote

Google — how 5xx is handled (HTTP status codes doc)

• “5xx and 429 server errors prompt Google’s crawlers to temporarily slow down with crawling. For Google Search, already indexed URLs are preserved in the index, but eventually dropped.” Jump to quote
• “Any content Google receives from URLs that return a 5xx status code is ignored.” Jump to quote
• “Google decreases the crawl rate for the site. The decrease in crawl rate is proportionate to the number of individual URLs that are returning a server error.” Jump to quote
• “Once the server starts responding with a 2xx status code, Google gradually increases the crawl rate for the site.” Jump to quote
• “Google’s crawlers treat the 429 status code as a signal that the server is overloaded, and it’s considered a server error.” Jump to quote

Google — taking a site down correctly (Pause your online business)

• “If you need to urgently disable the site for 1-2 days, then return an informational error page with a 503 HTTP response status code.” Jump to quote
• “This is an extreme measure that should only be taken for a very short period of time (a few days at most).” Jump to quote
• “Completely closing a site even for just a few weeks can have negative consequences on Google’s indexing of your site.” Jump to quote
• “Use the retry-after HTTP header with a best effort date or duration.” Jump to quote
• “Don’t return a 503 HTTP response status code for the robots.txt file because this blocks all crawling.” Jump to quote

Fixing a “Server error (5xx)” — checklist

Work top to bottom; the early steps tell you whether this is an emergency.

• Confirm the scale. GSC → Settings → Crawl stats → host availability and the by-response-code chart. Persistent/large-scale, or an isolated blip?
• Reproduce it. Load a flagged URL yourself, then run URL Inspection → Test Live URL to see the error from Google’s side right now.
• Check it’s not bot-specific. Request the URL with Googlebot’s user-agent (and from outside your network) — catch CDN/WAF/rate-limit rules that only fire for the bot (5xx or 429).
• Read the logs. Filter server/access logs to verified Googlebot; confirm which status codes it received and when the 5xx started.
• Identify the code. 500 (app/DB) · 502 (bad upstream) · 503 (overload / maintenance) · 504 (upstream timeout) — it points you at the layer to fix.
• Reduce load on dynamic requests. Cache expensive pages, optimize slow queries, fix the heavy server-side work behind 503/504.
• Fix host health. Capacity, config, restarts, database, dependencies.
• Audit Google-blocking. CDN/WAF/bot/rate-limit rules returning 5xx/429 to Googlebot.
• Verify robots.txt returns 200 — especially if a maintenance handler is involved (a 503 on robots.txt can pause crawling site-wide).
• Run Validate Fix once the server is healthy; watch the validation state.

Planned-downtime (“503 done right”) checklist

Before you put the site into maintenance mode:

• Serve 503 (Service Unavailable), not 200, 403, 404, or 410.
• Add a Retry-After header (a best-effort date or duration).
• Keep it short — 1–2 days, a few days at most. Weeks of downtime harms indexing regardless of the status code.
• Exclude robots.txt from the maintenance rule so it keeps returning 200 and stays crawlable.
• Serve a useful, human-readable maintenance page in the 503 body (the content is ignored by Google, but real users will see it).
• When you’re back, confirm pages return 200 and the crawl rate recovers (it ramps up gradually).

5xx cheat sheets

The common 5xx codes — what they mean and where to look

How Google reacts to a 5xx

Which status code for which situation

The mental models

1. A 5xx is “broken,” a 404 is “gone.” Google handles them very differently. A 404 is fine — once discovered, Googlebot just retries it and it drops gently. A 5xx says something is wrong with the server, so Google slows down, ignores the content, and — if it persists — drops your pages. When you’re choosing a status code on purpose, choose the one whose meaning matches reality.

2. Slow down → ignore → preserve → drop → recover. That’s the lifecycle of a 5xx, in order. The takeaways fall out of it: short outages are tolerated (you’re still in “preserve”), prolonged outages cost you the index (you’ve reached “drop”), and the cure is simply returning 2xx (which moves you to “recover” — gradually).

3. The proportionality rule. The crawl slowdown is proportionate to how many URLs are erroring. One flaky URL is a nudge; a site-wide 5xx is a hard brake. This tells you where to triage: a small flagged set is low-urgency; a host-level error is an emergency.

4. 503 is a feature, not just an error. For planned downtime, 503 + Retry-After is the correct, SEO-safe response — because it’s temporary by design. But it’s a short-term tool: a few days at most. No status code protects you from the indexing damage of being down for weeks.

5. robots.txt is load-bearing. Treat your robots.txt as the one URL that must never 5xx. A 503 on robots.txt doesn’t just hide robots.txt — it pauses crawling for the whole site. Whatever maintenance rule you write, carve robots.txt out of it.

6. Reproduce as the bot, not as yourself. The nastiest 5xx causes (WAF rules, rate limiting) fire only for Googlebot. If the page loads fine in your browser but GSC says 5xx, you haven’t reproduced the problem yet — test with Googlebot’s user-agent and read the logs.

Check what status code a URL actually returns

The first move is confirming the response code from the command line — and, ideally, as Googlebot, since some 5xx/429s only fire for the bot.

macOS / Linux

# Status code as a normal client
curl -s -o /dev/null -w "%{http_code}\n" https://www.example.com/page/

# Status code as Googlebot (catches WAF / rate-limit rules that target the bot)
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://www.example.com/page/

# Full response headers — confirm a 503 carries a Retry-After during maintenance
curl -sI https://www.example.com/page/

Windows (PowerShell)

# Status code (note: 5xx throws, so capture the response in the catch block)
try {
  (Invoke-WebRequest -Uri "https://www.example.com/page/" -UseBasicParsing).StatusCode
} catch {
  $_.Exception.Response.StatusCode.value__
}

# As Googlebot
try {
  (Invoke-WebRequest -Uri "https://www.example.com/page/" -UseBasicParsing `
    -UserAgent "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)").StatusCode
} catch { $_.Exception.Response.StatusCode.value__ }

Confirm the traffic was really Googlebot (before trusting your logs)

A “Googlebot” 5xx in your logs might be a spoofed user-agent. Verify with a reverse + forward DNS check — Google publishes no shortcut.

macOS / Linux

# 1) Reverse DNS the IP from your logs — it should end in googlebot.com or google.com
host 66.249.66.1
# 2) Forward DNS that hostname back — it must resolve to the same IP
host crawl-66-249-66-1.googlebot.com

Windows

nslookup 66.249.66.1
nslookup crawl-66-249-66-1.googlebot.com

Maintenance mode done right — 503 + Retry-After, robots.txt stays 200

The critical bit: serve 503 for the site, but never for robots.txt.

Apache (.htaccess)

# Let robots.txt through untouched so crawling isn't halted
RewriteEngine On
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteCond %{REQUEST_URI} !^/maintenance\.html$
RewriteRule ^ /maintenance.html [R=503,L]

# Attach the headers to the 503 response
<If "%{REQUEST_URI} != '/robots.txt'">
  Header always set Retry-After "86400"
</If>
ErrorDocument 503 /maintenance.html

Nginx

location / {
    # robots.txt is matched by a more specific block below, so it's exempt
    return 503;
}

location = /robots.txt {
    # Always serve the real robots.txt with a 200
    try_files $uri =404;
}

error_page 503 /maintenance.html;
location = /maintenance.html { internal; }

# Add Retry-After to 503 responses (a day, in seconds)
add_header Retry-After 86400 always;

Confirm it with curl -sI from the section above: the site URLs should return 503 with a Retry-After header, and https://www.example.com/robots.txt should still return 200.

Tools for finding and confirming a 5xx

Start with the tools that can check status directly against your flagged URLs, then narrow to logs if the answer isn’t obvious from a single request.

• Bulk HTTP Status Code Checker — paste every URL Search Console flagged under “Server error (5xx)” (up to 500 at once) and see the actual status code, redirect chain, and latency for each one right now. This is the fastest way to confirm whether a fix really took before you click Validate Fix in GSC, or whether some URLs are still erroring.
• Website Down Checker — a single-URL “is it actually down right now” check from a live vantage point, with response timing and redirects. Good for the first step in this article’s diagnosis flow: “visit a flagged page yourself” — before you go digging in logs.
• HTTP Header Checker — inspect the full response headers across every redirect hop. Use it to confirm a 503 actually carries a Retry-After header during planned maintenance, and to check that robots.txt is still answering 200 with no maintenance rule accidentally catching it.
• Log File Analyzer — drop in your server access log and see status-code waste and crawl activity by bot, plus a spoofer report that separates verified Googlebot from user-agents faking it. This is the tool for the trickiest cause in this article: a CDN/WAF or rate limiter that returns 5xx or 429 only to Googlebot while real visitors see the site fine — something a browser test alone won’t catch.

From Google and third-party crawlers:

• Search Console — Crawl Stats report (Settings → Crawl stats) is the whole-site view of host availability and the by-response-code breakdown covered in the Advanced tab.
• Screaming Frog SEO Spider / Ahrefs Site Audit can crawl your site the way Googlebot would and surface which URLs are erroring at scale, beyond the handful Search Console has already sampled.

Mistakes that turn a fixable 5xx into an indexing problem

• Serving a 200 “we’ll be back soon” page during maintenance instead of a 503. Google treats whatever comes back with a 200 as the real page and may index your maintenance message in place of your actual content. Serve a 503 instead — its content is ignored, which is what you want.
• Returning a 404, 410, or 403 during planned downtime. These say “gone” or “forbidden,” not “temporarily unavailable.” Google’s guidance is explicit: don’t block a paused site this way. A 503 preserves the page’s standing in the index; a 404/410 doesn’t.
• Letting maintenance mode return 503 for robots.txt too. A blanket rule that 503s everything, robots.txt included, pauses crawling for the entire site — not just the pages you meant to take down. Carve robots.txt out of the maintenance handler so it always answers 200.
• Assuming a page that loads fine right now rules out an earlier Googlebot-only 5xx. A CDN/WAF or rate limiter can return 5xx or 429 to Googlebot specifically while every human visitor sees a normal 200 — and the fact it works for you this minute doesn’t tell you what it returned when Google actually hit it. Test with Googlebot’s user-agent, read the logs, and confirm the traffic is really Googlebot (reverse DNS/IP ranges) before you trust the user-agent header alone.
• Leaving a “temporary” maintenance window up for weeks. A 503 is only correct for a short outage — Google’s own wording is “a few days at most.” Even a technically-correct 503 response causes indexing harm if the site stays down for weeks, and Google is explicit that there’s no fixed recovery time or way to speed one up after that.

Confirming a 5xx fix actually took

Each test below checks one specific claim after you believe the server error is resolved. Run them in order — the first two catch most false “it’s fixed” calls.

Test 1 — The flagged URLs return 200 right now

• Test to run: Run the Bulk HTTP Status Code Checker against every URL Search Console flagged, or curl -sI each one.
• Expected result: A clean 200 on every URL, consistently across repeat requests, not intermittent.
• Failure interpretation: Any URL still returning 5xx means the underlying server/host issue isn’t resolved; a mix of 200s and 5xxs across repeats points at an overloaded or unstable upstream, not a one-time bug.
• Monitoring window: Immediate — a single clean pass is enough to move on to clicking Validate Fix.
• Rollback trigger: Any 5xx recurrence before you click Validate Fix — hold off and keep diagnosing instead of validating a fix that hasn’t stuck.

Test 2 — Google’s own re-crawl clears it

• Test to run: Open the Server error (5xx) issue in the Page Indexing report and click Validate Fix.
• Expected result: The validation state moves to “Validation passed” as Google re-crawls the affected URLs and confirms 2xx.
• Failure interpretation: A “Validation failed” result on specific URLs — even after your own checks looked clean — usually means those URLs are still erroring for Googlebot specifically, which points back at a bot-targeted WAF/rate-limit rule.
• Monitoring window: Google re-crawls the flagged set in batches on its own schedule; there’s no published fixed cadence for this, so check back periodically rather than expecting a specific turnaround date. Validating isn’t even required — Google updates the issue count whenever it recrawls a page with known issues, validation or not — but it does prioritize the affected set and gives you a status to track.
• Rollback trigger: Any URL that gets re-flagged under Server error (5xx) after previously passing validation.

Test 3 — Crawl Stats host availability recovers

• Test to run: Search Console → Settings → Crawl stats → host status.
• Expected result: The host availability graph returns to normal, with no new red flags in the by-response-code breakdown.
• Failure interpretation: A graph that’s still showing red after you believe the fix landed means Google’s crawlers are still hitting errors — the fix hasn’t reached whatever Googlebot is actually requesting.
• Monitoring window: Allow a few days after the fix for Crawl Stats to reflect it; it’s a rolling report, not real-time.
• Rollback trigger: Host status flipping back to red after a period of green.

Test 4 — robots.txt still returns 200

• Test to run: curl -sI on your robots.txt URL, or run it through the HTTP Header Checker.
• Expected result: 200, exactly as it did before the incident or maintenance window.
• Failure interpretation: A 503 (or any non-200) on robots.txt means you’re blocking crawling for the whole site, not just the URLs you meant to affect.
• Monitoring window: Immediate.
• Rollback trigger: Any non-200 response on robots.txt — treat this as the single highest-priority thing to fix first.

Test 5 — Googlebot gets the same response a browser does

• Test to run: Request the URL with Googlebot’s user-agent (see the Scripts tab) and cross-check against the Log File Analyzer’s verified-Googlebot vs. spoofer report.
• Expected result: Googlebot receives the same 200 a normal browser request gets.
• Failure interpretation: A status mismatch between the Googlebot-UA request and a normal browser request means a WAF, rate limiter, or bot-detection rule is targeting the bot specifically.
• Monitoring window: Immediate.
• Rollback trigger: Any mismatch between what Googlebot receives and what real traffic receives.

The standing KPIs for server health

These aren’t a one-time fix check (that’s the Validation Tests tab) — they’re what to keep an eye on quarter over quarter so a 5xx pattern doesn’t build up unnoticed.

5xx rate in your server/access logs

• What it tells you: The ground-truth share of requests actually failing at the server, independent of what Search Console has gotten around to sampling and reporting.
• How to pull it: The Log File Analyzer — drop in your access log and see status-code waste, filterable to Googlebot.
• Benchmark / realistic range: There’s no universal healthy percentage here — establish your own baseline from a known-good period and watch for a rise against it, rather than chasing an invented target number.
• Cadence: Check after any deploy or traffic spike; a weekly glance is enough otherwise for most sites.

GSC Crawl Stats — host status by response code

• What it tells you: Google’s own recent read on how often its crawlers hit errors on your host, independent of your own logs.
• How to pull it: Search Console → Settings → Crawl stats.
• Benchmark / realistic range: Same honesty rule applies — track your own trend line rather than a fixed percentage; treat any new red flag in the host status section as worth investigating, whatever the number.
• Cadence: A weekly glance, or immediately after any known outage or maintenance window.

Uptime / origin availability

• What it tells you: Whether the origin server itself is reachable at all — upstream of whatever status code Googlebot ends up seeing.
• How to pull it: An uptime monitor for continuous coverage, or an ad hoc check with the Website Down Checker.
• Benchmark / realistic range: Depends entirely on your hosting tier and any SLA you’re on — there’s no universal “good” uptime figure to borrow; use your own host’s SLA, or your own historical baseline, as the reference point.
• Cadence: Continuous if you can, or at minimum a check after any known incident.

Server error (5xx) count in the Page Indexing report

• What it tells you: How many URLs Google currently has flagged as erroring — the lagging, official record, as opposed to the real-time server/log view.
• How to pull it: Search Console → Page Indexing → Not indexed → Server error (5xx).
• Benchmark / realistic range: Zero is the honest target for any URL you want indexed. Anything above zero is worth running through this article’s diagnosis steps, whatever the count.
• Cadence: Whenever you check Search Console generally, and always after a known incident or deploy.

AI prompts for 5xx diagnosis and maintenance-mode review

Two prompts built around the specific tasks this article walks through — paste your own data into either one.

Prompt: spot patterns in raw log lines around a 5xx incident

Here are raw server/access log lines from around the time my site started
returning 5xx errors:

<paste log lines>

From only what's in these lines, tell me:
1. Which HTTP status codes appear, and in what proportion.
2. Whether requests from Googlebot's IP ranges or user-agent get a different
   status code than other traffic (this could indicate a WAF or rate limiter
   targeting the bot specifically).
3. Any timing pattern — recurring at a fixed interval, clustered around a
   traffic spike, or starting right after a specific timestamp.

Don't guess at a root cause you can't see in the data — only report what the
log lines actually show.

Prompt: review a maintenance-mode config for the robots.txt trap

Here is my maintenance-mode configuration (nginx/Apache/other):

<paste config>

Check specifically whether this configuration:
1. Returns a 503 status code for regular site pages during maintenance.
2. Excludes /robots.txt from the 503 rule, so robots.txt keeps returning 200.
3. Sets a Retry-After header on the 503 responses.

Flag anything that would cause robots.txt to return a non-200 status, and
anything missing a Retry-After header.

Test yourself: Server error (5xx)

Five questions on what “Server error (5xx)” means, how Google reacts to it, and the right way to take a site down on purpose. Pick an answer for each, then check.

Server error (5xx)

A Page Indexing status in Google Search Console meaning Googlebot requested a URL and the server returned a 500-level HTTP error (500, 502, 503, 504) instead of a 200, so the page can't be indexed.

Related: Crawl Stats report, Crawl rate, Crawling

Server error (5xx)

Server error (5xx) is a “Not indexed” status in Google Search Console’s Page Indexing report. It means Googlebot requested the URL and the server replied with a 500-level HTTP status code — most often 500 (internal server error), 502 (bad gateway), 503 (service unavailable), or 504 (gateway timeout) — instead of a successful 200. Google couldn’t retrieve usable content, so the page isn’t in the index, or won’t stay there.

It matters because a 5xx is a “something broke on the server” signal, and Google reacts to it. Repeated 5xx errors make Googlebot slow its crawl rate, any content returned with a 5xx is ignored, and pages that were already indexed are preserved at first but eventually dropped if the errors persist. Crawling recovers on its own once the server starts returning 2xx again — gradually, not instantly.

The one nuance worth knowing: 503 (Service Unavailable) is the correct response for short, planned downtime, because it tells crawlers “temporary, come back later” rather than “gone.” Google’s default preference is to keep a site online with limited functionality rather than fully disable it; full disable is framed as an urgent, few-days-at-most measure with no fixed recovery time if you overstay it. Your robots.txt file must keep returning 200 throughout — a 503 on robots.txt can pause crawling for the whole site.

Related: Crawl Stats report, Crawl rate, Crawling

Changelog