500 Internal Server Error

TL;DR — A 500 Internal Server ErrorA 500 Internal Server Error is a generic HTTP status code — RFC 9110 defines it as an unexpected condition that stopped the server from fulfilling the request, and nothing more. The request may have been fine — something broke server-side while generating the response. For SEO, an isolated 500 is typically retried, but persistent, site-wide 500s get Google's documented response: slower crawling and, if the errors don't clear, eventual removal from the index. means the server broke while trying to build the page — it’s not a problem with your URL, your browser, or the search crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. A one-off 500 typically gets retried by Google, and there’s no rule saying it costs you anything. The documented danger is when lots of your pages throw 500s for a while — that’s when Google slows down crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and, eventually, can drop pages from search. Start fixing it by looking at your server’s error logs, not the browser.

What a 500 error is

A 500 Internal Server Error is the web’s version of “something went wrong and I can’t tell you exactly what.” The server received your request, started to build the page, hit a problem, and gave up — returning a generic error instead of the page. In the Ahrefs HTTP status codesAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. guide I wrote it this way: the server “encounters some kind of issue and doesn’t have a better or more specific error code.” Evidence for this claim A 500 response means the server encountered an unexpected condition that prevented it from fulfilling the request. Scope: RFC 9110 defines the generic response semantics; it does not diagnose the underlying server fault. Confidence: high · Verified: 2026-07-14 IETF: RFC 9110 §15.6.1 — 500 Internal Server Error

That’s the key thing to understand: a 500 is a catch-all. It tells you something broke on the server side. It does not tell you what. Figuring out the “what” is the whole job.

It’s a server problem, not your problem (usually)

500 errors are in the 5xx family — server errors. That’s different from 4xx errors like the 404 (page not found404 Not Found is the HTTP client-error status code a server returns when it can't find the requested URL — RFC 9110 defines it as no current representation, or unwillingness to disclose one. A \"hard 404\" actually returns the 404 status; a \"soft 404\" returns a success code (like 200) for a page that's really gone. 404s are normal and expected: the fact that some URLs 404 doesn't affect your site's other, successful pages, and Google de-indexes 404'd URLs over time (probably retrying for some period, less and less often).), which are about the request (a bad or missing URL). With a 500, the URL might be perfectly fine; the server just couldn’t finish the job. You’ll see cousins of the 500 too — 502, 503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime., and 504A 504 Gateway Timeout is an HTTP 5xx server error a gateway or proxy (a CDN, load balancer, or reverse proxy) returns when it doesn't get a timely response from the upstream server behind it. Unlike a 502 (a bad/garbled response) or a 503 (server explicitly unavailable), a 504 means no response arrived in time. For SEO, isolated 504s are retried and tolerated, but sustained 504s and timeouts make Googlebot slow its crawl and eventually drop pages from the index. — which are also server-side but point at more specific situations (a bad gateway, a server that’s temporarily unavailable, or a timeout).

Does a 500 error hurt SEO?

A single, occasional 500 on one page? Google will typically come back and try again, and if the page loads fine on retry, that’s usually the end of it. Google doesn’t publish a blanket promise that an isolated failure is harmless — it just doesn’t document any mechanism that penalizes a one-off blip either.

The real, documented risk is persistent 500s across lots of pages. When that happens:

1. Google keeps retrying, sees the errors continuing, and slows down how fast it crawls your site.
2. If the errors still don’t clear, Google eventually drops those pages from the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — meaning they stop showing up in search. Evidence for this claim Google reduces crawling in response to 5xx errors and eventually removes persistently failing URLs from its index. Scope: Google documents the general 5xx progression; it does not provide a guaranteed retry or recovery timeline for an individual URL. Confidence: high · Verified: 2026-07-14 Google: How HTTP status codes affect Google's crawlers

The good news: Google’s own documentation describes gradual recovery once the underlying problem is fixed and crawls start succeeding again — but it doesn’t guarantee a timeline or an outcome. “Recovery is usually quick” is the common case, not a promise.

How to start fixing it

• Check your server’s error logs first. They’re where the real reason lives — not the browser. A browser just shows you “500”; the logs show you why.
• Look at what changed recently. A new plugin, theme, or module? A recent code deploy? An edit to a config file like .htaccess? Recent changes are the usual suspects.
• Check Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.. The Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. has a “Server error (5xx)” section that shows which of your URLs Google is seeing 500s on.
• Ask your host. On shared hosting especially, a lot of 500s come from hitting memory or resource limits — your host can often see it from their side.

Want the deeper version — exactly how Google escalates, the ~1% rule of thumb, the 500-vs-503 distinction, and a full diagnosis order? Switch to the Advanced tab.

TL;DR — RFC 9110 defines 500 as an unexpected condition that stopped the server from fulfilling the request — that’s the entire boundary of the status code itself; cause, duration, and retry-worthiness are diagnosis, not semantics. Google’s documented reaction is graduated: isolated 500s are typically retried; persistent, site-wide 500s get slower crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and, if unresolved, eventual removal from the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Mueller has offered a rough, personal rule of thumb — error rates above roughly 1% probably mean something’s broken — but that’s not a documented Google threshold, and the retry → slow-crawl → drop sequence describes documented behaviors, not a fixed lockstep timer. Diagnose from server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. first, then GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s Server error (5xx) report and Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). “by response.” The 500-vs-503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. distinction matters: 503 is the sanctioned “come back later” code with a ~2-day grace window; an uncontrolled 500 gets no such grace.

What a 500 actually is

Start with the spec, not the practitioner shorthand. RFC 9110 — the HTTP semantics standard — defines a 500 Internal Server ErrorA 500 Internal Server Error is a generic HTTP status code — RFC 9110 defines it as an unexpected condition that stopped the server from fulfilling the request, and nothing more. The request may have been fine — something broke server-side while generating the response. For SEO, an isolated 500 is typically retried, but persistent, site-wide 500s get Google's documented response: slower crawling and, if the errors don't clear, eventual removal from the index. as an unexpected condition that prevented the server from fulfilling the request. That’s the entire boundary of what the status code itself tells you. It does not identify the root cause, the failing component, how long the problem will last, whether the same request would succeed on retry, or whether recovery is likely. Everything past “the server hit something it couldn’t handle” is diagnosis, not status-code semantics — and diagnosis lives in your server error logs, not in the spec or in the browser.

In the Ahrefs HTTP status codesAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. guide I keep the practitioner-facing definition deliberately blunt: the server “encounters some kind of issue and doesn’t have a better or more specific error code.” That’s a plain-language gloss of the same RFC boundary — still a catch-all, still a symptom rather than a diagnosis.

It sits in the 5xx family alongside 502 (bad gateway), 503 (service unavailable), and 504A 504 Gateway Timeout is an HTTP 5xx server error a gateway or proxy (a CDN, load balancer, or reverse proxy) returns when it doesn't get a timely response from the upstream server behind it. Unlike a 502 (a bad/garbled response) or a 503 (server explicitly unavailable), a 504 means no response arrived in time. For SEO, isolated 504s are retried and tolerated, but sustained 504s and timeouts make Googlebot slow its crawl and eventually drop pages from the index. (gateway timeout) — all server-side, but the 500 is the one that means “no better code applies.” Because the status code itself carries none of the diagnostic detail, a browser reload tells you nothing about why it happened; your server error logs are the ground truth.

How Googlebot treats a 500

Google’s crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. is polite by design — it throttles to your server’s health, and 5xx responses are one of the signals it reads as “slow down.” Google’s current documentation confirms the shape of that response: 5xx and 429 responses cause a temporary crawl-rate reduction (scaled to how many URLs are affected), and URLs that keep failing can eventually be removed from the index, while already-indexed content is preserved in the meantime pending a successful refresh. Evidence for this claim Google reduces crawling in response to 5xx errors and eventually removes persistently failing URLs from its index. Scope: Google documents the general 5xx progression; it does not provide a guaranteed retry or recovery timeline for an individual URL. Confidence: high · Verified: 2026-07-14 Google: How HTTP status codes affect Google's crawlers John Mueller has described that same progression in his own words, in a Google SEO Office Hours session relayed via Search Engine Journal:

“We don’t have any strong thresholds on that. But essentially what happens with 500 errors is we’ll try to retry them. And if we continue to see …the 500 errors then we will …slow down crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. And if we continue to see that there are 500 errors then we will drop those URLs from the index.”

Read that as a description of documented behaviors — retry, slower crawling, possible removal — not a fixed three-step timer with guaranteed transitions or timing; Google doesn’t publish exact thresholds for when one stage becomes the next. An isolated 500 on one URL is typically retried, and the next successful fetch is usually the end of it — but that’s a description of the common case, not a guarantee that a one-off failure carries zero cost. The real, documented risk lives in errors that don’t clear.

Why site-wide 500s are worse than isolated ones

There’s a nastier dynamic when a large share of your site 500s at once. Google’s current documentation confirms the crawl-rate cut scales with how many URLs are affected — the more of your site that’s failing, the more crawling slows. Mueller has put the reasoning behind that in more specific terms, framing it as Google suspecting its own crawling might be part of the overload:

“But if a large part of a site consistently has 500 errors and we might assume that maybe we’re causing the problem and we’ll slow down crawling of the whole site and at some point we’ll say well, it looks like these pages are really gone, we’re going to drop them.”

Treat that specific “we assume we’re causing it” causal framing as Mueller’s own characterization rather than wording pulled verbatim from Google’s current official docs — the underlying mechanism is documented (more failing URLs, more crawl-rate reduction), even though the exact causal reasoning is easier to verify from his own statement than from the docs themselves. Either way, the practical feedback loop is worth internalizing: aggressive crawling under resource exhaustion can trigger more 500s → Google backs off crawling of the whole site → and if the errors persist anyway, pages drop. It also means a 500 problem isn’t always a code bug — sometimes it’s your server buckling under concurrent load that only bites under crawler or traffic spikes.

How much is “too much”?

There’s no hard line — Google’s own troubleshooting documentation doesn’t publish an error-rate threshold at all. Mueller has offered a rough, personal rule of thumb in SEO Office Hours (again, relayed via Search Engine Journal, not an official Google publication):

“My feeling is if you’re seeing something more than one percent then that sounds like something is kind of broken.”

Treat ~1% as an unofficial smell test attributed to Mueller, not a documented or enforced Google limit. Below it you’re probably fine; above it, it’s worth investigating — but don’t treat crossing 1% as an automatic trigger, and don’t treat staying under it as a guarantee either. The one number Google commits to publicly is the absence of one: “we don’t have any strong thresholds.”

500 vs. 503: the distinction that matters

This is where a lot of people get it wrong. A 503 Service Unavailable503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. is the sanctioned way to tell a crawler “I’m temporarily down, come back later.” Google treats it as intentional and gives it a grace window. Google’s own crawling troubleshooting doc is explicit:

“Return 503 or 429 HTTP response status codes temporarily for GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. requests when your server is overloaded. Googlebot will retry these URLs for about 2 days. Note that returning ‘no availability’ codes for more than a few days will cause Google to permanently slow or stop crawling URLs on your site.”

The contrast: 503 is intentional and gets a ~2-day retry grace; an uncontrolled 500 is unintentional and gets no such grace — it’s simply retried until Google gives up. Practical upshot: for planned maintenance or a deliberate overload defense, return a 503 (ideally with a Retry-After header), not a 500 and not a 200 error page. Never dress a real outage up as a 200.

How to diagnose a 500

Work through this in layers — application/code, platform/CMSA content management system (CMS) is software that lets users create, manage, and publish digital content — like blog posts and pages — without writing raw code. WordPress, Drupal, and Joomla are the most common open-source CMS platforms., infrastructure and resources, then configuration — cheapest and most-likely-to-pay-off first. Steps marked (WordPress-specific) are common WordPress practice, not universal fixes; adapt them to whatever your actual stack is.

1. Server error logs. error.log / access.log (or your platform’s log viewer). Match timestamps to the failing requests. This is where the actual stack trace, PHP fatal error, or DB connection failure shows up. Everything downstream is guessing until you’ve read these.
2. GSC — Server error (5xx) report. Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s Page IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. report flags the URLs Google itself is seeing 500s on. Then open Crawl Stats and read the “by response” breakdown over time — that’s how you tell a transient blip from a real, sustained availability problem.
3. Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility.. Its crawl error alerts group Server Errors (5xx) and point you at the specific URLs and the Crawl Information tool.
4. Reproduce as a bot, not just a browser. A page can 500 for Googlebot while loading fine for you — under crawl-triggered load, bot-detection/firewall misconfiguration, or capacity limits that only bite under concurrent bot traffic. Use URL Inspection (GSC), Fetch as BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share., or curl with a bot user-agent to catch bot-only failures. “It works in my browser” is not the all-clear.
5. Plugin / theme / module conflicts (WordPress-specific pattern; adapt elsewhere). Take a backup first — always have a way back before you start disabling things. Then disable extensions and re-enable one at a time to isolate the offender, checking file permissions/ownership on anything you touch. Hosting vendors’ own WordPress troubleshooting guides report this pattern often, but that’s a stack-specific observation, not evidence it’s the leading cause everywhere — on a different CMS or a custom app, the equivalent is a third-party module, package, or middleware conflict.
6. Resource exhaustion. PHP memory limits, database connection limits, shared hosting capacity, traffic or crawl spikes. Your host can often confirm this from their side.
7. Configuration & recent changes. A broken .htaccess, a bad server-config edit, a recent deploy, or wrong database credentials. Recent changes are the highest-yield place to look.

How to fix it (match the fix to the cause)

Match the fix to whichever layer the diagnosis pointed at. These are common patterns reported across practitioner write-ups, not a ranked or universal list of what’s most likely on your specific stack:

• Config/deploy caused it → roll back the change; fix the .htaccess, config, or credentials.
• Resource exhaustion → raise limits (PHP memory, DB connections) or upgrade the hosting tier; if crawling is triggering the overload, that’s also a crawl-rate conversation.
• Plugin/module conflict → remove or replace the offending extension.
• Code bug → fix the code, add the missing error handling.
• You can’t tell → escalate to your host with the exact timestamps and the log lines. Don’t guess in production.

Preventing recurrence

Monitoring and alerting on 5xx rates, staging changes before they hit production, load testing ahead of known traffic spikes, and — if Googlebot’s crawling is itself the trigger — managing crawl load (and returning 503/429 deliberately during real overload, rather than letting the server emit uncontrolled 500s).

FAQ

Does a 500 error hurt SEO? The documented risk is mainly about persistence at scale. Isolated 500s are typically retried, with no documented penalty for a one-off blip; sustained, site-wide 500s slow crawling and can lead to deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist..

How long before Google deindexes a page with a 500? There’s no fixed timeline. Google retries first and slows crawling; dropping from the index comes only if the errors continue. Fix it and pages generally come back once crawls succeed again.

Why does my site 500 for Googlebot but load fine in my browser? Bot-only 500s usually mean capacity/bot-handling issues — crawl-triggered load, firewall/bot rules, or limits that only bite under concurrent bot traffic. Trust the logs, not a manual browser check.

Can 500s slow crawling of my whole site, not just the affected pages? Yes — Google’s documentation confirms the crawl-rate cut scales with how many URLs are failing, so a large share of the site returning 500s slows crawling site-wide. Mueller has additionally framed the reasoning as Google suspecting its own crawling might be part of the overload — his own characterization, not wording pulled verbatim from the current official docs.

What causes a 500 in WordPress? On WordPress specifically, hosting vendors and the WordPress community most often report plugin/theme conflicts, a corrupt .htaccess, or hitting the PHP memory limit — that’s what gets reported for that platform, not a claim that these are the universal leading causes of every 500. The diagnosis order above (logs first, then what changed) is the same regardless of CMS.

Is it safe to automatically retry a request after a 500? Only if you’ve checked the method and the request’s idempotency first — the 500 status itself doesn’t authorize a retry policy. GET, HEAD, PUT, and DELETE are generally safe to retry because they’re idempotent (repeating them shouldn’t cause extra side effects); a bare POST usually isn’t, unless your API explicitly guarantees idempotency (for example, via an idempotency key) — retrying it blindly risks a duplicate order, a duplicate email, or a double-charged payment. When you do retry, use exponential backoff with jitter, cap the number of attempts, and set a retry budget so a struggling server doesn’t get hit by a retry storm on top of whatever’s already failing.

AI summary

A condensed take on the Advanced version:

• 500 = RFC 9110’s unexpected-condition boundary. That’s the entire scope of the status code itself — no root cause, no duration, no retry-worthiness. It’s a symptom, not a diagnosis; root-cause it from server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened., not the browser. Distinct from 4xx (client/request) errors and from its 5xx siblings 502/503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime./504A 504 Gateway Timeout is an HTTP 5xx server error a gateway or proxy (a CDN, load balancer, or reverse proxy) returns when it doesn't get a timely response from the upstream server behind it. Unlike a 502 (a bad/garbled response) or a 503 (server explicitly unavailable), a 504 means no response arrived in time. For SEO, isolated 504s are retried and tolerated, but sustained 504s and timeouts make Googlebot slow its crawl and eventually drop pages from the index..
• Google’s documented reaction is graduated: temporary crawl-rate reduction (scaled to how many URLs are affected) and possible eventual indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. removal for URLs that keep failing, with recovery once fetches succeed again. Mueller describes this as retry → slow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. → drop from index — a description of documented behaviors, not a fixed lockstep timer. An isolated 500 is typically retried; there’s no promise it’s cost-free, but the real, documented risk is persistence at scale.
• Site-wide 500s are worse: the crawl-rate cut scales with how many URLs fail. Mueller frames the reasoning as Google suspecting its own crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. may be part of the overload — his own characterization, not verbatim official-docs wording — and either way it’s a feedback loop where crawl load can trigger more 500s.
• Rule of thumb, not a threshold: Mueller has said error rates above ~1% are “probably broken,” but that’s his personal framing from SEO Office Hours, not a documented Google limit — Google states it has no hard thresholds.
• 500 vs. 503: 503 (or 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content.) is the sanctioned “come back later” signal with a ~2-day retry grace per Google’s docs; an uncontrolled 500 gets no grace. Use 503 (with Retry-After) for planned downtime.
• Diagnosis order (by layer): server logs → GSC Server errorA Page Indexing status in Google Search Console meaning Googlebot requested a URL and the server returned a 500-level HTTP error (500, 502, 503, 504) instead of a 200, so the page can't be indexed. (5xx) + Crawl Stats “by response” → Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. → reproduce as a bot (URL Inspection / curl) → plugin/module conflicts (WordPress-specific pattern; adapt elsewhere, back up first) → resource exhaustion → config/deploy changes.
• Retry safety depends on the request, not the status code. Idempotent methods (GET/HEAD/PUT/DELETE) are generally safe to retry; a bare POST usually isn’t without an idempotency key. Use backoff, a retry cap, and a retry budget.
• Recovery is generally fast once fixed — dropped pages tend to return as crawls succeed again, though Google guarantees no timeline or outcome.

Official documentation

Primary-source documentation from the search engines and the HTTP spec.

Google

• Troubleshoot Google Search crawling errors — how Google handles server errors, and the sanctioned use of 503/429 for temporary overload.
• In-Depth Guide to How Google Search Works — the crawl scheduler and the fact that 5xx responses are read as “slow down.”
• Crawl Stats report — the “by response” breakdown (including Server error (5xx)) used to tell a blip from a sustained problem.
• Reduce the Googlebot crawl rate — the correct, deliberate way to slow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. (rather than letting the server emit uncontrolled 500s).

Bing / Microsoft

• Bing Webmaster Tools — List of crawl error alerts — how Bing groups Server Errors (5xx) and where to inspect the affected URLs.

HTTP spec / reference

• MDN — 500 Internal Server Error — the definition of the status code itself.
• RFC 9110 §15.6.1 — 500 Internal Server Error — the authoritative HTTP semantics.

Quotes from the source

On-the-record statements. Where a source page is JavaScript-rendered or was relayed through secondary coverage, that’s flagged with a <small> caveat.

Google — the escalation path

• “We don’t have any strong thresholds on that. But essentially what happens with 500 errors is we’ll try to retry them. And if we continue to see …the 500 errors then we will …slow down crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. And if we continue to see that there are 500 errors then we will drop those URLs from the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..” — John Mueller, Google. Read the coverage Relayed via Search Engine Journal’s transcript of a Google SEO Office Hours video; confirm exact wording against the source video before treating as final.
• “But if a large part of a site consistently has 500 errors and we might assume that maybe we’re causing the problem and we’ll slow down crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. of the whole site and at some point we’ll say well, it looks like these pages are really gone, we’re going to drop them.” — John Mueller, Google. Read the coverage Relayed via Search Engine Journal; verify verbatim against the original video.
• “My feeling is if you’re seeing something more than one percent then that sounds like something is kind of broken.” — John Mueller, Google, on a rough error-rate rule of thumb. Read the coverage Relayed via Search Engine Journal; verify verbatim against the original video.

Google — the sanctioned “slow down” signal (docs, verified)

• “Return 503 or 429 HTTP response status codes temporarily for GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. requests when your server is overloaded. GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. will retry these URLs for about 2 days. Note that returning ‘no availability’ codes for more than a few days will cause Google to permanently slow or stop crawling URLs on your site.” — Google Search Central docs. Jump to quote

Patrick Stox — the definition (Ahrefs, verified)

• “500 Internal Server ErrorA 500 Internal Server Error is a generic HTTP status code — RFC 9110 defines it as an unexpected condition that stopped the server from fulfilling the request, and nothing more. The request may have been fine — something broke server-side while generating the response. For SEO, an isolated 500 is typically retried, but persistent, site-wide 500s get Google's documented response: slower crawling and, if the errors don't clear, eventual removal from the index. – The server encounters some kind of issue and doesn’t have a better or more specific error code.” — my HTTP status codesAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. guide on the Ahrefs blog. Jump to quote

500-error triage checklist

Run top to bottom the moment 500s show up:

• Pulled the server error logs and foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. the actual error (stack trace / PHP fatal / DB failure) — not just the browser “500”.
• Checked what changed recently — new plugin/theme/module, recent deploy, .htaccess or server-config edit, changed DB credentials.
• Opened GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. → Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. → Server error (5xx) to see which URLs Google is seeing 500s on.
• Read GSC Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). “by response” over time to judge transient vs. sustained.
• Checked Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. crawl error alerts for the same URLs.
• Reproduced as a botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. (URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. / Fetch as BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. / curl with a bot user-agent) — not just in a browser.
• Ruled out resource exhaustion (PHP memory, DB connections, hosting capacity) with your host.
• Isolated any plugin/module conflict by disabling and re-enabling one at a time.
• Confirmed you’re not emitting uncontrolled 500s during planned downtime — use 503 (with Retry-After) instead.
• Verified error rate is back under the unofficial ~1% smell test (Mueller’s rule of thumb, not a documented Google threshold) once fixed, and that GSC shows crawls succeeding again.

Runbook: “The site is throwing 500s — what do I check first?”

A no-panic order of operations. Do these in sequence; stop when you’ve foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. and fixed the cause.

0. Scope it (2 minutes). Is it one URL, one template/section, or the whole site? One URL is low-stakes (Google retries). Site-wide is the emergency — that’s the pattern that makes Google slow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. of the whole site.

1. Read the server error logs. error.log first. Match timestamps to the failures. You’re looking for the real cause: a PHP fatal error, a DB connection failure, a segfault, an out-of-memory kill. Everything below is guessing until you’ve done this.

2. Correlate with “what changed.” Sort by recent changes: last deploy, a plugin/theme/module added or updated, an .htaccess edit, rotated DB credentials, a config push. Most 500s trace to a change in the last few hours or days. If you can, roll it back first, diagnose second — restoring service buys you time.

3. Confirm the search-engine view. GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. → Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. → Server error (5xx) for the affected URLs, then Crawl Stats → by response to see whether it’s a blip or sustained. Cross-check Bing Webmaster ToolsThe free, first-party consoles search engines give site owners — Google Search Console and Bing Webmaster Tools — to see how their pages are crawled, indexed, and ranked, and to fix problems. The legacy name is 'webmaster tools.' crawl error alerts. This tells you how urgent the SEO clock is.

4. Reproduce it the way the botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. sees it. If humans get the page fine but GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. 500s, test as a bot: URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version., Fetch as BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share., or curl -A "Googlebot" <url>. Bot-only 500s point at capacity, firewall/bot rules, or crawl-triggered load — a different fix than a code bug.

5. Bisect the usual suspects.

• Plugins/modules: disable all, re-enable one at a time until it breaks again.
• Resources: check PHP memory limit, DB connection ceiling, and hosting capacity with your host — especially if the 500s cluster around traffic or crawl spikes.
• Config: revert the .htaccess / server config to a known-good version.

6. If crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is the trigger, don’t just absorb it. When GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.’s own crawl volume is overloading you, the correct temporary lever is returning 503/429 (with Retry-After) — not letting the server emit uncontrolled 500s. Google honors 503 as “come back later” for ~2 days; a 500 gets no such grace.

7. Verify recovery. Error rate back under the unofficial ~1% smell test, logs clean, and GSC Crawl Stats showing successful fetches again. Dropped pages generally return as crawls succeed — no guaranteed timeline, but it’s usually fast.

Escalation pattern to keep in mind: occasional 500 → typically retried, low documented risk → persistent 500 → crawling slows → still persistent → URLs drop from the index. Google doesn’t publish exact timing for these transitions; your job is to break the chain before it gets that far.

Which server code should I be returning?

Use this when you’re deciding what to serve, or interpreting what you’re seeing.

Is the failure intentional (maintenance / deliberate overload defense)?

• Yes → Return 503 Service Unavailable503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime. with a Retry-After header. Google treats it as temporary and retries for ~2 days. Do not serve a 200 error page, and do not let it fall through to a 500.
• No (it’s an actual, unexpected failure) → continue.

Does everyone get the error, or only the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.?

• Everyone → It’s a code / config / DB problem. Go to the server error logs and the “what changed” list. Roll back recent changes.
• Only GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer./BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. → Suspect capacity, bot-detection/firewall rules, or crawl-triggered load. Reproduce as a bot; check server resources and any bot rules.

Is it one URL, or a large share of the site?

• One URL / occasional → Low urgency. Google retries; fix at leisure but confirm it’s genuinely isolated.
• Site-wide / persistent → Emergency. This is the pattern that makes Google slow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. of the whole site and eventually deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist.. Restore service first (roll back), root-cause second.

Is the error rate above ~1%?

• Yes → Worth investigating as a likely real problem — this is Mueller’s own unofficial rule of thumb, not a documented Google threshold.
• No → Probably fine — but keep monitoring the trend, not just the snapshot.

Prompt: correlate a 500 with logs and a deployment

Diagnose this HTTP 500 incident from the sanitized evidence I provide. Build a
timeline across deployment events, request IDs, access logs, application errors,
resource signals, and affected URL patterns. Rank likely causes by evidence, separate
the fastest service-restoration action from the root-cause fix, and give exact
validation and rollback checks. Do not invent missing stack traces or thresholds.

[PASTE TIMELINE, HEADERS, LOGS, AND RECENT CHANGES]

Prompt: turn a stack trace into a safe test plan

Explain this stack trace in plain language, identify the failing component and its
inputs, and propose the smallest reversible test that distinguishes code, dependency,
configuration, and resource-exhaustion causes. Include what evidence would falsify
each hypothesis and how to confirm the URL returns a stable non-5xx response afterward.
Redact secrets and do not suggest exposing debug output publicly.

[PASTE SANITIZED STACK TRACE]

Shell: sample a URL list for 5xx responses

Run this with one absolute URL per line in urls.txt.

while IFS= read -r url; do
  curl -sS -o /dev/null -w '%{http_code},%{time_total},%{url_effective}\n' "$url"
done < urls.txt

The output separates isolated routes from a broad failure and records latency without downloading response bodies.

PowerShell: export the same status sample

Get-Content .\urls.txt | ForEach-Object {
  $r = Invoke-WebRequest -Uri $_ -SkipHttpErrorCheck
  [PSCustomObject]@{ Status = $r.StatusCode; Url = $_ }
} | Export-Csv .\status-sample.csv -NoTypeInformation

Shell: count 5xx statuses in an access log

Adjust the status-field position to match your documented log format before relying on the result.

awk '$9 ~ /^5[0-9][0-9]$/ { count[$9]++ } END { for (code in count) print code, count[code] }' access.log

Tools for finding and diagnosing 500s

• Server error logs — error.log / access.log, or your host/platform’s log viewer. The single most important tool; the actual cause lives here.
• Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. — Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. — the “Server error (5xx)” section lists the URLs Google is seeing 500s on.
• GSC — Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). — the “by response” breakdown over time separates a transient blip from a sustained availability problem.
• GSC — URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. — fetch a single URL as Google to reproduce a botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-only 500.
• Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. — crawl error alerts group Server Errors (5xx) and point you at the Crawl Information tool.
• curl — reproduce with an arbitrary user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target.: curl -I -A "Googlebot" <url> to see the status code the way a bot does.
• CrawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. / site audits — Ahrefs Site Audit and Screaming Frog SEO Spider surface 5xx responses across the site and let you spot patterns (a whole template or section failing).
• Uptime / status monitoring — alerting on 5xx rates so you learn about a spike before Google does.

Resources worth your time

My related writing

• HTTP Status Codes: A Guide to How They Impact SEO & UX — my full status-codes reference, including the 500 definition and the broader 5xx-crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. framing.
• The Beginner’s Guide to Technical SEO — where server errors sit in the bigger technical picture.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and how server health feeds back into it. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From around the industry

• How 500 Error Codes Can Negatively Affect Indexing (Search Engine Journal) — Matt G. Southern’s write-up with John Mueller’s retry → slow-crawl → drop-from-indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. quotes and the ~1% rule of thumb.
• How to fix the “Server error (5xx)” error in Google Search Console (Search Engine Land) — a step-by-step GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.-report walkthrough.
• How To Fix “Server Error (5xx)” in Google Search Console (Onely) — what 5xx errorsThe 4xx (client error) and 5xx (server error) HTTP status codes a server returns instead of a successful 2xx — and how Google treats each class very differently. are, where to find them in GSC, and a fix path.
• How to Fix a 500 Internal Server Error on Your Site (Kinsta) — a detailed, WordPress-hosting-centric fix list (server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened., plugins/themes, PHP memory, .htaccess, permissions).
• 5xx Server Errors: A Guide to Finding & Fixing 5xx Issues (Lumar) — an enterprise/technical-audit angle strong on log-file and crawl-budget framing.
• How to Fix 5xx Server Error in Google Search Console (Sitechecker) — another GSC-report-walkthrough take.

Videos

• Google Search Central (YouTube) — the SEO Office Hours archive, where John Mueller’s server-error and crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. guidance (the retry → slow-crawl → drop-from-indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. framing quoted above) originates. Channel

Test yourself: 500 Internal Server Error

Five quick questions on what a 500 is and how it affects SEO. Pick an answer for each, then check.

500 Internal Server Error

A 500 Internal Server Error is a generic HTTP status code — RFC 9110 defines it as an unexpected condition that stopped the server from fulfilling the request, and nothing more. The request may have been fine — something broke server-side while generating the response. For SEO, an isolated 500 is typically retried, but persistent, site-wide 500s get Google's documented response: slower crawling and, if the errors don't clear, eventual removal from the index.

Related: HTTP Status Code, Crawling, Crawl Budget

500 Internal Server Error

A 500 Internal Server Error is a catch-all HTTP status codeAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. the web server returns when it ran into an unexpected condition and couldn’t complete the request. RFC 9110 defines it narrowly: an unexpected condition prevented the server from fulfilling the request — nothing about root cause, duration, or retry-worthiness. It’s a server-side failure — not a problem with the URL, the browser, or the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. making the request. As I put it in Ahrefs’ HTTP status codesAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. guide: “500 Internal Server Error – The server encounters some kind of issue and doesn’t have a better or more specific error code.”

The word to notice is generic. A 500 tells you something broke; it doesn’t tell you what. That’s why diagnosis always starts in your server’s error logs, not in the browser. Root causes reported by practitioners include a code bug, a plugin or module conflict (a common WordPress-specific pattern), resource exhaustion (PHP memory limits, database connection limits, shared-hosting capacity), a bad .htaccess or server-config change, or a failed deploy — but that’s what gets reported for particular stacks, not evidence of a universal ranking of causes.

For SEO, the risk is graduated, not instant. An isolated 500 is typically retried by Google, with no documented penalty for a one-off blip. But if a large share of a site consistently returns 500s, Google’s crawl-rate reduction scales with how many URLs are affected — Google slows crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and, if the errors don’t clear, eventually drops the affected URLs from the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. (John Mueller has separately framed the reasoning as Google suspecting its own crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. might be part of the overload — his own characterization, not wording from Google’s current official docs.) It’s distinct from its 5xx siblings (502, 503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime., 504A 504 Gateway Timeout is an HTTP 5xx server error a gateway or proxy (a CDN, load balancer, or reverse proxy) returns when it doesn't get a timely response from the upstream server behind it. Unlike a 502 (a bad/garbled response) or a 503 (server explicitly unavailable), a 504 means no response arrived in time. For SEO, isolated 504s are retried and tolerated, but sustained 504s and timeouts make Googlebot slow its crawl and eventually drop pages from the index.): 503 in particular is the intentional, Google-sanctioned “come back later” signal, which is why using the right code during planned downtime matters.

Related: HTTP Status Code, Crawling, Crawl Budget

Changelog