500 Internal Server Error

TL;DR — A 500 Internal Server Error means the server broke while trying to build the page — it’s not a problem with your URL, your browser, or the search crawler. A one-off 500 typically gets retried by Google, and there’s no rule saying it costs you anything. The documented danger is when lots of your pages throw 500s for a while — that’s when Google slows down crawling and, eventually, can drop pages from search. Start fixing it by looking at your server’s error logs, not the browser.

What a 500 error is

A 500 Internal Server Error is the web’s version of “something went wrong and I can’t tell you exactly what.” The server received your request, started to build the page, hit a problem, and gave up — returning a generic error instead of the page. In the Ahrefs HTTP status codes guide I wrote it this way: the server “encounters some kind of issue and doesn’t have a better or more specific error code.” Evidence for this claim A 500 response means the server encountered an unexpected condition that prevented it from fulfilling the request. Scope: RFC 9110 defines the generic response semantics; it does not diagnose the underlying server fault. Confidence: high · Verified: 2026-07-14 IETF: RFC 9110 §15.6.1 — 500 Internal Server Error

That’s the key thing to understand: a 500 is a catch-all. It tells you something broke on the server side. It does not tell you what. Figuring out the “what” is the whole job.

It’s a server problem, not your problem (usually)

500 errors are in the 5xx family — server errors. That’s different from 4xx errors like the 404 (page not found), which are about the request (a bad or missing URL). With a 500, the URL might be perfectly fine; the server just couldn’t finish the job. You’ll see cousins of the 500 too — 502, 503, and 504 — which are also server-side but point at more specific situations (a bad gateway, a server that’s temporarily unavailable, or a timeout).

Does a 500 error hurt SEO?

A single, occasional 500 on one page? Google will typically come back and try again, and if the page loads fine on retry, that’s usually the end of it. Google doesn’t publish a blanket promise that an isolated failure is harmless — it just doesn’t document any mechanism that penalizes a one-off blip either.

The real, documented risk is persistent 500s across lots of pages. When that happens:

1. Google keeps retrying, sees the errors continuing, and slows down how fast it crawls your site.
2. If the errors still don’t clear, Google eventually drops those pages from the index — meaning they stop showing up in search. Evidence for this claim Google reduces crawling in response to 5xx errors and eventually removes persistently failing URLs from its index. Scope: Google documents the general 5xx progression; it does not provide a guaranteed retry or recovery timeline for an individual URL. Confidence: high · Verified: 2026-07-14 Google: How HTTP status codes affect Google's crawlers

The good news: Google’s own documentation describes gradual recovery once the underlying problem is fixed and crawls start succeeding again — but it doesn’t guarantee a timeline or an outcome. “Recovery is usually quick” is the common case, not a promise.

How to start fixing it

• Check your server’s error logs first. They’re where the real reason lives — not the browser. A browser just shows you “500”; the logs show you why.
• Look at what changed recently. A new plugin, theme, or module? A recent code deploy? An edit to a config file like .htaccess? Recent changes are the usual suspects.
• Check Google Search Console. The Page Indexing report has a “Server error (5xx)” section that shows which of your URLs Google is seeing 500s on.
• Ask your host. On shared hosting especially, a lot of 500s come from hitting memory or resource limits — your host can often see it from their side.

Want the deeper version — exactly how Google escalates, the ~1% rule of thumb, the 500-vs-503 distinction, and a full diagnosis order? Switch to the Advanced tab.

TL;DR — RFC 9110 defines 500 as an unexpected condition that stopped the server from fulfilling the request — that’s the entire boundary of the status code itself; cause, duration, and retry-worthiness are diagnosis, not semantics. Google’s documented reaction is graduated: isolated 500s are typically retried; persistent, site-wide 500s get slower crawling and, if unresolved, eventual removal from the index. Mueller has offered a rough, personal rule of thumb — error rates above roughly 1% probably mean something’s broken — but that’s not a documented Google threshold, and the retry → slow-crawl → drop sequence describes documented behaviors, not a fixed lockstep timer. Diagnose from server logs first, then GSC’s Server error (5xx) report and Crawl Stats “by response.” The 500-vs-503 distinction matters: 503 is the sanctioned “come back later” code with a ~2-day grace window; an uncontrolled 500 gets no such grace.

What a 500 actually is

Start with the spec, not the practitioner shorthand. RFC 9110 — the HTTP semantics standard — defines a 500 Internal Server Error as an unexpected condition that prevented the server from fulfilling the request. That’s the entire boundary of what the status code itself tells you. It does not identify the root cause, the failing component, how long the problem will last, whether the same request would succeed on retry, or whether recovery is likely. Everything past “the server hit something it couldn’t handle” is diagnosis, not status-code semantics — and diagnosis lives in your server error logs, not in the spec or in the browser.

In the Ahrefs HTTP status codes guide I keep the practitioner-facing definition deliberately blunt: the server “encounters some kind of issue and doesn’t have a better or more specific error code.” That’s a plain-language gloss of the same RFC boundary — still a catch-all, still a symptom rather than a diagnosis.

It sits in the 5xx family alongside 502 (bad gateway), 503 (service unavailable), and 504 (gateway timeout) — all server-side, but the 500 is the one that means “no better code applies.” Because the status code itself carries none of the diagnostic detail, a browser reload tells you nothing about why it happened; your server error logs are the ground truth.

How Googlebot treats a 500

Google’s crawler is polite by design — it throttles to your server’s health, and 5xx responses are one of the signals it reads as “slow down.” Google’s current documentation confirms the shape of that response: 5xx and 429 responses cause a temporary crawl-rate reduction (scaled to how many URLs are affected), and URLs that keep failing can eventually be removed from the index, while already-indexed content is preserved in the meantime pending a successful refresh. Evidence for this claim Google reduces crawling in response to 5xx errors and eventually removes persistently failing URLs from its index. Scope: Google documents the general 5xx progression; it does not provide a guaranteed retry or recovery timeline for an individual URL. Confidence: high · Verified: 2026-07-14 Google: How HTTP status codes affect Google's crawlers John Mueller has described that same progression in his own words, in a Google SEO Office Hours session relayed via Search Engine Journal:

“We don’t have any strong thresholds on that. But essentially what happens with 500 errors is we’ll try to retry them. And if we continue to see …the 500 errors then we will …slow down crawling. And if we continue to see that there are 500 errors then we will drop those URLs from the index.”

Read that as a description of documented behaviors — retry, slower crawling, possible removal — not a fixed three-step timer with guaranteed transitions or timing; Google doesn’t publish exact thresholds for when one stage becomes the next. An isolated 500 on one URL is typically retried, and the next successful fetch is usually the end of it — but that’s a description of the common case, not a guarantee that a one-off failure carries zero cost. The real, documented risk lives in errors that don’t clear.

Why site-wide 500s are worse than isolated ones

There’s a nastier dynamic when a large share of your site 500s at once. Google’s current documentation confirms the crawl-rate cut scales with how many URLs are affected — the more of your site that’s failing, the more crawling slows. Mueller has put the reasoning behind that in more specific terms, framing it as Google suspecting its own crawling might be part of the overload:

“But if a large part of a site consistently has 500 errors and we might assume that maybe we’re causing the problem and we’ll slow down crawling of the whole site and at some point we’ll say well, it looks like these pages are really gone, we’re going to drop them.”

Treat that specific “we assume we’re causing it” causal framing as Mueller’s own characterization rather than wording pulled verbatim from Google’s current official docs — the underlying mechanism is documented (more failing URLs, more crawl-rate reduction), even though the exact causal reasoning is easier to verify from his own statement than from the docs themselves. Either way, the practical feedback loop is worth internalizing: aggressive crawling under resource exhaustion can trigger more 500s → Google backs off crawling of the whole site → and if the errors persist anyway, pages drop. It also means a 500 problem isn’t always a code bug — sometimes it’s your server buckling under concurrent load that only bites under crawler or traffic spikes.

How much is “too much”?

There’s no hard line — Google’s own troubleshooting documentation doesn’t publish an error-rate threshold at all. Mueller has offered a rough, personal rule of thumb in SEO Office Hours (again, relayed via Search Engine Journal, not an official Google publication):

“My feeling is if you’re seeing something more than one percent then that sounds like something is kind of broken.”

Treat ~1% as an unofficial smell test attributed to Mueller, not a documented or enforced Google limit. Below it you’re probably fine; above it, it’s worth investigating — but don’t treat crossing 1% as an automatic trigger, and don’t treat staying under it as a guarantee either. The one number Google commits to publicly is the absence of one: “we don’t have any strong thresholds.”

500 vs. 503: the distinction that matters

This is where a lot of people get it wrong. A 503 Service Unavailable is the sanctioned way to tell a crawler “I’m temporarily down, come back later.” Google treats it as intentional and gives it a grace window. Google’s own crawling troubleshooting doc is explicit:

“Return 503 or 429 HTTP response status codes temporarily for Googlebot requests when your server is overloaded. Googlebot will retry these URLs for about 2 days. Note that returning ‘no availability’ codes for more than a few days will cause Google to permanently slow or stop crawling URLs on your site.”

The contrast: 503 is intentional and gets a ~2-day retry grace; an uncontrolled 500 is unintentional and gets no such grace — it’s simply retried until Google gives up. Practical upshot: for planned maintenance or a deliberate overload defense, return a 503 (ideally with a Retry-After header), not a 500 and not a 200 error page. Never dress a real outage up as a 200.

How to diagnose a 500

Work through this in layers — application/code, platform/CMS, infrastructure and resources, then configuration — cheapest and most-likely-to-pay-off first. Steps marked (WordPress-specific) are common WordPress practice, not universal fixes; adapt them to whatever your actual stack is.

1. Server error logs. error.log / access.log (or your platform’s log viewer). Match timestamps to the failing requests. This is where the actual stack trace, PHP fatal error, or DB connection failure shows up. Everything downstream is guessing until you’ve read these.
2. GSC — Server error (5xx) report. Google Search Console’s Page Indexing report flags the URLs Google itself is seeing 500s on. Then open Crawl Stats and read the “by response” breakdown over time — that’s how you tell a transient blip from a real, sustained availability problem.
3. Bing Webmaster Tools. Its crawl error alerts group Server Errors (5xx) and point you at the specific URLs and the Crawl Information tool.
4. Reproduce as a bot, not just a browser. A page can 500 for Googlebot while loading fine for you — under crawl-triggered load, bot-detection/firewall misconfiguration, or capacity limits that only bite under concurrent bot traffic. Use URL Inspection (GSC), Fetch as Bingbot, or curl with a bot user-agent to catch bot-only failures. “It works in my browser” is not the all-clear.
5. Plugin / theme / module conflicts (WordPress-specific pattern; adapt elsewhere). Take a backup first — always have a way back before you start disabling things. Then disable extensions and re-enable one at a time to isolate the offender, checking file permissions/ownership on anything you touch. Hosting vendors’ own WordPress troubleshooting guides report this pattern often, but that’s a stack-specific observation, not evidence it’s the leading cause everywhere — on a different CMS or a custom app, the equivalent is a third-party module, package, or middleware conflict.
6. Resource exhaustion. PHP memory limits, database connection limits, shared hosting capacity, traffic or crawl spikes. Your host can often confirm this from their side.
7. Configuration & recent changes. A broken .htaccess, a bad server-config edit, a recent deploy, or wrong database credentials. Recent changes are the highest-yield place to look.

How to fix it (match the fix to the cause)

Match the fix to whichever layer the diagnosis pointed at. These are common patterns reported across practitioner write-ups, not a ranked or universal list of what’s most likely on your specific stack:

• Config/deploy caused it → roll back the change; fix the .htaccess, config, or credentials.
• Resource exhaustion → raise limits (PHP memory, DB connections) or upgrade the hosting tier; if crawling is triggering the overload, that’s also a crawl-rate conversation.
• Plugin/module conflict → remove or replace the offending extension.
• Code bug → fix the code, add the missing error handling.
• You can’t tell → escalate to your host with the exact timestamps and the log lines. Don’t guess in production.

Preventing recurrence

Monitoring and alerting on 5xx rates, staging changes before they hit production, load testing ahead of known traffic spikes, and — if Googlebot’s crawling is itself the trigger — managing crawl load (and returning 503/429 deliberately during real overload, rather than letting the server emit uncontrolled 500s).

FAQ

Does a 500 error hurt SEO? The documented risk is mainly about persistence at scale. Isolated 500s are typically retried, with no documented penalty for a one-off blip; sustained, site-wide 500s slow crawling and can lead to deindexing.

How long before Google deindexes a page with a 500? There’s no fixed timeline. Google retries first and slows crawling; dropping from the index comes only if the errors continue. Fix it and pages generally come back once crawls succeed again.

Why does my site 500 for Googlebot but load fine in my browser? Bot-only 500s usually mean capacity/bot-handling issues — crawl-triggered load, firewall/bot rules, or limits that only bite under concurrent bot traffic. Trust the logs, not a manual browser check.

Can 500s slow crawling of my whole site, not just the affected pages? Yes — Google’s documentation confirms the crawl-rate cut scales with how many URLs are failing, so a large share of the site returning 500s slows crawling site-wide. Mueller has additionally framed the reasoning as Google suspecting its own crawling might be part of the overload — his own characterization, not wording pulled verbatim from the current official docs.

What causes a 500 in WordPress? On WordPress specifically, hosting vendors and the WordPress community most often report plugin/theme conflicts, a corrupt .htaccess, or hitting the PHP memory limit — that’s what gets reported for that platform, not a claim that these are the universal leading causes of every 500. The diagnosis order above (logs first, then what changed) is the same regardless of CMS.

Is it safe to automatically retry a request after a 500? Only if you’ve checked the method and the request’s idempotency first — the 500 status itself doesn’t authorize a retry policy. GET, HEAD, PUT, and DELETE are generally safe to retry because they’re idempotent (repeating them shouldn’t cause extra side effects); a bare POST usually isn’t, unless your API explicitly guarantees idempotency (for example, via an idempotency key) — retrying it blindly risks a duplicate order, a duplicate email, or a double-charged payment. When you do retry, use exponential backoff with jitter, cap the number of attempts, and set a retry budget so a struggling server doesn’t get hit by a retry storm on top of whatever’s already failing.

AI summary

A condensed take on the Advanced version:

• 500 = RFC 9110’s unexpected-condition boundary. That’s the entire scope of the status code itself — no root cause, no duration, no retry-worthiness. It’s a symptom, not a diagnosis; root-cause it from server logs, not the browser. Distinct from 4xx (client/request) errors and from its 5xx siblings 502/503/504.
• Google’s documented reaction is graduated: temporary crawl-rate reduction (scaled to how many URLs are affected) and possible eventual index removal for URLs that keep failing, with recovery once fetches succeed again. Mueller describes this as retry → slow crawling → drop from index — a description of documented behaviors, not a fixed lockstep timer. An isolated 500 is typically retried; there’s no promise it’s cost-free, but the real, documented risk is persistence at scale.
• Site-wide 500s are worse: the crawl-rate cut scales with how many URLs fail. Mueller frames the reasoning as Google suspecting its own crawling may be part of the overload — his own characterization, not verbatim official-docs wording — and either way it’s a feedback loop where crawl load can trigger more 500s.
• Rule of thumb, not a threshold: Mueller has said error rates above ~1% are “probably broken,” but that’s his personal framing from SEO Office Hours, not a documented Google limit — Google states it has no hard thresholds.
• 500 vs. 503: 503 (or 429) is the sanctioned “come back later” signal with a ~2-day retry grace per Google’s docs; an uncontrolled 500 gets no grace. Use 503 (with Retry-After) for planned downtime.
• Diagnosis order (by layer): server logs → GSC Server error (5xx) + Crawl Stats “by response” → Bing Webmaster Tools → reproduce as a bot (URL Inspection / curl) → plugin/module conflicts (WordPress-specific pattern; adapt elsewhere, back up first) → resource exhaustion → config/deploy changes.
• Retry safety depends on the request, not the status code. Idempotent methods (GET/HEAD/PUT/DELETE) are generally safe to retry; a bare POST usually isn’t without an idempotency key. Use backoff, a retry cap, and a retry budget.
• Recovery is generally fast once fixed — dropped pages tend to return as crawls succeed again, though Google guarantees no timeline or outcome.

Official documentation

Primary-source documentation from the search engines and the HTTP spec.

Google

• Troubleshoot Google Search crawling errors — how Google handles server errors, and the sanctioned use of 503/429 for temporary overload.
• In-Depth Guide to How Google Search Works — the crawl scheduler and the fact that 5xx responses are read as “slow down.”
• Crawl Stats report — the “by response” breakdown (including Server error (5xx)) used to tell a blip from a sustained problem.
• Reduce the Googlebot crawl rate — the correct, deliberate way to slow crawling (rather than letting the server emit uncontrolled 500s).

Bing / Microsoft

• Bing Webmaster Tools — List of crawl error alerts — how Bing groups Server Errors (5xx) and where to inspect the affected URLs.

HTTP spec / reference

• MDN — 500 Internal Server Error — the definition of the status code itself.
• RFC 9110 §15.6.1 — 500 Internal Server Error — the authoritative HTTP semantics.

Quotes from the source

On-the-record statements. Where a source page is JavaScript-rendered or was relayed through secondary coverage, that’s flagged with a <small> caveat.

Google — the escalation path

• “We don’t have any strong thresholds on that. But essentially what happens with 500 errors is we’ll try to retry them. And if we continue to see …the 500 errors then we will …slow down crawling. And if we continue to see that there are 500 errors then we will drop those URLs from the index.” — John Mueller, Google. Read the coverage Relayed via Search Engine Journal’s transcript of a Google SEO Office Hours video; confirm exact wording against the source video before treating as final.
• “But if a large part of a site consistently has 500 errors and we might assume that maybe we’re causing the problem and we’ll slow down crawling of the whole site and at some point we’ll say well, it looks like these pages are really gone, we’re going to drop them.” — John Mueller, Google. Read the coverage Relayed via Search Engine Journal; verify verbatim against the original video.
• “My feeling is if you’re seeing something more than one percent then that sounds like something is kind of broken.” — John Mueller, Google, on a rough error-rate rule of thumb. Read the coverage Relayed via Search Engine Journal; verify verbatim against the original video.

Google — the sanctioned “slow down” signal (docs, verified)

• “Return 503 or 429 HTTP response status codes temporarily for Googlebot requests when your server is overloaded. Googlebot will retry these URLs for about 2 days. Note that returning ‘no availability’ codes for more than a few days will cause Google to permanently slow or stop crawling URLs on your site.” — Google Search Central docs. Jump to quote

Patrick Stox — the definition (Ahrefs, verified)

• “500 Internal Server Error – The server encounters some kind of issue and doesn’t have a better or more specific error code.” — my HTTP status codes guide on the Ahrefs blog. Jump to quote

500-error triage checklist

Run top to bottom the moment 500s show up:

• Pulled the server error logs and found the actual error (stack trace / PHP fatal / DB failure) — not just the browser “500”.
• Checked what changed recently — new plugin/theme/module, recent deploy, .htaccess or server-config edit, changed DB credentials.
• Opened GSC → Page Indexing → Server error (5xx) to see which URLs Google is seeing 500s on.
• Read GSC Crawl Stats “by response” over time to judge transient vs. sustained.
• Checked Bing Webmaster Tools crawl error alerts for the same URLs.
• Reproduced as a bot (URL Inspection / Fetch as Bingbot / curl with a bot user-agent) — not just in a browser.
• Ruled out resource exhaustion (PHP memory, DB connections, hosting capacity) with your host.
• Isolated any plugin/module conflict by disabling and re-enabling one at a time.
• Confirmed you’re not emitting uncontrolled 500s during planned downtime — use 503 (with Retry-After) instead.
• Verified error rate is back under the unofficial ~1% smell test (Mueller’s rule of thumb, not a documented Google threshold) once fixed, and that GSC shows crawls succeeding again.

Runbook: “The site is throwing 500s — what do I check first?”

A no-panic order of operations. Do these in sequence; stop when you’ve found and fixed the cause.

0. Scope it (2 minutes). Is it one URL, one template/section, or the whole site? One URL is low-stakes (Google retries). Site-wide is the emergency — that’s the pattern that makes Google slow crawling of the whole site.

1. Read the server error logs. error.log first. Match timestamps to the failures. You’re looking for the real cause: a PHP fatal error, a DB connection failure, a segfault, an out-of-memory kill. Everything below is guessing until you’ve done this.

2. Correlate with “what changed.” Sort by recent changes: last deploy, a plugin/theme/module added or updated, an .htaccess edit, rotated DB credentials, a config push. Most 500s trace to a change in the last few hours or days. If you can, roll it back first, diagnose second — restoring service buys you time.

3. Confirm the search-engine view. GSC → Page Indexing → Server error (5xx) for the affected URLs, then Crawl Stats → by response to see whether it’s a blip or sustained. Cross-check Bing Webmaster Tools crawl error alerts. This tells you how urgent the SEO clock is.

4. Reproduce it the way the bot sees it. If humans get the page fine but Googlebot 500s, test as a bot: URL Inspection, Fetch as Bingbot, or curl -A "Googlebot" <url>. Bot-only 500s point at capacity, firewall/bot rules, or crawl-triggered load — a different fix than a code bug.

5. Bisect the usual suspects.

• Plugins/modules: disable all, re-enable one at a time until it breaks again.
• Resources: check PHP memory limit, DB connection ceiling, and hosting capacity with your host — especially if the 500s cluster around traffic or crawl spikes.
• Config: revert the .htaccess / server config to a known-good version.

6. If crawling is the trigger, don’t just absorb it. When Googlebot’s own crawl volume is overloading you, the correct temporary lever is returning 503/429 (with Retry-After) — not letting the server emit uncontrolled 500s. Google honors 503 as “come back later” for ~2 days; a 500 gets no such grace.

7. Verify recovery. Error rate back under the unofficial ~1% smell test, logs clean, and GSC Crawl Stats showing successful fetches again. Dropped pages generally return as crawls succeed — no guaranteed timeline, but it’s usually fast.

Escalation pattern to keep in mind: occasional 500 → typically retried, low documented risk → persistent 500 → crawling slows → still persistent → URLs drop from the index. Google doesn’t publish exact timing for these transitions; your job is to break the chain before it gets that far.

Which server code should I be returning?

Use this when you’re deciding what to serve, or interpreting what you’re seeing.

Is the failure intentional (maintenance / deliberate overload defense)?

• Yes → Return 503 Service Unavailable with a Retry-After header. Google treats it as temporary and retries for ~2 days. Do not serve a 200 error page, and do not let it fall through to a 500.
• No (it’s an actual, unexpected failure) → continue.

Does everyone get the error, or only the crawler?

• Everyone → It’s a code / config / DB problem. Go to the server error logs and the “what changed” list. Roll back recent changes.
• Only Googlebot/Bingbot → Suspect capacity, bot-detection/firewall rules, or crawl-triggered load. Reproduce as a bot; check server resources and any bot rules.

Is it one URL, or a large share of the site?

• One URL / occasional → Low urgency. Google retries; fix at leisure but confirm it’s genuinely isolated.
• Site-wide / persistent → Emergency. This is the pattern that makes Google slow crawling of the whole site and eventually deindex. Restore service first (roll back), root-cause second.

Is the error rate above ~1%?

• Yes → Worth investigating as a likely real problem — this is Mueller’s own unofficial rule of thumb, not a documented Google threshold.
• No → Probably fine — but keep monitoring the trend, not just the snapshot.

Prompt: correlate a 500 with logs and a deployment

Diagnose this HTTP 500 incident from the sanitized evidence I provide. Build a
timeline across deployment events, request IDs, access logs, application errors,
resource signals, and affected URL patterns. Rank likely causes by evidence, separate
the fastest service-restoration action from the root-cause fix, and give exact
validation and rollback checks. Do not invent missing stack traces or thresholds.

[PASTE TIMELINE, HEADERS, LOGS, AND RECENT CHANGES]

Prompt: turn a stack trace into a safe test plan

Explain this stack trace in plain language, identify the failing component and its
inputs, and propose the smallest reversible test that distinguishes code, dependency,
configuration, and resource-exhaustion causes. Include what evidence would falsify
each hypothesis and how to confirm the URL returns a stable non-5xx response afterward.
Redact secrets and do not suggest exposing debug output publicly.

[PASTE SANITIZED STACK TRACE]

Shell: sample a URL list for 5xx responses

Run this with one absolute URL per line in urls.txt.

while IFS= read -r url; do
  curl -sS -o /dev/null -w '%{http_code},%{time_total},%{url_effective}\n' "$url"
done < urls.txt

The output separates isolated routes from a broad failure and records latency without downloading response bodies.

PowerShell: export the same status sample

Get-Content .\urls.txt | ForEach-Object {
  $r = Invoke-WebRequest -Uri $_ -SkipHttpErrorCheck
  [PSCustomObject]@{ Status = $r.StatusCode; Url = $_ }
} | Export-Csv .\status-sample.csv -NoTypeInformation

Shell: count 5xx statuses in an access log

Adjust the status-field position to match your documented log format before relying on the result.

awk '$9 ~ /^5[0-9][0-9]$/ { count[$9]++ } END { for (code in count) print code, count[code] }' access.log

Tools for finding and diagnosing 500s

• Server error logs — error.log / access.log, or your host/platform’s log viewer. The single most important tool; the actual cause lives here.
• Google Search Console — Page Indexing — the “Server error (5xx)” section lists the URLs Google is seeing 500s on.
• GSC — Crawl Stats report — the “by response” breakdown over time separates a transient blip from a sustained availability problem.
• GSC — URL Inspection — fetch a single URL as Google to reproduce a bot-only 500.
• Bing Webmaster Tools — crawl error alerts group Server Errors (5xx) and point you at the Crawl Information tool.
• curl — reproduce with an arbitrary user-agent: curl -I -A "Googlebot" <url> to see the status code the way a bot does.
• Crawlers / site audits — Ahrefs Site Audit and Screaming Frog SEO Spider surface 5xx responses across the site and let you spot patterns (a whole template or section failing).
• Uptime / status monitoring — alerting on 5xx rates so you learn about a spike before Google does.

Resources worth your time

My related writing

• HTTP Status Codes: A Guide to How They Impact SEO & UX — my full status-codes reference, including the 500 definition and the broader 5xx-crawling framing.
• The Beginner’s Guide to Technical SEO — where server errors sit in the bigger technical picture.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawling and how server health feeds back into it. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From around the industry

• How 500 Error Codes Can Negatively Affect Indexing (Search Engine Journal) — Matt G. Southern’s write-up with John Mueller’s retry → slow-crawl → drop-from-index quotes and the ~1% rule of thumb.
• How to fix the “Server error (5xx)” error in Google Search Console (Search Engine Land) — a step-by-step GSC-report walkthrough.
• How To Fix “Server Error (5xx)” in Google Search Console (Onely) — what 5xx errors are, where to find them in GSC, and a fix path.
• How to Fix a 500 Internal Server Error on Your Site (Kinsta) — a detailed, WordPress-hosting-centric fix list (server logs, plugins/themes, PHP memory, .htaccess, permissions).
• 5xx Server Errors: A Guide to Finding & Fixing 5xx Issues (Lumar) — an enterprise/technical-audit angle strong on log-file and crawl-budget framing.
• How to Fix 5xx Server Error in Google Search Console (Sitechecker) — another GSC-report-walkthrough take.

Videos

• Google Search Central (YouTube) — the SEO Office Hours archive, where John Mueller’s server-error and crawling guidance (the retry → slow-crawl → drop-from-index framing quoted above) originates. Channel

Test yourself: 500 Internal Server Error

Five quick questions on what a 500 is and how it affects SEO. Pick an answer for each, then check.

500 Internal Server Error

A 500 Internal Server Error is a generic HTTP status code — RFC 9110 defines it as an unexpected condition that stopped the server from fulfilling the request, and nothing more. The request may have been fine — something broke server-side while generating the response. For SEO, an isolated 500 is typically retried, but persistent, site-wide 500s get Google's documented response: slower crawling and, if the errors don't clear, eventual removal from the index.

Related: HTTP Status Code, Crawling, Crawl Budget

500 Internal Server Error

A 500 Internal Server Error is a catch-all HTTP status code the web server returns when it ran into an unexpected condition and couldn’t complete the request. RFC 9110 defines it narrowly: an unexpected condition prevented the server from fulfilling the request — nothing about root cause, duration, or retry-worthiness. It’s a server-side failure — not a problem with the URL, the browser, or the crawler making the request. As I put it in Ahrefs’ HTTP status codes guide: “500 Internal Server Error – The server encounters some kind of issue and doesn’t have a better or more specific error code.”

The word to notice is generic. A 500 tells you something broke; it doesn’t tell you what. That’s why diagnosis always starts in your server’s error logs, not in the browser. Root causes reported by practitioners include a code bug, a plugin or module conflict (a common WordPress-specific pattern), resource exhaustion (PHP memory limits, database connection limits, shared-hosting capacity), a bad .htaccess or server-config change, or a failed deploy — but that’s what gets reported for particular stacks, not evidence of a universal ranking of causes.

For SEO, the risk is graduated, not instant. An isolated 500 is typically retried by Google, with no documented penalty for a one-off blip. But if a large share of a site consistently returns 500s, Google’s crawl-rate reduction scales with how many URLs are affected — Google slows crawling and, if the errors don’t clear, eventually drops the affected URLs from the index. (John Mueller has separately framed the reasoning as Google suspecting its own crawling might be part of the overload — his own characterization, not wording from Google’s current official docs.) It’s distinct from its 5xx siblings (502, 503, 504): 503 in particular is the intentional, Google-sanctioned “come back later” signal, which is why using the right code during planned downtime matters.

Related: HTTP Status Code, Crawling, Crawl Budget

Changelog