401 Unauthorized

TL;DR — A 401 Unauthorized means the server wants you to log in before it hands over the page — your credentials are missing, wrong, or expired. Search engines never log in, so if a page returns 401 to Googlebot, Google can’t see it and won’t put it in search results. That’s exactly what you want for private pages (staging, members-only) — and a problem only when it happens to a page you actually want people to find.

What a 401 means

When your browser (or a search engine’s crawler) asks a server for a page, the server answers with a status code. 401 Unauthorized is the server’s way of saying: “I’m not going to show you this until you prove who you are.” The credentials you sent were missing, invalid, or expired — so you need to log in. Evidence for this claim A 401 response means the request lacks valid authentication credentials and the response must include a WWW-Authenticate challenge. Scope: RFC 9110 defines the status semantics and required challenge header; it does not prescribe a site's authentication product or login flow. Confidence: high · Verified: 2026-07-14 IETF: RFC 9110 §15.5.2 — 401 Unauthorized

You’ve seen this in action any time a page pops up a username/password box, or kicks you to a login screen. Behind the scenes, that’s often a 401.

Why it matters for SEO

Here’s the catch: Googlebot and Bingbot never log in. They don’t have a username and password for your site, and they don’t type into login boxes. So when a crawler hits a 401, it simply can’t get past the gate. To Google, the content on that page effectively doesn’t exist.

The practical result:

• A page that returns 401 won’t be indexed — it can’t show up in search.
• A page that used to rank but now returns 401 will drop out of the index over time. Evidence for this claim Google does not index 4xx URLs and removes already-indexed 4xx URLs over time. Scope: Google's crawler documentation supports the Search indexing outcome for 401 responses; it does not address access decisions for private users. Confidence: high · Verified: 2026-07-14 Google: How HTTP status codes affect Google's crawlers

When a 401 is fine (and when it’s a problem)

A 401 isn’t automatically a bug to fix. It’s the right answer for pages that are supposed to be private:

• Good 401: your staging site, a members-only area, an internal admin tool, a client portal. You want those hidden from Google, and a login wall does the job.
• Bad 401: a blog post, product page, or landing page that’s supposed to be public suddenly asking for a login — usually because of a misconfigured firewall, a leftover password protection, an expired token, or a plugin conflict.

The most common false alarm is “but it loads fine for me!” — that’s because you’re logged in and the crawler isn’t. To see what Googlebot sees, open the page in a private/incognito window, or use the URL Inspection tool in Google Search Console.

401 vs 403 — the quick version

You’ll often see 401 mentioned next to 403 Forbidden. They sound similar but mean different things:

• 401 = “Who are you?” Credentials are missing — log in.
• 403 = “I know who you are, and no.” The server understood the request but is refusing access anyway.

For Google, though, the outcome is the same: neither one gets indexed, because the crawler can’t reach the content either way.

Want the technical detail — the exact header a 401 must send, what Google’s docs actually say, and how to fix an unwanted 401? Switch to the Advanced tab.

TL;DR — 401 Unauthorized is a client error (RFC 9110 §15.5.2, which superseded RFC 7235) meaning the request lacks valid authentication credentials — including credentials that were sent but rejected, not just credentials that were never sent; a compliant 401 also sends a WWW-Authenticate header naming the expected scheme. It differs from 403 (a refusal that doesn’t require that challenge and can be unrelated to credentials entirely), but Google treats all 4xx except 429 the same for indexing: the content “doesn’t exist,” so it’s not indexed and previously indexed URLs drop over time. Because ordinary Googlebot never sends credentials, a 403 to Googlebot is — per Google’s own words — usually a server misconfiguration. 401/403 have no effect on site-wide crawl rate (a widely repeated myth), though an individual URL that keeps 4xx’ing does get recrawled less often over time. A 401 is the correct, Google-endorsed way to gate genuinely private content; it’s only a problem when it hits a page you want indexed. For paywalled content there’s a sanctioned path that isn’t a blanket 401.

What 401 actually is

401 Unauthorized is a client-error response. MDN’s definition is the cleanest technical one:

“The HTTP 401 Unauthorized client error response status code indicates that a request was not successful because it lacks valid authentication credentials for the requested resource. This status code is sent with an HTTP WWW-Authenticate response header that contains information on the authentication scheme the server expects the client to include to make the request successfully.”

Two things worth pulling out of that. First, “authentication” — the requester hasn’t proven who they are. That usually means missing, invalid, or expired credentials — but a 401 can also follow credentials that were sent and got rejected, so don’t assume “no auth header at all” without actually checking the request. Second, a spec-compliant 401 must carry a WWW-Authenticate header — per RFC 9110 §15.5.2, which superseded the older RFC 7235 — telling the client which scheme(s) it expects (HTTP Basic Auth, a Bearer token, a session-cookie flow, and so on). If you’re debugging a 401, that header is the first thing to inspect: whether it’s present at all, and which scheme it names. Evidence for this claim A 401 response means the request lacks valid authentication credentials and the response must include a WWW-Authenticate challenge. Scope: RFC 9110 defines the status semantics and required challenge header; it does not prescribe a site's authentication product or login flow. Confidence: high · Verified: 2026-07-14 IETF: RFC 9110 §15.5.2 — 401 Unauthorized

In my own status-codes guide I sum the 401 up as: “the client hasn’t identified or verified itself when needed.” That’s the whole idea — nobody’s said who they are yet.

401 vs 403 Forbidden — the distinction that matters

This is where most people get fuzzy, so let me make it sharp. In one line:

• 401 = “Who are you?” — credentials are missing or invalid; authenticate and try again.
• 403 = “I know who you are, but no.” — the request was understood, but access is denied regardless of credentials.

MDN frames it the same way:

“A 401 Unauthorized is similar to the 403 Forbidden response, except that a 403 is returned when a request contains valid credentials, but the client does not have permissions to perform a certain action.”

I draw the same contrast in my own guide: 401 is “the client hasn’t identified or verified itself when needed,” while 403 is “the client is known but doesn’t have access rights.” Authentication vs. authorization.

That one-liner is the reliable shorthand, but here’s the sharper protocol test if you need to pick between the two in code or a WAF rule: 401 requires that WWW-Authenticate challenge — RFC 9110 makes it a MUST — while 403 doesn’t require one, because a 403 refusal can be for reasons that have nothing to do with credentials at all (an IP block, a permissions rule, a rate-limit policy). So “were credentials present?” isn’t a reliable way to tell them apart on its own — a 401 can follow rejected credentials, not just missing ones. If you’re deciding which code to return, ask instead: am I issuing an authentication challenge (401), or a flat refusal (403)?

Here’s the non-obvious SEO twist, and it’s Google’s own point about 403. Ordinary Googlebot never sends credentials. So a 403 served to Googlebot specifically is, by Google’s own words, the server getting it wrong:

“HTTP 403 means that the user agent provided credentials, but was not granted access. However, Googlebot never provides credentials, so your server is returning this error incorrectly. The page will not be indexed.”

That’s a useful diagnostic. A 401 to Googlebot can be intentional (the page is gated by design). A 403 to Googlebot usually points at a misconfiguration — Googlebot didn’t send credentials, so nothing should be triggering a “credentials-were-denied” response for it. If you’re seeing 403s on pages Googlebot should reach, suspect your CDN, WAF, or server config first. (This describes Google’s common crawlers. Google documents special-case crawlers and user-triggered fetchers as separate categories with their own behaviors, so don’t assume the “never sends credentials” rule generalizes to every Google product integration without checking.)

The last row is the punchline: for indexing, Google treats them identically.

How Google treats 401 pages

Google’s HTTP status codes doc is blunt about the 4xx family:

“Google doesn’t use the content from URLs that return 4xx status codes. If a URL was previously used but is now returning 4xx status code, Google systems will stop using the URL over time. In the case of Google Search, Google doesn’t index URLs that return a 4xx status code, and URLs that are already indexed and return a 4xx status code are removed from the index.” Evidence for this claim Google does not index 4xx URLs and removes already-indexed 4xx URLs over time. Scope: Google's crawler documentation supports the Search indexing outcome for 401 responses; it does not address access decisions for private users. Confidence: high · Verified: 2026-07-14 Google: How HTTP status codes affect Google's crawlers

And 401 isn’t a special case within that family. In the doc’s table, 401 (unauthorized) and 403 (forbidden) are listed as separate rows that share a single explanation:

“All 4xx errors, except 429, are treated the same: Google crawlers inform the next processing system that the content doesn’t exist.”

So the outcome is binary, not a demotion. A 401’d page doesn’t “rank lower” — it doesn’t get indexed at all, or gets dropped entirely if it was indexed before. There’s no partial penalty. This lines up with the baseline claim from my own guide: 4xxs will cause pages to drop from the index. This article is the 401-specific deep dive on top of that.

One nuance on timing: removal happens “over time,” not on the first bad fetch. Google’s docs describe a gradual process, and the crawling system has historically been described as tolerating short-lived errors before treating a URL as genuinely gone.

The crawl-rate myth — 401 does NOT throttle crawling

This one trips up a lot of otherwise-good articles, so I want to be precise. A 401 (or 403) does not slow Google’s crawl rate. Google says so directly:

“Don’t use 401 and 403 status codes for limiting the crawl rate. The 4xx status codes, except 429, have no effect on crawl rate.”

This matters because you’ll see advice that gating pages behind a 401 “saves crawl budget” or “wastes crawl budget” — both framings are wrong. Only 429 (and 5xx-style signals like 503) tell Googlebot to back off. A 401 is not a throttle; it’s a “content doesn’t exist” signal for indexing, full stop. If you actually want to slow a crawl temporarily, that’s 429/503 — not 401/403.

One scope note, since these two claims are easy to conflate: “no effect on crawl rate” is about your site’s overall crawl rate. Separately, Google says an individual URL that keeps returning a 4xx gets recrawled less often over time — its own retry frequency gradually drops. Those are different scopes: your site-wide crawl budget isn’t throttled, but a persistently-401’ing URL specifically does get checked less frequently as Google deprioritizes it.

There’s also a special case worth knowing about: a 401 on an ordinary page and a 401 on /robots.txt itself are not handled the same way. If your robots.txt file returns a non-429 4xx (including 401), Google treats that as if no robots.txt exists at all — it assumes no crawl restrictions from that file, not that your whole site just went unreachable. Don’t put /robots.txt behind the same auth wall as your private pages.

Is a 401 always a problem? No.

A 401 is only a bug when it’s unintentional on a page you want public. When the page is genuinely private, a 401 is the correct way to keep it out of search — and it’s what Google recommends. John Mueller has put it plainly (relayed by Search Engine Journal): the ideal approach is server-side authentication that blocks normal users from seeing the content — “that would include GoogleBot.” (Relayed by Search Engine Journal from a 2019 Google hangout; treat as an accurately paraphrased rep statement rather than a fragment-verified verbatim quote.)

Server-side auth (which is what produces a 401) is his recommended mechanism for hiding non-public content — ahead of robots.txt — precisely because it actually blocks access rather than just requesting bots stay out.

So the decision framework is simple:

• Should stay 401’d: staging environments, member-only areas, internal tools, anything genuinely private. Working as designed. Don’t “fix” it.
• Needs fixing: a public, indexable page returning 401 by accident — a CDN/WAF false positive, leftover Basic Auth, an expired token, or a plugin/middleware conflict.

SEO implications, and the “it works for me” trap

The practical failure modes:

• Pages you want indexed stay invisible until the gate is removed.
• Previously ranking pages disappear if they start returning 401.
• The “it works for me” trap: the person testing is authenticated, logged in, or on an allowlisted IP — the crawler isn’t. Google’s own guidance for the 401 case: “You can verify this error by visiting the page in incognito mode.” Better still, test unauthenticated with curl -I https://example.com/page, or use Search Console’s URL Inspection / Live Test.

When you do need to let a real crawler through, verify it by IP / reverse-DNS, never by trusting the user-agent string alone — user-agent strings are trivially spoofable, so allowlisting “Googlebot” by name is a security hole, not a fix.

What about paywalled or subscriber-only content?

A blanket 401 to everyone (Googlebot included) isn’t your only option if you want gated content to still rank. Google supports indexing paywalled content via the isAccessibleForFree structured data, paired with granting access to Google’s dedicated crawler identities for subscriber/registered-user content. The point of the markup:

“This structured data helps Google differentiate paywalled content from the practice of cloaking, which violates spam policies.”

The warning baked into that: silently serving full content to Googlebot only, without the structured-data disclosure, is cloaking — a spam-policy risk. If you want gated content indexed, do it the sanctioned way (markup + crawler access), not a quiet bypass.

How to fix an unwanted 401

1. Confirm it’s actually unwanted. Is this page supposed to be public? If it’s staging or members-only, there’s nothing to fix.
2. Remove the auth requirement on public pages — strip leftover Basic Auth (.htaccess/nginx), fix expired tokens, resolve plugin/middleware conflicts.
3. Check your CDN/WAF — and don’t assume you already know which layer issued the 401. The response (and its WWW-Authenticate header) tells you a challenge happened; it doesn’t tell you where it came from — your application, an identity/auth proxy, the CDN or WAF’s bot-management rules, and the origin server can all generate one. False positives from edge bot-management rules are a common cause; inspect logs at each layer and verify Googlebot by reverse DNS before allowlisting anything.
4. Let verified crawlers through by IP/reverse-DNS, not user-agent.
5. Don’t reach for 401 to deindex a page you could make public — use noindex (with crawling allowed) instead. A login wall is for content that must be genuinely private.
6. Validate the fix with URL Inspection’s Live Test — but treat a pass as confirmation of the current fetch, not a guarantee. Google doesn’t commit to a fixed recrawl, re-indexing, or ranking-recovery timeline after you fix a 401; a passing Live Test shows Google’s live fetch got through, not that scheduled crawling or indexing has caught up. Give it time and recheck the Page Indexing report rather than expecting an instant reversal.

For the full Google Search Console diagnose-fix-validate workflow on the “Blocked due to unauthorized request (401)” Page Indexing status specifically, see the dedicated companion article — this piece stays at the protocol/concept level.

Bing

Bing behaves functionally the same way: a URL returning 401 (or 403) to Bingbot is inaccessible and won’t be indexed. Bingbot needs unauthenticated access just like Googlebot, and you verify it by its published IP ranges for allowlisting rather than by user-agent.

AI summary

A condensed take on the Advanced version:

• 401 = missing, invalid, or rejected credentials (“who are you?”). RFC 9110 §15.5.2 (which superseded RFC 7235) requires a WWW-Authenticate header naming the scheme the client should use — and a 401 doesn’t prove no credentials were sent; it can follow credentials that were sent and refused.
• 401 vs 403, the reliable test: 401 requires that WWW-Authenticate challenge; 403 doesn’t, because a 403 refusal can be unrelated to credentials entirely (IP block, permission rule, rate limit). Because ordinary Googlebot never sends credentials, a 403 to Googlebot is, per Google, usually a server misconfiguration — Google documents other crawler/fetcher identities separately, so don’t generalize this to every Google client.
• Indexing outcome is identical: Google treats all 4xx except 429 the same — the content “doesn’t exist,” so it’s not indexed, and previously indexed URLs drop over time. It’s binary, not a demotion.
• No site-wide crawl-rate effect, but per-URL frequency still drops: 401/403 don’t throttle overall crawl rate — only 429/5xx-style signals do. Separately, an individual URL that keeps 4xx’ing gets recrawled less often over time. Also: a 401 on /robots.txt itself is treated as “no robots.txt exists,” not a block.
• A 401 is often correct: server-side auth is the Google-endorsed way to hide staging/member/private pages. It’s only a problem when it hits a page you want indexed — and the response alone doesn’t tell you which layer (app, identity proxy, CDN/WAF, origin) produced it.
• “It works for me” trap: you’re authenticated; the crawler isn’t. Test in incognito, with curl -I, or URL Inspection — a passing Live Test only confirms the current fetch, not a guaranteed recrawl/recovery timeline.
• Paywalls: use isAccessibleForFree markup + crawler access, not a blanket 401. Serving Googlebot full content silently is cloaking.
• Fixing: remove the auth requirement / verify crawlers by IP/reverse-DNS (never user-agent) / use noindex — not 401 — to deindex a page you could make public.

Official documentation

Primary-source documentation from Google and the HTTP spec.

Google

• How HTTP Status Codes, Network and DNS Errors Affect Google Search — the 4xx handling, the “treated the same” rule, and the crawl-rate warning for 401/403.
• Page Indexing report — the “Blocked due to unauthorized request (401)” and “Blocked due to access forbidden (403)” status definitions.
• Structured data for subscription and paywalled content — isAccessibleForFree and the sanctioned alternative to a blanket 401.
• Verifying Googlebot and other Google crawlers — verify by IP/reverse-DNS, not user-agent, before allowlisting.

Standards / reference

• RFC 9110 §15.5.2 — HTTP Semantics: 401 Unauthorized — the current 401 + WWW-Authenticate specification (supersedes RFC 7235).
• MDN — 401 Unauthorized — the developer-audience technical definition (secondary, not SEO-specific).

Quotes from the source

On-the-record statements. Each link is a deep link that jumps to the quoted passage on the source page.

Google — how 4xx (including 401) is handled

• “Google doesn’t use the content from URLs that return 4xx status codes… Google doesn’t index URLs that return a 4xx status code, and URLs that are already indexed and return a 4xx status code are removed from the index.” — Google Search Central docs. Jump to quote
• “All 4xx errors, except 429, are treated the same: Google crawlers inform the next processing system that the content doesn’t exist.” Jump to quote
• “Don’t use 401 and 403 status codes for limiting the crawl rate. The 4xx status codes, except 429, have no effect on crawl rate.” Jump to quote

Google — Search Console Help, 401 and 403 entries

• “The page was blocked to Googlebot by a request for authorization (401 response)… You can verify this error by visiting the page in incognito mode.” — Google Search Console Help, Page Indexing report. Jump to quote
• “HTTP 403 means that the user agent provided credentials, but was not granted access. However, Googlebot never provides credentials, so your server is returning this error incorrectly. The page will not be indexed.” Jump to quote

Google — paywalled content

• “This structured data helps Google differentiate paywalled content from the practice of cloaking, which violates spam policies.” — Google Search Central docs. Jump to quote

MDN — the technical definition

• “The HTTP 401 Unauthorized client error response status code indicates that a request was not successful because it lacks valid authentication credentials for the requested resource.” — MDN Web Docs (secondary technical reference, not SEO-specific). Jump to quote
• “A 401 Unauthorized is similar to the 403 Forbidden response, except that a 403 is returned when a request contains valid credentials, but the client does not have permissions to perform a certain action.” Jump to quote

John Mueller, Google (relayed by Search Engine Journal)

• On hiding non-public content: server-side authentication that blocks normal users from seeing the content — “that would include GoogleBot” — is the ideal approach. Relayed by Search Engine Journal from a 2019 Google hangout; treat as an accurately paraphrased rep statement, not a fragment-verified verbatim quote.

401 vs 403 — which am I actually seeing, and do I need to fix it?

The two questions readers actually have: is this a 401 or a 403? and is it a problem? Click through it.

Prompt: diagnose an unintended 401

Paste one affected URL’s anonymous response headers plus relevant auth, CDN, or middleware configuration. Remove tokens, cookies, and secrets first.

Diagnose why this public URL returns HTTP 401 to an unauthenticated client. Check for
a valid WWW-Authenticate challenge, separate origin authentication from CDN/WAF and
application middleware, and distinguish an intentional private gate from a public-page
regression. Return: evidence, likely issuing layer, minimum safe fix, and exact
anonymous validation requests. Do not recommend user-agent-only bot allowlisting.

[PASTE SANITIZED HEADERS, LOG EVENT, AND CONFIG]

Prompt: review an authentication exception

Review this proposed crawler-access rule for a gated site. Identify whether it relies
only on a spoofable user-agent, whether crawler identity is verified by published IP
ranges or reverse-and-forward DNS, and whether serving different content would create
a cloaking risk. Give a least-privilege alternative and a rollback test.

[PASTE SANITIZED RULE]

Shell: inspect the anonymous challenge

Run this in a terminal without browser cookies or credentials.

URL='https://example.com/private'
curl -sS -D - -o /dev/null "$URL"

For a real 401, inspect WWW-Authenticate. A public URL returning 401 anonymously but 200 in your logged-in browser is the authentication-state trap described above.

PowerShell: inspect the same anonymous response

$response = Invoke-WebRequest -Uri 'https://example.com/private' -SkipHttpErrorCheck
$response.StatusCode
$response.Headers['WWW-Authenticate']

Shell: compare generic and claimed-bot user agents

URL='https://example.com/page'
curl -sS -o /dev/null -w 'generic %{http_code}\n' "$URL"
curl -sS -o /dev/null -w 'claimed-bot %{http_code}\n' -A 'Googlebot' "$URL"

The comparison can expose a user-agent rule, but it does not verify Googlebot; any client can send that string. Confirm real crawler traffic separately by IP or reverse-and-forward DNS before changing access rules.

Tools for finding the layer that returns 401

• Bulk HTTP Status Code Checker: confirm which public URLs return 401 without your browser session and export the affected set.
• HTTP Header Checker: inspect WWW-Authenticate, cache/CDN headers, and redirect hops for clues about the issuing layer.
• Googlebot Verifier: validate crawler IP evidence before creating an access exception; a user-agent string alone proves nothing.
• Search Console URL Inspection, Live Test: confirm what Google’s live fetch can access after the fix.
• CDN/WAF security events and origin logs: correlate the request ID and time to determine whether the edge, application, or origin generated the 401.

Test yourself: 401 Unauthorized

Five quick questions on what a 401 means and how search engines handle it. Pick an answer for each, then check.

401 Unauthorized

401 Unauthorized is a client-error HTTP status code meaning the request lacks valid authentication credentials — the server wants you to log in before it serves the page. For SEO it means the content is gated: Googlebot never sends credentials, so a 401'd page can't be seen and won't be indexed.

Related: 403 Forbidden, HTTP Status Code, Blocked due to unauthorized request (401)

401 Unauthorized

HTTP 401 Unauthorized is a client-error status code that means the request wasn’t successful because it lacks valid authentication credentials for the resource. In plain terms, the server is saying “log in first” — the credentials are missing, invalid, expired, or were sent and rejected. Per the current HTTP spec (RFC 9110 §15.5.2, which superseded RFC 7235), a 401 response must include a WWW-Authenticate header naming the authentication scheme the client should use.

It’s easy to confuse 401 with 403 Forbidden, but the reliable distinction is that a 401 requires that WWW-Authenticate challenge, while a 403 doesn’t — a 403 refusal can be unrelated to credentials entirely (an IP block, a permissions rule). In my own status-codes guide I describe 401 as “the client hasn’t identified or verified itself when needed” and 403 as “the client is known but doesn’t have access rights.”

For SEO, the outcome is what matters: a page returning 401 is gated, and since ordinary Googlebot never sends credentials, it can’t get past the gate. Google doesn’t use content from any 4xx URL, so a 401’d page won’t be indexed — and a previously indexed URL that starts returning 401 drops from the index over time. That’s fine (and correct) for genuinely private pages like staging sites and member areas; it’s only a problem when it happens to a page you actually want ranked. One exception: a 401 on /robots.txt itself is treated as if no robots.txt exists, not as a crawl block.

Related: 403 Forbidden, HTTP Status Code, Blocked due to unauthorized request (401)

Changelog