401 Unauthorized

What an HTTP 401 Unauthorized response means, how it differs from 403 Forbidden, how Google treats authentication-gated pages, and the SEO implications.

First published: Jun 28, 2026 · Last updated: Jul 17, 2026 · Advanced
demand #8 in HTTP Errors#11 in HTTP Status Codes#68 in Technical SEO#92 on the site
AIAuthored byAI v2
1 evidence signal on this page

401 Unauthorized means the request lacks valid authentication credentials — the server wants you to log in. It's distinct from 403 Forbidden (credentials given but access denied), but for indexing Google treats them the same: Googlebot never sends credentials, so a 401'd page's content effectively doesn't exist to Google, won't be indexed, and drops from the index over time if it was already there. A 401 isn't automatically bad — it's the correct way to gate staging sites and member areas — it's only a problem when it hits a page you want ranked. It has no effect on crawl rate, despite a common myth to the contrary.

TL;DR — 401 Unauthorized401 Unauthorized is a client-error HTTP status code meaning the request lacks valid authentication credentials — the server wants you to log in before it serves the page. For SEO it means the content is gated: Googlebot never sends credentials, so a 401'd page can't be seen and won't be indexed. is a client error (RFC 9110 §15.5.2, which superseded RFC 7235) meaning the request lacks valid authentication credentials — including credentials that were sent but rejected, not just credentials that were never sent; a compliant 401 also sends a WWW-Authenticate header naming the expected scheme. It differs from 403 (a refusal that doesn’t require that challenge and can be unrelated to credentials entirely), but Google treats all 4xx except 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. the same for indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.: the content “doesn’t exist,” so it’s not indexed and previously indexed URLs drop over time. Because ordinary GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. never sends credentials, a 403 to GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. is — per Google’s own words — usually a server misconfiguration. 401/403 have no effect on site-wide crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. (a widely repeated myth), though an individual URL that keeps 4xx’ing does get recrawled less often over time. A 401 is the correct, Google-endorsed way to gate genuinely private content; it’s only a problem when it hits a page you want indexed. For paywalled content there’s a sanctioned path that isn’t a blanket 401.

What 401 actually is

401 Unauthorized is a client-error response. MDN’s definition is the cleanest technical one:

“The HTTP 401 Unauthorized client error response status code indicates that a request was not successful because it lacks valid authentication credentials for the requested resource. This status code is sent with an HTTP WWW-Authenticate response header that contains information on the authentication scheme the server expects the client to include to make the request successfully.”

Two things worth pulling out of that. First, “authentication” — the requester hasn’t proven who they are. That usually means missing, invalid, or expired credentials — but a 401 can also follow credentials that were sent and got rejected, so don’t assume “no auth header at all” without actually checking the request. Second, a spec-compliant 401 must carry a WWW-Authenticate header — per RFC 9110 §15.5.2, which superseded the older RFC 7235 — telling the client which scheme(s) it expects (HTTP Basic Auth, a Bearer tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory., a session-cookie flow, and so on). If you’re debugging a 401, that header is the first thing to inspect: whether it’s present at all, and which scheme it names. Evidence for this claim A 401 response means the request lacks valid authentication credentials and the response must include a WWW-Authenticate challenge. Scope: RFC 9110 defines the status semantics and required challenge header; it does not prescribe a site's authentication product or login flow. Confidence: high · Verified: IETF: RFC 9110 §15.5.2 — 401 Unauthorized

In my own status-codes guide I sum the 401 up as: “the client hasn’t identified or verified itself when needed.” That’s the whole idea — nobody’s said who they are yet.

401 vs 403 Forbidden — the distinction that matters

This is where most people get fuzzy, so let me make it sharp. In one line:

  • 401 = “Who are you?” — credentials are missing or invalid; authenticate and try again.
  • 403 = “I know who you are, but no.” — the request was understood, but access is denied regardless of credentials.

MDN frames it the same way:

“A 401 Unauthorized is similar to the 403 ForbiddenAn HTTP 403 client-error status code meaning the server understood the request but refuses to fulfill it — access is denied, whether or not credentials are involved. For SEO, a 403 served to Googlebot on a page meant to be public and indexable usually points to a misconfigured rule; a persistent 403 keeps a page out of the index either way. response, except that a 403 is returned when a request contains valid credentials, but the client does not have permissions to perform a certain action.”

I draw the same contrast in my own guide: 401 is “the client hasn’t identified or verified itself when needed,” while 403 is “the client is known but doesn’t have access rights.” Authentication vs. authorization.

That one-liner is the reliable shorthand, but here’s the sharper protocol test if you need to pick between the two in code or a WAF rule: 401 requires that WWW-Authenticate challenge — RFC 9110 makes it a MUST — while 403 doesn’t require one, because a 403 refusal can be for reasons that have nothing to do with credentials at all (an IP block, a permissions rule, a rate-limit policy). So “were credentials present?” isn’t a reliable way to tell them apart on its own — a 401 can follow rejected credentials, not just missing ones. If you’re deciding which code to return, ask instead: am I issuing an authentication challenge (401), or a flat refusal (403)?

Here’s the non-obvious SEO twist, and it’s Google’s own point about 403. Ordinary Googlebot never sends credentials. So a 403 served to Googlebot specifically is, by Google’s own words, the server getting it wrong:

“HTTP 403 means that the user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. provided credentials, but was not granted access. However, Googlebot never provides credentials, so your server is returning this error incorrectly. The page will not be indexed.”

That’s a useful diagnostic. A 401 to Googlebot can be intentional (the page is gated by design). A 403 to Googlebot usually points at a misconfiguration — Googlebot didn’t send credentials, so nothing should be triggering a “credentials-were-denied” response for it. If you’re seeing 403s on pages Googlebot should reach, suspect your CDN, WAF, or server config first. (This describes Google’s common crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. Google documents special-case crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. and user-triggered fetchers as separate categories with their own behaviors, so don’t assume the “never sends credentials” rule generalizes to every Google product integration without checking.)

401 Unauthorized403 ForbiddenAn HTTP 403 client-error status code meaning the server understood the request but refuses to fulfill it — access is denied, whether or not credentials are involved. For SEO, a 403 served to Googlebot on a page meant to be public and indexable usually points to a misconfigured rule; a persistent 403 keeps a page out of the index either way.
MeaningCredentials missing/invalid — “who are you?”Understood, but access denied — “I know you, no”
Required headerWWW-Authenticate (RFC 9110)None required
Typical triggerLogin wall, expired token, Basic Auth, session timeoutPermission rules, IP/geoGenerative Engine Optimization — visibility inside AI answer engines. blocks, WAF rules, directory restrictions
To GooglebotCan be intentional (page is gated)Usually a server misconfiguration (Googlebot sends no credentials)
IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. outcomeNot indexed; dropped over timeNot indexed; dropped over time

The last row is the punchline: for indexing, Google treats them identically.

How Google treats 401 pages

Google’s HTTP status codes doc is blunt about the 4xx family:

“Google doesn’t use the content from URLs that return 4xx status codes. If a URL was previously used but is now returning 4xx status code, Google systems will stop using the URL over time. In the case of Google Search, Google doesn’t index URLs that return a 4xx status code, and URLs that are already indexed and return a 4xx status code are removed from the index.” Evidence for this claim Google does not index 4xx URLs and removes already-indexed 4xx URLs over time. Scope: Google's crawler documentation supports the Search indexing outcome for 401 responses; it does not address access decisions for private users. Confidence: high · Verified: Google: How HTTP status codes affect Google's crawlers

And 401 isn’t a special case within that family. In the doc’s table, 401 (unauthorized) and 403 (forbidden) are listed as separate rows that share a single explanation:

“All 4xx errors, except 429, are treated the same: Google crawlers inform the next processing system that the content doesn’t exist.”

So the outcome is binary, not a demotion. A 401’d page doesn’t “rank lower” — it doesn’t get indexed at all, or gets dropped entirely if it was indexed before. There’s no partial penalty. This lines up with the baseline claim from my own guide: 4xxs will cause pages to drop from the index. This article is the 401-specific deep dive on top of that.

One nuance on timing: removal happens “over time,” not on the first bad fetch. Google’s docs describe a gradual process, and the crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. system has historically been described as tolerating short-lived errors before treating a URL as genuinely gone.

The crawl-rate myth — 401 does NOT throttle crawling

This one trips up a lot of otherwise-good articles, so I want to be precise. A 401 (or 403) does not slow Google’s crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.. Google says so directly:

“Don’t use 401 and 403 status codes for limiting the crawl rate. The 4xx status codes, except 429, have no effect on crawl rate.”

This matters because you’ll see advice that gating pages behind a 401 “saves crawl budget” or “wastes crawl budgetThe number of URLs an engine will crawl in a timeframe.” — both framings are wrong. Only 429 (and 5xx-style signals like 503) tell Googlebot to back off. A 401 is not a throttle; it’s a “content doesn’t exist” signal for indexing, full stop. If you actually want to slow a crawl temporarily, that’s 429/503 — not 401/403.

One scope note, since these two claims are easy to conflate: “no effect on crawl rate” is about your site’s overall crawl rate. Separately, Google says an individual URL that keeps returning a 4xx gets recrawled less often over time — its own retry frequency gradually drops. Those are different scopes: your site-wide crawl budget isn’t throttled, but a persistently-401’ing URL specifically does get checked less frequently as Google deprioritizes it.

There’s also a special case worth knowing about: a 401 on an ordinary page and a 401 on /robots.txt itself are not handled the same way. If your robots.txt file returns a non-429 4xx (including 401), Google treats that as if no robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. exists at all — it assumes no crawl restrictions from that file, not that your whole site just went unreachable. Don’t put /robots.txt behind the same auth wall as your private pages.

Is a 401 always a problem? No.

A 401 is only a bug when it’s unintentional on a page you want public. When the page is genuinely private, a 401 is the correct way to keep it out of search — and it’s what Google recommends. John Mueller has put it plainly (relayed by Search Engine Journal): the ideal approach is server-side authentication that blocks normal users from seeing the content — “that would include GoogleBot.” (Relayed by Search Engine Journal from a 2019 Google hangout; treat as an accurately paraphrased rep statement rather than a fragment-verified verbatim quote.)

Server-side auth (which is what produces a 401) is his recommended mechanism for hiding non-public content — ahead of robots.txt — precisely because it actually blocks access rather than just requesting bots stay out.

So the decision framework is simple:

  • Should stay 401’d: staging environments, member-only areas, internal tools, anything genuinely private. Working as designed. Don’t “fix” it.
  • Needs fixing: a public, indexable page returning 401 by accident — a CDN/WAF false positive, leftover Basic Auth, an expired token, or a plugin/middleware conflict.

SEO implications, and the “it works for me” trap

The practical failure modes:

  • Pages you want indexed stay invisible until the gate is removed.
  • Previously ranking pages disappear if they start returning 401.
  • The “it works for me” trap: the person testing is authenticated, logged in, or on an allowlisted IP — the crawler isn’t. Google’s own guidance for the 401 case: “You can verify this error by visiting the page in incognito mode.” Better still, test unauthenticated with curl -I https://example.com/page, or use Search Console’s URL Inspection / Live Test.

When you do need to let a real crawler through, verify it by IP / reverse-DNS, never by trusting the user-agent string alone — user-agent strings are trivially spoofable, so allowlisting “Googlebot” by name is a security hole, not a fix.

What about paywalled or subscriber-only content?

A blanket 401 to everyone (Googlebot included) isn’t your only option if you want gated content to still rank. Google supports indexing paywalled content via the isAccessibleForFree structured data, paired with granting access to Google’s dedicated crawler identities for subscriber/registered-user content. The point of the markup:

“This structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding. helps Google differentiate paywalled content from the practice of cloaking, which violates spam policies.”

The warning baked into that: silently serving full content to Googlebot only, without the structured-data disclosure, is cloaking — a spam-policy risk. If you want gated content indexed, do it the sanctioned way (markup + crawler access), not a quiet bypass.

How to fix an unwanted 401

  1. Confirm it’s actually unwanted. Is this page supposed to be public? If it’s staging or members-only, there’s nothing to fix.
  2. Remove the auth requirement on public pages — strip leftover Basic Auth (.htaccess/nginx), fix expired tokens, resolve plugin/middleware conflicts.
  3. Check your CDN/WAF — and don’t assume you already know which layer issued the 401. The response (and its WWW-Authenticate header) tells you a challenge happened; it doesn’t tell you where it came from — your application, an identity/auth proxy, the CDN or WAF’s bot-management rules, and the origin server can all generate one. False positives from edge bot-management rules are a common cause; inspect logs at each layer and verify Googlebot by reverse DNS before allowlisting anything.
  4. Let verified crawlers through by IP/reverse-DNS, not user-agent.
  5. Don’t reach for 401 to deindex a page you could make public — use noindex (with crawling allowed) instead. A login wall is for content that must be genuinely private.
  6. Validate the fix with URL Inspection’s Live Test — but treat a pass as confirmation of the current fetch, not a guarantee. Google doesn’t commit to a fixed recrawl, re-indexing, or ranking-recovery timeline after you fix a 401; a passing Live Test shows Google’s live fetch got through, not that scheduled crawling or indexing has caught up. Give it time and recheck the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. rather than expecting an instant reversal.

For the full Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. diagnose-fix-validate workflow on the “Blocked due to unauthorized request (401)” Page Indexing status specifically, see the dedicated companion article — this piece stays at the protocol/concept level.

Bing

Bing behaves functionally the same way: a URL returning 401 (or 403) to BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. is inaccessible and won’t be indexed. Bingbot needs unauthenticated access just like Googlebot, and you verify it by its published IP ranges for allowlisting rather than by user-agent.

TIP Check whether a public URL challenges every crawler

A 401 is appropriate for genuinely protected content but can accidentally remove public pages from crawl access. The external response is the evidence that matters.

Run the URL through my HTTP Status Checker to confirm the final status and redirect path without an authenticated browser session. HTTP Status Checker Free

  1. Test the public URL without cookies or credentials.
  2. Confirm protected content returns 401 with the intended authentication challenge and public content does not.
  3. Retest with crawler and browser user agents after changing CDN, WAF, or application rules.

Try it live

This is a real endpoint on this site — not a simulation. Hit it from the button, open it in a new tab, or curl -i it from your terminal, and the server answers with the actual status code this article is about.

Open in new tab ↗

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.