Blocked Due to Unauthorized Request (401)

What the Google Search Console Page Indexing status "Blocked due to unauthorized request (401)" means, how it differs from 403 under RFC 9110, the common causes, how to diagnose it as a bot, and the right fix by page type — public, private, WAF false positive, or paywalled.

First published: Jun 23, 2026 · Last updated: Jul 17, 2026 · Advanced
demand #18 in Indexing#53 in How Search Works#285 in Technical SEO#381 on the site
AIAuthored byAI v2
2 evidence signals on this page

"Blocked due to unauthorized request (401)" is a Google Search Console Page Indexing status meaning Googlebot got an HTTP 401 (authentication required) when it tried to crawl the URL. Google never supplies credentials, so it can't see the page — it won't be indexed, and a previously indexed URL returning 401 eventually gets dropped. By RFC 9110, 401 means the request lacks valid credentials (none sent, or the ones sent were refused); 403 means the server understood the request and refused it, for reasons that aren't always about credentials. Google treats all 4xx except 429 the same for indexing, so the outcome matches, but the cause and fix differ by what the page actually is: remove the auth requirement on an accidentally-gated public page; let verified Googlebot through by IP/reverse-DNS (not a spoofable user-agent) on a page falsely blocked by bot security; keep authentication on genuinely private or staging content rather than opening it to clear the report; use Google's paywall structured data instead of a blanket 401 on indexable subscription content. "Loads fine in my browser" is a trap — you're authenticated; Googlebot isn't. Don't use 401/403 to throttle crawling. Diagnose with URL Inspection's Live Test and curl -I (a missing WWW-Authenticate header means the response is malformed, not proof of a WAF). Live Test and Validate Fix confirm access, not indexing — there's no published retry cadence.

TL;DR — “Blocked due to unauthorized request (401)” means GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. got an HTTP 401 (Unauthorized) — an authentication gate it can’t pass. Google never supplies credentials, so the content is never seen: the page won’t be indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and a previously indexed URL returning 401 gets dropped over time. 401 vs 403, precisely (RFC 9110): 401 = the request lacks valid credentials (none sent, or the ones sent were refused); 403 = the server understood the request and refused it, for reasons that aren’t always about credentials. Google treats all 4xx except 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. the same for indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., so the outcome converges — but the cause and fix differ, and the fix depends on what the page actually is: remove the auth requirement on an accidentally-gated public page; let verified GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. through by IP/reverse-DNS (never a spoofable user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target.) on a page falsely blocked by botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. security; keep authentication on genuinely private or staging content; use Google’s paywall structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding., not a blanket 401, on indexable subscription content. “Loads fine in my browser” is a trap — you’re authenticated; Googlebot isn’t. Don’t use 401/403 to throttle crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. Diagnose as a bot (URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. Live Test, curl -I) — a missing WWW-Authenticate header means the response is malformed, not that a WAF caused it. Live Test and Validate Fix confirm access, not indexing, and there’s no published retry cadence.

What Google is actually telling you

The Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. label reports the response Google observed; it does not identify which authentication, CDN, or application rule produced it. Evidence for this claim The Page Indexing report identifies URLs where Google encountered an authorization request. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: Google: Page indexing report The underlying indexing behavior comes from Google’s documented handling of 4xx responses. Evidence for this claim Google treats 4xx responses other than 429 as if the content does not exist for indexing. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: Google: HTTP status codes

The status comes straight from the server’s response code. Googlebot requested the URL and got an HTTP 401, the “Unauthorized” status — the page is behind an authentication gate (HTTP Basic Auth, a login wall, or an access-control rule). Google’s own definition in the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. is that the page was blocked to Googlebot by a request for authorization, and that if you want it indexed you need to either remove the authorization requirement or let Googlebot through by verifying its identity.

In my own HTTP Status Codes & Their SEO Impact writeup I describe a 401 as the client not having identified or verified itself when needed — a useful mental model, but not the full protocol definition. The precise rule, from RFC 9110: a 401 means the request lacks valid authentication credentials for the resource, and it can also follow credentials the server refuses — so a 401 doesn’t always mean Googlebot (or a browser) sent nothing at all, just that whatever was presented wasn’t valid. Either way the practical outcome for Googlebot is the same: it has no credentials to offer, so it never gets past the gate.

What Google does with a 401

Nothing good, if you wanted the page indexed. Google’s HTTP-status documentation is explicit that all 4xx errors except 429 are treated the same — the crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. inform the next processing system that the content doesn’t exist. So a 401 effectively tells Google “there’s nothing here.” The consequences:

  • The page won’t be indexed. Google never saw content, so there’s nothing to index.
  • A previously indexed page gets dropped. This isn’t unique to 401 — it’s the 4xx family behavior. As I put it in the Ahrefs HTTP status codes writeup, 4xxs will cause pages to drop from the index. A 401 that appears on a page Google had already indexed will, over repeated crawls, lead to that URL falling out.
  • It does not throttle crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. Google says plainly: don’t use 401 and 403 status codes for limiting the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.. 4xx (except 429) has no crawl-rate effect, so you can’t use a 401 as a “slow down” lever — that’s what 503/429 are for.

One thing I’m not going to give you is a specific retry cadence. Google re-crawls over time, but the docs don’t publish a guaranteed “Googlebot retries a 401 every N days” schedule, so I won’t pretend one exists. Treat re-crawl as “it’ll come back around, eventually,” not a stopwatch.

401 vs 403 vs other 4xx — the part that matters

This is the distinction most write-ups blur, and it’s where the real value is. For indexing, Google handles 401 and 403 identically (the 4xx-except-429 rule). But the cause and fix are different because the codes mean different things:

  • 401 Unauthorized401 Unauthorized is a client-error HTTP status code meaning the request lacks valid authentication credentials — the server wants you to log in before it serves the page. For SEO it means the content is gated: Googlebot never sends credentials, so a 401'd page can't be seen and won't be indexed., by the RFC 9110 definition — the request lacks valid authentication credentials for the target resource. That covers two cases: no credentials were sent at all, or credentials were sent and the server refused them. A conformant 401 response must carry a WWW-Authenticate header naming at least one challenge. My own compact version — “the client hasn’t identified or verified itself when needed” — is a useful shorthand for the common case, but treat it as a mental model, not the exhaustive rule.
  • 403 ForbiddenAn HTTP 403 client-error status code meaning the server understood the request but refuses to fulfill it — access is denied, whether or not credentials are involved. For SEO, a 403 served to Googlebot on a page meant to be public and indexable usually points to a misconfigured rule; a persistent 403 keeps a page out of the index either way., by the same RFC — the server understood the request but refuses to fulfill it. Credentials are one possible reason, but the spec is explicit that a request “might be forbidden for reasons unrelated to the credentials.” So 403 does not always mean “the client is known/authenticated but lacks rights” — that’s a common real-world pattern, not a guarantee. On Google’s Page Indexing report specifically, a 403 to Googlebot (which never sends credentials) usually does mean the server is returning that error incorrectly — often a misconfigured firewall, WAF, or bot rule. (There’s a sibling Blocked due to access forbidden (403) status for that one; the diagnosis flow overlaps heavily.)

The practical mental shortcut still holds for triage: 401 ≈ “an auth gate I forgot to lift” (a staging site, leftover HTTP auth), while 403 ≈ “a security rule wrongly blocking Googlebot.” Same indexing outcome, different root cause — just don’t treat either shorthand as the protocol’s actual boundary. The decision table is in the Cheat Sheets tab.

Why you’re seeing it — the common causes

On a page you actually want indexed, a 401 is almost always one of these:

  • A staging or dev site behind Basic Auth. You password-protected a staging environment (the right instinct), but Google discovered the URL somehow — an internal linkAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., a sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., a leaked reference — and is now reporting the 401. If the staging URL genuinely shouldn’t be public, this is expected (see the intentional-401 section).
  • Accidental HTTP auth on a public section. A .htpasswd rule, a “coming soon” plugin, or a maintenance-mode gate left switched on over a directory that’s supposed to be live.
  • A WAF / CDN / IP-allowlist blocking Googlebot. This is the sneaky one. Cloudflare, Akamai, Sucuri, or a geoGenerative Engine Optimization — visibility inside AI answer engines./IP allowlist returns 401 (or 403) to Googlebot’s IPs while serving humans fine. The page “works for everyone” because everyone testing it is coming from an allowed IP.
  • A login wall on subscription or paywalled content you actually want indexed. A blanket 401 keeps Googlebot out entirely — the fix isn’t to weaken the gate, it’s Google’s supported paywall structured data (see the fourth branch below).

How to diagnose it — test as a bot, not a browser

The single biggest trap here is “it works for me.” Of course it does — you’re authenticated, or your IP is on the allowlist, or your browser has a session cookie. Googlebot has none of that. So diagnose like a bot:

  • URL Inspection → Live Test (GSC). This is the closest you’ll get to seeing the real response Googlebot receives. Run it on the affected URL; if it can’t fetch because of authorization, you’ve confirmed the 401 is real and reproducible.

  • curl -I from an unauthenticated context. Request the URL with no cookies and no credentials and read the status line:

    curl -I https://www.example.com/page/
# Look for:  HTTP/1.1 401 Unauthorized
# and a WWW-Authenticate: header confirming an auth gate

    If curl (which sends no session and no auth) gets a 401 while your browser gets a 200, that gap is the bug — your browser is authenticated and Googlebot isn’t. RFC 9110 requires a conformant 401 to carry a WWW-Authenticate header — its presence confirms a real auth challenge. Its absence doesn’t hand you the answer, though: it tells you the response is malformed or incomplete, not which layer produced it. Don’t jump straight to “it must be the WAF” from a missing header alone.

  • Check whether it’s IP-scoped. If curl from your own machine returns 200 but GSC’s Live Test fails, that’s a real signal something is keyed to source IP or request routing — but confirm it by comparing edge/CDN logs, origin logs, the application layer, and any identity provider before naming the WAF as the cause. A HEAD request (what curl -I sends) can also route or cache differently than a GET, so cross-check with an anonymous GET too.

TIP Verify the claimed crawler before changing an allowlist

A Googlebot user-agent string is only a claim. Check the source IP against published ranges before opening a WAF rule; then diagnose the 401 separately.

Spot-check whether a claimed crawler request matches the operator's published ranges with my free Googlebot Verifier Free

  1. Take the source IP and claimed user-agent from the same blocked request in the WAF or access log.
  2. Verify the IP against published crawler ranges rather than trusting the user-agent alone.
  3. Only after identity is established, inspect the authentication or IP rule that returned 401 and retest the live URL.
This request should not be allowlisted as Googlebot from the user-agent claim alone.

The Googlebot Verifier result lists IP address 203.0.113.9 with a Googlebot user-agent claim. The published-range result is mismatch, and the row is labeled for a spot-check.

How to fix it — four branches, by what the page actually is

There isn’t one fix — there are four, and picking the wrong one either exposes content you meant to protect or leaves an indexable page permanently gated. Sort the URL into one of these before touching any config:

1. Genuinely private or staging content → keep the authentication, don’t touch it. If the URL truly shouldn’t be public, the 401 is working as designed — server-side authentication is a legitimate, Google-recommended way to keep content away from everyone, Googlebot included. John Mueller has made this point for staging sites: the right way to hide a site is server-side authentication, by IP, a cookie, or normal server auth, so that normal users — and that includes Googlebot — are blocked from seeing the content. Don’t allowlist Googlebot through a gate that’s protecting genuinely private content just to clear this report row; that defeats the point of the gate. The fix here is discovery hygiene, not access: confirm the URL isn’t linked, sitemapped, or submitted into this Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. property, and leave the authentication in place.

2. A public page accidentally gated → remove the authorization requirement. If the page should be indexed and the gate is a leftover — Basic Auth, a login wall, a maintenance-mode plugin — turn it off for that path. This is the straightforward case: once an anonymous request gets a 200, the page is open to Googlebot.

3. A public page falsely blocked by bot security → admit verified Googlebot, not the literal user-agent string. When a WAF, CDN, or IP-allowlist is rejecting Googlebot on a page you actually want public, allowlist verified Googlebot by IP / reverse-DNS. The user-agent is trivially spoofable; anyone can claim to be Googlebot. Google’s recommended path is to verify the crawler by its published IP ranges or a reverse-then-forward DNS check, then allow those specific requests past the gate — you’re not removing the security rule, you’re carving out a verified exception to it. To fix the false positive itself:

  • Identify the rule returning 401/403 to Googlebot (Cloudflare Firewall Events, Akamai/Sucuri logs, or your own edge/origin/application logs).
  • Allowlist Google’s verified IP ranges (or the bot category) rather than disabling protection wholesale.
  • Re-test with URL Inspection Live Test until Google can fetch.

4. Subscription or paywalled content you want indexed → don’t use a blanket 401 at all. If the page is registration- or subscription-gated but you want it discoverable in Search, a hard 401 is the wrong tool regardless of cause — Googlebot still can’t fetch it. Google documents a supported paywall implementation instead: serve the page with the appropriate paywalled-content structured data (isAccessibleForFree, hasPart, and the related properties) so Google can index the free-preview portion without you having to open the whole page. That’s a markup and server-response change, not an auth-gate change.

Validate the fix and set expectations

Once you’ve genuinely opened the page (and confirmed with curl/Live Test that an unauthenticated request now returns 200), be precise about what each step actually confirms:

  • Live Test confirms access, not indexing. Google’s own URL Inspection documentation says the live test only confirms whether Google-InspectionTool can currently access and parse the page — there’s no test that guarantees the page will end up in the index or appear in Search results. A passing Live Test means the gate is open; it doesn’t promise anything about what happens next.
  • Validate Fix is optional, not required. Google updates the issue count whenever it recrawls a page, whether or not you clicked Validate Fix. Use it for your own tracking on a real correction — don’t validate a URL that’s supposed to stay private, and don’t treat validation itself as something that speeds up re-indexing.
  • Don’t expect instant re-indexing. Re-crawl and re-index take time, and — as noted above — there’s no published retry cadence I’ll quote you. Monitor the actual indexed result and search performance separately from the report status; neither Live Test nor Validate Fix guarantees canonical selection or search appearance.
  • A myth to drop on the way out: a 401 in GSC is not a penalty or a manual action. It’s a crawl-access status. It doesn’t hurt rankings of your other pages and it doesn’t “blacklist” your site — it just keeps the gated page out of the index.

Where this sits

This is one of the HTTP-status statuses in the Page Indexing report — siblings include Blocked due to access forbidden (403), the other 4xx and 404 statuses, and the 5xx server-error status. For the report itself and how to read the “Why pages aren’t indexed” table, see the Page Indexing report hub. The underlying mechanics — how Googlebot fetches and what status codes mean to it — are covered in crawling and indexing.

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.