Blocked Due to Unauthorized Request (401)

TL;DR — “Blocked due to unauthorized request (401)” means Googlebot tried to read your page and got asked to log in. Google doesn’t have a password for your site, so it gives up and the page can’t be indexed. If you want that page in Google, something is gating it that shouldn’t be — usually leftover login protection, a staging-site password, or a security rule that’s blocking Google by mistake. If the page is meant to be private, this is normal and there’s nothing to fix.

What this status means

This report label means Google received an HTTP 401 authorization response for the URL. Evidence for this claim The Page Indexing report identifies URLs where Google encountered an authorization request. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report Google treats a persistent 4xx response other than 429 as unavailable content for indexing. Evidence for this claim Google treats 4xx responses other than 429 as if the content does not exist for indexing. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: HTTP status codes

When you see “Blocked due to unauthorized request (401)” in Google Search Console’s Page Indexing report, it’s telling you this: Googlebot went to crawl the URL, and your server answered with an HTTP 401, which is the code for “authentication required” — basically “you need to log in to see this.”

Googlebot doesn’t have a username and password for your site, and it never will. So when a page demands a login, Googlebot can’t get in, can’t read the content, and can’t index the page. If the page used to be in Google and then started returning 401, Google eventually drops it from search.

Is it a problem?

It depends on whether you want that page in Google:

• You want it indexed → yes, it’s a problem. Something is putting a login wall in front of a page that should be public. You need to find what’s gating it and open it up.
• The page is private/staging → no, it’s working as intended. A 401 is a perfectly good way to keep a private area out of Google, and you shouldn’t weaken that gate just to clear this row from the report. The only thing to check is whether that URL should even be linked, sitemapped, or submitted into this Search Console property.

”But the page loads fine for me!”

This is the most common confusion. You open the URL in your browser and it works — so how can Google say it’s blocked? Because you’re logged in (or your office IP is whitelisted) and Googlebot isn’t. You’re seeing the page past the gate; Googlebot is hitting the gate. To see what Google sees, you have to test the page as an anonymous visitor — the Advanced tab shows you how.

What usually causes it

• A staging or test site left behind a password.
• Login protection accidentally left on a section that should be public.
• A security tool or CDN (like Cloudflare) blocking Googlebot by mistake.
• A real login wall on content you meant to gate (members-only, etc.).

Want the real diagnosis steps, the 401-vs-403 difference, and the exact fixes? Switch to the Advanced tab.

TL;DR — “Blocked due to unauthorized request (401)” means Googlebot got an HTTP 401 (Unauthorized) — an authentication gate it can’t pass. Google never supplies credentials, so the content is never seen: the page won’t be indexed, and a previously indexed URL returning 401 gets dropped over time. 401 vs 403, precisely (RFC 9110): 401 = the request lacks valid credentials (none sent, or the ones sent were refused); 403 = the server understood the request and refused it, for reasons that aren’t always about credentials. Google treats all 4xx except 429 the same for indexing, so the outcome converges — but the cause and fix differ, and the fix depends on what the page actually is: remove the auth requirement on an accidentally-gated public page; let verified Googlebot through by IP/reverse-DNS (never a spoofable user-agent) on a page falsely blocked by bot security; keep authentication on genuinely private or staging content; use Google’s paywall structured data, not a blanket 401, on indexable subscription content. “Loads fine in my browser” is a trap — you’re authenticated; Googlebot isn’t. Don’t use 401/403 to throttle crawling. Diagnose as a bot (URL Inspection Live Test, curl -I) — a missing WWW-Authenticate header means the response is malformed, not that a WAF caused it. Live Test and Validate Fix confirm access, not indexing, and there’s no published retry cadence.

What Google is actually telling you

The Page Indexing label reports the response Google observed; it does not identify which authentication, CDN, or application rule produced it. Evidence for this claim The Page Indexing report identifies URLs where Google encountered an authorization request. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report The underlying indexing behavior comes from Google’s documented handling of 4xx responses. Evidence for this claim Google treats 4xx responses other than 429 as if the content does not exist for indexing. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: HTTP status codes

The status comes straight from the server’s response code. Googlebot requested the URL and got an HTTP 401, the “Unauthorized” status — the page is behind an authentication gate (HTTP Basic Auth, a login wall, or an access-control rule). Google’s own definition in the Page Indexing report is that the page was blocked to Googlebot by a request for authorization, and that if you want it indexed you need to either remove the authorization requirement or let Googlebot through by verifying its identity.

In my own HTTP Status Codes & Their SEO Impact writeup I describe a 401 as the client not having identified or verified itself when needed — a useful mental model, but not the full protocol definition. The precise rule, from RFC 9110: a 401 means the request lacks valid authentication credentials for the resource, and it can also follow credentials the server refuses — so a 401 doesn’t always mean Googlebot (or a browser) sent nothing at all, just that whatever was presented wasn’t valid. Either way the practical outcome for Googlebot is the same: it has no credentials to offer, so it never gets past the gate.

What Google does with a 401

Nothing good, if you wanted the page indexed. Google’s HTTP-status documentation is explicit that all 4xx errors except 429 are treated the same — the crawlers inform the next processing system that the content doesn’t exist. So a 401 effectively tells Google “there’s nothing here.” The consequences:

• The page won’t be indexed. Google never saw content, so there’s nothing to index.
• A previously indexed page gets dropped. This isn’t unique to 401 — it’s the 4xx family behavior. As I put it in the Ahrefs HTTP status codes writeup, 4xxs will cause pages to drop from the index. A 401 that appears on a page Google had already indexed will, over repeated crawls, lead to that URL falling out.
• It does not throttle crawling. Google says plainly: don’t use 401 and 403 status codes for limiting the crawl rate. 4xx (except 429) has no crawl-rate effect, so you can’t use a 401 as a “slow down” lever — that’s what 503/429 are for.

One thing I’m not going to give you is a specific retry cadence. Google re-crawls over time, but the docs don’t publish a guaranteed “Googlebot retries a 401 every N days” schedule, so I won’t pretend one exists. Treat re-crawl as “it’ll come back around, eventually,” not a stopwatch.

401 vs 403 vs other 4xx — the part that matters

This is the distinction most write-ups blur, and it’s where the real value is. For indexing, Google handles 401 and 403 identically (the 4xx-except-429 rule). But the cause and fix are different because the codes mean different things:

• 401 Unauthorized, by the RFC 9110 definition — the request lacks valid authentication credentials for the target resource. That covers two cases: no credentials were sent at all, or credentials were sent and the server refused them. A conformant 401 response must carry a WWW-Authenticate header naming at least one challenge. My own compact version — “the client hasn’t identified or verified itself when needed” — is a useful shorthand for the common case, but treat it as a mental model, not the exhaustive rule.
• 403 Forbidden, by the same RFC — the server understood the request but refuses to fulfill it. Credentials are one possible reason, but the spec is explicit that a request “might be forbidden for reasons unrelated to the credentials.” So 403 does not always mean “the client is known/authenticated but lacks rights” — that’s a common real-world pattern, not a guarantee. On Google’s Page Indexing report specifically, a 403 to Googlebot (which never sends credentials) usually does mean the server is returning that error incorrectly — often a misconfigured firewall, WAF, or bot rule. (There’s a sibling Blocked due to access forbidden (403) status for that one; the diagnosis flow overlaps heavily.)

The practical mental shortcut still holds for triage: 401 ≈ “an auth gate I forgot to lift” (a staging site, leftover HTTP auth), while 403 ≈ “a security rule wrongly blocking Googlebot.” Same indexing outcome, different root cause — just don’t treat either shorthand as the protocol’s actual boundary. The decision table is in the Cheat Sheets tab.

Why you’re seeing it — the common causes

On a page you actually want indexed, a 401 is almost always one of these:

• A staging or dev site behind Basic Auth. You password-protected a staging environment (the right instinct), but Google discovered the URL somehow — an internal link, a sitemap, a leaked reference — and is now reporting the 401. If the staging URL genuinely shouldn’t be public, this is expected (see the intentional-401 section).
• Accidental HTTP auth on a public section. A .htpasswd rule, a “coming soon” plugin, or a maintenance-mode gate left switched on over a directory that’s supposed to be live.
• A WAF / CDN / IP-allowlist blocking Googlebot. This is the sneaky one. Cloudflare, Akamai, Sucuri, or a geo/IP allowlist returns 401 (or 403) to Googlebot’s IPs while serving humans fine. The page “works for everyone” because everyone testing it is coming from an allowed IP.
• A login wall on subscription or paywalled content you actually want indexed. A blanket 401 keeps Googlebot out entirely — the fix isn’t to weaken the gate, it’s Google’s supported paywall structured data (see the fourth branch below).

How to diagnose it — test as a bot, not a browser

The single biggest trap here is “it works for me.” Of course it does — you’re authenticated, or your IP is on the allowlist, or your browser has a session cookie. Googlebot has none of that. So diagnose like a bot:

• URL Inspection → Live Test (GSC). This is the closest you’ll get to seeing the real response Googlebot receives. Run it on the affected URL; if it can’t fetch because of authorization, you’ve confirmed the 401 is real and reproducible.

• curl -I from an unauthenticated context. Request the URL with no cookies and no credentials and read the status line:

  curl -I https://www.example.com/page/
  # Look for:  HTTP/1.1 401 Unauthorized
  # and a WWW-Authenticate: header confirming an auth gate

  If curl (which sends no session and no auth) gets a 401 while your browser gets a 200, that gap is the bug — your browser is authenticated and Googlebot isn’t. RFC 9110 requires a conformant 401 to carry a WWW-Authenticate header — its presence confirms a real auth challenge. Its absence doesn’t hand you the answer, though: it tells you the response is malformed or incomplete, not which layer produced it. Don’t jump straight to “it must be the WAF” from a missing header alone.

• Check whether it’s IP-scoped. If curl from your own machine returns 200 but GSC’s Live Test fails, that’s a real signal something is keyed to source IP or request routing — but confirm it by comparing edge/CDN logs, origin logs, the application layer, and any identity provider before naming the WAF as the cause. A HEAD request (what curl -I sends) can also route or cache differently than a GET, so cross-check with an anonymous GET too.

How to fix it — four branches, by what the page actually is

There isn’t one fix — there are four, and picking the wrong one either exposes content you meant to protect or leaves an indexable page permanently gated. Sort the URL into one of these before touching any config:

1. Genuinely private or staging content → keep the authentication, don’t touch it. If the URL truly shouldn’t be public, the 401 is working as designed — server-side authentication is a legitimate, Google-recommended way to keep content away from everyone, Googlebot included. John Mueller has made this point for staging sites: the right way to hide a site is server-side authentication, by IP, a cookie, or normal server auth, so that normal users — and that includes Googlebot — are blocked from seeing the content. Don’t allowlist Googlebot through a gate that’s protecting genuinely private content just to clear this report row; that defeats the point of the gate. The fix here is discovery hygiene, not access: confirm the URL isn’t linked, sitemapped, or submitted into this Search Console property, and leave the authentication in place.

2. A public page accidentally gated → remove the authorization requirement. If the page should be indexed and the gate is a leftover — Basic Auth, a login wall, a maintenance-mode plugin — turn it off for that path. This is the straightforward case: once an anonymous request gets a 200, the page is open to Googlebot.

3. A public page falsely blocked by bot security → admit verified Googlebot, not the literal user-agent string. When a WAF, CDN, or IP-allowlist is rejecting Googlebot on a page you actually want public, allowlist verified Googlebot by IP / reverse-DNS. The user-agent is trivially spoofable; anyone can claim to be Googlebot. Google’s recommended path is to verify the crawler by its published IP ranges or a reverse-then-forward DNS check, then allow those specific requests past the gate — you’re not removing the security rule, you’re carving out a verified exception to it. To fix the false positive itself:

• Identify the rule returning 401/403 to Googlebot (Cloudflare Firewall Events, Akamai/Sucuri logs, or your own edge/origin/application logs).
• Allowlist Google’s verified IP ranges (or the bot category) rather than disabling protection wholesale.
• Re-test with URL Inspection Live Test until Google can fetch.

4. Subscription or paywalled content you want indexed → don’t use a blanket 401 at all. If the page is registration- or subscription-gated but you want it discoverable in Search, a hard 401 is the wrong tool regardless of cause — Googlebot still can’t fetch it. Google documents a supported paywall implementation instead: serve the page with the appropriate paywalled-content structured data (isAccessibleForFree, hasPart, and the related properties) so Google can index the free-preview portion without you having to open the whole page. That’s a markup and server-response change, not an auth-gate change.

Validate the fix and set expectations

Once you’ve genuinely opened the page (and confirmed with curl/Live Test that an unauthenticated request now returns 200), be precise about what each step actually confirms:

• Live Test confirms access, not indexing. Google’s own URL Inspection documentation says the live test only confirms whether Google-InspectionTool can currently access and parse the page — there’s no test that guarantees the page will end up in the index or appear in Search results. A passing Live Test means the gate is open; it doesn’t promise anything about what happens next.
• Validate Fix is optional, not required. Google updates the issue count whenever it recrawls a page, whether or not you clicked Validate Fix. Use it for your own tracking on a real correction — don’t validate a URL that’s supposed to stay private, and don’t treat validation itself as something that speeds up re-indexing.
• Don’t expect instant re-indexing. Re-crawl and re-index take time, and — as noted above — there’s no published retry cadence I’ll quote you. Monitor the actual indexed result and search performance separately from the report status; neither Live Test nor Validate Fix guarantees canonical selection or search appearance.
• A myth to drop on the way out: a 401 in GSC is not a penalty or a manual action. It’s a crawl-access status. It doesn’t hurt rankings of your other pages and it doesn’t “blacklist” your site — it just keeps the gated page out of the index.

Where this sits

This is one of the HTTP-status statuses in the Page Indexing report — siblings include Blocked due to access forbidden (403), the other 4xx and 404 statuses, and the 5xx server-error status. For the report itself and how to read the “Why pages aren’t indexed” table, see the Page Indexing report hub. The underlying mechanics — how Googlebot fetches and what status codes mean to it — are covered in crawling and indexing.

AI summary

A condensed take on the Advanced version:

• What it is. A GSC Page Indexing status meaning Googlebot got an HTTP 401 (Unauthorized) — an authentication gate it can’t pass. Google supplies no credentials, so it never sees the content.
• What Google does. Not indexed; a previously indexed URL returning 401 gets dropped over time. All 4xx except 429 are treated the same — Google is told “the content doesn’t exist.” 4xx has no crawl-rate effect, so don’t use 401/403 to throttle Googlebot.
• 401 vs 403, precisely. By RFC 9110: 401 = the request lacks valid credentials (none sent, or the ones sent were refused); 403 = the server understood the request and refused it, for reasons that aren’t always about credentials. “401 = auth gate, 403 = security rule wrongly blocking Googlebot” is a useful triage shortcut, not the exhaustive protocol rule.
• Common causes (on pages you want indexed): staging site behind Basic Auth, accidental HTTP auth on a public section, a WAF/CDN/IP-allowlist excluding Googlebot, or a login wall on subscription/paywalled content.
• “Loads for me” is a trap. You’re authenticated or IP-allowlisted; Googlebot isn’t. Diagnose as a bot: URL Inspection Live Test and curl -I (no cookies, no auth) — a 401 there confirms it. A WWW-Authenticate header confirms a real auth gate (RFC 9110 requires it on a conformant 401); its absence just means the response is malformed — check edge, origin, application, and identity-provider logs before blaming a WAF.
• Fix by what the page is, not one universal fix. Public page accidentally gated → remove the authorization requirement. Public page falsely blocked by bot security → allowlist verified Googlebot by IP/reverse-DNS (never the spoofable user-agent string), without disabling the rule wholesale. Genuinely private/staging content → keep the authentication; never open it just to clear this report row — only clean up its discovery (sitemaps/links/property). Indexable subscription/paywalled content → use Google’s paywall structured data instead of a blanket 401.
• Validate & wait, without over-promising. Live Test confirms current access only, not indexing. Validate Fix is optional — Google updates the count on its next recrawl regardless. Re-indexing takes time, there’s no published retry cadence, and neither step guarantees indexing, canonical selection, or search appearance. A 401 is not a penalty.

Official documentation

Primary-source documentation from the search engines.

Google

• Page Indexing report — the report itself and the definition of the “Blocked due to unauthorized request (401)” status (plus the 403 sibling and the other HTTP-status statuses).
• How HTTP status codes, and network and DNS errors affect Google Search — what Googlebot does with a 401: the 4xx-except-429 handling and the “don’t use 401/403 to limit crawl rate” rule.
• Verifying Googlebot and other Google crawlers — the recommended way to allowlist Googlebot: verify by IP / reverse-DNS, not by user-agent string.
• Googlebot IP ranges (googlebot.json) — the published IP ranges to allowlist past a WAF/CDN.

Bing / Microsoft

• Bing Webmaster Tools Help — Bing doesn’t surface an identical status string, but a URL returning 401/403 is likewise treated as inaccessible and won’t be indexed; bingbot also needs to reach the page anonymously and publishes verified IP ranges / reverse-DNS verification for the same allowlist fix.

Quotes from the source

On-the-record statements. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what the 401 status means (Page Indexing report)

• “The page was blocked to Googlebot by a request for authorization (401 response). If you do want Googlebot to be able to index this page, either remove authorization requirements for this page, or else allow Googlebot to access your pages by verifying its identity.” — Google Search Console Help, Page Indexing report. Jump to quote

Google — what Googlebot does with a 401 (HTTP status codes doc)

• “All 4xx errors, except 429, are treated the same: Google crawlers inform the next processing system that the content doesn’t exist.” — Google Search Central docs. Jump to quote
• “Don’t use 401 and 403 status codes for limiting the crawl rate.” Jump to quote

Patrick Stox — my own 401/403 definitions and the 4xx index impact

• “The client hasn’t identified or verified itself when needed.” (401) — Patrick Stox, HTTP Status Codes & Their SEO Impact, Ahrefs. Jump to quote
• “The client is known but doesn’t have access rights.” (403) Jump to quote
• “4xxs will cause pages to drop from the index.” Jump to quote

John Mueller, Google — server-side auth is the right way to gate a site

• “Ideally, what you would want to do is provide some kind of server side authentication on the server so that normal users when they go there they would get blocked from being able to see the content; that would include GoogleBot.” (Webmaster Hangout, Sep 25 2019, relayed via Search Engine Journal.) Read the coverage

401 diagnosis & fix checklist

Run this when GSC flags “Blocked due to unauthorized request (401)”:

• Decide intent first — do you actually want this URL indexed? If it’s genuinely private/staging, the 401 is correct; skip to the last item. If it’s subscription/paywalled content you do want indexed, skip the auth-removal steps and use the paywall structured-data fix instead.
• Reproduce as a bot — run URL Inspection → Live Test on the URL and confirm the authorization failure (don’t trust your logged-in browser).
• curl -I with no cookies/credentials — confirm it returns 401 and look for a WWW-Authenticate header. A 401 here while your browser gets 200 is the bug. A header confirms a real auth gate; its absence flags a malformed response, not the layer that produced it.
• Identify the gate — check edge/CDN, origin, application, and identity logs to narrow it down: Basic Auth (.htpasswd), a maintenance/“coming soon” plugin, a login wall, or a WAF/CDN/IP-allowlist rule.
• If it’s a WAF/CDN — check firewall events for the rule blocking Google; allowlist verified Googlebot IP ranges, don’t disable protection wholesale.
• Fix by the right method — remove the auth requirement on an accidentally gated public page, or allow verified Googlebot by IP / reverse-DNS (never by the spoofable user-agent string) for a bot-security false positive. Never allowlist Googlebot through a gate that’s protecting genuinely private content.
• Re-test — curl -I (still unauthenticated) now returns 200, and URL Inspection Live Test can fetch.
• Validate Fix (optional) in the Page Indexing report and re-inspect a sample URL — Google updates the count on its next recrawl either way; a passing Live Test confirms access, not indexing.
• Set expectations — re-crawl/re-index takes time; no guaranteed retry cadence, and no guarantee of indexing or search appearance. It’s not a penalty.
• If intentional — confirm the staging/private URL shouldn’t be in this property (stop linking/submitting it), and leave the gate in place.

401 vs 403 vs other 4xx — cheat sheet

What each code actually means (and the typical cause)

How Google treats them for indexing

The fix-it map

Two official fixes for a public page wrongly gated (Google’s wording)

1. Remove the authorization requirement for the page.
2. Let Googlebot through by verifying its identity — allowlist by IP / reverse-DNS, not by the (spoofable) user-agent string.

Neither applies to genuinely private/staging content (keep the gate) or to indexable paywalled content (use the paywall markup instead of opening the gate).

The mental models

1. Gate vs. content. A 401 isn’t a content problem — Googlebot never reached the content. It’s a gate problem. So you don’t edit the page; you change what the gate does to an unauthenticated bot. Always separate “is the page good?” from “can Googlebot get past the door?”

2. Intent decides everything. Before any fix, answer one question: should this URL be indexed? If yes, a 401 is a misconfiguration to remove. If no, the 401 is working as designed and the real question is why that URL is even in this Search Console property. Don’t fix a 401 that’s doing its job.

3. Test as a bot, not as yourself. “It works for me” is the default failure of judgment here. You carry a session, a cookie, an allowlisted IP. Googlebot carries none of that. Every diagnosis starts by stripping that away — curl -I with no credentials, or URL Inspection’s Live Test.

4. 401 vs 403 — same outcome, different door. For indexing they’re identical (4xx-except-429). By the RFC, 401 = the request lacks valid credentials (none sent, or the ones sent were refused); 403 = the server understood the request and refused it, for reasons that may or may not be about credentials. Treat “401 = auth gate I control, 403 = security rule misfiring” as the practical triage shortcut, not the protocol’s actual boundary — it’s right often enough to be useful, but the RFC’s wording is what’s actually true. Diagnose toward the door: 401 → look at auth/staging; 403 → look at the WAF/firewall.

5. Verify the bot, don’t trust the name. The fix that bites people later is “allowlist the Googlebot user-agent.” That string is free for anyone to claim. The durable fix is identity by IP / reverse-DNS — allow the bot you can prove is Googlebot, not the one that merely says so.

Diagnosing a 401: intentional gate or misconfiguration?

The first fork is intent, not technology — decide whether the URL should be indexed before you touch any config. Once you know it’s a real misconfiguration, the second fork is whether you’re looking at a login wall you forgot to lift or a WAF/CDN rule that’s blocking Googlebot by accident. Click through it.

Track the 401 bucket, not just its existence

The one number worth watching is how many URLs sit in the “Blocked due to unauthorized request (401)” row of the Page Indexing report over time, not just whether the row exists — a single snapshot can’t tell you whether you’re fixing the problem or accumulating more gated URLs.

401-bucket count over time

• Metric — The count of URLs under “Blocked due to unauthorized request (401)” in GSC’s Page Indexing report, tracked week over week.
• What it tells you — Whether a fix actually worked and whether new 401s are creeping in. After you remove an auth requirement or allowlist verified Googlebot, this count should fall toward zero for the URLs you fixed. For URLs that are intentionally gated (staging, private sections), the count should stay flat — a rising count there usually means new staging/private URLs are getting linked or sitemapped into this property by mistake.
• How to pull it — GSC Page Indexing report, filtered to the 401 row; spot-check individual URLs with URL Inspection → Live Test to confirm they’re not a stale snapshot.
• Benchmark / realistic range — There’s no universal target — it depends entirely on how many pages you intentionally gate. The honest bar: zero 401s among URLs you want indexed, and a flat (not growing) count for URLs you deliberately keep behind auth. Establish your own baseline count before judging trend direction.
• Cadence — Weekly right after a fix, until the count stabilizes; monthly afterward as a regression check.

Runbook: diagnose, fix, validate

The 401 workflow is a short, ordered loop — don’t skip straight to “fix” before you’ve confirmed intent and reproduced the block as a bot.

1. Decide intent first. Should this URL actually be indexed? If it’s genuinely private or staging, stop here — the 401 is correct, and the only follow-up is making sure the URL isn’t linked or sitemapped into this property; don’t weaken the gate. If it’s subscription/paywalled content you do want indexed, skip to the paywall structured-data fix instead of touching the auth gate.

2. Reproduce it as a bot, not as yourself. Run URL Inspection → Live Test in GSC, and separately request the URL with curl -I using no cookies and no credentials. If curl gets a 401 while your logged-in browser gets a 200, that gap is the bug — you’re authenticated or IP-allowlisted, Googlebot isn’t.

3. Identify the gate. Check whether the anonymous response carries a WWW-Authenticate header — a conformant 401 must include one, so its presence confirms a real auth gate (Basic Auth, login wall, maintenance mode). Its absence doesn’t hand you the cause; it means the response is malformed, so cross-check edge/CDN, origin, application, and identity-provider logs before concluding it’s a WAF/CDN/ allowlist rule.

4. Fix by cause. Auth gate on an accidentally-public page → remove the authorization requirement for that path. WAF/CDN rule falsely blocking a public page → allowlist verified Googlebot by IP or reverse-DNS in the firewall, never by the spoofable user-agent string alone. Never allowlist Googlebot through a gate protecting genuinely private content.

5. Validate. Re-run the same anonymous curl -I and confirm it now returns 200. Validate Fix in the Page Indexing report is optional tracking — Google updates the issue count on its next recrawl regardless — and URL Inspection’s Live Test only confirms Google-InspectionTool can currently access the page, not that it will be indexed.

6. Set expectations and stop. Re-crawl and re-index take time, there’s no published retry cadence, and neither Live Test nor Validate Fix guarantees indexing or search appearance — don’t “fix” the same URL twice while waiting. If the 401-bucket count in GSC isn’t trending down after a couple of weeks, go back to step 2 and re-reproduce rather than guessing at a new cause.

Ready-to-use AI prompts

Copy-paste prompts for classifying a 401 from raw diagnostic output. These help you triage the likely cause fast — always confirm the conclusion against the actual server/firewall config before changing anything.

Classify the likely cause from curl output

I'm diagnosing a "Blocked due to unauthorized request (401)" status in Google
Search Console. Below is the raw output of an unauthenticated request to the
URL (curl -I, no cookies, no credentials). Based on this output alone,
classify the most likely cause as one of: (1) staging/dev site behind Basic
Auth, (2) accidental HTTP auth or maintenance-mode gate left on a public
section, (3) WAF/CDN/IP-allowlist blocking Googlebot, (4) a login wall on
content that's meant to be gated. Explain which specific header or detail in
the output pointed you to that answer, and tell me what's missing if you
can't tell for sure.

CURL OUTPUT:
[paste]

Classify the cause from a WAF/firewall log line

I'm investigating why Googlebot is getting a 401 or 403 on a page I want
indexed. Below is a log line (or a few) from my WAF/CDN showing a blocked
request. Tell me whether this looks like a bot-identity rule (blocking by
user-agent or IP range), a rate-limit/challenge rule, or something else, and
what I'd need to allowlist — by IP range or reverse-DNS — to let verified
Googlebot through without disabling the rule entirely.

LOG LINE(S):
[paste]

Sanity-check a fix before shipping it

I'm about to remove an authorization requirement from a URL that's currently
returning 401 to Googlebot, because I want it indexed. Here's a short
description of the current gate and what I'm about to change: [describe].
Point out anything I might be missing — e.g. whether this could accidentally
expose a section I meant to keep private, or whether I should allowlist
verified Googlebot instead of removing the auth requirement outright.

Test yourself: “Blocked due to unauthorized request (401)”

Five questions on what a 401 means, how it differs from a 403, and how to diagnose and fix it. Pick an answer for each, then check.

Reproduce the 401 as an anonymous request

The point of these checks is the same one the Advanced tab makes: your browser is authenticated, Googlebot isn’t. Strip out cookies and credentials entirely and see what an anonymous client actually gets.

macOS / Linux

# Fetch headers only, with no cookies and no credentials
curl -sI https://www.example.com/page/
# Look for:
#   HTTP/1.1 401 Unauthorized
#   WWW-Authenticate: Basic realm="..."   <- confirms a real auth gate

Windows (PowerShell)

# -SkipHttpErrorCheck so PowerShell shows the 401 instead of throwing
Invoke-WebRequest -Uri "https://www.example.com/page/" -Method Head `
  -SkipHttpErrorCheck | Select-Object StatusCode, Headers
# Check the Headers output for WWW-Authenticate

If this returns 401 with a WWW-Authenticate header, you’ve confirmed a real authentication gate (Basic Auth, login wall) — RFC 9110 requires a conformant 401 to send that header. If it returns 401/403 with no WWW-Authenticate header, that tells you the response is malformed or incomplete, not which layer produced it — the header’s absence alone doesn’t prove a WAF or CDN. If the block also only happens from outside your own network, that IP-scoping is the stronger lead toward a WAF/CDN/IP-allowlist rule; confirm it in the firewall/CDN logs, re-test from a different network, or use GSC’s URL Inspection Live Test, which requests as Googlebot itself.

Verify a bot is really Googlebot (before you allowlist)

Before you open a WAF rule to let a “Googlebot” request through, confirm the IP is genuinely Google’s — plenty of traffic spoofs the user-agent string.

macOS / Linux

# 1) Reverse DNS the IP from your logs — must end in googlebot.com / google.com
host 66.249.66.1
# → ... domain name pointer crawl-66-249-66-1.googlebot.com

# 2) Forward DNS that hostname back — it must resolve to the same IP
host crawl-66-249-66-1.googlebot.com
# → crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Windows

nslookup 66.249.66.1
nslookup crawl-66-249-66-1.googlebot.com

If the reverse lookup doesn’t end in a Google domain, or the forward lookup doesn’t match the original IP, don’t allowlist it — it isn’t Googlebot. You can also check the IP against Google’s published ranges (googlebot.json), or verify it directly with the Googlebot Verifier tool. Allowlist by verified identity, never by the user-agent string alone.

Tools for diagnosing and fixing a 401

• HTTP Status Checker — paste the affected URL (or a batch) to confirm the status code, see the full response chain, and catch anything that redirects before it hits the auth gate.
• Googlebot Verifier — check whether an IP claiming to be Googlebot is genuine (published IP ranges plus reverse-DNS confirmation) before you allowlist it past a WAF or firewall rule.
• Google Search Console — URL Inspection → Live Test — the closest you’ll get to seeing the exact response Googlebot receives, including whether it’s currently blocked by authorization.
• curl -I — the fastest way to request a URL with no cookies or credentials and read the raw status line and headers, including WWW-Authenticate.
• Your WAF/CDN’s firewall event log (Cloudflare, Akamai, Sucuri, etc.) — find the specific rule returning 401/403 to Googlebot’s IP ranges.

Prove the fix actually worked

Once you’ve removed an auth requirement or allowlisted verified Googlebot, these are the checks that separate “the config changed” from “Google can now actually reach the page.” Run them in order.

Test 1 — An anonymous request now returns 200

• Test to run — Run curl -I with no cookies or credentials on the affected URL (or check it with the HTTP Status Checker).
• Expected result — The status line reads HTTP/1.1 200 OK with no WWW-Authenticate header.
• Failure interpretation — Still 401 means the auth requirement wasn’t actually removed for that path, or you’re testing the wrong URL/environment. A 403 instead of 401 means you’ve traded one gate for another — check the WAF/CDN rule.
• Monitoring window — Immediate — the server answers as soon as the change is live.
• Rollback trigger — If removing the gate exposes content you meant to keep private, put the auth requirement back immediately and allowlist verified Googlebot by IP/reverse-DNS instead.

Test 2 — Google confirms it can now reach the page

• Test to run — Run URL Inspection → Test Live URL in Google Search Console on the affected URL.
• Expected result — The live test succeeds and shows the page content — no authorization failure reported. This confirms Google-InspectionTool can currently access and parse the page — it’s an access result, not an indexing guarantee; Google’s own docs say the live test doesn’t test every indexing condition and a pass doesn’t guarantee inclusion in the index.
• Failure interpretation — If Live Test still reports an authorization block after the anonymous curl test passes, suspect a rule that’s IP-scoped specifically to Googlebot’s ranges (a WAF/CDN allowlist issue), not a general auth gate.
• Monitoring window — Immediate to a few minutes after the fix.
• Rollback trigger — N/A — this is a read-only test; if it still fails, go back to Test 1’s failure branch rather than rolling anything back.

Test 3 — The 401 status clears in the Page Indexing report

• Test to run — Use Validate Fix (optional — Google updates the count on its next recrawl regardless) on the “Blocked due to unauthorized request (401)” issue in the Page Indexing report, and watch the bucket count over the following weeks (see the How to Measure tab).
• Expected result — The URL moves out of the 401 bucket, and — if it was previously indexed — reappears in the index over time. Neither Validate Fix nor a passing Live Test guarantees indexing, canonical selection, or search appearance — track the actual indexed state and search performance separately.
• Failure interpretation — No published retry cadence exists, so don’t read a slow validation as a new failure; if the count in the 401 bucket isn’t trending down after a couple of weeks, re-run Test 1 to confirm the fix is still in place (a redeploy or CDN cache can silently reintroduce the gate).
• Monitoring window — Days to a few weeks, tracked via the 401-bucket count.
• Rollback trigger — Only revisit the gate itself if Test 1 starts failing again — don’t chase the Page Indexing report’s timing.

Blocked due to unauthorized request (401)

A Google Search Console Page Indexing status: Googlebot was refused with an HTTP 401 (authentication required), so the page is gated and can't be indexed.

Parent concept: Page Indexing report · Related: Crawling, Indexing, Page Indexing report

Blocked due to unauthorized request (401)

“Blocked due to unauthorized request (401)” is a Google Search Console Page Indexing status. It means Googlebot tried to crawl the URL but the server answered with an HTTP 401 (Unauthorized) — the page sits behind an authentication gate (HTTP auth, a login wall, or an access-control rule) that Googlebot couldn’t get past. Because Googlebot never sees the content, the page is not indexed, and a URL that was previously indexed and then starts returning 401 eventually gets dropped from the index.

By RFC 9110, a 401 means the request lacks valid authentication credentials — either none were sent, or the ones sent were refused. A 403 (Forbidden) means the server understood the request but refused it, for reasons the spec explicitly says aren’t always about credentials — “the client is known but lacks access rights” is a common real-world pattern, not the universal definition. Google treats all 4xx except 429 the same for indexing, so the outcome is identical — but the cause and the fix differ, which is why the 401-vs-403 distinction matters.

The right fix depends on what the page actually is, not one universal move: remove the authorization requirement on a public page that was accidentally gated; allowlist verified Googlebot by IP/reverse-DNS (never a spoofable user-agent) on a page falsely blocked by a WAF/CDN/IP-allowlist; keep the authentication in place on genuinely private or staging content — don’t open it just to clear this report row, only clean up its discovery via sitemaps and links; and use Google’s paywall structured data, not a blanket 401, on subscription content you want indexed.

Parent concept: Page Indexing report · Related: Crawling, Indexing, Page Indexing report

Changelog