Blocked Due to Access Forbidden (403)

TL;DR — “Blocked due to access forbiddenAn HTTP 403 client-error status code meaning the server understood the request but refuses to fulfill it — access is denied, whether or not credentials are involved. For SEO, a 403 served to Googlebot on a page meant to be public and indexable usually points to a misconfigured rule; a persistent 403 keeps a page out of the index either way. (403)” in Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. means GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. tried to read your page and your server said “no, you’re not allowed” (an HTTP 403). The page won’t be indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. If the page is supposed to be public, that’s a bug to fix — often a firewall, CDN, or security plugin blocking Google by mistake, which is why the page can look fine in your browser but get blocked for Google. If the page is supposed to stay private, the 403 may be doing exactly what it should — the fix is different.

What this status means

This report label means Google received an HTTP 403 response while requesting the URL. Evidence for this claim The Page Indexing report identifies URLs where Google encountered a forbidden-access response. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report Google treats persistent 4xx responses other than 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. as unavailable content for indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Evidence for this claim Google treats 4xx responses other than 429 as if the content does not exist for indexing. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: HTTP status codes

When you open the Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. report in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. and see “Blocked due to access forbidden (403),” it’s telling you something simple: GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. asked your server for the page, and the server refused with a 403 ForbiddenAn HTTP 403 client-error status code meaning the server understood the request but refuses to fulfill it — access is denied, whether or not credentials are involved. For SEO, a 403 served to Googlebot on a page meant to be public and indexable usually points to a misconfigured rule; a persistent 403 keeps a page out of the index either way. error.

Because Google got a 403, it can’t read the page’s content, so it won’t index it — and if the page was already in Google, it gets dropped from the results.

Should you actually fix it?

Before you chase a rule, decide what the URL is supposed to do:

• Meant to be public and indexed — the 403 is a bug. Keep reading; the rest of this page is the fix.
• Meant to be public but not indexed — don’t rely on robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. for this; a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. disallow is a separate gate with its own report reason, and it can’t by itself produce an HTTP 403. Serve the normal 200 response and use a noindex directive instead.
• Meant to stay private — the 403 can be a correct, working access control. The fix isn’t the firewall rule; it’s making sure the URL isn’t sitting in your sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. or internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. where Google keeps finding it.

Why it loads for you but not for Google

This is the confusing part for the “meant to be public” case. You click the URL in your browser and it loads perfectly, so how can Google be getting blocked?

Your browser sends signals a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. doesn’t: cookies, a “normal” browser user-agent, and (for JS-challenge or CAPTCHA gates) the ability to pass the challenge. Googlebot’s request typically looks different on most of those — no cookies, a crawler user-agent — which is often enough for a bot rule aimed at scrapers to fire only for the crawler while real visitors sail right through. The page usually isn’t broken — a rule is blocking the wrong visitor. (The Advanced tab shows how to confirm the exact difference instead of guessing.)

A firewall or CDN is a frequently reported cause

For URLs that should be public, a firewall, CDN, or security tool (think Cloudflare, a WAF, or a WordPress security plugin) blocking Googlebot by mistake is one of the most commonly reported causes — though there’s no independently verified figure for exactly how often. Other causes include rules that block by user-agent or country, hotlink protection, or putting content behind a login that should be public.

How to start fixing it

• In Search Console, use URL Inspection on an affected URL and click Test live URL to confirm Google is really getting a 403.
• Check your CDN / firewall / security plugin settings (and their logs) for anything blocking Googlebot or its IP ranges.
• Make sure it’s really Googlebot before you allow it in, and keep the exception narrow — the specific route and rule, not a blanket allow. Plenty of bots fake the Googlebot name. (The Advanced and Scripts tabs show how to verify.)
• Once you fix the rule, the page should return a normal 200. Then use Validate Fix in the report — re-indexing follows automatically once Google re-crawls the 200, though there’s no guaranteed timeline for it.

One sibling status to know: “Blocked due to unauthorized request (401)” is the same idea but for pages behind a login wall. Different mechanism, same outcome and mostly the same fix.

Want the full diagnosis-and-fix loop, the WAF specifics, and the verify-Googlebot commands? Switch to the Advanced tab.

TL;DR — A 403 in the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. means GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. got an HTTP 403An HTTP 403 client-error status code meaning the server understood the request but refuses to fulfill it — access is denied, whether or not credentials are involved. For SEO, a 403 served to Googlebot on a page meant to be public and indexable usually points to a misconfigured rule; a persistent 403 keeps a page out of the index either way. on the URL, so Google won’t indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. it (and removes it if it was indexed). Google’s Page IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. help frames a 403 as credentials supplied but denied — that’s Google’s report wording, not the full HTTP definition; RFC 9110 defines 403 more broadly as “understood but refused,” with or without credentials, so treat Google’s framing as a heuristic, not proof of misconfiguration. Decide first whether the URL should be public and indexable at all — for URLs that should be public, a firewall/CDN/WAF unintentionally blocking GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. is a frequently reported cause, which is why the page often renders for you but 403s for Google, but a 403 on a URL meant to stay private can be working as intended. Google treats 401 and 403 the same for indexing, and explicitly says don’t use 401/403 to limit crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. — use 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. (or 503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime.). Verify the real Googlebot (reverse DNS / published IP ranges / correct client category) before allowlisting, scope any exception to the specific route and rule, then confirm a 200 in URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. → Test live URL and run Validate Fix.

What Google actually says about a 403 — and what HTTP actually says

The label reports the observed response, not the specific WAF, CDN, or access rule that caused it. Evidence for this claim The Page Indexing report identifies URLs where Google encountered a forbidden-access response. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report Its indexing consequence follows Google’s general 4xx handling. Evidence for this claim Google treats 4xx responses other than 429 as if the content does not exist for indexing. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: HTTP status codes

Google’s Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. documentation describes it this way: an HTTP 403 means the user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. provided credentials but was not granted access — but Googlebot never provides credentials, so, in Google’s words, your server is returning the error incorrectly. The page will not be indexed. If you want it indexed, Google’s fix is to either admit non-signed-in users or explicitly allow Googlebot’s requests without authentication.

That’s Google’s Help-doc framing, though, not the complete HTTP definition. RFC 9110 (the current HTTP semantics spec) defines 403 more broadly: the server understood the request and refuses to fulfill it. Credentials can be part of that — but the RFC is explicit that a request can be forbidden for reasons unrelated to credentials entirely: policy, an access-control rule, an edge security decision, whatever the server owner decided. So the useful takeaway isn’t “a 403 to Googlebot always means a broken rule by definition” — it’s “figure out whether this URL is supposed to be reachable, then find the exact rule that returned 403.” For a page that should be public and indexed, a 403 to Googlebot is almost always something to fix. For a page that’s deliberately private or forbidden, the 403 can be a valid, working access decision — the problem there is usually that Google shouldn’t have discovered the URL in the first place (see the intent check below), not that the firewall is wrong.

Whichever case it is, Google’s separate HTTP status-codes documentation is blunt about the indexing consequence: it doesn’t use the content from URLs returning 4xx codes, all 4xx errors except 429 are treated the same (the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. tells the next system the content doesn’t exist), and the indexing pipeline removes the URL from the index if it was previously indexed.

Decide what the URL is supposed to do before you fix anything

Not every row in this report needs the same fix. Sort the URL into one of four buckets first:

1. Public and meant to be indexed. The 403 is unintended — work through the diagnose → fix → validate loop below.
2. Public but not meant to be indexed. Don’t reach for robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. to solve this — a robots.txt disallow is a separate crawl gate with its own Page Indexing reason (“blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.”); it doesn’t itself produce the HTTP 403 this status reports. Serve the normal 200 response and use a noindex directive (or a proper redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't./removal) instead.
3. Deliberately private or forbidden. A 403 can be the correct, intended outcome for a resource the requester shouldn’t reach. If it’s showing up in this report, the actionable problem is usually that the URL is discoverable — in your sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., or elsewhere — not that the access rule is wrong. Clean up the discovery signal rather than opening the door.
4. A URL that shouldn’t exist, or whose existence you don’t want to confirm. Choose the response based on your application’s security policy, not an SEO shortcut — a 403 (or a 404, if you don’t want to confirm the URL exists at all) can both be legitimate here.

Everything from here on assumes case 1: a URL you actually want crawled and indexed.

403 vs 401 — how to tell them apart

These two are siblings in the report and most write-ups blur them with a rule of thumb (“403 = no credentials asked, 401 = login wall”) that isn’t reliable on its own — a misconfigured auth layer can return 403 too, and still land in this row. The response itself is the real signal:

• 401 (unauthorized) — per RFC 9110, a 401 specifically means valid authentication credentials are missing, and the response is required to carry a WWW-Authenticate challenge header (HTTP auth, a login wall). Google was told it needed to authenticate and couldn’t.
• 403 (access forbidden) — a broader refusal. RFC 9110 defines it as “the server understood the request but refuses to fulfill it,” with or without credentials. In practice this row is most often a firewall/CDN/WAF or security-rule block, but it can also be an application returning 403 from a broken auth check, a rate limiter, or a deliberate access-control decision.

To actually tell them apart on a given URL, check the returned status and headers — does the response carry a WWW-Authenticate header? — rather than assuming a mechanism from the report label alone. They share the outcome (not indexed) and often the fix (let verified Googlebot through, or open the content to anonymous users), so once you know which one you’re looking at, the same diagnosis loop below applies to either row.

What a 403 does to indexing

Three things follow once Googlebot gets a 403:

1. The content is ignored. Google doesn’t use content from 4xx URLs, so nothing on the page can be indexed.
2. The URL is removed if it was indexed. A 403 can deindex a page that was previously ranking — this isn’t a “won’t get added” problem only, it’s a “drops out” problem.
3. Crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. decays. Google gradually crawls 4xx URLs less often over time, so a long-standing 403 gets visited less and recovers more slowly once you fix it.

Why Googlebot gets a 403 when your browser doesn’t

This is the question that sends people in circles. The page loads in your browser, so the content is obviously fine — but Google reports a 403. The likely explanation is that your browser and Googlebot make very different-looking requests, and a bot-management rule built to stop scrapers can fire for the crawler while serving humans normally — but treat that as a hypothesis to confirm, not a foregone conclusion, because the attribute that actually matters varies by site:

• Your browser sends cookies, a real browser user-agent, and can solve a JS or CAPTCHA challenge; Googlebot’s request typically carries none of that — no cookies, a crawler user-agent, and it can’t solve an interactive challenge (though Google-InspectionTool does render JavaScript and report the rendered output, so “Googlebot has no JavaScript” isn’t accurate for every part of this).
• Beyond client-side differences, the two requests can also differ in source IP/category, geography, method, referrer, cache state, and which rule actually fired — any one of those can be the real trigger.

Don’t guess which attribute caused it. Compare the exact edge and origin logs for a real blocked request against a normal browser hit — same rule ID, response headers, method, UA, source IP and its verified category, geoGenerative Engine Optimization — visibility inside AI answer engines., cookies, referrer, cache status, and challenge outcome — and change one attribute at a time until you find the one that flips the response. That’s how you confirm the 403 is selective rather than just assume it.

Common causes

• CDN / WAF bot management. Cloudflare (Bot Fight Mode / Super Bot Fight Mode, Browser Integrity Check, Managed Challenge), Akamai, Imperva/Incapsula, Sucuri, AWS WAF — these can challenge or 403 non-browser clients, including Googlebot. For URLs that should be public, this is one of the most frequently reported causes, and almost always unintended — though there’s no independently verified figure for what share of cases it accounts for.
• Server firewall / security rules. mod_security / the OWASP CRS, fail2ban, or host-level firewalls flagging Googlebot’s pattern as abusive.
• User-agent / referrer / hotlink rules. Rules that 403 any request lacking a browser-like user-agent or referrer.
• Geo / IP blocking. Excluding the IP ranges Googlebot crawls from (Googlebot crawls primarily from US IPs), so a country block can silently catch it.
• Login / credential walls. Requiring cookies or a login to view content that should be public — this overlaps with the 401 case.
• Rate-limiting rules that 403 after N requests. Don’t do this — see below.

Don’t use 403 to throttle Googlebot

This is the one anti-pattern worth calling out by name. People (and some CDNs) return 403s or 404s to slow Googlebot down and reduce server load. Google has been explicit that this doesn’t work and shouldn’t be done: don’t use 401 and 403 status codes for limiting the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.. They have no effect on crawl rate — they just deindex the page. If you genuinely need Googlebot to back off, return 429 (or a 5xx like 503); those are the codes Google reads as “slow down,” and the effect is temporary, not permanent.

How to diagnose

1. URL Inspection → Test live URL. Run an affected URL through Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. and use Test live URL to confirm Google is currently getting a 403 (not a stale report). Keep in mind this test is Google-InspectionTool, a specific Google client — a passing live test tells you InspectionTool got through right then, not that scheduled common Googlebot will take the identical WAF, geo, cache, or rate-limit path on its next crawl.
2. Reproduce as Googlebot. Don’t test from your normal browser — fetch the URL with Googlebot’s user-agent, ideally from outside your network and from more than one region (Google mostly crawls from US IPs but can shift to other countries if US requests get blocked, so a single-region pass doesn’t prove global access), to trip the same rule. (Commands are in the Scripts tab.)
3. Read the CDN / WAF / firewall logs. Find the exact rule that fired on Googlebot’s request — compare its response headers, rule ID, and trigger (user-agent, IP, challenge, rate limit) against a normal browser hit’s log entry, rather than assuming which attribute caused it.
4. Verify the real Googlebot — and its category. Before you allowlist anything, confirm the requests are genuinely Googlebot via reverse + forward DNS or Google’s published IP ranges, and check which Google client made the request — common Googlebot, special-case crawlers, and user-triggered fetchers (like Google-InspectionTool) use different host masks and IP lists, so verifying against the wrong list can make a real request look spoofed.

TIP Verify the claimed crawler before changing a firewall rule

The same user-agent can be sent by a real crawler or an impostor. Confirm the request identity before allowlisting it, then trace the rule that returned 403.

Spot-check the source IP against published crawler ranges with my free Googlebot Verifier Free

1. Copy the source IP and claimed user-agent from the blocked WAF or access-log event.
2. Verify the IP against the crawler operator’s published ranges instead of trusting the user-agent string.
3. Allow only verified crawler traffic on the specific route and rule that blocked it, fix that rule, and confirm the URL now returns 200 in a live test.

Tap to enlarge

The Googlebot Verifier result lists IP address 203.0.113.9 with a Googlebot user-agent claim. The published-range result is mismatch, and the row is labeled for a spot-check.

How to fix it

• Scope a narrow exception for verified Googlebot in your WAF / firewall — by verified identity (reverse + forward DNS, matched to the correct client category) or Google’s current published crawler IP ranges, not by trusting the user-agent string alone. Keep the exception narrow: the specific route(s) that need it and the specific rule that’s blocking, rather than a blanket allow-all-Google exception — leave your other security controls, rate limits, and logging in place, and set an expiry/review date so a stale allowlist doesn’t quietly outlive its reason. Ask your CDN/firewall vendor to confirm Googlebot is allowed, and automate a check of your block rules against Google’s current published IP subnets so a future IP-range update doesn’t silently re-block you.
• Open public content to anonymous users. If the 403 comes from a login/cookie requirement on content that should be public, remove that requirement (this is also the shared fix with the 401 sibling).
• Swap 403 rate-limits for 429. If a rule 403s after a request threshold, change it to return 429 (or 503) so Googlebot reads it as “slow down” instead of “go away.”

Validate the fix and get re-indexed

Once the rule is fixed, the URL should return 200. Then:

1. URL Inspection → Test live URL to confirm the live response is now 200.
2. Request indexing for high-priority URLs, and/or click Validate Fix on the “Blocked due to access forbidden (403)” row in the Page Indexing report.
3. Re-indexing follows automatically once Google re-crawls a 200 — there’s no official guarantee of exactly how fast, and the URL still needs to clear a normal crawl before it can reappear. You don’t need to trigger anything manually beyond removing the block, confirming the healthy response, and giving Google time to recrawl.

For the bigger picture of which not-indexed reasons show up here and how the report groups them, see the Page Indexing report overview; for the sibling status, “Blocked due to unauthorized request (401)” is the 401 version of this same problem.

AI summary

A condensed take on the Advanced version:

• What it is. “Blocked due to access forbiddenAn HTTP 403 client-error status code meaning the server understood the request but refuses to fulfill it — access is denied, whether or not credentials are involved. For SEO, a 403 served to Googlebot on a page meant to be public and indexable usually points to a misconfigured rule; a persistent 403 keeps a page out of the index either way. (403)” in GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. report = GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. crawled the URL and the server returned HTTP 403. Google won’t indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. it, and removes it if it was previously indexed.
• Google’s framing vs. HTTP’s. Google’s Help doc describes 403 as credentials provided but denied — that’s Google’s report wording, not the full HTTP definition. RFC 9110 defines 403 more broadly: a refusal the server can issue with or without credentials, for any reason.
• Decide the URL’s intent first. Public and meant to be indexed → fix the 403. Public but not meant to be indexed → use noindex, not robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. (which is a separate gate and can’t itself produce a 403). Deliberately private → the 403 may be correct; clean up how Google is discovering the URL instead.
• A frequently reported cause (for public URLs). A firewall / CDN / WAF (Cloudflare, Akamai, Imperva/Incapsula, Sucuri, AWS WAF) or a security plugin blocking GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., almost always unintentionally — though there’s no verified figure for how often this specific cause applies.
• Why it can load for you but not Google. Your browser typically carries cookies, a real user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., and challenge-solving that Googlebot’s request doesn’t — but confirm the actual trigger by comparing logs rather than assuming.
• 401 vs 403401 Unauthorized is a client-error HTTP status code meaning the request lacks valid authentication credentials — the server wants you to log in before it serves the page. For SEO it means the content is gated: Googlebot never sends credentials, so a 401'd page can't be seen and won't be indexed.. Check the response: 401 requires a WWW-Authenticate challenge; 403 is a broader refusal that can happen with or without credentials. Google treats both the same for indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and the fixes often overlap.
• Don’t throttle with it. Don’t use 401/403 to limit crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. — they just deindex. Use 429 (or 503) for “slow down.”
• Diagnose. URL Inspection → Test live URL (confirms Google-InspectionTool access at that moment, not scheduled Googlebot’s exact path); reproduce as Googlebot from more than one region; compare WAF/firewall logs line by line; verify the real Googlebot and its client category before allowlisting.
• Fix + validate. Scope a narrow, verified-Googlebot exception (or open the content); swap 403 rate-limits for 429; confirm a 200 in Test live URL; run Validate Fix. Re-indexing follows automatically once Google re-crawls a 200, without a guaranteed timeline.

Official documentation

Primary-source documentation from the search engines.

Google

• Page Indexing report — the report itself, including the “Blocked due to access forbiddenAn HTTP 403 client-error status code meaning the server understood the request but refuses to fulfill it — access is denied, whether or not credentials are involved. For SEO, a 403 served to Googlebot on a page meant to be public and indexable usually points to a misconfigured rule; a persistent 403 keeps a page out of the index either way. (403)” and “Blocked due to unauthorized request (401)” entries and what each means.
• How HTTP status codes, and network and DNS errors affect Google Search — how Google handles 4xx (content ignored, URL removed from the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.) and the rule against using 401/403 to limit crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor..
• Verifying Googlebot and other Google crawlers — reverse + forward DNS and the published crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. IP ranges, so you allowlist the real GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. and not a spoofer.
• Overview of Google crawlers and fetchers — every Google user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. and the IP-range JSON files to match against.
• Don’t 404 my yum (Search Central Blog, 2023) — Google asking site owners and CDNs to stop using 403/404 to throttle GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., and what to use instead.

Bing / Microsoft

• Bing Webmaster Tools — Crawl Control — BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share., like Googlebot, can’t index a page it gets a 403 on; verify firewall/CDN rules aren’t blocking BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. (hostnames resolve to *.search.msn.com) and use Crawl Control to manage speed rather than blocking.

HTTP standard

• RFC 9110 — HTTP Semantics, §15.5.2 (401) and §15.5.4 (403) — the protocol definitions this page uses to correct Google’s narrower, credential-based Help-doc wording: 401 requires a WWW-Authenticate challenge, while 403 is a broader refusal that can happen with or without credentials.

Quotes from the source

On-the-record statements from Google. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what a 403 means for indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. (Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.)

• “HTTP 403An HTTP 403 client-error status code meaning the server understood the request but refuses to fulfill it — access is denied, whether or not credentials are involved. For SEO, a 403 served to Googlebot on a page meant to be public and indexable usually points to a misconfigured rule; a persistent 403 keeps a page out of the index either way. means that the user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. provided credentials, but was not granted access. However, GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. never provides credentials, so your server is returning this error incorrectly. The page will not be indexed.” — Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. Help, “Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason..” Jump to quote

Google — how 4xx (including 403) is handled

• “Google doesn’t use the content from URLs that return 4xx status codes.” — Google Search Central docs. Jump to quote
• “All 4xx errors, except 429, are treated the same: Google crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. inform the next processing system that the content doesn’t exist.” Jump to quote

Google — don’t use 401/403 to limit crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.

• “Don’t use 401 and 403 status codes for limiting the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor..” — Google Search Central docs. Jump to quote

403 diagnose → fix → validate checklist

Run this against any URL showing “Blocked due to access forbiddenAn HTTP 403 client-error status code meaning the server understood the request but refuses to fulfill it — access is denied, whether or not credentials are involved. For SEO, a 403 served to Googlebot on a page meant to be public and indexable usually points to a misconfigured rule; a persistent 403 keeps a page out of the index either way. (403)”:

• Decide the URL’s intent — public/indexable, public/non-indexable, deliberately private, or shouldn’t be discoverable at all. Only the public-and-indexable case needs the rest of this checklist.
• Confirm it’s live, not stale — URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. → Test live URL shows a current 403 (this confirms Google-InspectionTool access, not necessarily scheduled GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.’s exact path).
• Reproduce as GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. — fetch the URL with Googlebot’s user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. (and from outside your network, ideally more than one region), not your normal browser.
• Find the rule — compare the CDN / WAF / firewall / plugin logs for the blocked request against a normal browser hit (rule ID + trigger: user-agent, IP, challenge, or rate limit) rather than assuming the cause.
• Verify the real Googlebot and its client category — reverse + forward DNS (host ends in googlebot.com / google.com / googleusercontent.com) or match the IP-range list for the correct category, before allowlisting anything.
• Scope a narrow exception for verified Googlebot — the specific route(s) and rule, by identity/IP range, with logging and an expiry/review date — not a blanket allow and never by trusting the user-agent string alone.
• Open public content — if the 403 is a login/cookie requirement on content that should be public, remove it.
• Swap any 403 rate-limit for 429 (or 503) — never use 401/403 to slow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..
• Confirm a 200 — URL Inspection → Test live URL now returns 200.
• Validate — click Validate Fix on the 403 row and/or Request indexing for priority URLs; re-indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. then happens automatically on re-crawl.
• Future-proof — automate a check of your block rules against Google’s published IP subnets so an IP-range update doesn’t silently re-block you.

The mental models

1. Ask “should this be public?” before “who’s misconfigured?” Google’s Help wording frames 403 as credentials sent and refused, but RFC 9110 defines it more broadly — a request the server understood and refused, for any reason, credentials included or not. For a page that’s supposed to be public and indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., a 403 to GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. is almost always a rule firing on the wrong visitor — treat it as “find the rule,” not “judge the page.” For a page that’s deliberately private, the 403 might be doing its job; the fix there is removing the URL from Google’s discovery path, not opening the firewall.

2. Same rule, a different-looking visitor — confirm which difference matters. The page can load for you and 403 for Google because your browser carries cookies, a real user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., and challenge-solving that GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.’s request doesn’t. But don’t assume which attribute triggered the rule — compare logs and change one attribute at a time. Debug by making Googlebot’s request, not yours, and let the logs tell you why, not a guess.

3. Verify identity, then scope the exception narrowly. Anyone can spoof the Googlebot user-agent. The order is always: confirm identity and client category (reverse DNS / the matching IP-range list) → then allow — and even then, scope the exception to the specific route and rule, with logging and an expiry/review date, rather than a blanket Google allowlist. Allowlisting on user-agent alone, or too broadly, opens a hole for scrapers pretending to be Google.

4. 403 is the wrong throttle. If your goal is to slow Googlebot, 401/403 don’t do it — they deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist.. The “slow down” codes are 429 and 5xx/503. Pick the code that matches your intent: “go away forever” (deindex) vs. “come back later” (throttle).

5. 403 and 401 are close cousins, told apart by the response. 401 requires a WWW-Authenticate challenge — a real authorization wall. 403 is a broader refusal that can happen with or without credentials, so don’t infer the mechanism from the label alone; check the actual response. Once you know which one you’re looking at, they share the outcome (not indexed) and often the fix (admit verified Googlebot / open the content) — diagnose them with the same loop.

403 cheat sheet

The sibling statuses in Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.

Status codes and crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. — pick the right one

What 403 does to a page meant to be indexed

• Content ignored (Google doesn’t use 4xx content).
• URL removed from the index if it was previously indexed.
• Crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. decays while the 403 persists.

Verify-Googlebot quick reference

• Reverse-DNS the IP → host must end in googlebot.com, google.com, or googleusercontent.com.
• Forward-DNS that host → must resolve back to the same IP.
• Or match the IP against Google’s current published crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. IP-range JSON files for the correct client category — common Googlebot, special-case crawlers, and user-triggered fetchers (like Google-InspectionTool) publish separate lists.

Reproduce the 403 as Googlebot

Don’t test from your normal browser — it won’t trip the botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. rule. Fetch the URL with GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.’s user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. so you see what Google sees.

macOS / Linux

# Fetch headers only, as Googlebot's user-agent — look at the status line
curl -s -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  -I https://www.example.com/page/
# A "HTTP/1.1 403 Forbidden" here reproduces what Googlebot is getting.

Windows (PowerShell)

# -SkipHttpErrorCheck so PowerShell shows the 403 instead of throwing
$ua = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Invoke-WebRequest -Uri "https://www.example.com/page/" -Method Head `
  -UserAgent $ua -SkipHttpErrorCheck | Select-Object StatusCode, StatusDescription

If that returns 403 but a normal browser request returns 200, you’ve confirmed a bot-specific block. Note: a sophisticated WAF may also key off IP, so for a full reproduction test from outside your own network.

Verify a bot is really Googlebot (before you allowlist)

Plenty of traffic fakes the GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. user-agent. Confirm identity with a reverse

• forward DNS check before you allow an IP through your firewall.

macOS / Linux

# 1) Reverse DNS the IP from your logs — must end in googlebot.com / google.com
host 66.249.66.1
# → ... domain name pointer crawl-66-249-66-1.googlebot.com

# 2) Forward DNS that hostname back — it must resolve to the same IP
host crawl-66-249-66-1.googlebot.com
# → crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Windows

nslookup 66.249.66.1
nslookup crawl-66-249-66-1.googlebot.com

If the reverse lookup doesn’t end in a Google domain, or the forward lookup doesn’t match the original IP, it isn’t Googlebot — don’t allowlist it. You can also match the IP against Google’s published ranges (googlebot.json). Note that googlebot.json covers common Googlebot only — special-case crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. and user-triggered fetchers (like Google-InspectionTool) publish separate IP-range files, so match against the list for the actual client you’re trying to verify. Confirm identity first, then scope the exception to the specific route and rule; never allowlist by the user-agent string alone or open a blanket exception.

Diagnosing a 403: WAF block, login wall, or rate-limit misuse?

The first fork checks the response itself — a 403 with no WWW-Authenticate header or login prompt points away from the 401 sibling and toward a security-rule or access-control refusal. From there the branches follow the diagnose-then-fix loop already in the Advanced tab; treat each result as a hypothesis to confirm in your logs, not a certainty. Click through it.

Track the 403 bucket, not just its existence

The one number worth watching is how many URLs sit in the “Blocked due to access forbiddenAn HTTP 403 client-error status code meaning the server understood the request but refuses to fulfill it — access is denied, whether or not credentials are involved. For SEO, a 403 served to Googlebot on a page meant to be public and indexable usually points to a misconfigured rule; a persistent 403 keeps a page out of the index either way. (403)” row of the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. over time, not just whether the row exists — a single snapshot can’t tell you whether a fix actually worked or whether new 403s are accumulating.

403-bucket count over time

• Metric — The count of URLs under “Blocked due to access forbidden (403)” in GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s Page IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. report, tracked week over week.
• What it tells you — Whether an allowlist or firewall-rule fix actually worked and whether new 403s are creeping in. After you allowlist verified GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. or open a public path, this count should fall toward zero for the URLs you fixed. If the count stays flat, that’s fine only when you’ve deliberately decided those URLs should never be crawled; a rising count usually means a WAF/CDN rule or IP-range update is newly catching GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer..
• How to pull it — GSC Page IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. report, filtered to the 403 row; spot-check individual URLs with URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. → Test live URL to confirm they’re not a stale snapshot.
• Benchmark / realistic range — There’s no universal target — it depends on how many URLs you actually want indexed. The honest bar: zero 403s among URLs you want indexed, and a flat (not growing) count elsewhere. Establish your own baseline count before judging trend direction.
• Cadence — Weekly right after a fix, until the count stabilizes; monthly afterward as a regression check, since a CDN or Googlebot IP-range update can silently reintroduce the block.

Runbook: diagnose, fix, validate

The 403 workflow is a short, ordered loop — don’t jump to allowlisting before you’ve reproduced the block as GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. and confirmed the identity of what you’re about to let through.

1. Confirm it’s live, not stale. Run URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. → Test live URL in GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. on the affected URL to confirm Google is currently getting a 403, not a cached report.

2. Reproduce it as GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., not as yourself. Fetch the URL with Googlebot’s user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., ideally from outside your own network and from more than one region — your normal browser sends cookies, a real user-agent, and can pass JS/CAPTCHA challenges that Googlebot’s request typically can’t, so it may not trip the same rule. Compare the resulting log entries side by side rather than assuming which difference caused the block.

3. Find the rule. Read the CDN/WAF/firewall logs for the exact rule that fired on Googlebot’s request — the rule ID and trigger (user-agent, IP, challenge, or rate limit) tell you what to change.

4. Verify the real Googlebot — and its category — before allowlisting. Confirm the requests you’re about to allow are genuinely Googlebot via reverse

• forward DNS or the IP-range list matching that client’s category (common Googlebot, special-case crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., and user-triggered fetchers publish separate lists) — otherwise you punch a hole for spoofers or verify against the wrong list.

5. Fix by cause. WAF/CDN rule → scope a narrow, verified-Googlebot exception to the specific route and rule, by identity/IP range. Login/cookie requirement on public content → remove it. Rate-limit rule returning 403 → swap it for 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. (or 503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime.).

6. Validate. Confirm URL Inspection → Test live URL now returns 200, then click Validate Fix on the 403 row and/or Request indexing for priority URLs. Re-indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. happens automatically once Google re-crawls a 200 — if the 403-bucket count isn’t trending down after a couple of weeks, go back to step 2 and re-reproduce rather than guessing at a new cause.

Ready-to-use AI prompts

Copy-paste prompts for classifying a 403 from raw diagnostic output. These help you triage the likely cause fast — always confirm the conclusion against the actual server/firewall config before changing anything.

Classify the likely cause from curl output

I'm diagnosing a "Blocked due to access forbidden (403)" status in Google
Search Console. Below is the raw output of a request made with Googlebot's
user-agent (curl -A "...Googlebot..." -I). Based on this output alone,
classify the most likely cause as one of: (1) CDN/WAF bot-management
challenging or blocking non-browser clients, (2) server firewall / mod_security
rule flagging the crawl pattern as abuse, (3) user-agent or referrer/hotlink
rule, (4) geo/IP blocking excluding Googlebot's IP ranges, (5) a login/cookie
requirement on content that should be public, (6) a rate-limit rule misusing
403 instead of 429. Explain which specific header or detail in the output
pointed you to that answer, and tell me what's missing if you can't tell for
sure.

CURL OUTPUT:
[paste]

Classify the cause from a WAF/firewall log line

I'm investigating why Googlebot is getting a 403 on a page I want indexed.
Below is a log line (or a few) from my WAF/CDN showing a blocked request. Tell
me whether this looks like a bot-identity rule (blocking by user-agent or IP
range), a challenge/CAPTCHA rule, or a rate-limit rule misusing 403, and what
I'd need to allowlist — by IP range or reverse-DNS — to let verified Googlebot
through without disabling the rule entirely.

LOG LINE(S):
[paste]

Sanity-check a fix before shipping it

I'm about to allowlist a set of IPs as "Googlebot" in my WAF, because a page
I want indexed is currently returning 403 to the crawler. Here's the evidence
I have that these requests are genuinely Googlebot: [describe reverse-DNS or
IP-range check]. Point out anything I might be missing — e.g. whether I should
verify by both reverse and forward DNS, whether I should match Google's
published IP-range JSON instead, or whether allowlisting by user-agent alone
would leave a hole for spoofers.

Test yourself: “Blocked due to access forbidden (403)”

Five questions on what a 403 means, how it differs from a 401, and how to diagnose and fix it. Pick an answer for each, then check.

Tools for diagnosing and fixing a 403

• HTTP Status Checker — paste the affected URL (or a batch) to confirm the 403 status code, see the full response chain, and catch anything that redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. before it hits the block.
• Googlebot Verifier — check whether an IP claiming to be GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. is genuine (published IP ranges plus reverse-DNS confirmation) before you allowlist it past a WAF or firewall rule.
• Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. — URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. → Test live URL — the closest you’ll get to seeing the exact response GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. receives right now.
• curl -A "...Googlebot..." — the fastest way to fetch a URL with Googlebot’s user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. and read the raw status line.
• Your WAF/CDN’s firewall event log (Cloudflare, Akamai, Sucuri, etc.) — find the specific rule returning 403 to Googlebot’s IP ranges.

Prove the fix actually worked

Once you’ve allowlisted verified GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., opened public content, or swapped a rate-limit rule to 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content., these are the checks that separate “the config changed” from “Google can now actually reach the page.” Run them in order.

Test 1 — A Googlebot-user-agent request now returns 200

• Test to run — Fetch the affected URL with GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.’s user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. (or check it with the HTTP Status Checker).
• Expected result — The status line reads HTTP/1.1 200 OK.
• Failure interpretation — Still 403 means the firewall/WAF rule wasn’t actually updated, or you allowlisted the wrong IP range. A 401 instead of 403 means you’ve traded a block for an auth wall — check that path separately.
• Monitoring window — Immediate — the server answers as soon as the rule change is live.
• Rollback trigger — If opening the path exposes content you meant to keep restricted, put the block back and instead allowlist verified Googlebot by IP/reverse-DNS on a narrower rule.

Test 2 — Google confirms it can now reach the page

• Test to run — Run URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. → Test live URL in Google Search Console on the affected URL.
• Expected result — The live test succeeds and shows the page content — no access-forbidden error reported.
• Failure interpretation — If Test live URL still reports a 403 after the Googlebot-user-agent fetch passes, suspect a rule that’s scoped specifically to Google’s published IP ranges rather than the user-agent string — check the firewall log for a second rule.
• Monitoring window — Immediate to a few minutes after the fix.
• Rollback trigger — N/A — this is a read-only test; if it still fails, go back to Test 1’s failure branch rather than rolling anything back.

Test 3 — The 403 status clears in the Page Indexing report

• Test to run — Use Validate Fix on the “Blocked due to access forbidden (403)” issue in the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason., and watch the bucket count over the following weeks (see the How to Measure tab).
• Expected result — The URL moves out of the 403 bucket, and — if it was previously indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — reappears in the index over time as Google re-crawls the now-200 page.
• Failure interpretation — Re-indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. is automatic but not instant; don’t read a slow validation as a new failure. If the count in the 403 bucket isn’t trending down after a couple of weeks, re-run Test 1 to confirm the fix is still in place — a CDN cache or a Googlebot IP-range update can silently reintroduce the block.
• Monitoring window — Days to a few weeks, tracked via the 403-bucket count.
• Rollback trigger — Only revisit the rule itself if Test 1 starts failing again — don’t chase the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.’s timing.

Resources worth your time

Official

• Page Indexing report (Google) — the 403 and 401 entries, verbatim.
• How HTTP status codes affect Google Search (Google) — how 4xx is handled and the no-401/403-for-rate-limiting rule.
• Verifying Googlebot (Google) — reverse DNS + published IP ranges.
• Don’t 404 my yum (Google, 2023) — don’t use 403/404 to throttle GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer..

My related writing

• The Beginner’s Guide to Technical SEO — where crawl-access problems like this fit in the bigger picture.
• Robots.txt and SEO: Everything You Need to Know — the other common way pages get unintentionally blocked from crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..

From others

• r/TechSEO — the community for crawl/indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. debugging, including GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.-blocked-by-CDN threads.
• Google warns against using 403 or 404 status codes for Googlebot crawl-rate limiting (Search Engine Land) — coverage of Gary Illyes’ “Don’t 404 my yum” guidance and what to use instead (429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content./503503 Service Unavailable is the HTTP status code a server returns when it's temporarily unable to handle a request — usually because of overload or planned maintenance. It tells crawlers 'come back later' instead of 'this page is gone,' which is why Google recommends it (paired with a Retry-After header) for short, planned downtime.).
• Google: Don’t use 403/400 error responses for rate-limiting Googlebot (Search Engine Journal) — SEJ’s relay of the same Illyes post, with extra context.
• Most common reason for blocking Googlebot is firewalls/CDNs (Search Engine Roundtable) — Google Search Relations explaining that the vast majority of Googlebot blocks are unintentional CDN/firewall rules.
• Bots that impersonate Googlebot (johnmu.com) — John Mueller on why you must verify Googlebot via reverse DNS before allowlisting, not just trust the user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string.
• How to fix “Blocked due to access forbidden (403)” (Onely) — practitioner walkthrough covering WAF/CDN specifics and the diagnose → fix loop.

Blocked due to access forbidden (403)

A Google Search Console Page Indexing status meaning Googlebot tried to crawl the URL but the server returned an HTTP 403 (Forbidden). Google won't index the page. For a URL meant to be public, this is often a firewall/CDN/WAF unintentionally blocking Googlebot; for a URL meant to stay private, the 403 can be a valid, working access decision.

Parent concept: Indexing · Related: Page Indexing report, Crawling, Indexing

Blocked due to access forbidden (403)

“Blocked due to access forbiddenAn HTTP 403 client-error status code meaning the server understood the request but refuses to fulfill it — access is denied, whether or not credentials are involved. For SEO, a 403 served to Googlebot on a page meant to be public and indexable usually points to a misconfigured rule; a persistent 403 keeps a page out of the index either way. (403)” is a status in Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.. It means GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. requested the URL and your server answered with an HTTP 403 (Forbidden). Because the page returns 403, Google doesn’t use its content, the page won’t be indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and if it was indexed before, it gets removed.

Google’s Help doc frames this as a mismatch: a 403 is supposed to mean the client sent credentials but wasn’t allowed in, and GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. never sends credentials. That’s Google’s report wording, though, not the full HTTP definition — RFC 9110 defines 403 more broadly as a refusal the server can issue with or without credentials, for any reason. So whether a 403 is a bug depends on the URL: for a page you want indexed, it almost always means a firewall, CDN, or WAF is unintentionally blocking the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., and it’s an error to fix. For a page that’s deliberately private or forbidden, the 403 can be doing exactly what it’s supposed to — the fix there is cleaning up how Google discovered the URL, not opening the firewall.

It’s the close sibling of “Blocked due to unauthorized request (401).” 401 requires a WWW-Authenticate challenge — a real authorization/login wall; 403 is a broader refusal that can happen with or without credentials, so check the actual response rather than assuming the mechanism. They differ in mechanism but often share the outcome (not indexed) and the fix (let verified Googlebot through, or open the content to anonymous users). Don’t reach for 401 or 403 to slow Googlebot down — that’s what 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content. is for.

Parent concept: Indexing · Related: Page Indexing report, Crawling, Indexing

Changelog