Blocked by robots.txt (GSC Status)

TL;DR — “Blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.” in Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. means Google foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. the page but didn’t read it, because your robots.txt file tells crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. to stay out. Usually that’s on purpose and it’s fine — it’s the normal result of a block, not an error. It’s only a problem if you blocked a page you actually wanted in Google.

What this status means

Open the Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. report in Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results., click into “Blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.,” and you’ll see a list of URLs Google chose not to indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. The reason is simple: Google found those URLs, but your site’s robots.txt file says crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. aren’t allowed to fetch them. Evidence for this claim Google reports Blocked by robots.txt when crawling is disallowed and warns that this does not guarantee the URL cannot be indexed by other means. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report The label reports the crawl block; it does not guarantee that the URL can never be indexed through other signals.

The first thing I tell people is: don’t panic. This status is usually intentional. If you (or your CMSA content management system (CMS) is software that lets users create, manage, and publish digital content — like blog posts and pages — without writing raw code. WordPress, Drupal, and Joomla are the most common open-source CMS platforms., or a plugin) blocked something on purpose — internal search results, faceted-navigation URLs, a staging path, cart and checkout URLs — then seeing it here is exactly what’s supposed to happen. It’s Google confirming the block worked, not flagging a mistake.

The one idea to hold onto

robots.txt controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Those are two different things. Blocking a page in robots.txt stops Google from reading it; it does not delete Evidence for this claim Google says robots.txt manages crawler access and is not a mechanism for keeping a page out of Google. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: robots.txt introduction it from Google, and it isn’t a way to remove a page from search. I’ve tested the “what happens if you block a page” side of this myself — more on that in the Advanced tab — but the headline is: a block stops the crawl, full stop.

Is it a problem?

Run one check: did you mean to block these URLs?

• Yes, on purpose → leave it alone. This is working as intended.
• No, that page should be in Google → that’s the one case to fix. Find the rule in your robots.txt that’s catching the URL and remove or loosen it, then request indexing.

There’s a related-but-different status you might also see: “Indexed, though blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing..” That one’s a warning, not an exclusion — it means Google indexed a blocked URL anyway (usually because other pages link to it). Same cause, opposite outcome. If that’s what you’re looking at, it’s covered separately.

One more thing robots.txt can’t do: keep something private. It’s a request, not an access control — well-behaved bots honor it, but nothing forces them to, and nothing stops the URL from surfacing elsewhere (a link, a screenshot, a scraper that ignores robots.txt entirely). If a page holds sensitive content, protect it with a login or password, not a disallow.

The mistake to avoid

A lot of people try to remove a page from Google by blocking it in robots.txt, sometimes adding a noindex tag at the same time for good measure. That doesn’t work — and it backfires. If Google can’t crawl the page, it can never see the noindex on it, so the page can stay in Google. To actually take a page out of search, you do the opposite: let Google crawl it, and add noindex.

Want the full version — how to find the exact blocking rule, why the noindex + disallow combo conflicts, and what my own experiment showed — switch to the Advanced tab.

TL;DR — “Blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.” is a Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. exclusion: Google discovered the URL but didn’t crawl it because robots.txt disallows it, so it isn’t indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. in this state. It’s normally intentional and benign — robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. governs crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., so a disallow is never a deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. tool. Don’t confuse it with the warning “Indexed, though blocked by robots.txtA Google Search Console Page Indexing warning: Google indexed the URL anyway despite your robots.txt disallowing crawling it — Google names other pages linking to it as the likely path. robots.txt blocks crawling, not indexing.” (Google indexed a blocked URL anyway, usually via inbound links). The classic mistake is pairing disallow with noindex: Google can’t crawl the page, so it never sees the noindex and the page can stay indexed. To remove a page, allow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and serve noindex. Only treat this status as a bug when you blocked a URL you actually wanted indexed.

What the status actually reports

In the Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. Page Indexing report, “Blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” is an excluded state, not an error and not a warning. Google’s own wording is plain: the page “was blocked by your site’s robots.txt file,” and Google adds the important caveat that this “does not guarantee that the page won’t be indexed through some other means.” Evidence for this claim Google reports Blocked by robots.txt when crawling is disallowed and warns that this does not guarantee the URL cannot be indexed by other means. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report That caveat is the whole topic in one line — see below.

Mechanically: GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. knows the URL exists (it foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. it via a link, a sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., or history), tried to respect your rules, hit a matching Disallow, and stopped. No fetch means no content to index, so the URL sits in this excluded bucket. This is the expected result of a disallow — most URLs in here are supposed to be in here.

Crawling is not indexing — why a disallow doesn’t deindex

Google documents robots.txt as crawl-access control, not a reliable removal mechanism. Evidence for this claim Google says robots.txt manages crawler access and is not a mechanism for keeping a page out of Google. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: robots.txt introduction

This is the accuracy spine of the whole status. robots.txt is a crawl control. Google is explicit that it “is not a mechanism for keeping a web page out of Google.” Blocking a URL prevents the fetch; it does not remove the URL from the index, and it is not a deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. tool.

As I’ve put it in my Ahrefs article on the sibling status: “crawling and indexing are two different things.” A page you block can still end up indexed if other pages link to it — Google just can’t see the content, and it can’t see any noindex you tried to put there either. Which leads to the single most common mistake people make when they land on this report.

”Blocked by robots.txt” vs “Indexed, though blocked by robots.txt”

These two get conflated constantly, and they’re the same cause with opposite outcomes:

If you’re staring at the warning version — a blocked URL that got indexed anyway, showing as a bare URL with no description — that’s the indexed-though-blocked case, and it’s handled in its own write-up. This article is about the plain exclusion: blocked, and not indexed.

For utility URLs, “indexed anyway” is often nothing to worry about. John Mueller, addressing a site owner whose WooCommerce ?add-to-cart= URLs showed up as indexed-though-blocked, said you don’t need those URLs indexed, that blocking them with robots.txt is fine, and that even if they get “indexed” while blocked, they’re unlikely to actually surface in search unless someone runs a very specific query for those URLs — which real users don’t.

The noindex + disallow conflict (the #1 fix that backfires)

Here’s the trap. Someone wants a page gone, so they Disallow it in robots.txt and add a noindex meta tag — belt and suspenders. It doesn’t work, because the two instructions contradict each other.

Google states the rule directly: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index..” If the page is disallowed, Google never crawls it, never sees the noindex, and the page can stay indexed. As I’ve written in my article on “Indexed, though blocked by robots.txt”: unless Google can crawl a page, it won’t see the noindex tag and may still index it because it has links.

So the deindex sequence is the opposite of what people reach for:

1. Allow crawling of the URL (remove the Disallow).
2. Serve noindex (meta robots tag or X-Robots-Tag header) and let Google re-crawl to see it.
3. Leave the page crawlable with noindex in place once it’s dropped. Re-blocking it afterward isn’t a safe “finishing move” — if you disallow the URL again, Google can lose visibility into the noindex rule on the next crawl, and a blocked-but-linked URL can get indexed again from other pages’ links, which is the exact “indexed, though blocked” outcome you were trying to avoid. If crawl budget on the removed page is a real concern, that’s a case for authentication or deletion (404/410) — not a return trip through robots.txt.

For urgent removals, the Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. Removals toolA Google Search Console feature for verified owners to manage how their URLs appear in Search. A Temporary Removal hides a URL for about six months — it does not delete the page from the index., password protection, or simply deleting the page (returning 404/410) are the faster routes.

How to find which rule is blocking the URL

Three tools, each doing a different job — don’t expect one to do the others’ work:

• URL Inspection (GSC). Paste the specific URL. It tells you whether that URL is currently blocked and is the quickest per-URL check.
• robots.txt report (GSC). A domain-property-level monitoring report — not an editable tester — showing the robots.txt files Google found for your top hosts, the last-fetched time, fetch status, and any parsing warnings, plus a “request a recrawl” action after you change the file. It tells you Google can see your file; it doesn’t test individual URLs against it.
• A robots.txt validator or Google’s open-source robots.txt parser. To see which line matches a given URL, run it through a validator — because the longest, most specific matching rule wins (and an Allow can override a broader Disallow).

Once you’ve found the offending line, the fix depends on where your robots.txt lives. If you control the file directly, remove or correct the rule (and mind the syntax). If you’re on a hosted platform like Wix, Shopify, or Squarespace, you may need to follow that provider’s specific docs to change it, since some platforms manage robots.txt for you.

TIP Find the rule that is actually blocking the URL

To test the current policy and a proposed edit against the same URL set, use the robots.txt Tester Free

1. Paste the current robots.txt file and test the affected URL or a representative batch.
2. Inspect the winning Allow or Disallow rule for each URL before changing the file.
3. Use diff mode to test the proposed policy, publish it, then confirm the live result in Search Console.

Tap to enlarge

The completed Robots.txt Tester result checks two URLs for Googlebot. The first URL is allowed. The second URL, /search?q=redirects, is blocked by line 2. The file-health check reports no issues, so the problem is the policy decision rather than malformed syntax.

My own experiment: what happens when you block a page you wanted indexed

The genuinely useful question is: if you accidentally block a page that should rank, how bad is it? I ran that test directly. On January 30, 2023 I blocked two of our actually-ranking pages — “Top Bing Searches” and “Top YouTube Searches” — with robots.txt and tracked what happened.

The damage was real but smaller than I expected. We lost a position here or there (a few keywords slipped one or two spots, a couple even gained), and we lost all the featured snippets for those pages while they were blocked — they came back after I unblocked. The SERP appearance degraded too: Google showed “no information is available for this page” instead of the meta descriptionThe meta description is an HTML head tag — `<meta name=\"description\" content=\"…\">` — that suggests a short summary of the page for the search snippet. It's not a Google ranking factor, and Google rewrites it the majority of the time, but a good one can still lift click-through., and lost our custom titles. And because the listing looked worse, clicks dropped more than impressions did — the CTR took the hit.

My summary at the time: “We lost a position here or there and all of the featured snippets for the pages. I expected a lot more impact, but the world didn’t end.” And the takeaway I’d still stand behind: don’t block pages you want indexed. It hurts. Not as bad as you might think — but it still hurts. That’s the right mental frame for this report. If everything in your “Blocked by robots.txt” bucket is stuff you meant to block, you’re fine. If a page you care about is in there, get it out.

One scope note on this experiment: both pages were already ranking and indexed before I blocked them, so what I measured is what happens when an indexed page gets pushed into the “indexed, though blocked” state — not what happens to a URL that was never indexed and is simply sitting in this exclusion bucket. If a URL here was never indexed to begin with, unblocking it just lets Google crawl and consider it normally; there’s no featured-snippet or CTR history to lose, because it was never there.

The decision tree

• Did you mean to block it? → Leave it. Working as intended.
• No — you want it indexed? → Remove/loosen the robots.txt rule, then request indexing.
• You want it gone from Google? → Don’t use a disallow. Allow crawling + noindex (or Removals tool / delete), then leave it crawlable — re-blocking it afterward can hide the noindex rule again.
• It’s a blocked URL that got indexed anyway? → That’s the indexed-though-blocked case, not this one — handle it there.
• The page has sensitive or private content? → robots.txt isn’t an access control; it’s a request bots can ignore. Use authentication or password protection, not a disallow.

The crawl-vs-index principle is universal, by the way: Bing honors robots.txt for crawling the same way, and removing a URL from Bing similarly uses its Block URLs tool or a noindex on a crawlable page — not a bare disallow.

For the broader file itself — syntax, wildcards, where it lives, and what it can and can’t do — see the robots.txt guide. For how indexing decisions get made upstream, see the indexing hub.

AI summary

A condensed take on the Advanced version:

• “Blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.” = a Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. exclusion. Google foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. the URL but didn’t crawl it because robots.txt disallows it, so it isn’t indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. in this state. It’s the normal result of a block — usually intentional and benign.
• robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Google says it “is not a mechanism for keeping a web page out of Google.” A disallow stops the fetch; it does not deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. a page.
• Don’t confuse it with the warning “Indexed, though blocked by robots.txtA Google Search Console Page Indexing warning: Google indexed the URL anyway despite your robots.txt disallowing crawling it — Google names other pages linking to it as the likely path. robots.txt blocks crawling, not indexing..” Same cause, opposite outcome: that one means a blocked URL got indexed anyway, usually via inbound links. For utility URLs, Mueller says that’s typically fine — blocked-and-indexed URLs rarely surface for normal queries.
• The #1 backfire: noindex + disallow together. Google can’t crawl a disallowed page, so it never sees the noindex and the page can stay indexed. Google’s rule: for noindex to work, the page must not be blocked by robots.txt.
• To actually remove a page: allow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., serve noindex, let Google re-crawl, then leave it crawlable. Re-blocking it afterward can hide the noindex rule again and let the URL get indexed from links — the opposite of what you wanted. For urgent cases use the Removals toolA Google Search Console feature for verified owners to manage how their URLs appear in Search. A Temporary Removal hides a URL for about six months — it does not delete the page from the index., password protection, or delete the page.
• Find the blocking rule with URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. (per-URL) and a robots.txt validator (which line matches; longest/most-specific rule wins). The GSC robots.txt report monitors file fetch status — it doesn’t test individual URLs. Fix depends on where the file lives — hosted platforms may need provider-specific steps.
• Patrick’s experiment: blocking two already-ranking, already-indexed pages cost a position here or there and all featured snippets, degraded the SERP listing, and dropped clicks more than impressions — “the world didn’t end,” but “don’t block pages you want indexed.” This shows what happens to an indexed page pushed into “indexed, though blocked” — it doesn’t forecast outcomes for a URL that was never indexed.
• robots.txt isn’t a security control. It’s a request bots can choose to ignore. Protect sensitive content with authentication, not a disallow.
• Decision rule: meant to block it → leave it; want it indexed → unblock + request indexing; want it gone → allow-crawl + noindex (and stay crawlable).

Official documentation

Primary-source documentation from the search engines.

Google

• Page indexing report — the report this status lives in, with the verbatim definitions of “Blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.” and “IndexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., though blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing..”
• Unblock a page blocked by robots.txt — Google’s step-by-step for finding and fixing the rule that blocks a URL.
• robots.txt report — the domain-property report showing the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. files Google foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore., fetch status, warnings, and the recrawlCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. request.
• Introduction to robots.txt — what robots.txt does and doesn’t do, including that it “is not a mechanism for keeping a web page out of Google.”
• Block Search indexing with noindex — the rule that noindex only works on a page that isn’t blocked by robots.txt (the noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. + disallow conflict).

Bing / Microsoft

• Bing Webmaster Tools Help — Bing honors robots.txt for crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. too; its robots.txt Tester and Block URLs tool follow the same crawl-vs-index principle.

Quotes from the source

On-the-record statements from Google. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what the status means

• “This page was blocked by your site’s robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file. You can verify this using the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. tester. Note that this does not guarantee that the page won’t be indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. through some other means.” — Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. Help, Page indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.. Jump to quote

Google — robots.txt controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.

• “it is not a mechanism for keeping a web page out of Google.” — Google Search Central, Introduction to robots.txt. Jump to quote

Google — why noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. + disallow conflict

• “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index..” — Google Search Central, Block Search indexing with noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed.. Jump to quote

Stats & first-party data worth citing

• Blocking ranking pages: real but survivable damage. In my experiment blocking two already-ranking, already-indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. pages with robots.txt (started Jan 30, 2023), the pages lost a position here or there and all of their featured snippets, the SERP listing degraded to “no information is available for this page,” and clicks dropped more than impressions. My summary: “I expected a lot more impact, but the world didn’t end” — but “don’t block pages you want indexed. It hurts.” This tests what happens to an indexed page pushed into the “indexed, though blocked” state; it isn’t a forecast for a URL that was never indexed in the first place. Source
• Blocked utility URLs that get indexed are usually harmless. Per John Mueller, blocking add-to-cart-style URLs with robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. is fine; even if they show as indexed-though-blocked, they’re unlikely to surface for normal queries. Coverage
• How common robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. blocking is across the web — block-rate data on which crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. ~140M sites disallow the most, from my study with Xibeijia Guan. Source

The mistake that defines this status: noindex + disallow

Mistake: pairing Disallow with noindex to try to remove a page. This is the single most common mistake tied to “Blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.,” and it’s worth naming directly because it feels like belt-and-suspenders and is actually self-defeating. Someone wants a page gone from Google, so they Disallow it in robots.txt and add a noindex meta tag for good measure. It doesn’t work. Google states the rule plainly: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file, and it has to be otherwise accessible to the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index..” If the page is disallowed, Google never crawls it, never sees the noindex, and the page can stay indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — the exact opposite of the intended outcome. The fix is the opposite instinct: allow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., then serve noindex.

Myth: “A disallow in robots.txt is a deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. tool.” It isn’t, and Google says so directly: robots.txt “is not a mechanism for keeping a web page out of Google.” Blocking a URL stops the fetch — it does not remove the URL from the index, and a blocked-but-linked page can still show up (that’s the sibling “Indexed, though blocked by robots.txtA Google Search Console Page Indexing warning: Google indexed the URL anyway despite your robots.txt disallowing crawling it — Google names other pages linking to it as the likely path. robots.txt blocks crawling, not indexing.” status). Use noindex (with crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. allowed) or the Removals toolA Google Search Console feature for verified owners to manage how their URLs appear in Search. A Temporary Removal hides a URL for about six months — it does not delete the page from the index. when the goal is actually removal.

Myth: “Everything in the ‘Blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.’ bucket is a problem to fix.” It’s usually the opposite. This status is the expected outcome of an intentional disallow — internal search results, faceted-nav parameters, staging paths. The only version of this status worth treating as a bug is a URL you actually wanted indexed sitting in the list by accident.

Mistake: blocking a page you want to rank, without checking the cost first. Blocking a page you want ranking to “clean things up” or “save crawl budgetThe number of URLs an engine will crawl in a timeframe.” is riskier than it looks. In my own experiment blocking two ranking pages, I lost all featured snippets while they were blocked and clicks dropped more than impressions because the SERP listing degraded to “no information is available for this page.” The rankings themselves mostly held — but treat a robots.txt block on a page you care about as a real decision, not a housekeeping tweak.

”Blocked by robots.txt” triage checklist

Work through this list top to bottom when you open the status in Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.:

• Open the Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. report and click into “Blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.” to see the actual list of affected URLs.
• For each URL (or pattern of URLs), ask: did I mean to block this?
• If yes — internal search, faceted nav, staging, cart/checkout — leave it alone. This status is working as intended for that URL.
• If no, or you’re not sure — run the URL through URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. to confirm it’s currently blocked (not a stale snapshot).
• Find the specific rule catching the URL with the robots.txt Tester (or another robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. validator) — the longest, most specific matching rule wins, and an Allow can override a broader Disallow. GSC’s robots.txt report monitors file fetch status; it doesn’t test individual URLs.
• Confirm you’re not also relying on noindex on the same disallowed URL — that combination can’t work; pick allow-crawl-plus-noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. instead if the goal is removal.
• Remove or loosen the blocking rule (directly if you control robots.txt, or via your hosting platform’s docs if it’s managed for you, e.g. Wix, Shopify, Squarespace).
• Request indexing for the fixed URL in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. once the block is cleared.
• Double-check you’re not confusing this status with “IndexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., though blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” — that’s the opposite outcome (indexed anyway) and needs a different fix.
• Re-check the bucket in a few weeks to confirm the count trended the way you expected (down for fixed URLs, flat for intentional blocks).
• If any of these URLs hold sensitive content, don’t rely on the disallow to keep it private — add authentication or password protection instead.

Work the decision tree

Answer one question at a time to land on the right move for a URL sitting in the “Blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.” bucket.

The mental models

1. CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. ≠ indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. The whole status collapses into confusion if you conflate the two. robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. — whether GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. fetches a page. It has no direct control over indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — whether the URL ends up in Google’s index. A disallowed page isn’t indexed via crawling, but it can still be indexed without contentA Google Search Console status meaning the URL is in Google's index, but Googlebot couldn't read any usable content from it — possible causes include a server/CDN block, cloaking, an empty render, or an unsupported format. Not the same as a page-level robots.txt block. if other pages link to it. Keep the two ideas separate and most of this status stops being confusing.

2. The one-question triage. Before doing anything else, ask: did I mean to block this? Yes → leave it, it’s working as intended. No → that’s the one case worth fixing. This single question sorts almost every URL in the bucket correctly.

3. To remove a page, invert your instinct. The natural move — disallow plus noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed., “extra safe” — is exactly backwards. Google can’t see a directive on a page it never fetches. The correct sequence is the opposite of the intuitive one: allow crawling, then serve noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed.. Anytime someone reaches for a disallow to make a page disappear, that’s the signal to apply this model.

4. robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. is a crawl-budget tool, not a search-results tool. Use it to keep botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. off low-value or infinite URL spacesA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content. (internal search, facets, staging) so crawl effort goes toward pages you want found — not as a lever for what shows up in search results. That job belongs to noindex, password protection, or deletion.

Track the bucket count, not just its existence

The one number worth watching is how many URLs sit in the “Blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.” row of the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. over time — a single snapshot can’t tell you whether a fix worked or whether new unintended blocks are creeping in.

”Blocked by robots.txt” bucket count over time

• Metric — The count of URLs under “Blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” in GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s Page IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. report, tracked over time.
• What it tells you — Whether the bucket is behaving the way you expect. For URLs you intentionally block (internal search, facets, staging), the count should stay roughly flat as those URL spaces stay stable — a sudden jump usually means a new section got swept into an existing Disallow pattern by accident. For a URL you deliberately unblocked because you wanted it indexed, that specific URL should drop out of the bucket after the next crawl.
• How to pull it — GSC Page IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. report, filtered to the “Blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.” row; spot-check specific URLs with URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. to confirm current status rather than a stale snapshot.
• Benchmark / realistic range — There’s no universal target — it depends entirely on how much of your site is intentionally disallowed. The honest bar: zero URLs in this bucket that you actually wanted indexed, and a flat trend for everything else. Baseline your own count before judging whether a change moved it in the right direction.
• Cadence — Check right after any robots.txt change; otherwise a periodic glance (monthly is plenty for most sites) is enough to catch drift.

The playbook: actually removing a page from Google

A repeatable sequence for when a URL needs to come out of Google’s indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. for real — not just get blocked from future crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..

1. Confirm the goal is removal, not just crawl control. If the goal is really “stop wasting crawl budgetThe number of URLs an engine will crawl in a timeframe. on this,” a plain robots.txt disallow is the right tool and this playbook doesn’t apply — that page can stay indexed via links and that’s fine. Only use this sequence when the page must actually leave the index.

2. Allow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. Remove or loosen the Disallow rule blocking the URL. Confirm with the robots.txt Tester that the URL now comes back allowed, not blocked.

3. Serve noindex. Add a noindex meta robotsThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen. tag or X-Robots-Tag HTTP header to the now-crawlable page. This is the actual removal instruction — the disallow was never doing that job.

4. Let Google re-crawl and pick it up. Google has to fetch the page again to see the noindex. There’s no way to force an instant fetch beyond requesting indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. in Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. and waiting for the next crawl.

5. Confirm removal. Check URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. for the page’s current indexed status. Once it reports the page as excluded due to noindex, the removal has worked.

6. Don’t treat re-blocking as the finishing step. Once the page has dropped, adding the Disallow back isn’t free: Google can lose visibility into the noindex rule again on the next crawl, and a blocked-but-linked URL can come back as indexed via other pages’ links — the exact “indexed, though blocked” outcome you were trying to avoid. If crawl budgetThe number of URLs an engine will crawl in a timeframe. is the concern, weigh that against the risk, or reach for authentication/deletion instead for pages that truly need to stay gone for good.

7. For urgent cases, skip the wait. If the page needs to be out now, use the Search Console Removals tool, password-protect the page, or delete it outright (return 404/410) instead of waiting on the crawl-then-noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. cycle.

Ready-to-use AI prompts

Copy-paste prompts for working through a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. block with an LLMA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4).. Treat the output as a starting hypothesis — confirm anything load-bearing with the robots.txt Tester or Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. tool before acting on it.

Figure out which rule blocks which URL

Here is my robots.txt file and a list of URLs. For each URL, tell me whether it
is blocked or allowed, and quote the exact line in robots.txt responsible
(remember: the longest, most specific matching rule wins, and an Allow can
override a broader Disallow). If a URL isn't matched by any rule, say so.

ROBOTS.TXT:
[paste]

URLS:
[paste list, one per line]

Sanity-check whether a block looks intentional

I run a [type of site]. Here is my robots.txt file. For each Disallow rule, tell
me what kind of URLs it likely targets (e.g. internal search, faceted
navigation, staging, cart/checkout, admin) and flag any rule that looks broad
enough it might be catching content pages by accident. Don't guess about pages
you can't see — just reason from the rule pattern and ask me to confirm
anything ambiguous.

ROBOTS.TXT:
[paste]

Draft the noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed.-then-crawl sequence for a specific page

I need to remove [URL] from Google. It's currently blocked by this robots.txt
rule: [paste rule]. Walk me through the exact sequence of changes in order
(robots.txt edit, meta tag or header change, what to check in Search Console at
each step) so I don't accidentally leave the noindex tag unseen by Google.

Test a URL against your robots.txt rules

Before you change anything, confirm exactly which rule is catching a given URL. The robots.txt Tester does this with a matcher ported from Google’s own open-source robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. parser and shows the winning rule per URL — that’s the fastest and most accurate way to check. For a quick manual sanity check without a tool, here’s the core logic in Python:

import re

def rule_matches(path, rule):
    """Very simplified robots.txt path matcher: '*' = wildcard, '$' = end anchor."""
    pattern = re.escape(rule).replace(r'\*', '.*')
    if pattern.endswith(r'\$'):
        pattern = pattern[:-2] + '$'
    return re.match(pattern, path) is not None

def find_blocking_rule(path, disallow_rules, allow_rules):
    """Longest matching rule wins; ties go to Allow (mirrors Google's documented
    precedence). Returns the winning rule string, or None if nothing matches."""
    matches = [r for r in disallow_rules if rule_matches(path, r)]
    matches += [r for r in allow_rules if rule_matches(path, r)]
    if not matches:
        return None
    return max(matches, key=len)

# Example
disallow = ["/search", "/*?*sort="]
allow = ["/search/help"]
print(find_blocking_rule("/search/help", disallow, allow))  # -> "/search/help" (Allow wins, longer)
print(find_blocking_rule("/search/results", disallow, allow))  # -> "/search" (Disallow, no competing Allow)

This is a simplification of Google’s actual matcher (real robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. parsing has more edge cases around escaping and group selection) — use it to sanity-check your mental model, and confirm anything that matters with the real tester before you rely on it.

Tools for diagnosing and fixing this status

• Google Index Checker — once you’ve fixed a block, use this to check the page’s observable indexability signals (status code, redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., noindex, canonical) before routing to Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. for Google’s actual answer.
• URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. (Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.) — the source of truth for whether a specific URL is currently blocked, its live-tested indexability, and (once fixed) whether Google has picked it back up.
• robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. report (Google Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance.) — a domain-property-level view of the robots.txt files Google foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. for your top hosts, last-fetched time, and fetch status, with a “request a recrawlCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial.” action after you change the file.

Test yourself: Blocked by robots.txt

Five questions on the “Blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.” status and the mistakes people make around it. Pick an answer for each, then check.

Prove the removal actually worked

“I set noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed.” or “I edited robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.” isn’t proof of anything on its own — the only result that counts is what Google’s indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. actually reflects afterward. These tests separate the change you made from the outcome Google reached.

Test 1 — Crawling is genuinely allowed

• Test to run — Run the URL through the robots.txt Tester after editing the file.
• Expected result — The URL comes back allowed, with no Disallow rule matching it (or a more specific Allow winning over a broader Disallow).
• Failure interpretation — Still blocked means a different, more specific rule is catching the URL, or the edit didn’t deploy — check the file’s actual live contents, not just what you intended to change.
• Monitoring window — Immediate — this is a static check of the file as served.
• Rollback trigger — N/A — fix the rule before moving to the next test; nothing downstream works until crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is actually allowed.

Test 2 — noindex is being served correctly

• Test to run — URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. → Live Test in Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. (or fetch the page directly and check the response headers/HTML) on the now-crawlable URL.
• Expected result — The live test shows noindex present, either as a meta robots tag in the <head> or an X-Robots-Tag HTTP header.
• Failure interpretation — Missing noindex here means the removal will never happen no matter how long you wait — the crawl-allowed step alone doesn’t remove anything.
• Monitoring window — Immediate — live test reflects the current served page.
• Rollback trigger — N/A — this must pass before Google has any chance to act on it.

Test 3 — Google has actually dropped the page

• Test to run — URL Inspection (the indexed-status field, not just Live Test) on the URL, checked periodically after the previous two tests pass.
• Expected result — The status changes to excluded due to noindex, and the URL stops appearing for a site: search or in your rank-tracking data.
• Failure interpretation — Still indexed after a full crawl cycle usually means cachingCaching stores a copy of a page or resource — in a browser, a CDN edge node, or a search crawler's own cache — so it can be served again without regenerating or re-downloading it. It isn't a direct ranking factor, but it feeds page speed and crawl efficiency./propagation delay, not a broken setup — Google has to re-crawl to notice the change, and that isn’t instant.
• Monitoring window — 1–3 weeks, depending on how often Google was already crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. the URL — don’t judge this the day after shipping the change.
• Rollback trigger — the named abort condition — Still indexed after a full crawl cycle and the page keeps showing as freshly crawled in server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. → stop assuming it’s just propagation and re-check Tests 1 and 2 for a mistake (a caching layer serving a stale robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., a CDN stripping the X-Robots-Tag header, or a second conflicting rule are the usual culprits).

Blocked by robots.txt

A Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.

Related: Indexed, though blocked by robots.txt, Robots.txt, Crawling

Blocked by robots.txt

“Blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.” is an exclusion status in Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.. It means GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. the URL but didn’t crawl it, because your site’s robots.txt file disallows it — so the page isn’t being indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. in this state. Most of the time this is exactly what you intended, and it’s nothing to panic over.

The key thing to keep straight: robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. A disallow stops the fetch; it doesn’t guarantee a page stays out of Google. That’s why this status has a sibling — the “Indexed, though blocked by robots.txtA Google Search Console Page Indexing warning: Google indexed the URL anyway despite your robots.txt disallowing crawling it — Google names other pages linking to it as the likely path. robots.txt blocks crawling, not indexing.” warning, which is the other outcome: Google indexed a blocked URL anyway, usually because something links to it.

Because Google can’t crawl a disallowed URL, it also can’t see a noindex you put on that page — so you can’t reliably remove a page from Search with a robots.txt disallow. To actually deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist., allow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and serve noindex instead. It’s only worth “fixing” this status when you blocked a page you actually wanted indexed.

Related: Indexed, though blocked by robots.txt, Robots.txt, Crawling

Changelog