Indexed, Though Blocked by robots.txt

TL;DR — This Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. warning means Google indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. a page even though your robots.txt blocks it. That sounds like a contradiction, but robots.txt only stops Google from reading a page — it doesn’t keep the URL out of search. If you actually want the page gone, you have to unblock it and add a noindex tag. If it’s a junk URL, you can usually just leave it.

What the warning means

When you block a URL in robots.txt, you’re telling Google “don’t crawl this.” Google obeys that. But “don’t crawl” is not the same as “don’t index.” If other pages link to that blocked URL, Google can still add it to its index — it just can’t open the page to see what’s on it.

So you end up with a URL in Google’s results that Google never actually read. Google itself says any snippet for a page like this will probably be very limited — sometimes no proper title, and a note that no information is available for the page. That’s the warning: indexed, though blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.. Evidence for this claim Google reports this warning when a URL is indexed even though robots.txt blocks crawling. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report

The one thing to understand

Blocking a page in robots.txt does not remove it from Google. Evidence for this claim Google documents that robots.txt controls crawling and does not reliably prevent indexing from other signals. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: robots.txt introduction People assume a Disallow deletes a page from search. It doesn’t — and worse, it can lock the page in, because Google can’t crawl in to see the noindex tag that would actually remove it.

How to fix it (it depends on what you want)

• You want the page in Google. Unblock it in robots.txt so Google can crawl and index it properly.
• You want the page out of Google. Unblock it and add a noindex tag (or password-protect it). Then Google can crawl in, see the noindex, and drop it.
• It’s a junk URL (an add-to-cart link, a filtered/parameter URL, internal search results). It’s often fine to leave, but not automatically — check whether it’s actually surfacing for real searches and whether it’s sensitive before you decide. The Advanced tab has the fuller triage.

The trap to avoid: don’t block a page in robots.txt and add noindex. Google can’t read the noindex on a blocked page, so the page stays stuck.

Want the full decision tree — including the case where the page should point at another URL instead of being removed — switch to the Advanced tab.

TL;DR — “IndexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., though blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” is a warning: Google indexed the URL despite a robots.txt disallow — Google names other pages linking to it as the likely path, without publishing how often that’s the actual cause. It couldn’t fetch the content, so Google says the resulting snippet will probably be very limited. The accuracy spine: robots.txt controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — a Disallow can’t deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. a page and can trap it, because Google never crawls in to see a noindex. Fix by intent: want it indexed → unblock; want it gone → unblock + noindex; should it consolidate → unblock + canonical, no noindex; links are the cause → remove the offending links. Never pair Disallow with noindex. For cart/parameter/faceted junk, triage first — it’s often fine to ignore, but not automatically. Distinct from the excluded status “Blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.” (blocked and not indexed).

What this status actually means

This is a warning, not an error. Google puts it plainly: the page was indexed despite being blocked by your robots.txt, and Google always respects robots.txt — but that doesn’t necessarily prevent indexing if someone else links to your page. In other words, Google added the URL to its index, obeyed your crawl block, and never fetched the content. Evidence for this claim Google reports this warning when a URL is indexed even though robots.txt blocks crawling. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report The result is a listing Google itself says will probably be very limited — sometimes with no real title or description — that can still surface for queries that specifically target that URL.

The reason this confuses people is that it looks like Google ignored your robots.txt. It didn’t. It obeyed the crawl directive perfectly. Google names external signals such as links pointing at the URL as the likely path it used to index it anyway, though Google doesn’t publish how often that’s actually the cause. Evidence for this claim Google documents that robots.txt controls crawling and does not reliably prevent indexing from other signals. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: robots.txt introduction

Why a blocked page gets indexed: crawling ≠ indexing

This is the whole game, and it’s the point I keep coming back to in my robots.txt guide. As I put it there: “If you block a page from being crawled, Google may still index it because crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and indexing are two different things. Unless Google can crawl a page, they won’t see the noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. meta tag and may still index it because it has links.”

robots.txt governs crawling — which URLs a botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. may request. The index is a separate system. When enough links point at a disallowed URL, Google can index that URL based on those external signals without ever downloading the page. The definition I use is exactly that: Google has indexed URLs that you blocked them from crawling using the robots.txt file on your site.

So the counterintuitive trap: a Disallow can lock a URL into the index. The tool that would remove it — noindex — only works if Google can crawl the page to see it. Block the crawl and you’ve blocked the cure.

”Indexed, though blocked” vs “Blocked by robots.txt” — two different statuses

These look almost identical and mean opposite things:

• “Blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” is an excluded status. The URL is blocked and not indexed. Usually intentional and benign — it’s robots.txt doing its job.
• “Indexed, though blocked by robots.txtA Google Search Console Page Indexing warning: Google indexed the URL anyway despite your robots.txt disallowing crawling it — Google names other pages linking to it as the likely path. robots.txt blocks crawling, not indexing.” is a warning. The URL is blocked but indexed anyway. Google got in through the back door of inbound links.

If you only remember one thing: the excluded version is “kept out,” the warning version is “snuck in.”

Is it actually a problem? Triage first

Before you touch anything, decide whether the flagged URL even matters. A lot of the time, this warning is cosmetic — but “it’s a junk URL” isn’t a blanket pass by itself. Work through these before deciding to leave it:

1. What template generated it? Add-to-cart links, faceted-navigation and parameter URLs, and internal search results are the classic low-value triggers — but confirm the flagged URLs actually match one of those patterns rather than assuming from the volume alone.
2. Is it actually showing up for real queries? Check Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. performance data for the affected pattern. If nothing’s getting impressions, it’s a much safer thing to leave.
3. Does it carry anything sensitive? A blocked-but-indexed URL that exposes pricing logic, internal search terms, or anything else you wouldn’t want public is worth fixing even if it never gets clicks.
4. What’s actually generating the links? Since inbound links are what gets these URLs indexed in the first place, a large or fast-growing count of flagged URLs can point at a template or internal-linking bug worth fixing at the source, not just triaging away one warning at a time.
5. Is a directive-level fix even practical? Applying noindex selectively to one URL pattern inside a shared template isn’t always feasible without development work — that’s a real cost to weigh against how much the URL actually matters.
6. How much does this URL matter to the business? Weigh the fix effort against the actual downside of leaving it.

Google’s public position on one narrow case — John Mueller, responding to a WooCommerce site with a large batch of add-to-cart URLs flagged this way — was that you don’t need them indexed, blocking them with robots.txt is fine, and even when they get “indexed” they’re unlikely to actually show in search unless someone runs a very specific query for that exact URL. Treat that as relayed, scoped evidence for that specific case (it was reported secondhand, and the outlet that covered it flagged that the “just leave it” framing doesn’t generalize cleanly to every template or business) — not a rule that every low-value URL pattern is automatically safe to ignore. Reserve “leave it” for URLs that clear the checklist above; reserve the deindex workflow for pages that genuinely matter and genuinely appear.

The fix: an intent-based decision tree

The right fix depends entirely on what you want the URL to do. Walk these in order:

1. You want it indexed. The block was a mistake. Remove the Disallow from robots.txt so Google can crawl and index the page properly. Evidence for this claim If you want the URL indexed, Google's stated next step is to update robots.txt to unblock the page. Scope: Google Search Console Page Indexing report, recommended action for this warning. Confidence: high · Verified: 2026-07-18 Google: Page indexing report (Use the robots.txt tester / report to find which rule is catching it.)

2. You want it out of the index. Here’s the recipe — and the order matters. Allow crawling, then add a noindex (meta robots tag in the <head>, or an X-Robots-Tag: noindex HTTP header for non-HTML files). Evidence for this claim If you want an accessible page excluded from Google Search, Google's stated path is to remove the robots.txt block and use noindex (or password-protect / remove the content). Scope: Google Search Console Page Indexing report plus the robots.txt introduction's alternatives for keeping content out of Search. Confidence: high · Verified: 2026-07-18 Google: Page indexing report Google: robots.txt introduction My own short version of this: “Add a noindex meta robots tag and make sure to allow crawling — assuming it’s canonical.” Google then crawls in, sees the noindex, and drops the page. You can also password-protect it, or return a 404/410 if it should truly be gone.

3. It should consolidate to another URL. This is the case almost nobody covers, and it’s where a reflexive noindex does damage. If the URL canonicalizes to another page, don’t noindex it. As I’ve written in my canonicalization deep dive: “If the URL canonicalizes to another page, don’t add a noindex meta robots tag. Just make sure proper canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. signals are in place, including a canonical tagA rel=\"canonical\" annotation — in the HTML <head> or an HTTP Link header — that tells search engines which URL is the preferred version of duplicate or near-duplicate content. on the canonical page, and allow crawling so signals pass and consolidate correctly.” A noindex here would throw away the consolidation you actually want.

4. Links are the cause. Since inbound links are why the URL got indexed, if those are internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. you control, removing or fixing them cuts off the signal feeding the index.

TIP Find the rule blocking this exact URL

Test the URL against the crawler-specific policy before changing robots.txt; the winning rule is the useful evidence.

Paste the current file and affected URL to see the selected crawler's actual allow-or-block decision with my free robots.txt Tester Free

1. Select the crawler and test the exact canonical URL, not only the directory pattern.
2. Remove the winning Disallow only when the page should be crawled or needs Google to see a noindex directive.
3. Rerun the same decision after publishing the file.

Tap to enlarge

The trap to avoid: never pair Disallow with noindex

This is the single most common self-inflicted version of this problem. People see “indexed despite robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.,” panic, and add a noindex on top of the existing Disallow — to be “extra safe.” It does the opposite. Because the page is disallowed, Google can’t crawl it, so it never sees the noindex, so the page stays indexed. noindex and Disallow on the same URL cancel each other out. Pick one based on intent. To remove a page, the block has to come off.

I tested the blocking side myself

I blocked two of our high-ranking Ahrefs pages with robots.txt as an experiment — deliberately, to see what would happen. They stayed indexed. They lost their featured snippets and slipped a position or two, but they didn’t vanish. That’s the whole lesson in one test: blocking the crawl didn’t remove the pages from Google; it just degraded the listings (Google could no longer read them properly) while they kept ranking. Blocking a page you want indexed hurts — not as catastrophically as you’d expect, but it still hurts, and it’s never the way to remove something.

Special cases: parameter, faceted, cart, internal-search URLs

Run these through the triage checklist above rather than assuming “leave it” by default. When they do clear the checklist (not surfacing, nothing sensitive, no runaway link-source problem), the right move is usually not a noindex race — it’s fixing the architecture so the junk URLs don’t get linked and discovered in the first place. If they’re already indexed and you genuinely want them gone, allow the crawl and noindex them; just weigh the development cost of a template-level directive change against whether it’s worth the effort for URLs that don’t actually surface.

How to validate the fix in GSC

Once you’ve changed the directive:

1. Confirm the new state on the URL — for a removal, check that robots.txt now allows the URL and the page returns the noindex (URL Inspection → live test shows the rendered page and tags).
2. Request a recrawl of the URL, and recrawl your robots.txt from the GSC settings if you changed it.
3. Use “Validate Fix” on the warning in the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason..
4. Be patient. Recrawl and reprocessing take days to weeks — the status won’t flip the moment you save the change.

One distinction worth keeping straight: a robots.txt change plus noindex is how you deindex going forward. The URL Removal tool is only a temporary hide (roughly six months) — it doesn’t remove the page from the index, so it’s a stopgap, not the fix.

The sibling pages here — the excluded “Blocked by robots.txt” status, noindex itself, and robots.txt as a whole — go deeper on each piece.

AI summary

A condensed take on the Advanced version:

• It’s a warning, not an error. Google indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. the URL despite your robots.txt block — Google names other pages linking to it as the likely path, without publishing how often that’s the actual cause. Google obeyed the crawl block; it just indexed the URL from the outside without reading it, so Google says the resulting snippet will probably be very limited (sometimes no title).
• CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. ≠ indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. robots.txt controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexing. A Disallow can’t deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. a page and can trap it — Google can’t crawl in to see the noindex that would remove it.
• Not the same as “Blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing..” That sibling is excluded (blocked AND not indexed). This one is indexed anyway.
• Triage before fixing. For cart/parameter/faceted/internal-search junk, run it through the checklist first: which template, does it actually surface, is anything sensitive exposed, what’s generating the links, is a fix even practical. It’s frequently fine to ignore once it clears that — but not automatically. Reserve the fix for valuable pages that actually appear.
• Fix by intent: want it indexed → unblock; want it gone → unblock + noindex (or password-protect / 404·410); should it consolidate → unblock + canonical, no noindex; links are the cause → remove the offending internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them..
• Never pair Disallow with noindex — Google can’t see the noindex on a blocked page, so it stays indexed.
• My experiment: blocking two high-ranking pages with robots.txt kept them indexed but cost them featured snippets and a position or two — proof that blocking degrades the listing rather than removing the page.
• Validate in GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.: confirm the new directive, request recrawlCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. (+ recrawl robots.txt), hit “Validate Fix,” and wait days to weeks. The Removal tool is only a temporary hide.

Official documentation

Primary-source documentation from the search engines.

Google

• Page Indexing report — the status definitions, including “IndexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., though blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” and the sibling “Blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.,” plus the recommended actions and the Validate Fix flow.
• Block search indexing with noindex — the correct way to remove a page, and the critical caveat that noindex can’t be seen on a robots.txt-blocked page.
• Introduction to robots.txt — what robots.txt is (and isn’t) for, including the warning not to use it to hide pages from Search.
• How to remove information from Google — noindex, password protection, removal, and why the Removal tool is temporary.

Bing / Microsoft

• Bing Webmaster Tools — Block URLs — Bing’s fast, temporary removalA Google Search Console feature for verified owners to manage how their URLs appear in Search. A Temporary Removal hides a URL for about six months — it does not delete the page from the index. tool; like Google, Bing recommends noindex (not robots.txt) to keep a URL out of the index, and can likewise list a well-linked URL it hasn’t crawled.

Quotes from the source

On-the-record statements from Google. Each link is a deep link that jumps to the quoted passage on the source page.

Google — the status definition

• “The page was indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. despite being blocked by your website’s robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file. Google always respects robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., but this doesn’t necessarily prevent indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. if someone else links to your page.” — Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. Help, Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.. Jump to quote

Google — robots.txt is not for hiding pages

• “Warning: Don’t use a robots.txt file as a means to hide your web pages (including PDFs and other text-based formats supported by Google) from Google Search results.” — Google Search Central docs, Introduction to robots.txt. Jump to quote

Google — why noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. needs crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. (the trap, stated)

• “Important: For the noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. If the page is blocked by a robots.txt file or the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.” — Google Search Central docs, Block search indexing with noindex. Jump to quote

What should you actually do about this URL?

Start from what you want the URL to do — not from the warning itself.

One follow-up question applies regardless of which branch you land on: are internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. still pointing at this URL? If you control them, removing or fixing those links cuts off the signal that got the URL indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. in the first place.

Playbook: clear an indexed-but-blocked incident

1. Sample the affected URLs. Separate pages that should be indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., removed, consolidated, or left blocked. Do not apply one fix to every row in the report.
2. Open the crawl path. For URLs that need noindex or a canonical read by the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., remove the relevant robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. block first.
3. Apply the intent-specific control. Keep wanted pages crawlable and indexable; add noindex to removal candidates; use a crawlable canonical or redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. for duplicates. Leave genuinely low-value crawl trapsA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content. blocked when indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. is harmless.
4. Remove conflicting signals. Update internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. and sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. so they no longer promote URLs meant to disappear or consolidate.
5. Validate a small live sample. Confirm robots access, the rendered directive or canonical, and the live response before starting Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. validation.
6. Monitor the affected pattern. Exit when the warning clears for the sampled template and new URLs are no longer entering the same state.

Mistakes that make this worse

• Pairing Disallow with noindex on the same URL. Adding noindex “to be extra safe” on a page that’s still blocked in robots.txt does nothing — Google can’t crawl in to see the tag, so the page stays indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Instead: unblock the page first, then add noindex. The two directives only work in sequence, never together.
• Assuming a Disallow deletes a page from Google. robots.txt controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Treating a block as a removal mechanism is exactly how pages get “indexed, though blocked” in the first place. Instead: use noindex (with crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. allowed) or the Removal tool for an actual removal.
• noindex-ing a URL that should canonicalize elsewhere. If the real fix is consolidation, a reflexive noindex throws away the signal you wanted to pass to the canonical page. Instead: allow crawling, fix the canonical tagA rel=\"canonical\" annotation — in the HTML <head> or an HTTP Link header — that tells search engines which URL is the preferred version of duplicate or near-duplicate content., and leave noindex off entirely.
• Using robots.txt to “hide” a page from search results. Google’s own docs warn against this directly — a block only stops crawling, and inbound links can still get the URL indexed anyway, often with a thinner, less controlled listing than if you’d just left it crawlable and used noindex.
• Treating the URL Removal tool as a permanent fix. It hides a URL for roughly six months but doesn’t touch the index or the underlying robots.txt / noindex state. Instead: use it as a stopgap while the real fix (unblock + noindex, or unblock + canonical) propagates.
• Expecting the warning to clear the moment you save the change. Recrawling and reprocessing take days to weeks. Checking back an hour later and assuming the fix “didn’t work” leads people to make a second, conflicting change on top of the first.

Common issues

The warning covers URLs you never meant to block

Cause: a Disallow rule in robots.txt is broader than intended — a wildcard or a directory-level rule catching URLs you didn’t think about. Fix: open the rule in a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. tester against the specific URL to see which line matches, then narrow the pattern. The robots-txt-tester tool on this site will show you exactly which directive is catching a given path.

You added noindex, but the page still shows as indexed

Cause: the page is still disallowed in robots.txt, so Google’s crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. never reaches the page to see the noindex tag. Fix: remove the block first. In URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version., run a live test — if the “crawl allowed” indicator is No, that’s the whole problem; the noindex is irrelevant until crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is allowed.

”Validate Fix” keeps failing or the status won’t change

Cause: recrawlCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. hasn’t happened yet, or robots.txt itself is cached and Google hasn’t re-fetched it since your edit. Fix: request a recrawl of both the URL and robots.txt from Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results., then wait — reprocessing typically takes days to weeks, not hours.

The page is unblocked and noindex-ed, but it’s still appearing in search

Cause: either the crawl genuinely hasn’t happened yet, or the noindex was placed somewhere Google can’t see it (added by JavaScript without server-side renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., or missing from the HTTP header on a non-HTML file). Fix: confirm the rendered page (not just the source) actually carries the noindex via URL Inspection’s live test; for PDFs and other non-HTML files, confirm the X-Robots-Tag: noindex header with curl -I.

The warning reappears after you thought you’d fixed it

Cause: internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. pointing at the URL are still live, or a new external link surfaced, feeding the same indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. signal that caused the problem originally. Fix: audit inbound links to the URL (site search or a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. like Screaming Frog) and remove or redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. the ones you control.

If no current robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. rule explains it, or it keeps coming back, the cause isn’t always a stable Disallow line. This is practitioner diagnosis, not something Google documents directly — work through it as an escalation checklist, not a first resort:

• Historical or intermittent robots.txt responses. A server error, deploy glitch, or CDN hiccup can have served a blocking robots.txt (or a 5xx, which Google can treat as a block) at some point even if the live file looks fine now.
• Crawler-specific rules. Confirm the block isn’t scoped to a specific user-agent — test the exact URL against GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. specifically, not just the generic ruleset.
• CDN, firewall, or WAF layers. A rule blocking Googlebot’s IP range or user agent at the network layer won’t show up in robots.txt at all.
• Hosting-provider controls. Some hosts and site builders have their own crawler-blocking or “hide from search” setting that’s independent of your robots.txt file — check the platform’s own indexing controls.
• Caching. A CDN or reverse proxy can keep serving a stale, cached robots.txt after you’ve published a fix, so Google keeps re-fetching the old rules until the cache clears.

If you’ve ruled out the standard Disallow/noindex explanations, escalate through this list before assuming the fix didn’t work.

Annotated examples

1. The trap — blocked and noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed.-ed at the same time

# robots.txt
User-agent: *
Disallow: /old-campaign/

<!-- /old-campaign/page.html -->
<meta name="robots" content="noindex">

Wrong: Google can’t crawl /old-campaign/page.html to ever see that noindex tag, so the page stays indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. on the strength of whatever links point at it. The two directives cancel each other out.

2. The correct removal — unblock, then noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed.

# robots.txt
User-agent: *
Allow: /old-campaign/

<!-- /old-campaign/page.html -->
<meta name="robots" content="noindex">

Right: crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is allowed, so Google reaches the page, reads the noindex, and drops it from the index on the next crawl/reprocess cycle.

3. The consolidation case — canonical, no noindex

# robots.txt
User-agent: *
Allow: /products/?variant=blue

<!-- /products/?variant=blue -->
<link rel="canonical" href="https://example.com/products/" />

Right: the variant URL is crawlable (so the canonical signal can pass) and carries no noindex — it’s meant to consolidate into the base product page, not disappear.

4. A junk URL that’s fine to leave alone

# robots.txt
User-agent: *
Disallow: /cart/add*

A simplified example: this add-to-cart pattern is blocked and may show as “indexed, though blocked” if anything links to it. Run it through the triage checklist — if it’s not surfacing for real queries and carries nothing sensitive, it’s usually safe to ignore.

Quick reference

Status comparison

Directive combinations and what they actually do

Fix by intent — one line each

• Want it indexed → remove the Disallow.
• Want it gone → allow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. + add noindex (or password-protect, or 404/410).
• Should consolidate → allow crawling + fix the canonical tagA rel=\"canonical\" annotation — in the HTML <head> or an HTTP Link header — that tells search engines which URL is the preferred version of duplicate or near-duplicate content., no noindex.
• Low-value junk URL that clears the triage checklist (not surfacing, nothing sensitive) → leave it.
• Links are the cause → remove or fix the internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. pointing at it.

Validation timing

• RecrawlCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. + reprocessing: days to weeks, not hours.
• URL Removal tool: temporary, roughly six months — not a real fix.

Toolkit for diagnosing and fixing this

Check the response headers and whether noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. is present (macOS/Linux)

curl -sI "https://example.com/path/to/page" | grep -i "x-robots-tag"
curl -s "https://example.com/robots.txt"

Run this to confirm whether a noindex is being served via HTTP header (needed for non-HTML files like PDFs) and to eyeball the live robots.txt for the exact Disallow line catching the URL.

Same check on Windows (PowerShell)

(Invoke-WebRequest -Uri "https://example.com/path/to/page" -Method Head).Headers["X-Robots-Tag"]
(Invoke-WebRequest -Uri "https://example.com/robots.txt").Content

Regex to pull every Disallow rule out of a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file

^Disallow:\s*(.+)$

Capture group 1 is the path pattern. Run this against a saved copy of robots.txt in your editor or a script to list every blocked path at once, so you can eyeball which rule is catching the flagged URL.

Chrome DevTools Console — check the rendered meta robotsThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen. tag

Run in the Console panel on the live page (confirms what Google’s renderer would actually see, not just the page source):

document.querySelector('meta[name="robots"]')?.content ?? 'no meta robots tag found'

Bookmarklet — check meta robots on any page in one click

Drag this to your bookmarks bar, then click it on any page:

javascript:(function(){alert(document.querySelector('meta[name="robots"]')?.content||'no meta robots tag found');})();

Tools for this task

• robots-txt-tester — paste the URL and your robots.txt to see exactly which rule is blocking it, before you change anything.
• robots-txt-generator — build a corrected robots.txt once you know which rule needs to change or narrow.
• canonical-checker — confirm the canonical tagA rel=\"canonical\" annotation — in the HTML <head> or an HTTP Link header — that tells search engines which URL is the preferred version of duplicate or near-duplicate content. is actually in place and pointing where you expect, for the consolidation branch of the fix.
• site-audit-lite — crawl the site to find internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. still pointing at the blocked/indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. URL, since those links are usually why it got indexed.
• gsc-workbench — pull the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. data and cross-check which URLs carry this warning versus the “Blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” excluded status.

Third-party: Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. (Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason., URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version., Validate Fix) is the primary place this warning appears and where you confirm the fix. Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. has an equivalent Block URLs report.

Proving the fix worked

Test 1 — robots.txt now allows the URL

Test to run: fetch robots.txt directly (curl -s https://example.com/robots.txt) or run it through the robots-txt-tester tool against the URL. Expected result: the URL is no longer matched by any Disallow rule. Failure interpretation: if it’s still matched, the block wasn’t fully removed or Google hasn’t re-fetched the updated file yet. Monitoring window: immediate for the file itself; Google typically re-fetches robots.txt within a day of a requested recrawlCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial.. Rollback trigger: none — this step only removes a block, it doesn’t itself change indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..

Test 2 — noindex is visible to the crawler (removal path only)

Test to run: URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. → live test in Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results., checking the rendered HTML for the noindex tag (or curl -I for the X-Robots-Tag header on non-HTML files). Expected result: the live test shows crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. allowed AND the noindex directive present in the rendered output. Failure interpretation: if crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is still blocked, the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. change hasn’t propagated; if crawling is allowed but no noindex shows, the tag was placed somewhere the renderer can’t see (JS-injected, or missing from the header on a non-HTML file). Monitoring window: immediate once the live test runs. Rollback trigger: n/a — this is a diagnostic check, not a change to undo.

Test 3 — the warning clears in the Page Indexing report

Test to run: “Validate Fix” on the “Indexed, though blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” warning in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance.’s Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.. Expected result: the URL moves off the warning into either the indexed/valid set (unblock path) or the excluded set (noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. path). Failure interpretation: a failed validation usually means the crawl hasn’t happened yet, not that the fix is wrong — check Tests 1 and 2 before changing anything further. Monitoring window: days to weeks; Validate Fix reprocessing is not immediate. Rollback trigger: if validation fails repeatedly (multiple weeks) after Tests 1 and 2 both pass, re-check for a caching layer or CDN serving a stale robots.txt/page.

Test 4 — consolidation path: canonical is being honored

Test to run: URL Inspection on the non-canonical URL, checking “Google-selected canonical” against the canonical-checker tool’s output. Expected result: Google’s selected canonical matches the URL you declared. Failure interpretation: a mismatch usually means competing signals (internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. entries) are still pointing search engines at the wrong URL as canonical. Monitoring window: 2–4 weeks — canonical selection is not instant even once crawling is allowed. Rollback trigger: if Google keeps selecting the wrong canonical after 4+ weeks, revisit internal linkingLinks between pages on the same site. and sitemap entries rather than re-touching the tag itself.

Quiz

Check what you took away from this one.

Indexed, though blocked by robots.txt

A Google Search Console Page Indexing warning: Google indexed the URL anyway despite your robots.txt disallowing crawling it — Google names other pages linking to it as the likely path. robots.txt blocks crawling, not indexing.

Related: Blocked by robots.txt, Robots.txt, Noindex

Indexed, though blocked by robots.txt

“IndexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., though blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” is a warning in the Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.. It means Google added the URL to its index even though your robots.txt file disallows crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. it. Because Google obeyed the disallow, it never fetched the page — so it indexed the URL based on external signals, which Google names other pages linking to it as the likely path, without ever seeing the content. Google itself says any resulting snippet will probably be very limited — sometimes title-less.

The whole point of this status is the thing most people get backwards: robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. A Disallow can’t remove a page from GoogleDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist., and it can actually trap one in the index, because Google can’t crawl in to see a noindex that would drop it.

The fix turns on intent. Want the page indexed? Unblock it so Google can crawl and properly index it. Want it out? Allow crawling and add a noindex (or password-protect / remove the page) — never a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. disallow. If the URL canonicalizes elsewhere, allow crawling and let canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. signals consolidate instead of adding noindex. For junk URLs like cart or parameter pages, triage first (does it surface, is anything sensitive) — it’s often fine to ignore once it clears that, but not automatically.

This is distinct from the sibling status “Blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” (excluded), where the page was blocked and is not indexed.

Related: Blocked by robots.txt, Robots.txt, Noindex

Changelog