Blocked by robots.txt (GSC Status)

TL;DR — “Blocked by robots.txt” in Search Console means Google found the page but didn’t read it, because your robots.txt file tells crawlers to stay out. Usually that’s on purpose and it’s fine — it’s the normal result of a block, not an error. It’s only a problem if you blocked a page you actually wanted in Google.

What this status means

Open the Page Indexing report in Google Search Console, click into “Blocked by robots.txt,” and you’ll see a list of URLs Google chose not to index. The reason is simple: Google found those URLs, but your site’s robots.txt file says crawlers aren’t allowed to fetch them. Evidence for this claim Google reports Blocked by robots.txt when crawling is disallowed and warns that this does not guarantee the URL cannot be indexed by other means. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report The label reports the crawl block; it does not guarantee that the URL can never be indexed through other signals.

The first thing I tell people is: don’t panic. This status is usually intentional. If you (or your CMS, or a plugin) blocked something on purpose — internal search results, faceted-navigation URLs, a staging path, cart and checkout URLs — then seeing it here is exactly what’s supposed to happen. It’s Google confirming the block worked, not flagging a mistake.

The one idea to hold onto

robots.txt controls crawling, not indexing. Those are two different things. Blocking a page in robots.txt stops Google from reading it; it does not delete Evidence for this claim Google says robots.txt manages crawler access and is not a mechanism for keeping a page out of Google. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: robots.txt introduction it from Google, and it isn’t a way to remove a page from search. I’ve tested the “what happens if you block a page” side of this myself — more on that in the Advanced tab — but the headline is: a block stops the crawl, full stop.

Is it a problem?

Run one check: did you mean to block these URLs?

• Yes, on purpose → leave it alone. This is working as intended.
• No, that page should be in Google → that’s the one case to fix. Find the rule in your robots.txt that’s catching the URL and remove or loosen it, then request indexing.

There’s a related-but-different status you might also see: “Indexed, though blocked by robots.txt.” That one’s a warning, not an exclusion — it means Google indexed a blocked URL anyway (usually because other pages link to it). Same cause, opposite outcome. If that’s what you’re looking at, it’s covered separately.

One more thing robots.txt can’t do: keep something private. It’s a request, not an access control — well-behaved bots honor it, but nothing forces them to, and nothing stops the URL from surfacing elsewhere (a link, a screenshot, a scraper that ignores robots.txt entirely). If a page holds sensitive content, protect it with a login or password, not a disallow.

The mistake to avoid

A lot of people try to remove a page from Google by blocking it in robots.txt, sometimes adding a noindex tag at the same time for good measure. That doesn’t work — and it backfires. If Google can’t crawl the page, it can never see the noindex on it, so the page can stay in Google. To actually take a page out of search, you do the opposite: let Google crawl it, and add noindex.

Want the full version — how to find the exact blocking rule, why the noindex + disallow combo conflicts, and what my own experiment showed — switch to the Advanced tab.

TL;DR — “Blocked by robots.txt” is a Page Indexing exclusion: Google discovered the URL but didn’t crawl it because robots.txt disallows it, so it isn’t indexed in this state. It’s normally intentional and benign — robots.txt governs crawling, not indexing, so a disallow is never a deindexing tool. Don’t confuse it with the warning “Indexed, though blocked by robots.txt” (Google indexed a blocked URL anyway, usually via inbound links). The classic mistake is pairing disallow with noindex: Google can’t crawl the page, so it never sees the noindex and the page can stay indexed. To remove a page, allow crawling and serve noindex. Only treat this status as a bug when you blocked a URL you actually wanted indexed.

What the status actually reports

In the Search Console Page Indexing report, “Blocked by robots.txt” is an excluded state, not an error and not a warning. Google’s own wording is plain: the page “was blocked by your site’s robots.txt file,” and Google adds the important caveat that this “does not guarantee that the page won’t be indexed through some other means.” Evidence for this claim Google reports Blocked by robots.txt when crawling is disallowed and warns that this does not guarantee the URL cannot be indexed by other means. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report That caveat is the whole topic in one line — see below.

Mechanically: Googlebot knows the URL exists (it found it via a link, a sitemap, or history), tried to respect your rules, hit a matching Disallow, and stopped. No fetch means no content to index, so the URL sits in this excluded bucket. This is the expected result of a disallow — most URLs in here are supposed to be in here.

Crawling is not indexing — why a disallow doesn’t deindex

Google documents robots.txt as crawl-access control, not a reliable removal mechanism. Evidence for this claim Google says robots.txt manages crawler access and is not a mechanism for keeping a page out of Google. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: robots.txt introduction

This is the accuracy spine of the whole status. robots.txt is a crawl control. Google is explicit that it “is not a mechanism for keeping a web page out of Google.” Blocking a URL prevents the fetch; it does not remove the URL from the index, and it is not a deindexing tool.

As I’ve put it in my Ahrefs article on the sibling status: “crawling and indexing are two different things.” A page you block can still end up indexed if other pages link to it — Google just can’t see the content, and it can’t see any noindex you tried to put there either. Which leads to the single most common mistake people make when they land on this report.

”Blocked by robots.txt” vs “Indexed, though blocked by robots.txt”

These two get conflated constantly, and they’re the same cause with opposite outcomes:

If you’re staring at the warning version — a blocked URL that got indexed anyway, showing as a bare URL with no description — that’s the indexed-though-blocked case, and it’s handled in its own write-up. This article is about the plain exclusion: blocked, and not indexed.

For utility URLs, “indexed anyway” is often nothing to worry about. John Mueller, addressing a site owner whose WooCommerce ?add-to-cart= URLs showed up as indexed-though-blocked, said you don’t need those URLs indexed, that blocking them with robots.txt is fine, and that even if they get “indexed” while blocked, they’re unlikely to actually surface in search unless someone runs a very specific query for those URLs — which real users don’t.

The noindex + disallow conflict (the #1 fix that backfires)

Here’s the trap. Someone wants a page gone, so they Disallow it in robots.txt and add a noindex meta tag — belt and suspenders. It doesn’t work, because the two instructions contradict each other.

Google states the rule directly: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.” If the page is disallowed, Google never crawls it, never sees the noindex, and the page can stay indexed. As I’ve written in my article on “Indexed, though blocked by robots.txt”: unless Google can crawl a page, it won’t see the noindex tag and may still index it because it has links.

So the deindex sequence is the opposite of what people reach for:

1. Allow crawling of the URL (remove the Disallow).
2. Serve noindex (meta robots tag or X-Robots-Tag header) and let Google re-crawl to see it.
3. Leave the page crawlable with noindex in place once it’s dropped. Re-blocking it afterward isn’t a safe “finishing move” — if you disallow the URL again, Google can lose visibility into the noindex rule on the next crawl, and a blocked-but-linked URL can get indexed again from other pages’ links, which is the exact “indexed, though blocked” outcome you were trying to avoid. If crawl budget on the removed page is a real concern, that’s a case for authentication or deletion (404/410) — not a return trip through robots.txt.

For urgent removals, the Search Console Removals tool, password protection, or simply deleting the page (returning 404/410) are the faster routes.

How to find which rule is blocking the URL

Three tools, each doing a different job — don’t expect one to do the others’ work:

• URL Inspection (GSC). Paste the specific URL. It tells you whether that URL is currently blocked and is the quickest per-URL check.
• robots.txt report (GSC). A domain-property-level monitoring report — not an editable tester — showing the robots.txt files Google found for your top hosts, the last-fetched time, fetch status, and any parsing warnings, plus a “request a recrawl” action after you change the file. It tells you Google can see your file; it doesn’t test individual URLs against it.
• A robots.txt validator or Google’s open-source robots.txt parser. To see which line matches a given URL, run it through a validator — because the longest, most specific matching rule wins (and an Allow can override a broader Disallow).

Once you’ve found the offending line, the fix depends on where your robots.txt lives. If you control the file directly, remove or correct the rule (and mind the syntax). If you’re on a hosted platform like Wix, Shopify, or Squarespace, you may need to follow that provider’s specific docs to change it, since some platforms manage robots.txt for you.

My own experiment: what happens when you block a page you wanted indexed

The genuinely useful question is: if you accidentally block a page that should rank, how bad is it? I ran that test directly. On January 30, 2023 I blocked two of our actually-ranking pages — “Top Bing Searches” and “Top YouTube Searches” — with robots.txt and tracked what happened.

The damage was real but smaller than I expected. We lost a position here or there (a few keywords slipped one or two spots, a couple even gained), and we lost all the featured snippets for those pages while they were blocked — they came back after I unblocked. The SERP appearance degraded too: Google showed “no information is available for this page” instead of the meta description, and lost our custom titles. And because the listing looked worse, clicks dropped more than impressions did — the CTR took the hit.

My summary at the time: “We lost a position here or there and all of the featured snippets for the pages. I expected a lot more impact, but the world didn’t end.” And the takeaway I’d still stand behind: don’t block pages you want indexed. It hurts. Not as bad as you might think — but it still hurts. That’s the right mental frame for this report. If everything in your “Blocked by robots.txt” bucket is stuff you meant to block, you’re fine. If a page you care about is in there, get it out.

One scope note on this experiment: both pages were already ranking and indexed before I blocked them, so what I measured is what happens when an indexed page gets pushed into the “indexed, though blocked” state — not what happens to a URL that was never indexed and is simply sitting in this exclusion bucket. If a URL here was never indexed to begin with, unblocking it just lets Google crawl and consider it normally; there’s no featured-snippet or CTR history to lose, because it was never there.

The decision tree

• Did you mean to block it? → Leave it. Working as intended.
• No — you want it indexed? → Remove/loosen the robots.txt rule, then request indexing.
• You want it gone from Google? → Don’t use a disallow. Allow crawling + noindex (or Removals tool / delete), then leave it crawlable — re-blocking it afterward can hide the noindex rule again.
• It’s a blocked URL that got indexed anyway? → That’s the indexed-though-blocked case, not this one — handle it there.
• The page has sensitive or private content? → robots.txt isn’t an access control; it’s a request bots can ignore. Use authentication or password protection, not a disallow.

The crawl-vs-index principle is universal, by the way: Bing honors robots.txt for crawling the same way, and removing a URL from Bing similarly uses its Block URLs tool or a noindex on a crawlable page — not a bare disallow.

For the broader file itself — syntax, wildcards, where it lives, and what it can and can’t do — see the robots.txt guide. For how indexing decisions get made upstream, see the indexing hub.

AI summary

A condensed take on the Advanced version:

• “Blocked by robots.txt” = a Page Indexing exclusion. Google found the URL but didn’t crawl it because robots.txt disallows it, so it isn’t indexed in this state. It’s the normal result of a block — usually intentional and benign.
• robots.txt controls crawling, not indexing. Google says it “is not a mechanism for keeping a web page out of Google.” A disallow stops the fetch; it does not deindex a page.
• Don’t confuse it with the warning “Indexed, though blocked by robots.txt.” Same cause, opposite outcome: that one means a blocked URL got indexed anyway, usually via inbound links. For utility URLs, Mueller says that’s typically fine — blocked-and-indexed URLs rarely surface for normal queries.
• The #1 backfire: noindex + disallow together. Google can’t crawl a disallowed page, so it never sees the noindex and the page can stay indexed. Google’s rule: for noindex to work, the page must not be blocked by robots.txt.
• To actually remove a page: allow crawling, serve noindex, let Google re-crawl, then leave it crawlable. Re-blocking it afterward can hide the noindex rule again and let the URL get indexed from links — the opposite of what you wanted. For urgent cases use the Removals tool, password protection, or delete the page.
• Find the blocking rule with URL Inspection (per-URL) and a robots.txt validator (which line matches; longest/most-specific rule wins). The GSC robots.txt report monitors file fetch status — it doesn’t test individual URLs. Fix depends on where the file lives — hosted platforms may need provider-specific steps.
• Patrick’s experiment: blocking two already-ranking, already-indexed pages cost a position here or there and all featured snippets, degraded the SERP listing, and dropped clicks more than impressions — “the world didn’t end,” but “don’t block pages you want indexed.” This shows what happens to an indexed page pushed into “indexed, though blocked” — it doesn’t forecast outcomes for a URL that was never indexed.
• robots.txt isn’t a security control. It’s a request bots can choose to ignore. Protect sensitive content with authentication, not a disallow.
• Decision rule: meant to block it → leave it; want it indexed → unblock + request indexing; want it gone → allow-crawl + noindex (and stay crawlable).

Official documentation

Primary-source documentation from the search engines.

Google

• Page indexing report — the report this status lives in, with the verbatim definitions of “Blocked by robots.txt” and “Indexed, though blocked by robots.txt.”
• Unblock a page blocked by robots.txt — Google’s step-by-step for finding and fixing the rule that blocks a URL.
• robots.txt report — the domain-property report showing the robots.txt files Google found, fetch status, warnings, and the recrawl request.
• Introduction to robots.txt — what robots.txt does and doesn’t do, including that it “is not a mechanism for keeping a web page out of Google.”
• Block Search indexing with noindex — the rule that noindex only works on a page that isn’t blocked by robots.txt (the noindex + disallow conflict).

Bing / Microsoft

• Bing Webmaster Tools Help — Bing honors robots.txt for crawling too; its robots.txt Tester and Block URLs tool follow the same crawl-vs-index principle.

Quotes from the source

On-the-record statements from Google. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what the status means

• “This page was blocked by your site’s robots.txt file. You can verify this using the robots.txt tester. Note that this does not guarantee that the page won’t be indexed through some other means.” — Google Search Console Help, Page indexing report. Jump to quote

Google — robots.txt controls crawling, not indexing

• “it is not a mechanism for keeping a web page out of Google.” — Google Search Central, Introduction to robots.txt. Jump to quote

Google — why noindex + disallow conflict

• “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.” — Google Search Central, Block Search indexing with noindex. Jump to quote

Stats & first-party data worth citing

• Blocking ranking pages: real but survivable damage. In my experiment blocking two already-ranking, already-indexed pages with robots.txt (started Jan 30, 2023), the pages lost a position here or there and all of their featured snippets, the SERP listing degraded to “no information is available for this page,” and clicks dropped more than impressions. My summary: “I expected a lot more impact, but the world didn’t end” — but “don’t block pages you want indexed. It hurts.” This tests what happens to an indexed page pushed into the “indexed, though blocked” state; it isn’t a forecast for a URL that was never indexed in the first place. Source
• Blocked utility URLs that get indexed are usually harmless. Per John Mueller, blocking add-to-cart-style URLs with robots.txt is fine; even if they show as indexed-though-blocked, they’re unlikely to surface for normal queries. Coverage
• How common robots.txt blocking is across the web — block-rate data on which crawlers ~140M sites disallow the most, from my study with Xibeijia Guan. Source

The mistake that defines this status: noindex + disallow

Mistake: pairing Disallow with noindex to try to remove a page. This is the single most common mistake tied to “Blocked by robots.txt,” and it’s worth naming directly because it feels like belt-and-suspenders and is actually self-defeating. Someone wants a page gone from Google, so they Disallow it in robots.txt and add a noindex meta tag for good measure. It doesn’t work. Google states the rule plainly: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.” If the page is disallowed, Google never crawls it, never sees the noindex, and the page can stay indexed — the exact opposite of the intended outcome. The fix is the opposite instinct: allow crawling, then serve noindex.

Myth: “A disallow in robots.txt is a deindexing tool.” It isn’t, and Google says so directly: robots.txt “is not a mechanism for keeping a web page out of Google.” Blocking a URL stops the fetch — it does not remove the URL from the index, and a blocked-but-linked page can still show up (that’s the sibling “Indexed, though blocked by robots.txt” status). Use noindex (with crawling allowed) or the Removals tool when the goal is actually removal.

Myth: “Everything in the ‘Blocked by robots.txt’ bucket is a problem to fix.” It’s usually the opposite. This status is the expected outcome of an intentional disallow — internal search results, faceted-nav parameters, staging paths. The only version of this status worth treating as a bug is a URL you actually wanted indexed sitting in the list by accident.

Mistake: blocking a page you want to rank, without checking the cost first. Blocking a page you want ranking to “clean things up” or “save crawl budget” is riskier than it looks. In my own experiment blocking two ranking pages, I lost all featured snippets while they were blocked and clicks dropped more than impressions because the SERP listing degraded to “no information is available for this page.” The rankings themselves mostly held — but treat a robots.txt block on a page you care about as a real decision, not a housekeeping tweak.

”Blocked by robots.txt” triage checklist

Work through this list top to bottom when you open the status in Search Console:

• Open the Page Indexing report and click into “Blocked by robots.txt” to see the actual list of affected URLs.
• For each URL (or pattern of URLs), ask: did I mean to block this?
• If yes — internal search, faceted nav, staging, cart/checkout — leave it alone. This status is working as intended for that URL.
• If no, or you’re not sure — run the URL through URL Inspection to confirm it’s currently blocked (not a stale snapshot).
• Find the specific rule catching the URL with the robots.txt Tester (or another robots.txt validator) — the longest, most specific matching rule wins, and an Allow can override a broader Disallow. GSC’s robots.txt report monitors file fetch status; it doesn’t test individual URLs.
• Confirm you’re not also relying on noindex on the same disallowed URL — that combination can’t work; pick allow-crawl-plus-noindex instead if the goal is removal.
• Remove or loosen the blocking rule (directly if you control robots.txt, or via your hosting platform’s docs if it’s managed for you, e.g. Wix, Shopify, Squarespace).
• Request indexing for the fixed URL in Search Console once the block is cleared.
• Double-check you’re not confusing this status with “Indexed, though blocked by robots.txt” — that’s the opposite outcome (indexed anyway) and needs a different fix.
• Re-check the bucket in a few weeks to confirm the count trended the way you expected (down for fixed URLs, flat for intentional blocks).
• If any of these URLs hold sensitive content, don’t rely on the disallow to keep it private — add authentication or password protection instead.

Work the decision tree

Answer one question at a time to land on the right move for a URL sitting in the “Blocked by robots.txt” bucket.

The mental models

1. Crawling ≠ indexing. The whole status collapses into confusion if you conflate the two. robots.txt controls crawling — whether Googlebot fetches a page. It has no direct control over indexing — whether the URL ends up in Google’s index. A disallowed page isn’t indexed via crawling, but it can still be indexed without content if other pages link to it. Keep the two ideas separate and most of this status stops being confusing.

2. The one-question triage. Before doing anything else, ask: did I mean to block this? Yes → leave it, it’s working as intended. No → that’s the one case worth fixing. This single question sorts almost every URL in the bucket correctly.

3. To remove a page, invert your instinct. The natural move — disallow plus noindex, “extra safe” — is exactly backwards. Google can’t see a directive on a page it never fetches. The correct sequence is the opposite of the intuitive one: allow crawling, then serve noindex. Anytime someone reaches for a disallow to make a page disappear, that’s the signal to apply this model.

4. robots.txt is a crawl-budget tool, not a search-results tool. Use it to keep bots off low-value or infinite URL spaces (internal search, facets, staging) so crawl effort goes toward pages you want found — not as a lever for what shows up in search results. That job belongs to noindex, password protection, or deletion.

Track the bucket count, not just its existence

The one number worth watching is how many URLs sit in the “Blocked by robots.txt” row of the Page Indexing report over time — a single snapshot can’t tell you whether a fix worked or whether new unintended blocks are creeping in.

”Blocked by robots.txt” bucket count over time

• Metric — The count of URLs under “Blocked by robots.txt” in GSC’s Page Indexing report, tracked over time.
• What it tells you — Whether the bucket is behaving the way you expect. For URLs you intentionally block (internal search, facets, staging), the count should stay roughly flat as those URL spaces stay stable — a sudden jump usually means a new section got swept into an existing Disallow pattern by accident. For a URL you deliberately unblocked because you wanted it indexed, that specific URL should drop out of the bucket after the next crawl.
• How to pull it — GSC Page Indexing report, filtered to the “Blocked by robots.txt” row; spot-check specific URLs with URL Inspection to confirm current status rather than a stale snapshot.
• Benchmark / realistic range — There’s no universal target — it depends entirely on how much of your site is intentionally disallowed. The honest bar: zero URLs in this bucket that you actually wanted indexed, and a flat trend for everything else. Baseline your own count before judging whether a change moved it in the right direction.
• Cadence — Check right after any robots.txt change; otherwise a periodic glance (monthly is plenty for most sites) is enough to catch drift.

The playbook: actually removing a page from Google

A repeatable sequence for when a URL needs to come out of Google’s index for real — not just get blocked from future crawling.

1. Confirm the goal is removal, not just crawl control. If the goal is really “stop wasting crawl budget on this,” a plain robots.txt disallow is the right tool and this playbook doesn’t apply — that page can stay indexed via links and that’s fine. Only use this sequence when the page must actually leave the index.

2. Allow crawling. Remove or loosen the Disallow rule blocking the URL. Confirm with the robots.txt Tester that the URL now comes back allowed, not blocked.

3. Serve noindex. Add a noindex meta robots tag or X-Robots-Tag HTTP header to the now-crawlable page. This is the actual removal instruction — the disallow was never doing that job.

4. Let Google re-crawl and pick it up. Google has to fetch the page again to see the noindex. There’s no way to force an instant fetch beyond requesting indexing in Search Console and waiting for the next crawl.

5. Confirm removal. Check URL Inspection in Search Console for the page’s current indexed status. Once it reports the page as excluded due to noindex, the removal has worked.

6. Don’t treat re-blocking as the finishing step. Once the page has dropped, adding the Disallow back isn’t free: Google can lose visibility into the noindex rule again on the next crawl, and a blocked-but-linked URL can come back as indexed via other pages’ links — the exact “indexed, though blocked” outcome you were trying to avoid. If crawl budget is the concern, weigh that against the risk, or reach for authentication/deletion instead for pages that truly need to stay gone for good.

7. For urgent cases, skip the wait. If the page needs to be out now, use the Search Console Removals tool, password-protect the page, or delete it outright (return 404/410) instead of waiting on the crawl-then-noindex cycle.

Ready-to-use AI prompts

Copy-paste prompts for working through a robots.txt block with an LLM. Treat the output as a starting hypothesis — confirm anything load-bearing with the robots.txt Tester or Search Console’s URL Inspection tool before acting on it.

Figure out which rule blocks which URL

Here is my robots.txt file and a list of URLs. For each URL, tell me whether it
is blocked or allowed, and quote the exact line in robots.txt responsible
(remember: the longest, most specific matching rule wins, and an Allow can
override a broader Disallow). If a URL isn't matched by any rule, say so.

ROBOTS.TXT:
[paste]

URLS:
[paste list, one per line]

Sanity-check whether a block looks intentional

I run a [type of site]. Here is my robots.txt file. For each Disallow rule, tell
me what kind of URLs it likely targets (e.g. internal search, faceted
navigation, staging, cart/checkout, admin) and flag any rule that looks broad
enough it might be catching content pages by accident. Don't guess about pages
you can't see — just reason from the rule pattern and ask me to confirm
anything ambiguous.

ROBOTS.TXT:
[paste]

Draft the noindex-then-crawl sequence for a specific page

I need to remove [URL] from Google. It's currently blocked by this robots.txt
rule: [paste rule]. Walk me through the exact sequence of changes in order
(robots.txt edit, meta tag or header change, what to check in Search Console at
each step) so I don't accidentally leave the noindex tag unseen by Google.

Test a URL against your robots.txt rules

Before you change anything, confirm exactly which rule is catching a given URL. The robots.txt Tester does this with a matcher ported from Google’s own open-source robots.txt parser and shows the winning rule per URL — that’s the fastest and most accurate way to check. For a quick manual sanity check without a tool, here’s the core logic in Python:

import re

def rule_matches(path, rule):
    """Very simplified robots.txt path matcher: '*' = wildcard, '$' = end anchor."""
    pattern = re.escape(rule).replace(r'\*', '.*')
    if pattern.endswith(r'\$'):
        pattern = pattern[:-2] + '$'
    return re.match(pattern, path) is not None

def find_blocking_rule(path, disallow_rules, allow_rules):
    """Longest matching rule wins; ties go to Allow (mirrors Google's documented
    precedence). Returns the winning rule string, or None if nothing matches."""
    matches = [r for r in disallow_rules if rule_matches(path, r)]
    matches += [r for r in allow_rules if rule_matches(path, r)]
    if not matches:
        return None
    return max(matches, key=len)

# Example
disallow = ["/search", "/*?*sort="]
allow = ["/search/help"]
print(find_blocking_rule("/search/help", disallow, allow))  # -> "/search/help" (Allow wins, longer)
print(find_blocking_rule("/search/results", disallow, allow))  # -> "/search" (Disallow, no competing Allow)

This is a simplification of Google’s actual matcher (real robots.txt parsing has more edge cases around escaping and group selection) — use it to sanity-check your mental model, and confirm anything that matters with the real tester before you rely on it.

Tools for diagnosing and fixing this status

• Google Index Checker — once you’ve fixed a block, use this to check the page’s observable indexability signals (status code, redirects, noindex, canonical) before routing to Search Console for Google’s actual answer.
• URL Inspection (Google Search Console) — the source of truth for whether a specific URL is currently blocked, its live-tested indexability, and (once fixed) whether Google has picked it back up.
• robots.txt report (Google Search Console) — a domain-property-level view of the robots.txt files Google found for your top hosts, last-fetched time, and fetch status, with a “request a recrawl” action after you change the file.

Test yourself: Blocked by robots.txt

Five questions on the “Blocked by robots.txt” status and the mistakes people make around it. Pick an answer for each, then check.

Prove the removal actually worked

“I set noindex” or “I edited robots.txt” isn’t proof of anything on its own — the only result that counts is what Google’s index actually reflects afterward. These tests separate the change you made from the outcome Google reached.

Test 1 — Crawling is genuinely allowed

• Test to run — Run the URL through the robots.txt Tester after editing the file.
• Expected result — The URL comes back allowed, with no Disallow rule matching it (or a more specific Allow winning over a broader Disallow).
• Failure interpretation — Still blocked means a different, more specific rule is catching the URL, or the edit didn’t deploy — check the file’s actual live contents, not just what you intended to change.
• Monitoring window — Immediate — this is a static check of the file as served.
• Rollback trigger — N/A — fix the rule before moving to the next test; nothing downstream works until crawling is actually allowed.

Test 2 — noindex is being served correctly

• Test to run — URL Inspection → Live Test in Search Console (or fetch the page directly and check the response headers/HTML) on the now-crawlable URL.
• Expected result — The live test shows noindex present, either as a meta robots tag in the <head> or an X-Robots-Tag HTTP header.
• Failure interpretation — Missing noindex here means the removal will never happen no matter how long you wait — the crawl-allowed step alone doesn’t remove anything.
• Monitoring window — Immediate — live test reflects the current served page.
• Rollback trigger — N/A — this must pass before Google has any chance to act on it.

Test 3 — Google has actually dropped the page

• Test to run — URL Inspection (the indexed-status field, not just Live Test) on the URL, checked periodically after the previous two tests pass.
• Expected result — The status changes to excluded due to noindex, and the URL stops appearing for a site: search or in your rank-tracking data.
• Failure interpretation — Still indexed after a full crawl cycle usually means caching/propagation delay, not a broken setup — Google has to re-crawl to notice the change, and that isn’t instant.
• Monitoring window — 1–3 weeks, depending on how often Google was already crawling the URL — don’t judge this the day after shipping the change.
• Rollback trigger — the named abort condition — Still indexed after a full crawl cycle and the page keeps showing as freshly crawled in server logs → stop assuming it’s just propagation and re-check Tests 1 and 2 for a mistake (a caching layer serving a stale robots.txt, a CDN stripping the X-Robots-Tag header, or a second conflicting rule are the usual culprits).

Blocked by robots.txt

A Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.

Related: Indexed, though blocked by robots.txt, Robots.txt, Crawling

Blocked by robots.txt

“Blocked by robots.txt” is an exclusion status in Google Search Console’s Page Indexing report. It means Googlebot found the URL but didn’t crawl it, because your site’s robots.txt file disallows it — so the page isn’t being indexed in this state. Most of the time this is exactly what you intended, and it’s nothing to panic over.

The key thing to keep straight: robots.txt controls crawling, not indexing. A disallow stops the fetch; it doesn’t guarantee a page stays out of Google. That’s why this status has a sibling — the “Indexed, though blocked by robots.txt” warning, which is the other outcome: Google indexed a blocked URL anyway, usually because something links to it.

Because Google can’t crawl a disallowed URL, it also can’t see a noindex you put on that page — so you can’t reliably remove a page from Search with a robots.txt disallow. To actually deindex, allow crawling and serve noindex instead. It’s only worth “fixing” this status when you blocked a page you actually wanted indexed.

Related: Indexed, though blocked by robots.txt, Robots.txt, Crawling

Changelog