robots.txt Tester

Free, no signup. robots.txt gives no warning when a misplaced Disallow hides half a site from Google — it just quietly works. Test any list of URLs against it with a matcher ported from Google's own open-source parser — longest-match-wins, wildcards, sub-agent fallback, and status-code semantics handled exactly the way Google does it.

Built by Patrick Stox

1. robots.txt
2. Pages to test
3. Bots

Example data — replace with your own

Runs entirely in your browser — nothing you paste is uploaded or stored. The server is only used to fetch a site's robots.txt or sitemap (browsers can't — no CORS). Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.

Feedback
Report a bug

Found something broken in Robots Txt Tester? Let us know what happened — this goes straight to a private triage queue, not a public list.

What will be sent 
 No tool inputs, uploads, pasted source, complete results, query parameters, or URL fragments are attached automatically. You can edit or remove the selected passage above. Browser and anti-abuse metadata is processed for spam prevention. 
  
 
  
 
 
  
 
 
   
    
 
Sample report Example data — real, verified result
 
Given this robots.txt:
 
 
User-agent: *
Disallow: /search
Allow: /search/howsearchworks
 
 
…then test /search/howsearchworks and /search?q=redirects against Googlebot. The matrix comes back:
  
     
  • Longest match wins, not first match. /search/howsearchworks is
            allowed even though Disallow: /search also matches it — the longer
Allow: /search/howsearchworks on line 3 is more specific and overrides it.
    •  
  • The second URL has no exception. /search?q=redirects is blocked by
Disallow: /search on line 2 — nothing more specific matches it.
    •  
  • Click a verdict to see why. Clicking either cell highlights the exact line that
            decided it — see what "blocked"/"allowed" and the L-numbers mean
and the common-mistakes list below for how order-independent matching trips people up.
    •  
  • Background reading. The robots.txt guideA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.
covers every directive this matcher supports.
    •  
 
 
 
How to use it
 
     
  1. Load a robots.txt. Enter a domain to fetch the live file, or switch to
Paste to test a draft that isn't published yet. In fetch mode, full URLs on
            other subdomains automatically use that origin’s own /robots.txt.
    2.  
  2. Add the pages to test. Paste URLs (full or bare paths), upload a
.txt/.csv list, or pull them straight from the site's
sitemap.xml.
    3.  
  3. Pick the bots. Choose the crawlers you care about, or hit
AI crawler audit to load the AI bots (GPTBot, ClaudeBot, Google-Extended, and
            the rest) in one click.
    4.  
  4. Read the matrix. Every URL × bot cell shows allowed or blocked. Click a cell
            to see the exact rule that decided it, highlighted in the file.
    5.  
 
 
 
     
     
       
 
How it works
 

The matcher is ported from Google's open-source robots.txt parser, so it decides
          allow/block the way Googlebot actually does — not with a naive "first match wins" rule that most
          testers use. That means longest-match-wins (the most specific
Allow/Disallow path takes precedence regardless of order),
* and $ wildcards, percent-encoding normalisation, and
sub-agent fallback (a bot with no group of its own falls back to
User-agent: *).

 
 Crawl-delay is crawler-specific, not part of this Google matcher. Google does not process it.
Bing documents 1–20 second values.
          This tool reports Google-style Allow/Disallow decisions and never pretends a Crawl-delay line changes Googlebot’s result.

 

All of that runs in your browser. The server is only touched to fetch a live
          robots.txt or sitemap, because browsers can't request them cross-origin. It also models
          robots.txt status-code semantics: most 4xx responses are treated as "no robots.txt,"
          but 429 is a temporary failure like a 5xx or timeout. Temporary failures can initially pause crawling;
          Google may also reuse its last known good file, which a point-in-time tester cannot observe.

 
Features
 
     
  • Google-accurate matching: longest-match-wins, wildcards, sub-agent fallback, status-code rules.
    •  
  • Blocked/allowed matrix across as many URLs and bots as you like.
    •  
  • Click any verdict to see the exact matching line highlighted in the file.
    •  
  • One-click AI crawler audit for the major LLM and AI-search bots.
    •  
  • Built-in linter that flags common robots.txt mistakes.
    •  
  • Fetch by domain, upload a URL list, or pull URLs from a sitemap — including a separate live robots.txt for each entered subdomain (up to 10 origins per run).
    •  
  • Privacy: matching is fully client-side — pasted files never leave your machine.
    •  
 
Common mistakes it catches
 
     
  • Assuming rule order matters — it doesn't for Google; the most specific path wins.
    •  
  • Blocking a resource (CSS/JS) that the page needs to render, then wondering why rankings drop.
    •  
  • Using robots.txt to keep a page out of the index — disallow stops crawling, not indexing; use
NoindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. for that.
    •  
  • A stray Disallow: / under User-agent: * that quietly blocks the whole site.
    •  
 
Limitations
 

This checks Google-style robots.txt matching only — it tells you whether a bot is allowed to crawl a
          URL, not whether the page is indexable (a page can be crawlable and still carry a
noindex). Fetch-by-domain reads the robots.txt exactly as served; if a site returns
          different robots.txt to different user-agents or regions, you'll see the version your fetch
          received. A robots.txt file is scoped to its origin, so fetch mode requests each
          entered subdomain’s own file. Paste mode deliberately uses the one pasted draft for every URL.
          Non-Google crawler behavior is not inferred here; consult that crawler’s own documentation before
          relying on system-specific extensions.

 
Frequently asked questions
 
Does robots.txt stop a page from being indexed?
No. Disallow blocks crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. If other pages link to a disallowed URL, Google can still index it (usually without a snippet). To keep a page out of the index, let it be crawled and add a noindex meta tag or X-Robots-Tag header — which means you must not also disallow it, or Google can never see the noindex.
Does the order of rules in robots.txt matter?
Not for Google. Google applies the most specific rule — the longest matching path — regardless of where it sits in the file, so an Allow can override an earlier Disallow. This tool matches the same way. Some other crawlers use first-match-wins, so keep rules unambiguous.
What is the difference between Disallow and noindex?
Disallow (in robots.txt) tells crawlers not to fetch a URL. noindex (a meta tag or HTTP header on the page) tells them not to keep it in the index. They solve different problems, and combining them backfires: a disallowed page is never fetched, so its noindex is never seen.
Will blocking CSS or JavaScript in robots.txt hurt SEO?
It can. Google renders pages to understand them, so if it cannot fetch the CSS and JS a page needs, it may misjudge the layout, content, or mobile-friendliness. Allow the resources required to render your pages and only block genuinely non-essential paths.
How do I block AI crawlers with robots.txt?
Add a user-agent group for each AI bot — such as GPTBot, Google-Extended, ClaudeBot, CCBot, and PerplexityBot — with Disallow: /. Use the AI crawler audit button to load them and confirm your rules match. Note that robots.txt compliance is voluntary; reputable bots honour it, but it is not an access control.
 
 
 
     
Next steprobots.txt Generator — generate a replacement robots.txt with the directives you meant to ship.

Explore more
ToolLog File AnalyzerToolAI-Crawler Access CheckerToolGooglebot VerifierArticleUser AgentArticleAI CrawlersArticleNuxt SEO
 
 
 
Feature requests for Robots Txt Tester
 

Upvote what you want most. New ideas can be submitted from the floating Feedback menu;
      requests appear here once approved, and the most-wanted rise to the top.

   
Loading…
  
  ➕ Request a feature   
New requests are reviewed before they appear here.
 
     
  
  
 
 
 
 
   
        
 
Common issues & how to fix them
 
 
 
 
 
robots.txt Tester · one finding
 
Unsupported noindex directive is used
 
Recommended action: Remove noindex from robots.txt and use a meta robots tag or X-Robots-Tag on crawlable URLs instead.
 
Fix principle: Remove every unintended noindex from its confirmed owner; absence of noindex is sufficient, while an explicit index directive cannot override another noindex because restrictive directives combine.
 
 
 
 Implementation for your setup 
 
Confirm first: The final HTML or HTTP response contains a noindex directive, including one added only by client-side script during rendering. The page is intended to be an indexable canonical URL. A robots meta tag or X-Robots-Tag intended to keep the page indexable is neutralized by a conflicting directive elsewhere, because directives combine using the most restrictive rule. Robots.txt disallows the URL itself, or disallows a resource the page depends on to render, so the crawler cannot reach or fully evaluate the intended indexable page.
 
Do not apply when: Exclusion is intentional for the page type, environment, or workflow state. The directive appears only on a noncanonical or otherwise ineligible URL. Robots.txt does not disallow the URL or the resources it needs to render, and the effective combination of every directive already resolves to the intended indexable state.
 
Likely control points: cms-app · server · edge-cdn
 
   
Choose the layer that owns the final response.
 
 
 
Generic HTTP / HTML
 
Status: supported · Location: The application, server, or edge owner that emits the directive, or the robots.txt file at the site root
 
Steps
  1. Follow redirects, identify the exact owner of each unintended robots/googlebot meta or X-Robots-Tag noindex, and remove it only at that owner.
  2. Return the intended canonical page with a crawlable 200 response, align its canonical, and purge the caches that can retain the directive.
  3. If a robots meta tag or X-Robots-Tag is present but rendered ineffective by a conflicting directive elsewhere, remove the actual unwanted restrictive directive rather than adding a permissive one to compete with it — the most restrictive rule always wins. Google also honors a robots meta tag placed outside <head>, so an out-of-head placement is a specification and cross-crawler risk to correct, not by itself proof Google is ignoring the directive.
  4. If robots.txt disallows the URL itself, or disallows a resource (such as CSS or JS) the page needs to render its content, correct only the specific Disallow rule at the robots.txt file that owns it — keep the rule as narrow as its original intent, and remember Disallow blocks crawling, not indexing: a disallowed URL can still be indexed if discovered elsewhere.
  5. Keep the URL crawlable so Google can observe the removal.
  
Verify on this platform
 
  • After redirects and cache purge, inspect every robots and googlebot meta tag plus every X-Robots-Tag value on the final response.
  • Fetch and parse robots.txt for every group that applies to the relevant user-agent — Google consolidates every group matching the same user-agent token into one effective ruleset, though not every crawler does — and confirm the URL and its critical rendering resources are not disallowed.
  • Use Search Console URL Inspection’s live test on the production URL, then request indexing when appropriate.
 
Rollback
  1. Restore the prior directive or robots.txt rule if the page or a resource it needs was exposed or blocked unintentionally, then correct the underlying publication rule before trying again.
 
Caveats
  • Do not add index as a counter-directive: the most restrictive directive applies, so another noindex still wins.
  • The live test checks current accessibility; it does not guarantee selection, canonical choice, or indexing.
  • Google’s own parser combines every robots.txt group matching the same user-agent token into one effective ruleset, but many other crawlers and Robots Exclusion Protocol implementations honor only the first matching group — treat a directive split across groups as a real interoperability risk even when Google itself is unaffected.
  • If the page combines a canonical to a different URL with a noindex directive, remove the noindex only if this URL itself should be indexable; if the canonical target is the one meant to be indexed, keep the noindex here and verify the canonical target separately rather than treating its indexability as confirmed by removing this noindex.
 
Official sources
 
 
WordPress
 
Status: supported · Location: Global Settings → Reading/blog_public, or the scoped plugin/theme callback that owns wp_robots
 
Steps
  1. If WordPress core globally owns the directive, confirm the scope and change Settings → Reading so blog_public permits search-engine visibility.
  2. For a scoped page-type or URL rule, identify and change the plugin or theme callback filtering wp_robots or invoking the core noindex behavior; do not invent a plugin UI.
  3. Purge page, object, host, and edge caches that can retain the old output.
  
Verify on this platform
 
  • While logged out, follow redirects and inspect all robots/googlebot meta plus X-Robots-Tag values on the final public response.
 
Rollback
  1. Restore the previous visibility or plugin setting if unintended pages become indexable.
 
Caveats
  • Do not use the global Reading setting to solve a scoped plugin or theme rule.
  • WordPress exposes the wp_robots API, but a third-party owner’s exact UI is not verified here.
  • Robots.txt is a separate control point from this meta/header directive — correct a Disallow rule using the generic-http-html guidance above regardless of which CMS serves the site.
 
Official sources
 
 
Cloudflare Workers / Rules
 
Status: supported · Location: Confirmed Response Header Transform Rule or Worker emitter
 
Steps
  1. Use Cloudflare Trace and configuration review to confirm which Transform Rule or Worker adds the final X-Robots-Tag and account for response-header rule ordering.
  2. Narrow or remove only that confirmed edge emitter; in a Worker, clone the response before modifying its headers.
  3. Purge the affected cache entries and leave any valid origin-owned robots header intact.
  
Verify on this platform
 
  • Compare origin and final Cloudflare responses after redirects, then confirm every final robots header and meta directive is free of unintended noindex.
 
Rollback
  1. Restore the prior edge rule if the scope change exposes excluded routes, then replace it with a narrower condition.
 
Caveats
  • Never blanket-delete a valid origin X-Robots-Tag when Cloudflare is not the confirmed emitter.
  • Later response-header rules can override earlier changes, so verify the final rule order and response.
  • Robots.txt is a separate control point from this response-header directive — correct a Disallow rule using the generic-http-html guidance above.
 
Official sources
 
 
Next.js / Vercel
 
Status: supported · Location: App Router metadata/generateMetadata, Pages Router next/head, or next.config headers()
 
Steps
  1. In the App Router, remove the unintended noindex from static metadata.robots or generateMetadata at the route that owns it.
  2. In the Pages Router, correct the owning next/head robots meta; for an X-Robots-Tag, correct the scoped next.config headers() rule.
  3. Deploy to the intended production environment and purge or bypass relevant caches before verification.
  
Verify on this platform
 
  • Inspect all final robots/googlebot meta and X-Robots-Tag values after redirects on the current production deployment.
 
Rollback
  1. Revert the route metadata, Head output, or headers() rule if the deployment exposes a route that should remain excluded.
 
Caveats
  • Vercel intentionally serves X-Robots-Tag: noindex for Preview Deployments and outdated Production Deployments; preserve that protection.
  • Do not diagnose an intentional preview response as a production metadata defect.
  • Robots.txt (a static file or a route handler such as app/robots.ts) is a separate control point from this metadata/header directive — correct a Disallow rule using the generic-http-html guidance above.
 
Official sources
 
 
Platform-managed / ask the vendor
 
Status: platform-managed · Location: Vendor-owned publication, visibility, or response-header control
 
Steps
  1. Ask the vendor to identify the exact global or scoped control that emits noindex and remove it only for the confirmed public canonical URL.
  
Verify on this platform
 
  • Follow redirects and inspect all robots/googlebot meta and X-Robots-Tag values on the final public response, then use the Search Console live test.
 
Rollback
  1. Restore the vendor visibility control if an unintended page becomes public while the owning rule is corrected.
 
Caveats
  • Do not add index to compete with a noindex emitted elsewhere.
  • Keep the URL crawlable long enough for search engines to observe the removal.
  • Ask the vendor to confirm whether it also owns robots.txt for the domain — a Disallow rule there is a separate control point from a meta/header directive.
 
Official sources
 
 
 
 
 How to verify 
 
  1. Follow redirects and confirm a crawlable 200 final response with no noindex in any robots/googlebot meta or X-Robots-Tag value.
  2. Confirm the canonical points to the intended URL and relevant caches no longer serve the directive.
  3. Fetch and parse robots.txt for the applicable user-agent group(s) and confirm the URL and its critical rendering resources are not disallowed.
  4. Rerun SEO Site Audit Crawler, On-Page SEO Checker, Google Index Checker, Render Gap, or Robots.txt Tester (whichever surfaced the finding), and use Search Console URL Inspection’s live test for Google’s current view.
 
Escalate when
 
Escalate when page ownership or publication intent is unclear, noindex or a robots.txt Disallow rule is applied by multiple layers, or removing it would expose private, duplicate, or staging content.
 
 
 
 
 
 
 
robots.txt Tester · one finding
 
User-agent is split across groups
 
Recommended action: Merge rules for the same user-agent into one group so the effective policy is clear and maintainable.
 
Fix principle: Remove every unintended noindex from its confirmed owner; absence of noindex is sufficient, while an explicit index directive cannot override another noindex because restrictive directives combine.
 
 
 
 Implementation for your setup 
 
Confirm first: The final HTML or HTTP response contains a noindex directive, including one added only by client-side script during rendering. The page is intended to be an indexable canonical URL. A robots meta tag or X-Robots-Tag intended to keep the page indexable is neutralized by a conflicting directive elsewhere, because directives combine using the most restrictive rule. Robots.txt disallows the URL itself, or disallows a resource the page depends on to render, so the crawler cannot reach or fully evaluate the intended indexable page.
 
Do not apply when: Exclusion is intentional for the page type, environment, or workflow state. The directive appears only on a noncanonical or otherwise ineligible URL. Robots.txt does not disallow the URL or the resources it needs to render, and the effective combination of every directive already resolves to the intended indexable state.
 
Likely control points: cms-app · server · edge-cdn
 
   
Choose the layer that owns the final response.
 
 
 
Generic HTTP / HTML
 
Status: supported · Location: The application, server, or edge owner that emits the directive, or the robots.txt file at the site root
 
Steps
  1. Follow redirects, identify the exact owner of each unintended robots/googlebot meta or X-Robots-Tag noindex, and remove it only at that owner.
  2. Return the intended canonical page with a crawlable 200 response, align its canonical, and purge the caches that can retain the directive.
  3. If a robots meta tag or X-Robots-Tag is present but rendered ineffective by a conflicting directive elsewhere, remove the actual unwanted restrictive directive rather than adding a permissive one to compete with it — the most restrictive rule always wins. Google also honors a robots meta tag placed outside <head>, so an out-of-head placement is a specification and cross-crawler risk to correct, not by itself proof Google is ignoring the directive.
  4. If robots.txt disallows the URL itself, or disallows a resource (such as CSS or JS) the page needs to render its content, correct only the specific Disallow rule at the robots.txt file that owns it — keep the rule as narrow as its original intent, and remember Disallow blocks crawling, not indexing: a disallowed URL can still be indexed if discovered elsewhere.
  5. Keep the URL crawlable so Google can observe the removal.
  
Verify on this platform
 
  • After redirects and cache purge, inspect every robots and googlebot meta tag plus every X-Robots-Tag value on the final response.
  • Fetch and parse robots.txt for every group that applies to the relevant user-agent — Google consolidates every group matching the same user-agent token into one effective ruleset, though not every crawler does — and confirm the URL and its critical rendering resources are not disallowed.
  • Use Search Console URL Inspection’s live test on the production URL, then request indexing when appropriate.
 
Rollback
  1. Restore the prior directive or robots.txt rule if the page or a resource it needs was exposed or blocked unintentionally, then correct the underlying publication rule before trying again.
 
Caveats
  • Do not add index as a counter-directive: the most restrictive directive applies, so another noindex still wins.
  • The live test checks current accessibility; it does not guarantee selection, canonical choice, or indexing.
  • Google’s own parser combines every robots.txt group matching the same user-agent token into one effective ruleset, but many other crawlers and Robots Exclusion Protocol implementations honor only the first matching group — treat a directive split across groups as a real interoperability risk even when Google itself is unaffected.
  • If the page combines a canonical to a different URL with a noindex directive, remove the noindex only if this URL itself should be indexable; if the canonical target is the one meant to be indexed, keep the noindex here and verify the canonical target separately rather than treating its indexability as confirmed by removing this noindex.
 
Official sources
 
 
WordPress
 
Status: supported · Location: Global Settings → Reading/blog_public, or the scoped plugin/theme callback that owns wp_robots
 
Steps
  1. If WordPress core globally owns the directive, confirm the scope and change Settings → Reading so blog_public permits search-engine visibility.
  2. For a scoped page-type or URL rule, identify and change the plugin or theme callback filtering wp_robots or invoking the core noindex behavior; do not invent a plugin UI.
  3. Purge page, object, host, and edge caches that can retain the old output.
  
Verify on this platform
 
  • While logged out, follow redirects and inspect all robots/googlebot meta plus X-Robots-Tag values on the final public response.
 
Rollback
  1. Restore the previous visibility or plugin setting if unintended pages become indexable.
 
Caveats
  • Do not use the global Reading setting to solve a scoped plugin or theme rule.
  • WordPress exposes the wp_robots API, but a third-party owner’s exact UI is not verified here.
  • Robots.txt is a separate control point from this meta/header directive — correct a Disallow rule using the generic-http-html guidance above regardless of which CMS serves the site.
 
Official sources
 
 
Cloudflare Workers / Rules
 
Status: supported · Location: Confirmed Response Header Transform Rule or Worker emitter
 
Steps
  1. Use Cloudflare Trace and configuration review to confirm which Transform Rule or Worker adds the final X-Robots-Tag and account for response-header rule ordering.
  2. Narrow or remove only that confirmed edge emitter; in a Worker, clone the response before modifying its headers.
  3. Purge the affected cache entries and leave any valid origin-owned robots header intact.
  
Verify on this platform
 
  • Compare origin and final Cloudflare responses after redirects, then confirm every final robots header and meta directive is free of unintended noindex.
 
Rollback
  1. Restore the prior edge rule if the scope change exposes excluded routes, then replace it with a narrower condition.
 
Caveats
  • Never blanket-delete a valid origin X-Robots-Tag when Cloudflare is not the confirmed emitter.
  • Later response-header rules can override earlier changes, so verify the final rule order and response.
  • Robots.txt is a separate control point from this response-header directive — correct a Disallow rule using the generic-http-html guidance above.
 
Official sources
 
 
Next.js / Vercel
 
Status: supported · Location: App Router metadata/generateMetadata, Pages Router next/head, or next.config headers()
 
Steps
  1. In the App Router, remove the unintended noindex from static metadata.robots or generateMetadata at the route that owns it.
  2. In the Pages Router, correct the owning next/head robots meta; for an X-Robots-Tag, correct the scoped next.config headers() rule.
  3. Deploy to the intended production environment and purge or bypass relevant caches before verification.
  
Verify on this platform
 
  • Inspect all final robots/googlebot meta and X-Robots-Tag values after redirects on the current production deployment.
 
Rollback
  1. Revert the route metadata, Head output, or headers() rule if the deployment exposes a route that should remain excluded.
 
Caveats
  • Vercel intentionally serves X-Robots-Tag: noindex for Preview Deployments and outdated Production Deployments; preserve that protection.
  • Do not diagnose an intentional preview response as a production metadata defect.
  • Robots.txt (a static file or a route handler such as app/robots.ts) is a separate control point from this metadata/header directive — correct a Disallow rule using the generic-http-html guidance above.
 
Official sources
 
 
Platform-managed / ask the vendor
 
Status: platform-managed · Location: Vendor-owned publication, visibility, or response-header control
 
Steps
  1. Ask the vendor to identify the exact global or scoped control that emits noindex and remove it only for the confirmed public canonical URL.
  
Verify on this platform
 
  • Follow redirects and inspect all robots/googlebot meta and X-Robots-Tag values on the final public response, then use the Search Console live test.
 
Rollback
  1. Restore the vendor visibility control if an unintended page becomes public while the owning rule is corrected.
 
Caveats
  • Do not add index to compete with a noindex emitted elsewhere.
  • Keep the URL crawlable long enough for search engines to observe the removal.
  • Ask the vendor to confirm whether it also owns robots.txt for the domain — a Disallow rule there is a separate control point from a meta/header directive.
 
Official sources
 
 
 
 
 How to verify 
 
  1. Follow redirects and confirm a crawlable 200 final response with no noindex in any robots/googlebot meta or X-Robots-Tag value.
  2. Confirm the canonical points to the intended URL and relevant caches no longer serve the directive.
  3. Fetch and parse robots.txt for the applicable user-agent group(s) and confirm the URL and its critical rendering resources are not disallowed.
  4. Rerun SEO Site Audit Crawler, On-Page SEO Checker, Google Index Checker, Render Gap, or Robots.txt Tester (whichever surfaced the finding), and use Search Console URL Inspection’s live test for Google’s current view.
 
Escalate when
 
Escalate when page ownership or publication intent is unclear, noindex or a robots.txt Disallow rule is applied by multiple layers, or removing it would expose private, duplicate, or staging content.
 
 
 
 
 
 
 
robots.txt Tester · one finding
 
Test URL is blocked for the selected crawler
 
Recommended action: Remove or narrow the winning Disallow rule, or add a more specific Allow rule, when the tested URL should be crawlable.
 
Fix principle: Remove every unintended noindex from its confirmed owner; absence of noindex is sufficient, while an explicit index directive cannot override another noindex because restrictive directives combine.
 
 
 
 Implementation for your setup 
 
Confirm first: The final HTML or HTTP response contains a noindex directive, including one added only by client-side script during rendering. The page is intended to be an indexable canonical URL. A robots meta tag or X-Robots-Tag intended to keep the page indexable is neutralized by a conflicting directive elsewhere, because directives combine using the most restrictive rule. Robots.txt disallows the URL itself, or disallows a resource the page depends on to render, so the crawler cannot reach or fully evaluate the intended indexable page.
 
Do not apply when: Exclusion is intentional for the page type, environment, or workflow state. The directive appears only on a noncanonical or otherwise ineligible URL. Robots.txt does not disallow the URL or the resources it needs to render, and the effective combination of every directive already resolves to the intended indexable state.
 
Likely control points: cms-app · server · edge-cdn
 
   
Choose the layer that owns the final response.
 
 
 
Generic HTTP / HTML
 
Status: supported · Location: The application, server, or edge owner that emits the directive, or the robots.txt file at the site root
 
Steps
  1. Follow redirects, identify the exact owner of each unintended robots/googlebot meta or X-Robots-Tag noindex, and remove it only at that owner.
  2. Return the intended canonical page with a crawlable 200 response, align its canonical, and purge the caches that can retain the directive.
  3. If a robots meta tag or X-Robots-Tag is present but rendered ineffective by a conflicting directive elsewhere, remove the actual unwanted restrictive directive rather than adding a permissive one to compete with it — the most restrictive rule always wins. Google also honors a robots meta tag placed outside <head>, so an out-of-head placement is a specification and cross-crawler risk to correct, not by itself proof Google is ignoring the directive.
  4. If robots.txt disallows the URL itself, or disallows a resource (such as CSS or JS) the page needs to render its content, correct only the specific Disallow rule at the robots.txt file that owns it — keep the rule as narrow as its original intent, and remember Disallow blocks crawling, not indexing: a disallowed URL can still be indexed if discovered elsewhere.
  5. Keep the URL crawlable so Google can observe the removal.
  
Verify on this platform
 
  • After redirects and cache purge, inspect every robots and googlebot meta tag plus every X-Robots-Tag value on the final response.
  • Fetch and parse robots.txt for every group that applies to the relevant user-agent — Google consolidates every group matching the same user-agent token into one effective ruleset, though not every crawler does — and confirm the URL and its critical rendering resources are not disallowed.
  • Use Search Console URL Inspection’s live test on the production URL, then request indexing when appropriate.
 
Rollback
  1. Restore the prior directive or robots.txt rule if the page or a resource it needs was exposed or blocked unintentionally, then correct the underlying publication rule before trying again.
 
Caveats
  • Do not add index as a counter-directive: the most restrictive directive applies, so another noindex still wins.
  • The live test checks current accessibility; it does not guarantee selection, canonical choice, or indexing.
  • Google’s own parser combines every robots.txt group matching the same user-agent token into one effective ruleset, but many other crawlers and Robots Exclusion Protocol implementations honor only the first matching group — treat a directive split across groups as a real interoperability risk even when Google itself is unaffected.
  • If the page combines a canonical to a different URL with a noindex directive, remove the noindex only if this URL itself should be indexable; if the canonical target is the one meant to be indexed, keep the noindex here and verify the canonical target separately rather than treating its indexability as confirmed by removing this noindex.
 
Official sources
 
 
WordPress
 
Status: supported · Location: Global Settings → Reading/blog_public, or the scoped plugin/theme callback that owns wp_robots
 
Steps
  1. If WordPress core globally owns the directive, confirm the scope and change Settings → Reading so blog_public permits search-engine visibility.
  2. For a scoped page-type or URL rule, identify and change the plugin or theme callback filtering wp_robots or invoking the core noindex behavior; do not invent a plugin UI.
  3. Purge page, object, host, and edge caches that can retain the old output.
  
Verify on this platform
 
  • While logged out, follow redirects and inspect all robots/googlebot meta plus X-Robots-Tag values on the final public response.
 
Rollback
  1. Restore the previous visibility or plugin setting if unintended pages become indexable.
 
Caveats
  • Do not use the global Reading setting to solve a scoped plugin or theme rule.
  • WordPress exposes the wp_robots API, but a third-party owner’s exact UI is not verified here.
  • Robots.txt is a separate control point from this meta/header directive — correct a Disallow rule using the generic-http-html guidance above regardless of which CMS serves the site.
 
Official sources
 
 
Cloudflare Workers / Rules
 
Status: supported · Location: Confirmed Response Header Transform Rule or Worker emitter
 
Steps
  1. Use Cloudflare Trace and configuration review to confirm which Transform Rule or Worker adds the final X-Robots-Tag and account for response-header rule ordering.
  2. Narrow or remove only that confirmed edge emitter; in a Worker, clone the response before modifying its headers.
  3. Purge the affected cache entries and leave any valid origin-owned robots header intact.
  
Verify on this platform
 
  • Compare origin and final Cloudflare responses after redirects, then confirm every final robots header and meta directive is free of unintended noindex.
 
Rollback
  1. Restore the prior edge rule if the scope change exposes excluded routes, then replace it with a narrower condition.
 
Caveats
  • Never blanket-delete a valid origin X-Robots-Tag when Cloudflare is not the confirmed emitter.
  • Later response-header rules can override earlier changes, so verify the final rule order and response.
  • Robots.txt is a separate control point from this response-header directive — correct a Disallow rule using the generic-http-html guidance above.
 
Official sources
 
 
Next.js / Vercel
 
Status: supported · Location: App Router metadata/generateMetadata, Pages Router next/head, or next.config headers()
 
Steps
  1. In the App Router, remove the unintended noindex from static metadata.robots or generateMetadata at the route that owns it.
  2. In the Pages Router, correct the owning next/head robots meta; for an X-Robots-Tag, correct the scoped next.config headers() rule.
  3. Deploy to the intended production environment and purge or bypass relevant caches before verification.
  
Verify on this platform
 
  • Inspect all final robots/googlebot meta and X-Robots-Tag values after redirects on the current production deployment.
 
Rollback
  1. Revert the route metadata, Head output, or headers() rule if the deployment exposes a route that should remain excluded.
 
Caveats
  • Vercel intentionally serves X-Robots-Tag: noindex for Preview Deployments and outdated Production Deployments; preserve that protection.
  • Do not diagnose an intentional preview response as a production metadata defect.
  • Robots.txt (a static file or a route handler such as app/robots.ts) is a separate control point from this metadata/header directive — correct a Disallow rule using the generic-http-html guidance above.
 
Official sources
 
 
Platform-managed / ask the vendor
 
Status: platform-managed · Location: Vendor-owned publication, visibility, or response-header control
 
Steps
  1. Ask the vendor to identify the exact global or scoped control that emits noindex and remove it only for the confirmed public canonical URL.
  
Verify on this platform
 
  • Follow redirects and inspect all robots/googlebot meta and X-Robots-Tag values on the final public response, then use the Search Console live test.
 
Rollback
  1. Restore the vendor visibility control if an unintended page becomes public while the owning rule is corrected.
 
Caveats
  • Do not add index to compete with a noindex emitted elsewhere.
  • Keep the URL crawlable long enough for search engines to observe the removal.
  • Ask the vendor to confirm whether it also owns robots.txt for the domain — a Disallow rule there is a separate control point from a meta/header directive.
 
Official sources
 
 
 
 
 How to verify 
 
  1. Follow redirects and confirm a crawlable 200 final response with no noindex in any robots/googlebot meta or X-Robots-Tag value.
  2. Confirm the canonical points to the intended URL and relevant caches no longer serve the directive.
  3. Fetch and parse robots.txt for the applicable user-agent group(s) and confirm the URL and its critical rendering resources are not disallowed.
  4. Rerun SEO Site Audit Crawler, On-Page SEO Checker, Google Index Checker, Render Gap, or Robots.txt Tester (whichever surfaced the finding), and use Search Console URL Inspection’s live test for Google’s current view.
 
Escalate when
 
Escalate when page ownership or publication intent is unclear, noindex or a robots.txt Disallow rule is applied by multiple layers, or removing it would expose private, duplicate, or staging content.
 
 
 
 
  
 
 
     
  •  Warnings Directive is misspelled Fix: Correct the directive spelling to a supported name so crawlers do not ignore it. 
  •  Information Crawl-delay is ignored by Google Fix: Remove Crawl-delay for Google and manage Googlebot crawl rate through Search Console or server capacity controls. 
  •  Errors Rule appears before any user-agent group Fix: Move the rule below the User-agent line for the crawler group that should apply it. 
  •  Warnings Line has no colon separator Fix: Add the missing colon between the directive name and value so parsers can read the line. 
  •  Information Unknown directive is ignored Fix: Remove the nonstandard directive or replace it with a supported robots.txt directive that expresses the intended crawl rule. 
  •  Information Crawler group contains no rules Fix: Add the intended Allow or Disallow rules to this group, or remove the group so the crawler can fall through to the wildcard group. 
  •  Errors Rule pattern can never match Fix: Start the path pattern with / or * so matching can begin at the start of the URL path. 
  •  Warnings Dollar anchor appears inside a pattern Fix: Move $ to the end of the pattern if it should anchor the path, or remove it if a literal dollar sign was not intended. 
  •  Errors Sitemap URL is not absolute Fix: Replace the relative sitemap location with a fully qualified URL including https:// and the hostname. 
  •  Errors Robots file exceeds 500 KiB Fix: Reduce robots.txt below 500 KiB by consolidating patterns and removing redundant rules before Google’s truncation point. 
  •  Information Robots file is empty Fix: Leave the file empty if all crawling is intentionally allowed; otherwise add explicit user-agent groups and the minimum required rules. 
    •