Langues

robots.txt Tester

Free, no signup. robots.txt gives no warning when a misplaced Disallow hides half a site from Google — it just quietly works. Test any list of URLs against it with a matcher ported from Google's own open-source parser — longest-match-wins, wildcards, sub-agent fallback, and status-code semantics handled exactly the way Google does it.

Built by Patrick Stox

1. robots.txt
2. Pages to test
3. Bots

Example data — replace with your own

Runs entirely in your browser — nothing you paste is uploaded or stored. The server is only used to fetch a site's robots.txt or sitemap (browsers can't — no CORS). Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.

Feedback
Report a bug

Found something broken in Robots Txt Tester? Let us know what happened — this goes straight to a private triage queue, not a public list.

What will be sent 
 No tool inputs, uploads, pasted source, complete results, query parameters, or URL fragments are attached automatically. You can edit or remove the selected passage above. Browser and anti-abuse metadata is processed for spam prevention. 
  
 
  
 
 
  
 
 
   
    
 
Sample report Example data — real, verified result
 
Given this robots.txt:
 
 
User-agent: *
Disallow: /search
Allow: /search/howsearchworks
 
 
…then test /search/howsearchworks and /search?q=redirects against Googlebot. The matrix comes back:
  
     
  • Longest match wins, not first match. /search/howsearchworks is
            allowed even though Disallow: /search also matches it — the longer
Allow: /search/howsearchworks on line 3 is more specific and overrides it.
    •  
  • The second URL has no exception. /search?q=redirects is blocked by
Disallow: /search on line 2 — nothing more specific matches it.
    •  
  • Click a verdict to see why. Clicking either cell highlights the exact line that
            decided it — see what "blocked"/"allowed" and the L-numbers mean
and the common-mistakes list below for how order-independent matching trips people up.
    •  
  • Background reading. The robots.txt guideA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.
covers every directive this matcher supports.
    •  
 
Accidental-indexing example: robots.txt hides the noindex
 

A staging template returns a public 200 response with this directive:

 
 
<meta name="robots" content="noindex">
 
 
But the production robots.txt also contains:
 
 
User-agent: *
Disallow: /staging/
 
 

Test /staging/pricing/ and this tool correctly reports it as
blocked. That is evidence about crawling only—not evidence that the URL is
          excluded from search. Because Google cannot fetch the page, it cannot see the
noindex. Links or an old sitemap can still expose the URL for indexing.

 
     
  1. Remove the Disallow: /staging/ rule temporarily while keeping the server-delivered noindex.
    2.  
  2. Rerun this tool and confirm the URL is crawlable, then inspect the live response and request recrawling.
    3.  
  3. After removal is verified, protect private staging environments with authentication—not robots.txt.
    4.  
 
 
 
How to use it
 
     
  1. Load a robots.txt. Enter a domain to fetch the live file, or switch to
Paste to test a draft that isn't published yet. In fetch mode, full URLs on
            other subdomains automatically use that origin’s own /robots.txt.
    2.  
  2. Add the pages to test. Paste URLs (full or bare paths), upload a
.txt/.csv list, or pull them straight from the site's
sitemap.xml.
    3.  
  3. Pick the bots. Choose the crawlers you care about, or hit
AI crawler audit to load the AI bots (GPTBot, ClaudeBot, Google-Extended, and
            the rest) in one click.
    4.  
  4. Read the matrix. Every URL × bot cell shows allowed or blocked. Click a cell
            to see the exact rule that decided it, highlighted in the file.
    5.  
 
 
 
       
     
       
 
How it works
 

The matcher is ported from Google's open-source robots.txt parser, so it decides
          allow/block the way Googlebot actually does — not with a naive "first match wins" rule that most
          testers use. That means longest-match-wins (the most specific
Allow/Disallow path takes precedence regardless of order),
* and $ wildcards, percent-encoding normalisation, and
sub-agent fallback (a bot with no group of its own falls back to
User-agent: *).

 
 Crawl-delay is crawler-specific, not part of this Google matcher. Google does not process it.
Bing documents 1–20 second values.
          This tool reports Google-style Allow/Disallow decisions and never pretends a Crawl-delay line changes Googlebot’s result.

 

All of that runs in your browser. The server is only touched to fetch a live
          robots.txt or sitemap, because browsers can't request them cross-origin. It also models
          robots.txt status-code semantics: most 4xx responses are treated as "no robots.txt,"
          but 429 is a temporary failure like a 5xx or timeout. Temporary failures can initially pause crawling;
          Google may also reuse its last known good file, which a point-in-time tester cannot observe.

 
Features
 
     
  • Google-accurate matching: longest-match-wins, wildcards, sub-agent fallback, status-code rules.
    •  
  • Blocked/allowed matrix across as many URLs and bots as you like.
    •  
  • Click any verdict to see the exact matching line highlighted in the file.
    •  
  • One-click AI crawler audit for the major LLM and AI-search bots.
    •  
  • Built-in linter that flags common robots.txt mistakes.
    •  
  • Fetch by domain, upload a URL list, or pull URLs from a sitemap — including a separate live robots.txt for each entered subdomain (up to 10 origins per run).
    •  
  • Privacy: matching is fully client-side — pasted files never leave your machine.
    •  
 
Common mistakes it catches
 
     
  • Assuming rule order matters — it doesn't for Google; the most specific path wins.
    •  
  • Blocking a resource (CSS/JS) that the page needs to render, then wondering why rankings drop.
    •  
  • Using robots.txt to keep a page out of the index — disallow stops crawling, not indexing; use
NoindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. for that.
    •  
  • A stray Disallow: / under User-agent: * that quietly blocks the whole site.
    •  
 
Limitations
 

This checks Google-style robots.txt matching only — it tells you whether a bot is allowed to crawl a
          URL, not whether the page is indexable (a page can be crawlable and still carry a
noindex). Fetch-by-domain reads the robots.txt exactly as served; if a site returns
          different robots.txt to different user-agents or regions, you'll see the version your fetch
          received. A robots.txt file is scoped to its origin, so fetch mode requests each
          entered subdomain’s own file. Paste mode deliberately uses the one pasted draft for every URL.
          Non-Google crawler behavior is not inferred here; consult that crawler’s own documentation before
          relying on system-specific extensions.

 
Frequently asked questions
 
Does robots.txt stop a page from being indexed?
No. Disallow blocks crawling, not indexing. If other pages link to a disallowed URL, Google can still index it (usually without a snippet). To keep a page out of the index, let it be crawled and add a noindex meta tag or X-Robots-Tag header — which means you must not also disallow it, or Google can never see the noindex.
Does the order of rules in robots.txt matter?
Not for Google. Google applies the most specific rule — the longest matching path — regardless of where it sits in the file, so an Allow can override an earlier Disallow. This tool matches the same way. Some other crawlers use first-match-wins, so keep rules unambiguous.
What is the difference between Disallow and noindex?
Disallow (in robots.txt) tells crawlers not to fetch a URL. noindex (a meta tag or HTTP header on the page) tells them not to keep it in the index. They solve different problems, and combining them backfires: a disallowed page is never fetched, so its noindex is never seen.
Will blocking CSS or JavaScript in robots.txt hurt SEO?
It can. Google renders pages to understand them, so if it cannot fetch the CSS and JS a page needs, it may misjudge the layout, content, or mobile-friendliness. Allow the resources required to render your pages and only block genuinely non-essential paths.
How do I block AI crawlers with robots.txt?
Add a user-agent group for each AI bot — such as GPTBot, Google-Extended, ClaudeBot, CCBot, and PerplexityBot — with Disallow: /. Use the AI crawler audit button to load them and confirm your rules match. Note that robots.txt compliance is voluntary; reputable bots honour it, but it is not an access control.
 
 
 
     
Next steprobots.txt Generator — generate a replacement robots.txt with the directives you meant to ship.

People also explore
ToolLog File AnalyzerToolAI-Crawler Access CheckerToolGooglebot VerifierArticleAI CrawlersArticleUser AgentArticleNuxt SEO
 
 
 
Feature requests for Robots Txt Tester
 

Upvote what you want most. New ideas can be submitted from the floating Feedback menu;
      requests appear here once approved, and the most-wanted rise to the top.

   
Loading…
  
  ➕ Request a feature   
New requests are reviewed before they appear here.
 
     
  
  
 
 
 
 
   
   
À propos de l’outil
Testez des URL contre robots.txt avec un moteur de correspondance inspiré de l’analyseur open source de Google. Examinez une matrice bloqué/autorisé, la règle gagnante, les erreurs du fichier, les conflits de sitemap et les changements proposés.
Gratuit, sans inscription : les fichiers collés et les essais restent dans votre navigateur.
Fonctionnalités
  • Matrice par robot et par URL avec la règle exacte qui gagne.
  • Lint robots.txt, détection des conflits de sitemap et mode différence.
  • Préréglages Googlebot et robots IA, avec user-agent personnalisé.
  • Récupération séparée du robots.txt en ligne pour chaque origine saisie.
Fonctionnement
Le parseur regroupe les directives User-agent, applique les règles Allow et Disallow selon la priorité du moteur, puis affiche la règle gagnante pour chaque cellule. Les contrôles de sitemap et de récupération en ligne restent séparés afin que chaque preuve soit identifiable.
Limites
  • Le testeur modélise la correspondance d’un fichier à un instant donné ; il ne voit pas les caches, les fichiers précédents de Google ni les changements déployés après le test.
  • Robots.txt contrôle l’exploration, pas l’indexation. Les liens, les sitemaps et les directives noindex peuvent encore exposer une URL.
Questions fréquentes
Un blocage robots.txt prouve-t-il qu’une URL n’est pas indexée ?
Non. Il montre seulement que le robot sélectionné ne peut pas explorer le chemin. Une URL peut rester connue par des liens ou un ancien sitemap.
Google peut-il utiliser un ancien robots.txt ?
Oui. Google peut réutiliser son dernier fichier connu comme valide, ce qu’un test ponctuel ne peut pas observer.
Le testeur récupère-t-il mon site ?
Le mode collé fonctionne localement dans le navigateur. La récupération en ligne est une requête séparée et bornée vers l’origine que vous saisissez.
Crawl-delay est-il respecté par Google ?
Non. Google ignore Crawl-delay ; gérez le débit via Search Console ou la capacité du serveur.
Que signifie une règle gagnante ?
C’est la règle qui décide la cellule pour le robot et le chemin donnés ; elle décrit l’exploration, pas le statut d’indexation.