Lingue

robots.txt Tester

Free, no signup. robots.txt gives no warning when a misplaced Disallow hides half a site from Google — it just quietly works. Test any list of URLs against it with a matcher ported from Google's own open-source parser — longest-match-wins, wildcards, sub-agent fallback, and status-code semantics handled exactly the way Google does it.

Built by Patrick Stox

1. robots.txt
2. Pages to test
3. Bots

Example data — replace with your own

Runs entirely in your browser — nothing you paste is uploaded or stored. The server is only used to fetch a site's robots.txt or sitemap (browsers can't — no CORS). Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.

Feedback
Report a bug

Found something broken in Robots Txt Tester? Let us know what happened — this goes straight to a private triage queue, not a public list.

What will be sent 
 No tool inputs, uploads, pasted source, complete results, query parameters, or URL fragments are attached automatically. You can edit or remove the selected passage above. Browser and anti-abuse metadata is processed for spam prevention. 
  
 
  
 
 
  
 
 
   
    
 
Sample report Example data — real, verified result
 
Given this robots.txt:
 
 
User-agent: *
Disallow: /search
Allow: /search/howsearchworks
 
 
…then test /search/howsearchworks and /search?q=redirects against Googlebot. The matrix comes back:
  
     
  • Longest match wins, not first match. /search/howsearchworks is
            allowed even though Disallow: /search also matches it — the longer
Allow: /search/howsearchworks on line 3 is more specific and overrides it.
    •  
  • The second URL has no exception. /search?q=redirects is blocked by
Disallow: /search on line 2 — nothing more specific matches it.
    •  
  • Click a verdict to see why. Clicking either cell highlights the exact line that
            decided it — see what "blocked"/"allowed" and the L-numbers mean
and the common-mistakes list below for how order-independent matching trips people up.
    •  
  • Background reading. The robots.txt guideA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.
covers every directive this matcher supports.
    •  
 
Accidental-indexing example: robots.txt hides the noindex
 

A staging template returns a public 200 response with this directive:

 
 
<meta name="robots" content="noindex">
 
 
But the production robots.txt also contains:
 
 
User-agent: *
Disallow: /staging/
 
 

Test /staging/pricing/ and this tool correctly reports it as
blocked. That is evidence about crawling only—not evidence that the URL is
          excluded from search. Because Google cannot fetch the page, it cannot see the
noindex. Links or an old sitemap can still expose the URL for indexing.

 
     
  1. Remove the Disallow: /staging/ rule temporarily while keeping the server-delivered noindex.
    2.  
  2. Rerun this tool and confirm the URL is crawlable, then inspect the live response and request recrawling.
    3.  
  3. After removal is verified, protect private staging environments with authentication—not robots.txt.
    4.  
 
 
 
How to use it
 
     
  1. Load a robots.txt. Enter a domain to fetch the live file, or switch to
Paste to test a draft that isn't published yet. In fetch mode, full URLs on
            other subdomains automatically use that origin’s own /robots.txt.
    2.  
  2. Add the pages to test. Paste URLs (full or bare paths), upload a
.txt/.csv list, or pull them straight from the site's
sitemap.xml.
    3.  
  3. Pick the bots. Choose the crawlers you care about, or hit
AI crawler audit to load the AI bots (GPTBot, ClaudeBot, Google-Extended, and
            the rest) in one click.
    4.  
  4. Read the matrix. Every URL × bot cell shows allowed or blocked. Click a cell
            to see the exact rule that decided it, highlighted in the file.
    5.  
 
 
 
       
     
       
 
How it works
 

The matcher is ported from Google's open-source robots.txt parser, so it decides
          allow/block the way Googlebot actually does — not with a naive "first match wins" rule that most
          testers use. That means longest-match-wins (the most specific
Allow/Disallow path takes precedence regardless of order),
* and $ wildcards, percent-encoding normalisation, and
sub-agent fallback (a bot with no group of its own falls back to
User-agent: *).

 
 Crawl-delay is crawler-specific, not part of this Google matcher. Google does not process it.
Bing documents 1–20 second values.
          This tool reports Google-style Allow/Disallow decisions and never pretends a Crawl-delay line changes Googlebot’s result.

 

All of that runs in your browser. The server is only touched to fetch a live
          robots.txt or sitemap, because browsers can't request them cross-origin. It also models
          robots.txt status-code semantics: most 4xx responses are treated as "no robots.txt,"
          but 429 is a temporary failure like a 5xx or timeout. Temporary failures can initially pause crawling;
          Google may also reuse its last known good file, which a point-in-time tester cannot observe.

 
Features
 
     
  • Google-accurate matching: longest-match-wins, wildcards, sub-agent fallback, status-code rules.
    •  
  • Blocked/allowed matrix across as many URLs and bots as you like.
    •  
  • Click any verdict to see the exact matching line highlighted in the file.
    •  
  • One-click AI crawler audit for the major LLM and AI-search bots.
    •  
  • Built-in linter that flags common robots.txt mistakes.
    •  
  • Fetch by domain, upload a URL list, or pull URLs from a sitemap — including a separate live robots.txt for each entered subdomain (up to 10 origins per run).
    •  
  • Privacy: matching is fully client-side — pasted files never leave your machine.
    •  
 
Common mistakes it catches
 
     
  • Assuming rule order matters — it doesn't for Google; the most specific path wins.
    •  
  • Blocking a resource (CSS/JS) that the page needs to render, then wondering why rankings drop.
    •  
  • Using robots.txt to keep a page out of the index — disallow stops crawling, not indexing; use
NoindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. for that.
    •  
  • A stray Disallow: / under User-agent: * that quietly blocks the whole site.
    •  
 
Limitations
 

This checks Google-style robots.txt matching only — it tells you whether a bot is allowed to crawl a
          URL, not whether the page is indexable (a page can be crawlable and still carry a
noindex). Fetch-by-domain reads the robots.txt exactly as served; if a site returns
          different robots.txt to different user-agents or regions, you'll see the version your fetch
          received. A robots.txt file is scoped to its origin, so fetch mode requests each
          entered subdomain’s own file. Paste mode deliberately uses the one pasted draft for every URL.
          Non-Google crawler behavior is not inferred here; consult that crawler’s own documentation before
          relying on system-specific extensions.

 
Frequently asked questions
 
Does robots.txt stop a page from being indexed?
No. Disallow blocks crawling, not indexing. If other pages link to a disallowed URL, Google can still index it (usually without a snippet). To keep a page out of the index, let it be crawled and add a noindex meta tag or X-Robots-Tag header — which means you must not also disallow it, or Google can never see the noindex.
Does the order of rules in robots.txt matter?
Not for Google. Google applies the most specific rule — the longest matching path — regardless of where it sits in the file, so an Allow can override an earlier Disallow. This tool matches the same way. Some other crawlers use first-match-wins, so keep rules unambiguous.
What is the difference between Disallow and noindex?
Disallow (in robots.txt) tells crawlers not to fetch a URL. noindex (a meta tag or HTTP header on the page) tells them not to keep it in the index. They solve different problems, and combining them backfires: a disallowed page is never fetched, so its noindex is never seen.
Will blocking CSS or JavaScript in robots.txt hurt SEO?
It can. Google renders pages to understand them, so if it cannot fetch the CSS and JS a page needs, it may misjudge the layout, content, or mobile-friendliness. Allow the resources required to render your pages and only block genuinely non-essential paths.
How do I block AI crawlers with robots.txt?
Add a user-agent group for each AI bot — such as GPTBot, Google-Extended, ClaudeBot, CCBot, and PerplexityBot — with Disallow: /. Use the AI crawler audit button to load them and confirm your rules match. Note that robots.txt compliance is voluntary; reputable bots honour it, but it is not an access control.
 
 
 
     
Next steprobots.txt Generator — generate a replacement robots.txt with the directives you meant to ship.

People also explore
ToolLog File AnalyzerToolAI-Crawler Access CheckerToolGooglebot VerifierArticleAI CrawlersArticleUser AgentArticleNuxt SEO
 
 
 
Feature requests for Robots Txt Tester
 

Upvote what you want most. New ideas can be submitted from the floating Feedback menu;
      requests appear here once approved, and the most-wanted rise to the top.

   
Loading…
  
  ➕ Request a feature   
New requests are reviewed before they appear here.
 
     
  
  
 
 
 
 
   
   
Informazioni sullo strumento
Verifica elenchi di URL contro robots.txt con un matcher accurato secondo Google, controlla i crawler AI, analizza gli errori comuni e confronta le modifiche proposte. Il test resta nel browser; il recupero live è limitato alle origini che inserisci.
Funzionalità
  • Matcher Google-accurato con regola vincente e riga esatta per ogni cella URL × crawler.
  • Audit per crawler AI con gruppi GPTBot, ClaudeBot, Google-Extended e altri bot principali.
  • Lint dei file robots.txt, rilevamento dei conflitti sitemap e modalità diff per le modifiche.
  • Recupero per dominio e controllo di più origini, con risultati e limiti dichiarati chiaramente.
Come funziona
Il parser applica la semantica Google: la corrispondenza di percorso più lunga vince, i wildcard vengono normalizzati e il gruppo user-agent effettua il fallback quando necessario. Ogni risultato conserva la regola determinante, mentre il recupero live resta separato dal test del testo incollato.
Limitazioni
  • Il matcher descrive decisioni Allow/Disallow; non misura l’indicizzazione e non trasforma Crawl-delay in un verdetto Google.
  • Le verifiche live usano richieste limitate e dipendono dalle risposte dei domini inseriti; la conformità robots.txt è volontaria e non sostituisce un controllo degli accessi.
Domande frequenti
L’ordine delle regole in robots.txt è importante?
Per Google vince il percorso corrispondente più specifico, indipendentemente dalla posizione nel file. Un Allow può quindi sovrascrivere un Disallow precedente.
Qual è la differenza tra Disallow e noindex?
Disallow impedisce la scansione; noindex comunica di non mantenere la pagina nell’indice. Per vedere noindex, il crawler deve poter recuperare la pagina.
robots.txt impedisce l’indicizzazione?
No. Una pagina vietata può ancora essere indicizzata senza snippet se esistono collegamenti; per escluderla, consenti la scansione e usa noindex.
Bloccare CSS o JavaScript in robots.txt danneggia la SEO?
Può farlo: Google deve recuperare le risorse necessarie per comprendere il layout. Consenti le risorse di rendering e blocca solo percorsi davvero non essenziali.