Idiomas

robots.txt Tester

Free, no signup. robots.txt gives no warning when a misplaced Disallow hides half a site from Google — it just quietly works. Test any list of URLs against it with a matcher ported from Google's own open-source parser — longest-match-wins, wildcards, sub-agent fallback, and status-code semantics handled exactly the way Google does it.

Built by Patrick Stox

1. robots.txt
2. Pages to test
3. Bots

Example data — replace with your own

Runs entirely in your browser — nothing you paste is uploaded or stored. The server is only used to fetch a site's robots.txt or sitemap (browsers can't — no CORS). Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.

Feedback
Report a bug

Found something broken in Robots Txt Tester? Let us know what happened — this goes straight to a private triage queue, not a public list.

What will be sent 
 No tool inputs, uploads, pasted source, complete results, query parameters, or URL fragments are attached automatically. You can edit or remove the selected passage above. Browser and anti-abuse metadata is processed for spam prevention. 
  
 
  
 
 
  
 
 
   
    
 
Sample report Example data — real, verified result
 
Given this robots.txt:
 
 
User-agent: *
Disallow: /search
Allow: /search/howsearchworks
 
 
…then test /search/howsearchworks and /search?q=redirects against Googlebot. The matrix comes back:
  
     
  • Longest match wins, not first match. /search/howsearchworks is
            allowed even though Disallow: /search also matches it — the longer
Allow: /search/howsearchworks on line 3 is more specific and overrides it.
    •  
  • The second URL has no exception. /search?q=redirects is blocked by
Disallow: /search on line 2 — nothing more specific matches it.
    •  
  • Click a verdict to see why. Clicking either cell highlights the exact line that
            decided it — see what "blocked"/"allowed" and the L-numbers mean
and the common-mistakes list below for how order-independent matching trips people up.
    •  
  • Background reading. The robots.txt guideA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.
covers every directive this matcher supports.
    •  
 
Accidental-indexing example: robots.txt hides the noindex
 

A staging template returns a public 200 response with this directive:

 
 
<meta name="robots" content="noindex">
 
 
But the production robots.txt also contains:
 
 
User-agent: *
Disallow: /staging/
 
 

Test /staging/pricing/ and this tool correctly reports it as
blocked. That is evidence about crawling only—not evidence that the URL is
          excluded from search. Because Google cannot fetch the page, it cannot see the
noindex. Links or an old sitemap can still expose the URL for indexing.

 
     
  1. Remove the Disallow: /staging/ rule temporarily while keeping the server-delivered noindex.
    2.  
  2. Rerun this tool and confirm the URL is crawlable, then inspect the live response and request recrawling.
    3.  
  3. After removal is verified, protect private staging environments with authentication—not robots.txt.
    4.  
 
 
 
How to use it
 
     
  1. Load a robots.txt. Enter a domain to fetch the live file, or switch to
Paste to test a draft that isn't published yet. In fetch mode, full URLs on
            other subdomains automatically use that origin’s own /robots.txt.
    2.  
  2. Add the pages to test. Paste URLs (full or bare paths), upload a
.txt/.csv list, or pull them straight from the site's
sitemap.xml.
    3.  
  3. Pick the bots. Choose the crawlers you care about, or hit
AI crawler audit to load the AI bots (GPTBot, ClaudeBot, Google-Extended, and
            the rest) in one click.
    4.  
  4. Read the matrix. Every URL × bot cell shows allowed or blocked. Click a cell
            to see the exact rule that decided it, highlighted in the file.
    5.  
 
 
 
       
     
       
 
How it works
 

The matcher is ported from Google's open-source robots.txt parser, so it decides
          allow/block the way Googlebot actually does — not with a naive "first match wins" rule that most
          testers use. That means longest-match-wins (the most specific
Allow/Disallow path takes precedence regardless of order),
* and $ wildcards, percent-encoding normalisation, and
sub-agent fallback (a bot with no group of its own falls back to
User-agent: *).

 
 Crawl-delay is crawler-specific, not part of this Google matcher. Google does not process it.
Bing documents 1–20 second values.
          This tool reports Google-style Allow/Disallow decisions and never pretends a Crawl-delay line changes Googlebot’s result.

 

All of that runs in your browser. The server is only touched to fetch a live
          robots.txt or sitemap, because browsers can't request them cross-origin. It also models
          robots.txt status-code semantics: most 4xx responses are treated as "no robots.txt,"
          but 429 is a temporary failure like a 5xx or timeout. Temporary failures can initially pause crawling;
          Google may also reuse its last known good file, which a point-in-time tester cannot observe.

 
Features
 
     
  • Google-accurate matching: longest-match-wins, wildcards, sub-agent fallback, status-code rules.
    •  
  • Blocked/allowed matrix across as many URLs and bots as you like.
    •  
  • Click any verdict to see the exact matching line highlighted in the file.
    •  
  • One-click AI crawler audit for the major LLM and AI-search bots.
    •  
  • Built-in linter that flags common robots.txt mistakes.
    •  
  • Fetch by domain, upload a URL list, or pull URLs from a sitemap — including a separate live robots.txt for each entered subdomain (up to 10 origins per run).
    •  
  • Privacy: matching is fully client-side — pasted files never leave your machine.
    •  
 
Common mistakes it catches
 
     
  • Assuming rule order matters — it doesn't for Google; the most specific path wins.
    •  
  • Blocking a resource (CSS/JS) that the page needs to render, then wondering why rankings drop.
    •  
  • Using robots.txt to keep a page out of the index — disallow stops crawling, not indexing; use
NoindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. for that.
    •  
  • A stray Disallow: / under User-agent: * that quietly blocks the whole site.
    •  
 
Limitations
 

This checks Google-style robots.txt matching only — it tells you whether a bot is allowed to crawl a
          URL, not whether the page is indexable (a page can be crawlable and still carry a
noindex). Fetch-by-domain reads the robots.txt exactly as served; if a site returns
          different robots.txt to different user-agents or regions, you'll see the version your fetch
          received. A robots.txt file is scoped to its origin, so fetch mode requests each
          entered subdomain’s own file. Paste mode deliberately uses the one pasted draft for every URL.
          Non-Google crawler behavior is not inferred here; consult that crawler’s own documentation before
          relying on system-specific extensions.

 
Frequently asked questions
 
Does robots.txt stop a page from being indexed?
No. Disallow blocks crawling, not indexing. If other pages link to a disallowed URL, Google can still index it (usually without a snippet). To keep a page out of the index, let it be crawled and add a noindex meta tag or X-Robots-Tag header — which means you must not also disallow it, or Google can never see the noindex.
Does the order of rules in robots.txt matter?
Not for Google. Google applies the most specific rule — the longest matching path — regardless of where it sits in the file, so an Allow can override an earlier Disallow. This tool matches the same way. Some other crawlers use first-match-wins, so keep rules unambiguous.
What is the difference between Disallow and noindex?
Disallow (in robots.txt) tells crawlers not to fetch a URL. noindex (a meta tag or HTTP header on the page) tells them not to keep it in the index. They solve different problems, and combining them backfires: a disallowed page is never fetched, so its noindex is never seen.
Will blocking CSS or JavaScript in robots.txt hurt SEO?
It can. Google renders pages to understand them, so if it cannot fetch the CSS and JS a page needs, it may misjudge the layout, content, or mobile-friendliness. Allow the resources required to render your pages and only block genuinely non-essential paths.
How do I block AI crawlers with robots.txt?
Add a user-agent group for each AI bot — such as GPTBot, Google-Extended, ClaudeBot, CCBot, and PerplexityBot — with Disallow: /. Use the AI crawler audit button to load them and confirm your rules match. Note that robots.txt compliance is voluntary; reputable bots honour it, but it is not an access control.
 
 
 
     
Next steprobots.txt Generator — generate a replacement robots.txt with the directives you meant to ship.

People also explore
ToolLog File AnalyzerToolAI-Crawler Access CheckerToolGooglebot VerifierArticleAI CrawlersArticleUser AgentArticleNuxt SEO
 
 
 
Feature requests for Robots Txt Tester
 

Upvote what you want most. New ideas can be submitted from the floating Feedback menu;
      requests appear here once approved, and the most-wanted rise to the top.

   
Loading…
  
  ➕ Request a feature   
New requests are reviewed before they appear here.
 
     
  
  
 
 
 
 
   
   
Sobre a ferramenta
Audite em um clique o acesso de rastreadores de IA, LLMs e bots de pesquisa importantes.
Pesquise um domínio, envie uma lista de URLs ou carregue URLs de um sitemap — incluindo um robots.txt ativo separado para cada subdomínio informado, com até 10 origens por execução.
Recursos
  • Correspondência compatível com o Google: vence a regra mais específica, com curingas, fallback de subagente e regras por código de status.
  • Adicione um grupo de user-agent para cada bot de IA — como GPTBot, Google-Extended, ClaudeBot, CCBot e PerplexityBot — com Disallow: /. Use o botão de auditoria de rastreadores de IA para carregá-los e confirmar a correspondência das regras. A conformidade com robots.txt é voluntária; bots confiáveis a respeitam, mas isso não é controle de acesso.
  • O Google renderiza páginas para entendê-las. Se não conseguir buscar o CSS e o JavaScript necessários, pode interpretar incorretamente o layout, o conteúdo ou a compatibilidade móvel. Permita os recursos necessários para renderização e bloqueie apenas caminhos genuinamente dispensáveis.
  • O robots.txt não avisa quando um Disallow colocado incorretamente oculta metade de um site do Google. Teste as URLs com um comparador baseado no parser do próprio Google.
Como funciona
O comparador usa o parser de robots.txt de código aberto do Google: a regra mais específica vence, com curingas, fallback de subagente e regras por código de status. Ele compara cada URL com cada grupo de user-agent e mostra a regra vencedora, conflitos e avisos de lint.
Limitações
  • A comparação é totalmente executada no navegador: arquivos colados nunca saem da sua máquina.
  • O comportamento de rastreadores que não são do Google não é inferido aqui; consulte a documentação do rastreador antes de interpretar o resultado.
Perguntas frequentes
Qual é a diferença entre Disallow e noindex?
Disallow bloqueia o rastreamento, não a indexação. Se outras páginas vincularem uma URL bloqueada, o Google ainda poderá indexá-la, normalmente sem snippet. Para manter uma página fora do índice, permita o rastreamento e adicione uma meta tag noindex ou um cabeçalho X-Robots-Tag; não bloqueie a URL, ou o Google talvez nunca veja o noindex.
A ordem das regras no robots.txt importa?
Não para o Google. Ele aplica a regra mais específica — o caminho correspondente mais longo — independentemente da posição no arquivo, portanto um Allow pode substituir um Disallow anterior. Esta ferramenta usa a mesma lógica. Alguns outros rastreadores usam a primeira correspondência, então mantenha as regras sem ambiguidade.
O robots.txt impede que uma página seja indexada?
Não. Disallow bloqueia o rastreamento, não a indexação. Se outras páginas vincularem a URL bloqueada, o Google ainda poderá indexá-la, normalmente sem snippet. Para removê-la do índice, permita o rastreamento e use noindex ou X-Robots-Tag sem bloquear a URL.
Bloquear CSS ou JavaScript no robots.txt prejudica o SEO?
Pode. O Google renderiza páginas para entendê-las; se não conseguir buscar o CSS e o JavaScript necessários, poderá interpretar incorretamente layout, conteúdo ou compatibilidade móvel. Permita recursos necessários e bloqueie apenas caminhos dispensáveis.
Como bloquear rastreadores de IA com robots.txt?
Inclua um grupo de user-agent por bot de IA, como GPTBot, Google-Extended, ClaudeBot, CCBot ou PerplexityBot, com Disallow: /. A auditoria confirma como cada regra corresponde; isso não substitui controle de acesso.