Permission
Is the crawler allowed?
AI-Crawler Access Checker
Free, no signup. “Am I letting ChatGPT, Claude, Perplexity, and Google’s AI read this page — and did I mean to?” Enter a URL and get the answer for every major AI crawler, with the exact robots.txt rule that decided it.
Five separate tests · registry 2026-07-29
Crawler access is only the first layer
Extractability
Are useful facts present in raw HTML?
Renderability
Does JavaScript successfully expose them?
Operability
Can an agent identify and use semantic controls?
Commerce readiness
Do page, schema, feed and checkout facts agree?
HTML remains primary. llms.txt, UCP, and other protocol files are optional distribution layers and cannot compensate for inaccessible or unextractable HTML.
Validate a protocol response
Upload verified-bot log evidence
Expected columns: crawler, timestamp, url, status, verification. A user-agent string alone is never treated as verified identity.
No log evidence uploaded.
How it works: matching runs in your browser with a matcher ported from Google’s open-source
robots.txt parser. The server only fetches the site’s robots.txt and checks for an llms.txt (browsers can’t — CORS).
robots.txt is advisory, not a lock. It tells cooperating crawlers what you prefer — it does
not enforce anything. Some crawlers below are reported to ignore it (flagged ⚠). To actually stop a
crawler, block it at your CDN/WAF and verify it's genuine by IP.
Checks run from our server; we fetch the URL you enter and don't keep the results. Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.
Sample report
Say a site's robots.txt contains:
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /private/…and you check https://example.com/blog/post. The tool returns a per-crawler verdict like this
(run through the same matcher this tool ships with, not hand-typed):
Example data — captured by running this page's own matcher against the robots.txt above
| Crawler | Operator | Access | Winning rule |
|---|---|---|---|
OAI-SearchBot verifiableoai-searchbot | OpenAI | Allowed | no matching rule → default allow |
ClaudeBot published rangesclaudebot | Anthropic | Allowed | no matching rule → default allow |
GPTBot verifiablegptbot | OpenAI | Blocked | Disallow: / (line 2) |
- GPTBot is blocked — its own
User-agent: GPTBotgroup carriesDisallow: /, so this site opts out of OpenAI's model training. See what Allowed/Blocked mean ↓. - OAI-SearchBot and ClaudeBot are allowed — the
User-agent: *group only disallows/private/, and/blog/postdoesn't match it, so both fall through to the default allow rather than matching any rule at all. - Winning rule is shown per bot, not just a pass/error — that's how you'd confirm which line to edit if you wanted a different outcome. See how matching works ↓.
- Training vs. retrieval are separate decisions here: blocking GPTBot (training) doesn't touch OAI-SearchBot (the bot that fetches pages to cite in ChatGPT answers) — see the AI crawlers guide for which bot does which job.
How to use it
- Paste the page URL you care about — a full address like
https://example.com/some/page. The scheme is optional; a bare host is assumedhttps://. The exact path matters, because aDisallowrule can block one folder while the rest of the site stays open. - Press Check access. The tool fetches that site's
robots.txt, checks for anllms.txt, and evaluates every crawler in the roster against your URL. - Read the two tables — Live retrieval & answer citation and Model training. Each row shows Allowed or Blocked and the exact rule that decided it.
- To change your posture, use the robots.txt block generator below the results: tick training and/or retrieval, set a path, and copy the generated rules.
+ saves the current site or page. Use ☆ beside any saved site, page, or list to favorite it. Recent check history appears below.
Create a named list
Target filled from your local choices.
Site passport Local context for this saved site
Local data
Saved targets, named lists, and recent check summaries remain only in this browser.
Google AI Overviews: there is no separate AI Overviews crawler and no full AI-Overviews-only opt-out that leaves ordinary Search unchanged. Blocking Googlebot prevents crawling but does not by itself guarantee that a known URL disappears from Search.
nosnippet, max-snippet, and data-nosnippet can limit preview/use but also affect normal Search snippets. Google-Extended only controls Gemini/Vertex training and grounding, not Search or Overviews.
Engine-specific limits: this checker evaluates the currently fetched file. Amazon says its crawlers may use a robots.txt copy cached within the previous 30 days and may behave as if the file is absent when they cannot fetch it; a failed checker fetch is not proof that Amazon saw the same failure. Page-level controls are separate: Apple says
nosnippet can block content from generative-model context without preventing title-based discovery, while Bing documents AI-answer/training effects for noarchive and noindex. Use the page audit or extension to inspect those meta and HTTP-header directives.
Search & answer citation
These fetch pages to answer questions and cite sources. Blocking a cooperating crawler can prevent that crawler from accessing the page, but it does not guarantee absence from every answer or citation.
User-triggered assistant fetches
Model training
These collect pages for model training. A provider's documented robots control governs future crawling; it does not remove content already collected or guarantee deletion from an existing training dataset.
Rate this tool
Set your posture → robots.txt block
Pick what to block; copy the generated rules into your robots.txt.
Blocking here means “documented posture,” not enforcement — a cooperating crawler will honor it; a hostile one may not.
Crawler roster last reviewed . Data is hand-maintained and editable (src/data/ai-crawlers.json).
What the results mean
- Allowed (green) — no rule blocks this crawler from this path, so a cooperating crawler may fetch it.
- Blocked (red) — a
Disallowrule in the crawler's group (or the wildcard*group) covers this path. The Winning rule column shows the exact directive and line number. - Winning rule → "no matching rule → default allow" — nothing in
robots.txttouches this crawler for this path, so it defaults to allowed. - ⚠ ignores robots tag — the operator is reported to ignore
robots.txt, so an "Allowed"/"Blocked" verdict here describes documented posture, not enforcement. - verifiable / unverifiable tag — whether the operator publishes IP ranges or reverse DNS so you could confirm a real hit. "Unverifiable" means the user-agent can be spoofed and you can't prove it.
The crawlers are split into Live retrieval & answer citation (bots that fetch pages now to answer and cite) and Model training (bots that collect pages to train models) so you can set each posture deliberately.
How it works
Matching runs in your browser with a matcher ported from Google's open-source
robots.txt parser — the same engine behind the site's robots.txt tester and Render Gap
tool — so verdicts follow Google's real precedence rules (most-specific path wins, not first-match).
The only server work is a small endpoint that fetches the target's robots.txt and checks
for an llms.txt, because a browser can't request another domain's files directly (CORS).
If robots.txt is missing or returns a 4xx, every crawler defaults to allowed
(Google's behavior). If it returns a 5xx or is unreachable, Google treats the whole site as temporarily
disallowed. The tool shows amber Not evaluated rows and withholds access verdicts until
the file can be fetched successfully again.
Features
- Per-crawler Allowed/Blocked verdict for every major AI crawler, with the exact winning rule and line number.
- Crawlers grouped by purpose — live retrieval & citation vs. model training — so you set each posture on purpose.
- Flags crawlers reported to ignore
robots.txt, and tags each as verifiable or unverifiable by IP (cross-referenced with the Googlebot Verifier roster). - Checks for an
llms.txtand reports it honestly (no crawler is documented to read it, so it changes no verdict). - Posture generator: tick training and/or retrieval, set a path, and copy a ready
robots.txtblock. It does not add a Googlebot disallow because that would prevent Google from crawling affected URLs. - Runs against any public URL — no sign-in, nothing stored; share a checked URL via a copyable link.
Limitations
robots.txt is advisory, not a lock — it tells cooperating crawlers what
you prefer and enforces nothing. A crawler tagged "⚠ ignores robots" may fetch a "Blocked" page anyway;
to actually stop it you must block at your CDN/WAF and verify by IP. The roster is hand-maintained, so a
brand-new crawler may not appear until it's added. And a "verifiable" tag means the operator publishes a
way to check identity — it does not guarantee any given request is genuine.
Frequently asked questions
Does blocking GPTBot stop ChatGPT from citing my site?
Not on its own. GPTBot is OpenAI’s training crawler — blocking it opts your content out of future model training, but it does not stop the live-retrieval bots that fetch pages to answer questions and cite sources. Those are OAI-SearchBot (ChatGPT search results and citations) and ChatGPT-User (a page fetched when a user follows a link). To stay out of ChatGPT answers you would have to block those too, which the checker lists separately under "Live retrieval & answer citation."
Can I opt out of Google AI Overviews with robots.txt?
There is no separate AI Overviews crawler, and Google does not offer a full AI-Overviews-only opt-out that keeps the page otherwise unchanged in Search. Blocking Googlebot prevents crawling but does not by itself guarantee that a known URL disappears from Search. Snippet controls also affect ordinary Search snippets. Google-Extended separately controls Gemini/Vertex training and grounding, not Search or AI Overviews.
Will robots.txt actually stop an AI crawler?
robots.txt is advisory, not a lock. It tells cooperating crawlers what you prefer; it does not enforce anything. Well-behaved operators honor it, but some crawlers are reported to ignore it — the checker flags those with a "⚠ ignores robots" tag. To actually stop a crawler you have to block it at your CDN or WAF, and verify the request is genuine by IP rather than trusting the user-agent string.
Why does a crawler show as "unverifiable"?
The "verifiable" / "unverifiable" tag reflects whether the operator publishes a way to check a crawler claim — current IP ranges or reverse-DNS plus forward confirmation (FCrDNS), cross-referenced from the Bot Verifier roster. A user-agent alone is always spoofable. Anthropic now publishes an official crawler IP list at claude.com/crawling/bots.json, so Claude traffic can be checked for a published-range match; that is not the same as reverse-DNS confirmation.
Does an llms.txt file change any of these verdicts?
No. llms.txt is a proposed community convention, and no major AI crawler is documented to read it, so its presence or absence does not affect whether any crawler is allowed or blocked. The checker reports whether an llms.txt exists at the domain purely for information — every access verdict is decided by robots.txt alone.
Feature requests for Ai Crawler Checker
Upvote what you want most. New ideas can be submitted from the floating Feedback menu; requests appear here once approved, and the most-wanted rise to the top.
You won't be emailed about that request anymore.
Loading…
➕ Request a feature
New requests are reviewed before they appear here.
Sobre a ferramenta
Grátis e sem cadastro. “Estou permitindo que o ChatGPT, Claude, Perplexity e a IA do Google leiam esta página — e essa era a minha intenção?” Informe uma URL e obtenha a resposta para cada grande rastreador de IA, com a regra exata do robots.txt que determinou o resultado.
Veja se o ChatGPT, Claude, Perplexity e a IA do Google podem ler uma página — e se você pretendia permitir isso — com a regra exata do robots.txt que decidiu o resultado.
Recursos
- Os rastreadores são divididos entre recuperação ativa e citação de respostas, para bots que buscam páginas agora para responder e citar, e treinamento de modelos, para bots que coletam páginas para treinar modelos.
- Não existe um rastreador separado para as Visões Gerais Criadas por IA, e o Google não oferece uma desativação completa apenas para esse recurso que mantenha a página inalterada na Busca. Bloquear o Googlebot impede o rastreamento, mas não garante por si só que uma URL conhecida desapareça da Busca. Os controles de snippets também afetam os snippets comuns da Busca. O Google-Extended controla separadamente o treinamento e a fundamentação do Gemini/Vertex, não a Busca nem as Visões Gerais Criadas por IA.
- Sinaliza rastreadores relatados como ignorando robots.txt e identifica cada um como verificável ou não verificável por IP, com referência cruzada ao cadastro do Verificador de bots.
- Assim, você pode definir deliberadamente a postura para cada tipo de modelo.
Como funciona
Como funciona: a correspondência é executada no navegador com um comparador adaptado do analisador de robots.txt de código aberto do Google. O servidor apenas busca o robots.txt do site e verifica se existe um llms.txt, pois o navegador não pode solicitar diretamente arquivos de outro domínio por causa do CORS.
Limitações
- robots.txt é uma orientação, não um bloqueio. Ele informa aos rastreadores cooperativos o que você prefere, mas não impõe nada. Alguns rastreadores abaixo são relatados como ignorando o arquivo, e a ferramenta os sinaliza com ⚠. Para realmente impedir um rastreador, bloqueie-o na CDN ou no WAF e verifique pelo IP se a solicitação é genuína.
- O arquivo robots.txt não impõe bloqueios. Operadores responsáveis o respeitam, mas um rastreador hostil pode ignorá-lo; a aplicação efetiva exige controles na CDN ou no WAF e verificação da identidade da solicitação.
Perguntas frequentes
Bloquear o GPTBot impede o ChatGPT de citar meu site?
Não por si só. O GPTBot é o rastreador de treinamento da OpenAI; bloqueá-lo impede que o conteúdo seja coletado para futuros treinamentos, mas não bloqueia os bots de recuperação ativa que buscam páginas para responder perguntas e citar fontes. Esses bots são o OAI-SearchBot, usado nos resultados e citações da busca do ChatGPT, e o ChatGPT-User, usado quando um usuário abre um link. Para não aparecer nas respostas do ChatGPT, você também teria de bloquear esses bots, listados separadamente em “Recuperação ativa e citação de respostas”.
Posso desativar as Visões Gerais Criadas por IA do Google com robots.txt?
Não existe um rastreador separado para as Visões Gerais Criadas por IA, e o Google não oferece uma desativação completa apenas para esse recurso que mantenha a página inalterada na Busca. Bloquear o Googlebot impede o rastreamento, mas não garante por si só que uma URL conhecida desapareça da Busca. Os controles de snippets também afetam os snippets comuns da Busca. O Google-Extended controla separadamente o treinamento e a fundamentação do Gemini/Vertex, não a Busca nem as Visões Gerais Criadas por IA.
O robots.txt realmente impede um rastreador de IA?
robots.txt é uma orientação, não um bloqueio. Ele informa aos rastreadores cooperativos o que você prefere, mas não impõe nada. Alguns rastreadores abaixo são relatados como ignorando o arquivo, e a ferramenta os sinaliza com ⚠. Para realmente impedir um rastreador, bloqueie-o na CDN ou no WAF e verifique pelo IP se a solicitação é genuína.
Por que um rastreador aparece como “não verificável”?
A tag “verificável” ou “não verificável” indica se o operador publica uma forma de confirmar a identidade do rastreador: intervalos de IP atuais ou DNS reverso com confirmação direta (FCrDNS), com referência cruzada ao cadastro do Verificador de bots. Um user-agent isolado sempre pode ser falsificado. A Anthropic publica uma lista oficial de IPs de rastreadores em claude.com/crawling/bots.json; assim, o tráfego do Claude pode ser comparado aos intervalos publicados, o que não equivale à confirmação por DNS reverso.
Um arquivo llms.txt altera algum desses vereditos?
Não. llms.txt é uma convenção proposta pela comunidade, e nenhum grande rastreador de IA está documentado como leitor desse arquivo. Sua presença ou ausência não muda se um rastreador é permitido ou bloqueado. A ferramenta relata a existência de llms.txt no domínio apenas como informação; todos os vereditos de acesso são determinados exclusivamente pelo robots.txt.