Permission
Is the crawler allowed?
AI-Crawler Access Checker
Free, no signup. “Am I letting ChatGPT, Claude, Perplexity, and Google’s AI read this page — and did I mean to?” Enter a URL and get the answer for every major AI crawler, with the exact robots.txt rule that decided it.
Five separate tests · registry 2026-07-29
Crawler access is only the first layer
Extractability
Are useful facts present in raw HTML?
Renderability
Does JavaScript successfully expose them?
Operability
Can an agent identify and use semantic controls?
Commerce readiness
Do page, schema, feed and checkout facts agree?
HTML remains primary. llms.txt, UCP, and other protocol files are optional distribution layers and cannot compensate for inaccessible or unextractable HTML.
Validate a protocol response
Upload verified-bot log evidence
Expected columns: crawler, timestamp, url, status, verification. A user-agent string alone is never treated as verified identity.
No log evidence uploaded.
How it works: matching runs in your browser with a matcher ported from Google’s open-source
robots.txt parser. The server only fetches the site’s robots.txt and checks for an llms.txt (browsers can’t — CORS).
robots.txt is advisory, not a lock. It tells cooperating crawlers what you prefer — it does
not enforce anything. Some crawlers below are reported to ignore it (flagged ⚠). To actually stop a
crawler, block it at your CDN/WAF and verify it's genuine by IP.
Checks run from our server; we fetch the URL you enter and don't keep the results. Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.
Sample report
Say a site's robots.txt contains:
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /private/…and you check https://example.com/blog/post. The tool returns a per-crawler verdict like this
(run through the same matcher this tool ships with, not hand-typed):
Example data — captured by running this page's own matcher against the robots.txt above
| Crawler | Operator | Access | Winning rule |
|---|---|---|---|
OAI-SearchBot verifiableoai-searchbot | OpenAI | Allowed | no matching rule → default allow |
ClaudeBot published rangesclaudebot | Anthropic | Allowed | no matching rule → default allow |
GPTBot verifiablegptbot | OpenAI | Blocked | Disallow: / (line 2) |
- GPTBot is blocked — its own
User-agent: GPTBotgroup carriesDisallow: /, so this site opts out of OpenAI's model training. See what Allowed/Blocked mean ↓. - OAI-SearchBot and ClaudeBot are allowed — the
User-agent: *group only disallows/private/, and/blog/postdoesn't match it, so both fall through to the default allow rather than matching any rule at all. - Winning rule is shown per bot, not just a pass/error — that's how you'd confirm which line to edit if you wanted a different outcome. See how matching works ↓.
- Training vs. retrieval are separate decisions here: blocking GPTBot (training) doesn't touch OAI-SearchBot (the bot that fetches pages to cite in ChatGPT answers) — see the AI crawlers guide for which bot does which job.
How to use it
- Paste the page URL you care about — a full address like
https://example.com/some/page. The scheme is optional; a bare host is assumedhttps://. The exact path matters, because aDisallowrule can block one folder while the rest of the site stays open. - Press Check access. The tool fetches that site's
robots.txt, checks for anllms.txt, and evaluates every crawler in the roster against your URL. - Read the two tables — Live retrieval & answer citation and Model training. Each row shows Allowed or Blocked and the exact rule that decided it.
- To change your posture, use the robots.txt block generator below the results: tick training and/or retrieval, set a path, and copy the generated rules.
+ saves the current site or page. Use ☆ beside any saved site, page, or list to favorite it. Recent check history appears below.
Create a named list
Target filled from your local choices.
Site passport Local context for this saved site
Local data
Saved targets, named lists, and recent check summaries remain only in this browser.
Google AI Overviews: there is no separate AI Overviews crawler and no full AI-Overviews-only opt-out that leaves ordinary Search unchanged. Blocking Googlebot prevents crawling but does not by itself guarantee that a known URL disappears from Search.
nosnippet, max-snippet, and data-nosnippet can limit preview/use but also affect normal Search snippets. Google-Extended only controls Gemini/Vertex training and grounding, not Search or Overviews.
Engine-specific limits: this checker evaluates the currently fetched file. Amazon says its crawlers may use a robots.txt copy cached within the previous 30 days and may behave as if the file is absent when they cannot fetch it; a failed checker fetch is not proof that Amazon saw the same failure. Page-level controls are separate: Apple says
nosnippet can block content from generative-model context without preventing title-based discovery, while Bing documents AI-answer/training effects for noarchive and noindex. Use the page audit or extension to inspect those meta and HTTP-header directives.
Search & answer citation
These fetch pages to answer questions and cite sources. Blocking a cooperating crawler can prevent that crawler from accessing the page, but it does not guarantee absence from every answer or citation.
User-triggered assistant fetches
Model training
These collect pages for model training. A provider's documented robots control governs future crawling; it does not remove content already collected or guarantee deletion from an existing training dataset.
Rate this tool
Set your posture → robots.txt block
Pick what to block; copy the generated rules into your robots.txt.
Blocking here means “documented posture,” not enforcement — a cooperating crawler will honor it; a hostile one may not.
Crawler roster last reviewed . Data is hand-maintained and editable (src/data/ai-crawlers.json).
What the results mean
- Allowed (green) — no rule blocks this crawler from this path, so a cooperating crawler may fetch it.
- Blocked (red) — a
Disallowrule in the crawler's group (or the wildcard*group) covers this path. The Winning rule column shows the exact directive and line number. - Winning rule → "no matching rule → default allow" — nothing in
robots.txttouches this crawler for this path, so it defaults to allowed. - ⚠ ignores robots tag — the operator is reported to ignore
robots.txt, so an "Allowed"/"Blocked" verdict here describes documented posture, not enforcement. - verifiable / unverifiable tag — whether the operator publishes IP ranges or reverse DNS so you could confirm a real hit. "Unverifiable" means the user-agent can be spoofed and you can't prove it.
The crawlers are split into Live retrieval & answer citation (bots that fetch pages now to answer and cite) and Model training (bots that collect pages to train models) so you can set each posture deliberately.
How it works
Matching runs in your browser with a matcher ported from Google's open-source
robots.txt parser — the same engine behind the site's robots.txt tester and Render Gap
tool — so verdicts follow Google's real precedence rules (most-specific path wins, not first-match).
The only server work is a small endpoint that fetches the target's robots.txt and checks
for an llms.txt, because a browser can't request another domain's files directly (CORS).
If robots.txt is missing or returns a 4xx, every crawler defaults to allowed
(Google's behavior). If it returns a 5xx or is unreachable, Google treats the whole site as temporarily
disallowed. The tool shows amber Not evaluated rows and withholds access verdicts until
the file can be fetched successfully again.
Features
- Per-crawler Allowed/Blocked verdict for every major AI crawler, with the exact winning rule and line number.
- Crawlers grouped by purpose — live retrieval & citation vs. model training — so you set each posture on purpose.
- Flags crawlers reported to ignore
robots.txt, and tags each as verifiable or unverifiable by IP (cross-referenced with the Googlebot Verifier roster). - Checks for an
llms.txtand reports it honestly (no crawler is documented to read it, so it changes no verdict). - Posture generator: tick training and/or retrieval, set a path, and copy a ready
robots.txtblock. It does not add a Googlebot disallow because that would prevent Google from crawling affected URLs. - Runs against any public URL — no sign-in, nothing stored; share a checked URL via a copyable link.
Limitations
robots.txt is advisory, not a lock — it tells cooperating crawlers what
you prefer and enforces nothing. A crawler tagged "⚠ ignores robots" may fetch a "Blocked" page anyway;
to actually stop it you must block at your CDN/WAF and verify by IP. The roster is hand-maintained, so a
brand-new crawler may not appear until it's added. And a "verifiable" tag means the operator publishes a
way to check identity — it does not guarantee any given request is genuine.
Frequently asked questions
Does blocking GPTBot stop ChatGPT from citing my site?
Not on its own. GPTBot is OpenAI’s training crawler — blocking it opts your content out of future model training, but it does not stop the live-retrieval bots that fetch pages to answer questions and cite sources. Those are OAI-SearchBot (ChatGPT search results and citations) and ChatGPT-User (a page fetched when a user follows a link). To stay out of ChatGPT answers you would have to block those too, which the checker lists separately under "Live retrieval & answer citation."
Can I opt out of Google AI Overviews with robots.txt?
There is no separate AI Overviews crawler, and Google does not offer a full AI-Overviews-only opt-out that keeps the page otherwise unchanged in Search. Blocking Googlebot prevents crawling but does not by itself guarantee that a known URL disappears from Search. Snippet controls also affect ordinary Search snippets. Google-Extended separately controls Gemini/Vertex training and grounding, not Search or AI Overviews.
Will robots.txt actually stop an AI crawler?
robots.txt is advisory, not a lock. It tells cooperating crawlers what you prefer; it does not enforce anything. Well-behaved operators honor it, but some crawlers are reported to ignore it — the checker flags those with a "⚠ ignores robots" tag. To actually stop a crawler you have to block it at your CDN or WAF, and verify the request is genuine by IP rather than trusting the user-agent string.
Why does a crawler show as "unverifiable"?
The "verifiable" / "unverifiable" tag reflects whether the operator publishes a way to check a crawler claim — current IP ranges or reverse-DNS plus forward confirmation (FCrDNS), cross-referenced from the Bot Verifier roster. A user-agent alone is always spoofable. Anthropic now publishes an official crawler IP list at claude.com/crawling/bots.json, so Claude traffic can be checked for a published-range match; that is not the same as reverse-DNS confirmation.
Does an llms.txt file change any of these verdicts?
No. llms.txt is a proposed community convention, and no major AI crawler is documented to read it, so its presence or absence does not affect whether any crawler is allowed or blocked. The checker reports whether an llms.txt exists at the domain purely for information — every access verdict is decided by robots.txt alone.
Feature requests for Ai Crawler Checker
Upvote what you want most. New ideas can be submitted from the floating Feedback menu; requests appear here once approved, and the most-wanted rise to the top.
You won't be emailed about that request anymore.
Loading…
➕ Request a feature
New requests are reviewed before they appear here.
À propos de l’outil
Saisissez une URL pour vérifier, robot par robot, ce que votre robots.txt autorise ou bloque, quelle règle gagne et si un fichier llms.txt est présent.
Fonctionnalités
- Roster documenté de robots de recherche, de citation et d’entraînement.
- Moteur de correspondance robots.txt avec règle gagnante et ligne source affichées.
- Presets pour séparer récupération, citation et entraînement, plus générateur de règles à copier.
- Résultats explicites sur la différence entre posture déclarée et véritable blocage réseau.
Fonctionnement
Le contrôle récupère le robots.txt demandé, applique les règles selon leur priorité réelle pour chaque user-agent, puis affiche le verdict, la directive et la ligne gagnante. Les limites propres aux opérateurs et aux contrôles de page restent séparées.
Limites
- robots.txt est consultatif : un opérateur coopératif peut le respecter, mais un opérateur hostile peut l’ignorer.
- Un user-agent ne prouve pas l’identité d’une requête ; vérifiez les preuves réseau lorsque c’est nécessaire.
- La présence de llms.txt est informative et ne modifie aucun verdict d’accès.
Questions fréquentes
Bloquer GPTBot empêche-t-il ChatGPT de citer mon site ?
Cela peut empêcher un accès coopératif de ce robot, mais ne garantit pas l’absence de votre contenu dans toutes les réponses. Les contrôles de page et les autres robots sont distincts.
Que signifie « non vérifiable » ?
Le registre ne fournit pas de plage IP ou de preuve opérateur suffisante pour ce robot. Le contrôle montre la posture déclarée, pas une identité authentifiée.
llms.txt change-t-il les verdicts ?
Non. Le fichier est signalé pour information seulement ; chaque accès est décidé par robots.txt et par la règle qui correspond au chemin.
Comment bloquer un robot qui ignore robots.txt ?
Utilisez une règle CDN ou WAF et vérifiez l’adresse IP ou l’ASN de la requête. robots.txt ne constitue pas un verrou réseau.