Permission
Is the crawler allowed?
AI-Crawler Access Checker
Free, no signup. “Am I letting ChatGPT, Claude, Perplexity, and Google’s AI read this page — and did I mean to?” Enter a URL and get the answer for every major AI crawler, with the exact robots.txt rule that decided it.
Five separate tests · registry 2026-07-29
Crawler access is only the first layer
Extractability
Are useful facts present in raw HTML?
Renderability
Does JavaScript successfully expose them?
Operability
Can an agent identify and use semantic controls?
Commerce readiness
Do page, schema, feed and checkout facts agree?
HTML remains primary. llms.txt, UCP, and other protocol files are optional distribution layers and cannot compensate for inaccessible or unextractable HTML.
Validate a protocol response
Upload verified-bot log evidence
Expected columns: crawler, timestamp, url, status, verification. A user-agent string alone is never treated as verified identity.
No log evidence uploaded.
How it works: matching runs in your browser with a matcher ported from Google’s open-source
robots.txt parser. The server only fetches the site’s robots.txt and checks for an llms.txt (browsers can’t — CORS).
robots.txt is advisory, not a lock. It tells cooperating crawlers what you prefer — it does
not enforce anything. Some crawlers below are reported to ignore it (flagged ⚠). To actually stop a
crawler, block it at your CDN/WAF and verify it's genuine by IP.
Checks run from our server; we fetch the URL you enter and don't keep the results. Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.
Sample report
Say a site's robots.txt contains:
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /private/…and you check https://example.com/blog/post. The tool returns a per-crawler verdict like this
(run through the same matcher this tool ships with, not hand-typed):
Example data — captured by running this page's own matcher against the robots.txt above
| Crawler | Operator | Access | Winning rule |
|---|---|---|---|
OAI-SearchBot verifiableoai-searchbot | OpenAI | Allowed | no matching rule → default allow |
ClaudeBot published rangesclaudebot | Anthropic | Allowed | no matching rule → default allow |
GPTBot verifiablegptbot | OpenAI | Blocked | Disallow: / (line 2) |
- GPTBot is blocked — its own
User-agent: GPTBotgroup carriesDisallow: /, so this site opts out of OpenAI's model training. See what Allowed/Blocked mean ↓. - OAI-SearchBot and ClaudeBot are allowed — the
User-agent: *group only disallows/private/, and/blog/postdoesn't match it, so both fall through to the default allow rather than matching any rule at all. - Winning rule is shown per bot, not just a pass/error — that's how you'd confirm which line to edit if you wanted a different outcome. See how matching works ↓.
- Training vs. retrieval are separate decisions here: blocking GPTBot (training) doesn't touch OAI-SearchBot (the bot that fetches pages to cite in ChatGPT answers) — see the AI crawlers guide for which bot does which job.
How to use it
- Paste the page URL you care about — a full address like
https://example.com/some/page. The scheme is optional; a bare host is assumedhttps://. The exact path matters, because aDisallowrule can block one folder while the rest of the site stays open. - Press Check access. The tool fetches that site's
robots.txt, checks for anllms.txt, and evaluates every crawler in the roster against your URL. - Read the two tables — Live retrieval & answer citation and Model training. Each row shows Allowed or Blocked and the exact rule that decided it.
- To change your posture, use the robots.txt block generator below the results: tick training and/or retrieval, set a path, and copy the generated rules.
+ saves the current site or page. Use ☆ beside any saved site, page, or list to favorite it. Recent check history appears below.
Create a named list
Target filled from your local choices.
Site passport Local context for this saved site
Local data
Saved targets, named lists, and recent check summaries remain only in this browser.
Google AI Overviews: there is no separate AI Overviews crawler and no full AI-Overviews-only opt-out that leaves ordinary Search unchanged. Blocking Googlebot prevents crawling but does not by itself guarantee that a known URL disappears from Search.
nosnippet, max-snippet, and data-nosnippet can limit preview/use but also affect normal Search snippets. Google-Extended only controls Gemini/Vertex training and grounding, not Search or Overviews.
Engine-specific limits: this checker evaluates the currently fetched file. Amazon says its crawlers may use a robots.txt copy cached within the previous 30 days and may behave as if the file is absent when they cannot fetch it; a failed checker fetch is not proof that Amazon saw the same failure. Page-level controls are separate: Apple says
nosnippet can block content from generative-model context without preventing title-based discovery, while Bing documents AI-answer/training effects for noarchive and noindex. Use the page audit or extension to inspect those meta and HTTP-header directives.
Search & answer citation
These fetch pages to answer questions and cite sources. Blocking a cooperating crawler can prevent that crawler from accessing the page, but it does not guarantee absence from every answer or citation.
User-triggered assistant fetches
Model training
These collect pages for model training. A provider's documented robots control governs future crawling; it does not remove content already collected or guarantee deletion from an existing training dataset.
Rate this tool
Set your posture → robots.txt block
Pick what to block; copy the generated rules into your robots.txt.
Blocking here means “documented posture,” not enforcement — a cooperating crawler will honor it; a hostile one may not.
Crawler roster last reviewed . Data is hand-maintained and editable (src/data/ai-crawlers.json).
What the results mean
- Allowed (green) — no rule blocks this crawler from this path, so a cooperating crawler may fetch it.
- Blocked (red) — a
Disallowrule in the crawler's group (or the wildcard*group) covers this path. The Winning rule column shows the exact directive and line number. - Winning rule → "no matching rule → default allow" — nothing in
robots.txttouches this crawler for this path, so it defaults to allowed. - ⚠ ignores robots tag — the operator is reported to ignore
robots.txt, so an "Allowed"/"Blocked" verdict here describes documented posture, not enforcement. - verifiable / unverifiable tag — whether the operator publishes IP ranges or reverse DNS so you could confirm a real hit. "Unverifiable" means the user-agent can be spoofed and you can't prove it.
The crawlers are split into Live retrieval & answer citation (bots that fetch pages now to answer and cite) and Model training (bots that collect pages to train models) so you can set each posture deliberately.
How it works
Matching runs in your browser with a matcher ported from Google's open-source
robots.txt parser — the same engine behind the site's robots.txt tester and Render Gap
tool — so verdicts follow Google's real precedence rules (most-specific path wins, not first-match).
The only server work is a small endpoint that fetches the target's robots.txt and checks
for an llms.txt, because a browser can't request another domain's files directly (CORS).
If robots.txt is missing or returns a 4xx, every crawler defaults to allowed
(Google's behavior). If it returns a 5xx or is unreachable, Google treats the whole site as temporarily
disallowed. The tool shows amber Not evaluated rows and withholds access verdicts until
the file can be fetched successfully again.
Features
- Per-crawler Allowed/Blocked verdict for every major AI crawler, with the exact winning rule and line number.
- Crawlers grouped by purpose — live retrieval & citation vs. model training — so you set each posture on purpose.
- Flags crawlers reported to ignore
robots.txt, and tags each as verifiable or unverifiable by IP (cross-referenced with the Googlebot Verifier roster). - Checks for an
llms.txtand reports it honestly (no crawler is documented to read it, so it changes no verdict). - Posture generator: tick training and/or retrieval, set a path, and copy a ready
robots.txtblock. It does not add a Googlebot disallow because that would prevent Google from crawling affected URLs. - Runs against any public URL — no sign-in, nothing stored; share a checked URL via a copyable link.
Limitations
robots.txt is advisory, not a lock — it tells cooperating crawlers what
you prefer and enforces nothing. A crawler tagged "⚠ ignores robots" may fetch a "Blocked" page anyway;
to actually stop it you must block at your CDN/WAF and verify by IP. The roster is hand-maintained, so a
brand-new crawler may not appear until it's added. And a "verifiable" tag means the operator publishes a
way to check identity — it does not guarantee any given request is genuine.
Frequently asked questions
Does blocking GPTBot stop ChatGPT from citing my site?
Not on its own. GPTBot is OpenAI’s training crawler — blocking it opts your content out of future model training, but it does not stop the live-retrieval bots that fetch pages to answer questions and cite sources. Those are OAI-SearchBot (ChatGPT search results and citations) and ChatGPT-User (a page fetched when a user follows a link). To stay out of ChatGPT answers you would have to block those too, which the checker lists separately under "Live retrieval & answer citation."
Can I opt out of Google AI Overviews with robots.txt?
There is no separate AI Overviews crawler, and Google does not offer a full AI-Overviews-only opt-out that keeps the page otherwise unchanged in Search. Blocking Googlebot prevents crawling but does not by itself guarantee that a known URL disappears from Search. Snippet controls also affect ordinary Search snippets. Google-Extended separately controls Gemini/Vertex training and grounding, not Search or AI Overviews.
Will robots.txt actually stop an AI crawler?
robots.txt is advisory, not a lock. It tells cooperating crawlers what you prefer; it does not enforce anything. Well-behaved operators honor it, but some crawlers are reported to ignore it — the checker flags those with a "⚠ ignores robots" tag. To actually stop a crawler you have to block it at your CDN or WAF, and verify the request is genuine by IP rather than trusting the user-agent string.
Why does a crawler show as "unverifiable"?
The "verifiable" / "unverifiable" tag reflects whether the operator publishes a way to check a crawler claim — current IP ranges or reverse-DNS plus forward confirmation (FCrDNS), cross-referenced from the Bot Verifier roster. A user-agent alone is always spoofable. Anthropic now publishes an official crawler IP list at claude.com/crawling/bots.json, so Claude traffic can be checked for a published-range match; that is not the same as reverse-DNS confirmation.
Does an llms.txt file change any of these verdicts?
No. llms.txt is a proposed community convention, and no major AI crawler is documented to read it, so its presence or absence does not affect whether any crawler is allowed or blocked. The checker reports whether an llms.txt exists at the domain purely for information — every access verdict is decided by robots.txt alone.
Feature requests for Ai Crawler Checker
Upvote what you want most. New ideas can be submitted from the floating Feedback menu; requests appear here once approved, and the most-wanted rise to the top.
You won't be emailed about that request anymore.
Loading…
➕ Request a feature
New requests are reviewed before they appear here.
Tentang alat
verifiable / unverifiable tag — apakah operator menerbitkan IP ranges atau
halaman sekarang untuk jawaban dan cite) dan Model pelatihan (bot yang kumpulkan halaman untuk latih
robots.txt adalah advisory, tidak lock. Ini tells cooperating perayap apa Anda prefer — ini melakukan
Fitur
- Home › Alat › AI-Perayap Akses Pemeriksa
- way untuk periksa identitas — ini tidak jaminan apa pun given permintaan adalah genuine.
- robots.txt adalah advisory, tidak lock — ini tells cooperating perayap apa
- memeriksa untuk llms.TXT, dan evaluates setiap perayap di roster terhadap Anda URL.
- Anda prefer dan enforces tidak ada. perayap tagged "⚠ mengabaikan robots" mungkin ambil "Diblokir" halaman anyway;
Cara kerja
Berjalan terhadap apa pun publik URL — tidak ada sign-di, tidak ada disimpan; share diperiksa URL melalui copyable tautan. balik DNS so Anda dapat konfirmasi nyata hit. "Unverifiable" berarti pengguna-agen dapat menjadi spoofed dan Perayap grouped oleh tujuan — langsung retrieval & citation vs. model pelatihan — so Anda atur setiap posture pada tujuan. Posture generator: tick pelatihan dan/or retrieval, atur jalur, dan salin ready robots.txt block. Ini tidak tambah Googlebot larang karena yang akan prevent Google dari perayapan terdampak URLs.
Batasan
- Tidak pada sendiri-nya. GPTBot adalah OpenAI’s pelatihan perayap — memblokir ini opts Anda konten keluar dari future model pelatihan, tetapi ini tidak hentikan langsung-retrieval bot yang ambil halaman untuk jawaban pertanyaan dan cite sumber. Itu adalah OAI-SearchBot (ChatGPT pencarian hasil dan citations) dan ChatGPT-Pengguna ( halaman diambil ketika pengguna mengikuti tautan). Untuk stay keluar dari ChatGPT menjawab Anda akan memiliki untuk block itu terlalu, yang pemeriksa daftar secara terpisah di bawah "Langsung retrieval & jawaban citation."
- Di sana tidak ada terpisah AI Overviews perayap, dan Google tidak offer penuh AI-Overviews-hanya opt-keluar yang mempertahankan halaman jika tidak tidak berubah di Pencarian. Memblokir Googlebot prevents perayapan tetapi tidak oleh dirinya sendiri jaminan yang diketahui URL disappears dari Pencarian. Snippet kontrol juga affect ordinary Pencarian snippets. Google-Extended secara terpisah kontrol Gemini/Vertex pelatihan dan grounding, tidak Pencarian atau AI Overviews.
- tidak enforce anything. Beberapa perayap di bawah adalah reported untuk abaikan ini (flagged ⚠). Untuk sebenarnya hentikan
- perayap adalah split menjadi Langsung retrieval & jawaban citation (bot yang ambil
Pertanyaan umum
Untuk perubahan Anda posture, gunakan robots.txt block generator di bawah hasil:
merek-baru perayap mungkin tidak muncul until ini's added. Dan "verifiable" tag berarti operator menerbitkan
Melakukan llms.TXT berkas perubahan apa pun dari ini verdicts?
direktif dan baris number.
Diblokir (red) — Larang aturan di perayap's kelompokkan (atau
models) so Anda dapat atur setiap posture dengan sengaja.
Mengapa melakukan perayap tunjukkan sebagai "unverifiable"?
⚠ mengabaikan robots tag — operator adalah reported untuk abaikan robots.txt,