Permission
Is the crawler allowed?
AI-Crawler Access Checker
Free, no signup. “Am I letting ChatGPT, Claude, Perplexity, and Google’s AI read this page — and did I mean to?” Enter a URL and get the answer for every major AI crawler, with the exact robots.txt rule that decided it.
Five separate tests · registry 2026-07-29
Crawler access is only the first layer
Extractability
Are useful facts present in raw HTML?
Renderability
Does JavaScript successfully expose them?
Operability
Can an agent identify and use semantic controls?
Commerce readiness
Do page, schema, feed and checkout facts agree?
HTML remains primary. llms.txt, UCP, and other protocol files are optional distribution layers and cannot compensate for inaccessible or unextractable HTML.
Validate a protocol response
Upload verified-bot log evidence
Expected columns: crawler, timestamp, url, status, verification. A user-agent string alone is never treated as verified identity.
No log evidence uploaded.
How it works: matching runs in your browser with a matcher ported from Google’s open-source
robots.txt parser. The server only fetches the site’s robots.txt and checks for an llms.txt (browsers can’t — CORS).
robots.txt is advisory, not a lock. It tells cooperating crawlers what you prefer — it does
not enforce anything. Some crawlers below are reported to ignore it (flagged ⚠). To actually stop a
crawler, block it at your CDN/WAF and verify it's genuine by IP.
Checks run from our server; we fetch the URL you enter and don't keep the results. Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.
Sample report
Say a site's robots.txt contains:
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /private/…and you check https://example.com/blog/post. The tool returns a per-crawler verdict like this
(run through the same matcher this tool ships with, not hand-typed):
Example data — captured by running this page's own matcher against the robots.txt above
| Crawler | Operator | Access | Winning rule |
|---|---|---|---|
OAI-SearchBot verifiableoai-searchbot | OpenAI | Allowed | no matching rule → default allow |
ClaudeBot published rangesclaudebot | Anthropic | Allowed | no matching rule → default allow |
GPTBot verifiablegptbot | OpenAI | Blocked | Disallow: / (line 2) |
- GPTBot is blocked — its own
User-agent: GPTBotgroup carriesDisallow: /, so this site opts out of OpenAI's model training. See what Allowed/Blocked mean ↓. - OAI-SearchBot and ClaudeBot are allowed — the
User-agent: *group only disallows/private/, and/blog/postdoesn't match it, so both fall through to the default allow rather than matching any rule at all. - Winning rule is shown per bot, not just a pass/error — that's how you'd confirm which line to edit if you wanted a different outcome. See how matching works ↓.
- Training vs. retrieval are separate decisions here: blocking GPTBot (training) doesn't touch OAI-SearchBot (the bot that fetches pages to cite in ChatGPT answers) — see the AI crawlers guide for which bot does which job.
How to use it
- Paste the page URL you care about — a full address like
https://example.com/some/page. The scheme is optional; a bare host is assumedhttps://. The exact path matters, because aDisallowrule can block one folder while the rest of the site stays open. - Press Check access. The tool fetches that site's
robots.txt, checks for anllms.txt, and evaluates every crawler in the roster against your URL. - Read the two tables — Live retrieval & answer citation and Model training. Each row shows Allowed or Blocked and the exact rule that decided it.
- To change your posture, use the robots.txt block generator below the results: tick training and/or retrieval, set a path, and copy the generated rules.
+ saves the current site or page. Use ☆ beside any saved site, page, or list to favorite it. Recent check history appears below.
Create a named list
Target filled from your local choices.
Site passport Local context for this saved site
Local data
Saved targets, named lists, and recent check summaries remain only in this browser.
Google AI Overviews: there is no separate AI Overviews crawler and no full AI-Overviews-only opt-out that leaves ordinary Search unchanged. Blocking Googlebot prevents crawling but does not by itself guarantee that a known URL disappears from Search.
nosnippet, max-snippet, and data-nosnippet can limit preview/use but also affect normal Search snippets. Google-Extended only controls Gemini/Vertex training and grounding, not Search or Overviews.
Engine-specific limits: this checker evaluates the currently fetched file. Amazon says its crawlers may use a robots.txt copy cached within the previous 30 days and may behave as if the file is absent when they cannot fetch it; a failed checker fetch is not proof that Amazon saw the same failure. Page-level controls are separate: Apple says
nosnippet can block content from generative-model context without preventing title-based discovery, while Bing documents AI-answer/training effects for noarchive and noindex. Use the page audit or extension to inspect those meta and HTTP-header directives.
Search & answer citation
These fetch pages to answer questions and cite sources. Blocking a cooperating crawler can prevent that crawler from accessing the page, but it does not guarantee absence from every answer or citation.
User-triggered assistant fetches
Model training
These collect pages for model training. A provider's documented robots control governs future crawling; it does not remove content already collected or guarantee deletion from an existing training dataset.
Rate this tool
Set your posture → robots.txt block
Pick what to block; copy the generated rules into your robots.txt.
Blocking here means “documented posture,” not enforcement — a cooperating crawler will honor it; a hostile one may not.
Crawler roster last reviewed . Data is hand-maintained and editable (src/data/ai-crawlers.json).
What the results mean
- Allowed (green) — no rule blocks this crawler from this path, so a cooperating crawler may fetch it.
- Blocked (red) — a
Disallowrule in the crawler's group (or the wildcard*group) covers this path. The Winning rule column shows the exact directive and line number. - Winning rule → "no matching rule → default allow" — nothing in
robots.txttouches this crawler for this path, so it defaults to allowed. - ⚠ ignores robots tag — the operator is reported to ignore
robots.txt, so an "Allowed"/"Blocked" verdict here describes documented posture, not enforcement. - verifiable / unverifiable tag — whether the operator publishes IP ranges or reverse DNS so you could confirm a real hit. "Unverifiable" means the user-agent can be spoofed and you can't prove it.
The crawlers are split into Live retrieval & answer citation (bots that fetch pages now to answer and cite) and Model training (bots that collect pages to train models) so you can set each posture deliberately.
How it works
Matching runs in your browser with a matcher ported from Google's open-source
robots.txt parser — the same engine behind the site's robots.txt tester and Render Gap
tool — so verdicts follow Google's real precedence rules (most-specific path wins, not first-match).
The only server work is a small endpoint that fetches the target's robots.txt and checks
for an llms.txt, because a browser can't request another domain's files directly (CORS).
If robots.txt is missing or returns a 4xx, every crawler defaults to allowed
(Google's behavior). If it returns a 5xx or is unreachable, Google treats the whole site as temporarily
disallowed. The tool shows amber Not evaluated rows and withholds access verdicts until
the file can be fetched successfully again.
Features
- Per-crawler Allowed/Blocked verdict for every major AI crawler, with the exact winning rule and line number.
- Crawlers grouped by purpose — live retrieval & citation vs. model training — so you set each posture on purpose.
- Flags crawlers reported to ignore
robots.txt, and tags each as verifiable or unverifiable by IP (cross-referenced with the Googlebot Verifier roster). - Checks for an
llms.txtand reports it honestly (no crawler is documented to read it, so it changes no verdict). - Posture generator: tick training and/or retrieval, set a path, and copy a ready
robots.txtblock. It does not add a Googlebot disallow because that would prevent Google from crawling affected URLs. - Runs against any public URL — no sign-in, nothing stored; share a checked URL via a copyable link.
Limitations
robots.txt is advisory, not a lock — it tells cooperating crawlers what
you prefer and enforces nothing. A crawler tagged "⚠ ignores robots" may fetch a "Blocked" page anyway;
to actually stop it you must block at your CDN/WAF and verify by IP. The roster is hand-maintained, so a
brand-new crawler may not appear until it's added. And a "verifiable" tag means the operator publishes a
way to check identity — it does not guarantee any given request is genuine.
Frequently asked questions
Does blocking GPTBot stop ChatGPT from citing my site?
Not on its own. GPTBot is OpenAI’s training crawler — blocking it opts your content out of future model training, but it does not stop the live-retrieval bots that fetch pages to answer questions and cite sources. Those are OAI-SearchBot (ChatGPT search results and citations) and ChatGPT-User (a page fetched when a user follows a link). To stay out of ChatGPT answers you would have to block those too, which the checker lists separately under "Live retrieval & answer citation."
Can I opt out of Google AI Overviews with robots.txt?
There is no separate AI Overviews crawler, and Google does not offer a full AI-Overviews-only opt-out that keeps the page otherwise unchanged in Search. Blocking Googlebot prevents crawling but does not by itself guarantee that a known URL disappears from Search. Snippet controls also affect ordinary Search snippets. Google-Extended separately controls Gemini/Vertex training and grounding, not Search or AI Overviews.
Will robots.txt actually stop an AI crawler?
robots.txt is advisory, not a lock. It tells cooperating crawlers what you prefer; it does not enforce anything. Well-behaved operators honor it, but some crawlers are reported to ignore it — the checker flags those with a "⚠ ignores robots" tag. To actually stop a crawler you have to block it at your CDN or WAF, and verify the request is genuine by IP rather than trusting the user-agent string.
Why does a crawler show as "unverifiable"?
The "verifiable" / "unverifiable" tag reflects whether the operator publishes a way to check a crawler claim — current IP ranges or reverse-DNS plus forward confirmation (FCrDNS), cross-referenced from the Bot Verifier roster. A user-agent alone is always spoofable. Anthropic now publishes an official crawler IP list at claude.com/crawling/bots.json, so Claude traffic can be checked for a published-range match; that is not the same as reverse-DNS confirmation.
Does an llms.txt file change any of these verdicts?
No. llms.txt is a proposed community convention, and no major AI crawler is documented to read it, so its presence or absence does not affect whether any crawler is allowed or blocked. The checker reports whether an llms.txt exists at the domain purely for information — every access verdict is decided by robots.txt alone.
Feature requests for Ai Crawler Checker
Upvote what you want most. New ideas can be submitted from the floating Feedback menu; requests appear here once approved, and the most-wanted rise to the top.
You won't be emailed about that request anymore.
Loading…
➕ Request a feature
New requests are reviewed before they appear here.
حول الأداة
انظر ما إذا ChatGPT, Claude, Perplexity, و Google AI يمكن اقرأ واحد صفحة — و ما إذا أنت meant هم إلى — مع ال دقيق Robots.txt قاعدة ذلك حُسم إنه.
مجاني, من دون تسجيل. “Am I letting ChatGPT, Claude, Perplexity, و Google AI اقرأ هذا صفحة — و فعل I يعني إلى?” أدخل واحد عنوان URL و احصل ال إجابة من أجل كل رئيسي AI زاحف, مع ال دقيق Robots.txt قاعدة ذلك حُسم إنه.
الميزات
- Per-crawler Allowed/Blocked حكم من أجل كل رئيسي AI زاحف, مع ال دقيق فائز قاعدة و سطر number.
- زواحف مجمّع بواسطة غرض — حي استرجاع & citation vs. نموذج تدريب — لذلك أنت مجموعة كل وضعية على غرض.
- علامات زواحف مُبلّغ عنه إلى تجاهل Robots.txt, و وسوم كل باعتباره قابل للتحقق أو غير قابل للتحقق بواسطة IP (cross-referenced مع ال Googlebot متحقّق قائمة).
- تحقّقات من أجل واحد llms.txt و يبلّغ إنه بصدق (لا زاحف هو موثق إلى اقرأ إنه, لذلك إنه تغييرات لا حكم).
- وضعية مولّد: tick تدريب و/أو استرجاع, مجموعة واحد مسار, و نسخ واحد جاهز Robots.txt حظر. إنه يفعل ليس أضف واحد Googlebot منع لأن ذلك سيكون يمنع محرك Google من زحف متأثر عناوين عنوان URL.
- تشغيل مقابل أي عام عنوان URL — لا sign-in, لا شيء مخزّن; مشاركة واحد تم التحقق منه عنوان URL via واحد copyable رابط.
كيفية العمل
مطابق تشغيل في الخاص بك متصفح مع واحد مطابق منقول من Google open-source Robots.txt محلل — ال نفس محرك خلف ال الموقع Robots.txt فاحص و تصيير فجوة الأداة — لذلك أحكام اتبع Google فعلي أسبقية قواعد (الأكثر تحديدًا مسار يفوز, ليس أول تطابق). ال فقط خادم عمل هو واحد صغير نقطة نهاية ذلك يجلب ال المستهدف Robots.txt و تحقّقات من أجل واحد llms.txt, لأن واحد متصفح لا يمكنه طلب آخر النطاق ملفات مباشرة (CORS). إذا Robots.txt هو مفقود أو يعيد واحد 4xx, كل زاحف الإعدادات الافتراضية إلى allowed (Google سلوك). إذا إنه يعيد واحد 5xx أو هو غير قابل للوصول, محرك Google يعالج ال كامل موقع باعتباره مؤقتًا ممنوع. ال الأداة يعرض كهرماني ليس مُقيَّم صفوف و يحجب وصول أحكام حتى ال ملف يمكن يكون مجلوب بنجاح مرة أخرى.
القيود
- Robots.txt هو advisory, ليس واحد lock — إنه يخبر متعاون زواحف ما أنت يفضّل و enforces لا شيء. واحد زاحف tagged “⚠ يتجاهل Robots” قد جلب واحد “محظور” صفحة anyway; إلى فعليًا أوقف إنه أنت يجب حظر في الخاص بك CDN/WAF و تحقّق بواسطة IP. ال قائمة هو مُدار يدويًا, لذلك واحد brand-new زاحف قد ليس يظهر حتى إنه مضاف. و واحد قابل للتحقق وسم يعني ال مشغّل ينشر واحد way إلى تحقّق هوية — إنه يفعل ليس يضمن أي معطى طلب هو حقيقي.
الأسئلة الشائعة
يفعل حظر GPTBot أوقف ChatGPT من citing my موقع?
ليس على الخاص به الخاص. GPTBot هو OpenAI’s تدريب زاحف — حظر إنه يختار الخاص بك محتوى خارج من مستقبل نموذج تدريب, لكن إنه يفعل ليس أوقف ال live-retrieval bots ذلك جلب صفحات إلى إجابة أسئلة و استشهد مصادر. تلك هي OAI-SearchBot (ChatGPT بحث نتائج و citations) و ChatGPT-User (واحد صفحة مجلوب عندما واحد مستخدم يتبع واحد رابط). إلى يبقى خارج من ChatGPT إجابات أنت سيكون يحتوي إلى حظر تلك أيضًا, أي ال فاحص قوائم بشكل منفصل تحت "حي استرجاع & إجابة citation."
يمكن I opt خارج من محرك Google AI نظرات عامة مع Robots.txt?
هناك هو لا منفصل AI نظرات عامة زاحف, و محرك Google يفعل ليس يقدّم واحد كامل AI-Overviews-only إلغاء الاشتراك ذلك يحتفظ ال صفحة وإلا دون تغيير في بحث. حظر Googlebot يمنع زحف لكن يفعل ليس بواسطة نفسه يضمن ذلك واحد معروف عنوان URL يختفي من بحث. مقتطف ضوابط أيضًا يؤثر عادي بحث مقتطفات. Google-Extended بشكل منفصل ضوابط Gemini/Vertex تدريب و تأسيس, ليس بحث أو AI نظرات عامة.
سوف Robots.txt فعليًا أوقف واحد AI زاحف?
Robots.txt هو advisory, ليس واحد lock. إنه يخبر متعاون زواحف ما أنت يفضّل; إنه يفعل ليس enforce أي شيء. Well-behaved operators يقرّ إنه, لكن بعض زواحف هي مُبلّغ عنه إلى تجاهل إنه — ال فاحص علامات تلك مع واحد "⚠ يتجاهل Robots" وسم. إلى فعليًا أوقف واحد زاحف أنت يحتوي إلى حظر إنه في الخاص بك CDN أو WAF, و تحقّق ال طلب هو حقيقي بواسطة IP بدلًا من trusting ال user-agent string.
لماذا يفعل واحد زاحف اعرض باعتباره "غير قابل للتحقق"?
ال "قابل للتحقق" / "غير قابل للتحقق" وسم يعكس ما إذا ال مشغّل ينشر واحد way إلى تحقّق واحد زاحف يدّعي — حالي IP نطاقات أو reverse-DNS بالإضافة أمامي تأكيد (FCrDNS), cross-referenced من ال Bot متحقّق قائمة. واحد user-agent بمفرده هو دائمًا spoofable. Anthropic الآن ينشر واحد رسمي زاحف IP قائمة في claude.com/الزحف/bots.json, لذلك Claude زيارات يمكن يكون تم التحقق منه من أجل واحد published-range تطابق; ذلك هو ليس ال نفس باعتباره reverse-DNS تأكيد.
يفعل واحد llms.txt ملف غيّر أي من هذه أحكام?
لا. llms.txt هو واحد مقترح community اصطلاح, و لا رئيسي AI زاحف هو موثق إلى اقرأ إنه, لذلك الخاص به وجود أو غياب يفعل ليس يؤثر ما إذا أي زاحف هو allowed أو محظور. ال فاحص يبلّغ ما إذا واحد llms.txt موجود في ال نطاق بحتًا من أجل معلومات — كل وصول حكم هو حُسم بواسطة Robots.txt بمفرده.