Free, no signup. “Am I letting ChatGPT, Claude, Perplexity, and Google’s AI read this page — and did I mean to?” Enter a URL and get the answer for every major AI crawler, with the exact robots.txt rule that decided it.
How it works: matching runs in your browser with a matcher ported from Google’s open-source
robots.txt parser. The server only fetches the site’s robots.txt and checks for an llms.txt (browsers can’t — CORS).
robots.txt is advisory, not a lock. It tells cooperating crawlers what you prefer — it does
not enforce anything. Some crawlers below are reported to ignore it (flagged ⚠). To actually stop a
crawler, block it at your CDN/WAF and verify it's genuine by IP.
AI-crawler roster data last synced Jul 25, 2026.
Checks run from our server; we fetch the URL you enter and don't keep the results. Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.
Feedback
Report a bug
Found something broken in Ai Crawler Checker? Let us know what happened — this goes straight to a private triage queue, not a public list.
…and you check https://example.com/blog/post. The tool returns a per-crawler verdict like this
(run through the same matcher this tool ships with, not hand-typed):
Example data — captured by running this page's own matcher against the robots.txt above
Crawler
Operator
Access
Winning rule
OAI-SearchBotverifiable oai-searchbot
OpenAI
Allowed
no matching rule → default allow
ClaudeBotpublished ranges claudebot
Anthropic
Allowed
no matching rule → default allow
GPTBotverifiable gptbot
OpenAI
Blocked
Disallow: /(line 2)
GPTBot is blocked — its own User-agent: GPTBot group carries
Disallow: /, so this site opts out of OpenAI's model training. See
what Allowed/Blocked mean ↓.
OAI-SearchBot and ClaudeBot are allowed — the User-agent: * group only
disallows /private/, and /blog/post doesn't match it, so both fall through
to the default allow rather than matching any rule at all.
Winning rule is shown per bot, not just a pass/error — that's how you'd confirm
which line to edit if you wanted a different outcome. See how matching
works ↓.
Training vs. retrieval are separate decisions here: blocking GPTBot (training)
doesn't touch OAI-SearchBot (the bot that fetches pages to cite in ChatGPT answers) — see the
AI crawlers guide for which bot does which job.
How to use it
Paste the page URL you care about — a full address like
https://example.com/some/page. The scheme is optional; a bare host is assumed
https://. The exact path matters, because a Disallow rule can block one
folder while the rest of the site stays open.
Press Check access. The tool fetches that site's robots.txt,
checks for an llms.txt, and evaluates every crawler in the roster against your URL.
Read the two tables — Live retrieval & answer citation and
Model training. Each row shows Allowed or Blocked and the exact rule that decided it.
To change your posture, use the robots.txt block generator below the results:
tick training and/or retrieval, set a path, and copy the generated rules.
Current target
Recent checks appear below. Use the star beside any saved site, page, or list to favorite it.
Saved targets, named lists, and recent check summaries remain only in this browser.
Google AI Overviews: there is no separate AI Overviews crawler and no full AI-Overviews-only opt-out that leaves ordinary Search unchanged. Blocking Googlebot prevents crawling but does not by itself guarantee that a known URL disappears from Search. nosnippet, max-snippet, and data-nosnippet can limit preview/use but also affect normal Search snippets. Google-Extended only controls Gemini/Vertex training and grounding, not Search or Overviews.
Engine-specific limits: this checker evaluates the currently fetched file. Amazon says its crawlers may use a robots.txt copy cached within the previous 30 days and may behave as if the file is absent when they cannot fetch it; a failed checker fetch is not proof that Amazon saw the same failure. Page-level controls are separate: Apple says nosnippet can block content from generative-model context without preventing title-based discovery, while Bing documents AI-answer/training effects for noarchive and noindex. Use the page audit or extension to inspect those meta and HTTP-header directives.
Search & answer citation
These fetch pages to answer questions and cite sources. Blocking a cooperating crawler can prevent that crawler from accessing the page, but it does not guarantee absence from every answer or citation.
User-triggered assistant fetches
Model training
These collect pages for model training. A provider's documented robots control governs future crawling; it does not remove content already collected or guarantee deletion from an existing training dataset.
Rate this tool
Set your posture → robots.txt block
Pick what to block; copy the generated rules into your robots.txt.
Blocking here means “documented posture,” not enforcement — a cooperating crawler will honor it; a hostile one may not.
Crawler roster last reviewed . Data is hand-maintained and editable (src/data/ai-crawlers.json).
What the results mean
Allowed (green) — no rule blocks this crawler from this path, so a cooperating
crawler may fetch it.
Blocked (red) — a Disallow rule in the crawler's group (or the
wildcard * group) covers this path. The Winning rule column shows the exact
directive and line number.
Winning rule → "no matching rule → default allow" — nothing in
robots.txt touches this crawler for this path, so it defaults to allowed.
⚠ ignores robots tag — the operator is reported to ignore robots.txt,
so an "Allowed"/"Blocked" verdict here describes documented posture, not enforcement.
verifiable / unverifiable tag — whether the operator publishes IP ranges or
reverse DNS so you could confirm a real hit. "Unverifiable" means the user-agent can be spoofed and
you can't prove it.
The crawlers are split into Live retrieval & answer citation (bots that fetch
pages now to answer and cite) and Model training (bots that collect pages to train
models) so you can set each posture deliberately.
How it works
Matching runs in your browser with a matcher ported from Google's open-source
robots.txt parser — the same engine behind the site's robots.txt tester and Render Gap
tool — so verdicts follow Google's real precedence rules (most-specific path wins, not first-match).
The only server work is a small endpoint that fetches the target's robots.txt and checks
for an llms.txt, because a browser can't request another domain's files directly (CORS).
If robots.txt is missing or returns a 4xx, every crawler defaults to allowed
(Google's behavior). If it returns a 5xx or is unreachable, Google treats the whole site as temporarily
disallowed. The tool shows amber Not evaluated rows and withholds access verdicts until
the file can be fetched successfully again.
Features
Per-crawler Allowed/Blocked verdict for every major AI crawler, with the exact winning rule and line number.
Crawlers grouped by purpose — live retrieval & citation vs. model training — so you set each posture on purpose.
Flags crawlers reported to ignore robots.txt, and tags each as verifiable or unverifiable by IP (cross-referenced with the Googlebot Verifier roster).
Checks for an llms.txt and reports it honestly (no crawler is documented to read it, so it changes no verdict).
Posture generator: tick training and/or retrieval, set a path, and copy a ready robots.txt block. It does not add a Googlebot disallow because that would prevent Google from crawling affected URLs.
Runs against any public URL — no sign-in, nothing stored; share a checked URL via a copyable link.
Limitations
robots.txt is advisory, not a lock — it tells cooperating crawlers what
you prefer and enforces nothing. A crawler tagged "⚠ ignores robots" may fetch a "Blocked" page anyway;
to actually stop it you must block at your CDN/WAF and verify by IP. The roster is hand-maintained, so a
brand-new crawler may not appear until it's added. And a "verifiable" tag means the operator publishes a
way to check identity — it does not guarantee any given request is genuine.
Frequently asked questions
Does blocking GPTBot stop ChatGPT from citing my site?
Not on its own. GPTBot is OpenAI’s training crawler — blocking it opts your content out of future model training, but it does not stop the live-retrieval bots that fetch pages to answer questions and cite sources. Those are OAI-SearchBot (ChatGPT search results and citations) and ChatGPT-User (a page fetched when a user follows a link). To stay out of ChatGPT answers you would have to block those too, which the checker lists separately under "Live retrieval & answer citation."
Can I opt out of Google AI Overviews with robots.txt?
There is no separate AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. crawler, and Google does not offer a full AI-Overviews-only opt-out that keeps the page otherwise unchanged in Search. Blocking GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. prevents crawling but does not by itself guarantee that a known URL disappears from Search. Snippet controls also affect ordinary Search snippets. Google-Extended separately controls Gemini/Vertex training and grounding, not Search or AI Overviews.
Will robots.txt actually stop an AI crawler?
robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. is advisory, not a lock. It tells cooperating crawlers what you prefer; it does not enforce anything. Well-behaved operators honor it, but some crawlers are reported to ignore it — the checker flags those with a "⚠ ignores robots" tag. To actually stop a crawler you have to block it at your CDN or WAF, and verify the request is genuine by IP rather than trusting the user-agent string.
Why does a crawler show as "unverifiable"?
The "verifiable" / "unverifiable" tag reflects whether the operator publishes a way to check a crawler claim — current IP ranges or reverse-DNS plus forward confirmation (FCrDNS), cross-referenced from the Googlebot Verifier roster. A user-agent alone is always spoofable. Anthropic now publishes an official crawler IP list at claude.com/crawling/bots.json, so Claude traffic can be checked for a published-range match; that is not the same as reverse-DNS confirmation.
Does an llms.txt file change any of these verdicts?
No. llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. is a proposed community convention, and no major AI crawlerAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. is documented to read it, so its presence or absence does not affect whether any crawler is allowed or blocked. The checker reports whether an llms.txt exists at the domain purely for information — every access verdict is decided by robots.txt alone.
Upvote what you want most. New ideas can be submitted from the floating Feedback menu;
requests appear here once approved, and the most-wanted rise to the top.
You won't be emailed about that request anymore.
Loading…
➕ Request a feature
New requests are reviewed before they appear here.
Common issues & how to fix them
AI-Crawler Access Checker · one finding
Selected AI crawler is blocked
Recommended action: Remove or narrow the selected AI crawler’s Disallow rule for content you intend it to access, then retest that exact user-agent and path.
Fix principle: Crawler access is a policy decision, not a score optimization. Identify the exact current user-agent token from the operator, decide whether that crawler may fetch each path class, and test the complete robots.txt group selection so wildcard and specific rules do not create an accidental outcome.
Implementation for your setup
Confirm first: A named AI crawler is blocked or partially blocked contrary to the site owner's documented policy. Wildcard and crawler-specific groups produce a result different from the author's intention. Generated robots.txt output blocks search crawlers, exposes sensitive paths, or introduces contradictory group definitions.
Do not apply when: The owner has an explicit allow/block decision for each crawler class and path class, using operator-documented current tokens. A standards-conformant parser selects the intended group and result for representative URLs, with no reliance on rule order or unsupported directives.
Likely control points: server · edge-cdn · cms-app · managed-vendor
Choose the layer that owns the final response.
Generic HTTP / HTML
Status: supported · Location: The single robots.txt source served from the origin root
Steps
Document the business policy first: which crawler operators and product uses are allowed, partially allowed, or blocked, and which path classes contain public versus sensitive material.
Confirm each current crawler token from that operator's official documentation; do not infer policy from a display name or stale third-party list.
Consolidate duplicate groups for the same token and make the wildcard group independently safe. Specific and wildcard groups may not combine the way the author expects.
Use only supported allow/disallow path rules for access control, keep sensitive systems protected by authentication rather than robots.txt, and preserve sitemap declarations as absolute URLs.
Test representative public, blocked, and borderline paths for every named token before publishing.
Verify on this platform
Fetch the production robots.txt and confirm it is reachable, parseable, and the exact reviewed revision.
Evaluate representative URLs with the exact crawler token and a wildcard token using a standards-conformant parser.
Re-run AI-Crawler Access Checker and manually confirm any intentional blocks remain described as policy, not defects.
Rollback
Retain the prior robots.txt revision.
If the new file accidentally blocks important search or service crawlers, restore the prior revision immediately and re-author the affected groups offline.
Caveats
robots.txt is publicly readable and is not an access-control system for confidential data.
Allowing a crawler does not guarantee crawling, training, citation, ranking, or referral traffic.
Different operators and products may use different tokens; a generic “AI bots” rule can overreach.
Record the owner-approved policy and operator-documented current user-agent tokens.
Test each representative path against the exact specific and wildcard group selection.
Fetch the final public file after deployment and rerun the checker.
Keep sensitive content behind authentication regardless of the robots result.
Escalate when
Escalate when the operator token or intended product use cannot be confirmed, generated and custom robots rules conflict, or the requested policy could expose confidential content.
AI-Crawler Access Checker · one finding
Selected AI crawler is blocked on some paths
Recommended action: Review each blocked path and narrow the crawler-specific Disallow patterns so only intentionally private sections remain excluded.
Fix principle: Crawler access is a policy decision, not a score optimization. Identify the exact current user-agent token from the operator, decide whether that crawler may fetch each path class, and test the complete robots.txt group selection so wildcard and specific rules do not create an accidental outcome.
Implementation for your setup
Confirm first: A named AI crawler is blocked or partially blocked contrary to the site owner's documented policy. Wildcard and crawler-specific groups produce a result different from the author's intention. Generated robots.txt output blocks search crawlers, exposes sensitive paths, or introduces contradictory group definitions.
Do not apply when: The owner has an explicit allow/block decision for each crawler class and path class, using operator-documented current tokens. A standards-conformant parser selects the intended group and result for representative URLs, with no reliance on rule order or unsupported directives.
Likely control points: server · edge-cdn · cms-app · managed-vendor
Choose the layer that owns the final response.
Generic HTTP / HTML
Status: supported · Location: The single robots.txt source served from the origin root
Steps
Document the business policy first: which crawler operators and product uses are allowed, partially allowed, or blocked, and which path classes contain public versus sensitive material.
Confirm each current crawler token from that operator's official documentation; do not infer policy from a display name or stale third-party list.
Consolidate duplicate groups for the same token and make the wildcard group independently safe. Specific and wildcard groups may not combine the way the author expects.
Use only supported allow/disallow path rules for access control, keep sensitive systems protected by authentication rather than robots.txt, and preserve sitemap declarations as absolute URLs.
Test representative public, blocked, and borderline paths for every named token before publishing.
Verify on this platform
Fetch the production robots.txt and confirm it is reachable, parseable, and the exact reviewed revision.
Evaluate representative URLs with the exact crawler token and a wildcard token using a standards-conformant parser.
Re-run AI-Crawler Access Checker and manually confirm any intentional blocks remain described as policy, not defects.
Rollback
Retain the prior robots.txt revision.
If the new file accidentally blocks important search or service crawlers, restore the prior revision immediately and re-author the affected groups offline.
Caveats
robots.txt is publicly readable and is not an access-control system for confidential data.
Allowing a crawler does not guarantee crawling, training, citation, ranking, or referral traffic.
Different operators and products may use different tokens; a generic “AI bots” rule can overreach.
Record the owner-approved policy and operator-documented current user-agent tokens.
Test each representative path against the exact specific and wildcard group selection.
Fetch the final public file after deployment and rerun the checker.
Keep sensitive content behind authentication regardless of the robots result.
Escalate when
Escalate when the operator token or intended product use cannot be confirmed, generated and custom robots rules conflict, or the requested policy could expose confidential content.
AI-Crawler Access Checker · one finding
Wildcard rules block AI crawlers
Recommended action: Adjust the User-agent: * group because its Disallow rules also apply to AI crawlers without a more specific applicable policy.
Fix principle: Crawler access is a policy decision, not a score optimization. Identify the exact current user-agent token from the operator, decide whether that crawler may fetch each path class, and test the complete robots.txt group selection so wildcard and specific rules do not create an accidental outcome.
Implementation for your setup
Confirm first: A named AI crawler is blocked or partially blocked contrary to the site owner's documented policy. Wildcard and crawler-specific groups produce a result different from the author's intention. Generated robots.txt output blocks search crawlers, exposes sensitive paths, or introduces contradictory group definitions.
Do not apply when: The owner has an explicit allow/block decision for each crawler class and path class, using operator-documented current tokens. A standards-conformant parser selects the intended group and result for representative URLs, with no reliance on rule order or unsupported directives.
Likely control points: server · edge-cdn · cms-app · managed-vendor
Choose the layer that owns the final response.
Generic HTTP / HTML
Status: supported · Location: The single robots.txt source served from the origin root
Steps
Document the business policy first: which crawler operators and product uses are allowed, partially allowed, or blocked, and which path classes contain public versus sensitive material.
Confirm each current crawler token from that operator's official documentation; do not infer policy from a display name or stale third-party list.
Consolidate duplicate groups for the same token and make the wildcard group independently safe. Specific and wildcard groups may not combine the way the author expects.
Use only supported allow/disallow path rules for access control, keep sensitive systems protected by authentication rather than robots.txt, and preserve sitemap declarations as absolute URLs.
Test representative public, blocked, and borderline paths for every named token before publishing.
Verify on this platform
Fetch the production robots.txt and confirm it is reachable, parseable, and the exact reviewed revision.
Evaluate representative URLs with the exact crawler token and a wildcard token using a standards-conformant parser.
Re-run AI-Crawler Access Checker and manually confirm any intentional blocks remain described as policy, not defects.
Rollback
Retain the prior robots.txt revision.
If the new file accidentally blocks important search or service crawlers, restore the prior revision immediately and re-author the affected groups offline.
Caveats
robots.txt is publicly readable and is not an access-control system for confidential data.
Allowing a crawler does not guarantee crawling, training, citation, ranking, or referral traffic.
Different operators and products may use different tokens; a generic “AI bots” rule can overreach.
Record the owner-approved policy and operator-documented current user-agent tokens.
Test each representative path against the exact specific and wildcard group selection.
Fetch the final public file after deployment and rerun the checker.
Keep sensitive content behind authentication regardless of the robots result.
Escalate when
Escalate when the operator token or intended product use cannot be confirmed, generated and custom robots rules conflict, or the requested policy could expose confidential content.
AI-Crawler Access Checker · one finding
Specific and wildcard crawler policies conflict
Recommended action: Consolidate wildcard and crawler-specific groups into an intentional precedence-safe policy, then test representative allowed and blocked paths.
Fix principle: Crawler access is a policy decision, not a score optimization. Identify the exact current user-agent token from the operator, decide whether that crawler may fetch each path class, and test the complete robots.txt group selection so wildcard and specific rules do not create an accidental outcome.
Implementation for your setup
Confirm first: A named AI crawler is blocked or partially blocked contrary to the site owner's documented policy. Wildcard and crawler-specific groups produce a result different from the author's intention. Generated robots.txt output blocks search crawlers, exposes sensitive paths, or introduces contradictory group definitions.
Do not apply when: The owner has an explicit allow/block decision for each crawler class and path class, using operator-documented current tokens. A standards-conformant parser selects the intended group and result for representative URLs, with no reliance on rule order or unsupported directives.
Likely control points: server · edge-cdn · cms-app · managed-vendor
Choose the layer that owns the final response.
Generic HTTP / HTML
Status: supported · Location: The single robots.txt source served from the origin root
Steps
Document the business policy first: which crawler operators and product uses are allowed, partially allowed, or blocked, and which path classes contain public versus sensitive material.
Confirm each current crawler token from that operator's official documentation; do not infer policy from a display name or stale third-party list.
Consolidate duplicate groups for the same token and make the wildcard group independently safe. Specific and wildcard groups may not combine the way the author expects.
Use only supported allow/disallow path rules for access control, keep sensitive systems protected by authentication rather than robots.txt, and preserve sitemap declarations as absolute URLs.
Test representative public, blocked, and borderline paths for every named token before publishing.
Verify on this platform
Fetch the production robots.txt and confirm it is reachable, parseable, and the exact reviewed revision.
Evaluate representative URLs with the exact crawler token and a wildcard token using a standards-conformant parser.
Re-run AI-Crawler Access Checker and manually confirm any intentional blocks remain described as policy, not defects.
Rollback
Retain the prior robots.txt revision.
If the new file accidentally blocks important search or service crawlers, restore the prior revision immediately and re-author the affected groups offline.
Caveats
robots.txt is publicly readable and is not an access-control system for confidential data.
Allowing a crawler does not guarantee crawling, training, citation, ranking, or referral traffic.
Different operators and products may use different tokens; a generic “AI bots” rule can overreach.
Record the owner-approved policy and operator-documented current user-agent tokens.
Test each representative path against the exact specific and wildcard group selection.
Fetch the final public file after deployment and rerun the checker.
Keep sensitive content behind authentication regardless of the robots result.
Escalate when
Escalate when the operator token or intended product use cannot be confirmed, generated and custom robots rules conflict, or the requested policy could expose confidential content.
InformationCrawler is not in the current rosterFix: Confirm the bot’s published user-agent and operator documentation before adding a dedicated group or changing access rules.
