Languages

AI-Crawler Access Checker

Free, no signup. “Am I letting ChatGPT, Claude, Perplexity, and Google’s AI read this page — and did I mean to?” Enter a URL and get the answer for every major AI crawler, with the exact robots.txt rule that decided it.

Built by Patrick Stox

Five separate tests · registry 2026-07-29

Crawler access is only the first layer

Permission

Is the crawler allowed?

Extractability

Are useful facts present in raw HTML?

Renderability

Does JavaScript successfully expose them?

Operability

Can an agent identify and use semantic controls?

Commerce readiness

Do page, schema, feed and checkout facts agree?

HTML remains primary. llms.txt, UCP, and other protocol files are optional distribution layers and cannot compensate for inaccessible or unextractable HTML.

Validate a protocol response

Upload verified-bot log evidence

Expected columns: crawler, timestamp, url, status, verification. A user-agent string alone is never treated as verified identity.

No log evidence uploaded.

How it works: matching runs in your browser with a matcher ported from Google’s open-source robots.txt parser. The server only fetches the site’s robots.txt and checks for an llms.txt (browsers can’t — CORS).

robots.txt is advisory, not a lock. It tells cooperating crawlers what you prefer — it does not enforce anything. Some crawlers below are reported to ignore it (flagged ⚠). To actually stop a crawler, block it at your CDN/WAF and verify it's genuine by IP.

Checks run from our server; we fetch the URL you enter and don't keep the results. Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.

Feedback
Report a bug

Found something broken in Ai Crawler Checker? Let us know what happened — this goes straight to a private triage queue, not a public list.

What will be sent 
 No tool inputs, uploads, pasted source, complete results, query parameters, or URL fragments are attached automatically. You can edit or remove the selected passage above. Browser and anti-abuse metadata is processed for spam prevention. 
  
 
  
 
 
  
 
 
   
 
 
Sample report
 
Say a site's robots.txt contains:
 
 
User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /private/
 
 
…and you check https://example.com/blog/post. The tool returns a per-crawler verdict like this
          (run through the same matcher this tool ships with, not hand-typed):
 
Example data — captured by running this page's own matcher against the robots.txt above
 
 
 
CrawlerOperatorAccessWinning rule
  
 OAI-SearchBot verifiable
oai-searchbot		 OpenAI Allowed no matching rule → default allow 
 
 ClaudeBot published ranges
claudebot		 Anthropic Allowed no matching rule → default allow 
 
 GPTBot verifiable
gptbot		 OpenAI Blocked Disallow: / (line 2) 
  
 
 
     
  • GPTBot is blocked — its own User-agent: GPTBot group carries
Disallow: /, so this site opts out of OpenAI's model training. See
what Allowed/Blocked mean ↓.
    •  
  • OAI-SearchBot and ClaudeBot are allowed — the User-agent: * group only
            disallows /private/, and /blog/post doesn't match it, so both fall through
            to the default allow rather than matching any rule at all.
    •  
  • Winning rule is shown per bot, not just a pass/error — that's how you'd confirm
            which line to edit if you wanted a different outcome. See how matching
            works ↓.
    •  
  • Training vs. retrieval are separate decisions here: blocking GPTBot (training)
            doesn't touch OAI-SearchBot (the bot that fetches pages to cite in ChatGPT answers) — see the
AI crawlers guide for which bot does which job.
    •  
 
 
 
How to use it
 
     
  1. Paste the page URL you care about — a full address like
https://example.com/some/page. The scheme is optional; a bare host is assumed
https://. The exact path matters, because a Disallow rule can block one
            folder while the rest of the site stays open.
    2.  
  2. Press Check access. The tool fetches that site's robots.txt,
            checks for an llms.txt, and evaluates every crawler in the roster against your URL.
    3.  
  3. Read the two tables — Live retrieval & answer citation and
Model training. Each row shows Allowed or Blocked and the exact rule that decided it.
    4.  
  4. To change your posture, use the robots.txt block generator below the results:
            tick training and/or retrieval, set a path, and copy the generated rules.
    5.  
 
 
 
       
     
      
 
Set your posture → robots.txt block
 
Pick what to block; copy the generated rules into your robots.txt.
 
     
 
   
 
  
Blocking here means “documented posture,” not enforcement — a cooperating crawler will honor it; a hostile one may not.
 
 
Crawler roster last reviewed . Data is hand-maintained and editable (src/data/ai-crawlers.json).
  
 
What the results mean
 
     
  • Allowed (green) — no rule blocks this crawler from this path, so a cooperating
            crawler may fetch it.
    •  
  • Blocked (red) — a Disallow rule in the crawler's group (or the
            wildcard * group) covers this path. The Winning rule column shows the exact
            directive and line number.
    •  
  • Winning rule → "no matching rule → default allow" — nothing in
robots.txt touches this crawler for this path, so it defaults to allowed.
    •  
  • ⚠ ignores robots tag — the operator is reported to ignore robots.txt,
            so an "Allowed"/"Blocked" verdict here describes documented posture, not enforcement.
    •  
  • verifiable / unverifiable tag — whether the operator publishes IP ranges or
            reverse DNS so you could confirm a real hit. "Unverifiable" means the user-agent can be spoofed and
            you can't prove it.
    •  
 
The crawlers are split into Live retrieval & answer citation (bots that fetch
          pages now to answer and cite) and Model training (bots that collect pages to train
          models) so you can set each posture deliberately.
 
How it works
 

Matching runs in your browser with a matcher ported from Google's open-source
robots.txt parser — the same engine behind the site's robots.txt tester and Render Gap
          tool — so verdicts follow Google's real precedence rules (most-specific path wins, not first-match).
          The only server work is a small endpoint that fetches the target's robots.txt and checks
          for an llms.txt, because a browser can't request another domain's files directly (CORS).

 

If robots.txt is missing or returns a 4xx, every crawler defaults to allowed
(Google's behavior). If it returns a 5xx or is unreachable, Google treats the whole site as temporarily
          disallowed. The tool shows amber Not evaluated rows and withholds access verdicts until
          the file can be fetched successfully again.

 
Features
 
     
  • Per-crawler Allowed/Blocked verdict for every major AI crawler, with the exact winning rule and line number.
    •  
  • Crawlers grouped by purpose — live retrieval & citation vs. model training — so you set each posture on purpose.
    •  
  • Flags crawlers reported to ignore robots.txt, and tags each as verifiable or unverifiable by IP (cross-referenced with the Googlebot Verifier roster).
    •  
  • Checks for an llms.txt and reports it honestly (no crawler is documented to read it, so it changes no verdict).
    •  
  • Posture generator: tick training and/or retrieval, set a path, and copy a ready robots.txt block. It does not add a Googlebot disallow because that would prevent Google from crawling affected URLs.
    •  
  • Runs against any public URL — no sign-in, nothing stored; share a checked URL via a copyable link.
    •  
 
Limitations
 
 robots.txt is advisory, not a lock — it tells cooperating crawlers what
          you prefer and enforces nothing. A crawler tagged "⚠ ignores robots" may fetch a "Blocked" page anyway;
          to actually stop it you must block at your CDN/WAF and verify by IP. The roster is hand-maintained, so a
          brand-new crawler may not appear until it's added. And a "verifiable" tag means the operator publishes a
          way to check identity — it does not guarantee any given request is genuine.

 
Frequently asked questions
 
Does blocking GPTBot stop ChatGPT from citing my site?
Not on its own. GPTBot is OpenAI’s training crawler — blocking it opts your content out of future model training, but it does not stop the live-retrieval bots that fetch pages to answer questions and cite sources. Those are OAI-SearchBot (ChatGPT search results and citations) and ChatGPT-User (a page fetched when a user follows a link). To stay out of ChatGPT answers you would have to block those too, which the checker lists separately under "Live retrieval & answer citation."
Can I opt out of Google AI Overviews with robots.txt?
There is no separate AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. crawler, and Google does not offer a full AI-Overviews-only opt-out that keeps the page otherwise unchanged in Search. Blocking GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. prevents crawling but does not by itself guarantee that a known URL disappears from Search. Snippet controls also affect ordinary Search snippets. Google-Extended separately controls Gemini/Vertex training and grounding, not Search or AI Overviews.
Will robots.txt actually stop an AI crawler?
robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. is advisory, not a lock. It tells cooperating crawlers what you prefer; it does not enforce anything. Well-behaved operators honor it, but some crawlers are reported to ignore it — the checker flags those with a "⚠ ignores robots" tag. To actually stop a crawler you have to block it at your CDN or WAF, and verify the request is genuine by IP rather than trusting the user-agent string.
Why does a crawler show as "unverifiable"?
The "verifiable" / "unverifiable" tag reflects whether the operator publishes a way to check a crawler claim — current IP ranges or reverse-DNS plus forward confirmation (FCrDNS), cross-referenced from the Bot Verifier roster. A user-agent alone is always spoofable. Anthropic now publishes an official crawler IP list at claude.com/crawling/bots.json, so Claude traffic can be checked for a published-range match; that is not the same as reverse-DNS confirmation.
Does an llms.txt file change any of these verdicts?
No. llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. is a proposed community convention, and no major AI crawlerAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. is documented to read it, so its presence or absence does not affect whether any crawler is allowed or blocked. The checker reports whether an llms.txt exists at the domain purely for information — every access verdict is decided by robots.txt alone.
 
 
 
      
Next steprobots.txt Generator — generate the corrected version.

People also explore
Toolrobots.txt TesterToolLog File AnalyzerToolGooglebot VerifierArticleAI CrawlersArticleAI Crawler Log AnalysisArticleUser Agent
 
 
 
Feature requests for Ai Crawler Checker
 

Upvote what you want most. New ideas can be submitted from the floating Feedback menu;
      requests appear here once approved, and the most-wanted rise to the top.

   
Loading…
  
  ➕ Request a feature   
New requests are reviewed before they appear here.
 
     
  
  
 
 
 
 
     
 
Where this tool helps
Common use cases
Audit AI crawler access
See whether robots.txt allows or blocks major named AI crawlers and inspect the exact winning rule.
Design a crawler posture
Compare current directives with the access policy the organization actually wants before changing robots.txt.
Troubleshoot an unexpected block
Find a broad or conflicting rule that prevents a crawler from reaching an intended public page.
Check adjacent discovery signals
Confirm whether llms.txt exists while keeping that community proposal separate from documented crawler controls.
Watch the full workflow
AI-Crawler Access Checker walkthrough
Read the transcript

      
      

        
AI-Crawler Access Checker

        

        
A-I providers use different crawlers for search, assistant fetches, and training. I’ll show you when this checker helps, how to test a page, read the verified sample, interpret allowed, blocked, and identity tags, generate a posture, understand enforcement limits, and choose the next verification steps.

      

      

        
Step 1

        

        
This tool evaluates one public page path against a maintained A-I crawler roster and the site’s fetched robots dot t-x-t. It separates search and citation, user-triggered assistant fetches, and model training so you can make distinct choices instead of treating one provider as one bot.

      

      

        
Step 2

        

        
Use it when reviewing A-I access, debugging an unexpected block, separating training from live retrieval, checking a new directory rule, or documenting policy before a launch. Test the exact public U-R-L because a folder-specific directive can differ from the homepage result.

      

      

        
Step 3

        

        
Enter a full page U-R-L and select Check access. The service fetches only the domain’s robots dot t-x-t and checks for l-l-m-s dot t-x-t, while matching runs in the browser. A live lookup is not triggered in this walkthrough; the matcher-verified sample provides honest evidence.

      

      

        
Step 4

        

        
The sample blocks G-P-T-Bot at slash, while the wildcard group blocks only slash private. For slash blog slash post, G-P-T-Bot is blocked for training, but O-A-I SearchBot and ClaudeBot default to allowed. Blocking one OpenAI crawler does not automatically block the others.

      

      

        
Step 5

        

        
Each verdict includes the applied user-agent group, exact winning directive, and line number. No matching rule means default allow. The matcher follows Google-style precedence where the most-specific path wins, not simply the first line. Review this evidence before editing a broad policy.

      

      

        
Step 6

        

        
Allowed means no applicable rule blocks a cooperating crawler from this path. Blocked means a matching Disallow expresses that posture. Neither is proof of a fetch, citation, training, deletion from old datasets, or enforcement. Reported ignores-robots tags make that distinction especially important.

      

      

        
Step 7

        

        
Verifiable means the operator publishes I-P ranges or reverse-D-N-S guidance that can help check a request. Unverifiable means the claim cannot be proven with the available roster evidence. A user-agent string is always spoofable, and a verifiable crawler row does not validate any particular log hit.

      

      

        
Step 8

        

        
The posture generator can block training while leaving retrieval and citation available, block all listed A-I crawlers, or create a path-specific selection. Review every generated user agent before copying. It intentionally does not add a broad Googlebot block that would affect ordinary Search crawling.

      

      

        
Step 9

        

        
There is no separate A-I Overviews crawler or complete Overviews-only robots opt-out that leaves Search unchanged. Google-Extended covers Gemini and Vertex training or grounding, not ordinary Search crawling. Snippet controls can also affect standard Search presentation, so evaluate those consequences separately.

      

      

        
Step 10

        

        
Features include purpose groups, per-crawler verdicts, exact winning rules, identity-verification tags, documented noncompliance flags, informational l-l-m-s presence, a path-aware posture generator, public-U-R-L checks, data freshness, and share links. L-l-m-s presence never changes a robots verdict.

      

      

        
Step 11

        

        
Robots dot t-x-t is advisory and cannot enforce access against hostile clients. The roster and operator documentation can change, caches may delay behavior, failed fetches may create not-evaluated results, and page-level meta or header controls are separate. The checker cannot prove past collection or future compliance.

      

      

        
Step 12

        

        
Record the checked U-R-L, robots fetch state, crawler token, purpose, verdict, winning rule, roster date, and intended business policy. Test representative allowed and blocked paths. Then review server logs, verify suspicious identities, use edge enforcement where required, and recheck after robots, provider, or crawler changes.

      

      

        
Set the policy—then enforce and verify it.

        

        
Publish the reviewed robots policy, test representative paths for every intended crawler purpose, and monitor access logs. Where enforcement matters, verify genuine networks and apply scoped edge controls. Recheck provider documentation and the roster as crawler identities and product behaviors change.

      

    
 
          
 
Common issues & how to fix them
 
     
  •  Error Selected AI crawler is blocked Fix: Remove or narrow the selected AI crawler’s Disallow rule for content you intend it to access, then retest that exact user-agent and path. 
  •  Warning Selected AI crawler is blocked on some paths Fix: Review each blocked path and narrow the crawler-specific Disallow patterns so only intentionally private sections remain excluded. 
  •  Error Wildcard rules block AI crawlers Fix: Adjust the User-agent: * group because its Disallow rules also apply to AI crawlers without a more specific applicable policy. 
  •  Warning Specific and wildcard crawler policies conflict Fix: Consolidate wildcard and crawler-specific groups into an intentional precedence-safe policy, then test representative allowed and blocked paths. 
  •  Information Crawler is not in the current roster Fix: Confirm the bot’s published user-agent and operator documentation before adding a dedicated group or changing access rules. 
    •