AI Crawlers

OpenAI publishes separate controls for its training, search, and user-action botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. Evidence for this claim OpenAI publishes separate controls for GPTBot, OAI-SearchBot, and ChatGPT-User, and says each setting is independent. Scope: OpenAI-declared crawlers and user agents; their purposes and robots behavior are platform-specific. Confidence: high · Verified: 2026-07-14 OpenAI: Crawlers Anthropic likewise documents distinct training, search, and user-request agents. Evidence for this claim Anthropic documents ClaudeBot, Claude-SearchBot, and Claude-User separately and says its bots honor robots.txt directives. Scope: Anthropic-declared bots; user-initiated retrieval has additional product-specific conditions. Confidence: high · Verified: 2026-07-14 Anthropic: Bot information

Tap to expand

The chart compares request trends for the five most active AI bots observed by Cloudflare during the selected four-week period, including Googlebot, ClaudeBot, Meta-ExternalAgent, GPTBot, and Bingbot.

TL;DR — “AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls.” isn’t one thing — it’s three. Some bots scrape your site to train AI models (like GPTBot and ClaudeBot). Some build an indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. so AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. can cite you (like OAI-SearchBot and PerplexityBot). And some fetch a page live because a person just asked a question (like ChatGPT-User). Each one has its own name in your robots.txt, so you can allow or block them separately. Blocking the training bots won’t hurt your Google rankings — blocking the AI-search bots can hide you from AI answers.

What AI crawlers are

You already know about search crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — a crawler like GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. or BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. that visits your pages so they can show up in search. AI crawlers are the newer crowd: bots run by AI companies (OpenAI, Anthropic, Google, Apple, Perplexity, and others) that fetch your pages for their purposes.

The key thing to understand before you block anything: they don’t all do the same job. There are three kinds.

1. Training bots read your pages to help build and improve AI models. GPTBot (OpenAI) and ClaudeBot (Anthropic) are the famous ones.
2. AI-search bots read your pages to build a searchable index, so when someone asks ChatGPT or Perplexity a question, it can answer and link to you. OAI-SearchBot and PerplexityBot do this.
3. User-triggered fetchers grab one specific page in real time because a person asked something. ChatGPT-User is an example — it’s not roaming the web on its own, it’s running an errand for a user.

Why the difference matters

Each bot has its own name (its user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target.), and your robots.txt file can allow or block each one by name. So you get to decide, separately:

• Do I mind AI companies training on my content? → that’s the training bots.
• Do I want to show up in ChatGPT and Perplexity answers? → that’s the AI-search bots. Most people want to keep these.

Here’s the big myth to kill: blocking AI bots does not hurt your Google rankings. Google has said so directly, and publishers who block AI training bots keep ranking fine. But if you block the AI-search bots, you can disappear from AI answers — so don’t block the wrong category by accident.

The names you’ll see in your logs

• GPTBot — OpenAI, training.
• OAI-SearchBot — OpenAI, the one that powers ChatGPT search.
• ChatGPT-User — OpenAI, live fetch when a user asks.
• ClaudeBot — Anthropic, training.
• PerplexityBot — Perplexity, search/citations.
• CCBot — Common Crawl, a non-profit whose data trains lots of AI models.
• Bytespider — ByteDance (TikTok’s parent), training.
• Applebot — Apple, powers Siri/Spotlight and Apple Intelligence.

Two more you’ll see — Google-Extended and Applebot-Extended — are not actual crawlers. They’re just switches in robots.txt that say “don’t use my content to train your AI.” They don’t fetch anything.

Want the full bot-by-bot table, the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. recipes, the “do they actually obey robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.?” debate, and whether to block them at all? Switch to the Advanced tab.

OpenAI says the settings for GPTBot, OAI-SearchBot, and ChatGPT-User are independent. Evidence for this claim OpenAI publishes separate controls for GPTBot, OAI-SearchBot, and ChatGPT-User, and says each setting is independent. Scope: OpenAI-declared crawlers and user agents; their purposes and robots behavior are platform-specific. Confidence: high · Verified: 2026-07-14 OpenAI: Crawlers Anthropic separately identifies ClaudeBot, Claude-SearchBot, and Claude-User and documents robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. handling. Evidence for this claim Anthropic documents ClaudeBot, Claude-SearchBot, and Claude-User separately and says its bots honor robots.txt directives. Scope: Anthropic-declared bots; user-initiated retrieval has additional product-specific conditions. Confidence: high · Verified: 2026-07-14 Anthropic: Bot information

TL;DR — AI crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. split into three categories, and conflating them is the mistake almost everyone makes. Training bots (GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot, Meta-ExternalAgent) scrape to build model corpora. AI-search bots (OAI-SearchBot, PerplexityBot, Applebot, Amzn-SearchBot) build a citing indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. User-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User) grab a page live when a person asks. Each has its own user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. and its own robots.txt rule, so blocking GPTBot doesn’t touch OAI-SearchBot. Google-Extended and Applebot-Extended are tokensA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory., not bots — they don’t crawl, they only opt you out of training/groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it.. Blocking training bots has no effect on Google rankings (Google says so, and empirical data agrees); blocking AI-search bots does reduce your visibility in AILLM visibility (or AI visibility) is the aggregate measure of how often and how prominently a brand or page shows up in AI-generated answers — across AI Overviews, ChatGPT, Perplexity, Copilot, and Gemini. It's the AI-search analog of organic visibility, but it's driven by different signals. answers. The robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. debate is real for user fetchers (operators say it “may not apply”) and for Perplexity (reported stealth crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.). llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. is a proposal, largely unread, not adopted by the majors.

The three categories — the spine of this whole topic

Every named AI bot belongs in one of three buckets. Get the bucket right and every downstream decision (block it? allow it? what happens to my SEO?) falls out of it.

1. Training / data-collection crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. These crawl at scale to build corpora for training and fine-tuning LLMs. They’re the bots most publishers want to block, and disallowing them has no effect on search rankings. Members: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance), Amazonbot (Amazon), Meta-ExternalAgent (Meta), AI2Bot (Allen Institute), cohere-training-data-crawler (Cohere), Diffbot, Timpibot (Timpi), webzio-extended (Webz.io). Plus two control tokens that are NOT separate crawlers — Google-Extended and Applebot-Extended — covered below.

2. AI-search / indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. crawlers. These build a live index used to answer queries with citations and links back to you. Blocking these can remove you from AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. results. Members: OAI-SearchBot (OpenAI — distinct from GPTBot), PerplexityBot (Perplexity), Applebot (Apple — powers Spotlight/Siri/Safari suggestions), Amzn-SearchBot (Amazon).

3. User-triggered / on-demand fetchers. These fetch a specific page live because a user asked a question, not because a bot is roaming. Operators typically claim robots.txt doesn’t bind a user-initiated fetch — contested territory. Members: ChatGPT-User (OpenAI), Perplexity-User (Perplexity), Claude-User and Claude-SearchBot (Anthropic), Amzn-User (Amazon, for Alexa), Meta-ExternalFetcher (Meta).

The full table with user-agent tokens and robots.txt compliance is in the Cheat Sheets tab — that table is the centerpiece of this article.

The “token, not bot” trap (Google-Extended and Applebot-Extended)

This is the single most misunderstood fact in the space, so I’ll be precise. Google-Extended and Applebot-Extended do not crawl anything. They have no independent user-agent and generate no crawl traffic. They are robots.txt control tokens that change what existing crawls are allowed to do with your content.

• Google-Extended controls use for improving Gemini Apps and Vertex AI generative models, including groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. outside Google Search. It does not control Google Search’s AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. or AI Mode; those use GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. access. Google is explicit: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” Disallowing it does nothing to Googlebot or your rankings.
• Applebot-Extended is the same idea for Apple. Apple states: “Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results.” It opts you out of training Apple’s foundation models (Apple Intelligence) — your pages still appear in Spotlight/Siri via the real crawler, Applebot.

So Disallow: Google-Extended and Disallow: Applebot-Extended are product-use controls rather than independent crawl blocks. Their published scopes do not remove pages from the corresponding search index, but they are not interchangeable and should not both be shortened to “training bot.”

The major operators, bot by bot

OpenAI runs four declared bots, each independent: GPTBot (training — “used to make our generative AI foundation models more useful and safe”), OAI-SearchBot (search — “used to surface websites in search results in ChatGPT’s search features”), ChatGPT-User (user fetcher — “used for certain user actions in ChatGPT”), and OAI-AdsBot (an independent ads scope, not training and not the search-index crawler). OpenAI’s policy line is the model for the whole space: “Each bot setting is independent, allowing webmasters granular control over how their content interacts with different OpenAI systems.” Disallowing GPTBot “signals that future data from these materials should be excluded from model training datasets”; disallowing OAI-SearchBot means your site “won’t appear in ChatGPT search results.”

Anthropic runs three: ClaudeBot (training), Claude-SearchBot (search indexing), and Claude-User (user fetch). Anthropic states its bots “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt”. Its current support page also points to a shared published IP list for Anthropic-specific verification. That provider-specific list is evidence of identity, not a universal IP or reverse-DNS verification rule for every bot.

Google — the key fact is the one above: Google-Extended is a token, not a bot. Google also runs Google-CloudVertexBot (crawls for site-owner-requested Vertex AI Agents; doesn’t affect ranking) and user-triggered agents like Gemini-Deep-Research and Google-NotebookLM.

Apple runs Applebot (the real crawler powering Spotlight/Siri/Safari suggestions, and “may also be used to help train Apple foundation models”) and the Applebot-Extended token. Blocking Applebot removes you from Apple search; blocking Applebot-Extended only opts out of training.

Perplexity declares two bots: PerplexityBot (search/citations — explicitly “not used to crawl content for AI foundation models”, so Perplexity has no declared training crawler) and Perplexity-User (user fetch). Its docs state Perplexity-User “generally ignores robots.txt rules” because the fetch is user-initiated. Perplexity is also the subject of reported stealth-crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. — see the controversy below.

Amazon runs three, and this is a useful exception: Amazonbot (improves products and “may be used to train Amazon AI models”), Amzn-SearchBot (search), and Amzn-User (Alexa user fetches). Per Amazon’s docs, Amzn-SearchBot and Amzn-User explicitly do NOT crawl for generative AI model training — so “all AI companies scrape for training” is a myth. Amazon also honors a noarchive meta tag with a special meaning: do not use the page for model training — a rare per-page training opt-out.

Amazon’s robots handling has two unusual operational edges: its crawlers may use a robots.txt copy cached within the previous 30 days, and Amazon says that if the file cannot be fetched, its crawlers behave as though it does not exist. That does not mean a failed audit fetch proves Amazon saw an allow-all state; the audit and Amazon may be reaching the host at different times or through different network paths. Evidence for this claim Amazon says its crawlers may use a robots.txt copy cached within the previous 30 days. Scope: Amazon crawler behavior only; a current audit fetch cannot identify which cached copy Amazon used. Confidence: high · Verified: 2026-07-19 Amazon: Amazonbot Evidence for this claim Amazon says its crawlers behave as if robots.txt does not exist when they cannot fetch it. Scope: Amazon crawler behavior only; a tool-side fetch failure is not proof that Amazon observed the same failure. Confidence: high · Verified: 2026-07-19 Amazon: Amazonbot

Meta runs Meta-ExternalAgent (training/indexing for Llama and Meta AI), Meta-ExternalFetcher (user fetch), and separately Facebookbot (link previews — not an AI crawler). Blocking Meta-ExternalAgent doesn’t affect your Facebook link previews.

Common Crawl (CCBot) is run by a non-profit foundation that maintains an open web archive. It’s the hidden layer: the Common Crawl dataset has been used to train ChatGPT, Claude, LLaMA, and many others — so blocking CCBot has downstream effects on models that never crawl you directly. Common Crawl also warns that impostors spoof CCBot.

ByteDance (Bytespider) is the aggressive one: declared as respecting robots.txt, but widely reported violating it, with very high request volumes. Others worth knowing: AI2Bot (Allen Institute, open models / the Dolma dataset), Cohere, Diffbot (structured-data extraction whose output feeds many downstream AI pipelines), and Timpibot / webzio-extended.

Do AI bots actually obey robots.txt?

For training and search crawlers, the major operators (OpenAI, Anthropic, Google, Apple, Amazon) all state in their docs that they respect robots.txt. The contested zone is user-triggered fetchers, where several operators explicitly say robots.txt may not apply because a person initiated the request — OpenAI says “these actions are initiated by a user, robots.txt rules may not apply”; Perplexity says Perplexity-User “generally ignores robots.txt rules.” That’s stated policy, not a violation.

The Perplexity controversy is the headline dispute, and I’ll flag it as reported, not settled. In June 2024, Wired and developer Robb Knight reported that Perplexity accessed content from sites that had blocked it, apparently using unlisted IPs. In August 2025, Cloudflare reported that Perplexity used a stealth user-agent impersonating Chrome on macOS, rotated IPs and ASNs to evade blocks, and fetched content from newly created test domains that carried a full Disallow: / — and Cloudflare subsequently delisted Perplexity as a verified bot. Present this as documented reporting from those sources; Perplexity’s own docs maintain that PerplexityBot respects robots.txt.

Because some bots ignore robots.txt at some level, Cloudflare has become the practical enforcement layer — network-level blocking, a one-click “AI Scrapers and Crawlers” toggle, managed robots.txt, robots.txt-violation tracking, and a pay-per-crawl beta. As of July 2025, new Cloudflare domains block known AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. by default.

How to control AI crawlers

• Per-user-agent robots.txt is the primary lever, and it’s granular: blocking GPTBot does not block OAI-SearchBot or ChatGPT-User. The robots.txt recipes (block all training bots, block AI-search bots) are in the Scripts tab.
• The training tokens — Google-Extended and Applebot-Extended — opt you out of Gemini/Apple-Intelligence training with no search cost.
• Amazon’s noarchive meta tag is a per-page training opt-out for Amazon.
• Cloudflare AI Crawl Control — block/allow/charge at the network level, plus managed robots.txt and Content Signals directives (search, ai-input, ai-train); default is search=yes, ai-train=no.
• llms.txt is a proposal (Jeremy Howard / Answer.AI, Sept 2024), not an adopted standard — and not honored by major crawlers. It’s a curated markdown map meant to be read at inference time, but an Ahrefs study of 137K sites found 97% of published llms.txt files get zero requests, and no major LLM provider has formally adopted it. Treat it as documentation tooling, not AI-search tooling.
• The ai.robots.txt community project maintains a comprehensive Disallow: / file listing 150+ AI bot user-agents — a useful reference for blanket blocking.

A visibility-first crawler policy

For a public educational or commercial site that wants AI-search visibility, my default policy is:

1. Allow conventional search crawlers that feed the search surfaces you want, including Googlebot and BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share..
2. Allow documented AI-search/indexing crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot on public content.
3. Allow intended user-directed fetchers such as ChatGPT-User, Claude-User, and Perplexity-User, while recognizing that provider robots behavior differs. Evidence for this claim OpenAI publishes separate controls for GPTBot, OAI-SearchBot, and ChatGPT-User, and says each setting is independent. Scope: OpenAI-declared crawlers and user agents; their purposes and robots behavior are platform-specific. Confidence: high · Verified: 2026-07-14 OpenAI: Crawlers Evidence for this claim Anthropic documents ClaudeBot, Claude-SearchBot, and Claude-User separately and says its bots honor robots.txt directives. Scope: Anthropic-declared bots; user-initiated retrieval has additional product-specific conditions. Confidence: high · Verified: 2026-07-14 Anthropic: Bot information Evidence for this claim Perplexity documents PerplexityBot for search indexing separately from Perplexity-User for user-requested fetches, with different robots behavior. Scope: Perplexity-declared crawlers and published IP ranges; a user-agent string alone does not verify identity. Confidence: high · Verified: 2026-07-28 Perplexity: Crawlers
4. Make a separate rights decision for training/model-development access. GPTBot, ClaudeBot, Google-Extended, and other training controls do not need to share the search policy. Allow them only if that use matches your publishing policy.
5. Protect private and operational routes with authentication. Keep /admin/, previews, account data, and non-public APIs behind server-side authorization; a Disallow is only a crawler preference.
6. Verify identity before bypassing WAF controls. Combine the documented user-agent with current provider IP ranges, reverse-DNS verification, or supported bot authentication. Never create a broad WAF allow rule from a spoofable user-agent alone.

Do not add a specific user-agent group merely to say Allow: / unless you repeat every restriction that must still apply to that crawler. Under robots matching rules, a specific group can supersede the wildcard group rather than inherit its private-path exclusions under provider-specific robots parsing. The simplest safe file is often one independently safe wildcard group plus narrowly scoped training opt-outs.

For this site specifically, the current public policy is an explicit allow posture for search, AI input, and AI training while excluding admin, preview, API, trap, and gateway routes. I would keep search and user-directed retrieval allowed. I would change ai-train=yes only as an editorial-rights decision—not as an AI visibility tactic—because training access has no demonstrated citation benefit.

AI crawler access is not AI readiness

An allow rule proves, at most, that the declared robots policy permits a named agent to request a URL. It does not prove the operator obeyed the policy, fetched the page, rendered it, extracted the important information, placed it in model context, cited it, or completed an action.

Keep five layers separate:

A crawler-access checker should therefore report allowed, blocked, or indeterminate for policy—not “AI ready.” A successful log request proves that request occurred; it is not evidence that the page was used in an answer. A valid feed or tool declaration likewise does not prove that a public page renders correctly or that checkout can fulfill the represented offer.

Use How AI Search Works for retrieval and citation layers, and AI Shopping OptimizationAI shopping optimization is the practice of making a merchant's products discoverable, recommendable, and buyable across AI shopping surfaces — ChatGPT, Google AI Mode, Gemini, Copilot, and Perplexity — by treating the structured product feed as a first-class optimization surface alongside the human-facing webpage, not a replacement for it. for catalog and transaction readiness. Keeping the layers separate makes the remedy specific: fix access policy for a block, page output for extraction, rendering for a missing state, an interface contract for operability, or commerce data for a failed purchase.

For ecommerce pages, the extractability test should verify that core product identity and the selected offer—SKU, group ID, price, currency, and availability—exist in the initial response and still agree after rendering. The raw-versus-rendered product-page standardProduct page SEO is the practice of optimizing an individual product detail page (PDP) so it ranks in organic search and earns rich results. It blends unique product copy, structured data, variant canonicalization, image SEO, and customer reviews — but the structured data earns rich results and eligibility for free product listings, it doesn't make the page rank. defines what should be server-visible; crawler permission alone says nothing about that contract.

The myth that costs you traffic: training ≠ AI-search

Here’s the one to get right. Blocking training bots is not the same as blocking AI-search bots. They’re separate robots.txt rules with opposite consequences:

• Block training bots (GPTBot, ClaudeBot, CCBot, Bytespider, plus the Google-Extended/Applebot-Extended tokens) → no effect on Google rankings, confirmed by Google and by empirical publisher data. You’re opting out of model training, full stop.
• Block AI-search bots (OAI-SearchBot, PerplexityBot) → you lose visibility in ChatGPT and Perplexity answers. That’s a real trade-off, not a free lunch.

If you block too broadly — say, a blanket block that catches OAI-SearchBot while you only meant to stop training — you can quietly remove yourself from AI search. Be precise about which category each rule hits.

Why publishers block at all: the social contract

The reason this is even a debate comes down to traffic. As I’ve put it in my AI bots analysis, “there’s a cost to bots crawling your websites and there’s a social contract between search engines and website owners, where search engines add value by sending referral traffic to websites. Google sends traffic (for now), so they don’t get blocked.” AI crawlers, so far, mostly don’t send traffic back — and the data backs the unease. In my Cloudflare Radar analysis, AI bots are already firmly the #2 crawler behind search engines and on track to overtake them. More in the Stats tab.

For the broader pipeline this sits inside — discovery, the crawl scheduler, rendering, and how crawling differs from indexing — see the crawling hub, and the sibling topics on a crawler in general, the user-agent, Googlebot, Bingbot, and robots.txt.

AI summary

A condensed take on the Advanced version:

• AI crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. are three categories, not one — and conflating them is the core mistake.
  • Training: GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot, Meta-ExternalAgent, AI2Bot, Cohere, Diffbot — scrape to build model corpora.
  • AI-search/indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.: OAI-SearchBot, PerplexityBot, Applebot, Amzn-SearchBot — build a citing index.
  • User-triggered fetchers: ChatGPT-User, Perplexity-User, Claude-User — fetch a page live when a person asks.
• Each bot has its own user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. and its own robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. rule. Blocking GPTBot does not block OAI-SearchBot or ChatGPT-User.
• Google-Extended and Applebot-Extended are tokensA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory., not bots — they don’t crawl; they only opt you out of training/groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it.. Disallowing them has zero search cost.
• Blocking training bots ≠ hurting SEO. Google: Google-Extended “does not impact a site’s inclusion in Google Search.” Empirical publisher data agrees.
• Blocking AI-search bots DOES cut AI visibilityLLM visibility (or AI visibility) is the aggregate measure of how often and how prominently a brand or page shows up in AI-generated answers — across AI Overviews, ChatGPT, Perplexity, Copilot, and Gemini. It's the AI-search analog of organic visibility, but it's driven by different signals. in ChatGPT/Perplexity — a real trade-off.
• Amazon’s Amzn-SearchBot and Amzn-User do NOT train on crawled content (per Amazon). “All AI bots train” is a myth.
• robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. debate: user-triggered fetchers carve themselves out (“may not apply”); Perplexity is reported (Wired/Robb Knight 2024; Cloudflare 2025) to use stealth crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — present as reported, not settled.
• llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. is a proposal, not adopted — 97% of files unread; no major LLM supports it. Cloudflare is the practical enforcement layer.

Official documentation

Primary-source documentation from the AI operators and the relevant proposals.

OpenAI

• OpenAI bots / crawlers — GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, and the “each botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. setting is independent” policy.

Anthropic

• Does Anthropic crawl data from the web? — ClaudeBot, Claude-SearchBot, Claude-User, robots controls, and current IP-list verification.
• Anthropic crawler IP list — provider-specific verification data.

Google

• Google common crawlers (incl. Google-Extended) — the product tokensA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. and user-agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target..
• AI features and your website — GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., not Google-Extended, controls inclusion in AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. and AI Mode.

Apple

• About Applebot — Applebot vs Applebot-Extended, and the “Applebot-Extended does not crawl webpages” distinction.

Amazon

• Amazonbot — Amazonbot, Amzn-SearchBot, Amzn-User, the noarchive training opt-out, and which bots do not train.

Perplexity

• Perplexity crawlers — PerplexityBot and Perplexity-User, including the “generally ignores robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. rules” statement.

Common Crawl

• CCBot — the non-profit crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. behind much LLMA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). training data, plus the impersonation warning.

Allen Institute for AI

• AI2 crawler notice — AI2Bot and how to filter it.

Cloudflare (AI crawler controls)

• AI Crawl Control — block/allow/charge, monitoring, and robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. violation tracking.
• Managed robots.txt — auto-maintained AI-crawler directives.
• Control content use for AI training — managed robots.txt, Content Signals, and pay-per-crawl.

The proposal

• llms.txt specification — the inference-time map proposal (a proposal, not an adopted standard).
• ai.robots.txt project — community-maintained list of AI bot user-agents.

Quotes from the source

On-the-record statements from the operators and proposals. Each link is a deep link that jumps to the quoted passage where verified.

OpenAI — independent botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., independent purposes

• “Each bot setting is independent, allowing webmasters granular control over how their content interacts with different OpenAI systems.” — OpenAI bots documentation. Jump to quote
• GPTBot is “used to make our generative AI foundation models more useful and safe.” OAI-SearchBot is “used to surface websites in search results in ChatGPT’s search features.” ChatGPT-User: because “these actions are initiated by a user, robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. rules may not apply.” Jump to quote

Anthropic — robots controls and verification scope

• Anthropic documents ClaudeBot, Claude-SearchBot, and Claude-User, their robots controls, and a shared IP-list verification method. Source

Google — the tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory., not a bot

• “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” Jump to quote

Apple — Applebot-Extended doesn’t crawl

• “Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results.” Jump to quote

Perplexity — the user fetcher’s carve-out

• PerplexityBot is “designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.” Perplexity-User: “Since a user requested the fetch, this fetcher generally ignores robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. rules.” Jump to quote

Common Crawl — the hidden training layer

• CCBot exists for “democratizing access to web information by producing and maintaining an open repository of web crawl data.” Note: “we are aware of crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. falsely identifying themselves as CCBot.” Jump to quote

llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. — a proposal, by its own description

• “A proposal to standardise on using an /llms.txt file to provide information to help LLMsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). use a website at inference time.” — llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. specification. Jump to quote

Me — the social contract

• “There’s a cost to bots crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. your websites and there’s a social contract between search engines and website owners, where search engines add value by sending referral traffic to websites. Google sends traffic (for now), so they don’t get blocked.” — Patrick Stox, Ahrefs. Read the piece

The master AI-crawler table

The centerpiece. Every major AI botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., its user-agent tokenA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., operator, category, and stated robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. compliance. Copy tokensA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. exactly — spelling and case matter in robots.txt.

The three rules that fall out of the table

• Blocking a training bot (GPTBot, ClaudeBot, CCBot…) → no ranking effect.
• Blocking an AI-search bot (OAI-SearchBot, PerplexityBot) → lose AI-answer visibility.
• Google-Extended and Applebot-Extended are tokens, not crawlers — pure training opt-outs with no search cost.

Fast facts

• GPTBot ≠ OAI-SearchBot ≠ ChatGPT-User — three independent OpenAI rules.
• Amazon’s Amzn-SearchBot and Amzn-User do not train on crawled content.
• Amazon honors noarchive meta as a no-training per-page signal.
• llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. is a proposal, not an adopted standard; major crawlers don’t honor it.
• Block by user-agent in robots.txt, not by IP (IP blocks can stop a bot reading your robots.txt at all).

How to audit and decide what to allow

A practical pass for deciding which AI crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. to allow, block, or just watch.

• Pull your server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. and list which AI user-agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. are actually hitting you (GPTBot, ClaudeBot, CCBot, Bytespider, OAI-SearchBot, PerplexityBot, Applebot, etc.) and how often. Logs are the ground truth.
• Sort each one into its category — training, AI-search, or user fetch. Decisions follow from the category, not the brand.
• Decide your training stance. Comfortable being trained on? Allow the training bots. Not comfortable? Block GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot, Meta-ExternalAgent — and add the Google-Extended and Applebot-Extended tokensA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory.. None of this affects Google rankings.
• Decide your AI-search stance — carefully. If you want to appear in ChatGPT and Perplexity answers, keep OAI-SearchBot and PerplexityBot allowed. Only block these if you genuinely want out of AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity..
• Don’t catch the wrong bot. Confirm a “block training” rule isn’t also hitting an AI-search bot. GPTBot ≠ OAI-SearchBot.
• Block by user-agent, not by IP — IP blocks can prevent a bot from reading your robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. (Anthropic’s explicit warning).
• Verify suspicious bots. CCBot and others get impersonated; check published IP ranges / reverse DNS before trusting a user-agent string.
• Consider network-level enforcement (Cloudflare AI Crawl Control) for bots that don’t reliably honor robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere..
• Skip llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. as an SEO move — it’s a proposal, largely unread, and not honored by the major crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. Don’t expect AI-search visibility from it.
• Re-check quarterly. New bots appear constantly (the ai.robots.txt list is a good reference) and operators change behavior.

The mistake that costs you AI-search traffic

Conflating training botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. with AI-search bots is the single most expensive mistake in this space, and it’s the one I called out above as the myth to kill. A rule meant to stop model training and a rule meant to stop AI-search citation are not the same rule — mixing them up has opposite consequences.

• Wrong: blocking “AI bots” as one blob (a single Disallow rule that happens to catch GPTBot and OAI-SearchBot). Why it’s wrong: it silently removes you from ChatGPT and Perplexity answers while you thought you were only opting out of training. Do instead: write separate, named rules per bot — block GPTBot/ClaudeBot/CCBot/Bytespider for training, and leave OAI-SearchBot/PerplexityBot allowed unless you deliberately want out of AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity..

Confusing identity verification with access policy

A provider-published IP list can help verify who sent a request. It does not tell you what your policy should be, and it does not turn a network block into a robots rule.

• Wrong: treating a verified range as an automatic allow/block decision. Why it’s wrong: identity, declared crawl preference, and enforced network access are separate facts. Do instead: express crawl policy with the documented User-agent tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory.; add network-level enforcement (Cloudflare AI Crawl Control) on top for bots that don’t reliably respect the file.

Treating llms.txt as an AI-search lever

llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. gets pitched constantly as “the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. for AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity.,” and that framing is wrong per the article’s own sourcing.

• Wrong: publishing an llms.txt file expecting it to boost AI-search visibility or citations. Why it’s wrong: it’s a proposal, not an adopted standard — an Ahrefs study of 137K sites foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. 97% of published llmsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4)..txt files get zero requests, and no major LLM provider has formally adopted it. Do instead: treat llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. as documentation tooling at best, and put the effort into the robots.txt rules and AI-search bots that actually control visibility.

Trusting a user-agent string without verifying it

CCBot in particular gets impersonated, by Common Crawl’s own admission, and Bytespider is widely reported to ignore the rules it claims to respect.

• Wrong: assuming a User-agent string in your logs is who it claims to be. Why it’s wrong: a spoofed or non-compliant bot can hit you at full volume while a robots.txt rule sits there unenforced. Do instead: confirm high-volume or suspicious hits against the operator’s published IP ranges or reverse DNS before trusting the string — the Tools tab has a tool built for exactly this check.

The decision: allow, block, or monitor

1. Categorize first — it decides everything. Training vs AI-search vs user-fetch. Get the category right and the consequence is obvious: training blocks are SEO-safe; AI-search blocks cost you AI visibilityLLM visibility (or AI visibility) is the aggregate measure of how often and how prominently a brand or page shows up in AI-generated answers — across AI Overviews, ChatGPT, Perplexity, Copilot, and Gemini. It's the AI-search analog of organic visibility, but it's driven by different signals.; user-fetch is contested.

2. The allow / block / monitor rule.

• Allow AI-search botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. (OAI-SearchBot, PerplexityBot, Applebot) if you want AI referral visibility — most sites should.
• Block training bots (GPTBot, ClaudeBot, CCBot, Bytespider) if you don’t want to be trained on — zero ranking cost. Add the Google-Extended/Applebot-Extended tokensA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. for Gemini/Apple training.
• Monitor user-triggered fetchers and known bad actors (Bytespider, undeclared Perplexity traffic). robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. may not stop them; logs and network-level rules are how you actually see and gate them.

3. The “token, not bot” check. Before you reason about crawl traffic, ask: is this an actual crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. or a control token? Google-Extended and Applebot-Extended generate no traffic — they only flip a permission on traffic that already happened.

4. Training ≠ AI-search (the expensive mistake). Two separate robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. rules, opposite consequences. Never block “AI bots” as one blob — you’ll either fail to stop training or accidentally delete yourself from AI answers.

5. robots.txt is a request; enforcement is separate. The protocol works for compliant bots. For the ones that carve themselves out (user fetchers) or are reported to ignore it (Perplexity stealth crawlers, Bytespider), real enforcement is network-level (Cloudflare) — robots.txt alone won’t do it.

Should I block this AI bot?

This decision tree walks the allow/block/monitor rule from the Frameworks tab step by step. Start by identifying which of the three categories the botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. belongs to, then answer the one question that decides its fate.

Quarterly AI-bot robots.txt audit

A recurring check keeps your robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. rules matched to reality as new AI botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. appear and operators change behavior — the Checklists tab already flags “re-check quarterly,” and this is the step-by-step version of that flag.

1. Pull the quarter’s AI-bot hits. Run your server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. through the site’s Log File Analyzer or grep for the known user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. list (the Scripts tab has the exact command) to see which AI bots actually hit you and how often.
2. Re-check each bot’s category. Cross-reference every user-agent against the master table in the Cheat Sheets tab to confirm it’s still training, AI-search, or user-fetch — and flag any new, unrecognized bot against the ai.robots.txt project list.
3. Test your current rules. Run the live robots.txt through the robots.txt Tester against each bot’s user-agent to confirm training rules aren’t accidentally catching an AI-search bot.
4. Verify suspicious or high-volume hits. Confirm bots like CCBot or Bytespider against the operator’s published IP ranges or reverse DNS with the Googlebot Verifier before trusting the user-agent string — impersonation happens.
5. Log the quarter’s AI-bot crawl share. Note the split of training vs AI-search vs user-fetch traffic (see the How to Measure tab) so the next quarterly check has a trend to compare against, not just a snapshot.

Confirm your robots.txt edit did what you meant

After editing robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. to block training botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., run these checks before you consider the change done — the goal is proving AI-search bots still reach your pages while training bots are actually disallowed.

Test 1: AI-search bots still allowed

• Test to run: Check the URL against OAI-SearchBot and PerplexityBot user-agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. with the robots.txt Tester, or confirm real hits with the Googlebot Verifier.
• Expected result: “Allowed” for both on any page you want cited in AIThree distinct states of AI visibility: retrieved (an AI fetched your page as source material), mentioned (your brand appears in the answer text), and cited (your URL is linked as a source). They don't always happen together, and each is measured with a different tool. answers.
• Failure interpretation: a “Disallowed” result means your training-block rule is too broad and caught the wrong user-agent.
• Monitoring window: immediate for the tester; allow a few days for the bot’s own cached copy of robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. to refresh.
• Rollback trigger: an AI-search bot you meant to allow shows Disallowed, or citations/referrals from that engine drop after the change.

Test 2: Training bots actually disallowed

• Test to run: Check the same URL against GPTBot, ClaudeBot, CCBot, and Bytespider with the robots.txt Tester.
• Expected result: “Disallowed” for each.
• Failure interpretation: an “Allowed” result usually means a typo in the tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. — spelling and case matter in robots.txt.
• Monitoring window: immediate.
• Rollback trigger: none required — a training block doesn’t need a rollback path unless you decide you’re fine with training after all.

Test 3: Google rankings unaffected

• Test to run: Check Search Console PerformanceThe Google Search Console report that shows how your site actually performed in Google Search, built from real impressions and clicks. It reports four metrics — clicks, impressions, average CTR, and average position — and keeps the most recent 16 months of data./IndexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. Coverage for the affected URLs a few weeks after blocking training bots or the Google-Extended token.
• Expected result: no material impressions or ranking change — Google states Google-Extended “does not impact a site’s inclusion in Google Search.”
• Failure interpretation: a wide impressions drop coinciding with the change usually means GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. itself got caught by a too-broad rule.
• Monitoring window: 2-4 weeks (Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. reporting lag).
• Rollback trigger: impressions or rankings drop and GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. shows Disallowed on a URL that should be crawlable.

The standing KPIs for AI-bot traffic

Tracking these quarter over quarter tells you whether your AI-botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. posture is actually doing what you intend — separate from any single robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. edit.

robots.txt — block the major bots

Block training crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., keep AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity.. This opts you out of model training while staying visible in ChatGPT/Perplexity answers. Copy tokensA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. exactly.

# --- AI training crawlers ---
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

# --- Training control tokens (not crawlers) ---
# Opt out of Gemini/Vertex training; search unaffected
User-agent: Google-Extended
Disallow: /

# Opt out of Apple Intelligence training; Applebot search unaffected
User-agent: Applebot-Extended
Disallow: /

Block AI-search indexers (this REDUCES AI-answer visibility). Only do this if you genuinely want out of ChatGPT/Perplexity answers.

User-agent: OAI-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

The key principle: blocking GPTBot does not block OAI-SearchBot or ChatGPT-User — each is an independent rule.

Spot AI bots in your access logs

Find which AI user-agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. are hitting you and how often.

macOS / Linux

# Count hits per known AI bot in an Apache/Nginx access log
grep -iE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot|Amzn-SearchBot|meta-externalagent|Applebot|AI2Bot|Diffbot' access.log \
  | grep -oiE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot|Amzn-SearchBot|meta-externalagent|Applebot|AI2Bot|Diffbot' \
  | sort | uniq -c | sort -rn

Windows (PowerShell)

# Count hits per known AI bot in an access log
$bots = 'GPTBot','OAI-SearchBot','ChatGPT-User','ClaudeBot','Claude-User','Claude-SearchBot','PerplexityBot','Perplexity-User','CCBot','Bytespider','Amazonbot','Amzn-SearchBot','meta-externalagent','Applebot','AI2Bot','Diffbot'
Select-String -Path .\access.log -Pattern ($bots -join '|') |
  ForEach-Object { ($_ -split '"')[5] } |
  ForEach-Object { foreach ($b in $bots) { if ($_ -match $b) { $b } } } |
  Group-Object | Sort-Object Count -Descending | Select-Object Count, Name

A user-agent string can be spoofed (CCBot especially) — confirm important bots against the operator’s published IP ranges or via reverse + forward DNS before acting on them.

Classify your log hits by AI-bot category

Paste raw server-log lines or a list of user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. strings and have an AI classify each one using the master table from the Cheat Sheets tab — useful when you’re staring at a log file full of unfamiliar botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. names.

I have a list of user-agent strings from my server logs. Using this
classification (paste the Cheat Sheets table from this article, or the list
below), classify each user-agent as one of: Training, AI-search, User fetch,
or Control token (not a crawler). Flag any user-agent you don't recognize as
"unknown — verify."

Categories:
- Training: GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot, Meta-ExternalAgent,
  AI2Bot, cohere-training-data-crawler, Diffbot, Timpibot, webzio-extended
- AI-search: OAI-SearchBot, PerplexityBot, Applebot, Amzn-SearchBot
- User fetch: ChatGPT-User, Perplexity-User, Claude-User, Claude-SearchBot,
  Amzn-User, Meta-ExternalFetcher
- Control token (no crawl traffic): Google-Extended, Applebot-Extended

My user-agent list:
<paste your list here>

Draft a robots.txt block list from your stance

Describe your training and AI-search stance in plain language and have the AI draft the corresponding robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. rules, then verify the output against the robots.txt Tester before publishing it.

Draft a robots.txt block that does the following, using exact, correctly-cased
user-agent tokens for each rule (one User-agent/Disallow pair per bot, don't
combine bots under one User-agent line):

1. Block all training bots: GPTBot, ClaudeBot, CCBot, Bytespider,
   Meta-ExternalAgent, Amazonbot.
2. Also disallow the Google-Extended and Applebot-Extended control tokens
   (training/grounding opt-out only — not real crawlers).
3. Leave OAI-SearchBot and PerplexityBot allowed so AI-search visibility is
   unaffected.

Output only the robots.txt block, no explanation.

Tools for verifying and controlling AI bots

• Googlebot Verifier — paste server-log hits and confirm whether a “GPTBot” or “ClaudeBot” request is actually from that operator’s published IP range, or an impersonator. The direct fix for the “trusting a user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string” mistake in the What Not to Do tab.

• robots.txt Tester — check whether a specific URL is allowed or blocked for a specific botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.’s user-agent, so you can confirm a training-block rule isn’t also catching an AI-search bot.

• robots.txt Generator — build the training-block / AI-search-allow rules from the Scripts tab without hand-typing user-agent tokensA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. (where a typo silently breaks the rule).

• llms.txt Generator + Validator — if you still want an llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. file for documentation purposes, generate and validate one here — just don’t expect it to move AI-search visibility (see What Not to Do).

Resources worth your time

My related writing

• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — my Cloudflare Radar analysis; AI botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. as the clear #2 crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. and the social-contract framing.
• The AI Bots That ~140 Million Websites Block the Most — robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. block-rate data across the open web, and the request-volume / block-rate correlation.
• ChatGPT Has 12% of Google’s Search Volume but Google Sends 190x More Traffic — the AI-vs-search referral gap that drives the blocking debate.
• What We Actually Know About Optimizing for LLM Search — the broader GEOGenerative Engine Optimization (GEO) is the practice of optimizing content and brand presence so AI-powered search engines and assistants — Google AI Overviews, ChatGPT, Perplexity — cite, recommend, or mention you when generating answers. Google's position is that it's still SEO. context.

From others

• Cloudflare: Perplexity is using stealth, undeclared crawlers — the August 2025 investigation (reported findings).
• Robb Knight: Perplexity AI is lying about its user agent — the original June 2024 investigation.
• ai.robots.txt project — community-maintained list of 150+ AI bot user-agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target..
• Known Agents (formerly Dark Visitors) — comprehensive AI bot directory.
• Cloudflare: Declaring your AI independence (one-click AI bot blocking) — the announcement of Cloudflare’s one-click “AI Scrapers and Crawlers” toggle.
• Cloudflare: Control content use for AI training — managed robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., Content Signals directives, and the pay-per-crawl beta explained.
• Heise Online: Crawler without limits — Perplexity ignores robots.txt — June 2024 reporting on Perplexity’s robots.txt evasion.
• Gizmodo: Perplexity AI ignored the internet’s rules — further coverage of the robots.txt compliance controversy.

Stats worth citing

• AI botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. are the clear #2 crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. From my Cloudflare Radar analysis, search-engine bots still crawl the most, but AI bots are firmly second and on pace to overtake them — AI bots already represent roughly 18.9% of all bot requests. Source
• The referral gap that drives blocking. Google sends ~190x more traffic to websites than ChatGPT, despite ChatGPT having a meaningful share of Google’s search volume — the imbalance behind the “social contract” argument. Source
• Crawl-to-referral ratios (Cloudflare, 2025). Roughly 14 Google crawls per referral visit, versus far higher ratios for AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. — on the order of thousands of OpenAI crawls and tens of thousands of Anthropic crawls per referral. The numbers are why publishers question the trade.
• llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. reality check. In an Ahrefs study of 137K sites, ~28% published an llmsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4)..txt file but 97% of those files received zero requests, and no major LLM provider has formally adopted it. Source
• Block rates across the open web. From my study of ~140M sites, the most-blocked AI bots cluster around GPTBot, CCBot, ClaudeBot, Amazonbot, and Bytespider — block rates track request volume. Source
• Bytespider reach and enforcement. Bytespider accessed 40.40% of all Cloudflare-protected websites before Cloudflare began blocking it in July 2024; traffic declined 71.45% afterward. Meta-ExternalAgent accessed 22.16% of protected sites — second only to GPTBot (28.97%). Source
• AI bot traffic growth. Cloudflare Radar data shows AI bot traffic grew 65% in six months (2025), with GPTBot requests up 305% between May 2024 and May 2025. As of July 2025, new Cloudflare domains block known AI crawlers by default, and over 1 million Cloudflare customers have enabled the AI bot blocking feature. Source

Test yourself: AI Crawlers

Five quick questions on the three AI-crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. categories and the traps around them. Pick an answer for each, then check.

AI Crawlers

AI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls.

Related: Crawler, User Agent, Googlebot, Bingbot, Robots.txt

AI Crawlers

AI crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. are automated programs operated by AI companies that fetch web pages for one of three distinct purposes — and the distinction is the whole point. Lumping them together is how publishers accidentally block the wrong thing.

The three categories are:

• Training / data-collection crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. scrape the web at scale to build datasets for training large language modelsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4).. Examples: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance), Amazonbot, Meta-ExternalAgent. Two important controls in this bucket — Google-Extended and Applebot-Extended — are robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. tokensA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory., not crawlers: they don’t fetch anything themselves; they only opt your content out of training and groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it..
• AI-search / indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. crawlers build an index used to answer questions with citations and links back. Examples: OAI-SearchBot (OpenAI, distinct from GPTBot), PerplexityBot, Applebot (Spotlight/Siri/Apple Intelligence), Amzn-SearchBot. Blocking these can remove you from AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. answers.
• User-triggered / on-demand fetchers fetch a specific page live because a person asked a question. Examples: ChatGPT-User, Perplexity-User, Claude-User. Several operators state robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. “may not apply” here because the request is user-initiated — contested territory.

They differ from traditional search crawlers like a crawler such as GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. or BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. in one practical way that drives most of the debate: many AI crawlers return little or no referral traffic for the content they collect. Each major bot has its own user-agent token, so robots.txt control is granular — blocking a training bot does not block an AI-search bot, and vice versa.

Related: Crawler, User Agent, Googlebot, Bingbot, Robots.txt

Changelog