llms.txt

llms.txt is a community proposal for a Markdown file at /llms.txt, not an adopted web standard. Evidence for this claim llms.txt is a community proposal for a Markdown file at /llms.txt that offers LLM-friendly site information; it is not a web standard. Scope: The proposal's own specification and stated purpose. Confidence: high · Verified: 2026-07-14 llms.txt proposal OpenAI’s published crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. controls use robots.txt and do not document llms.txt as a control. Evidence for this claim OpenAI documents robots.txt controls for its declared crawlers and does not list llms.txt as a crawler control. Scope: OpenAI's published crawler controls; absence is not proof that no internal system ever fetches the file. Confidence: high · Verified: 2026-07-14 OpenAI: Crawlers

TL;DR — llmsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4)..txt is a Markdown file you put at /llms.txt that hands AI systems a short, curated list of your most important pages. A developer named Jeremy Howard proposed it in 2024. It’s a nice idea, but in practice almost nobody reads it — Google ignores it outright, and the main thing that does read it is AI coding assistants, not the search bots most people are hoping to reach. It costs little to add and probably won’t move your AI visibilityLLM visibility (or AI visibility) is the aggregate measure of how often and how prominently a brand or page shows up in AI-generated answers — across AI Overviews, ChatGPT, Perplexity, Copilot, and Gemini. It's the AI-search analog of organic visibility, but it's driven by different signals..

What llms.txt is

llms.txt is a plain Markdown file you place at the root of your site — yoursite.com/llms.txt. Inside, you write your site’s name, a one-line description, and then a short list of links to your most important pages, each with a quick note about what it covers. The idea is that an AI system can read that one file and immediately understand what your site is and where the good stuff lives, instead of having to crawl and parse every page.

It was proposed by Jeremy Howard — the person behind fast.ai (a popular deep learning course) and Answer.AI — back in September 2024.

What it’s supposed to do vs. what it actually does

The pitch: AI tools have limited “context windowsA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory.” (they can only read so much at once), and turning messy HTML into clean text is hard. So why not let the site owner hand the model a tidy, pre-written map?

The reality is more sobering. The major AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. providers haven’t adopted it. Google has flatly said it doesn’t use it. And when Ahrefs studied 137,000 sites, 97% of the published llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. files got zero requests in a month — as in, nothing fetched them at all.

The one place it genuinely gets used is AI coding assistants — tools like Claude Code that developers use to write software. Those read llms.txt to find their way around technical documentation quickly. That’s the original use case, and it works. It’s just very different from “this will get my business cited in ChatGPT.”

Is it like robots.txt?

A lot of people assume llms.txt is “robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. for AI.” It isn’t.

• robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. controls access — it tells crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. what they’re allowed to fetch, and well-behaved bots obey it.
• llms.txt offers guidance — it suggests what’s worth reading, and nothing is obligated to listen. It can’t block anything.

If your goal is to keep AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. out, llms.txt does nothing — that’s a job for robots.txt and your CDN.

Should you bother?

If you run developer documentation that coding agents read, sure — it’s cheap and it helps them. If you’re a normal business hoping llms.txt will boost your visibility in AI answers, the honest answer today is: don’t expect results. Spend that time on clear content and structure instead.

Want the spec, the provider-by-provider breakdown, the adoption data, and the security angle? Switch to the Advanced tab.

The specification describes a voluntary LLMA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4).-friendly site summary; it does not establish crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. compliance or indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. behavior. Evidence for this claim llms.txt is a community proposal for a Markdown file at /llms.txt that offers LLM-friendly site information; it is not a web standard. Scope: The proposal's own specification and stated purpose. Confidence: high · Verified: 2026-07-14 llms.txt proposal OpenAI currently documents named bots and independent robots.txt settings instead. Evidence for this claim OpenAI documents robots.txt controls for its declared crawlers and does not list llms.txt as a crawler control. Scope: OpenAI's published crawler controls; absence is not proof that no internal system ever fetches the file. Confidence: high · Verified: 2026-07-14 OpenAI: Crawlers

TL;DR — llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. is a proposal (Jeremy Howard, Answer.AI, Sept 3, 2024), not an adopted standard. It’s a curated Markdown index at /llms.txt, with an optional full-content /llms-full.txt. The bots people hope are reading it for AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. — OAI-SearchBot, PerplexityBot — barely register; the dominant consumer is Claude Code and other coding agents. Google ignores it (Illyes confirmed; Mueller compared it to the keywords meta tagThe meta keywords tag — <meta name=\"keywords\" content=\"...\"> — is a mid-1990s HTML head element meant to let a page declare its own topic keywords to search engines. Google has publicly ignored it for ranking since 2009, and no major search engine uses it as a ranking signal today. It's dead as SEO, and a populated one only leaks your target keywords to anyone who views your source.). In Ahrefs’ 137K-site study, 97% of files got zero requests, and SE Ranking foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. that removing llms.txt from a citation model improved its accuracy. It’s worth adding for developer docs consumed by coding agents; for GEOGenerative Engine Optimization (GEO) is the practice of optimizing content and brand presence so AI-powered search engines and assistants — Google AI Overviews, ChatGPT, Perplexity — cite, recommend, or mention you when generating answers. Google's position is that it's still SEO./AEOAnswer Engine Optimization (AEO) is the practice of structuring content so engines deliver it as a direct answer — featured snippets, voice assistants, and AI search — rather than just a ranked link. Coined for voice search in 2018 and revived for the LLM era. Google's position is that it's still SEO. on a normal site, the data doesn’t support it. There’s also a real prompt-injection risk.

What llms.txt actually is

llms.txt is a proposalllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. — I want to lead with that word because it’s the whole story. There’s no RFC, no W3C blessing, no IETF process. It’s one well-reasoned suggestion from Jeremy Howard (co-founder of Answer.AI and fast.ai), published September 3, 2024, that caught on hard in the developer-documentation world and got grafted onto SEO by a community hungry for an AI-visibility shortcut.

The design problem it targets is legitimate. As the spec puts it, “Large language models increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety.” And converting HTML to clean, LLM-friendly text is, in the spec’s words, “difficult and imprecise.” Howard’s fix is to let the site author pre-flatten the content they want models to see into curated Markdown.

The format

/llms.txt is an ordered Markdown document:

1. H1 (required) — project or site name.
2. Blockquote — a short summary with the key information.
3. Optional prose/lists — paragraphs or bullets, but no headings here.
4. H2-delimited link lists — - [name](url): optional notes.
5. An ## Optional section — secondary links a model can skip when context is tight.

A minimal example, from the spec:

# FastHTML

> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX,
> and fastcore's FT...

## Docs

- [FastHTML quick start](url): A brief overview of many features

## Optional

- [Starlette documentation](url): A subset useful for FastHTML development.

Two companion conventions exist:

• /llms-full.txt — the entire site content flattened into one Markdown file. Anthropic, Perplexity, and Stripe publish both this and the short form.
• The .md URL convention — offering page.md versions of individual pages so they’re LLM-ready without the full dump.

A newer variant skips the static files entirely: edge-served Markdown via content negotiation. Cloudflare’s Markdown for Agents converts any HTML page to clean Markdown on the fly when a client sends Accept: text/markdown — no per-page .md files, no /llms-full.txt to regenerate, no template changes, because the CDN does the conversion at the edge (Cloudflare cites ~80% fewer tokens for the agent). It’s a paid feature — Pro plan and up, not the free tier — so on this site it’s one Cloudflare upgrade away from being switched on rather than something I’ve hand-built. Whether it’s worth switching on is the same open question that hangs over llms.txt itself: serving crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. a parallel Markdown copy of every page is convenient, but it’s another representation to keep in sync, and Bing’s Fabrice Canel has flagged the doubled crawl load and cloaking-adjacent risk of edge-served alternate content. Check your logs for real agent demand before flipping it.

llms.txt vs robots.txt — they are not the same thing

The single most common misconception is that llms.txt is “robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. for AI.” It isn’t, and conflating them leads to bad decisions.

robots.txt is a real, enforced standard. llms.txt is a suggestion box that most systems aren’t reading yet. If you want to control AI crawler access — which is a separate, legitimate goal — that lives in robots.txt and AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls., not here.

Who actually reads it

This is the section that matters, because it’s where the data diverges hardest from the hype. In Ahrefs’ study of 137,210 domains (May 2026 traffic):

• 97% of published llms.txt files received zero requests. Only ~3% (about 1,100 domains) saw any traffic at all.
• Of the files that did get requested: 96% of requests came from bots, 4% from humans.
• Of the bot requests, 77% came from non-AI tools — SEO auditors (Ahrefs itself among them), anonymous crawlers, tech-profiling bots. The SEO irony writes itself.
• Only ~19.5% came from AI tools, and once you break that down it gets worse for the GEOGenerative Engine Optimization — visibility inside AI answer engines. theory:
  • AI agents & infrastructure (coding agents): ~10.5%
  • Training crawlers (GPTBot led at 4.51%): ~5.3%
  • AI assistants: ~2.5%
  • AI retrieval bots — the ones that build citation indexes: 1.1%.

Read that last line twice. The bots most people add llms.txt for — OAI-SearchBot, PerplexityBot — “barely registered.” The dominant AI consumer is Claude Code, Anthropic’s coding agent, which the study found “outfetched every AI retrieval bot, assistant, and training crawler except GPTBot.” And GPTBot is a training crawler that crawls everything regardless — its presence isn’t evidence of intentional llms.txt parsing.

Other independent measurements land in the same place. OtterlyAI watched one site for 90 days: of 62,100+ AI bot visits, exactly 84 hit /llms.txt — about 0.1%, performing “3x worse than average pages.” A separate analysis of 515M+ LLM bot events over 90 days found 408 requests total targeting /llms.txt — “statistically negligible.”

The takeaway is the developer-docs-vs-GEO split: llms.txt works for what Howard built it for (coding agents navigating API docs), and it largely doesn’t get touched by the search-citation bots the SEO community wants it to reach.

What the providers have said

Google — no, and no plans. Gary Illyes confirmed at Search Central Live (July 2025) that Google doesn’t support llms.txt and isn’t planning to. John Mueller, the most-quoted voice here, compared it to the keywords meta tag (April 2025) and called it a “temporary crutch, perhaps to save some tokens” for AI coding tools — explicitly not a search-visibility mechanism. Google’s May 2026 guidance stated machine-readable files like llms.txt aren’t necessary for AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. or AI Mode, which “continue to rely on traditional SEO signals.” (Google did briefly publish an llms.txt on its dev docs on December 3, 2025, then pulled it the same day — later attributed to a CMSA content management system (CMS) is software that lets users create, manage, and publish digital content — like blog posts and pages — without writing raw code. WordPress, Drupal, and Joomla are the most common open-source CMS platforms. update, not a strategy shift.)

OpenAI — nothing. No announcement that ChatGPT, GPTBot, or OAI-SearchBot parse it. Server logs contradict meaningful usage; OAI-SearchBot “barely registered.” OpenAI points people to robots.txt for crawler control.

Anthropic — the interesting one. Anthropic publishes both llms.txt and llms-full.txt at docs.anthropic.com/llms.txt, and Claude Code demonstrably fetches other sites’ llms.txt files — it’s the largest AI consumer in the data. But that’s a coding-tool story. There’s no confirmation that Claude.ai’s search or citation layer honors llms.txt. “Anthropic’s coding agent reads it” and “Anthropic’s search index trusts it” are different claims, and only the first is supported.

Perplexity — claims yes, data says rarely. Perplexity has stated it retrieves llms.txt and uses it to “prioritize page selection.” But proactive fetching is the catch: PerplexityBot “barely registered” in the request data, and there’s “almost zero activity from PerplexityBot requesting llms.txt files proactively.” Paste an llms.txt URL into Perplexity and it reads it fine; autonomous, proactive use is the gap between policy and practice.

Bing/Microsoft, Apple, Meta — no public position. Treat as unconfirmed.

Does it move AI citations?

No demonstrated benefit. SE Ranking modeled 300,000 domains with an XGBoost regression + SHAP analysis and found that removing llms.txt from the model improved its accuracy — their conclusion: “LLMs.txt doesn’t seem to directly impact AI citation frequency. At least not yet.” A Search Engine Land 10-site, 180-day study saw 8 of 10 sites show no measurable change, and the two “winners” had confounding changes (PR coverage, new FAQ pages, technical fixes) you can’t separate from the llms.txt itself.

The meta-keywords comparison — fair or not?

Mueller’s keywords-meta-tag analogy holds on the points that matter: both are site-owner self-descriptions, neither is verified, and both are open to gaming/cloaking — you could show one thing in llms.txt and another on the page. That manipulability is exactly why a serious AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. product won’t trust a self-description over the actual pages. The counter-argument (Carolyn Shelby) is that llms.txt at least points to real URLs that have to deliver — it’s “a spotlight, not a wish list.” Both are right, and the practical result is the same: AI search systems don’t lean on it.

When it’s worth adding — and when to skip it

Add it if you run developer documentation or an API reference whose primary audience is AI coding agents (Claude Code, Cursor, Copilot, Codeium). That’s the use case it was built for, and it works.

Skip it — or at least don’t expect AI-visibility returns — for content sites, media, ecommerce, and ordinary business sites. The cost is ~20 minutes; the GEO benefit today is near-zero; and that 20 minutes is better spent on content structure, FAQ coverage, and the AI search optimizationAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. fundamentals that actually move AI visibility. The high adoption headlines (BuiltWith’s 844K+ figure) are inflated by platform auto-deployment — Mintlify rolled llms.txt out to all its hosted docs sites at once. Adoption is not usage; 97% of files get zero requests.

The security angle

This is underreported and real: because AI agents are designed to trust what’s in llms.txt, the file is a natural attack surface. The Ahrefs study flagged bad actors probing llms.txt files for prompt-injection vulnerabilities. If you do publish one, treat it as security-sensitive: keep it in version control, alert on unauthorized changes, and never put anything in it you wouldn’t show publicly.

Implementing it (if you decide to)

• Placement: /llms.txt at the domain root, served as text/plain or text/markdown, HTTP 200.
• Curate, don’t dump: 20–50 of your most important links. If you list everything, a model might as well crawl directly — curation is the entire point.
• Optionally serve /llms-full.txt for contexts that want all your content.
• Validate: load the file into Claude, ChatGPT, or Perplexity and ask it to summarize your site; then check server logs after a few weeks for real requests.
• Generators exist for VitePress (vitepress-plugin-llms), Docusaurus (docusaurus-plugin-llms), WordPress (“Website LLMs.txt”), Drupal, plus a Python CLI (llms_txt2ctx).

The bottom line: low cost, currently near-zero GEO benefit, genuinely useful for the coding-agent use case it was born for. Check your own server logs before you decide — if you’re seeing Claude-Code/Cursor traffic, llms.txt may help them; if you’re chasing OAI-SearchBot and PerplexityBot, it isn’t the lever.

TIP

Validate the file's proposed Markdown structure and catch references an agent cannot fetch directly with my free llms.txt Generator & Validator Free

1. Paste or generate the curated file and run the structural checks.
2. Replace relative or inaccessible references with public absolute URLs an external client can fetch directly.
3. Verify the deployed response and linked resources separately; valid form does not mean any AI provider will read the file.

Tap to enlarge

The focused validator finding reports an error on line six: the link URL slash-private is not absolute, and llms.txt links should use absolute URLs an LLM can fetch directly.

AI summary

A condensed take on the Advanced version:

• llmsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4)..txt is a proposal, not a standard — Jeremy Howard (Answer.AI/fast.ai), Sept 3, 2024. A curated Markdown indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. at /llms.txt; optional full-content /llms-full.txt.
• It is not robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. for AI. robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. controls access and is enforced; llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. is advisory and can’t block anything.
• Almost nobody reads it. Ahrefs (137K sites): 97% of files got zero requests. Of files that were requested, 77% of botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. hits were non-AI tools (SEO auditors), and AI retrieval bots — the citation-builders — were just 1.1%.
• Claude Code is the dominant AI consumer, not search bots. OAI-SearchBot and PerplexityBot “barely registered.”
• Google ignores it. Illyes confirmed no support/plans; Mueller compared it to the keywords meta tagThe meta keywords tag — <meta name=\"keywords\" content=\"...\"> — is a mid-1990s HTML head element meant to let a page declare its own topic keywords to search engines. Google has publicly ignored it for ranking since 2009, and no major search engine uses it as a ranking signal today. It's dead as SEO, and a populated one only leaks your target keywords to anyone who views your source.. OpenAI: nothing. Perplexity: claims support, but proactive fetching is minimal. Anthropic publishes its own + Claude Code reads it, but search-layer use is unconfirmed.
• No citation benefit shown. SE Ranking (300K domains): removing llms.txt from the model improved accuracy.
• Worth it for developer docs consumed by coding agents; skip it as a GEOGenerative Engine Optimization (GEO) is the practice of optimizing content and brand presence so AI-powered search engines and assistants — Google AI Overviews, ChatGPT, Perplexity — cite, recommend, or mention you when generating answers. Google's position is that it's still SEO. play for normal sites. Prompt-injection risk — treat it as security-sensitive.

Official documentation

Primary-source material — the spec, the proposal, and the providers’ own files and statements.

The spec & proposal

• llmstxt.org — the specification: file format, /llms.txt vs /llms-full.txt, and the inference-time rationale.
• The original proposal (Jeremy Howard, Sept 3, 2024) — Answer.AI’s announcement post.
• AnswerDotAI/llms-txt on GitHub — the repo, tooling, and llms_txt2ctx CLI.

Providers’ own files

• Anthropic’s llms.txt — a live example (Anthropic also publishes llms-full.txt).
• Perplexity’s llms.txt — another live example.

Google’s official position

• Google has stated machine-readable files like llmsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4)..txt are not necessary for AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. or AI Mode, which rely on traditional SEO signals (May 2026 guidance); Gary Illyes confirmed no support and no plans at Search Central Live (July 2025). See the Search Engine Journal coverage of John Mueller’s statement for the most-cited articulation.

Quotes from the source

On-the-record statements on llmsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4)..txt — from Google, the spec’s creator, and the researchers who measured it.

John Mueller, Google Search Advocate (April 17, 2025)

• “AFAIK none of the AI services have said they’re using LLMs.TXTllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. (and you can tell when you look at your server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. that they don’t even check for it). To me, it’s comparable to the keywords meta tagThe meta keywords tag — <meta name=\"keywords\" content=\"...\"> — is a mid-1990s HTML head element meant to let a page declare its own topic keywords to search engines. Google has publicly ignored it for ranking since 2009, and no major search engine uses it as a ranking signal today. It's dead as SEO, and a populated one only leaks your target keywords to anyone who views your source. – this is what a site-owner claims their site is about … (Is the site really like that? well, you can check it. At that point, why not just check the site directly?)” Coverage

Gary Illyes, Google Search Central (Search Central Live, July 2025)

• Confirmed Google does not support llms.txt and has no plans to. (Reported from the event; no primary transcript URL identified — confirm before treating as final.)

Jeremy Howard, creator (Answer.AI)

• “A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.” llmstxt.org
• From the spec, on the core problem: “Large language models increasingly rely on website information, but face a critical limitation: context windowsA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. are too small to handle most websites in their entirety.” llmstxt.org

Brett Tabke, Pubcon / WebmasterWorld (March 2025)

• “we just don’t need people thinking they are different from any other spiderA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index..” Search Engine Land

Carolyn Shelby, Search Engine Land (July 9, 2025 — the dissenting view)

• “Llms.txt curates a list of real URLs, and the content has to exist – and deliver – when the model gets there.” and “Think of llms.txt like a treasure map for AI systems – one you draw yourself. It’s not a wish list. It’s a spotlight.” Search Engine Land

Should you implement llms.txt? — a decision checklist

Work top to bottom. The first “yes” that fits usually settles it.

• Is your primary audience AI coding agents (Claude Code, Cursor, Copilot, Codeium) reading developer docs / an API reference? → Yes, add it. This is the use case it was built for and it works.
• Are you adding it purely to improve AI search citationsAn AI citation is the visible source link an AI answer engine shows next to its generated text — the clickable reference that credits the web page it used. A citation's presence is a separate thing from whether the cited page actually supports the statement, and from being retrieved (read behind the scenes) or merely mentioned (named without a link); citation is driven more by brand mentions and being retrievable than by traditional ranking. for a content, media, ecommerce, or general business site? → Don’t expect results. The data shows no citation benefit; spend the time on content structure instead.
• Have you checked your server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. first? Look for Claude-Code/Cursor traffic (llmsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4)..txt may help them) vs OAI-SearchBot/PerplexityBot (it won’t move that needle).
• Are you treating it as robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. for AI? → Stop. It blocks nothing. Use robots.txt + your CDNAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. for access control.

If you do publish one:

• Place it at /llms.txt (root), serve text/plain/text/markdown, HTTP 200HTTP 200 OK is the standard 2xx success status code, meaning the server received, understood, and fulfilled the request and is returning the resource. It's the code every page you want indexed should return — but a 200 alone doesn't guarantee Google will index the page..
• Curate 20–50 of your most important URLs — don’t dump the sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing..
• Keep it in version control and alert on unauthorized changes (prompt-injection risk — agents are built to trust it).
• Put nothing in it you wouldn’t show publicly.
• Optionally serve /llms-full.txt for full-content contexts.
• Validate: load the URL into Claude/ChatGPT/Perplexity, ask for a site summary; recheck server logs after a few weeks for real requests.

Provider support — cheat sheet

Who actually supports llmsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4)..txt (as of mid-2026)

Fast facts

• Proposed: Jeremy Howard (Answer.AI/fast.ai), Sept 3, 2024. A proposal, not an adopted standard.
• Files: /llms.txt (short, curated) and optional /llms-full.txt (full content flattened to Markdown).
• Format: H1An H1 tag is the HTML `<h1>` element that marks a page's primary heading — the big visible headline at the top of the content. It helps users, search engines, and screen readers understand what the page is about. name → blockquote summary → H2 link lists (- [name](url): note) → ## Optional section.
• vs robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.: robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. = access control, enforced. llms.txt = guidance, advisory, blocks nothing.
• The number: 97% of published files get zero requests (Ahrefs, 137K sites).
• Dominant AI reader: Claude Code, not search bots.
• Citation benefit: none demonstrated (SE Ranking: removing it improved model accuracy).
• Security: prompt-injection risk — version-control it, alert on changes.

llms.txt mistakes to avoid

Treating the file as an access-control standard

llms.txt does not allow or block crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. Use robots.txt, authentication, and other real access controls for that job.

Promising an AI-visibility lift

Provider support and observed fetching are limited. Describe the file as an optional navigation aid, not a ranking or citation lever.

Publishing sensitive or private URLs

The file is public and easy to discover. Include only resources intended for public consumption; never use obscurity as protection.

Generating an uncurated sitemap in Markdown

The proposal is most useful as a short, human-readable guide to important resources. A huge dump adds maintenance cost without clarifying which pages matter.

Common llms.txt problems

The file returns HTML instead of Markdown

Symptom: /llms.txt shows an application shell, branded 404, or redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. target. Likely cause: A catch-all route or CDN rule intercepts the text path. Fix: Serve the file directly at the root with a successful response and plain text or Markdown content, then request it without cookies.

Links in the file fail or redirect repeatedly

Symptom: A reader cannot retrieve listed resources at their canonical URLsHow search engines pick one canonical URL among duplicates and consolidate signals onto it.. Likely cause: The file was generated from stale navigation or relative paths. Fix: validate every URL, replace retired locations, and prefer stable canonical HTTPSHTTPS is the encrypted version of HTTP — it uses TLS to authenticate the server and protect data in transit between a browser and a website. Google announced it as a lightweight ranking signal in 2014 and today conditionally prefers HTTPS pages as canonical; Chrome marks plain HTTP pages 'Not Secure.' URLs.

No bots request the file

Symptom: Logs show no traffic to /llms.txt. Likely cause: The systems you care about do not support or discover the proposal. Fix: Confirm log coverage and leave the file as a low-cost experiment; do not add crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-specific tricks or infer a site problem from zero requests.

Prompts for curating llms.txt

Build a proposed llms.txt outline from this public URL inventory. Keep only canonical,
durable resources that help an agent understand the site or complete a task. Group them
under short Markdown headings, write one factual description per link, and flag URLs
that redirect, duplicate another page, require authentication, or may expose sensitive
information. Do not claim the file affects rankings or citations.

[paste inventory]

Review this llms.txt file against the supplied crawl results. Report broken or
redirecting URLs, non-canonical links, descriptions unsupported by the destination,
missing high-value documentation, and sections that are too broad to be useful. Return
a corrected draft using only public URLs from the inputs.

[paste file and crawl results]

Frameworks for deciding whether to use llms.txt

The cost–consumer–content test

1. Cost: Can the team generate and maintain the file with little ongoing work?
2. Consumer: Is there an identified agent or workflow that actually reads it?
3. Content: Does the site have stable public documentation worth curating?

Implement when all three are credible. If the consumer is hypothetical, treat the file as an experiment and cap the maintenance budget.

Keep the roles separate

• robots.txt: crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. access guidance.
• XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags.: URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched. for search engines.
• llms.txt: proposed curated reading guide for AI systems.
• MCPMCP is an open protocol, created and open-sourced by Anthropic in November 2024, that standardizes how AI applications connect to external tools and data at runtime. It runs on a host–client–server architecture and lets agents call functions, read data, and take actions — the opposite of a static file like llms.txt.: runtime protocol through which an agent can discover and invoke resources or tools.

One file should not be evaluated as if it performs another system’s job.

Tools for llms.txt

• llms.txt Generator + Validator: Draft a curated file and check its structure and linked resources before publishing.
• AI-Crawler Access Checker: Audit actual crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. access controls separately; llms.txt is not a substitute for this check.
• curl: Verify status, redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., content type, and body at the exact root path.
• Server or CDN logs: Determine whether named user agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. request the file; absence is an observation, not proof of a ranking problem.
• A link checker: Revalidate every curated destination on a schedule so the guide does not decay.

Validate an llms.txt release

Test the root response

Test to run: Request /llms.txt directly without cookies and follow no redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't.. Expected result: A successful response contains the intended Markdown at the root path. Failure interpretation: Routing or deployment does not expose the file. Monitoring window: Immediate. Rollback trigger: The path serves an HTML shell, error document, or unrelated redirect.

Test every listed resource

Test to run: Crawl the URLs in the file and compare each destination with its description. Expected result: Public canonical resources resolve successfully and match the stated purpose. Failure interpretation: The guide is stale or misleading. Monitoring window: Immediate, then on the documentation update cadence. Rollback trigger: A listed URL exposes private material or consistently fails.

Test the stated experiment outcome honestly

Test to run: Query server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. for /llms.txt by user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. after publication. Expected result: Requests, if any, are recorded and attributable; zero is also a valid result. Failure interpretation: Logging may be incomplete or consumers may not support the proposal. Monitoring window: Several normal crawl cycles. Rollback trigger: None for zero traffic alone; remove only when maintenance or exposure risk exceeds the file’s value.

Test yourself: llms.txt

Resources worth your time

The original source

• llmstxt.org — the spec.
• Jeremy Howard’s proposal post (Sept 3, 2024) — the rationale, straight from the creator.
• AnswerDotAI/llms-txt — the repo and tooling.

The data (read these before you implement anything)

• What Is llms.txt, and Should You Care About It? — Ahrefs — Ryan Law’s overview; the 28%-publish / 97%-zero-requests framing.
• The llms.txt adoption study — Ahrefs — Louise Linehan & Xibeijia Guan, 137,210 domains: who’s actually requesting the files (Claude Code dominates AI; retrieval botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. barely register) and the prompt-injection finding.
• Does llms.txt matter? We tracked 10 sites — Search Engine Land — 180-day study; 8/10 sites showed no measurable change.
• The llms.txt experiment — OtterlyAI — 90 days, one site: 84 of 62,100+ AI botAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. visits hit /llms.txt.

The debate

• No, llms.txt is not the “new meta keywords” — Carolyn Shelby, Search Engine Land — the strongest defense of llmsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4)..txt.
• Meet llms.txt, a proposed standard — Search Engine Land — the pro/con summary and Brett Tabke’s critique.

My related writing on this site

• AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. — controlling the bots (which is what llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. is not for), and where the access-control story actually lives.

From around the industry

• SE Ranking: The State of llms.txt — 300,000-domain XGBoost/SHAP study finding that removing llms.txt from the citation model improved accuracy; also has adoption-rate breakdown by traffic tier.
• Wix AI Search Lab — llms.txt myths (Crystal Carter, June 2026) — more optimistic read; 1,400+ files reviewed, argues llms.txt files themselves surface in AI results even if adoption is patchy.
• Google Says LLMs.Txt Comparable To Keywords Meta Tag — Search Engine Journal — full John Mueller quote and context from April 2025.
• presenc.ai: State of llms.txt 2026 — provider-by-provider support matrix; treat Anthropic/Perplexity “confirmed” claims with care (primary sources not always cited).
• The llms.txt is dead. More precisely: a dud. — Kai Spriestersbach (Medium) — aggregates OtterlyAI 90-day data, Gary Illyes confirmation, and the Google 24-hour incident into a skeptic’s summary.

Stats worth citing

• 97% of published llmsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4)..txt files got zero requests in a month (Ahrefs study, 137,210 domains, May 2026). Only ~3% saw any traffic at all. Source
• Claude Code is the dominant AI consumer of llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. — it “outfetched every AI retrieval botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., assistant, and training crawlerAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. except GPTBot” (the training bot). The retrieval bots that build citation indexes were just 1.1% of requests. Source
• 77% of bot requests to llms.txt files came from non-AI tools — SEO auditors, anonymous crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., tech-profiling bots — not AI at all. Source
• 84 of 62,100+ AI bot visits hit /llms.txt over 90 days on one monitored site (~0.1%), performing “3x worse than average pages.” Source
• Removing llms.txt from a citation model improved its accuracy — SE Ranking, 300,000 domains, XGBoost + SHAP analysis. No demonstrated citation benefit “at least not yet.” Source
• Adoption ≠ usage. BuiltWith counted 844,000+ sites with llms.txt (Oct 2025), but much of that is platform auto-deployment (Mintlify rolled it out to all hosted docs sites at once), not informed individual decisions.
• Only 408 requests targeted /llms.txt across 515 million+ AI bot traffic events analyzed over 90 days (limy.ai analysis) — described as “statistically negligible.”
• 10.13% of sites have adopted llms.txt across a 300,000-domain sample (SE Ranking study, 2026) — roughly 1 in 10, with no meaningful difference by traffic tier (low-traffic 9.88%, high-traffic 8.27%). Source

llms.txt

llms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it.

Related: AI Crawlers, Crawling

llms.txt

llmsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4)..txt is a proposed standard — not an adopted one — that places a curated Markdown file at /llms.txt on a website to help AI systems find and use that site’s content more efficiently. It was proposed by Jeremy Howard (Answer.AI / fast.ai) on September 3, 2024.

The file works like a hand-crafted table of contents: an H1An H1 tag is the HTML `<h1>` element that marks a page's primary heading — the big visible headline at the top of the content. It helps users, search engines, and screen readers understand what the page is about. site name, a blockquote summary, and H2-grouped lists of links to the most important pages, each with a short note. The idea is that an AI agent reading the file can quickly find the right content without parsing the entire site. A companion /llms-full.txt flattens the whole site into one Markdown document.

Two things are easy to get wrong. First, llms.txt is not robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. for AI: robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. controls access and is enforced; llms.txt offers guidance and is purely advisory — it can’t block anything. Second, the major AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. providers haven’t adopted it. Google ignores it (Gary Illyes confirmed no support, no plans; John Mueller compared it to the keywords meta tagThe meta keywords tag — <meta name=\"keywords\" content=\"...\"> — is a mid-1990s HTML head element meant to let a page declare its own topic keywords to search engines. Google has publicly ignored it for ranking since 2009, and no major search engine uses it as a ranking signal today. It's dead as SEO, and a populated one only leaks your target keywords to anyone who views your source.), and in an Ahrefs study of 137K sites, 97% of published llms.txt files received zero requests. The dominant real-world consumer is developer tooling — coding agents like Claude Code — not the search botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. that build citation indexes.

Related: AI Crawlers, Crawling