llms.txt

llms.txt is a community proposal for a Markdown file at /llms.txt, not an adopted web standard. Evidence for this claim llms.txt is a community proposal for a Markdown file at /llms.txt that offers LLM-friendly site information; it is not a web standard. Scope: The proposal's own specification and stated purpose. Confidence: high · Verified: 2026-07-14 llms.txt proposal OpenAI’s published crawler controls use robots.txt and do not document llms.txt as a control. Evidence for this claim OpenAI documents robots.txt controls for its declared crawlers and does not list llms.txt as a crawler control. Scope: OpenAI's published crawler controls; absence is not proof that no internal system ever fetches the file. Confidence: high · Verified: 2026-07-14 OpenAI: Crawlers

TL;DR — llms.txt is a Markdown file you put at /llms.txt that hands AI systems a short, curated list of your most important pages. A developer named Jeremy Howard proposed it in 2024. It’s a nice idea, but in practice almost nobody reads it — Google ignores it outright, and the main thing that does read it is AI coding assistants, not the search bots most people are hoping to reach. It costs little to add and probably won’t move your AI visibility.

What llms.txt is

llms.txt is a plain Markdown file you place at the root of your site — yoursite.com/llms.txt. Inside, you write your site’s name, a one-line description, and then a short list of links to your most important pages, each with a quick note about what it covers. The idea is that an AI system can read that one file and immediately understand what your site is and where the good stuff lives, instead of having to crawl and parse every page.

It was proposed by Jeremy Howard — the person behind fast.ai (a popular deep learning course) and Answer.AI — back in September 2024.

What it’s supposed to do vs. what it actually does

The pitch: AI tools have limited “context windows” (they can only read so much at once), and turning messy HTML into clean text is hard. So why not let the site owner hand the model a tidy, pre-written map?

The reality is more sobering. The major AI search providers haven’t adopted it. Google has flatly said it doesn’t use it. And when Ahrefs studied 137,000 sites, 97% of the published llms.txt files got zero requests in a month — as in, nothing fetched them at all.

The one place it genuinely gets used is AI coding assistants — tools like Claude Code that developers use to write software. Those read llms.txt to find their way around technical documentation quickly. That’s the original use case, and it works. It’s just very different from “this will get my business cited in ChatGPT.”

Is it like robots.txt?

A lot of people assume llms.txt is “robots.txt for AI.” It isn’t.

• robots.txt controls access — it tells crawlers what they’re allowed to fetch, and well-behaved bots obey it.
• llms.txt offers guidance — it suggests what’s worth reading, and nothing is obligated to listen. It can’t block anything.

If your goal is to keep AI bots out, llms.txt does nothing — that’s a job for robots.txt and your CDN.

Should you bother?

If you run developer documentation that coding agents read, sure — it’s cheap and it helps them. If you’re a normal business hoping llms.txt will boost your visibility in AI answers, the honest answer today is: don’t expect results. Spend that time on clear content and structure instead.

Want the spec, the provider-by-provider breakdown, the adoption data, and the security angle? Switch to the Advanced tab.

The specification describes a voluntary LLM-friendly site summary; it does not establish crawler compliance or indexing behavior. Evidence for this claim llms.txt is a community proposal for a Markdown file at /llms.txt that offers LLM-friendly site information; it is not a web standard. Scope: The proposal's own specification and stated purpose. Confidence: high · Verified: 2026-07-14 llms.txt proposal OpenAI currently documents named bots and independent robots.txt settings instead. Evidence for this claim OpenAI documents robots.txt controls for its declared crawlers and does not list llms.txt as a crawler control. Scope: OpenAI's published crawler controls; absence is not proof that no internal system ever fetches the file. Confidence: high · Verified: 2026-07-14 OpenAI: Crawlers

TL;DR — llms.txt is a proposal (Jeremy Howard, Answer.AI, Sept 3, 2024), not an adopted standard. It’s a curated Markdown index at /llms.txt, with an optional full-content /llms-full.txt. The bots people hope are reading it for AI search — OAI-SearchBot, PerplexityBot — barely register; the dominant consumer is Claude Code and other coding agents. Google ignores it (Illyes confirmed; Mueller compared it to the keywords meta tag). In Ahrefs’ 137K-site study, 97% of files got zero requests, and SE Ranking found that removing llms.txt from a citation model improved its accuracy. It’s worth adding for developer docs consumed by coding agents; for GEO/AEO on a normal site, the data doesn’t support it. There’s also a real prompt-injection risk.

What llms.txt actually is

llms.txt is a proposal — I want to lead with that word because it’s the whole story. There’s no RFC, no W3C blessing, no IETF process. It’s one well-reasoned suggestion from Jeremy Howard (co-founder of Answer.AI and fast.ai), published September 3, 2024, that caught on hard in the developer-documentation world and got grafted onto SEO by a community hungry for an AI-visibility shortcut.

The design problem it targets is legitimate. As the spec puts it, “Large language models increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety.” And converting HTML to clean, LLM-friendly text is, in the spec’s words, “difficult and imprecise.” Howard’s fix is to let the site author pre-flatten the content they want models to see into curated Markdown.

The format

/llms.txt is an ordered Markdown document:

1. H1 (required) — project or site name.
2. Blockquote — a short summary with the key information.
3. Optional prose/lists — paragraphs or bullets, but no headings here.
4. H2-delimited link lists — - [name](url): optional notes.
5. An ## Optional section — secondary links a model can skip when context is tight.

A minimal example, from the spec:

# FastHTML

> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX,
> and fastcore's FT...

## Docs

- [FastHTML quick start](url): A brief overview of many features

## Optional

- [Starlette documentation](url): A subset useful for FastHTML development.

Two companion conventions exist:

• /llms-full.txt — the entire site content flattened into one Markdown file. Anthropic, Perplexity, and Stripe publish both this and the short form.
• The .md URL convention — offering page.md versions of individual pages so they’re LLM-ready without the full dump.

A newer variant skips the static files entirely: edge-served Markdown via content negotiation. Cloudflare’s Markdown for Agents converts any HTML page to clean Markdown on the fly when a client sends Accept: text/markdown — no per-page .md files, no /llms-full.txt to regenerate, no template changes, because the CDN does the conversion at the edge (Cloudflare cites ~80% fewer tokens for the agent). It’s a paid feature — Pro plan and up, not the free tier — so on this site it’s one Cloudflare upgrade away from being switched on rather than something I’ve hand-built. Whether it’s worth switching on is the same open question that hangs over llms.txt itself: serving crawlers a parallel Markdown copy of every page is convenient, but it’s another representation to keep in sync, and Bing’s Fabrice Canel has flagged the doubled crawl load and cloaking-adjacent risk of edge-served alternate content. Check your logs for real agent demand before flipping it.

llms.txt vs robots.txt — they are not the same thing

The single most common misconception is that llms.txt is “robots.txt for AI.” It isn’t, and conflating them leads to bad decisions.

robots.txt is a real, enforced standard. llms.txt is a suggestion box that most systems aren’t reading yet. If you want to control AI crawler access — which is a separate, legitimate goal — that lives in robots.txt and AI crawlers, not here.

Who actually reads it

This is the section that matters, because it’s where the data diverges hardest from the hype. In Ahrefs’ study of 137,210 domains (May 2026 traffic):

• 97% of published llms.txt files received zero requests. Only ~3% (about 1,100 domains) saw any traffic at all.
• Of the files that did get requested: 96% of requests came from bots, 4% from humans.
• Of the bot requests, 77% came from non-AI tools — SEO auditors (Ahrefs itself among them), anonymous crawlers, tech-profiling bots. The SEO irony writes itself.
• Only ~19.5% came from AI tools, and once you break that down it gets worse for the GEO theory:
  • AI agents & infrastructure (coding agents): ~10.5%
  • Training crawlers (GPTBot led at 4.51%): ~5.3%
  • AI assistants: ~2.5%
  • AI retrieval bots — the ones that build citation indexes: 1.1%.

Read that last line twice. The bots most people add llms.txt for — OAI-SearchBot, PerplexityBot — “barely registered.” The dominant AI consumer is Claude Code, Anthropic’s coding agent, which the study found “outfetched every AI retrieval bot, assistant, and training crawler except GPTBot.” And GPTBot is a training crawler that crawls everything regardless — its presence isn’t evidence of intentional llms.txt parsing.

Other independent measurements land in the same place. OtterlyAI watched one site for 90 days: of 62,100+ AI bot visits, exactly 84 hit /llms.txt — about 0.1%, performing “3x worse than average pages.” A separate analysis of 515M+ LLM bot events over 90 days found 408 requests total targeting /llms.txt — “statistically negligible.”

The takeaway is the developer-docs-vs-GEO split: llms.txt works for what Howard built it for (coding agents navigating API docs), and it largely doesn’t get touched by the search-citation bots the SEO community wants it to reach.

What the providers have said

Google — no, and no plans. Gary Illyes confirmed at Search Central Live (July 2025) that Google doesn’t support llms.txt and isn’t planning to. John Mueller, the most-quoted voice here, compared it to the keywords meta tag (April 2025) and called it a “temporary crutch, perhaps to save some tokens” for AI coding tools — explicitly not a search-visibility mechanism. Google’s May 2026 guidance stated machine-readable files like llms.txt aren’t necessary for AI Overviews or AI Mode, which “continue to rely on traditional SEO signals.” (Google did briefly publish an llms.txt on its dev docs on December 3, 2025, then pulled it the same day — later attributed to a CMS update, not a strategy shift.)

OpenAI — nothing. No announcement that ChatGPT, GPTBot, or OAI-SearchBot parse it. Server logs contradict meaningful usage; OAI-SearchBot “barely registered.” OpenAI points people to robots.txt for crawler control.

Anthropic — the interesting one. Anthropic publishes both llms.txt and llms-full.txt at docs.anthropic.com/llms.txt, and Claude Code demonstrably fetches other sites’ llms.txt files — it’s the largest AI consumer in the data. But that’s a coding-tool story. There’s no confirmation that Claude.ai’s search or citation layer honors llms.txt. “Anthropic’s coding agent reads it” and “Anthropic’s search index trusts it” are different claims, and only the first is supported.

Perplexity — claims yes, data says rarely. Perplexity has stated it retrieves llms.txt and uses it to “prioritize page selection.” But proactive fetching is the catch: PerplexityBot “barely registered” in the request data, and there’s “almost zero activity from PerplexityBot requesting llms.txt files proactively.” Paste an llms.txt URL into Perplexity and it reads it fine; autonomous, proactive use is the gap between policy and practice.

Bing/Microsoft, Apple, Meta — no public position. Treat as unconfirmed.

Does it move AI citations?

No demonstrated benefit. SE Ranking modeled 300,000 domains with an XGBoost regression + SHAP analysis and found that removing llms.txt from the model improved its accuracy — their conclusion: “LLMs.txt doesn’t seem to directly impact AI citation frequency. At least not yet.” A Search Engine Land 10-site, 180-day study saw 8 of 10 sites show no measurable change, and the two “winners” had confounding changes (PR coverage, new FAQ pages, technical fixes) you can’t separate from the llms.txt itself.

The meta-keywords comparison — fair or not?

Mueller’s keywords-meta-tag analogy holds on the points that matter: both are site-owner self-descriptions, neither is verified, and both are open to gaming/cloaking — you could show one thing in llms.txt and another on the page. That manipulability is exactly why a serious AI search product won’t trust a self-description over the actual pages. The counter-argument (Carolyn Shelby) is that llms.txt at least points to real URLs that have to deliver — it’s “a spotlight, not a wish list.” Both are right, and the practical result is the same: AI search systems don’t lean on it.

When it’s worth adding — and when to skip it

Add it if you run developer documentation or an API reference whose primary audience is AI coding agents (Claude Code, Cursor, Copilot, Codeium). That’s the use case it was built for, and it works.

Skip it — or at least don’t expect AI-visibility returns — for content sites, media, ecommerce, and ordinary business sites. The cost is ~20 minutes; the GEO benefit today is near-zero; and that 20 minutes is better spent on content structure, FAQ coverage, and the AI search optimization fundamentals that actually move AI visibility. The high adoption headlines (BuiltWith’s 844K+ figure) are inflated by platform auto-deployment — Mintlify rolled llms.txt out to all its hosted docs sites at once. Adoption is not usage; 97% of files get zero requests.

The security angle

This is underreported and real: because AI agents are designed to trust what’s in llms.txt, the file is a natural attack surface. The Ahrefs study flagged bad actors probing llms.txt files for prompt-injection vulnerabilities. If you do publish one, treat it as security-sensitive: keep it in version control, alert on unauthorized changes, and never put anything in it you wouldn’t show publicly.

Implementing it (if you decide to)

• Placement: /llms.txt at the domain root, served as text/plain or text/markdown, HTTP 200.
• Curate, don’t dump: 20–50 of your most important links. If you list everything, a model might as well crawl directly — curation is the entire point.
• Optionally serve /llms-full.txt for contexts that want all your content.
• Validate: load the file into Claude, ChatGPT, or Perplexity and ask it to summarize your site; then check server logs after a few weeks for real requests.
• Generators exist for VitePress (vitepress-plugin-llms), Docusaurus (docusaurus-plugin-llms), WordPress (“Website LLMs.txt”), Drupal, plus a Python CLI (llms_txt2ctx).

The bottom line: low cost, currently near-zero GEO benefit, genuinely useful for the coding-agent use case it was born for. Check your own server logs before you decide — if you’re seeing Claude-Code/Cursor traffic, llms.txt may help them; if you’re chasing OAI-SearchBot and PerplexityBot, it isn’t the lever.

AI summary

A condensed take on the Advanced version:

• llms.txt is a proposal, not a standard — Jeremy Howard (Answer.AI/fast.ai), Sept 3, 2024. A curated Markdown index at /llms.txt; optional full-content /llms-full.txt.
• It is not robots.txt for AI. robots.txt controls access and is enforced; llms.txt is advisory and can’t block anything.
• Almost nobody reads it. Ahrefs (137K sites): 97% of files got zero requests. Of files that were requested, 77% of bot hits were non-AI tools (SEO auditors), and AI retrieval bots — the citation-builders — were just 1.1%.
• Claude Code is the dominant AI consumer, not search bots. OAI-SearchBot and PerplexityBot “barely registered.”
• Google ignores it. Illyes confirmed no support/plans; Mueller compared it to the keywords meta tag. OpenAI: nothing. Perplexity: claims support, but proactive fetching is minimal. Anthropic publishes its own + Claude Code reads it, but search-layer use is unconfirmed.
• No citation benefit shown. SE Ranking (300K domains): removing llms.txt from the model improved accuracy.
• Worth it for developer docs consumed by coding agents; skip it as a GEO play for normal sites. Prompt-injection risk — treat it as security-sensitive.

Official documentation

Primary-source material — the spec, the proposal, and the providers’ own files and statements.

The spec & proposal

• llmstxt.org — the specification: file format, /llms.txt vs /llms-full.txt, and the inference-time rationale.
• The original proposal (Jeremy Howard, Sept 3, 2024) — Answer.AI’s announcement post.
• AnswerDotAI/llms-txt on GitHub — the repo, tooling, and llms_txt2ctx CLI.

Providers’ own files

• Anthropic’s llms.txt — a live example (Anthropic also publishes llms-full.txt).
• Perplexity’s llms.txt — another live example.

Google’s official position

• Google has stated machine-readable files like llms.txt are not necessary for AI Overviews or AI Mode, which rely on traditional SEO signals (May 2026 guidance); Gary Illyes confirmed no support and no plans at Search Central Live (July 2025). See the Search Engine Journal coverage of John Mueller’s statement for the most-cited articulation.

Quotes from the source

On-the-record statements on llms.txt — from Google, the spec’s creator, and the researchers who measured it.

John Mueller, Google Search Advocate (April 17, 2025)

• “AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it). To me, it’s comparable to the keywords meta tag – this is what a site-owner claims their site is about … (Is the site really like that? well, you can check it. At that point, why not just check the site directly?)” Coverage

Gary Illyes, Google Search Central (Search Central Live, July 2025)

• Confirmed Google does not support llms.txt and has no plans to. (Reported from the event; no primary transcript URL identified — confirm before treating as final.)

Jeremy Howard, creator (Answer.AI)

• “A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.” llmstxt.org
• From the spec, on the core problem: “Large language models increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety.” llmstxt.org

Brett Tabke, Pubcon / WebmasterWorld (March 2025)

• “we just don’t need people thinking they are different from any other spider.” Search Engine Land

Carolyn Shelby, Search Engine Land (July 9, 2025 — the dissenting view)

• “Llms.txt curates a list of real URLs, and the content has to exist – and deliver – when the model gets there.” and “Think of llms.txt like a treasure map for AI systems – one you draw yourself. It’s not a wish list. It’s a spotlight.” Search Engine Land

Should you implement llms.txt? — a decision checklist

Work top to bottom. The first “yes” that fits usually settles it.

• Is your primary audience AI coding agents (Claude Code, Cursor, Copilot, Codeium) reading developer docs / an API reference? → Yes, add it. This is the use case it was built for and it works.
• Are you adding it purely to improve AI search citations for a content, media, ecommerce, or general business site? → Don’t expect results. The data shows no citation benefit; spend the time on content structure instead.
• Have you checked your server logs first? Look for Claude-Code/Cursor traffic (llms.txt may help them) vs OAI-SearchBot/PerplexityBot (it won’t move that needle).
• Are you treating it as robots.txt for AI? → Stop. It blocks nothing. Use robots.txt + your CDN for access control.

If you do publish one:

• Place it at /llms.txt (root), serve text/plain/text/markdown, HTTP 200.
• Curate 20–50 of your most important URLs — don’t dump the sitemap.
• Keep it in version control and alert on unauthorized changes (prompt-injection risk — agents are built to trust it).
• Put nothing in it you wouldn’t show publicly.
• Optionally serve /llms-full.txt for full-content contexts.
• Validate: load the URL into Claude/ChatGPT/Perplexity, ask for a site summary; recheck server logs after a few weeks for real requests.

Provider support — cheat sheet

Who actually supports llms.txt (as of mid-2026)

Fast facts

• Proposed: Jeremy Howard (Answer.AI/fast.ai), Sept 3, 2024. A proposal, not an adopted standard.
• Files: /llms.txt (short, curated) and optional /llms-full.txt (full content flattened to Markdown).
• Format: H1 name → blockquote summary → H2 link lists (- [name](url): note) → ## Optional section.
• vs robots.txt: robots.txt = access control, enforced. llms.txt = guidance, advisory, blocks nothing.
• The number: 97% of published files get zero requests (Ahrefs, 137K sites).
• Dominant AI reader: Claude Code, not search bots.
• Citation benefit: none demonstrated (SE Ranking: removing it improved model accuracy).
• Security: prompt-injection risk — version-control it, alert on changes.

llms.txt mistakes to avoid

Treating the file as an access-control standard

llms.txt does not allow or block crawling. Use robots.txt, authentication, and other real access controls for that job.

Promising an AI-visibility lift

Provider support and observed fetching are limited. Describe the file as an optional navigation aid, not a ranking or citation lever.

Publishing sensitive or private URLs

The file is public and easy to discover. Include only resources intended for public consumption; never use obscurity as protection.

Generating an uncurated sitemap in Markdown

The proposal is most useful as a short, human-readable guide to important resources. A huge dump adds maintenance cost without clarifying which pages matter.

Common llms.txt problems

The file returns HTML instead of Markdown

Symptom: /llms.txt shows an application shell, branded 404, or redirect target. Likely cause: A catch-all route or CDN rule intercepts the text path. Fix: Serve the file directly at the root with a successful response and plain text or Markdown content, then request it without cookies.

Links in the file fail or redirect repeatedly

Symptom: A reader cannot retrieve listed resources at their canonical URLs. Likely cause: The file was generated from stale navigation or relative paths. Fix: validate every URL, replace retired locations, and prefer stable canonical HTTPS URLs.

No bots request the file

Symptom: Logs show no traffic to /llms.txt. Likely cause: The systems you care about do not support or discover the proposal. Fix: Confirm log coverage and leave the file as a low-cost experiment; do not add crawler-specific tricks or infer a site problem from zero requests.

Prompts for curating llms.txt

Build a proposed llms.txt outline from this public URL inventory. Keep only canonical,
durable resources that help an agent understand the site or complete a task. Group them
under short Markdown headings, write one factual description per link, and flag URLs
that redirect, duplicate another page, require authentication, or may expose sensitive
information. Do not claim the file affects rankings or citations.

[paste inventory]

Review this llms.txt file against the supplied crawl results. Report broken or
redirecting URLs, non-canonical links, descriptions unsupported by the destination,
missing high-value documentation, and sections that are too broad to be useful. Return
a corrected draft using only public URLs from the inputs.

[paste file and crawl results]

Frameworks for deciding whether to use llms.txt

The cost–consumer–content test

1. Cost: Can the team generate and maintain the file with little ongoing work?
2. Consumer: Is there an identified agent or workflow that actually reads it?
3. Content: Does the site have stable public documentation worth curating?

Implement when all three are credible. If the consumer is hypothetical, treat the file as an experiment and cap the maintenance budget.

Keep the roles separate

• robots.txt: crawler access guidance.
• XML sitemap: URL discovery for search engines.
• llms.txt: proposed curated reading guide for AI systems.
• MCP: runtime protocol through which an agent can discover and invoke resources or tools.

One file should not be evaluated as if it performs another system’s job.

Tools for llms.txt

• llms.txt Generator + Validator: Draft a curated file and check its structure and linked resources before publishing.
• AI-Crawler Access Checker: Audit actual crawler access controls separately; llms.txt is not a substitute for this check.
• curl: Verify status, redirects, content type, and body at the exact root path.
• Server or CDN logs: Determine whether named user agents request the file; absence is an observation, not proof of a ranking problem.
• A link checker: Revalidate every curated destination on a schedule so the guide does not decay.

Validate an llms.txt release

Test the root response

Test to run: Request /llms.txt directly without cookies and follow no redirects. Expected result: A successful response contains the intended Markdown at the root path. Failure interpretation: Routing or deployment does not expose the file. Monitoring window: Immediate. Rollback trigger: The path serves an HTML shell, error document, or unrelated redirect.

Test every listed resource

Test to run: Crawl the URLs in the file and compare each destination with its description. Expected result: Public canonical resources resolve successfully and match the stated purpose. Failure interpretation: The guide is stale or misleading. Monitoring window: Immediate, then on the documentation update cadence. Rollback trigger: A listed URL exposes private material or consistently fails.

Test the stated experiment outcome honestly

Test to run: Query server logs for /llms.txt by user agent after publication. Expected result: Requests, if any, are recorded and attributable; zero is also a valid result. Failure interpretation: Logging may be incomplete or consumers may not support the proposal. Monitoring window: Several normal crawl cycles. Rollback trigger: None for zero traffic alone; remove only when maintenance or exposure risk exceeds the file’s value.

Test yourself: llms.txt

Resources worth your time

The original source

• llmstxt.org — the spec.
• Jeremy Howard’s proposal post (Sept 3, 2024) — the rationale, straight from the creator.
• AnswerDotAI/llms-txt — the repo and tooling.

The data (read these before you implement anything)

• What Is llms.txt, and Should You Care About It? — Ahrefs — Ryan Law’s overview; the 28%-publish / 97%-zero-requests framing.
• The llms.txt adoption study — Ahrefs — Louise Linehan & Xibeijia Guan, 137,210 domains: who’s actually requesting the files (Claude Code dominates AI; retrieval bots barely register) and the prompt-injection finding.
• Does llms.txt matter? We tracked 10 sites — Search Engine Land — 180-day study; 8/10 sites showed no measurable change.
• The llms.txt experiment — OtterlyAI — 90 days, one site: 84 of 62,100+ AI bot visits hit /llms.txt.

The debate

• No, llms.txt is not the “new meta keywords” — Carolyn Shelby, Search Engine Land — the strongest defense of llms.txt.
• Meet llms.txt, a proposed standard — Search Engine Land — the pro/con summary and Brett Tabke’s critique.

My related writing on this site

• AI crawlers — controlling the bots (which is what llms.txt is not for), and where the access-control story actually lives.

From around the industry

• SE Ranking: The State of llms.txt — 300,000-domain XGBoost/SHAP study finding that removing llms.txt from the citation model improved accuracy; also has adoption-rate breakdown by traffic tier.
• Wix AI Search Lab — llms.txt myths (Crystal Carter, June 2026) — more optimistic read; 1,400+ files reviewed, argues llms.txt files themselves surface in AI results even if adoption is patchy.
• Google Says LLMs.Txt Comparable To Keywords Meta Tag — Search Engine Journal — full John Mueller quote and context from April 2025.
• presenc.ai: State of llms.txt 2026 — provider-by-provider support matrix; treat Anthropic/Perplexity “confirmed” claims with care (primary sources not always cited).
• The llms.txt is dead. More precisely: a dud. — Kai Spriestersbach (Medium) — aggregates OtterlyAI 90-day data, Gary Illyes confirmation, and the Google 24-hour incident into a skeptic’s summary.

Stats worth citing

• 97% of published llms.txt files got zero requests in a month (Ahrefs study, 137,210 domains, May 2026). Only ~3% saw any traffic at all. Source
• Claude Code is the dominant AI consumer of llms.txt — it “outfetched every AI retrieval bot, assistant, and training crawler except GPTBot” (the training bot). The retrieval bots that build citation indexes were just 1.1% of requests. Source
• 77% of bot requests to llms.txt files came from non-AI tools — SEO auditors, anonymous crawlers, tech-profiling bots — not AI at all. Source
• 84 of 62,100+ AI bot visits hit /llms.txt over 90 days on one monitored site (~0.1%), performing “3x worse than average pages.” Source
• Removing llms.txt from a citation model improved its accuracy — SE Ranking, 300,000 domains, XGBoost + SHAP analysis. No demonstrated citation benefit “at least not yet.” Source
• Adoption ≠ usage. BuiltWith counted 844,000+ sites with llms.txt (Oct 2025), but much of that is platform auto-deployment (Mintlify rolled it out to all hosted docs sites at once), not informed individual decisions.
• Only 408 requests targeted /llms.txt across 515 million+ AI bot traffic events analyzed over 90 days (limy.ai analysis) — described as “statistically negligible.”
• 10.13% of sites have adopted llms.txt across a 300,000-domain sample (SE Ranking study, 2026) — roughly 1 in 10, with no meaningful difference by traffic tier (low-traffic 9.88%, high-traffic 8.27%). Source

llms.txt

llms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it.

Related: AI Crawlers, Crawling

llms.txt

llms.txt is a proposed standard — not an adopted one — that places a curated Markdown file at /llms.txt on a website to help AI systems find and use that site’s content more efficiently. It was proposed by Jeremy Howard (Answer.AI / fast.ai) on September 3, 2024.

The file works like a hand-crafted table of contents: an H1 site name, a blockquote summary, and H2-grouped lists of links to the most important pages, each with a short note. The idea is that an AI agent reading the file can quickly find the right content without parsing the entire site. A companion /llms-full.txt flattens the whole site into one Markdown document.

Two things are easy to get wrong. First, llms.txt is not robots.txt for AI: robots.txt controls access and is enforced; llms.txt offers guidance and is purely advisory — it can’t block anything. Second, the major AI search providers haven’t adopted it. Google ignores it (Gary Illyes confirmed no support, no plans; John Mueller compared it to the keywords meta tag), and in an Ahrefs study of 137K sites, 97% of published llms.txt files received zero requests. The dominant real-world consumer is developer tooling — coding agents like Claude Code — not the search bots that build citation indexes.

Related: AI Crawlers, Crawling