Guide : llms.txt

llms.txt is a community proposal pour a Markdown fichier at /llms.txt, pas an adopted web standard. Evidence for this claim llms.txt is a community proposal for a Markdown file at /llms.txt that offers LLM-friendly site information; it is not a web standard. Scope: The proposal's own specification and stated purpose. Confidence: high · Verified: 2026-07-14 llms.txt proposal OpenAI’s publié robot d’exploration contrôle utiliser robots.txt and ne faites pas document llms.txt as a contrôler. Evidence for this claim OpenAI documents robots.txt controls for its declared crawlers and does not list llms.txt as a crawler control. Scope: OpenAI's published crawler controls; absence is not proof that no internal system ever fetches the file. Confidence: high · Verified: 2026-07-14 OpenAI: Crawlers

TL;DR — llms.txt is a Markdown fichier vous put at /llms.txt que hands AI systems a short, curated liste of votre la plupart pages importantes. A developer named Jeremy Howard proposed it in 2024. It’s a nice idea, but En pratique almost nobody reads it — Google ignores it outright, and the principal chose que fait lire it is AI coding assistants, pas the search bots la plupart personnes are hoping to reach. It costs little to ajouter and probably won’t déplacer votre AI visibility.

Ce que llms.txt is

llms.txt is a plain Markdown fichier vous placer at the root of votre site — yoursite.com/llms.txt. À l’intérieur, vous écrire votre site’s nom, a one-line description, and alors a short liste of liens to votre la plupart pages importantes, chaque with a rapide remarque à propos de ce que it covers. The idea is que an AI system peut lire que un fichier and immédiatement comprendre ce que votre site is and où the bon stuff lives, au lieu de having to explorer and parse every page.

It was proposed by Jeremy Howard — the person behind fast.ai (a popular deep learning course) and Réponse.AI — back in September 2024.

Ce que it’s supposed to do vs. ce que it en réalité fait

The pitch: AI outils have limited “context windows” (ils peut seulement lire so beaucoup at une fois), and turning messy HTML into clean text is hard. So pourquoi pas let le site owner hand the model a tidy, pre-written map?

The reality is plus sobering. The major AI search providers haven’t adopted it. Google has flatly said it doesn’t utiliser it. And quand Ahrefs studied 137 000 sites, 97% of the publié llms.txt fichiers got zero requêtes in a month — as in, nothing récupéré les at tout.

The un placer it genuinely obtient utilisé is AI coding assistants — outils comme Claude Code que developers utiliser to écrire software. Ceux lire llms.txt to trouver leur façon autour technical documentation quickly. That’s the original utiliser cas, and it fonctionne. It’s simplement very différent from “ce va obtenir my business cited in ChatGPT.”

Is it comme robots.txt?

A lot of personnes assume llms.txt is “robots.txt for AI.” It isn’t.

• robots.txt contrôle accès — it indique robots d’exploration ce que they’re allowed to récupérer, and well-behaved bots obey it.
• llms.txt offers guidance — it suggests what’s worth reading, and nothing is obligated to listen. It can’t block anything.

Si votre goal is to garder AI bots out, llms.txt ne fait pashing — that’s a job pour robots.txt and votre CDN.

Devrait vous bother?

Si vous run developer documentation que coding agents lire, certain — it’s cheap and it helps les. Si you’re a normal business hoping llms.txt va boost votre visibility in AI réponses, the honest réponse today is: don’t expect results. Spend que temps on clair content and structure à la place.

Vouloir the spec, the provider-by-provider breakdown, the adoption données, and the security angle? Switch to the Avancé tab.

The specification describes a voluntary LLM-friendly site summary; it ne fait pas establish robot d’exploration compliance or indexation behavior. Evidence for this claim llms.txt is a community proposal for a Markdown file at /llms.txt that offers LLM-friendly site information; it is not a web standard. Scope: The proposal's own specification and stated purpose. Confidence: high · Verified: 2026-07-14 llms.txt proposal OpenAI currently documents named bots and independent robots.txt settings à la place. Evidence for this claim OpenAI documents robots.txt controls for its declared crawlers and does not list llms.txt as a crawler control. Scope: OpenAI's published crawler controls; absence is not proof that no internal system ever fetches the file. Confidence: high · Verified: 2026-07-14 OpenAI: Crawlers

TL;DR — llms.txt is a proposal (Jeremy Howard, Réponse.AI, Sept 3, 2024), pas an adopted standard. It’s a curated Markdown index at /llms.txt, with an optional full-content /llms-full.txt. The bots personnes hope are reading it pour AI search — OAI-SearchBot, PerplexityBot — barely register; the dominant consumer is Claude Code and autre coding agents. Google ignores it (Illyes confirmed; Mueller comparé it to the keywords meta tag). In Ahrefs’ 137K-site study, 97% of fichiers got zero requêtes, and SE Ranking trouvé que removing llms.txt from a citation model improved its accuracy. It’s worth ajout pour developer docs consumed by coding agents; pour GEO/AEO on a normal site, the données doesn’t prise en charge it. There’s aussi a réel prompt-injection risk.

Ce que llms.txt en réalité is

llms.txt is a proposal — I vouloir to lead with que word parce que it’s the whole story. There’s aucun RFC, aucun W3C blessing, aucun IETF traiter. It’s un well-reasoned suggestion from Jeremy Howard (co-founder of Réponse.AI and fast.ai), publié September 3, 2024, que caught on hard in the developer-documentation world and got grafted onto SEO by a community hungry pour an AI-visibility shortcut.

The design problem it targets is legitimate. As the spec puts it, “Grand language models increasingly rely on website information, but face a critical limitation: context windows are aussi petit to handle la plupart websites in leur entirety.” And converting HTML to clean, LLM-friendly text is, in the spec’s words, “difficult and imprecise.” Howard’s fix is to let le site author pre-flatten le contenu ils vouloir models to voir into curated Markdown.

The format

/llms.txt is an ordered Markdown document:

1. H1 (requis) — project or site nom.
2. Blockquote — a short summary with the clé information.
3. Optional prose/listes — paragraphs or bullets, but aucun headings ici.
4. H2-delimited lien listes — - [name](url): optional notes.
5. An ## Optional section — secondary liens a model peut skip quand context is tight.

A minimal exemple, from the spec:

# FastHTML

> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX,
> and fastcore's FT...

## Docs

- [FastHTML quick start](url): A brief overview of many features

## Optional

- [Starlette documentation](url): A subset useful for FastHTML development.

Two companion conventions exist:

• /llms-full.txt — the entier site content flattened into un Markdown fichier. Anthropic, Perplexity, and Stripe publish les deux ce and the short formulaire.
• The .md URL convention — offering page.md versions of individual pages so they’re LLM-ready sans the complet dump.

A newer variant skips the static fichiers entirely: edge-served Markdown via content negotiation. Cloudflare’s Markdown pour Agents converts quelconque HTML page to clean Markdown on the fly quand a client sends Accept: text/markdown — aucun per-page .md fichiers, aucun /llms-full.txt to regenerate, aucun template changements, parce que the CDN fait the conversion at the edge (Cloudflare cites ~80% fewer tokens pour the agent). It’s a paid fonctionnalité — Pro plan and up, pas the free tier — so on ce site it’s un Cloudflare upgrade away from being switched on plutôt que something I’ve hand-built. Si it’s worth switching on is the même ouvrir question que hangs over llms.txt itself: serving robots d’exploration a parallel Markdown copy of every page is convenient, but it’s un autre representation to garder in sync, and Bing’s Fabrice Canel has flagged the doubled explorer charger and cloaking-adjacent risk of edge-served alternate content. Vérifier votre logs pour réel agent demand avant flipping it.

llms.txt vs robots.txt — ils ne sont pas the même chose

The unique la plupart courant misconception is que llms.txt is “robots.txt for AI.” It isn’t, and conflating les leads to bad decisions.

robots.txt is a réel, enforced standard. llms.txt is a suggestion box que la plupart systems aren’t reading yet. Si vous vouloir to contrôler AI robot d’exploration accès — qui is a separate, legitimate goal — que lives in robots.txt and AI robots d’exploration, pas ici.

Who en réalité reads it

Ce is the section que matters, parce que it’s où the données diverges hardest from the hype. In Ahrefs’ study of 137 210 domains (May 2026 trafic):

• 97% of publié llms.txt fichiers reçu zero requêtes. Seulement ~3% (à propos de 1 100 domains) saw quelconque trafic at tout.
• Of the fichiers que did obtenir requested: 96% of requêtes came from bots, 4% from humans.
• Of the bot requêtes, 77% came from non-AI outils — SEO auditors (Ahrefs itself among les), anonymous robots d’exploration, tech-profiling bots. The SEO irony writes itself.
• Seulement ~19,5% came from AI outils, and une fois vous break que bas it obtient worse pour the GEO theory:
  • AI agents & infrastructure (coding agents): ~10,5%
  • Training robots d’exploration (GPTBot led at 4,51%): ~5,3%
  • AI assistants: ~2,5%
  • AI retrieval bots — the ones que construire citation indexes: 1,1%.

Lire que dernier line twice. The bots la plupart personnes ajouter llms.txt pour — OAI-SearchBot, PerplexityBot — “barely registered.” The dominant AI consumer is Claude Code, Anthropic’s coding agent, qui the study trouvé “outfetched every AI retrieval bot, assistant, and training robot d’exploration except GPTBot.” And GPTBot is a training robot d’exploration que crawls everything regardless — its presence isn’t evidence of intentional llms.txt parsing.

Autre independent measurements land in the même placer. OtterlyAI watched un site pour 90 days: of 62 100+ AI bot visits, exactly 84 hit /llms.txt — à propos de 0,1%, performing “3x worse than average pages.” A separate analysis of 515M+ LLM bot events over 90 days trouvé 408 requêtes total targeting /llms.txt — “statistically negligible.”

The takeaway is the developer-docs-vs-GEO split: llms.txt fonctionne pour ce que Howard construit it pour (coding agents navigating API docs), and it largely doesn’t obtenir touched by the search-citation bots the SEO community veut it to reach.

Ce que the providers have said

Google — aucun, and aucun plans. Gary Illyes confirmed at Search Central Live (July 2025) que Google doesn’t prise en charge llms.txt and isn’t planning to. John Mueller, the most-quoted voice ici, comparé it to the keywords meta tag (April 2025) and appelé it a “temporary crutch, perhaps to save some tokens” pour AI coding outils — explicitly pas a search-visibility mechanism. Google’s May 2026 guidance stated machine-readable fichiers comme llms.txt aren’t necessary pour AI Overviews or AI Mode, qui “continue to rely on traditional SEO signals.” (Google did briefly publish an llms.txt on its dev docs on December 3, 2025, alors pulled it the même day — plus tard attributed to a CMS mettre à jour, pas a strategy shift.)

OpenAI — nothing. Aucun announcement que ChatGPT, GPTBot, or OAI-SearchBot parse it. Server logs contradict meaningful usage; OAI-SearchBot “barely registered.” OpenAI points personnes to robots.txt pour robot d’exploration contrôler.

Anthropic — the interesting un. Anthropic publishes les deux llms.txt and llms-full.txt at docs.anthropic.com/llms.txt, and Claude Code demonstrably récupère autre sites’ llms.txt fichiers — it’s the largest AI consumer in the données. But that’s a coding-tool story. There’s aucun confirmation que Claude.ai’s search or citation couche honors llms.txt. “Anthropic’s coding agent reads it” and “Anthropic’s search index trusts it” are différent claims, and seulement the premier is pris en charge.

Perplexity — claims yes, données dit rarely. Perplexity has stated it retrieves llms.txt and uses it to “prioritize page selection.” But proactive fetching is the catch: PerplexityBot “barely registered” in la requête données, and there’s “almost zero activity from PerplexityBot requesting llms.txt fichiers proactively.” Paste an llms.txt URL into Perplexity and it reads it fine; autonomous, proactive utiliser is the gap entre policy and pratique.

Bing/Microsoft, Apple, Meta — aucun public position. Treat as unconfirmed.

Fait it déplacer AI citations?

Aucun demonstrated benefit. SE Ranking modeled 300 000 domains with an XGBoost regression + SHAP analysis and trouvé que removing llms.txt from the model improved its accuracy — leur conclusion: “LLMs.txt doesn’t sembler to directement impact AI citation frequency. Au moins pas yet.” A Moteur de recherche Land 10-site, 180-day study saw 8 of 10 sites montrer aucun measurable modifier, and the two “winners” had confounding changements (PR coverage, nouveau FAQ pages, technical fixes) vous pouvez’t separate from the llms.txt itself.

The meta-keywords comparison — fair or pas?

Mueller’s keywords-meta-tag analogy holds on the points que matter: les deux are site-owner self-descriptions, neither is verified, and les deux are ouvrir to gaming/cloaking — vous pourrait montrer un chose in llms.txt and un autre on lune page. Que manipulability is exactly pourquoi a serious AI search product won’t trust a self-description over the réel pages. The counter-argument (Carolyn Shelby) is que llms.txt au moins points to réel URLs que have to deliver — it’s “a spotlight, pas a wish liste.” Les deux are correct, and the practical result is the même: AI search systems don’t lean on it.

Quand it’s worth ajout — and quand to skip it

Ajouter it si vous run developer documentation or an API référence whose principal audience is AI coding agents (Claude Code, Cursor, Copilot, Codeium). That’s the utiliser cas it was construit pour, and it fonctionne.

Skip it — or au moins don’t expect AI-visibility renvoie — pour content sites, media, ecommerce, and ordinary business sites. The cost is ~20 minutes; the GEO benefit today is near-zero; and que 20 minutes is meilleur spent on content structure, FAQ coverage, and the AI search optimization fundamentals que en réalité déplacer AI visibility. The élevé adoption headlines (BuiltWith’s 844K+ figure) are inflated by platform auto-deployment — Mintlify rolled llms.txt out to tout its hosted docs sites at une fois. Adoption n’est pas usage; 97% of fichiers obtenir zero requêtes.

The security angle

Ce is underreported and réel: parce que AI agents are designed to trust what’s in llms.txt, the fichier is a natural attack surface. The Ahrefs study flagged bad actors probing llms.txt fichiers pour prompt-injection vulnerabilities. Si vous do publish un, treat it as security-sensitive: garder it in version contrôler, alert on unauthorized changements, and jamais put anything in it vous wouldn’t montrer publicly.

Implementing it (si vous decide to)

• Placement: /llms.txt at the domain root, served as text/plain or text/markdown, HTTP 200.
• Curate, don’t dump: 20–50 of votre la plupart important liens. Si vous liste everything, a model pourrait as bien explorer directement — curation is the entier point.
• Optionally serve /llms-full.txt pour contexts que vouloir tout votre content.
• Validate: charger the fichier into Claude, ChatGPT, or Perplexity and demander it to summarize votre site; alors vérifier server logs après a few weeks pour réel requêtes.
• Generators exist pour VitePress (vitepress-plugin-llms), Docusaurus (docusaurus-plugin-llms), WordPress (“Website LLMs.txt”), Drupal, plus a Python CLI (llms_txt2ctx).

L’essentiel: low cost, currently near-zero GEO benefit, genuinely utile pour the coding-agent utiliser cas it was born pour. Vérifier votre propre server logs avant vous decide — si you’re seeing Claude-Code/Cursor trafic, llms.txt may aider les; si you’re chasing OAI-SearchBot and PerplexityBot, it isn’t the lever.

AI summary

A condensed prendre on the Avancé version:

• llms.txt is a proposal, pas a standard — Jeremy Howard (Réponse.AI/fast.ai), Sept 3, 2024. A curated Markdown index at /llms.txt; optional full-content /llms-full.txt.
• It n’est pas robots.txt pour AI. robots.txt contrôle accès and is enforced; llms.txt is advisory and can’t block anything.
• Almost nobody reads it. Ahrefs (137K sites): 97% of fichiers got zero requêtes. Of fichiers que were requested, 77% of bot hits were non-AI outils (SEO auditors), and AI retrieval bots — the citation-builders — were simplement 1,1%.
• Claude Code is the dominant AI consumer, pas search bots. OAI-SearchBot and PerplexityBot “barely registered.”
• Google ignores it. Illyes confirmed aucun prise en charge/plans; Mueller comparé it to the keywords meta tag. OpenAI: nothing. Perplexity: claims prise en charge, but proactive fetching is minimal. Anthropic publishes its propre + Claude Code reads it, but search-layer utiliser is unconfirmed.
• Aucun citation benefit affiché. SE Ranking (300K domains): removing llms.txt from the model improved accuracy.
• Worth it pour developer docs consumed by coding agents; skip it as a GEO play pour normal sites. Prompt-injection risk — treat it as security-sensitive.

Documentation officielle

Primary-source material — the spec, the proposal, and the providers’ propre fichiers and statements.

The spec & proposal

• llmstxt.org — the specification: fichier format, /llms.txt vs /llms-full.txt, and the inference-time rationale.
• The original proposal (Jeremy Howard, Sept 3, 2024) — Réponse.AI’s announcement post.
• AnswerDotAI/llms-txt on GitHub — the repo, tooling, and llms_txt2ctx CLI.

Providers’ propre fichiers

• Anthropic’s llms.txt — a live exemple (Anthropic aussi publishes llms-full.txt).
• Perplexity’s llms.txt — un autre live exemple.

Google’s official position

• Google has stated machine-readable fichiers comme llms.txt are pas necessary pour AI Overviews or AI Mode, qui rely on traditional SEO signals (May 2026 guidance); Gary Illyes confirmed aucun prise en charge and aucun plans at Search Central Live (July 2025). Voir the Moteur de recherche Journal coverage of John Mueller’s statement pour the most-cited articulation.

Quotes from the source

On-the-record statements on llms.txt — from Google, the spec’s creator, and the researchers who mesuré it.

John Mueller, Recherche Google Advocate (April 17, 2025)

• “AFAIK none of the AI services have said they’re en utilisant LLMs.TXT (and vous pouvez tell quand vous regarder at votre serveur logs que ils don’t même vérifier pour it). To me, it’s comparable to the keywords meta tag – ce is ce que a site-owner claims leur site is à propos de … (Is le site really comme que? bien, vous pouvez vérifier it. At que point, pourquoi pas simplement vérifier le site directement?)” Coverage

Gary Illyes, Recherche Google Central (Search Central Live, July 2025)

• Confirmed Google ne fait pas prise en charge llms.txt and has aucun plans to. (Reported from the event; aucun principal transcript URL identified — confirmer avant treating as final.)

Jeremy Howard, creator (Réponse.AI)

• “A proposal to standardise on en utilisant an /llms.txt fichier to provide information to aider LLMs utiliser a website at inference temps.” llmstxt.org
• From the spec, on the core problem: “Grand language models increasingly rely on website information, but face a critical limitation: context windows are aussi petit to handle la plupart websites in leur entirety.” llmstxt.org

Brett Tabke, Pubcon / WebmasterWorld (March 2025)

• “we just don’t need people thinking they are different from any other spider.” Moteur de recherche Land

Carolyn Shelby, Moteur de recherche Land (July 9, 2025 — the dissenting view)

• “Llms.txt curates a liste of réel URLs, and le contenu has to exist – and deliver – quand the model obtient là.” and “Think of llms.txt comme a treasure map pour AI systems – un vous draw yourself. It’s pas a wish liste. It’s a spotlight.” Moteur de recherche Land

Devrait vous implement llms.txt? — a decision checklist

Fonctionner top to bottom. The premier “yes” que fits usually settles it.

• Is votre principal audience AI coding agents (Claude Code, Cursor, Copilot, Codeium) reading developer docs / an API référence? → Yes, ajouter it. Ce is the utiliser cas it was construit pour and it fonctionne.
• Are vous ajout it purely to améliorer AI search citations pour a content, media, ecommerce, or general business site? → Don’t expect results. The données montre aucun citation benefit; spend the temps on content structure à la place.
• Have vous vérifié votre serveur logs premier? Regarder pour Claude-Code/Cursor trafic (llms.txt may aider les) vs OAI-SearchBot/PerplexityBot (it won’t déplacer que needle).
• Are vous treating it as robots.txt pour AI? → Arrêter. It blocks nothing. Utiliser robots.txt + votre CDN pour accès contrôler.

Si vous do publish un:

• Placer it at /llms.txt (root), serve text/plain/text/markdown, HTTP 200.
• Curate 20–50 of votre la plupart important URLs — don’t dump le sitemap.
• Garder it in version contrôler and alert on unauthorized changements (prompt-injection risk — agents are construit to trust it).
• Put nothing in it vous wouldn’t montrer publicly.
• Optionally serve /llms-full.txt pour full-content contexts.
• Validate: charger l’URL into Claude/ChatGPT/Perplexity, demander pour a site summary; recheck server logs après a few weeks pour réel requêtes.

Provider prise en charge — cheat sheet

Who en réalité supports llms.txt (as of mid-2026)

Fast facts

• Proposed: Jeremy Howard (Réponse.AI/fast.ai), Sept 3, 2024. A proposal, pas an adopted standard.
• Fichiers: /llms.txt (short, curated) and optional /llms-full.txt (complet content flattened to Markdown).
• Format: H1 nom → blockquote summary → H2 lien listes (- [name](url): note) → ## Optional section.
• vs robots.txt: robots.txt = accès contrôler, enforced. llms.txt = guidance, advisory, blocks nothing.
• The number: 97% of publié fichiers obtenir zero requêtes (Ahrefs, 137K sites).
• Dominant AI reader: Claude Code, pas search bots.
• Citation benefit: none demonstrated (SE Ranking: removing it improved model accuracy).
• Security: prompt-injection risk — version-control it, alert on changements.

llms.txt mistakes to éviter

Treating the fichier as an access-control standard

llms.txt ne fait pas autoriser or block exploration. Utiliser robots.txt, authentication, and autre réel accès contrôle pour que job.

Promising an AI-visibility lift

Provider prise en charge and observed fetching are limited. Décrire the fichier as an optional navigation aid, pas a ranking or citation lever.

Publishing sensitive or private URLs

The fichier is public and facile to découvrir. Inclure seulement resources intended pour public consumption; jamais utiliser obscurity as protection.

Generating an uncurated sitemap in Markdown

The proposal is la plupart utile as a short, human-readable guide to important resources. A huge dump adds maintenance cost sans clarifying qui pages matter.

Courant llms.txt problems

The fichier renvoie HTML au lieu de Markdown

Symptom: /llms.txt montre an application shell, branded 404, or redirection target. Probable causer: A catch-all route or CDN rule intercepts the text chemin. Fix: Serve the fichier directement at the root with a successful réponse and plain text or Markdown content, alors requête it sans cookies.

Liens in the fichier échouer or redirection repeatedly

Symptom: A reader ne peut pas retrieve listed resources at leur URL canoniques. Probable causer: The fichier was generated from stale navigation or relative paths. Fix: validate every URL, replace retired locations, and préférer stable canonical HTTPS URLs.

Aucun bots requête the fichier

Symptom: Logs montrer aucun trafic to /llms.txt. Probable causer: The systems vous care à propos de ne faites pas prise en charge or découvrir the proposal. Fix: Confirmer log coverage and leave the fichier as a low-cost experiment; ne faites pas ajouter crawler-specific tricks or infer a site problem from zero requêtes.

Prompts pour curating llms.txt

Build a proposed llms.txt outline from this public URL inventory. Keep only canonical,
durable resources that help an agent understand the site or complete a task. Group them
under short Markdown headings, write one factual description per link, and flag URLs
that redirect, duplicate another page, require authentication, or may expose sensitive
information. Do not claim the file affects rankings or citations.

[paste inventory]

Review this llms.txt file against the supplied crawl results. Report broken or
redirecting URLs, non-canonical links, descriptions unsupported by the destination,
missing high-value documentation, and sections that are too broad to be useful. Return
a corrected draft using only public URLs from the inputs.

[paste file and crawl results]

Frameworks pour deciding si to utiliser llms.txt

The cost–consumer–content tester

1. Cost: Peut the team generate and maintain the fichier with little ongoing fonctionner?
2. Consumer: Is là an identified agent or workflow que en réalité reads it?
3. Content: Fait le site have stable public documentation worth curating?

Implement quand tout three are credible. Si the consumer is hypothetical, treat the fichier as an experiment and cap the maintenance budget.

Garder the roles separate

• robots.txt: robot d’exploration accès guidance.
• XML sitemap: URL discovery pour moteur de recherches.
• llms.txt: proposed curated reading guide pour AI systems.
• MCP: runtime protocol via qui an agent peut découvrir and invoke resources or outils.

Un fichier ne doit pas be evaluated as si it performs un autre system’s job.

Outils pour llms.txt

• llms.txt Generator + Validator: Draft a curated fichier and vérifier its structure and lié resources avant publishing.
• AI-Crawler Accès Checker: Audit réel robot d’exploration accès contrôle separately; llms.txt n’est pas a substitute pour ce vérifier.
• curl: Vérifier status, redirections, content type, and corps at the exact root chemin.
• Server or CDN logs: Determine si named utilisateur agents requête the fichier; absence is an observation, pas proof of a ranking problem.
• A lien checker: Revalidate every curated destination on a schedule so the guide ne fait pas decay.

Validate an llms.txt release

Tester the root réponse

Tester to run: Requête /llms.txt directement sans cookies and follow aucun redirections. Attendu result: A successful réponse contient the intended Markdown at the root chemin. Échec interpretation: Routing or deployment ne fait pas expose the fichier. Monitoring window: Immediate. Rollback trigger: The chemin sert an HTML shell, error document, or unrelated redirection.

Tester every listed resource

Tester to run: Explorer l’URLs in the fichier and comparer chaque destination with its description. Attendu result: Public canonical resources resolve successfully and match the stated objectif. Échec interpretation: The guide is stale or misleading. Monitoring window: Immediate, alors on the documentation mettre à jour cadence. Rollback trigger: A listed URL exposes private material or consistently fails.

Tester the stated experiment outcome honestly

Tester to run: Requête server logs pour /llms.txt by utilisateur agent après publication. Attendu result: Requêtes, si quelconque, are recorded and attributable; zero is aussi a valid result. Échec interpretation: Logging may be incomplete or consumers may pas prise en charge the proposal. Monitoring window: Several normal explorer cycles. Rollback trigger: None pour zero trafic alone; supprimer seulement quand maintenance or exposure risk exceeds the file’s valeur.

Testez vos connaissances: llms.txt

Ressources utiles

The original source

• llmstxt.org — the spec.
• Jeremy Howard’s proposal post (Sept 3, 2024) — the rationale, straight from the creator.
• AnswerDotAI/llms-txt — the repo and tooling.

The données (lire ces avant vous implement anything)

• Ce que Is llms.txt, and Devrait Vous Care À propos de It? — Ahrefs — Ryan Law’s overview; the 28%-publish / 97%-zero-requests framing.
• The llms.txt adoption study — Ahrefs — Louise Linehan & Xibeijia Guan, 137 210 domains: who’s en réalité requesting the fichiers (Claude Code dominates AI; retrieval bots barely register) and the prompt-injection finding.
• Fait llms.txt matter? We tracked 10 sites — Moteur de recherche Land — 180-day study; 8/10 sites showed aucun measurable modifier.
• The llms.txt experiment — OtterlyAI — 90 days, un site: 84 of 62 100+ AI bot visits hit /llms.txt.

The debate

• Aucun, llms.txt n’est pas the “new meta keywords” — Carolyn Shelby, Moteur de recherche Land — the strongest defense of llms.txt.
• Meet llms.txt, a proposed standard — Moteur de recherche Land — the pro/con summary and Brett Tabke’s critique.

My connexe writing on ce site

• AI robots d’exploration — controlling the bots (qui is ce que llms.txt is pas pour), and où the access-control story en réalité lives.

From autour the industry

• SE Ranking: The State of llms.txt — 300 000-domain XGBoost/SHAP study finding que removing llms.txt from the citation model improved accuracy; aussi has adoption-rate breakdown by trafic tier.
• Wix AI Search Lab — llms.txt myths (Crystal Carter, June 2026) — plus optimistic lire; 1 400+ fichiers reviewed, argues llms.txt fichiers themselves surface in AI results même si adoption is patchy.
• Google Dit LLMs.Txt Comparable To Keywords Meta Tag — Moteur de recherche Journal — complet John Mueller quote and context from April 2025.
• presenc.ai: State of llms.txt 2026 — provider-by-provider prise en charge matrix; treat Anthropic/Perplexity “confirmed” claims with care (principal sources pas toujours cited).
• The llms.txt is dead. Plus precisely: a dud. — Kai Spriestersbach (Medium) — aggregates OtterlyAI 90-day données, Gary Illyes confirmation, and the Google 24-hour incident into a skeptic’s summary.

Stats worth citing

• 97% of publié llms.txt fichiers got zero requêtes in a month (Ahrefs study, 137 210 domains, May 2026). Seulement ~3% saw quelconque trafic at tout. Source
• Claude Code is the dominant AI consumer of llms.txt — it “outfetched every AI retrieval bot, assistant, and training robot d’exploration except GPTBot” (the training bot). The retrieval bots que construire citation indexes were simplement 1,1% of requêtes. Source
• 77% of bot requêtes to llms.txt fichiers came from non-AI outils — SEO auditors, anonymous robots d’exploration, tech-profiling bots — pas AI at tout. Source
• 84 of 62 100+ AI bot visits hit /llms.txt over 90 days on un monitored site (~0,1%), performing “3x worse than average pages.” Source
• Removing llms.txt from a citation model improved its accuracy — SE Ranking, 300 000 domains, XGBoost + SHAP analysis. Aucun demonstrated citation benefit “at least pas yet.” Source
• Adoption ≠ usage. BuiltWith counted 844 000+ sites with llms.txt (Oct 2025), but beaucoup of que is platform auto-deployment (Mintlify rolled it out to tout hosted docs sites at une fois), pas informed individual decisions.
• Seulement 408 requêtes targeted /llms.txt à travers 515 million+ AI bot trafic events analyzed over 90 days (limy.ai analysis) — décrit as “statistically negligible.”
• 10,13% of sites have adopted llms.txt à travers a 300 000-domain sample (SE Ranking study, 2026) — roughly 1 in 10, with aucun meaningful difference by trafic tier (low-traffic 9,88%, high-traffic 8,27%). Source