Tokens and Context Windows

TL;DR — A token is the little piece of text an AI reads — usually a chunk of a word, roughly ¾ of an English word. A context window is how much text an AI can hold in its head at once (its short-term memory). Both matter for search because search systems do not routinely hand a model every page or an entire website. They usually grab only the relevant parts before they answer.

What a token is

Language models process text as tokens, which may be whole words or smaller character sequences depending on the tokenizer. Evidence for this claim OpenAI models process text as tokens, and token boundaries may be whole words or parts of words. Scope: OpenAI tokenization; token counts depend on the model-compatible encoding. Confidence: high · Verified: 2026-07-14 OpenAI: What are tokens? A context window is a model-specific limit on the tokens available to a request and response, and published limits can change by model version. Evidence for this claim Context-window limits are documented per model and can differ across model versions. Scope: OpenAI model metadata; published limits are product-specific and can change. Confidence: high · Verified: 2026-07-14 OpenAI: Models

Large language modelsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). — the tech behind ChatGPT, Gemini, and AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. — don’t read text the way you do, word by word. They break it into tokens: small units that are often part of a word rather than a whole one. Google’s rule of thumb for its Gemini models is that one token is about 4 characters, and 100 tokens is roughly 60–80 English words. So a token is a bit less than a full word on average.

The process of chopping text into tokens is called tokenization, and the full set of tokens a model knows is its vocabulary. Common words might be a single token; longer or unusual words get split into several. And it’s not just text — Gemini tokenizes images, audio, and video too.

What a context window is

A context window is the most tokens a model can work with in one go. Think of it as short-term memory — that’s Google’s own analogy. Google describes that space as one shared pool covering the instructions the system gives the model, the back-and-forth of your conversation, any documents that got pulled in, your question, and the model’s answer — though the exact split isn’t identical everywhere; some providers cap how much output you can get back separately from the overall window, so check the specifics for whatever model you’re actually using.

The important part: once something falls outside the usable window, it’s gone — not slowly forgotten, just invisible, as if it was never there. The model can’t reach back for it unless the system deliberately re-feeds it, or the product instead rejects, truncates, or compacts your request before the model even sees it.

Why this matters for search

Here’s the connection people miss. Even when a page could fit inside a large context window, handing over every page or an entire site is expensive and usually irrelevant to the question. So when an AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. tool answers, it generally selects and chunks the most relevant passages first, then feeds only those to the model.

That’s the whole reason you hear advice like “put your answer up front” and “keep sections self-contained.” It’s not a style preference — it’s because the AI is working with a piece of your page, not the page. If a section only makes sense in the context of the rest of the page, it can lose its meaning the moment it’s pulled out on its own.

The thing people get wrong

A giant context window does not mean an AI reads your whole site. You’ll hear that Gemini can take in a million tokens, and think “great, it’ll read everything I publish.” It won’t. In a search or AI Overview scenario, the system still picks and chunks relevant passages before anything reaches the model — the huge window changes what’s possible in principle, not what a search engine actually feeds itself for your query.

And you don’t need to pre-cut your content into tiny token-sized blocks. Google says outright there’s no ideal page length and no requirement to break content into tiny pieces for AI. Write for people; the chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking. happens on the machine’s side.

Want the real mechanics — how tokenization works, how big context windows have gotten and why there’s a ceiling, and what “Lost in the Middle” means for your content? Switch to the Advanced tab.

TL;DR — A token is the unit an LLMA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). processes — a sub-word fragment, ~4 characters, with 100 tokens ≈ 60–80 English words (Google). Tokenization splits all input and output — text, images, audio, video — into tokens; the model’s known set is its vocabulary. A context window is the total token budget shared by input (system prompt + history + retrieved docs + your query) and output (the response) — Google’s analogy is short-term memory. Content outside the window is invisible, not gradually forgotten. Windows scaled from ~2K tokens to 1M+ (Gemini), with a hardware ceiling (“thermal limit” of TPUs). The reason this matters for search: systems manage cost and relevance by selecting and chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking. before an LLM sees your content — and “Lost in the Middle” means even what is in the window isn’t used evenly. A bigger window does not mean AI reads your whole site.

What a token actually is

Tokenization maps text into integer token IDs using a model-compatible encoding; token boundaries are not the same as word boundaries. Evidence for this claim OpenAI models process text as tokens, and token boundaries may be whole words or parts of words. Scope: OpenAI tokenization; token counts depend on the model-compatible encoding. Confidence: high · Verified: 2026-07-14 OpenAI: What are tokens? Context-window sizes are product and model metadata, not a permanent property of all language models. Evidence for this claim Context-window limits are documented per model and can differ across model versions. Scope: OpenAI model metadata; published limits are product-specific and can change. Confidence: high · Verified: 2026-07-14 OpenAI: Models

Google is blunt about the granularity: “Gemini and other generative AI models process input and output at a granularity called a token.” A token is not a word and not a character — it’s a fragment. As Google’s docs put it, “Long words are broken up into several tokens. The set of all tokens used by the model is called the vocabulary, and the process of splitting text into tokens is called tokenization.”

The citable rule of thumb, straight from Google: “For Gemini models, a token is equivalent to about 4 characters. 100 tokens is equal to about 60-80 English words.” So roughly ¾ of a word per token, on average — but average is doing work there. Punctuation, non-English text, code, and unusual or long words tokenize less efficiently (more tokens per word), which is exactly why counting words is a poor way to estimate token usage.

Tokens also aren’t just text. Google: “All input to and output from the Gemini API is tokenized, including text, image files, and other non-text modalities.” Images, audio, and video frames all become tokens too. Google DeepMind’s engineers describe the context-window measure as “how many tokens — the smallest building blocks, like part of a word, image or video — that the model can process at once.”

What a context window is

A context window is the maximum number of tokens a model can hold and reason over in a single interaction. Google’s framing is deliberately simple: “An analogy for the context window is short term memory.”

The key mechanical point is that it’s one shared budget. Everything competes for the same space:

• the system prompt (the instructions the application gives the model),
• the conversation history so far,
• any retrieved or injected documents (the passages a search/RAGRAG is the retrieve-then-generate pattern behind AI search: the system retrieves relevant passages from an external index at query time, injects them into the model's context, and generates an answer grounded in those sources — without changing the model's weights. system pulled in),
• your input (the current query), and
• the model’s own output (the answer it generates).

Google’s Gemini docs describe input and output drawing from one shared pool. That exact accounting isn’t universal, though: other providers publish a separate maximum- output figure alongside the context window rather than treating it as one undivided budget — Anthropic’s model docs, for instance, list a “context window” size and a distinct “max output” cap side by side for each Claude model. Evidence for this claim Context-window limits are documented per model and can differ across model versions. Scope: OpenAI model metadata; published limits are product-specific and can change. Confidence: high · Verified: 2026-07-14 OpenAI: Models Check the documentation for the specific model and product you’re actually using rather than assuming one formula applies everywhere.

Whatever the exact accounting, once content falls outside the usable window, it isn’t gradually forgotten — it’s simply invisible, as if it never existed, unless a system explicitly summarizes it and re-injects a shorter version, or the product instead rejects, truncates, or compacts the request before the model ever sees it. That “it’s gone, not fading” behavior is the part that trips people up when they assume an AI “remembers” a long conversation the way a person would.

How big context windows are — and how fast that changed

The scale-up here has been dramatic. Early generative models handled only a couple thousand tokens; the progression ran through roughly 8K, then 32K, then 128K, before the jump Google flags in its docs: “Gemini is the first model capable of accepting 1 million tokens.”

There’s a physical ceiling, though. Google DeepMind pushed to experimental 10-million- token windows, and research scientist Nikolay Savinov — one of the research leads on the long-context project, who originally targeted 128,000 tokens before landing on 1 million — put the limit plainly: “10 million tokens at once is already close to the thermal limit of our Tensor Processing Units.” Bigger context isn’t free — it costs memory, compute, and literally heat. (Google DeepMind’s Denis Teplyashin has described the same cascade from 128K to 512K to 1M to 10M, each step opening new possibilities but running into harder engineering limits; research scientist Machel Reid has described what teams actually do with that space, like feeding an entire codebase or a 45-minute film into a single prompt.)

Two consequences worth internalizing:

1. The numbers you see are moving targets. “The largest context window” is a figure that keeps changing; don’t hard-code strategy to a specific token count.
2. Bigger ≠ automatically better. More capacity is not more comprehension — see “Lost in the Middle” below.

What a million tokens actually looks like

Abstract token counts are hard to feel, so Google offers concrete sizing for what 1M tokens can hold:

• “50,000 lines of code (with the standard 80 characters per line),”
• “All the text messages you have sent in the last 5 years,”
• “8 average length English novels,”
• “Transcripts of over 200 average length podcast episodes.”

That’s genuinely enormous — and it’s exactly why the “so the AI just reads my whole site” assumption feels reasonable and is still wrong. Which brings us to the search angle.

Why tokens and context windows matter for AI search

This is the core of it for anyone doing SEO or content. No full page — and certainly no full site — is reliably handed to a model whole in a search scenario. Even when a page fits technically, feeding all of it may be uneconomical or irrelevant to the query. That is why retrieval systems — AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index., Copilot, RAG pipelines, chatbots with browsing — select, chunk, and pass only some of a page’s content to the model. (No search vendor publishes exactly what it retrieves per query, so this is inference from how retrieval architectures work generally and from public statements like the ones below — not a claim that every AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. product behaves identically.)

The token/context-window limit is the mechanical reason behind a whole stack of AI- search behavior you already know about on this site:

• It’s why chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking. exists — content gets split into retrievable passages because the whole thing won’t fit, and Microsoft’s own Azure guidance says partitioning large documents into smaller chunks “can help you stay under the maximum token input limits of chat completion and embeddingEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. models.”
• It’s why retrieval (the “R” in RAG) has to pick a handful of passages before generation — the model can only be handed what fits in its budget.
• It’s why embedding models have their own token caps (many top out around a few thousand tokens per input), so long passages have to be split before they’re even turned into vectors.

The industry’s own framing of the problem is explicit. The llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. spec opens with: “Large language models increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety.” That’s the whole motivation for the format — even if, as I’ll cover in the myths section, Google’s John Mueller has called llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. at best a token-saving “crutch” for coding tools, not a search-visibility mechanism.

”Lost in the Middle” — even what’s in the window isn’t used evenly

Here’s the subtler point. Getting your content into the window isn’t the finish line. The Stanford “Lost in the Middle” study (Liu et al., 2023) foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. that on the language models and tasks they tested, models used information at the start and end of their context far better than information buried in the middle. That’s a position effect measured on 2023-era models — not proof every current model always ignores the middle — but it’s held up as a general caution: stuffing more tokens in doesn’t guarantee better answers, and content that leads with its point survives both retrieval and in-context attention better than content that buries it.

The economics: why AI systems are selective on purpose

There’s a business-model reason retrieval systems chunk aggressively rather than feed whole pages. Google notes that “when billing is enabled, the cost of a call to the Gemini API is determined in part by the number of input and output tokens” — and output tokens typically cost several times more than input tokens. Tokens are literally metered. That gives AI systems a strong economic incentive to retrieve and pass only what’s necessary, which reinforces the same conclusion from a different direction: selective, passage-level retrieval isn’t a temporary limitation to wait out — it’s how these systems are designed to work.

Common myths, debunked

• “A 1-million-token context window means the AI reads my whole website at once.” No. In a search or RAG scenario, retrieval still selects and chunks content before it ever reaches the model’s window. A huge window changes what’s possible in principle, not what a search engine or AI Overview actually feeds itself per query. Google’s line is directly on point: “There’s no ideal page length, and in the end, make pages for your audience, not just for generative AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity..”

• “I need to write in exactly 200-word, token-sized chunks.” No. Google: “There’s no requirement to break your content into tiny pieces for AI to better understand it.” Chunking happens on the system side; your job is clear, well- structured content, not manual token accounting.

• “More tokens = a smarter model / better answers.” No. A larger window increases capacity, not comprehension — and “Lost in the Middle” shows models use the start and end of their context better than the middle. Bigger isn’t automatically better.

• “Tokens = words, so I can just count words to estimate usage.” Roughly, but not reliably. 100 tokens ≈ 60–80 English words is Google’s average, but punctuation, code, non-English text, and unusual words tokenize less efficiently — which matters when you’re estimating cost or budget.

• “llms.txt solves the context-window problem for my site.” The problem is real — it’s the spec’s own stated motivation — but Google doesn’t use llms.txt for search, and Mueller has framed it as a token-saving crutch for AI coding tools, not an SEO fix. The mitigations that actually matter are the same fundamentals: be crawlable and indexed, and write clear, retrieval-friendly content.

What this means for content and SEO

Strip the jargon and the takeaways are concrete — and they’re the same discipline that shows up across this cluster:

• Front-load the answer. Both retrieval and in-context attention favor the start (and end) of what a model sees. Lead with your point.
• Keep sections self-contained. Because a passage may be pulled out of your page and handed to a model on its own, it should stand on its own. This is the same logic behind good H2/H3 structure and clear topic sentences.
• Don’t obsess over manual chunking. No ideal page length, no requirement to fragment. Write for humans; the system chunks.
• “Bigger context window” ≠ “AI reads my whole site.” Optimize for being found and selected, not for being ingested whole.

Tokens and context windowsA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. are the low-level plumbing under most of this cluster: they’re the reason groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. and retrieval have to select passages, the reason chunking prepares text the way it does, the constraint embeddingsEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. and semantic and vector searchVector search finds content by comparing the meaning of a query against stored content as numerical vectors, retrieving the closest ones in a high-dimensional embedding space. At scale it uses approximate nearest neighbor (ANN) algorithms — not exact comparison — to search billions of vectors in milliseconds. work within, and the limit passage rankingPassage ranking is a Google AI system that scores individual sections ('passages') of a page so a single page can earn multiple relevance scores for different queries. Google still indexes whole pages — only the ranking changed. exists to work around. And they sit right next to the model’s other hard boundary — its knowledge cutoffA knowledge cutoff (or training data cutoff) is the date after which an LLM was no longer trained on new data. The model has no awareness of anything that happened later — unless the system bolts on live retrieval (RAG/web search) to fetch it at query time. — as one of the core limitations of any LLM.

AI summary

A condensed take on the Advanced version:

• A token = the unit an LLMA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). processes — a sub-word fragment, ~4 characters, with 100 tokens ≈ 60–80 English words (Google). Tokenization splits all input and output — text, images, audio, video — into tokens; the model’s known set is its vocabulary.
• Word counts estimate tokens poorly — punctuation, code, non-English text, and unusual words tokenize less efficiently (more tokens per word).
• A context window = the total token budget shared by input (system prompt + history + retrieved docs + query) and output. Google’s analogy: short-term memory.
• Content outside the window is invisible, not gradually forgotten — gone unless a system summarizes and re-injects it.
• Windows scaled fast: ~2K → 8K/32K/128K → 1M+ (Gemini), with experimental 10M near the “thermal limit” of Google’s TPUs (Nikolay Savinov). The numbers keep moving.
• Bigger ≠ better: more capacity isn’t more comprehension. “Lost in the Middle” — models use the start and end of their context better than the middle.
• Why it matters for search: no full page or site fits in one window, so retrieval systems (AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index., Copilot, RAGRAG is the retrieve-then-generate pattern behind AI search: the system retrieves relevant passages from an external index at query time, injects them into the model's context, and generates an answer grounded in those sources — without changing the model's weights.) select and chunk before an LLM ever sees your content. It’s the mechanical reason chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking., retrieval, and embeddingEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. token caps exist.
• Economics reinforce it: tokens are metered (output usually costs more than input), so systems are designed to retrieve selectively, not feed whole pages.
• Myths busted: a 1M window doesn’t mean AI reads your whole site; no need for manual token-sized chunks (Google says no ideal length, no requirement to fragment); more tokens ≠ smarter; llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. doesn’t solve it for search.
• Content upshot: front-load the answer, keep sections self-contained, don’t obsess over manual chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking., optimize for being selected, not ingested whole.

Official documentation

Primary-source documentation on tokens and context windowsA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory..

Google

• Understand and count tokens (Gemini API) — what a token is, the ~4-characters / 100-tokens-≈-60-80-words rule of thumb, tokenization and vocabulary, and multimodal tokenization.
• Long context (Gemini API) — the short-term-memory analogy, the 1-million-token milestone, the historical progression, and the concrete “what a million tokens looks like” examples.
• What is a long context window? Google DeepMind engineers explain (Google Blog) — DeepMind engineers on tokens as the smallest building blocks and the hardware ceiling.
• Gemini Developer API pricing — token-based billing; input vs. output token costs.
• Optimizing your website for generative AI features (Google Search Central) — “no ideal page length” and “no requirement to break your content into tiny pieces.”

Microsoft / Azure

• Chunk large documents for vector search (Azure AI Search) — chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking. to stay under the maximum token input limits of chat-completion and embeddingEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. models.
• What are AI Tokens? (Microsoft Copilot help) — Microsoft’s consumer-facing explainer of tokens and how token limits cap how much you can ask about at once.

Industry problem statement

• llms.txt spec (llmstxt.org) — frames the context-window limit as the origin problem: windows “too small to handle most websites in their entirety.”

Quotes from the source

On-the-record statements from Google and Microsoft, plus the industry framing of the context-window problem. Each link is a deep link to the quoted passage where the source page supports one.

Google — what a token is

• “Gemini and other generative AI models process input and output at a granularity called a token.” — Google AI for Developers, Understand and count tokens. Jump to quote
• “For Gemini models, a token is equivalent to about 4 characters. 100 tokens is equal to about 60-80 English words.” Jump to quote
• “Long words are broken up into several tokens. The set of all tokens used by the model is called the vocabulary, and the process of splitting text into tokens is called tokenization.” Jump to quote

Google — the context window

• “An analogy for the context window is short term memory.” — Google AI for Developers, Long context. Jump to quote
• “Gemini is the first model capable of accepting 1 million tokens.” Jump to quote

Google DeepMind engineers — tokens and the hardware ceiling (via Google Blog)

• “measures how many tokens — the smallest building blocks, like part of a word, image or video — that the model can process at once.” Jump to quote
• “10 million tokens at once is already close to the thermal limit of our Tensor Processing Units.” — Nikolay Savinov, Research Scientist, Google DeepMind. Read the coverage

Google — no ideal page length, no requirement to fragment

• “There’s no ideal page length, and in the end, make pages for your audience, not just for generative AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity..” — Google Search Central, Optimizing your website for generative AI features. Read the guide
• “There’s no requirement to break your content into tiny pieces for AI to better understand it.” — Google Search Central, same guide. Read the guide

Microsoft — chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking. to stay under token limits

• “Partitioning large documents into smaller chunks can help you stay under the maximum token input limits of chat completion and embeddingEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. models.” — Microsoft, Azure AI SearchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. documentation. Read the source

The industry problem statement — llms.txtllms.txt is a proposed (not adopted) Markdown file at /llms.txt that gives AI systems a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, it's read mostly by coding agents like Claude Code — not search crawlers — and Google ignores it. spec

• “Large language modelsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety.” — llmstxt.org spec. Jump to quote

Tokens and context windows — cheat sheet

What each term is in one line

Token ↔ word rule of thumb (Google, Gemini)

(Caveat: averages only — punctuation, code, and non-English text use more tokens per word.)

How context windows grew

What 1 million tokens can hold (Google’s examples)

• ~50,000 lines of code (80 chars/line)
• 5 years of your text messages
• 8 average-length English novels
• Transcripts of 200+ podcast episodes

Fast facts

• Context window = one shared budget for system prompt + history + retrieved docs + query + output.
• Content outside the window is invisible — gone, not gradually forgotten.
• Bigger window ≠ better answers — “Lost in the Middle”: start/end beat the middle.
• In search, retrieval chunks and selects before the model sees content — a 1M window doesn’t mean AI reads your whole site.
• Google: no ideal page length, no requirement to fragment content for AI.
• Tokens are metered/billed (output usually costs more than input) — the economic reason systems retrieve selectively.

The mental models

1. Token = the model’s unit of reading, not the word. LLMsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). read in sub-word fragments (~4 characters, ~¾ of a word). When you estimate cost or budget, count tokens, not words — punctuation, code, and non-English text inflate the ratio.

2. The context window is short-term memory — a shared budget. System prompt, history, retrieved documents, your query, and the model’s output all draw from one pool. Anything outside it is invisible, not fading. If a model seems to “forget,” ask what fell out of the window.

3. Bigger window ≠ more understanding. Capacity and comprehension are different axes. “Lost in the Middle” means the start and end of the context get used better than the middle — so lead with your point, don’t just add more.

4. No page fits — so retrieval selects first. The whole reason AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. chunks and retrieves is that a page (let alone a site) doesn’t fit, or isn’t economical to feed, whole. The model never sees your page; it sees the passages a retrieval system chose. Optimize to be selected, not ingested whole.

5. The decision rule for content. Don’t manually cut content into token-sized blocks (Google says you don’t need to). Do front-load answers and write self-contained sections — because both retrieval and in-context attention reward content that stands on its own and leads with its point.

Context-window mistakes

Treating words and tokens as interchangeable

Tokenization varies by model, language, punctuation, and code. Use the tokenizer for the actual model when a hard limit matters; a word-count ratio is only planning guidance.

Filling the entire advertised context window

Capacity does not mean every token receives equal attention or that output space is free. Reserve room for instructions and the answer, remove duplicated material, and test the full task rather than celebrating a maximum input size.

Splitting content at a fixed character count

Blind cuts can separate a heading, definition, table, or qualifier from the passage it explains. Chunk on semantic boundaries and preserve enough local context for each unit to stand alone.

Measure page sections before chunking

Run this in the Chrome DevTools Console. It reports characters and whitespace-split words for each headed section without pretending those counts equal model tokens:

const headings = [...document.querySelectorAll('main h2, main h3')];
console.table(headings.map((h, i) => { let text = ''; for (let n = h.nextElementSibling; n && !/^H[23]$/.test(n.tagName); n = n.nextElementSibling) text += ` ${n.innerText || ''}`; return { heading: h.textContent.trim(), characters: text.trim().length, words: text.trim() ? text.trim().split(/\s+/).length : 0, nextHeading: headings[i + 1]?.textContent.trim() || '' }; }));

Use the output to find oversized or empty sections, then run the final text through the tokenizer for the model you will actually use.

Find likely semantic boundaries in plain text

This regular expression matches Markdown second- and third-level headings and keeps the heading text in capture group 1:

^#{2,3}\s+(.+)$

Use multiline mode. Headings are candidate boundaries, not a guarantee that every section is complete enough to retrieve alone.

Resources worth your time

My related writing

• What We Actually Know About Optimizing for LLM Search — my Ahrefs piece on how AI retrieval treats content, including Dan Petrovic’s Chrome DocumentChunker research on how pages are chunked into ~200-word passages before they’re processed — the token/context-window limit is exactly why that chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking. happens.
• Generative Engine Optimization — my broader take on optimizing for a retrieval-and-generation search landscape shaped by these context limits.

My speaking

• GEO? AEO? LLMO? — my AI search webinar — where retrieval, chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking., and context limits fit into AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity.. (My standing disclaimer applies: this is my understanding of these systems, not going to be 100% complete or accurate.)

Official

• Understand and count tokens (Gemini API) and Long context (Gemini API) — Google’s own definitions and the million-token examples.
• Optimizing your website for generative AI features (Google Search Central) — “no ideal page length,” “no requirement to break your content into tiny pieces.”

From around the industry

• What is a long context window? Google DeepMind engineers explain — Nikolay Savinov on the TPU thermal ceiling, plus Machel Reid and Denis Teplyashin on what long context unlocks.
• Chunk large documents for vector search (Microsoft Azure AI SearchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity.) — chunking to stay under maximum token input limits.
• The llms.txt spec — the “context windows are too small to handle most websites in their entirety” problem statement, and the format built around it.
• SEO Chunk Optimization is Overrated (Despina Gavoyannis, Ahrefs) — “You can’t control how Google, ChatGPT, or Perplexity chunk your content. Their pipelines change based on cost, model, and context.”
• Chunked, Retrieved, Synthesized — Not Ranked (Duane Forrester) — “If traditional SEO optimized for clicks, GenAI systems optimize for chunks… Structure still wins.”
• What is a context window? (IBM) — a neutral, thorough reference explainer.

Stats worth citing

• ~4 characters per token; 100 tokens ≈ 60–80 English words — Google’s own rule of thumb for Gemini models, the most citable token-to-word conversion. Source
• 1,000,000 tokens — Gemini is the first model capable of accepting 1 million tokens in a single window, up from ~2K in early generative models. Source
• ~10,000,000 tokens = near the TPU thermal limit — Google DeepMind’s Nikolay Savinov on the current physical ceiling for context-window size. Source
• 1 million tokens ≈ 8 novels / 200+ podcast transcripts / 50,000 lines of code — Google’s concrete sizing for what a million-token window can hold. Source
• Token-based billing (output usually costs more than input) — per Google, the cost of a Gemini API call is determined in part by input and output token counts, the economic reason AI systems retrieve selectively rather than feed whole pages. Source

Test yourself: Tokens and Context Windows

Five quick questions on how LLMsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). read text and how much they can hold at once. Pick an answer for each, then check.

Tokens and Context Windows

A token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory.

Related: Chunking, Large Language Model (LLM), Knowledge Cutoff

Tokens and Context Windows

A token is the granularity at which large language modelsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). read and write. It isn’t a word or a character — it’s a sub-word fragment. Google’s rule of thumb for Gemini is that a token is about 4 characters, and 100 tokens ≈ 60–80 English words. Splitting text into tokens is called tokenization, and the full set of tokens a model knows is its vocabulary. Tokenization isn’t limited to text: Gemini also tokenizes images, audio, and video.

A context window is the maximum number of tokens a model can “see” in a single interaction — the system prompt, the conversation history, any retrieved or injected documents, your input, and the model’s own output all share that one budget. Google’s own analogy is short-term memory. Once content falls outside the window, it’s simply gone — not gradually forgotten, just invisible, as if it were never there (unless a system explicitly summarizes and re-injects it).

Context windows have grown fast — from a couple thousand tokens in early models to 1M+ tokens in Gemini — but a bigger window doesn’t mean an AI reads your whole site. In most search and RAGRAG is the retrieve-then-generate pattern behind AI search: the system retrieves relevant passages from an external index at query time, injects them into the model's context, and generates an answer grounded in those sources — without changing the model's weights. scenarios, retrieval systems still select and chunk content before it ever reaches the model. That’s the mechanical reason chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking., passage-level retrieval, and “keep content clear and self-contained” advice exist at all.

Related: Chunking, Large Language Model (LLM), Knowledge Cutoff

Changelog