Chunking

How AI systems split your pages into passages to embed, index, and retrieve — chunk size, overlap, semantic vs. fixed-size, and how it ties to Google's passage ranking.

First published: Jun 24, 2026 · Last updated: Jul 18, 2026 · Advanced
demand #4 in How Search Works#6 in AI Search#23 on the site
AIAuthored byAI v2

Chunking is the preprocessing step where AI systems split a document into smaller passages before embedding and retrieving them. The chunk — not the page — is the unit of retrieval in AI search, so being indexed isn't enough; you need a passage that answers a specific query in isolation. Chunk size is a trade-off (small = precise but thin on context; large = context-rich but noisy) with no size that guarantees a citation, overlap prevents boundary loss but duplicates tokens, contextual prefixes can disambiguate a chunk but mislead retrieval if they're stale, and 'Lost in the Middle' — a position effect measured on specific 2023 models and tasks, not every model — means LLMs often use the start and end of context best. You can't optimize for a specific chunk size — Google explicitly says you don't need to chop content into pieces — but answer-first, self-contained sections under a clear heading hierarchy chunk and cite cleanly. It's the same sub-document principle as Google's passage ranking, applied to RAG.

TL;DR — ChunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking. is the preprocessing step that splits a document into passages before they’re embedded, indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and retrieved. It exists because embeddingEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. models and context windowsA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. have token limits, and because passage-level retrieval is more precise than page-level. The chunk, not the page, is the unit of retrieval. Chunk size is a trade-off (small = precise/thin; large = rich/noisy) and no size, overlap value, or splitter guarantees a citation; overlap guards boundaries but costs index size and duplication; semantic chunking isn’t reliably better than fixed-size; contextual prefixes can disambiguate a chunk but mislead retrieval when stale. “Lost in the Middle” — a position effect documented on specific 2023 models and tasks — means an LLMA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). often uses the start and end of its context best. You can’t optimize for a specific chunk size — and Google says you shouldn’t try — but answer-first, self-contained sections under a clear heading hierarchy chunk and cite cleanly. It’s the same sub-document principle as Google’s passage rankingPassage ranking is a Google AI system that scores individual sections ('passages') of a page so a single page can earn multiple relevance scores for different queries. Google still indexes whole pages — only the ranking changed., applied to RAGRAG is the retrieve-then-generate pattern behind AI search: the system retrieves relevant passages from an external index at query time, injects them into the model's context, and generates an answer grounded in those sources — without changing the model's weights..

What chunking is, and why it exists

Retrieval systems operate over indexed units, but public search engines do not expose a universal publisher-controlled chunk size. Evidence for this claim Retrieval systems can split files into chunks that are embedded and indexed for later search. Scope: OpenAI's retrieval implementation; chunk sizes, overlap, and indexing behavior are implementation-dependent. Confidence: high · Verified: OpenAI: Retrieval guide RAG research supports retrieve-then-generate patterns without proving that every AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. product uses identical mechanics. Evidence for this claim Retrieval-augmented generation combines a generator with retrieved external passages or documents. Scope: The original RAG research architecture; it does not establish one optimal chunking strategy for every production system. Confidence: high · Verified: Lewis et al.: Retrieval-Augmented Generation

Chunking is the process of dividing a document into smaller, discrete segments — chunks or passages — before those segments are turned into embeddingsEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords., stored in a vector index, and retrieved to answer queries. It’s a foundational step in RAGRAG is the retrieve-then-generate pattern behind AI search: the system retrieves relevant passages from an external index at query time, injects them into the model's context, and generates an answer grounded in those sources — without changing the model's weights. (Retrieval-Augmented Generation) pipelines, and it’s the part of AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. SEOs underrate most, because it’s invisible: it happens during ingestion, not at query time.

Two problems force it:

  • The token-limit problem. Embedding models accept a bounded number of tokens per input — Microsoft notes the text-embedding-3-small model caps at 8,191 tokens, and other models are far smaller. LLMs have finite context windows too. A long page simply won’t fit as a single unit, so it gets split.
  • The retrieval-precision problem. A 5,000-word page about “technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own.” collapsed into one vector is a fuzzy, averaged signal. A 400-word section that’s specifically about crawl budgetThe number of URLs an engine will crawl in a timeframe., embedded as its own vector, is a sharp one. Sub-document granularity is what lets a system pull the one relevant passage out of a long, multi-topic page.

The consequence is the single most important idea here: the chunk, not the page, is the unit of retrieval in AI search. Being crawled and indexed is necessary but not sufficient. You need a chunk that answers a specific query, clearly, on its own. A page ranking #15 organically can earn an AI citation while the #1 result gets skipped — if the #15 page has the more extractable passage.

How chunking works, end to end

Chunking changes the retrieval unit: the page is published once, but its passages are stored and matched separately. Source: /ai-search/how-search-works/chunking/

A four-stage flow begins with one long document containing several topics. The system splits and embeds focused passages as separate vectors. A query retrieves one or more best-matching passages, and those selected passages enter the model context for answer generation.

© Patrick Stox LLC · CC BY 4.0 ·

  1. Ingestion + splitting. An AI crawlerAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. downloads the page; a chunking algorithm divides the text into (usually overlapping) segments.
  2. Embedding. Each chunk becomes a numerical vector and is stored in a vector index. (That’s the embeddingsEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. step.)
  3. Retrieval + generation. A query is embedded, the nearest chunk vectors are pulled via vector searchVector search finds content by comparing the meaning of a query against stored content as numerical vectors, retrieving the closest ones in a high-dimensional embedding space. At scale it uses approximate nearest neighbor (ANN) algorithms — not exact comparison — to search billions of vectors in milliseconds., and the retrieved chunks are passed to an LLM that writes the answer with citations.

This whole architecture traces back to Dense Passage Retrieval (Karpukhin et al., 2020), which showed that retrieving by dense vector similarity beat the old keyword approach (BM25) by 9–19% absolute on top-20 passage accuracy — and that passage-level retrieval works better than document-level for answering specific questions. The RAG paper (Lewis et al., 2020) named the pattern and used 100-word passages from Wikipedia as its chunks. Every AI search system retrieving web content is running a variant of that pipeline.

Chunking strategies

There’s no single algorithm — systems pick from a menu, and they change it over time:

  • Fixed-size — split by a token or character count, with some overlap. Most common. Microsoft’s example: “a fixed size sufficient for semantically meaningful paragraphs (for example, 200 words or 600 characters)” with 10–15% overlap.
  • Sentence / paragraph — split on natural language boundaries instead of an arbitrary token count, preserving semantic units.
  • Semantic / content-aware — group sentences by embedding similarity and split where the topic shifts, aiming to keep each chunk about one thing.
  • Hierarchical / recursive (RAPTOR) — build a tree of summaries (document → section → paragraph) so a query can be answered at the right level of abstraction. RAPTOR (Sarthi et al., 2024) reported a 20% absolute accuracy gain on a hard QA benchmark when paired with GPT-4.
  • Sliding window with overlap — each chunk shares some tokens with its neighbors so a sentence split across a boundary isn’t lost.
  • Adaptive / query-dependent (Mix-of-Granularity) — a trained router picks the chunk size per query. The most sophisticated approach; not yet standard in commercial systems.

Chunk size and overlap — the core trade-off

This is the lever everyone asks about, and the honest answer is it depends:

  • Small chunks (128–256 tokens): more precise retrieval, but they can lose the surrounding context the answer needs.
  • Large chunks (512–1,024 tokens): more context preserved, but noisier retrieval — you drag in irrelevant material with the relevant bit.

The research doesn’t crown a winner. LlamaIndex’s evaluation found 1,024 tokens optimal in their setup; Chroma’s benchmarks found a 200-token recursive splitter performed consistently across metrics. Ravi Theja’s takeaway is the right mindset: “Identifying the best chunk size for a RAG system is as much about intuition as it is empirical evidence.” Those numbers describe what won in one team’s benchmark, on one document set, with one embedding model and evaluation task — not a universal setting. No chunk size, overlap value, or splitter guarantees retrieval, citation, ranking, or inclusion in an answer from an external AI system; each provider’s pipeline picks its own defaults and can change them without notice.

Overlap is the underrated half of this. Without it, content near a boundary gets split and lost. Microsoft recommends starting at 25% overlap so there are “smoother transitions between chunks without excessive duplication”; other sources suggest 10–15%. Overlap isn’t free, though: overlapping tokens get embedded and stored twice, which inflates the index and can surface near-duplicate passages side by side in a retrieved set — so it’s a trade against index size and redundancy, not a costless safety net. Weigh it against how often a boundary actually costs you a missed fact, not on principle. The practical point for content: don’t assume the system will keep a key fact intact if you bury it exactly where a chunk is likely to break.

So, does semantic chunking always win? No — that’s the inconvenient finding. Vectara’s 2024 study found “performance differences are minimal” on real-world documents, and — crucially — the embedding model’s quality mattered more than the chunking strategy. When GPT-4o generated the answers, the differences between strategies were “negligible.” That result is specific to Vectara’s document set, models, and evaluation method — it’s evidence semantic chunking isn’t a reliable default win, not proof it never helps in any pipeline. Translation for SEOs: obsessing over an exact structure matters far less than content quality and semantic clarity.

Chunk context: what prefixes can (and can’t) fix

A chunk that reads clearly to a human can still retrieve badly once it’s separated from the document around it — the section it sits under, the entity it’s actually about, a qualifier stated two headings up. One documented mitigation is prepending a short, chunk-specific context string (the document title, the section it belongs to, what the chunk is actually about) before the chunk is embedded — an approach Anthropic calls contextual retrieval. Titles, section ancestry, and a contextual prefix can disambiguate an otherwise-orphaned chunk and cut down on retrieval misses caused by missing context.

That fix has a failure mode of its own, though: stale or wrong context doesn’t just fail to help — it actively misleads the retriever toward the wrong chunk. A prefix generated from a heading that no longer matches the section after a page reorg, or a contextual summary that misstates what the chunk covers, is worse than no prefix at all. Context preservation is a pipeline design choice with its own error mode, not a one-way improvement you can bolt on and forget.

Google’s passage ranking — the SEO ancestor of chunking

Google has been doing sub-document granularity since long before “RAG” was a buzzword. At Search On 2020, Prabhakar Raghavan announced passage rankingPassage ranking is a Google AI system that scores individual sections ('passages') of a page so a single page can earn multiple relevance scores for different queries. Google still indexes whole pages — only the ranking changed.: “By better understanding the relevancy of specific passages, not just the overall page, we can find that needle-in-a-haystack information you’re looking for.” It went live in U.S. English on February 10, 2021 and affects roughly 7% of queries.

Two things people get wrong about it:

  • It’s “passage ranking,” not “passage indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..” Google’s first announcement used “indexing,” then quickly corrected it: “this change doesn’t mean we’re indexing individual passages independently of pages.” The page is still indexed as a whole; the relevant passage is an additional ranking signal.
  • The page ranks, not the passage. John Mueller: “Passage ranking is not about ranking a specific passage but understanding the content on a really long, not SEO optimized page, and ranking that page (not the passage) for a query where the passage is relevant.”

So passage ranking and RAG chunking share a principle — a paragraph, not a page, is often the right unit to match against a specific query — but the outcome differs: passage ranking lifts the page’s ranking; RAG chunking retrieves a chunk to feed a generated answer. Same idea, different machinery. (The technical underpinning, per Dawn Anderson’s reporting, is DeepCT — BERT-derived contextual term weights replacing TF-IDF, so term frequency no longer equals term relevance.)

”Lost in the Middle” — where your answer sits matters

Even after your chunk is retrieved, where it lands in the LLM’s context affects whether the model actually uses it. The Stanford “Lost in the Middle” study (Liu et al., 2023) found that “performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts.”

That finding doesn’t apply universally by default — it’s a position effect measured on named multi-document QA and key-value retrieval tasks, on the specific model generation Liu et al. tested in 2023. It’s not proof that every current model ignores evidence placed centrally in its context; different architectures, longer effective context windows, and newer training can narrow or widen the effect. Treat it as a documented risk to design around, not a fixed law of every model you’ll ever be retrieved into.

The content implication is still concrete and low-risk either way: lead with your answer. Put the definition, the key finding, the direct answer in the first sentence of each section — not buried in paragraph four. This is the same answer-first (BLUF) discipline that serves human skimmers; it just happens to also hedge against being dropped into the middle of a context window on models where the effect holds.

The practical constraints you can’t see

  • Chrome’s ~30-passage limit. Dan Petrovic’s research suggests Chrome’s DocumentChunker analyzes content in ~200-word passages and “only ever considers the first 30 passages of a page.” It tree-walks the semantic HTMLSemantic HTML is the practice of using elements like <main>, <article>, <section>, <nav>, <header>, and <aside> to describe what content is, not just how it looks. It helps search engines and assistive tech identify a page's main content more reliably, but it isn't a ranking factor on its own. top to bottom. Implication: your most important content should appear early — not at the bottom of a 10,000-word page.
  • You don’t control the chunker. Despina Gavoyannis (Ahrefs) is blunt: “You can’t control how Google, ChatGPT, or Perplexity chunk your content. Their pipelines change based on cost, model, and context.” And: “Manual ‘chunk optimization’ is impossible in practice.”
TIP

Check whether a section still makes sense when a retrieval system separates it from the surrounding page with the Chunk Tester Free

  1. Paste a representative article or section and inspect the simulated passage boundaries.
  2. Review orphan headings, weak heading context, and dangling openers as editorial prompts—not as a model of any search engine’s private chunks.
  3. Fix passages that lose their subject or qualification, then reread the full page so retrieval clarity does not come at the expense of human flow.
A heading without supporting content creates a structurally empty passage. Add the section's answer, or remove the heading rather than leaving a retrieval-fragile fragment.

A focused Chunk Tester finding labeled orphan heading says the document ends on a heading with no section content and recommends adding section content after the heading or removing the empty heading.

What “chunk optimization” actually is (and the Google caveat)

Here’s the nuance the hype skips. Google’s 2026 AI optimization guide says plainly: “There’s no requirement to break your content into tiny pieces for AI to better understand it.” Its systems “are able to understand the nuance of multiple topics on a page and show the relevant piece to users.” So no — do not rewrite your content into rigid 300-word blocks.

But that doesn’t mean structure is irrelevant. As Gavoyannis puts it, “Most SEOs using the term [chunk optimization] are just talking about good content structure.” The advice underneath the buzzword is sound; the framing just inflates its novelty. Duane Forrester’s line captures the shift well: “If traditional SEO optimized for clicks, GenAI systems optimize for chunks… Structure still wins.”

So what do you actually do? Write chunk-ready content:

  • One topic per section. A focused H2/H3 maps cleanly to a coherent chunk.
  • Answer-first. Lead with the claim; support it below.
  • Self-contained sections. Test: would this paragraph make sense if it appeared in isolation? If not, it won’t survive being chunked out.
  • Appropriate length. 200–500 words per major section lines up naturally with 256–512-token chunks — complete enough to be useful, not artificially short.
  • Structured formats. Tables and lists give systems explicit boundaries to detect. (Onely’s research: tables increase citation rates ~2.5x.)

Mike King’s framing is the right reassurance: “chunking and writing for users is not mutually exclusive.” The structure that helps a reader skim is the structure that chunks cleanly. You’re not optimizing for a robot at the expense of a human — it’s the same content.

Passage ranking vs. RAG chunking — side by side

Google passage rankingRAG chunking
What it isA ranking signalA preprocessing step
GranularityPassage within a pageChunk split before embedding
OutcomeThe page ranks higherA chunk is retrieved into the answer
Where it runsAt ranking timeAt ingestion (then retrieval)
You control the split?NoNo
Shared principleSub-document granularity — a paragraph, not a page, often matches a specific query best

Where this sits in the pipeline

Chunking is the first move in AI retrieval: chunk → embed → store → retrieve → generate. It feeds embeddingsEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. (each chunk becomes a vector), which feed vector searchVector search finds content by comparing the meaning of a query against stored content as numerical vectors, retrieving the closest ones in a high-dimensional embedding space. At scale it uses approximate nearest neighbor (ANN) algorithms — not exact comparison — to search billions of vectors in milliseconds. (query matches nearest chunks), which feeds RAGRAG is the retrieve-then-generate pattern behind AI search: the system retrieves relevant passages from an external index at query time, injects them into the model's context, and generates an answer grounded in those sources — without changing the model's weights. (retrieved chunks become an answer). Upstream, AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. are how your content gets ingested in the first place. For the traditional-search version of this whole pipeline, see How Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank..

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.