Retrieval-Augmented Generation (RAG)

The original RAG architecture combined a language model with information retrieved from an external indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. during generation. Evidence for this claim The original RAG paper combined a pretrained sequence-to-sequence model with a non-parametric dense-vector index retrieved during generation. Scope: Lewis et al.'s 2020 RAG architecture and experiments, not every modern retrieval system. Confidence: high · Verified: 2026-07-14 Lewis et al.: Retrieval-Augmented Generation Modern platform documentation uses the same broad retrieve-then-generate idea. Evidence for this claim Google Cloud describes RAG as retrieving relevant information from external knowledge sources and providing it to a model to improve generated responses. Scope: General RAG architecture in Google Cloud documentation; quality depends on retrieval, source quality, and generation. Confidence: high · Verified: 2026-07-14 Google Cloud: RAG overview

TL;DR — RAG (Retrieval-Augmented Generation) is how AI search enginesAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. look things up before they answer. Instead of replying purely from memory, the system first retrieves relevant passages from a search index, then generates an answer based on what it foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore.. That’s why Google AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index., ChatGPT Search, and Perplexity can cite fresh web pages — and why being in the index still matters.

What RAG is

A large language modelA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). (an LLM, the thing behind ChatGPT and similar tools) learns from a huge pile of text during training. But that training has a cutoff date, and the model can’t possibly memorize everything — so on its own it either doesn’t know recent or niche facts, or it makes something up that sounds right.

RAG fixes that by letting the model look things up. When you ask a question, a RAG system does two things in order:

1. Retrieval — it searches an index (like Google’s or Bing’s) and pulls back the passages most relevant to your question.
2. Augmented generation — it hands those passages to the LLM, which writes an answer based on them and usually shows links to the sources.

The simplest way to picture it: instead of answering from memory alone, the AI does its homework first.

A quick example

Ask an AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. engine “what changed in the latest iPhone?” The model wasn’t trained on a product that launched last week. With RAG, it searches the web, retrieves a few recent articles, and writes its answer from those — with citations you can click. Without RAG, it would either say it doesn’t know or guess.

Why it matters to you

Here’s the part that surprises people: RAG doesn’t use a separate “AI index.” Google AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. retrieve from Google’s normal search index. ChatGPT Search launched on Bing’s index and also runs its own crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. (OAI-SearchBot) — OpenAI hasn’t said exactly how the two are mixed today. Either way, the same basics that have always mattered — being crawlable, getting indexed, writing clearly — are exactly what decides whether your content can be retrieved and cited in an AI answer.

The other thing to know: RAG reduces wrong answers (hallucinationsAn AI hallucination is when a large language model generates output that is confidently stated but factually wrong, made up, or unsupported by its source. It's a side effect of next-token prediction — not a bug that can be fully eliminated.) but doesn’t eliminate them. The AI can still misread what it retrieved. So being the clearest, most direct source on a topic genuinely helps.

Want the real mechanics — embeddingsEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords., chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking., re-ranking, naive vs. agentic RAG, and the SEO playbook? Switch to the Advanced tab.

Lewis and colleagues’ 2020 system paired sequence generation with dense retrieval from a non-parametric indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Evidence for this claim The original RAG paper combined a pretrained sequence-to-sequence model with a non-parametric dense-vector index retrieved during generation. Scope: Lewis et al.'s 2020 RAG architecture and experiments, not every modern retrieval system. Confidence: high · Verified: 2026-07-14 Lewis et al.: Retrieval-Augmented Generation Google Cloud’s current overview defines RAG more broadly as supplying retrieved external knowledge to a model. Evidence for this claim Google Cloud describes RAG as retrieving relevant information from external knowledge sources and providing it to a model to improve generated responses. Scope: General RAG architecture in Google Cloud documentation; quality depends on retrieval, source quality, and generation. Confidence: high · Verified: 2026-07-14 Google Cloud: RAG overview

TL;DR — RAG is a two-phase, inference-time pattern: retrieval (find relevant passages in an external corpus) then augmented generation (feed those passages to an LLMA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). to produce a grounded, cited answer). The weights never change — it combines the model’s parametric memory with non-parametric memory retrieved live. The retrieval phase chains chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking. → embeddingsEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. → vector searchVector search finds content by comparing the meaning of a query against stored content as numerical vectors, retrieving the closest ones in a high-dimensional embedding space. At scale it uses approximate nearest neighbor (ANN) algorithms — not exact comparison — to search billions of vectors in milliseconds. → re-rankingReranking is the second stage of a retrieval pipeline: after a cheap, broad first pass pulls a candidate set of documents or passages, a slower, more precise model re-scores and reorders that shortlist by true relevance before the results are served or handed to an LLM. → top-k. “Naive” RAG is retrieve-then-generate; advanced RAG adds query rewriting and re-ranking; agentic RAG adds iterative, multi-hop retrieval. Retrieval can ground answers but does not guarantee correctness; in one Gemma evaluation, insufficient context coincided with more incorrect answers. For SEO: there’s no separate AI index; crawlabilityCrawlability is how well search engine crawlers can discover, access, and fetch a site's pages. A crawlability issue is any technical condition — blocked access, broken links, server failures, or bloated URL inventory — that stops pages from reaching the index., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and passage-level clarity are the prerequisites for being retrieved.

The two phases (and why “inference time” is the whole point)

Break the acronym apart and you have the model: Retrieval plus Augmented Generation. A query comes in; the system retrieves the most relevant passages from an external corpus; it injects those passages into the LLM’s context windowA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory.; the LLM generates an answer grounded in them.

The detail that everyone gets wrong: this happens at inference time, and the model’s weights are never touched. RAG is not training and it is not fine-tuning. The original 2020 paper from Patrick Lewis and colleagues at Facebook AI Research framed it as combining two kinds of memory — parametric memory (knowledge baked into the weights during training) and non-parametric memory (knowledge retrieved live from an index). RAG uses both at once. AWS puts the practical case plainly: retraining a foundation model for fresh or domain-specific knowledge is expensive, and “RAG is a more cost-effective approach to introducing new data to the LLM.”

(The naming, for what it’s worth, was an accident. Lewis later admitted: “We definitely would have put more thought into the name had we known our work would become so widespread… We always planned to have a nicer sounding name, but when it came time to write the paper, no one had a better idea.”)

Inside the retrieval phase

“Retrieve the relevant passages” is doing a lot of work in that sentence. In a real system it’s a pipeline:

1. ChunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking.. Documents get split into retrievable pieces. Chunk size is a real tradeoff — too small and a passage loses its context; too large and it floods the token budget with irrelevance. Strategies range from fixed token counts (100/256/512) to recursive/sliding windows to “Small2Big” (retrieve a small sentence, return its parent chunk for generation).
2. EmbeddingsEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords.. Each chunk is turned into a dense vector — a numeric representation of its meaning — so similarity is computed semantically, not by keyword match. This is why content about a topic gets retrieved even when it doesn’t use the exact query phrasing.
3. Vector searchVector search finds content by comparing the meaning of a query against stored content as numerical vectors, retrieving the closest ones in a high-dimensional embedding space. At scale it uses approximate nearest neighbor (ANN) algorithms — not exact comparison — to search billions of vectors in milliseconds.. The query is embedded too, and the system finds the chunks whose vectors sit closest to it. Most production stacks run hybrid searchHybrid search runs keyword (lexical/BM25) search and vector (semantic) search together, then merges the two result lists into one ranking — commonly with Reciprocal Rank Fusion — so it catches both exact-term matches and meaning-based matches that either method alone would miss. — dense vector retrieval plus BM25 keyword search — because each catches recall the other misses.
4. Re-ranking. A separate model re-scores the candidates by relevance to the query and reorders them, “effectively reducing the overall document pool.” Only the top survivors make it into the context.
5. Top-k into the prompt. The best passages are concatenated with the user’s query and handed to the generator.

Chunking is the fragile link. Anthropic identified that “traditional RAG solutions remove context when encoding information” — a chunk pulled out of its document loses the surrounding context that made it meaningful. Their Contextual Retrieval technique (prepending chunk-specific context before indexing) reduced failed retrievals by 49%, and by 67% combined with re-ranking. That’s a strong signal that the chunking problem is real — and that self-contained, context-rich passages are easier to retrieve correctly.

Naive, advanced, and agentic RAG

The survey literature (Gao et al., 2023) splits RAG into a useful taxonomy:

• Naive RAG — “a traditional process that includes indexing, retrieval, and generation.” Retrieve top-k once, generate once. It “struggles with precision and recall, leading to the selection of misaligned or irrelevant chunks.”
• Advanced RAG — adds “pre-retrieval and post-retrieval strategies.” Pre-retrieval: query rewriting and better indexing (including HyDE, where the model generates a hypothetical answer, embeds that, and retrieves documents that look like answers rather than questions). Post-retrieval: re-ranking and context compressionCompression (HTTP content encoding) shrinks text-based responses — HTML, CSS, JS, JSON, SVG, XML sitemaps — before they're sent over the network, using an algorithm like Gzip, Brotli, or Zstd, so the browser or crawler downloads fewer bytes. It's not a ranking factor, but it speeds up page loads and helps pages stay under crawler fetch limits..
• Modular / agentic RAG — the model retrieves, reasons about what’s still missing, and retrieves again, iterating across multiple hops. This is the current state of AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity.. As Michael King put it: “The retrieve-once-then- generate pattern that defined the first wave is obsolete… Agentic RAG is now the default.”

This matters for SEO because content now has to survive multiple retrieval rounds and contradiction-checking — not just a single retrieval pass.

Does RAG eliminate hallucinations? No.

RAG can ground answers in retrieved sources, but the LLM can still misread or over-interpret what it pulled. Google Research (ICLR 2025) documented a counterintuitive result in one evaluation: Gemma produced incorrect answers on 10.2% of questions with no context and 66.1% with insufficient context. The researchers report that models can “excel with sufficient context but fail to recognize when context is insufficient.” Treat that as a model- and evaluation-specific warning, not proof that retrieval universally causes worse answers. The practical lesson is narrower: retrieval quality and context sufficiency need to be evaluated rather than assumed. Google operationalized the finding as an LLM re-ranker in its Vertex AI RAG Engine.

RAG vs. fine-tuning

These get conflated constantly, and they’re fundamentally different:

• RAG retrieves external information at query time. Weights unchanged. Best for fresh/changing information, citation requirements, and cost. The survey found “RAG consistently outperforms [unsupervised fine-tuning], for both existing knowledge encountered during training and entirely new knowledge.”
• Fine-tuning modifies the model’s weights in a separate training run. Best for changing style and behavior, or teaching stable domain knowledge that doesn’t change.

You’d reach for RAG to make a model know the latest facts; you’d reach for fine-tuning to change how it talks.

RAG in the wild: Google, ChatGPT, Perplexity

• Google AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index.. Google calls RAG “a technique (also known as groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it.)… relying on our core Search ranking systems to retrieve relevant, up-to-date web pages from our Search index.” Two things follow. First, there is no separate AI index — “our generative AI features on Google Search are rooted in our core Search ranking and quality systems.” Second, Google runs query fan-outQuery fan-out is the technique where an AI search system breaks a single user question into multiple related sub-queries, runs those searches concurrently, and synthesizes the retrieved results into one answer. Google confirms AI Overviews and AI Mode 'may use a query fan-out technique' issuing multiple related searches across subtopics.: “concurrent, related queries generated by the model to request more information.” A single question can spawn multiple sub-queries, each retrieving different content — so your content has to satisfy the implied sub-questions, not just the head query.
• ChatGPT Search. Launched (October 2024) with Bing as its data partner, and OpenAI’s own crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. documentation confirms OAI-SearchBot does independent fetching and indexing for search citations, separate from GPTBot’s training-crawl. OpenAI hasn’t published the current retrieval mix between Bing and its own index, and OpenAI has since positioned ChatGPT Search as a standalone competitor to Bing rather than a wrapper around it — so treat “it’s basically Bing” as a simplification. The documented, actionable lever is narrower and more durable: don’t block OAI-SearchBot in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., because that’s the crawler OpenAI itself names as the one that indexes content for search citations.
• Perplexity. Built on hybrid retrieval (Vespa.ai — BM25 + dense) with custom embedding models and a strict re-ranking threshold: by third-party analysis, only the top ~30% of 60-plus retrieved sources survive to the generation stage, and “citations are not retrofitted post-generation — they are structurally assigned during context assembly.” Deep Research runs the agentic loop across dozens of searches.

What RAG means for SEO

Strip away the jargon and the playbook is concrete:

• Being in the index is the prerequisite — full stop. No separate AI index means the crawl → index → retrieve chain has to be intact. If a page can’t be crawledCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and indexed, it can’t be retrieved into an AI answer. The same is true for the AI engines that build their own pools: AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. like OAI-SearchBot and PerplexityBot have to be allowed to fetch you, or you’re invisible to those answers.
• Write self-contained passages. RAG retrieves fragments, not whole pages. As iPullRank’s Francine Monahan put it, AI systems examine “fragments of pages rather than the page as a whole” — so craft “stand-out passages and phrases” that answer a specific question on their own. This is exactly the H2/H3 structure and clear topic sentences good SEO already rewards. Google explicitly says not to chop your content into tiny pieces for AI — well-structured content chunks well on its own.
• Cover the sub-topics. Query fan-out means one question can trigger many retrievals. Depth across related sub-questions beats one page stuffed around a single keyword.
• Authority drives citation more than rank position. From an 8,000-citation analysis: “Strong organic search presence and broad web visibility leads to AI citations, not the other way around” — and “highly authoritative content from a lower-ranking page” sometimes gets cited over a less credible top-ranking one. My own data lines up (from my AI Overview citation research): mentions on heavily-linked pages are the strongest predictor of AI Overview inclusion (ρ ≈ 0.70), and branded web mentions correlated ~0.66 across 75,000 brands.
• Fresh content has an edge. AI citationsAn AI citation is the visible source link an AI answer engine shows next to its generated text — the clickable reference that credits the web page it used. A citation's presence is a separate thing from whether the cited page actually supports the statement, and from being retrieved (read behind the scenes) or merely mentioned (named without a link); citation is driven more by brand mentions and being retrievable than by traditional ranking. skew meaningfully fresher than organic results, so currency matters.

If you want the one-sentence version: RAG didn’t replace SEO — it raised the stakes on the parts of SEO that were always about being findable and being clear.

AI summary

A condensed take on the Advanced version:

• RAG = Retrieval + Augmented Generation. Two phases at inference time: retrieve relevant passages from an external corpus, then feed them to an LLMA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). to generate a grounded, cited answer. The model’s weights never change — it’s not training and not fine-tuning.
• It combines two memories: parametric (baked into weights) + non-parametric (retrieved live). That’s how AI answers cover information past the training cutoffA knowledge cutoff (or training data cutoff) is the date after which an LLM was no longer trained on new data. The model has no awareness of anything that happened later — unless the system bolts on live retrieval (RAG/web search) to fetch it at query time..
• Retrieval is a pipeline: chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking. → embeddingsEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. → vector searchVector search finds content by comparing the meaning of a query against stored content as numerical vectors, retrieving the closest ones in a high-dimensional embedding space. At scale it uses approximate nearest neighbor (ANN) algorithms — not exact comparison — to search billions of vectors in milliseconds. (often hybrid with BM25Semantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it.) → re-rankingReranking is the second stage of a retrieval pipeline: after a cheap, broad first pass pulls a candidate set of documents or passages, a slower, more precise model re-scores and reorders that shortlist by true relevance before the results are served or handed to an LLM. → top-k passages into the prompt. ChunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking. is the fragile link; context-rich passages retrieve better (Anthropic cut failed retrievals 49%).
• Three flavors: naive (retrieve-once), advanced (query rewriting, HyDE, re-ranking), and agentic (iterative multi-hop) — agentic is now the AI-search default.
• It reduces, not eliminates, hallucinationsAn AI hallucination is when a large language model generates output that is confidently stated but factually wrong, made up, or unsupported by its source. It's a side effect of next-token prediction — not a bug that can be fully eliminated.. With insufficient context, one model’s hallucination rate jumped 10.2% → 66.1% — bad retrieval can beat no retrieval.
• RAG vs. fine-tuning: RAG for fresh/changing facts + citations + cost; fine-tuning for style/behavior and stable knowledge.
• Engines: Google AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. retrieve from the core index (no separate AI index) with query fan-outQuery fan-out is the technique where an AI search system breaks a single user question into multiple related sub-queries, runs those searches concurrently, and synthesizes the retrieved results into one answer. Google confirms AI Overviews and AI Mode 'may use a query fan-out technique' issuing multiple related searches across subtopics.; ChatGPT Search launched on Bing’s index and also runs its own crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., OAI-SearchBot — the exact current mix isn’t published, so don’t block OAI-SearchBot; Perplexity via hybrid retrieval with a strict re-ranking threshold and citations assigned during context assembly.
• SEO: being crawlable + indexed is the prerequisite; write self-contained passages; cover sub-topics (fan-out); authority/E-E-A-T drives citation more than rank position; fresh content has an edge.

Official documentation

Primary-source documentation and definitions from the providers.

Google

• Google’s Guide to Optimizing for Generative AI Features — defines RAG as groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. over the core Search indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.; covers query fan-outQuery fan-out is the technique where an AI search system breaks a single user question into multiple related sub-queries, runs those searches concurrently, and synthesizes the retrieved results into one answer. Google confirms AI Overviews and AI Mode 'may use a query fan-out technique' issuing multiple related searches across subtopics..
• AI Overviews and AI Mode in Search — confirms no additional requirements beyond standard indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. and snippet eligibility.
• RAG and grounding on Vertex AI — Google Cloud’s retrieve-then-generate definition (Burak Gokturk).
• Deeper insights into RAG: the role of sufficient context — Google Research (ICLR 2025) on the insufficient-context failure mode.

Microsoft / Azure

• RAG and generative AI — Azure AI Search — RAG defined as groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. in proprietary content; query understandingSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it., tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. constraints, and the move to agentic retrieval.

OpenAI

• Overview of OpenAI Crawlers — confirms OAI-SearchBot does independent fetching/indexing for ChatGPT Search citations, separate from GPTBot’s training crawl; doesn’t disclose the current mix with Bing’s index.

Anthropic

• Introducing Contextual Retrieval — the chunk-context-loss problem and a measured fix (49% / 67% fewer failed retrievals).

AWS

• What is Retrieval-Augmented Generation? — clean three-stage explainer and the RAG-vs-retraining cost argument.

Foundational papers

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., NeurIPS 2020 (the original RAG paper; parametric vs. non-parametric memory).
• Retrieval-Augmented Generation for LLMs: A Survey — Gao et al. (the naive / advanced / modular taxonomy, HyDE, re-rankingReranking is the second stage of a retrieval pipeline: after a cheap, broad first pass pulls a candidate set of documents or passages, a slower, more precise model re-scores and reorders that shortlist by true relevance before the results are served or handed to an LLM.).

Quotes from the source

On-the-record statements from the providers and the original researchers. Deep links jump to the quoted passage where available.

Google — RAG is groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it., over the core indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.

• “A technique (also known as groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it.) used to improve the quality, accuracy, and freshness of AI responses by relying on our core Search ranking systems to retrieve relevant, up-to-date web pages from our Search index.” — Google Search Central, AI optimization guide. Jump to quote
• “Our generative AI features on Google Search are rooted in our core Search ranking and quality systems.” — Google Search Central, AI optimization guide.

Google Cloud — the retrieve-then-generate definition

• “Retrieval Augmented Generation (RAG), a technique developed to mitigate these challenges, first ‘retrieves’ facts about a question, then provides those facts to the model before it ‘generates’ an answer – this is what we mean by grounding.” — Burak Gokturk, VP & GM, Cloud AI, Google Cloud (June 27, 2024). Jump to quote

The original RAG paper — parametric vs. non-parametric memory

• “retrieval-augmented generation (RAG) — models which combine pre-trained parametric and non-parametric memory for language generation.” — Lewis et al., NeurIPS 2020.

Patrick Lewis, lead author — on the name (via NVIDIA Blog, Rick Merritt)

• “We definitely would have put more thought into the name had we known our work would become so widespread.”
• “We always planned to have a nicer sounding name, but when it came time to write the paper, no one had a better idea.” Read the coverage

Microsoft — RAG as grounding in your content

• “Retrieval-augmented generation (RAG) is a pattern that extends LLMA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). capabilities by grounding responses in your proprietary content.” — Microsoft, Azure AI SearchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. documentation.

Anthropic — the chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking. problem

• “traditional RAG solutions remove context when encoding information.” — Anthropic, Contextual Retrieval (Sept 19, 2024). Read the post

AWS — RAG vs. retraining

• “Retrieval-Augmented Generation (RAG) is the process of optimizing the output of a large language model, so it references an authoritative knowledge base outside of its training data sources before generating a response.” — AWS.
• “RAG is a more cost-effective approach to introducing new data to the LLM.” — AWS.

OpenAI — its own crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. for ChatGPT Search

• “OpenAI uses OAI-SearchBot and GPTBot robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. tags to enable webmasters to manage how their sites and content work with AI… a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training.” — OpenAI, Overview of OpenAI CrawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. Read the docs

Michael King, iPullRank — the agentic shift (Search Engine Land)

• “The retrieve-once-then-generate pattern that defined the first wave is obsolete… Agentic RAG is now the default.” Read the coverage

RAG cheat sheet

The pipeline, end to end

query → [retrieval: chunk · embed · vector search (+BM25) · re-rank · top-k] → augment (passages into context) → generate (LLM writes grounded, cited answer)

RAG vs. fine-tuning

The three RAG generations

Engine retrieval pools at a glance

Fast facts

• RAG = Retrieval + Augmented Generation; coined in Lewis et al., 2020.
• It’s inference-time — weights never change.
• HallucinationAn AI hallucination is when a large language model generates output that is confidently stated but factually wrong, made up, or unsupported by its source. It's a side effect of next-token prediction — not a bug that can be fully eliminated. isn’t solved: insufficient context took one model from 10.2% → 66.1%.
• Context-aware chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking. cut failed retrievals by 49% (67% with re-ranking).
• Don’t pre-”chunk” your content for AI — clear H2/H3 structure chunks well on its own.

The mental models

1. Retrieve → Augment → Generate. Every RAG system is these three moves. When an AI answer is wrong, locate which stage failed: did it retrieve the right passages, did it pass enough context, or did the model misgenerate from good sources? Most AI-visibility problems are retrieval problems, not generation problems.

2. Parametric vs. non-parametric memory. The model has parametric knowledge (frozen in its weights, capped at its training cutoff) and non-parametric knowledge (retrieved live). Publishing content can’t touch the weights — but it can feed the live retrieval. That’s the entire reason SEO still applies to AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity..

3. RAG vs. fine-tuning is a knowledge-vs-behavior split. Need the model to know new or changing facts? RAG. Need to change how it behaves or writes? Fine-tuning. Don’t fine-tune to add facts that change weekly.

4. Retrieval quality is the bottleneck — and it cuts both ways. Better retrieval beats a bigger model. And insufficient retrieval can be worse than none. So the goal for your content isn’t just “get retrieved” — it’s “get retrieved as a sufficient, self-contained passage” that lets the model answer definitively.

5. The crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → retrieve chain. There’s no separate AI index. If a page fails at crawl or index, it can never reach retrieval — for Google’s RAG or for AI engines building their own pools. Fix the chain first; optimize passages second.

Test yourself: Retrieval-augmented generation

Resources worth your time

My related writing & research

• What We Actually Know About Optimizing for LLM Search — Ahrefs’ write-up using my data: mentions on heavily-linked pages are the strongest predictor of AI OverviewAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. inclusion (ρ ≈ 0.70).
• Generative Engine Optimization — the SEO response to a RAG-powered search landscape.
• GEO? AEO? LLMO? What’s With All This AI SEO Stuff? — my Ahrefs Evolve 2025 talk on the AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. landscape and why the indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. prerequisite hasn’t changed.

The foundational papers

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020 (the origin).
• RAG for LLMs: A Survey — Gao et al. (the naive/advanced/modular taxonomy).

From others

• How AI Search Engines Work — Ryan Law (Ahrefs) on RAG as the groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. mechanism.
• Google AI Overviews: All You Need to Know — Ong & Law (Ahrefs) on RAG over the core index.
• What Is Retrieval-Augmented Generation? — NVIDIA (includes the Lewis naming anecdote).
• How Retrieval-Augmented Generation is Redefining SEO — Francine Monahan, iPullRank (passage-level optimization).
• Beyond RAG: why every AI search platform is now agentic — Michael King, Search Engine Land.
• How Perplexity AI Answers Work — Ishtiaque Ahmed, a technical breakdown of the retrieval/ranking/citation pipeline.
• How to get cited by AI: SEO insights from 8,000 AI citations — James Allen, Search Engine Land; authority and E-E-A-T drive AI citationsAn AI citation is the visible source link an AI answer engine shows next to its generated text — the clickable reference that credits the web page it used. A citation's presence is a separate thing from whether the cited page actually supports the statement, and from being retrieved (read behind the scenes) or merely mentioned (named without a link); citation is driven more by brand mentions and being retrievable than by traditional ranking. more than rank position.
• How Perplexity uses Vespa.ai — Vespa.ai’s first-party account of Perplexity’s hybrid BM25Semantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it. + dense retrieval architecture.
• Retrieval-augmented generation — Wikipedia — useful reference overview; covers RAG poisoning and the hallucinationAn AI hallucination is when a large language model generates output that is confidently stated but factually wrong, made up, or unsupported by its source. It's a side effect of next-token prediction — not a bug that can be fully eliminated. caveat.

Stats worth citing

• 10.2% → 66.1% hallucinationAn AI hallucination is when a large language model generates output that is confidently stated but factually wrong, made up, or unsupported by its source. It's a side effect of next-token prediction — not a bug that can be fully eliminated. jump — one model’s hallucination rate with insufficient retrieved context vs. no context at all; bad retrieval can beat no retrieval. Google Research, ICLR 2025. Source
• 49% fewer failed retrievals from context-aware chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking. (Contextual EmbeddingsEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords.), rising to 67% when combined with re-rankingReranking is the second stage of a retrieval pipeline: after a cheap, broad first pass pulls a candidate set of documents or passages, a slower, more precise model re-scores and reorders that shortlist by true relevance before the results are served or handed to an LLM.. Anthropic, 2024. Source
• ρ ≈ 0.70 — mentions on heavily-linked pages are the strongest predictor of Google AI OverviewAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. inclusion in my research; branded web mentions correlated ~0.66 across 75,000 brands. Source
• ~30% survival rate — by third-party analysis, only roughly the top 30% of 60+ retrieved sources clear Perplexity’s re-ranking threshold into the generation stage. Source
• RAG > unsupervised fine-tuning for knowledge tasks — “for both existing knowledge encountered during training and entirely new knowledge.” Source

Retrieval-Augmented Generation (RAG)

RAG is the retrieve-then-generate pattern behind AI search: the system retrieves relevant passages from an external index at query time, injects them into the model's context, and generates an answer grounded in those sources — without changing the model's weights.

Related: Grounding, AI Search, Knowledge Cutoff

Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG) is the framework that lets a large language modelA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). answer with information it never memorized during training. Instead of relying only on the patterns frozen into its weights, the system runs two phases: retrieval — find the most relevant passages from an external corpus (a search indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., a knowledge base, the live web) — and augmented generation — pass those passages into the model’s context so it can write an answer grounded in real, citable sources. Coined in a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, RAG is now the mechanism powering Google AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index., ChatGPT Search, and Perplexity.

The crucial thing to understand is that RAG is not training and not fine-tuning. It happens at inference time — the moment you ask a question — and the model’s weights never change. The original paper framed this as combining parametric memory (knowledge baked into the weights) with non-parametric memory (knowledge retrieved live). That’s why RAG is the cost-effective way to give a model fresh or proprietary information: you retrieve it per query instead of retraining the whole model.

Under the hood, the retrieval phase usually chains several steps: documents are split into chunks, each chunk is turned into an embeddingEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords., vector searchVector search finds content by comparing the meaning of a query against stored content as numerical vectors, retrieving the closest ones in a high-dimensional embedding space. At scale it uses approximate nearest neighbor (ANN) algorithms — not exact comparison — to search billions of vectors in milliseconds. finds the chunks closest to the query, a re-rankingReranking is the second stage of a retrieval pipeline: after a cheap, broad first pass pulls a candidate set of documents or passages, a slower, more precise model re-scores and reorders that shortlist by true relevance before the results are served or handed to an LLM. pass reorders them by relevance, and the top-k survivors are handed to the model. Most production systems also blend in keyword (BM25Semantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it.) search for recall. “Naive RAG” is the simple retrieve-then-generate loop; advanced RAG adds query rewriting and re-ranking; agentic RAG — now the default in major AI search enginesAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. — retrieves, reasons about what’s missing, and retrieves again across multiple hops.

RAG reduces hallucinations but does not eliminate them. A model can still misread a retrieved passage, and Google research found that when the retrieved context is insufficient, hallucination rates can actually climb higher than with no retrieval at all. For SEO this is the whole point: there’s no separate AI index — being crawlable, indexed, and structured into clear, self-contained passages is the prerequisite for your content to be retrieved and cited.

Retrieval also isn’t guaranteed to happen at all. Production systems run a query classifier ahead of the retrieve step that decides, per query, whether to search — stable or well-known-to-the-model topics can get answered from parametric memory alone, with no retrieval and nothing to be cited. That’s a different measurement problem than “was my page retrieved”: see LLM visibilityLLM visibility (or AI visibility) is the aggregate measure of how often and how prominently a brand or page shows up in AI-generated answers — across AI Overviews, ChatGPT, Perplexity, Copilot, and Gemini. It's the AI-search analog of organic visibility, but it's driven by different signals. for why memory-driven answers don’t move the same way retrieval-driven ones do.

Related: Grounding, AI Search, Knowledge Cutoff

Changelog