How Search Works

TL;DR — Search works in three steps: a search engine crawls (finds and downloads your pages), indexes (figures out what each page is about and files it away), and serves results (matches a search to the best pages and puts them in order). The catch: a page has to pass each step to make it to the next. Getting foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. doesn’t mean you’re indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and being indexed doesn’t mean you’ll rank.

The three stages

When you search Google, you’re not searching the live web — you’re searching Google’s copy of it. Getting from “a page exists on the internet” to “a page shows up in your results” takes three stages:

1. CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. — automated programs (Google’s is GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., Bing’s is BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.) discover URLs by following links and reading sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., then download the pages.
2. IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — the engine looks at what it downloaded, works out what the page is about, and stores it in a giant database called the index.
3. Serving (ranking) — when someone searches, the engine pulls the best matching pages out of the index and puts them in order. Evidence for this claim Google describes Search as operating through crawling, indexing, and serving search results. Scope: Google's public conceptual model, not an exhaustive implementation description. Confidence: high · Verified: 2026-07-14 Google: How Search works

They’re gates, not a slide

The most important thing to understand: these aren’t a conveyor belt that carries every page to the end. They’re three gates, and a page can get stopped at any one of them. Google says it plainly: search works in three stages “and not all pages make it through each stage.” Evidence for this claim Google states that not all pages make it through each Search processing stage and that increased crawl rate is not a ranking signal. Scope: Google Search; crawling and indexing do not guarantee ranking. Confidence: high · Verified: 2026-07-14 Google: How Search works Google: Crawling myths

• A page can be crawled but not indexed — Google looked at it and decided not to store it.
• A page can be indexed but never ranked — it’s in the database, but it doesn’t win for the searches people make.

This is why “Google can see my page, so why isn’t it showing up?” is such a common question. Being crawled is just the first gate.

What this means for you

• If a page isn’t in search at all, figure out which gate it’s stuck at before you change anything — that tells you what to fix.
• You can’t pay Google to rank, and “submitting” a page to Google doesn’t force it to be indexed or to rank.
• Bing works the same way — crawl, index, rank — it just names some of the pieces differently.

Want the accurate, rep-sourced version — with renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it., why indexing isn’t guaranteed, and why ranking is many systems rather than “the algorithm”? Switch to the Advanced tab.

TL;DR — Search runs in three stages — crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve — and they’re gates: “not all pages make it through each stage.” CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. discovers and fetches; renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. (running JavaScript in a recent Chrome) is the bridge into indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and it can lag. Indexing = understand + canonicalize (~40 signals)

• store, and it isn’t guaranteed. Serving = understand the query, then rank with many automated ranking systems (RankBrainSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it., BERT, neural matching, link/PageRankPageRank is Google's original recursive link-graph algorithm: a page's score depends on the scores of the pages linking to it, and in the published model each page's score is split across its outbound links (the simplified version: links are weighted votes). Google says it's evolved since launch but still part of its core ranking systems. systems…) — not one “algorithm” — across hundreds of contextual factors, with search features layered on top. Bing runs the same shape. Treat any diagram (including mine) as a model, not the source code. The payoff: when a page isn’t showing up, diagnose which gate it’s stuck at.

Three stages, three gates

Google is blunt about the pipeline: “Google Search works in three stages, and not all pages make it through each stage” — crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., indexing, and serving. Evidence for this claim Google describes Search as operating through crawling, indexing, and serving search results. Scope: Google's public conceptual model, not an exhaustive implementation description. Confidence: high · Verified: 2026-07-14 Google: How Search works That last clause is the whole game. The pipeline is not a conveyor belt that carries every page to the finish; it’s three gates, each with its own pass/fail and its own failure mode.

The mental model to anchor everything:

Discover & fetch (crawl) → understand & store (index, including render + canonicalize) → match & order (serve, including query understanding + ranking systems + search features).

Keep crawl ≠ index ≠ rank separate in your head and most of technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own. stops being mysterious.

Stage 1 — Crawling (discover & fetch)

Crawling is discovery plus download. Google finds pages it already knows, follows links from known pages to new ones, and reads sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. you submit — Google calls the whole thing “URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched..” What gets crawled, how often, and how many pages per visit is decided algorithmically: “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.”

The one takeaway at this altitude: crawling is a prerequisite for indexing, not a ranking signal. Evidence for this claim Google states that not all pages make it through each Search processing stage and that increased crawl rate is not a ranking signal. Scope: Google Search; crawling and indexing do not guarantee ranking. Confidence: high · Verified: 2026-07-14 Google: How Search works Google: Crawling myths A higher crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. doesn’t lift your positions. I won’t re-derive the crawl pipeline here — discovery, the crawl scheduler, crawl budgetThe number of URLs an engine will crawl in a timeframe., log files, and the rest live in the crawling cluster (linked at the bottom). Crawling’s real job is to hand a fetched page off to the next stage.

One caveat before moving on: crawling is how web pages enter this pipeline, but it isn’t Google’s only source of information. As I cover below, the index also draws on partnerships, data feeds, and Google’s own Knowledge GraphThe Knowledge Graph is Google's database of entities — people, places, organizations, and things — and the factual relationships between them. It's separate from any single website's structured data: your schema markup is one of many possible inputs to the graph, not the graph itself. — so “not in the index” and “not on the crawled web” aren’t quite the same question.

The bridge — rendering

Here’s the step most overviews get wrong or skip. Between crawling and indexing, Google renders the page. “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome.” In Google’s own consumer words: “When crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. find a webpage, our systems render the content of the page, just as a browser does.”

RenderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. is part of indexing, not crawling — it runs in the Web Rendering Service (an evergreen, headless Chromium), and it can lag the initial fetch. If your content only appears after JavaScript runs, the version Google understands is the rendered version, and that may show up later than the raw HTML did. This is why JavaScript-heavy sites need to think about rendering as its own concern. (I go deep on this in my JavaScript SEO writing.)

Stage 2 — Indexing (understand & store)

Once a page is crawled and rendered, indexing tries to make sense of it. Google: “After a page is crawled, Google tries to understand what the page is about. This stage is called indexing and it includes processing and analyzing the textual content and key content tags and attributes, such as <title> elements and alt attributes, images, videos, and more.”

Three things happen here that overviews usually miss:

Understand. Google processes the text, tags, and media to work out the topic. In my own How Search Works deck I describe this as a content parser that normalizes the HTML, tokenizes it, and vectorizes the content (turning words into numbers) so the meaning can be analyzed — language, location, schema, and topic/semantic signals get extracted along the way. (That’s my paraphrase of how the processing works, not a Google quote.)

Canonicalize. Indexing also decides which URL represents a set of duplicates. “During the indexing process, Google determines if a page is a duplicate of another page on the internet or canonical. The canonical is the page that may be shown in search results.” Google clusters the duplicates and picks one canonical to stand for the group. This isn’t a coin flip — it’s a signal-weighted decision. As I cover in Google Uses ~40 Canonicalization Signals, “canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. is the process that search engines use to determine the main version of the URL that will be indexed and shown to users when there are duplicate URLs” — and according to Google’s Allan Scott, there are ~40 different canonical selection signals (redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., rel=canonical, sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. URLs, internal/external links, hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others., PageRankPageRank is Google's original recursive link-graph algorithm: a page's score depends on the scores of the pages linking to it, and in the published model each page's score is split across its outbound links (the simplified version: links are weighted votes). Google says it's evolved since launch but still part of its core ranking systems., HTTPSHTTPS is the encrypted version of HTTP — it uses TLS to authenticate the server and protect data in transit between a browser and a website. Google announced it as a lightweight ranking signal in 2014 and today conditionally prefers HTTPS pages as canonical; Chrome marks plain HTTP pages 'Not Secure.' over HTTP, shorter URLs over longer, and more).

Store. “The collected information about the canonical page and its cluster may be stored in the Google index, a large database hosted on thousands of computers.” That index is enormous — “hundreds of billions of webpages… well over 100,000,000 gigabytes in size” — and it’s not only crawled HTML: “we have multiple indexes of different types of information, which is gathered through crawling, through partnerships, through data feeds being sent to us and through our own encyclopedia of facts, the Knowledge Graph.”

Indexing isn’t guaranteed

The critical myth-buster: getting crawled doesn’t get you indexed. Google: “Indexing isn’t guaranteed; not every page that Google processes will be indexed.” Common causes are low-quality content, a robots noindex, and page design that makes indexing difficult.

John Mueller has put a human voice on this. On how long “discovered/crawled — currently not indexed” can last: “That can be forever. It’s something where we just don’t crawl and index all pages.” And the fix isn’t a trick — it’s substance: “continue working on the website and making sure that our systems recognize that there’s value in crawling and indexing more and then over time we will crawl and index more.”

I’ve proven crawl ≠ index from the other direction too. In Indexed, though blocked by robots.txt the point is simple: “crawling and indexing are two different things.” A page you block from crawling can still get indexed if other pages link to it — Google just can’t see its content (or any noindex you put there).

Stage 3 — Serving results (match & order)

Serving is what happens the instant someone searches. “When a user enters a query, our machines search the index for matching pages and return the results we believe are the highest quality and most relevant to the user’s query.” But there are two distinct moves inside that.

First, understand the query

Before anything gets ranked, Google has to understand what you asked. That means spelling correction, synonyms and query expansion, and figuring out intent. In my How Search Works deck I describe this as spelling correction via a deep neural net, query expansion across synonyms/acronyms/stemming, and neural matching that links conceptually related terms — with a meaningful share of daily queries being ones Google has never seen before. (Paraphrase, from the deck.) Query understanding is its own step; it’s not the same as ranking.

Then, rank — with many systems, not “the algorithm”

This is the big correction. Ranking is not one formula. “Google uses automated ranking systems that look at many factors and signals about hundreds of billions of web pages and other content in our Search index to present the most relevant, useful results, all in a fraction of a second.” Note the plural — ranking systems. Google names several: RankBrain (“an AI system that helps us understand how words are related to concepts”), BERT, neural matching (“an AI system that Google uses to understand representations of concepts in queries and pages and match them to one another”), and link analysis systems and PageRank (“we have various systems that understand how pages link to each other…”). The helpful content system is described in the past tense now — it was folded into core ranking in the March 2024 update, so it’s no longer a standalone live system.

Two distinctions worth teaching:

• Systems ≠ updates. The ranking systems are the ongoing machinery; ranking updates are the changes Google launches to them. They’re tracked separately.
• There’s no fixed “top 3 factors.” Gary Illyes has pushed back on the idea that links are a top-three factor — “I think they are important, but I think people overestimate the importance of links. I don’t agree it’s in the top three. It hasn’t been for some time.” He’s also said “it is possible to rank without links.” And Illyes has argued there isn’t even a universal “top three,” because different sites have different top signals (that framing is Search Engine Land’s paraphrase of him, not a verbatim quote).

The five factors Google names

For a beginner-friendly map, Google’s consumer site breaks ranking into five buckets:

1. Meaning of your query — “To return relevant results, we first need to establish what you’re looking for - the intent behind your query.”
2. Relevance of content — “The most basic signal that information is relevant is when content contains the same keywords as your search query.”
3. Quality of content — “they identify signals that can help determine which content demonstrates expertise, authoritativeness, and trustworthiness.” (Google’s own words here are E-A-T — expertise, authoritativeness, trustworthiness; the “experience” E that makes E-E-A-T is my gloss, not theirs in this quote.)
4. Usability of content — page experience and accessibility considerations.
5. Context and settings — “we use information such as your location, past Search history, and Search settings to determine what is most relevant for you.”

Relevance is contextual: “Relevancy is determined by hundreds of factors, which could include information such as the user’s location, language, and device (desktop or phone).” And the weighting shifts by query — “when searching for current news topics, content freshnessContent freshness is how recent or up-to-date a page is — by its original publish date, its last substantive revision, or the currency of the facts inside it. It only helps rankings when the query itself benefits from recent results (Query Deserves Freshness), and cosmetic date changes with no real update don't count. plays a bigger role than dictionary definitions.” There is no single “best” result that’s the same for everyone.

Serving isn’t ten blue links

Serving also decides what the page looks like. “Based on the user’s query the search features that appear on the search results page also change.” Knowledge panels, rich resultsRich results (formerly 'rich snippets') are enhanced search listings — stars, images, prices, breadcrumbs, video thumbnails, and more — that Google and Bing build from structured data. They're a display feature, not a ranking factor, and eligibility never guarantees they'll show., images, video, local packs — those are part of serving, not a separate system bolted on. And to kill two persistent myths: “we never provide special treatment to advertisers in how our search algorithms rank their websites, and nobody can pay us to do so.” You can’t buy organic position, and submitting a URL only signals — it doesn’t guarantee indexing or ranking.

Bing runs the same shape

Bing follows the same crawl → index → rank pipeline, just named its own way. Microsoft: “Crawling is how BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. discovers new and updated pages and content to add to our search index.” Then “as Bingbot crawls the web, it sends information to Bing about what it finds. These pages are then added to the Bing index.” And finally “complex algorithms generate Microsoft Bing search results using the user’s search query matching it with third-party webpages in our index.” Bing publishes its own named ranking factors — relevance, quality and credibility, user engagement, freshness, location, and page load time — the clean analog to Google’s five.

It’s a model, not the source code

One honest caveat before you treat any of this as gospel. Search engines tell us a lot, but not everything, and the internals change. Any pipeline diagram — including mine — is a useful approximation, not the actual code. I put this disclaimer right on slide 3 of my How Search Works deck and I’ll repeat it here: “This is my understanding of systems and is based on a lot of public statements from Google and my own knowledge. Warning: It’s not going to be 100% complete or accurate.” Hold the model loosely; use it to reason, not to litigate.

Which stage are you stuck at?

Here’s where the three-gates model pays off. When a page isn’t performing, don’t guess — locate the failed gate, then fix that stage:

The diagnosis tells you which deep dive you need next.

Where to go next

This page is the map. Each stage gets its own depth:

Crawling — the only stage fully built out so far, as its own cluster. Start with the crawling hub, then go deep on crawl budget (capacity + demand and who actually needs to care), crawl rate (how fast bots fetch and how to speed it up or slow it down), crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. (what makes Google re-crawl sooner), and crawl depthCrawl depth usually means click depth — how many clicks it takes to reach a page from the homepage by following internal links. It can also mean a crawler setting that limits how many levels deep a crawl goes before it stops. (why important pages belong close to the homepage). For diagnosing problems, see log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. (the ground truth for what bots actually did) and spider trapsA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content. (infinite URL spaces that burn crawl budget). To get pages discovered faster, there’s IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. (the push protocol Bing and others use) and the Google Indexing APIThe Google Indexing API lets site owners notify Google directly when a URL is added, updated, or removed. Google officially supports it only for pages with JobPosting or BroadcastEvent (livestream) structured data — not for general content. (and the myth that it indexes anything).

Rendering — the bridge between crawling and indexing, now built out in two deep dives. Rendering covers how Google runs your JavaScript in the Web Rendering Service and the rendering options (CSR, SSR, SSG, hydrationActivating server-rendered HTML in the browser by attaching JavaScript handlers., ISR, edge, dynamic) and their SEO trade-offs. JavaScript SEOMaking sure search engines can crawl, render, and index content that depends on JavaScript. covers the practical problems and fixes — real <a href> links, DOM parityWhether the rendered DOM matches what you expect the raw HTML to become., lazy content, soft-404s, and why infinite scrollInfinite scroll is a loading pattern where content appears automatically as a user scrolls, instead of via numbered pages. Because Googlebot doesn't scroll or click, indexable infinite scroll needs real per-chunk URLs (paginated loading) that update via the History API — otherwise deep content may never be crawled, or worse, get merged into the wrong page. can get two pages indexed as one.

Indexing — canonicalization and indexability have their own cluster now; start with the indexing hub.

Ranking — the ranking systems, the signals behind them, and how E-E-A-T fits in are a topic of their own, coming next.

For everything in this cluster in one place, see the How Search Works clusterSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank..

AI summary

A condensed take on the Advanced version:

• Search = three stages, three gates (crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve). Google: “not all pages make it through each stage.” Crawl ≠ index ≠ rank.
• CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. = discover + fetch (“URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched.,” algorithmic schedule). It’s a prerequisite for indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., not a ranking signal. Depth lives in the crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. cluster.
• RenderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. is the bridge: Google runs JavaScript in a recent Chrome (the Web RenderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. Service) as part of indexing; it can lag the initial crawl.
• Indexing = understand + canonicalize + store. Google picks a canonical among duplicates via ~40 signals (Allan Scott). The index is “well over 100,000,000 gigabytes” and includes multiple indexes + the Knowledge GraphThe Knowledge Graph is Google's database of entities — people, places, organizations, and things — and the factual relationships between them. It's separate from any single website's structured data: your schema markup is one of many possible inputs to the graph, not the graph itself..
• Indexing isn’t guaranteed: “not every page that Google processes will be indexed.” Mueller: “currently not indexed” can be “forever” — usually a quality call.
• Serving = understand the query, then rank. Query understandingSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it. (spelling, synonyms, intent) comes first.
• Ranking = many automated systems, not “the algorithm” (RankBrain, BERT, neural matching, link/PageRankPageRank is Google's original recursive link-graph algorithm: a page's score depends on the scores of the pages linking to it, and in the published model each page's score is split across its outbound links (the simplified version: links are weighted votes). Google says it's evolved since launch but still part of its core ranking systems. systems). Systems ≠ updates. No fixed “top 3”; Illyes: links aren’t top three.
• Relevance is contextual (location, language, device, history; freshness is query-dependent). Google’s five factors: meaning, relevance, quality, usability, context. You can’t pay for organic rank.
• Serving ≠ ten blue links — search features change by query.
• Bing runs the same shape with its own six named factors.
• It’s a model, not the source code (Patrick’s disclaimer). The payoff: diagnose which gate a page is stuck at.

Official documentation

Primary-source documentation from the search engines.

Google

• In-Depth Guide to How Google Search Works — the crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve overview, URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched., renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and serving.
• How Search organizes information — the consumer explainer of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., and the index (scale + multiple indexes + Knowledge GraphThe Knowledge Graph is Google's database of entities — people, places, organizations, and things — and the factual relationships between them. It's separate from any single website's structured data: your schema markup is one of many possible inputs to the graph, not the graph itself.).
• How Search ranking results — the five “key factors”: meaning, relevance, quality, usability, context.
• A guide to Google Search ranking systems — the ranking systems (RankBrainSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it., BERT, neural matching, link systems…), distinct from ranking updates.
• JavaScript SEO basics — rendering as part of indexing and how to keep JS content indexable.

Bing / Microsoft

• How Bing delivers search results — Bing’s crawl → index → rank pipeline and ranking factors.
• bingbot Series: Maximizing Crawl Efficiency — Bing’s definition of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and its “crawl efficiency north star.”

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — the three stages

• “Google Search works in three stages, and not all pages make it through each stage.” — Google Search Central docs. Jump to quote
• “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.” Jump to quote
• “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome.” Jump to quote

Google — indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.

• “After a page is crawled, Google tries to understand what the page is about. This stage is called indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.…” Jump to quote
• “During the indexing process, Google determines if a page is a duplicate of another page on the internet or canonical.” Jump to quote
• “Indexing isn’t guaranteed; not every page that Google processes will be indexed.” Jump to quote
• “When crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. find a webpage, our systems render the content of the page, just as a browser does.” — Google, How Search organizes information. Jump to quote
• “The Google Search index covers hundreds of billions of webpages and is well over 100,000,000 gigabytes in size.” Jump to quote

Google — serving & ranking

• “When a user enters a query, our machines search the index for matching pages and return the results we believe are the highest quality and most relevant to the user’s query.” Jump to quote
• “Relevancy is determined by hundreds of factors, which could include information such as the user’s location, language, and device (desktop or phone).” Jump to quote
• “Google uses automated ranking systems that look at many factors and signals about hundreds of billions of web pages and other content in our Search index to present the most relevant, useful results, all in a fraction of a second.” — Google, A guide to Google Search ranking systems. Jump to quote
• “RankBrainSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it. is an AI system that helps us understand how words are related to concepts.” Jump to quote
• “Neural matching is an AI system that Google uses to understand representations of concepts in queries and pages and match them to one another.” Jump to quote
• “To return relevant results, we first need to establish what you’re looking for - the intent behind your query.” — Google, How Search ranking results. Jump to quote
• “…they identify signals that can help determine which content demonstrates expertise, authoritativeness, and trustworthiness.” Jump to quote
• “We never provide special treatment to advertisers in how our search algorithms rank their websites, and nobody can pay us to do so.” Jump to quote

John Mueller, Google (via Search Engine Journal)

• “That can be forever. It’s something where we just don’t crawl and index all pages.” Read the coverage
• “…continue working on the website and making sure that our systems recognize that there’s value in crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and indexing more and then over time we will crawl and index more.” Read the coverage

Gary Illyes, Google (via Search Engine Land)

• “I don’t agree it’s in the top three. It hasn’t been for some time.” — on links as a ranking factor. Read the coverage
• “It is possible to rank without links.” Read the coverage

Microsoft Bing

• “CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is how BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. discovers new and updated pages and content to add to our search index.” Jump to quote
• “Complex algorithms generate Microsoft Bing search results using the user’s search query matching it with third-party webpages in our index.” Jump to quote

Patrick Stox (my own work)

• “CanonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. is the process that search engines use to determine the main version of the URL that will be indexed and shown to users when there are duplicate URLs.” — and per Google’s Allan Scott, there are ~40 canonical selection signals. Read the article
• “This is my understanding of systems and is based on a lot of public statements from Google and my own knowledge. Warning: It’s not going to be 100% complete or accurate.” — slide 3 of my How Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank. deck. See the slide

The mental models

1. Three gates, not a conveyor belt. Crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve. Each is a filter, and “not all pages make it through each stage.” When a page underperforms, locate which stage it failed before you change anything.

2. The three “not equals.”

• CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. ≠ indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. (a blocked page can still be indexed via links; indexing isn’t guaranteed even when crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. succeeds).
• Crawling ≠ ranking (crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. is not a ranking signal).
• Crawling ≠ renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. (JavaScript runs in a separate step that can lag).

3. Indexing = understand + canonicalize + store. Google processes text/tags/media, picks one canonical among duplicates (via ~40 signals), and stores the result. Two pages can both be crawled and still collapse into one indexed URL.

4. Serving = understand the query, then rank. Query understandingSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it. (spelling, synonyms, intent) is its own step before ranking. Don’t collapse them.

5. Ranking = many systems, not one algorithm. RankBrain, BERT, neural matching, link/PageRankPageRank is Google's original recursive link-graph algorithm: a page's score depends on the scores of the pages linking to it, and in the published model each page's score is split across its outbound links (the simplified version: links are weighted votes). Google says it's evolved since launch but still part of its core ranking systems. systems — “ranking systems” plural. And ranking systems ≠ ranking updates. There’s no fixed “top 3 factors.”

6. Relevance is contextual. Location, language, device, history, freshness — there is no single “best” result for everyone. The right answer depends on who’s asking.

7. It’s a model, not the source code. Use the pipeline to reason about problems, not to assert exactly how Google works internally. Hold it loosely.

Which stage are you stuck at? — diagnostic cheat sheet

Map the symptom to the failed gate, then go to the right deep dive.

Crawl ≠ index ≠ rank — what each really means

Resources worth your time

My related writing

• The Beginner’s Guide to Technical SEO — where crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and ranking fit together: “optimizing your website to help search engines find, crawl, understand, and index your pages.”
• Google Uses ~40 Canonicalization Signals — how indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. picks one URL among duplicates (Allan Scott’s ~40 signals).
• JavaScript SEO Issues & Best Practices — renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. as part of indexing, and why it can lag.
• Indexed, though blocked by robots.txt — proof that “crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and indexing are two different things.”
• The Story of Blocking 2 High-Ranking Pages With Robots.txt — my first-party experiment on crawl vs. rank.

My speaking

• How Search Works (SlideShare) — my full walkthrough of crawling, renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., indexing, query understandingSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it., and ranking. (My standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From around the industry

• r/TechSEO — the community for crawl/index/rank debugging.
• Google Says ‘Discovered - Currently Not Indexed’ Status Can Last Forever (Search Engine Journal) — John Mueller’s verbatim quotes on why “currently not indexed” can persist indefinitely.
• Links are not a top 3 Google Search ranking factor, says Gary Illyes (Search Engine Land) — Illyes on why ranking is many signals and links aren’t in the top three.
• Bing’s search ranking factors (Search Engine Land) — Bing’s six named factors (relevance, quality/credibility, user engagement, freshness, location, page load time).
• Google’s Gary Illyes Explains What Caffeine Does (Search Engine Roundtable) — Illyes on the six functions of Google’s Caffeine indexing system.
• JavaScript SEO basics (Google Search Central) — how rendering fits into indexing and how to keep JS-built content indexable.
• A guide to Google Search ranking systems (Google Search Central) — the authoritative list of ranking systems (RankBrain, BERT, neural matching, link systems) and the distinction from ranking updates.
• Search Off the Record podcast (Google Search Relations) — Gary Illyes, Martin Splitt, and John Mueller discussing crawling, indexing, and ranking from inside Google.

Podcasts

• Search Off the Record (Google Search Relations) — the Google team (Gary Illyes, Martin Splitt, John Mueller) talking through crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and ranking from the inside. The closest you’ll get to the people who actually build the pipeline. Listen

Which stage are you stuck at? — tools to check each gate

Use the same three-gates model, but with the site’s own tools wired to each symptom. Run the one that matches your problem before you touch anything.

Crawling — page isn’t crawled at all

• Check the server response first: HTTP Status Checker — confirms the URL isn’t returning a 4xx/5xx that’s blocking discovery.
• Check robots.txt rules: Robots.txt Tester — tests whether a specific URL is disallowed for GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer./BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.. If you need to fix the file itself, use the Robots.txt Generator.
• Check sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. coverage: Sitemap Validator — confirms the URL is actually listed and the sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. itself is well-formed. Build one with the XML Sitemap Generator if it’s missing entirely.
• Check for redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. problems eating the crawl: Redirect Checker for a single hop, Redirect Chain Mapper for chains and loops.
• Ground truth for what botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. actually did: Log File Analyzer — confirms whether GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer./BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. requested the URL at all, and how often.

Indexing — crawled but “currently not indexed,” or the wrong URL is showing

• Confirm which URL won the duplicate cluster: Canonical Checker — checks the declared canonical against what’s actually indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..
• Check whether Google is seeing the rendered version: Render Gap — compares raw HTML to the rendered DOM to catch content that never makes it past the Web Rendering ServiceTurning HTML, CSS, and JavaScript into the final visual page and DOM..
• Cross-reference indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. status and coverage issues directly in Google Search Console (third-party) — the Log File Analyzer above tells you if it was crawled; Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. tells you if and why it was or wasn’t indexed.

Serving / ranking — indexed but not ranking, or missing search features

• Confirm the page qualifies for the rich result you’re expecting: Rich Result Eligibility Checker and Schema Validator — a broken or missing schema won’t stop indexing, but it will stop the search feature.
• Rule out usability as the drag on ranking: Mobile-Friendly Tester, CWV Checker, and the Core Web Vitals History & Competitor Comparison for field dataPerformance metrics captured from real users, not lab tests. on Core Web VitalsGoogle's three real-user UX metrics — LCP (loading), INP (responsiveness), and CLS (visual stability) — used by Google's ranking systems, with no official weight attached, measured on field data. — usability is one of Google’s five named ranking factors.
• Check on-page relevance signals: On-Page SEO Checker — compares your content against the basics Google names for the “relevance” factor.
• Run a broader pass across a whole site or section: Scout Site Audit Free — useful when you’re not sure which gate is failing and want a first pass across many URLs at once.
• For deeper Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. analysis (query-level performance, indexing coverage, filters): Google Search Console CSV Analyzer.

Third-party tools worth knowing for this diagnosis (not built on this site): Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. (indexing status, coverage reports, URL Inspection), Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. (Bing’s equivalent), and Screaming Frog (crawl simulation and bulk canonical/status auditing).

Test yourself: How Search Works

Five quick questions on how Google crawls, renders, indexes, and serves. Pick an answer for each, then check.

How Search Works

Search works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank.

Related: Crawling, Crawl Budget

How Search Works

“How search works” describes the end-to-end pipeline a search engine runs to turn the open web into an answer. It happens in three stages: crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. (discovering and downloading pages), indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. (renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., understanding, deduplicating, and storing them in a giant database), and serving (understanding the query, then ranking and presenting the most relevant, useful pages for that specific user). Google states it as three stages “and not all pages make it through each stage.”

The mental model that makes the rest of technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own. click: these stages are three gates, not a conveyor belt. Each one has its own pass/fail. CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. does not guarantee indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — Google says “indexing isn’t guaranteed; not every page that Google processes will be indexed.” And indexing does not guarantee ranking — that’s a separate competition decided by many automated ranking systems and hundreds of contextual factors.

Keeping crawl ≠ index ≠ rank straight is the most useful diagnostic you have. When a page isn’t showing up, you don’t guess — you figure out which gate it’s stuck at (not crawled, crawled-but-not-indexed, or indexed-but-not-ranking) and fix that specific stage. Bing runs the same crawl → index → rank shape with its own named factors.

Related: Crawling, Crawl Budget

Changelog