How Search Works

TL;DR — Search works in three steps: a search engine crawls (finds and downloads your pages), indexes (figures out what each page is about and files it away), and serves results (matches a search to the best pages and puts them in order). The catch: a page has to pass each step to make it to the next. Getting found doesn’t mean you’re indexed, and being indexed doesn’t mean you’ll rank.

The three stages

When you search Google, you’re not searching the live web — you’re searching Google’s copy of it. Getting from “a page exists on the internet” to “a page shows up in your results” takes three stages:

1. Crawling — automated programs (Google’s is Googlebot, Bing’s is Bingbot) discover URLs by following links and reading sitemaps, then download the pages.
2. Indexing — the engine looks at what it downloaded, works out what the page is about, and stores it in a giant database called the index.
3. Serving (ranking) — when someone searches, the engine pulls the best matching pages out of the index and puts them in order. Evidence for this claim Google describes Search as operating through crawling, indexing, and serving search results. Scope: Google's public conceptual model, not an exhaustive implementation description. Confidence: high · Verified: 2026-07-14 Google: How Search works

They’re gates, not a slide

The most important thing to understand: these aren’t a conveyor belt that carries every page to the end. They’re three gates, and a page can get stopped at any one of them. Google says it plainly: search works in three stages “and not all pages make it through each stage.” Evidence for this claim Google states that not all pages make it through each Search processing stage and that increased crawl rate is not a ranking signal. Scope: Google Search; crawling and indexing do not guarantee ranking. Confidence: high · Verified: 2026-07-14 Google: How Search works Google: Crawling myths

• A page can be crawled but not indexed — Google looked at it and decided not to store it.
• A page can be indexed but never ranked — it’s in the database, but it doesn’t win for the searches people make.

This is why “Google can see my page, so why isn’t it showing up?” is such a common question. Being crawled is just the first gate.

What this means for you

• If a page isn’t in search at all, figure out which gate it’s stuck at before you change anything — that tells you what to fix.
• You can’t pay Google to rank, and “submitting” a page to Google doesn’t force it to be indexed or to rank.
• Bing works the same way — crawl, index, rank — it just names some of the pieces differently.

Want the accurate, rep-sourced version — with rendering, canonicalization, why indexing isn’t guaranteed, and why ranking is many systems rather than “the algorithm”? Switch to the Advanced tab.

TL;DR — Search runs in three stages — crawl → index → serve — and they’re gates: “not all pages make it through each stage.” Crawling discovers and fetches; rendering (running JavaScript in a recent Chrome) is the bridge into indexing, and it can lag. Indexing = understand + canonicalize (~40 signals)

• store, and it isn’t guaranteed. Serving = understand the query, then rank with many automated ranking systems (RankBrain, BERT, neural matching, link/PageRank systems…) — not one “algorithm” — across hundreds of contextual factors, with search features layered on top. Bing runs the same shape. Treat any diagram (including mine) as a model, not the source code. The payoff: when a page isn’t showing up, diagnose which gate it’s stuck at.

Three stages, three gates

Google is blunt about the pipeline: “Google Search works in three stages, and not all pages make it through each stage” — crawling, indexing, and serving. Evidence for this claim Google describes Search as operating through crawling, indexing, and serving search results. Scope: Google's public conceptual model, not an exhaustive implementation description. Confidence: high · Verified: 2026-07-14 Google: How Search works That last clause is the whole game. The pipeline is not a conveyor belt that carries every page to the finish; it’s three gates, each with its own pass/fail and its own failure mode.

The mental model to anchor everything:

Discover & fetch (crawl) → understand & store (index, including render + canonicalize) → match & order (serve, including query understanding + ranking systems + search features).

Keep crawl ≠ index ≠ rank separate in your head and most of technical SEO stops being mysterious.

Stage 1 — Crawling (discover & fetch)

Crawling is discovery plus download. Google finds pages it already knows, follows links from known pages to new ones, and reads sitemaps you submit — Google calls the whole thing “URL discovery.” What gets crawled, how often, and how many pages per visit is decided algorithmically: “Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.”

The one takeaway at this altitude: crawling is a prerequisite for indexing, not a ranking signal. Evidence for this claim Google states that not all pages make it through each Search processing stage and that increased crawl rate is not a ranking signal. Scope: Google Search; crawling and indexing do not guarantee ranking. Confidence: high · Verified: 2026-07-14 Google: How Search works Google: Crawling myths A higher crawl rate doesn’t lift your positions. I won’t re-derive the crawl pipeline here — discovery, the crawl scheduler, crawl budget, log files, and the rest live in the crawling cluster (linked at the bottom). Crawling’s real job is to hand a fetched page off to the next stage.

One caveat before moving on: crawling is how web pages enter this pipeline, but it isn’t Google’s only source of information. As I cover below, the index also draws on partnerships, data feeds, and Google’s own Knowledge Graph — so “not in the index” and “not on the crawled web” aren’t quite the same question.

The bridge — rendering

Here’s the step most overviews get wrong or skip. Between crawling and indexing, Google renders the page. “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome.” In Google’s own consumer words: “When crawlers find a webpage, our systems render the content of the page, just as a browser does.”

Rendering is part of indexing, not crawling — it runs in the Web Rendering Service (an evergreen, headless Chromium), and it can lag the initial fetch. If your content only appears after JavaScript runs, the version Google understands is the rendered version, and that may show up later than the raw HTML did. This is why JavaScript-heavy sites need to think about rendering as its own concern. (I go deep on this in my JavaScript SEO writing.)

Stage 2 — Indexing (understand & store)

Once a page is crawled and rendered, indexing tries to make sense of it. Google: “After a page is crawled, Google tries to understand what the page is about. This stage is called indexing and it includes processing and analyzing the textual content and key content tags and attributes, such as <title> elements and alt attributes, images, videos, and more.”

Three things happen here that overviews usually miss:

Understand. Google processes the text, tags, and media to work out the topic. In my own How Search Works deck I describe this as a content parser that normalizes the HTML, tokenizes it, and vectorizes the content (turning words into numbers) so the meaning can be analyzed — language, location, schema, and topic/semantic signals get extracted along the way. (That’s my paraphrase of how the processing works, not a Google quote.)

Canonicalize. Indexing also decides which URL represents a set of duplicates. “During the indexing process, Google determines if a page is a duplicate of another page on the internet or canonical. The canonical is the page that may be shown in search results.” Google clusters the duplicates and picks one canonical to stand for the group. This isn’t a coin flip — it’s a signal-weighted decision. As I cover in Google Uses ~40 Canonicalization Signals, “canonicalization is the process that search engines use to determine the main version of the URL that will be indexed and shown to users when there are duplicate URLs” — and according to Google’s Allan Scott, there are ~40 different canonical selection signals (redirects, rel=canonical, sitemap URLs, internal/external links, hreflang, PageRank, HTTPS over HTTP, shorter URLs over longer, and more).

Store. “The collected information about the canonical page and its cluster may be stored in the Google index, a large database hosted on thousands of computers.” That index is enormous — “hundreds of billions of webpages… well over 100,000,000 gigabytes in size” — and it’s not only crawled HTML: “we have multiple indexes of different types of information, which is gathered through crawling, through partnerships, through data feeds being sent to us and through our own encyclopedia of facts, the Knowledge Graph.”

Indexing isn’t guaranteed

The critical myth-buster: getting crawled doesn’t get you indexed. Google: “Indexing isn’t guaranteed; not every page that Google processes will be indexed.” Common causes are low-quality content, a robots noindex, and page design that makes indexing difficult.

John Mueller has put a human voice on this. On how long “discovered/crawled — currently not indexed” can last: “That can be forever. It’s something where we just don’t crawl and index all pages.” And the fix isn’t a trick — it’s substance: “continue working on the website and making sure that our systems recognize that there’s value in crawling and indexing more and then over time we will crawl and index more.”

I’ve proven crawl ≠ index from the other direction too. In Indexed, though blocked by robots.txt the point is simple: “crawling and indexing are two different things.” A page you block from crawling can still get indexed if other pages link to it — Google just can’t see its content (or any noindex you put there).

Stage 3 — Serving results (match & order)

Serving is what happens the instant someone searches. “When a user enters a query, our machines search the index for matching pages and return the results we believe are the highest quality and most relevant to the user’s query.” But there are two distinct moves inside that.

First, understand the query

Before anything gets ranked, Google has to understand what you asked. That means spelling correction, synonyms and query expansion, and figuring out intent. In my How Search Works deck I describe this as spelling correction via a deep neural net, query expansion across synonyms/acronyms/stemming, and neural matching that links conceptually related terms — with a meaningful share of daily queries being ones Google has never seen before. (Paraphrase, from the deck.) Query understanding is its own step; it’s not the same as ranking.

Then, rank — with many systems, not “the algorithm”

This is the big correction. Ranking is not one formula. “Google uses automated ranking systems that look at many factors and signals about hundreds of billions of web pages and other content in our Search index to present the most relevant, useful results, all in a fraction of a second.” Note the plural — ranking systems. Google names several: RankBrain (“an AI system that helps us understand how words are related to concepts”), BERT, neural matching (“an AI system that Google uses to understand representations of concepts in queries and pages and match them to one another”), and link analysis systems and PageRank (“we have various systems that understand how pages link to each other…”). The helpful content system is described in the past tense now — it was folded into core ranking in the March 2024 update, so it’s no longer a standalone live system.

Two distinctions worth teaching:

• Systems ≠ updates. The ranking systems are the ongoing machinery; ranking updates are the changes Google launches to them. They’re tracked separately.
• There’s no fixed “top 3 factors.” Gary Illyes has pushed back on the idea that links are a top-three factor — “I think they are important, but I think people overestimate the importance of links. I don’t agree it’s in the top three. It hasn’t been for some time.” He’s also said “it is possible to rank without links.” And Illyes has argued there isn’t even a universal “top three,” because different sites have different top signals (that framing is Search Engine Land’s paraphrase of him, not a verbatim quote).

The five factors Google names

For a beginner-friendly map, Google’s consumer site breaks ranking into five buckets:

1. Meaning of your query — “To return relevant results, we first need to establish what you’re looking for - the intent behind your query.”
2. Relevance of content — “The most basic signal that information is relevant is when content contains the same keywords as your search query.”
3. Quality of content — “they identify signals that can help determine which content demonstrates expertise, authoritativeness, and trustworthiness.” (Google’s own words here are E-A-T — expertise, authoritativeness, trustworthiness; the “experience” E that makes E-E-A-T is my gloss, not theirs in this quote.)
4. Usability of content — page experience and accessibility considerations.
5. Context and settings — “we use information such as your location, past Search history, and Search settings to determine what is most relevant for you.”

Relevance is contextual: “Relevancy is determined by hundreds of factors, which could include information such as the user’s location, language, and device (desktop or phone).” And the weighting shifts by query — “when searching for current news topics, content freshness plays a bigger role than dictionary definitions.” There is no single “best” result that’s the same for everyone.

Serving isn’t ten blue links

Serving also decides what the page looks like. “Based on the user’s query the search features that appear on the search results page also change.” Knowledge panels, rich results, images, video, local packs — those are part of serving, not a separate system bolted on. And to kill two persistent myths: “we never provide special treatment to advertisers in how our search algorithms rank their websites, and nobody can pay us to do so.” You can’t buy organic position, and submitting a URL only signals — it doesn’t guarantee indexing or ranking.

Bing runs the same shape

Bing follows the same crawl → index → rank pipeline, just named its own way. Microsoft: “Crawling is how Bingbot discovers new and updated pages and content to add to our search index.” Then “as Bingbot crawls the web, it sends information to Bing about what it finds. These pages are then added to the Bing index.” And finally “complex algorithms generate Microsoft Bing search results using the user’s search query matching it with third-party webpages in our index.” Bing publishes its own named ranking factors — relevance, quality and credibility, user engagement, freshness, location, and page load time — the clean analog to Google’s five.

It’s a model, not the source code

One honest caveat before you treat any of this as gospel. Search engines tell us a lot, but not everything, and the internals change. Any pipeline diagram — including mine — is a useful approximation, not the actual code. I put this disclaimer right on slide 3 of my How Search Works deck and I’ll repeat it here: “This is my understanding of systems and is based on a lot of public statements from Google and my own knowledge. Warning: It’s not going to be 100% complete or accurate.” Hold the model loosely; use it to reason, not to litigate.

Which stage are you stuck at?

Here’s where the three-gates model pays off. When a page isn’t performing, don’t guess — locate the failed gate, then fix that stage:

The diagnosis tells you which deep dive you need next.

Where to go next

This page is the map. Each stage gets its own depth:

Crawling — the only stage fully built out so far, as its own cluster. Start with the crawling hub, then go deep on crawl budget (capacity + demand and who actually needs to care), crawl rate (how fast bots fetch and how to speed it up or slow it down), crawl frequency (what makes Google re-crawl sooner), and crawl depth (why important pages belong close to the homepage). For diagnosing problems, see log file analysis (the ground truth for what bots actually did) and spider traps (infinite URL spaces that burn crawl budget). To get pages discovered faster, there’s IndexNow (the push protocol Bing and others use) and the Google Indexing API (and the myth that it indexes anything).

Rendering — the bridge between crawling and indexing, now built out in two deep dives. Rendering covers how Google runs your JavaScript in the Web Rendering Service and the rendering options (CSR, SSR, SSG, hydration, ISR, edge, dynamic) and their SEO trade-offs. JavaScript SEO covers the practical problems and fixes — real <a href> links, DOM parity, lazy content, soft-404s, and why infinite scroll can get two pages indexed as one.

Indexing — canonicalization and indexability have their own cluster now; start with the indexing hub.

Ranking — the ranking systems, the signals behind them, and how E-E-A-T fits in are a topic of their own, coming next.

For everything in this cluster in one place, see the How Search Works cluster.

AI summary

A condensed take on the Advanced version:

• Search = three stages, three gates (crawl → index → serve). Google: “not all pages make it through each stage.” Crawl ≠ index ≠ rank.
• Crawling = discover + fetch (“URL discovery,” algorithmic schedule). It’s a prerequisite for indexing, not a ranking signal. Depth lives in the crawling cluster.
• Rendering is the bridge: Google runs JavaScript in a recent Chrome (the Web Rendering Service) as part of indexing; it can lag the initial crawl.
• Indexing = understand + canonicalize + store. Google picks a canonical among duplicates via ~40 signals (Allan Scott). The index is “well over 100,000,000 gigabytes” and includes multiple indexes + the Knowledge Graph.
• Indexing isn’t guaranteed: “not every page that Google processes will be indexed.” Mueller: “currently not indexed” can be “forever” — usually a quality call.
• Serving = understand the query, then rank. Query understanding (spelling, synonyms, intent) comes first.
• Ranking = many automated systems, not “the algorithm” (RankBrain, BERT, neural matching, link/PageRank systems). Systems ≠ updates. No fixed “top 3”; Illyes: links aren’t top three.
• Relevance is contextual (location, language, device, history; freshness is query-dependent). Google’s five factors: meaning, relevance, quality, usability, context. You can’t pay for organic rank.
• Serving ≠ ten blue links — search features change by query.
• Bing runs the same shape with its own six named factors.
• It’s a model, not the source code (Patrick’s disclaimer). The payoff: diagnose which gate a page is stuck at.

Official documentation

Primary-source documentation from the search engines.

Google

• In-Depth Guide to How Google Search Works — the crawl → index → serve overview, URL discovery, rendering, indexing, and serving.
• How Search organizes information — the consumer explainer of crawling, rendering, and the index (scale + multiple indexes + Knowledge Graph).
• How Search ranking results — the five “key factors”: meaning, relevance, quality, usability, context.
• A guide to Google Search ranking systems — the ranking systems (RankBrain, BERT, neural matching, link systems…), distinct from ranking updates.
• JavaScript SEO basics — rendering as part of indexing and how to keep JS content indexable.

Bing / Microsoft

• How Bing delivers search results — Bing’s crawl → index → rank pipeline and ranking factors.
• bingbot Series: Maximizing Crawl Efficiency — Bing’s definition of crawling and its “crawl efficiency north star.”

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — the three stages

• “Google Search works in three stages, and not all pages make it through each stage.” — Google Search Central docs. Jump to quote
• “Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.” Jump to quote
• “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome.” Jump to quote

Google — indexing

• “After a page is crawled, Google tries to understand what the page is about. This stage is called indexing…” Jump to quote
• “During the indexing process, Google determines if a page is a duplicate of another page on the internet or canonical.” Jump to quote
• “Indexing isn’t guaranteed; not every page that Google processes will be indexed.” Jump to quote
• “When crawlers find a webpage, our systems render the content of the page, just as a browser does.” — Google, How Search organizes information. Jump to quote
• “The Google Search index covers hundreds of billions of webpages and is well over 100,000,000 gigabytes in size.” Jump to quote

Google — serving & ranking

• “When a user enters a query, our machines search the index for matching pages and return the results we believe are the highest quality and most relevant to the user’s query.” Jump to quote
• “Relevancy is determined by hundreds of factors, which could include information such as the user’s location, language, and device (desktop or phone).” Jump to quote
• “Google uses automated ranking systems that look at many factors and signals about hundreds of billions of web pages and other content in our Search index to present the most relevant, useful results, all in a fraction of a second.” — Google, A guide to Google Search ranking systems. Jump to quote
• “RankBrain is an AI system that helps us understand how words are related to concepts.” Jump to quote
• “Neural matching is an AI system that Google uses to understand representations of concepts in queries and pages and match them to one another.” Jump to quote
• “To return relevant results, we first need to establish what you’re looking for - the intent behind your query.” — Google, How Search ranking results. Jump to quote
• “…they identify signals that can help determine which content demonstrates expertise, authoritativeness, and trustworthiness.” Jump to quote
• “We never provide special treatment to advertisers in how our search algorithms rank their websites, and nobody can pay us to do so.” Jump to quote

John Mueller, Google (via Search Engine Journal)

• “That can be forever. It’s something where we just don’t crawl and index all pages.” Read the coverage
• “…continue working on the website and making sure that our systems recognize that there’s value in crawling and indexing more and then over time we will crawl and index more.” Read the coverage

Gary Illyes, Google (via Search Engine Land)

• “I don’t agree it’s in the top three. It hasn’t been for some time.” — on links as a ranking factor. Read the coverage
• “It is possible to rank without links.” Read the coverage

Microsoft Bing

• “Crawling is how Bingbot discovers new and updated pages and content to add to our search index.” Jump to quote
• “Complex algorithms generate Microsoft Bing search results using the user’s search query matching it with third-party webpages in our index.” Jump to quote

Patrick Stox (my own work)

• “Canonicalization is the process that search engines use to determine the main version of the URL that will be indexed and shown to users when there are duplicate URLs.” — and per Google’s Allan Scott, there are ~40 canonical selection signals. Read the article
• “This is my understanding of systems and is based on a lot of public statements from Google and my own knowledge. Warning: It’s not going to be 100% complete or accurate.” — slide 3 of my How Search Works deck. See the slide

The mental models

1. Three gates, not a conveyor belt. Crawl → index → serve. Each is a filter, and “not all pages make it through each stage.” When a page underperforms, locate which stage it failed before you change anything.

2. The three “not equals.”

• Crawling ≠ indexing (a blocked page can still be indexed via links; indexing isn’t guaranteed even when crawling succeeds).
• Crawling ≠ ranking (crawl rate is not a ranking signal).
• Crawling ≠ rendering (JavaScript runs in a separate step that can lag).

3. Indexing = understand + canonicalize + store. Google processes text/tags/media, picks one canonical among duplicates (via ~40 signals), and stores the result. Two pages can both be crawled and still collapse into one indexed URL.

4. Serving = understand the query, then rank. Query understanding (spelling, synonyms, intent) is its own step before ranking. Don’t collapse them.

5. Ranking = many systems, not one algorithm. RankBrain, BERT, neural matching, link/PageRank systems — “ranking systems” plural. And ranking systems ≠ ranking updates. There’s no fixed “top 3 factors.”

6. Relevance is contextual. Location, language, device, history, freshness — there is no single “best” result for everyone. The right answer depends on who’s asking.

7. It’s a model, not the source code. Use the pipeline to reason about problems, not to assert exactly how Google works internally. Hold it loosely.

Which stage are you stuck at? — diagnostic cheat sheet

Map the symptom to the failed gate, then go to the right deep dive.

Crawl ≠ index ≠ rank — what each really means

Resources worth your time

My related writing

• The Beginner’s Guide to Technical SEO — where crawling, indexing, and ranking fit together: “optimizing your website to help search engines find, crawl, understand, and index your pages.”
• Google Uses ~40 Canonicalization Signals — how indexing picks one URL among duplicates (Allan Scott’s ~40 signals).
• JavaScript SEO Issues & Best Practices — rendering as part of indexing, and why it can lag.
• Indexed, though blocked by robots.txt — proof that “crawling and indexing are two different things.”
• The Story of Blocking 2 High-Ranking Pages With Robots.txt — my first-party experiment on crawl vs. rank.

My speaking

• How Search Works (SlideShare) — my full walkthrough of crawling, rendering, indexing, query understanding, and ranking. (My standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From around the industry

• r/TechSEO — the community for crawl/index/rank debugging.
• Google Says ‘Discovered - Currently Not Indexed’ Status Can Last Forever (Search Engine Journal) — John Mueller’s verbatim quotes on why “currently not indexed” can persist indefinitely.
• Links are not a top 3 Google Search ranking factor, says Gary Illyes (Search Engine Land) — Illyes on why ranking is many signals and links aren’t in the top three.
• Bing’s search ranking factors (Search Engine Land) — Bing’s six named factors (relevance, quality/credibility, user engagement, freshness, location, page load time).
• Google’s Gary Illyes Explains What Caffeine Does (Search Engine Roundtable) — Illyes on the six functions of Google’s Caffeine indexing system.
• JavaScript SEO basics (Google Search Central) — how rendering fits into indexing and how to keep JS-built content indexable.
• A guide to Google Search ranking systems (Google Search Central) — the authoritative list of ranking systems (RankBrain, BERT, neural matching, link systems) and the distinction from ranking updates.
• Search Off the Record podcast (Google Search Relations) — Gary Illyes, Martin Splitt, and John Mueller discussing crawling, indexing, and ranking from inside Google.

Podcasts

• Search Off the Record (Google Search Relations) — the Google team (Gary Illyes, Martin Splitt, John Mueller) talking through crawling, indexing, and ranking from the inside. The closest you’ll get to the people who actually build the pipeline. Listen

Which stage are you stuck at? — tools to check each gate

Use the same three-gates model, but with the site’s own tools wired to each symptom. Run the one that matches your problem before you touch anything.

Crawling — page isn’t crawled at all

• Check the server response first: HTTP Status Checker — confirms the URL isn’t returning a 4xx/5xx that’s blocking discovery.
• Check robots.txt rules: Robots.txt Tester — tests whether a specific URL is disallowed for Googlebot/Bingbot. If you need to fix the file itself, use the Robots.txt Generator.
• Check sitemap coverage: Sitemap Validator — confirms the URL is actually listed and the sitemap itself is well-formed. Build one with the XML Sitemap Generator if it’s missing entirely.
• Check for redirect problems eating the crawl: Redirect Checker for a single hop, Redirect Chain Mapper for chains and loops.
• Ground truth for what bots actually did: Log File Analyzer — confirms whether Googlebot/Bingbot requested the URL at all, and how often.

Indexing — crawled but “currently not indexed,” or the wrong URL is showing

• Confirm which URL won the duplicate cluster: Canonical Checker — checks the declared canonical against what’s actually indexed.
• Check whether Google is seeing the rendered version: Render Gap — compares raw HTML to the rendered DOM to catch content that never makes it past the Web Rendering Service.
• Cross-reference indexing status and coverage issues directly in Google Search Console (third-party) — the Log File Analyzer above tells you if it was crawled; Search Console tells you if and why it was or wasn’t indexed.

Serving / ranking — indexed but not ranking, or missing search features

• Confirm the page qualifies for the rich result you’re expecting: Rich Result Eligibility Checker and Schema Validator — a broken or missing schema won’t stop indexing, but it will stop the search feature.
• Rule out usability as the drag on ranking: Mobile-Friendly Tester, CWV Checker, and the Core Web Vitals History & Competitor Comparison for field data on Core Web Vitals — usability is one of Google’s five named ranking factors.
• Check on-page relevance signals: On-Page SEO Checker — compares your content against the basics Google names for the “relevance” factor.
• Run a broader pass across a whole site or section: Scout Site Audit Free — useful when you’re not sure which gate is failing and want a first pass across many URLs at once.
• For deeper Search Console analysis (query-level performance, indexing coverage, filters): Google Search Console CSV Analyzer.

Third-party tools worth knowing for this diagnosis (not built on this site): Google Search Console (indexing status, coverage reports, URL Inspection), Bing Webmaster Tools (Bing’s equivalent), and Screaming Frog (crawl simulation and bulk canonical/status auditing).

Test yourself: How Search Works

Five quick questions on how Google crawls, renders, indexes, and serves. Pick an answer for each, then check.

How Search Works

Search works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank.

Related: Crawling, Crawl Budget

How Search Works

“How search works” describes the end-to-end pipeline a search engine runs to turn the open web into an answer. It happens in three stages: crawling (discovering and downloading pages), indexing (rendering, understanding, deduplicating, and storing them in a giant database), and serving (understanding the query, then ranking and presenting the most relevant, useful pages for that specific user). Google states it as three stages “and not all pages make it through each stage.”

The mental model that makes the rest of technical SEO click: these stages are three gates, not a conveyor belt. Each one has its own pass/fail. Crawling does not guarantee indexing — Google says “indexing isn’t guaranteed; not every page that Google processes will be indexed.” And indexing does not guarantee ranking — that’s a separate competition decided by many automated ranking systems and hundreds of contextual factors.

Keeping crawl ≠ index ≠ rank straight is the most useful diagnostic you have. When a page isn’t showing up, you don’t guess — you figure out which gate it’s stuck at (not crawled, crawled-but-not-indexed, or indexed-but-not-ranking) and fix that specific stage. Bing runs the same crawl → index → rank shape with its own named factors.

Related: Crawling, Crawl Budget

Changelog