Indexing

TL;DR — Indexing is how a search engine stores your page so it can show up in results. After a page is crawled (downloaded), the engine figures out what it’s about and decides whether to keep it. Getting crawled does not mean you’re indexed — Google picks what’s worth keeping. And being indexed isn’t the same as ranking; it just means you’re eligible to.

What indexing is

Search works in three steps, in order:

1. Crawl — a bot like Googlebot discovers a URL and downloads the page.
2. Index — the engine reads that page, figures out what it’s about, and files it away in a giant database of everything it might show in results.
3. Serve (rank) — when someone searches, the engine pulls the best matches from that database and puts them in order.

Indexing is step two. If a page isn’t indexed, it can’t rank — it simply isn’t in the database results are pulled from. But indexing on its own doesn’t lift you up the page; it just gets you into the running.

Crawled doesn’t mean indexed

This is the part people miss. Google doesn’t index every page it crawls — it chooses which ones are worth keeping. In Google’s own words, “indexing isn’t guaranteed; not every page that Google processes will be indexed.” Evidence for this claim Google does not guarantee that every processed page will be indexed. Scope: Google Search indexing; the source gives examples of possible causes rather than an exhaustive decision formula. Confidence: high · Verified: 2026-07-14 Google Search Central: In-depth guide to how Google Search works Pages get left out most often because the content is thin or low-value, because a noindex rule tells Google to skip it, or because something technical makes the page hard to process.

So if a page is missing from search, “Google hasn’t crawled it” and “Google crawled it but didn’t keep it” are two different problems with different fixes.

How to check if you’re indexed

• Google Search Console is the real answer. The URL Inspection tool tells you whether a specific page is indexed, and the Page indexing report shows why pages across your site were or weren’t indexed.
• For one specific URL, use URL Inspection — you can’t search or filter the Page indexing report by URL, so the report is for patterns across the site, not a single-page lookup. And a clean live test in URL Inspection isn’t the full story: it doesn’t check everything the report does, most notably duplicate and canonical conditions.
• A rough shortcut is the site: operator (e.g. site:example.com/page) — handy, but Search Console is the source of truth.

How to keep a page OUT of the index

This is where a lot of people get it backwards. If you want a page gone from search:

• Add a noindex tag (a meta robots tag or an X-Robots-Tag header), and
• Make sure the page is still crawlable — don’t block it in robots.txt.

Why? Because Google has to be able to crawl the page to see the noindex. If you block it in robots.txt, Google can’t read the tag — and the page can actually stay indexed anyway if other pages link to it. Evidence for this claim Google must be able to crawl a page to see and apply its noindex rule. Scope: Google-supported robots meta and X-Robots-Tag directives; a robots.txt block can prevent Google from seeing the rule. Confidence: high · Verified: 2026-07-14 Google Search Central: Block Search indexing with noindex

That covers the everyday case. For removal edge cases — pages you need gone fast, a password-protected or private page that leaked into the index, or a full 404/410 vs. noindex decision — see the dedicated how to deindex a page guide.

Want the deeper version — what actually happens during indexing, how canonicalization works, and how to read every status in the Search Console Page indexing report? Switch to the Advanced tab.

TL;DR — Indexing is the second of search’s three stages (crawl → index → serve): Google understands a crawled page (text, key tags, images, video; it renders JS), detects duplicates, clusters similar pages and picks the most representative one (canonicalization — rel=canonical is a hint, not a rule), computes signals, and stores the canonical in the index. Crawled ≠ indexed — “indexing isn’t guaranteed,” and the call is largely about quality/value. The Search Console Page indexing report is your cockpit. To deindex, use noindex and keep the page crawlable; never use robots.txt to remove a page, because a blocked page can still be indexed (just without a snippet).

Indexing is stage two of three

Google is blunt about the pipeline: “Google Search works in three stages, and not all pages make it through each stage” — crawling, indexing, and serving. Indexing is the middle stage, and the doc defines it cleanly: “Indexing: Google analyzes the text, images, and video files on the page, and stores the information in the Google index, which is a large database.”

A page has to be crawled before it can be indexed, and it has to be indexed before it can rank. But none of those are guarantees — each stage is a filter. Keeping the three stages separate in your head is the single most useful mental model in technical SEO, and it’s why I always ask which stage a page is failing at before changing anything. (For the stage before this one, see the crawling hub — crawl → index is the pipeline.)

What actually happens during indexing

Indexing isn’t one thing; it’s a sequence:

• Understanding the content. Google: “After a page is crawled, Google tries to understand what the page is about. This stage is called indexing.” That means “Google analyzes the textual content and key content tags and attributes, such as <title> elements and alt attributes, images, videos, and more.” JavaScript is rendered as part of this — if your content only appears after JS runs, it still has to render before it can be understood.
• Duplicate detection & canonicalization. This is the part most explainers skip, and it’s where a lot of “why isn’t this indexed?” mysteries live. Google “determines if a page is a duplicate of another page on the internet or canonical.” The mechanic: “we first group together (also known as clustering) the pages that we found on the internet that have similar content, and then we select the one that’s most representative of the group.” That representative is the canonical — “The canonical is the page that may be shown in search results.”
• Computing signals & storing. Finally, “The collected information about the canonical page and its cluster may be stored in the Google index, a large database hosted on thousands of computers.” Google’s named indexing system behind all this is Caffeine — the layer that ingests crawl data, renders and extracts, computes signals, and builds the index that gets served.

Canonicalization: a hint, not a command

Because canonicalization happens during indexing, it deserves its own note. “Canonicalization is the process of selecting the representative –canonical– URL of a piece of content,” and it exists because “this process helps Google show only one version of the otherwise duplicate content in its search results.”

The load-bearing detail: your rel=canonical is a suggestion. Google’s words: “indicating a canonical preference is a hint, not a rule.” Google weighs many signals — in my canonicalization guide I note that, per Google’s Allan Scott, there are roughly 40 different canonical selection signals — and it can pick a different URL than the one you flagged. That’s exactly what the “Duplicate, Google chose different canonical than user” status in Search Console is telling you.

Crawled ≠ indexed: why pages don’t get indexed

Here’s the myth-buster, straight from the docs: “Indexing isn’t guaranteed; not every page that Google processes will be indexed.” Evidence for this claim Google does not guarantee that every processed page will be indexed. Scope: Google Search indexing; the source gives examples of possible causes rather than an exhaustive decision formula. Confidence: high · Verified: 2026-07-14 Google Search Central: In-depth guide to how Google Search works Google lists common reasons it fails — “The quality of the content on page is low,” “Robots meta rules disallow indexing,” and “The design of the website might make indexing difficult.”

The reps are even more direct that this is a selection decision driven by value, not a quota you can buy past:

• John Mueller, on how long “Discovered/Crawled – currently not indexed” can persist: “That can be forever. It’s something where we just don’t crawl and index all pages.” The fix isn’t resubmitting — it’s making the systems recognize the value, to “continue working on the website and making sure that our systems recognize that there’s value in crawling and indexing more and then over time we will crawl and index more.”
• Mueller again: “it’s important to keep in mind that Google just doesn’t index every page on the web, even if it’s submitted directly.” And, bluntly: “Well, lots of SEOs & sites (perhaps not you/yours!) produce terrible content that’s not worth indexing.”
• Gary Illyes, on why it’s selective: “we don’t have infinite space, so we want to index stuff that we think– well, not we– but our algorithms determine that it might be searched for…”
• Martin Splitt frames it as a balancing act: “I usually describe it as a challenge with the balance between not overwhelming the website and also spending our resources where it matters.”

The practical takeaway: a sitemap or “request indexing” aids discovery, not selection. Submitting a page again won’t force it in. The lever is site quality and value.

Reading the Google Search Console Page indexing report

The Page indexing report is where indexing problems actually show up. Treat each status as a diagnosis. These are Google’s own verbatim descriptions:

• Crawled – currently not indexed: “The page was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.” Usually a quality/value judgment — improve the page, don’t spam the resubmit button.
• Discovered – currently not indexed: “The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl.” Technically a pre-crawl, capacity-driven status — but if it persists, reps tie that to value, same as the one above.
• Duplicate without user-selected canonical: “This page is a duplicate of another page, although it doesn’t indicate a preferred canonical page. Google has chosen the other page as the canonical for this page, and so will not serve this page in Search.”
• Duplicate, Google chose different canonical than user: “This page is marked as canonical for a set of pages, but Google thinks another URL makes a better canonical.” (The “hint, not a rule” outcome in the wild.)
• Alternate page with proper canonical tag: “This page is marked as an alternate of another page… This page correctly points to the canonical page, which is indexed, so there is nothing you need to do.”
• Indexed, though blocked by robots.txt: “The page was indexed despite being blocked by your website’s robots.txt file. Google always respects robots.txt, but this doesn’t necessarily prevent indexing if someone else links to your page.” This is the proof that blocking crawling does not block indexing.
• URL blocked by robots.txt: “This page was blocked by your site’s robots.txt file.”
• URL marked ‘noindex’: “When Google tried to index the page it encountered a ‘noindex’ directive and therefore did not index it.” (This is the deindex working as intended.)
• Page with redirect: “This is a non-canonical URL that redirects to another page. As such, this URL will not be indexed.”
• Soft 404: “The page request returns what we think is a soft 404 response. This means that it returns a user-friendly ‘not found’ message but not a 404 HTTP response code.”

How to control indexing the right way

To get a page indexed: make it crawlable, link to it internally, include it in your sitemap — and, above all, make it worth indexing. Discovery aids don’t override the value judgment.

To keep a page OUT — the most-botched control in SEO: use noindex, “a rule set with either a <meta> tag or HTTP response header,” and keep the page crawlable. Google’s load-bearing warning: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.”

The mistake I see constantly is adding noindex and blocking the page in robots.txt. That’s counterproductive. As I put it in How to Remove URLs From Google Search: “For these tags to be seen, a search engine needs to be able to crawl the pages—so make sure they aren’t blocked in robots.txt,” and “Crawling is not the same thing as indexing. Even if Google is blocked from crawling pages, if there are any internal or external links to a page they can still index it.” In my piece on the Indexed, though blocked by robots.txt status I say it even more plainly: “Unless Google can crawl a page, they won’t see the noindex meta tag and may still index it because it has links.” The fix: “Just add a noindex meta robots tag and make sure to allow crawling—assuming it’s canonical.”

For the fuller removal decision tree — 404/410 vs. noindex, the Removals tool’s ~6-month hold, and password protection — see how to deindex a page.

Indexing in Bing

Bing runs the same pipeline. As Microsoft describes it: “As Bingbot crawls the web, it sends information to Bing about what it finds. These pages are then added to the Bing index.” The same controls apply — a noindex directive keeps a page out, and an over-restrictive robots.txt can stop Bingbot from ever crawling it. Bing also needs at least one link pointing to your site to find it in the first place.

Not every page belongs in the index

A simple filter, not a universal rule: a page is worth indexing if it can show up for a search with a distinct, useful result. That’s the bar to check duplicates, parameter variants, private or staging URLs, and thin or repetitive inventory pages against — not a reason to noindex a whole page type by default. Run the full audit on the pages built to own it, next.

Where to go next: the indexing cluster

This hub is the overview. Two things go wrong at scale, and each gets its own deep dive:

• Index bloat — when too many low-value, duplicate, or thin URLs end up in the index, diluting your site and wasting crawl/index resources. How to diagnose it and prune it safely.
• Mobile-first indexing — Google indexes the mobile version of your pages, so content, links, and structured data have to reach parity between mobile and desktop. What to check and what breaks.

Both topics are nested under this hub — they’re in the sidebar too, and they’ll link back here.

The stage before this one — how bots discover and download your pages — lives in the crawling hub; crawl → index is the pipeline, and a page has to clear crawling before any of this applies. For the broader picture, see How Search Works.

AI summary

A condensed take on the Advanced version:

• Indexing = stage two of search (crawl → index → serve). A page must be crawled to be indexed, and indexed to rank — but none of those are guaranteed. It is not a ranking factor, just eligibility.
• What happens during indexing: Google understands the content (text, key tags, images, video; renders JS), detects duplicates, clusters similar pages and picks the most representative (canonical), computes signals, and stores it in the Google index (system: Caffeine).
• Canonicalization happens during indexing. rel=canonical is “a hint, not a rule” — Google can pick a different canonical (~40 selection signals).
• Crawled ≠ indexed. “Indexing isn’t guaranteed.” It’s a quality/value decision: Mueller — “That can be forever”; Illyes — “we don’t have infinite space.” Resubmitting won’t force a page in; improving the site is the lever.
• The GSC Page indexing report is the cockpit: “Crawled/Discovered – currently not indexed” (often value), the duplicate/canonical statuses, “Indexed, though blocked by robots.txt,” “URL marked ‘noindex’.”
• To deindex: use noindex (meta or X-Robots-Tag) and keep the page crawlable — Google must crawl it to see the tag. Never use robots.txt to deindex: a blocked page can still be indexed via links (without a snippet).
• Bing mirrors the pipeline. Same noindex rules apply.
• At scale, two failure modes: index bloat (too many low-value URLs) and mobile-first indexing (which version Google indexes).

Official documentation

Primary-source documentation from the search engines.

Google

• In-Depth Guide to How Google Search Works — the crawl → index → serve overview, the indexing stage, and why “indexing isn’t guaranteed.”
• Canonicalization and duplicate URLs — how Google clusters duplicates and selects a canonical (the hint-not-a-rule doc).
• Block Search indexing with noindex — the correct deindex tool, and why the page must stay crawlable.
• Page Indexing report — Search Console help for every indexing status and what it means.
• Crawling and Indexing — the hub for robots, sitemaps, canonicalization, and indexing controls.

Bing / Microsoft

• How Bing delivers search results — Bing’s crawl → index pipeline in Microsoft’s own words.
• Why is My Site Not in the Index? — Bing Webmaster Tools help on indexing barriers.

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what indexing is

• “Google Search works in three stages, and not all pages make it through each stage.” — Google Search Central docs. Jump to quote
• “After a page is crawled, Google tries to understand what the page is about. This stage is called indexing.” Jump to quote
• “Google analyzes the textual content and key content tags and attributes, such as <title> elements and alt attributes, images, videos, and more.” Jump to quote
• “The canonical is the page that may be shown in search results… we first group together (also known as clustering) the pages that we found on the internet that have similar content, and then we select the one that’s most representative of the group.” Jump to quote
• “The collected information about the canonical page and its cluster may be stored in the Google index, a large database hosted on thousands of computers.” Jump to quote
• “Indexing isn’t guaranteed; not every page that Google processes will be indexed.” Jump to quote

Google — canonicalization & noindex

• “Canonicalization is the process of selecting the representative –canonical– URL of a piece of content.” — Google Search Central docs. Jump to quote
• “That is, indicating a canonical preference is a hint, not a rule.” Jump to quote
• “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.” — Google Search Central docs. Jump to quote

John Mueller, Google (via Search Engine Journal)

• “That can be forever. It’s something where we just don’t crawl and index all pages.” Read the coverage
• “it’s important to keep in mind that Google just doesn’t index every page on the web, even if it’s submitted directly.” Read the coverage

Gary Illyes & Martin Splitt, Google (via Search Engine Journal)

• Illyes: “we don’t have infinite space, so we want to index stuff that we think– well, not we– but our algorithms determine that it might be searched for…” Read the coverage
• Splitt: “I usually describe it as a challenge with the balance between not overwhelming the website and also spending our resources where it matters.” Read the coverage

Bing / Microsoft

• “As Bingbot crawls the web, it sends information to Bing about what it finds. These pages are then added to the Bing index.” — Microsoft Support. Read the source

Indexing health checklist

A quick pass to confirm the right pages are indexed and the wrong ones aren’t:

• Pages you want indexed are crawlable (not blocked in robots.txt) and reachable via internal links — no orphans.
• Important pages return 200, aren’t noindex’d by accident, and point their rel=canonical at themselves (or the right canonical).
• The Search Console Page indexing report shows your key templates as “Indexed,” and you’ve triaged “Crawled/Discovered – currently not indexed.”
• Duplicate/canonical statuses reviewed — confirm Google’s chosen canonical matches your intent.
• Pages you want out use noindex (meta or X-Robots-Tag) and remain crawlable so Google can see the tag.
• You’re not using robots.txt to deindex anything (blocked ≠ removed).
• No “Indexed, though blocked by robots.txt” surprises in the report.
• JS-dependent content renders to indexable HTML (it has to render before it can be understood).
• Sitemap lists only canonical, indexable URLs (it aids discovery, not selection).

The mental models

1. The pipeline — crawl → index → serve. Each stage is a filter, and “not all pages make it through each stage.” Before changing anything, locate which stage a page is failing at: Was it crawled? Indexed? Served for the query?

2. Crawled ≠ indexed ≠ ranking. Crawled means downloaded. Indexed means stored and eligible. Ranking is a separate contest among indexed pages. Indexing is not a ranking factor — but you can’t rank without it.

3. Indexing is a selection decision. Google chooses what to keep — “indexing isn’t guaranteed.” If a page isn’t indexed, the question is rarely “did I submit it?” and almost always “is it worth keeping?” Improve value; don’t resubmit.

4. Canonicalization is part of indexing. Duplicates get clustered and one representative URL is stored. rel=canonical is a hint — Google can overrule it. If the wrong URL is indexed, look at signals (internal links, sitemaps, redirects), not just the tag.

5. The decision rule for keeping a page out. Want it gone from the index? Allow crawling + noindex. Want bots to skip a URL space entirely (and don’t care whether stray copies get indexed via links)? robots.txt disallow. Never use disallow to deindex.

Indexing controls & statuses — cheat sheet

What each control actually does

Reading the Page indexing report (high-value statuses)

Fast facts

• “Indexing isn’t guaranteed” — Google selects what to keep.
• noindex only works if the page is crawlable.
• robots.txt blocks crawling, not indexing — a blocked page can still be indexed.
• rel=canonical is a hint, not a directive (~40 canonical selection signals).
• Indexing is not a ranking factor — it’s the gate to being eligible to rank.

Tools for seeing and managing indexing

Check whether a specific page is indexed with the Google Index Checker:

1. Paste the exact public URL into the tool.
2. Run Check signals to fetch the live response.
3. Read the status, redirect, noindex, and canonical signals it reports.
4. Follow the Open URL Inspection handoff for Google’s actual index verdict — the tool only reports what’s observable from the public page.

• Google Search Console — Page indexing report — Google’s own view of which pages are indexed and why others aren’t, status by status.
• URL Inspection (GSC) — check whether a single URL is indexed, see Google’s chosen canonical, and view the rendered HTML.
• Bing Webmaster Tools — index coverage, URL inspection, and submission for Bing.
• The site: operator — a quick rough check of what’s indexed (not a substitute for Search Console).
• Crawlers / site audits — Ahrefs Site Audit and Screaming Frog SEO Spider surface noindex tags, canonical conflicts, duplicate clusters, and indexability issues at scale.
• Ahrefs Webmaster Tools — free crawl + audit for sites you verify, flagging indexability problems.

Resources worth your time

My related writing

• The Beginner’s Guide to Technical SEO — where indexing fits in the bigger picture.
• How to Remove URLs From Google Search (5 Methods) — the right (and wrong) ways to deindex.
• Indexed, though blocked by robots.txt — why blocked pages still get indexed, and the fix.
• Canonicalization: A Beginner’s Guide — how Google picks the URL it indexes.
• What “Crawled - Currently Not Indexed” Means in Google Search Console — diagnosing the most common “not indexed” status.

From others

• Robots Meta Tag & X-Robots-Tag: Everything You Need to Know (Michal Pecánek, Ahrefs) — the definitive reference on noindex directives, including: “Never disallow crawling of content that you’re trying to get deindexed in robots.txt.”
• r/TechSEO — the community for crawl/index debugging.

From around the industry

• Google Says ‘Discovered - Currently Not Indexed’ Status Can Last Forever (Search Engine Journal) — Mueller’s “That can be forever” quote in context, with practical takeaways for sites stuck in this status.
• Google Shares Insights into Indexing & Crawl Budget (Search Engine Journal) — Mueller, Illyes, and Splitt on why Google doesn’t index everything and how to think about the crawl/index value decision.
• Gary Illyes Talks On Information Retrieval At Google Search (Search Engine Roundtable) — Illyes on finite index space and why Google is selective about what it stores.
• Spilling the Beans on Caffeine (Google’s Indexing System) (Search Off the Record, Google) — the Google Search Relations team explains how Caffeine ingests crawl data, renders pages, extracts signals, and builds the index.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawling, rendering, indexing, and ranking. (My standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

Podcasts

• Search Off the Record (Google Search Relations) — Spilling the beans on Caffeine (Google’s indexing system) and more! The Google team on how the indexing system ingests crawl data, renders and extracts, computes signals, and builds the index. Listen

Videos

• Google Search Central (YouTube) — the How Google Search Works series and Martin Splitt’s indexing/rendering explainers, including the canonicalization and JavaScript SEO videos. Channel

Test yourself: Indexing

Five quick questions on how pages get indexed (and why they might not). Pick an answer for each, then check.

Indexing

Storing a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.

Related: Canonicalization, Duplicate Content, Crawling

Indexing

Indexing is the second of search’s three stages — crawl → index → serve. After a page is crawled, the engine tries to understand what it’s about: it processes the text, key tags and attributes, images, and video, and (if the page qualifies) stores that information in the search index, a giant database it queries at search time.

Crucially, crawled is not the same as indexed. Google selects which crawled pages to keep — “indexing isn’t guaranteed.” Pages get left out for low quality, a noindex rule, or technical barriers. And a page can’t rank until it’s indexed, so indexing is a prerequisite for ranking, not a guarantee of it.

Indexing also includes canonicalization: the engine clusters pages with similar content and picks the single most representative URL to store and show. A rel=canonical tag is a hint, not a directive — Google can choose a different canonical. To keep a page out of the index, use noindex (a meta robots tag or X-Robots-Tag header) and leave the page crawlable; blocking it in robots.txt stops crawling, not indexing, so a blocked page can still be indexed via links.

Related: Canonicalization, Duplicate Content, Crawling

Changelog