Indexing

TL;DR — IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. is how a search engine stores your page so it can show up in results. After a page is crawled (downloaded), the engine figures out what it’s about and decides whether to keep it. Getting crawled does not mean you’re indexed — Google picks what’s worth keeping. And being indexed isn’t the same as ranking; it just means you’re eligible to.

What indexing is

Search works in three steps, in order:

1. Crawl — a botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. like GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. discovers a URL and downloads the page.
2. Index — the engine reads that page, figures out what it’s about, and files it away in a giant database of everything it might show in results.
3. Serve (rank) — when someone searches, the engine pulls the best matches from that database and puts them in order.

Indexing is step two. If a page isn’t indexed, it can’t rank — it simply isn’t in the database results are pulled from. But indexing on its own doesn’t lift you up the page; it just gets you into the running.

Crawled doesn’t mean indexed

This is the part people miss. Google doesn’t index every page it crawls — it chooses which ones are worth keeping. In Google’s own words, “indexing isn’t guaranteed; not every page that Google processes will be indexed.” Evidence for this claim Google does not guarantee that every processed page will be indexed. Scope: Google Search indexing; the source gives examples of possible causes rather than an exhaustive decision formula. Confidence: high · Verified: 2026-07-14 Google Search Central: In-depth guide to how Google Search works Pages get left out most often because the content is thin or low-value, because a noindex rule tells Google to skip it, or because something technical makes the page hard to process.

So if a page is missing from search, “Google hasn’t crawled it” and “Google crawled it but didn’t keep it” are two different problems with different fixes.

How to check if you’re indexed

• Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. is the real answer. The URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. tool tells you whether a specific page is indexed, and the Page indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. shows why pages across your site were or weren’t indexed.
• For one specific URL, use URL Inspection — you can’t search or filter the Page indexing report by URL, so the report is for patterns across the site, not a single-page lookup. And a clean live test in URL Inspection isn’t the full story: it doesn’t check everything the report does, most notably duplicate and canonical conditions.
• A rough shortcut is the site: operator (e.g. site:example.com/page) — handy, but Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. is the source of truth.

How to keep a page OUT of the index

This is where a lot of people get it backwards. If you want a page gone from search:

• Add a noindex tag (a meta robotsThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen. tag or an X-Robots-Tag header), and
• Make sure the page is still crawlable — don’t block it in robots.txt.

Why? Because Google has to be able to crawl the page to see the noindex. If you block it in robots.txt, Google can’t read the tag — and the page can actually stay indexed anyway if other pages link to it. Evidence for this claim Google must be able to crawl a page to see and apply its noindex rule. Scope: Google-supported robots meta and X-Robots-Tag directives; a robots.txt block can prevent Google from seeing the rule. Confidence: high · Verified: 2026-07-14 Google Search Central: Block Search indexing with noindex

That covers the everyday case. For removal edge cases — pages you need gone fast, a password-protected or private page that leaked into the index, or a full 404/410 vs. noindex decision — see the dedicated how to deindex a pageDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. guide.

Want the deeper version — what actually happens during indexing, how canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. works, and how to read every status in the Search Console Page indexing report? Switch to the Advanced tab.

TL;DR — IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. is the second of search’s three stages (crawl → index → serve): Google understands a crawled page (text, key tags, images, video; it renders JS), detects duplicates, clusters similar pages and picks the most representative one (canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. — rel=canonical is a hint, not a rule), computes signals, and stores the canonical in the index. Crawled ≠ indexed — “indexing isn’t guaranteed,” and the call is largely about quality/value. The Search Console Page indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. is your cockpit. To deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist., use noindex and keep the page crawlable; never use robots.txt to remove a page, because a blocked page can still be indexed (just without a snippet).

Indexing is stage two of three

Google is blunt about the pipeline: “Google Search works in three stages, and not all pages make it through each stage” — crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., indexing, and serving. Indexing is the middle stage, and the doc defines it cleanly: “Indexing: Google analyzes the text, images, and video files on the page, and stores the information in the Google index, which is a large database.”

A page has to be crawled before it can be indexed, and it has to be indexed before it can rank. But none of those are guarantees — each stage is a filter. Keeping the three stages separate in your head is the single most useful mental model in technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own., and it’s why I always ask which stage a page is failing at before changing anything. (For the stage before this one, see the crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. hub — crawl → index is the pipeline.)

What actually happens during indexing

Indexing isn’t one thing; it’s a sequence:

• Understanding the content. Google: “After a page is crawled, Google tries to understand what the page is about. This stage is called indexing.” That means “Google analyzes the textual content and key content tags and attributes, such as <title> elements and alt attributesAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search., images, videos, and more.” JavaScript is rendered as part of this — if your content only appears after JS runs, it still has to render before it can be understood.
• Duplicate detection & canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it.. This is the part most explainers skip, and it’s where a lot of “why isn’t this indexed?” mysteries live. Google “determines if a page is a duplicate of another page on the internet or canonical.” The mechanic: “we first group together (also known as clustering) the pages that we foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. on the internet that have similar content, and then we select the one that’s most representative of the group.” That representative is the canonical — “The canonical is the page that may be shown in search results.”
• Computing signals & storing. Finally, “The collected information about the canonical page and its cluster may be stored in the Google index, a large database hosted on thousands of computers.” Google’s named indexing system behind all this is Caffeine — the layer that ingests crawl data, renders and extracts, computes signals, and builds the index that gets served.

Canonicalization: a hint, not a command

Because canonicalization happens during indexing, it deserves its own note. “Canonicalization is the process of selecting the representative –canonical– URL of a piece of content,” and it exists because “this process helps Google show only one version of the otherwise duplicate contentThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling. in its search results.”

The load-bearing detail: your rel=canonical is a suggestion. Google’s words: “indicating a canonical preference is a hint, not a rule.” Google weighs many signals — in my canonicalization guide I note that, per Google’s Allan Scott, there are roughly 40 different canonical selection signals — and it can pick a different URL than the one you flagged. That’s exactly what the “Duplicate, Google chose different canonical than userA Google Search Console Page Indexing status: you declared a canonical for this URL, but Google overrode your choice, picked a different page as the canonical, and indexed that one instead.” status in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. is telling you.

Crawled ≠ indexed: why pages don’t get indexed

Here’s the myth-buster, straight from the docs: “Indexing isn’t guaranteed; not every page that Google processes will be indexed.” Evidence for this claim Google does not guarantee that every processed page will be indexed. Scope: Google Search indexing; the source gives examples of possible causes rather than an exhaustive decision formula. Confidence: high · Verified: 2026-07-14 Google Search Central: In-depth guide to how Google Search works Google lists common reasons it fails — “The quality of the content on page is low,” “Robots meta rules disallow indexing,” and “The design of the website might make indexing difficult.”

The reps are even more direct that this is a selection decision driven by value, not a quota you can buy past:

• John Mueller, on how long “Discovered/Crawled – currently not indexedA Google Search Console Page Indexing status meaning Googlebot fetched the page but Google decided not to index it — usually a content- or site-quality signal, not a technical error.” can persist: “That can be forever. It’s something where we just don’t crawl and index all pages.” The fix isn’t resubmitting — it’s making the systems recognize the value, to “continue working on the website and making sure that our systems recognize that there’s value in crawling and indexing more and then over time we will crawl and index more.”
• Mueller again: “it’s important to keep in mind that Google just doesn’t index every page on the web, even if it’s submitted directly.” And, bluntly: “Well, lots of SEOs & sites (perhaps not you/yours!) produce terrible content that’s not worth indexing.”
• Gary Illyes, on why it’s selective: “we don’t have infinite space, so we want to index stuff that we think– well, not we– but our algorithms determine that it might be searched for…”
• Martin Splitt frames it as a balancing act: “I usually describe it as a challenge with the balance between not overwhelming the website and also spending our resources where it matters.”

The practical takeaway: a sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. or “request indexing” aids discovery, not selection. Submitting a page again won’t force it in. The lever is site quality and value.

Reading the Google Search Console Page indexing report

The Page indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. is where indexing problems actually show up. Treat each status as a diagnosis. These are Google’s own verbatim descriptions:

• Crawled – currently not indexed: “The page was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.” Usually a quality/value judgment — improve the page, don’t spam the resubmit button.
• Discovered – currently not indexedA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal.: “The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl.” Technically a pre-crawl, capacity-driven status — but if it persists, reps tie that to value, same as the one above.
• Duplicate without user-selected canonicalA Google Search Console Page Indexing status: Google found this page to be a duplicate, you didn't declare a canonical, so Google chose a different page as the canonical — and this URL isn't indexed.: “This page is a duplicate of another page, although it doesn’t indicate a preferred canonical page. Google has chosen the other page as the canonical for this page, and so will not serve this page in Search.”
• Duplicate, Google chose different canonical than user: “This page is marked as canonical for a set of pages, but Google thinks another URL makes a better canonical.” (The “hint, not a rule” outcome in the wild.)
• Alternate page with proper canonical tagA Google Search Console Page Indexing status meaning a page is a duplicate or alternate version that correctly points its canonical at another, indexed page. It's normal, healthy behavior — Google says there is nothing you need to do.: “This page is marked as an alternate of another page… This page correctly points to the canonical page, which is indexed, so there is nothing you need to do.”
• Indexed, though blocked by robots.txtA Google Search Console Page Indexing warning: Google indexed the URL anyway despite your robots.txt disallowing crawling it — Google names other pages linking to it as the likely path. robots.txt blocks crawling, not indexing.: “The page was indexed despite being blocked by your website’s robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file. Google always respects robots.txt, but this doesn’t necessarily prevent indexing if someone else links to your page.” This is the proof that blocking crawling does not block indexing.
• URL blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.: “This page was blocked by your site’s robots.txt file.”
• URL marked ‘noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed.’: “When Google tried to index the page it encountered a ‘noindex’ directive and therefore did not index it.” (This is the deindex working as intended.)
• Page with redirectA Google Search Console Page Indexing status for a URL that redirects elsewhere. It's not indexed by design because it's a redirect — the destination is a separate question, and Google says the target may or may not end up indexed — usually expected, not an error (unlike the separate \"Redirect error\").: “This is a non-canonical URL that redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. to another page. As such, this URL will not be indexed.”
• Soft 404A soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing.: “The page request returns what we think is a soft 404 response. This means that it returns a user-friendly ‘not found’ message but not a 404 HTTP response code.”

How to control indexing the right way

To get a page indexed: make it crawlable, link to it internally, include it in your sitemap — and, above all, make it worth indexing. Discovery aids don’t override the value judgment.

To keep a page OUT — the most-botched control in SEO: use noindex, “a rule set with either a <meta> tag or HTTP response header,” and keep the page crawlable. Google’s load-bearing warning: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index..”

The mistake I see constantly is adding noindex and blocking the page in robots.txt. That’s counterproductive. As I put it in How to Remove URLs From Google Search: “For these tags to be seen, a search engine needs to be able to crawl the pages—so make sure they aren’t blocked in robots.txt,” and “Crawling is not the same thing as indexing. Even if Google is blocked from crawling pages, if there are any internal or external links to a page they can still index it.” In my piece on the Indexed, though blocked by robots.txt status I say it even more plainly: “Unless Google can crawl a page, they won’t see the noindex meta tag and may still index it because it has links.” The fix: “Just add a noindex meta robots tag and make sure to allow crawling—assuming it’s canonical.”

For the fuller removal decision tree — 404/410 vs. noindex, the Removals toolA Google Search Console feature for verified owners to manage how their URLs appear in Search. A Temporary Removal hides a URL for about six months — it does not delete the page from the index.’s ~6-month hold, and password protection — see how to deindex a pageDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist..

Indexing in Bing

Bing runs the same pipeline. As Microsoft describes it: “As BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. crawls the web, it sends information to Bing about what it finds. These pages are then added to the Bing index.” The same controls apply — a noindex directive keeps a page out, and an over-restrictive robots.txt can stop Bingbot from ever crawling it. Bing also needs at least one link pointing to your site to find it in the first place.

Not every page belongs in the index

A simple filter, not a universal rule: a page is worth indexing if it can show up for a search with a distinct, useful result. That’s the bar to check duplicates, parameter variants, private or staging URLs, and thin or repetitive inventory pages against — not a reason to noindex a whole page type by default. Run the full audit on the pages built to own it, next.

Where to go next: the indexing cluster

This hub is the overview. Two things go wrong at scale, and each gets its own deep dive:

• Index bloatAn SEO term for when a search engine has indexed a lot of low-value, thin, or duplicate URLs that don't serve search demand. It's a quality and crawl-efficiency problem, not a penalty. — when too many low-value, duplicate, or thin URLs end up in the index, diluting your site and wasting crawl/index resources. How to diagnose it and prune it safely.
• Mobile-first indexingGoogle's practice of using the mobile version of a page's content — crawled by Googlebot smartphone — for indexing and ranking. It is not a separate index and not a ranking boost. — Google indexes the mobile version of your pages, so content, links, and structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding. have to reach parity between mobile and desktop. What to check and what breaks.

Both topics are nested under this hub — they’re in the sidebar too, and they’ll link back here.

The stage before this one — how bots discover and download your pages — lives in the crawling hub; crawl → index is the pipeline, and a page has to clear crawling before any of this applies. For the broader picture, see How Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank..

AI summary

A condensed take on the Advanced version:

• IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. = stage two of search (crawl → index → serve). A page must be crawled to be indexed, and indexed to rank — but none of those are guaranteed. It is not a ranking factor, just eligibility.
• What happens during indexing: Google understands the content (text, key tags, images, video; renders JS), detects duplicates, clusters similar pages and picks the most representative (canonical), computes signals, and stores it in the Google index (system: Caffeine).
• CanonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. happens during indexing. rel=canonical is “a hint, not a rule” — Google can pick a different canonical (~40 selection signals).
• Crawled ≠ indexed. “Indexing isn’t guaranteed.” It’s a quality/value decision: Mueller — “That can be forever”; Illyes — “we don’t have infinite space.” Resubmitting won’t force a page in; improving the site is the lever.
• The GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. Page indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. is the cockpit: “Crawled/Discovered – currently not indexed” (often value), the duplicate/canonical statuses, “Indexed, though blocked by robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.,” “URL marked ‘noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed.’.”
• To deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist.: use noindex (meta or X-Robots-TagThe X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos.) and keep the page crawlable — Google must crawl it to see the tag. Never use robots.txt to deindex: a blocked page can still be indexed via links (without a snippet).
• Bing mirrors the pipeline. Same noindex rules apply.
• At scale, two failure modes: index bloatAn SEO term for when a search engine has indexed a lot of low-value, thin, or duplicate URLs that don't serve search demand. It's a quality and crawl-efficiency problem, not a penalty. (too many low-value URLs) and mobile-first indexingGoogle's practice of using the mobile version of a page's content — crawled by Googlebot smartphone — for indexing and ranking. It is not a separate index and not a ranking boost. (which version Google indexes).

Official documentation

Primary-source documentation from the search engines.

Google

• In-Depth Guide to How Google Search Works — the crawl → index → serve overview, the indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. stage, and why “indexing isn’t guaranteed.”
• Canonicalization and duplicate URLs — how Google clusters duplicates and selects a canonical (the hint-not-a-rule doc).
• Block Search indexing with noindex — the correct deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. tool, and why the page must stay crawlable.
• Page Indexing report — Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. help for every indexing status and what it means.
• Crawling and Indexing — the hub for robots, sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it., and indexing controls.

Bing / Microsoft

• How Bing delivers search results — Bing’s crawl → index pipeline in Microsoft’s own words.
• Why is My Site Not in the Index? — Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. help on indexing barriers.

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. is

• “Google Search works in three stages, and not all pages make it through each stage.” — Google Search Central docs. Jump to quote
• “After a page is crawled, Google tries to understand what the page is about. This stage is called indexing.” Jump to quote
• “Google analyzes the textual content and key content tags and attributes, such as <title> elements and alt attributesAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search., images, videos, and more.” Jump to quote
• “The canonical is the page that may be shown in search results… we first group together (also known as clustering) the pages that we foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. on the internet that have similar content, and then we select the one that’s most representative of the group.” Jump to quote
• “The collected information about the canonical page and its cluster may be stored in the Google index, a large database hosted on thousands of computers.” Jump to quote
• “Indexing isn’t guaranteed; not every page that Google processes will be indexed.” Jump to quote

Google — canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. & noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed.

• “CanonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. is the process of selecting the representative –canonical– URL of a piece of content.” — Google Search Central docs. Jump to quote
• “That is, indicating a canonical preference is a hint, not a rule.” Jump to quote
• “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file, and it has to be otherwise accessible to the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index..” — Google Search Central docs. Jump to quote

John Mueller, Google (via Search Engine Journal)

• “That can be forever. It’s something where we just don’t crawl and index all pages.” Read the coverage
• “it’s important to keep in mind that Google just doesn’t index every page on the web, even if it’s submitted directly.” Read the coverage

Gary Illyes & Martin Splitt, Google (via Search Engine Journal)

• Illyes: “we don’t have infinite space, so we want to index stuff that we think– well, not we– but our algorithms determine that it might be searched for…” Read the coverage
• Splitt: “I usually describe it as a challenge with the balance between not overwhelming the website and also spending our resources where it matters.” Read the coverage

Bing / Microsoft

• “As BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. crawls the web, it sends information to Bing about what it finds. These pages are then added to the Bing index.” — Microsoft Support. Read the source

Indexing health checklist

A quick pass to confirm the right pages are indexed and the wrong ones aren’t:

• Pages you want indexed are crawlable (not blocked in robots.txt) and reachable via internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. — no orphans.
• Important pages return 200, aren’t noindex’d by accident, and point their rel=canonical at themselves (or the right canonical).
• The Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. Page indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. shows your key templates as “Indexed,” and you’ve triaged “Crawled/Discovered – currently not indexedA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal..”
• Duplicate/canonical statuses reviewed — confirm Google’s chosen canonical matches your intent.
• Pages you want out use noindex (meta or X-Robots-Tag) and remain crawlable so Google can see the tag.
• You’re not using robots.txt to deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. anything (blocked ≠ removed).
• No “Indexed, though blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” surprises in the report.
• JS-dependent content renders to indexable HTML (it has to render before it can be understood).
• SitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. lists only canonical, indexable URLs (it aids discovery, not selection).

The mental models

1. The pipeline — crawl → index → serve. Each stage is a filter, and “not all pages make it through each stage.” Before changing anything, locate which stage a page is failing at: Was it crawled? Indexed? Served for the query?

2. Crawled ≠ indexed ≠ ranking. Crawled means downloaded. Indexed means stored and eligible. Ranking is a separate contest among indexed pages. IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. is not a ranking factor — but you can’t rank without it.

3. Indexing is a selection decision. Google chooses what to keep — “indexing isn’t guaranteed.” If a page isn’t indexed, the question is rarely “did I submit it?” and almost always “is it worth keeping?” Improve value; don’t resubmit.

4. CanonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. is part of indexing. Duplicates get clustered and one representative URL is stored. rel=canonical is a hint — Google can overrule it. If the wrong URL is indexed, look at signals (internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't.), not just the tag.

5. The decision rule for keeping a page out. Want it gone from the index? Allow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. + noindex. Want botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. to skip a URL space entirely (and don’t care whether stray copies get indexed via links)? robots.txt disallow. Never use disallow to deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist..

Indexing controls & statuses — cheat sheet

What each control actually does

Reading the Page indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. (high-value statuses)

Fast facts

• “Indexing isn’t guaranteed” — Google selects what to keep.
• noindex only works if the page is crawlable.
• robots.txt blocks crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexing — a blocked page can still be indexed.
• rel=canonical is a hint, not a directive (~40 canonical selection signals).
• Indexing is not a ranking factor — it’s the gate to being eligible to rank.

Tools for seeing and managing indexing

Check whether a specific page is indexed with the Google Index Checker:

1. Paste the exact public URL into the tool.
2. Run Check signals to fetch the live response.
3. Read the status, redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed., and canonical signals it reports.
4. Follow the Open URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. handoff for Google’s actual index verdict — the tool only reports what’s observable from the public page.

• Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. — Page indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. — Google’s own view of which pages are indexed and why others aren’t, status by status.
• URL Inspection (GSC) — check whether a single URL is indexed, see Google’s chosen canonical, and view the rendered HTML.
• Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. — index coverage, URL inspection, and submission for Bing.
• The site: operator — a quick rough check of what’s indexed (not a substitute for Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance.).
• CrawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. / site audits — Ahrefs Site Audit and Screaming Frog SEO Spider surface noindex tags, canonical conflicts, duplicate clusters, and indexability issues at scale.
• Ahrefs Webmaster ToolsThe free, first-party consoles search engines give site owners — Google Search Console and Bing Webmaster Tools — to see how their pages are crawled, indexed, and ranked, and to fix problems. The legacy name is 'webmaster tools.' — free crawl + audit for sites you verify, flagging indexability problems.

Resources worth your time

My related writing

• The Beginner’s Guide to Technical SEO — where indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. fits in the bigger picture.
• How to Remove URLs From Google Search (5 Methods) — the right (and wrong) ways to deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist..
• Indexed, though blocked by robots.txt — why blocked pages still get indexed, and the fix.
• Canonicalization: A Beginner’s Guide — how Google picks the URL it indexes.
• What “Crawled - Currently Not Indexed” Means in Google Search Console — diagnosing the most common “not indexed” status.

From others

• Robots Meta Tag & X-Robots-Tag: Everything You Need to Know (Michal Pecánek, Ahrefs) — the definitive reference on noindex directives, including: “Never disallow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. of content that you’re trying to get deindexed in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere..”
• r/TechSEO — the community for crawl/index debugging.

From around the industry

• Google Says ‘Discovered - Currently Not Indexed’ Status Can Last Forever (Search Engine Journal) — Mueller’s “That can be forever” quote in context, with practical takeaways for sites stuck in this status.
• Google Shares Insights into Indexing & Crawl Budget (Search Engine Journal) — Mueller, Illyes, and Splitt on why Google doesn’t index everything and how to think about the crawl/index value decision.
• Gary Illyes Talks On Information Retrieval At Google Search (Search Engine Roundtable) — Illyes on finite index space and why Google is selective about what it stores.
• Spilling the Beans on Caffeine (Google’s Indexing System) (Search Off the Record, Google) — the Google Search Relations team explains how Caffeine ingests crawl data, renders pages, extracts signals, and builds the index.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., indexing, and ranking. (My standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

Podcasts

• Search Off the Record (Google Search Relations) — Spilling the beans on Caffeine (Google’s indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. system) and more! The Google team on how the indexing system ingests crawl data, renders and extracts, computes signals, and builds the index. Listen

Videos

• Google Search Central (YouTube) — the How Google Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank. series and Martin Splitt’s indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed./renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. explainers, including the canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. and JavaScript SEOMaking sure search engines can crawl, render, and index content that depends on JavaScript. videos. Channel

Test yourself: Indexing

Five quick questions on how pages get indexed (and why they might not). Pick an answer for each, then check.

Indexing

Storing a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.

Related: Canonicalization, Duplicate Content, Crawling

Indexing

Indexing is the second of search’s three stages — crawl → index → serve. After a page is crawled, the engine tries to understand what it’s about: it processes the text, key tags and attributes, images, and video, and (if the page qualifies) stores that information in the search index, a giant database it queries at search time.

Crucially, crawled is not the same as indexed. Google selects which crawled pages to keep — “indexing isn’t guaranteed.” Pages get left out for low quality, a noindex rule, or technical barriers. And a page can’t rank until it’s indexed, so indexing is a prerequisite for ranking, not a guarantee of it.

Indexing also includes canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it.: the engine clusters pages with similar content and picks the single most representative URL to store and show. A rel=canonical tag is a hint, not a directive — Google can choose a different canonical. To keep a page out of the index, use noindex (a meta robotsThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen. tag or X-Robots-Tag headerThe X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos.) and leave the page crawlable; blocking it in robots.txt stops crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexing, so a blocked page can still be indexed via links.

Related: Canonicalization, Duplicate Content, Crawling

Changelog