Indexing at Scale

TL;DR — Indexing at scale is about getting a huge number of pages into Google — think thousands to millions, usually made from a template (product pages, location pages, listings). On a small site, getting indexed mostly just happens. At scale it doesn’t, and the reason is almost always one of two things: Google never got around to crawling the page, or it crawled it and decided the page wasn’t worth keeping.

What “indexing at scale” means

When you have a few hundred pages, indexing is barely a topic. You publish, Google finds the pages through your links and sitemap, crawls them, and indexes them. Done.

When you have a few hundred thousand pages — all generated from the same template with different data plugged in — that easy path breaks. You’ll publish a big new section, watch Google Search Console, and see the indexed count creep up to a fraction of what you published and then just… stop. That gap is the whole problem this article is about.

The one distinction that explains everything

There are two different words that people mush together, and keeping them apart solves half your confusion:

• Crawled means Googlebot came and downloaded the page.
• Indexed means Google processed it into the index, making it eligible to show in search. Evidence for this claim Google Search uses crawling, indexing, and serving stages, and not every page makes it through each stage. Scope: Google's public Search processing model. Confidence: high · Verified: 2026-07-14 Google: How Search works

A page can be crawled and never indexed. On a small site that’s a curiosity. At scale it’s the main event — and Google’s reports even tell you which of the two is happening:

• “Discovered — currently not indexed” means Google knows the URL exists (from your sitemap or a link) but hasn’t crawled it yet. Google says this often happens when crawling was expected to overload the site, but the status alone does not prove one root cause. Evidence for this claim Google defines Discovered - currently not indexed as found but not yet crawled, often because crawling was expected to overload the site. Scope: Page Indexing report status; the label alone does not prove every contributing cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report
• “Crawled — currently not indexed” means Google did fetch the page and chose not to keep it. That’s usually a quality problem — the page didn’t earn its spot.

Those two need completely different fixes. The first is “help Google crawl more of my site.” The second is “give Google a reason to keep what it finds.” Reaching for the wrong fix is the most common mistake at scale.

The three things that actually move the needle

1. Crawl budget. On a big site, Google won’t crawl everything constantly — it rations. Waste (duplicate URLs, endless filter combinations, session IDs) burns that ration on junk instead of your real pages. Cleaning up waste is how you free crawling for the pages that matter. (This is a big topic on its own — see crawl budget.)
2. Sitemaps you can actually read. Instead of one giant sitemap, split it by section (products, categories, locations) so you can check each one separately in Search Console. When indexing stalls, this tells you which section is failing instead of one useless overall number.
3. Quality at the template level. If a whole batch of your templated pages is thin or near-identical, Google can decide the template isn’t worth indexing — and it can even lower how much of your whole site it bothers with. This is the part people underestimate the most.

The thing most people get wrong

Getting a page crawled more often does not make it rank, and re-submitting a page over and over does not force Google to keep it. If Google crawled your programmatic pages and left them out, the fix is almost never technical — it’s the content. Google has said, repeatedly and plainly, that quality is the biggest driver of what gets indexed. Pinging, re-submitting, and hammering the URL Inspection “Request indexing” button won’t fix a thin-template problem.

Want the operational version — how crawl budget actually works at scale, how to segment sitemaps as a diagnostic, reading log files, and how one bad template can sink a whole domain? Switch to the Advanced tab.

TL;DR — Indexing at scale is getting thousands-to-millions of URLs reliably through discovery → crawl → index, instead of troubleshooting one page at a time. Crawl budget is the binding constraint but it’s necessary-not-sufficient — being crawled doesn’t guarantee being kept. Split “Discovered — currently not indexed” (capacity/priority) from “Crawled — currently not indexed” (quality/duplication); they have different fixes. Segment sitemaps by section/template/status and submit via a sitemap index so you can filter the Page Indexing report per segment and see which pattern is failing. Log files are the ground truth GSC’s sampled numbers can’t give you. The failure mode that defines scale: one thin/near-duplicate template can depress crawl demand and indexing willingness for the whole domain — Google’s stated primary lever is quality. Don’t chase a benchmark indexing rate; chase the trend of a segment after you fix its cause.

Why “at scale” is a different problem

Ordinary indexing troubleshooting is a page-level investigation: one URL isn’t indexed, you figure out why, you fix it. That approach doesn’t survive contact with a programmatic site. When you’ve generated 500,000 templated URLs from a database, a “not indexed” result isn’t 500,000 separate problems — it’s usually one problem (the template) repeated 500,000 times. The unit of work shifts from the URL to the URL pattern.

That shift changes everything downstream. You stop asking “why isn’t this page indexed” and start asking “which of my templates is failing, and is it a crawl problem or a quality problem.” The rest of this article is a way to answer that at scale.

The literal phrase “indexing at scale” isn’t owned by any single SEO explainer — most of the top results for it are database/search-engineering content. But the practitioner need is real and validated: there are monitoring products (like Indexing Insight) built specifically for the 100k-to-1M+ page segment, tracking why URLs fail to index. This piece is the operational synthesis those tools assume you already understand.

Crawled vs. indexed — the distinction everything depends on

Google is explicit that search runs in stages and “not all pages make it through each stage.” Evidence for this claim Google Search uses crawling, indexing, and serving stages, and not every page makes it through each stage. Scope: Google's public Search processing model. Confidence: high · Verified: 2026-07-14 Google: How Search works Simplified, the pipeline is:

Discovered → Crawled → Indexed.

• Discovered — Google knows the URL exists (from a sitemap or a link) but hasn’t fetched it yet.
• Crawled — Googlebot fetched the URL and got a response.
• Indexed — Google decided to add (some version of) that content to its index, making it eligible to appear in results.

The two failure states that matter most at scale map directly onto this pipeline, and they are the Page Indexing report’s two most-argued-about buckets:

• “Discovered — currently not indexed” — the URL never made it past discovery to a crawl. Evidence for this claim Google defines Discovered - currently not indexed as found but not yet crawled, often because crawling was expected to overload the site. Scope: Page Indexing report status; the label alone does not prove every contributing cause. Confidence: high · Verified: 2026-07-14 Google: Page indexing report This can reflect a crawl-capacity / priority issue: server strain, low perceived value, thin internal linking into the section, or crawl waste elsewhere starving the section. The fix lives on the crawl side — internal links, sitemap inclusion, server performance, and removing waste.
• “Crawled — currently not indexed” — the URL was fetched and not kept. This is predominantly a quality / duplication verdict. Gary Illyes has laid out the causes bluntly (see the Quotes tab): duplicate elimination against a version already in the index with better signals, general site quality, and site errors that serve the same page to many URLs. The fix lives on the content side — improve or consolidate; a stronger tag or another re-submit won’t move it.

Conflating these two is the single most expensive mistake at scale. Do a content-quality pass when the real problem is thin internal linking and you burn a sprint; ping IndexNow repeatedly when the real problem is a duplicate template and you accomplish nothing. The Page Indexing report exists to tell you which bucket you’re in — reading it correctly is the whole skill, and it’s covered in depth in the Page Indexing (Index Coverage) report deep dive.

Crawl budget is the binding constraint — but it’s not sufficient

You can’t index a page you never crawl, so crawl budget is the floor. Google defines it as “the set of URLs that Google can and wants to crawl” — crawl capacity (what your server can take) times crawl demand (how much Google wants to, driven by popularity and staleness). I’ve written the full mechanics up in Ahrefs’ crawl-budget guide, and the on-site crawl budget article covers capacity vs. demand in beginner-friendly terms — I’ll assume that background and go straight to the at-scale implications.

Two things to keep straight from the start:

It’s not a ranking lever. As I put it in that Ahrefs guide: “The rate of crawling isn’t going to impact your rankings.” Crawl-budget work gets pages indexed, it doesn’t move positions. Don’t sell it internally as a ranking hack — you’ll be wrong and you’ll lose credibility.

Most sites genuinely don’t need to think about it. Google scopes its own large-site crawl-budget guide to “Large sites (1 million+ unique pages) with content that changes moderately often (once a week)” and “Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily).” And it says outright: “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.” Treat those numbers as Google’s rough “who should care” markers, not hard thresholds — don’t publish “1 million pages” as a cutoff.

What wastes crawl budget on programmatic sites

The at-scale waste is almost always self-inflicted URL multiplication: faceted navigation generating every filter combination, tracking and sort parameters, session IDs, HTTP/HTTPS and www/non-www variants, and infinite spaces (calendars, relative-link explosions — see spider traps). Roughly 60% of the web is duplicate content, much of it these boring technical variants, and every duplicate URL is a fetch Google spent on a copy instead of a page you actually want indexed. Consolidate duplicates (canonicalization), block genuinely low-value spaces in robots.txt, return real 404/410 for gone pages, and keep redirect chains short. On a big site, freeing crawl budget is mostly about removing waste, not asking Google to crawl more.

Sitemap segmentation as a monitoring strategy, not just a size fix

Google’s documentation frames splitting sitemaps as a size-limit thing: “If you have a sitemap that exceeds the size limits, you’ll need to split up your large sitemap into multiple sitemaps such that each new sitemap is below the size limit.” Each sitemap caps at 50,000 URLs, and “the referenced sitemaps must be hosted on the same site as your sitemap index file” and “must be in the same directory as the sitemap index file, or lower in the site hierarchy.”

That’s true, but the far more valuable use of segmentation is diagnosis. Here’s the synthesis Google doesn’t spell out in one place: the Page Indexing report has a sitemap filter — you can view all known pages, all submitted pages, unsubmitted pages only, or a specific submitted sitemap. Combine those two features and one aggregate “X% indexed” number becomes a per-segment breakdown.

Concretely: instead of one sitemap.xml, ship a sitemap index that references sitemap-products.xml, sitemap-categories.xml, sitemap-locations.xml, sitemap-blog.xml, and so on — and submit each. Now you can filter the Page Indexing report to each sitemap independently and see indexed-vs-not-indexed per section. Instead of “68% of the site is indexed” (useless — which 32% is missing?) you get “products are 91% indexed, locations are 12% indexed” — and now you know exactly which template to investigate.

You can slice sitemaps three useful ways, and each reveals something different:

• By section / template (products, categories, locations) — isolates which URL pattern is failing. The most common and most useful cut.
• By date / cohort (sitemap-products-2026-07.xml) — shows how fast a newly published batch gets picked up over time, separate from your established inventory.
• By status (a sitemap-priority.xml of only your most important URLs) — a validation trick straight from Search Console Help: submit a sitemap of only the pages you care most about and filter the report to it to check a fix faster, without waiting on the whole site.

One caveat worth stating: a URL is treated as submitted by a sitemap even if Google also found it another way, so these segments overlap discovery from links — the filter is about attribution to a sitemap, not exclusivity.

Log files: the ground truth GSC can’t give you

The Page Indexing report is sampled and aggregated — its numbers are rounded and it doesn’t show you every request. Your raw server logs do. For a site big enough to need this article, log-file analysis is usually the only way to see what Googlebot and Bingbot actually fetched: which URL patterns are eating crawl budget, which important sections bots rarely reach, and where an infinite parameter space is quietly absorbing thousands of fetches that never appear as a tidy line in GSC. If the Page Indexing report tells you what Google decided, logs tell you what Google did — and at scale you need both. (Full treatment in the log file analysis deep dive.)

The quality gate — why “just get it crawled” isn’t enough

This is the part that most differentiates scale from ordinary indexing work, and it’s the reason crawl-budget engineering alone doesn’t fix stalled indexing.

Google’s stated primary lever isn’t technical — it’s quality. Gary Illyes, on Google’s Search Off the Record podcast: “The most important is quality. It’s always quality. And I think externally, people don’t necessarily want to believe it, but the quality, that’s the biggest driver for most of the indexing and crawling decisions that we make.” That single line undercuts the whole “indexing problems are robots.txt-and-sitemap problems” mindset.

At scale this compounds in a way it can’t on a small site. Google doesn’t evaluate each low-value programmatic page in a vacuum. A large cluster of thin or near-duplicate templated pages can depress Google’s overall quality perception of the site and reduce its willingness to crawl and index broadly — not just for the offending URLs, but for the whole domain. So one bad template is not a local problem; it’s a tax on everything. This is exactly the pattern the thin content and scaled content abuse articles cover: the trigger for trouble isn’t volume — large, genuinely useful programmatic sites are fine — it’s low value at volume. Programmatic SEO isn’t penalized for being programmatic; it gets suppressed when the marginal page has nothing to offer.

The spring-2026 wave of mass-deindexing reports fits this frame. Google’s John Mueller, responding on Bluesky, characterized the movement of pages into not-indexed status as unremarkable rather than a special event (Search Engine Journal coverage). Whether or not that reassured anyone, it’s consistent with the throughline: Google frames indexing tightening through a quality lens, and “not indexed” at scale is more often a verdict than a bug.

A practical workflow for indexing at scale

Put the pieces in operational order. The whole point is to work at the template level, not the URL level:

1. Segment sitemaps by section/template (and add date-cohort and priority-subset sitemaps), submitted via a sitemap index.
2. Monitor each segment in the Page Indexing report — track indexed share per sitemap, not one site-wide number.
3. Triage by cause. For a failing segment, split the not-indexed URLs: a wall of “Discovered — currently not indexed” points at crawl capacity/priority; a wall of “Crawled — currently not indexed” points at quality/duplication.
4. Fix the matching side. Capacity problem → internal links into the section, sitemap inclusion, server performance, and remove crawl waste elsewhere. Quality problem → improve or consolidate the template, or deliberately keep the truly low-value variants out (noindex/canonical/robots) so they stop dragging the set down.
5. Confirm with logs. Verify bots are actually reaching the section and not spending the budget on parameter junk.
6. Watch the trend, not a number. After a fix, the question is whether that segment’s indexed share is climbing — not whether it hit some universal target (there isn’t one).

Don’t invent an indexing-rate benchmark

There is no “you should be X% indexed” number, and repeating third-party figures as if they were Google-sanctioned targets is a trap. Every site’s ratio depends on the uniqueness, value, and demand of its pages. The actionable signal is the trend of a specific segment after you change something — full stop. If someone hands you a target percentage, ask where it came from; it’s almost certainly someone else’s illustrative number, not a rate Google promises.

IndexNow and push-based indexing — a supplement, not a strategy

Push notification has its place. IndexNow (backed by Bing/Microsoft, and adopted by Yandex, Naver, and others — Google does not use it for general pages) lets you tell engines about changes instead of waiting for a re-crawl. Bing’s framing: “Whether you’re adding, updating, or deleting content, IndexNow notifies multiple search engines of your content changes as soon as they happen.”

But be clear about what it does and doesn’t solve. IndexNow accelerates the freshness half of the problem — getting new/changed/deleted URLs seen faster. It does nothing for the crawl-budget and quality-gating half, which is Google’s larger share of the problem for most readers here. And it carries a daily per-domain URL quota (commonly cited around 10,000), which is far too small to bulk-notify a multi-million-URL catalog — so it can’t replace sitemap-based discovery at true scale. Batch and queue submissions rather than firing everything at once on a bulk republish. Treat IndexNow as an “also do this,” not a plan. (See the on-site IndexNow and Google Indexing API coverage for what each is actually for.)

Common myths about indexing at scale

• “Getting more pages crawled will improve rankings.” No. Crawling is a prerequisite for indexing, indexing for ranking — but crawling more ≠ ranking higher. “The rate of crawling isn’t going to impact your rankings.”
• “If a page isn’t indexed, it’s a technical bug.” At scale, usually not. “Crawled — currently not indexed” is most often a quality/duplication verdict, and re-submitting or pinging it won’t help.
• “Crawl budget matters for every site.” No — it’s a large / rapidly-changing site concern, per Google’s own scoping. Most sites never need to manage it.
• “More indexed pages = better SEO.” No — this is the index bloat myth in reverse. The goal is more valuable pages indexed, which sometimes means deliberately keeping low-value variants out.
• “Submitting a sitemap forces indexing.” No. A sitemap is a discovery and priority hint; Google still evaluates each URL independently.
• “Programmatic SEO is inherently penalized.” No. Volume isn’t the trigger; low value at volume with manipulative intent is (see scaled content abuse).
• “A high ‘Discovered — not indexed’ count always means a quality problem.” No — that bucket skews toward crawl priority/capacity; the quality signal is “Crawled — not indexed.” Fixing the wrong one wastes the sprint.

Where this sits

Indexing at scale is the operational counterpart to a few neighbors: crawl budget is the mechanism underneath it, the Page Indexing report and log file analysis are how you see it, and thin content / scaled content abuse are what happens if you get the quality gate wrong. It’s also the mirror image of index bloat — that’s “too many low-value pages got in,” this is “not enough valuable pages are getting in,” and the fixes overlap because both are really about concentrating index inclusion on pages that earn it. For the broader discipline of building templated pages that deserve to rank, see the Programmatic SEO pillar; for the org side of doing this on a giant site, see Enterprise SEO.

AI summary

A condensed take on the Advanced version:

• Indexing at scale = getting thousands-to-millions of URLs reliably through discovery → crawl → index, at the template level rather than one page at a time. On a programmatic site, “not indexed” is usually one problem (a template) repeated, not thousands of separate problems.
• Crawled ≠ indexed. “Discovered — currently not indexed” is a crawl capacity/priority problem (fix: internal links, sitemap inclusion, server speed, remove crawl waste). “Crawled — currently not indexed” is a quality/duplication verdict (fix: improve or consolidate content). Reaching for the wrong fix is the costliest mistake.
• Crawl budget is the binding constraint but necessary-not-sufficient. Google: “the set of URLs that Google can and wants to crawl.” It’s not a ranking lever — “the rate of crawling isn’t going to impact your rankings.” Most sites don’t need to manage it; Google scopes it to ~1M+ pages (weekly changes) or ~10k+ (daily). Free budget by removing waste (duplicates, parameters, faceted nav, spider traps).
• Segment sitemaps as a monitoring tool, not just a size fix. Split by section/template/date/status, submit via a sitemap index, then filter the Page Indexing report per sitemap to see which pattern is failing — turning one useless aggregate into a per-segment breakdown.
• Log files are ground truth the sampled Page Indexing report can’t give you — what bots actually crawled and where budget leaks.
• Quality is Google’s stated primary lever. Illyes: “The most important is quality. It’s always quality… the biggest driver for most of the indexing and crawling decisions.” At scale, one thin/near-duplicate template can depress crawl demand and indexing willingness for the whole domain — the thin content / scaled content abuse failure mode.
• Don’t chase a benchmark indexing rate; chase the trend of a segment after you fix its cause.
• IndexNow is a freshness supplement, not a scale strategy — daily per-domain quota (~10k) can’t bulk-notify huge catalogs, and Google doesn’t use it for general pages.

Official documentation

Primary-source documentation that bears on getting large URL sets indexed.

Google

• Optimize your crawl budget (Google Search Central; formerly titled “Large site owner’s guide to managing crawl budget”) — the one official page written for scale: crawl capacity + demand, who should care, and the URL-hygiene levers that protect your budget.
• Manage your sitemaps with sitemap index files — splitting large sitemaps, the 50,000-URL cap, and the same-site / same-directory hosting rules for a sitemap index.
• Page indexing report (Search Console Help) — the not-indexed reason taxonomy and the sitemap filter that makes per-segment monitoring possible.
• In-Depth Guide to How Google Search Works — the crawl → index → serve pipeline and “not all pages make it through each stage.”
• Spam policies for Google Web Search — the scaled-content-abuse policy that defines the quality-at-volume line for programmatic pages.

Bing / Microsoft

• IndexNow / indexnow.org — the push protocol for signaling new/changed/deleted URLs; a freshness supplement to sitemaps, not a bulk-discovery replacement.
• Keeping Content Discoverable with Sitemaps in AI-Powered Search (Bing Webmaster Blog, July 2025) — Bing’s position on sitemaps + IndexNow at scale.

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page where the anchor is supported.

Google — quality is the primary driver of indexing

• “The most important is quality. It’s always quality. And I think externally, people don’t necessarily want to believe it, but the quality, that’s the biggest driver for most of the indexing and crawling decisions that we make.” — Gary Illyes, Google, Search Off the Record (via Search Engine Journal). Read the coverage

Google — crawl budget, and who needs it

• “Google defines a site’s crawl budget as the set of URLs that Google can and wants to crawl.” Jump to quote
• “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.” Jump to quote

Google — splitting sitemaps at scale

• “If you have a sitemap that exceeds the size limits, you’ll need to split up your large sitemap into multiple sitemaps such that each new sitemap is below the size limit.” Jump to quote
• “The referenced sitemaps must be hosted on the same site as your sitemap index file.” Jump to quote
• “Sitemaps that are referenced in the sitemap index file must be in the same directory as the sitemap index file, or lower in the site hierarchy.” Jump to quote

Bing / Microsoft — IndexNow

• “Whether you’re adding, updating, or deleting content, IndexNow notifies multiple search engines of your content changes as soon as they happen.” — Microsoft Bing. Read on bing.com/indexnow

Patrick Stox — crawling is not a ranking lever

• “The rate of crawling isn’t going to impact your rankings.” — me, in my Ahrefs crawl-budget guide. Read the guide

Which path should I take?

A section isn’t getting indexed. Work the branches — the goal is to land on capacity vs. quality before you spend any effort, because the fixes don’t overlap.

Start: is the section even in a submitted sitemap you can filter?

• No → Segment it first. Ship a sitemap-<section>.xml (via your sitemap index) and submit it. You can’t diagnose per-segment until you can filter the Page Indexing report to it. Come back in a couple of weeks with data.
• Yes → Filter the Page Indexing report to that sitemap and look at the dominant not-indexed reason. ↓

What’s the dominant reason for the failing segment?

• Mostly “Discovered — currently not indexed” → Capacity / priority problem. Google knows the URLs but isn’t crawling them.

  • Are you wasting crawl budget elsewhere (parameters, facets, spider traps, duplicates)? → Remove the waste first — that’s usually the highest-leverage fix. Confirm in log files.
  • Is the section thinly linked internally? → Add internal links into it from strong pages; make sure it’s not buried deep.
  • Is the server slow / throwing 5xx under crawl? → Fix server performance; a strained server makes Google back off.

• Mostly “Crawled — currently not indexed” → Quality / duplication verdict. Google fetched the pages and chose not to keep them.

  • Are the pages near-duplicates of each other or of a better existing version? → Consolidate (canonicalization) or differentiate the template.
  • Is the template thin — little unique value per page? → Raise the value per page, or don’t publish the variants that can’t clear the bar (noindex/keep them out) so they stop dragging the set down.
  • Is a bug serving the same page to many URLs? → Fix the bug — this alone can inflate the bucket.

• Mostly “Duplicate” / “Alternate page with proper canonical” → Working as intended in most cases. These are canonicalized dupes, not a failure. Don’t chase them to 100%.

After any fix: watch that segment’s indexed share trend over the following weeks — not a target percentage, and not the site-wide number.

Separate question — “I just want it indexed faster.”

• New/changed URLs and you’re on Bing? → IndexNow helps with freshness.
• Multi-million-URL bulk push? → IndexNow’s daily quota won’t cover it; lean on segmented sitemaps and strong internal linking. There is no shortcut that bypasses the quality gate.

The mental models

1. Discovered → Crawled → Indexed is a funnel, and each stage is a filter. Before changing anything, locate which stage a section is stuck at. “Discovered — not indexed” = stuck before the crawl (capacity). “Crawled — not indexed” = stuck after the crawl (quality). Same words, opposite fixes.

2. Crawl budget is necessary, not sufficient. You can’t index what you don’t crawl, so budget is the floor — but clearing the floor doesn’t get you indexed. Split your effort: half is “get Google to fetch the section” (capacity), half is “give Google a reason to keep it” (quality). Most teams over-invest in the first half.

3. The unit of work is the template, not the URL. At scale, one problem repeats across a whole pattern. Don’t debug 500,000 pages; find the one template that’s failing and fix it once. Segmented sitemaps exist to turn an aggregate number into a per-template diagnosis.

4. Segment to see. Aggregate hides. “68% indexed” is useless. “Products 91%, locations 12%” is a work order. Split sitemaps by section, then date, then priority-subset, and read the Page Indexing report per segment. You can’t fix a pattern you can’t isolate.

5. Quality is a site-level signal, not a page-level one. Google doesn’t judge a thin template in a vacuum — a big enough cluster of low-value pages can lower crawl demand and indexing willingness for the whole domain. That’s why “just noindex the junk” is sometimes the pro-indexing move: keeping bad pages out protects the good ones.

6. Trend, not target. There is no universal indexing rate to hit. The only honest KPI is whether a segment’s indexed share is climbing after you addressed its specific cause.

Indexing-at-scale checklist

A pass for a large or programmatic site where indexed counts are lagging published counts:

• Confirm you’re actually at scale. Near Google’s rough markers (~1M+ pages, or ~10k+ rapidly changing)? If not, crawl budget probably isn’t your problem — look at ordinary page-level indexing issues first.
• Segment sitemaps by section/template, submitted via a sitemap index (each sitemap ≤ 50,000 URLs; referenced sitemaps on the same site and in the same directory or lower).
• Add a date-cohort sitemap for newly published batches and a priority-subset sitemap of your most important URLs (for fast fix validation).
• Read the Page Indexing report per sitemap — track indexed share by segment, not one site-wide number.
• For each failing segment, split the not-indexed reasons: “Discovered — not indexed” (capacity) vs. “Crawled — not indexed” (quality).
• Capacity fixes: internal links into the section, server performance, and remove crawl waste (parameters, faceted nav, spider traps, duplicate variants).
• Quality fixes: improve/consolidate the template; deliberately keep truly low-value variants out (noindex/canonical/robots) so they stop dragging the set down.
• Pull server logs to confirm bots reach the section and aren’t burning budget on junk URLs.
• Don’t chase a benchmark % — watch each segment’s indexed-share trend after a fix.
• IndexNow (Bing/others) wired up for new/changed/deleted URLs — as a freshness supplement, batched, not as your bulk-discovery strategy.
• Sanity-check for index bloat too — you want fewer low-value pages indexed and more high-value ones; the fixes overlap.

Indexing-at-scale cheat sheet

The two failure states (memorize these)

Sitemap segmentation cuts, and what each reveals

Scale limits & facts

• Sitemap cap: 50,000 URLs each; sitemap index references up to 50,000 sitemaps; referenced sitemaps on the same site, same directory or lower.
• Google’s “large site” markers: ~1M+ pages (weekly changes) or ~10k+ (daily) — rough guidance, not a hard cutoff.
• Crawl budget = capacity × demand; it is not a ranking lever.
• IndexNow: Bing/Yandex/others, not Google; ~10k URLs/day per domain (commonly cited) — a freshness supplement, not bulk discovery.
• Indexing benchmark: there isn’t one — track the trend, not a target %.

Do / don’t

• Do: segment → monitor → triage by cause → fix at the template level → watch trend.
• Don’t: re-submit/ping a thin page repeatedly, treat “Crawled — not indexed” as a technical bug, or read one aggregate indexed % as if it told you anything.

Indexing mistakes that multiply at scale

Treating every excluded URL as the same problem. “Discovered — currently not indexed” points mainly to crawl capacity or priority; “Crawled — currently not indexed” points mainly to quality or duplication. Applying one fix to both wastes time. Split the statuses first, then work on the crawl side or content side accordingly.

Repeatedly submitting a thin page. Another request cannot make a low-value template earn index inclusion. Improve or consolidate the template, or keep the weak variants out of the index instead of resubmitting them.

Reading one site-wide indexed percentage. An aggregate can hide a healthy product section and a failing location template. Segment sitemaps by section or template and compare each segment’s own trend.

Putting every URL in one giant sitemap. A monolithic sitemap may aid discovery, but it removes the diagnostic cut you need in the Page Indexing report. Use a sitemap index with meaningful segments, plus date cohorts or a priority subset when useful.

Optimizing crawl budget before proving it is constrained. Most smaller sites have an ordinary page-quality or technical issue, not a large-site crawl-capacity problem. Check the failure status and logs before treating crawl budget as the cause.

Chasing a universal indexing-rate benchmark. There is no honest percentage that every site should hit. Establish a baseline per template and judge whether the segment moves after its specific cause is addressed.

Using IndexNow as the indexing strategy. It can notify participating engines about new, changed, or deleted URLs, but it does not replace useful pages, internal links, sitemaps, or crawlable architecture. Use it as a freshness supplement.

Prompts for indexing-at-scale diagnosis

Separate capacity problems from quality problems

Paste a GSC export with URL, sitemap segment, and Page Indexing reason. If available, add last Googlebot hit and status code from server logs. Expect a segment-level triage, not a page-by-page rewrite list.

Classify this large-site indexing export using this framework:
- “Discovered — currently not indexed” is primarily a crawl-capacity or priority lead.
- “Crawled — currently not indexed” is primarily a quality or duplication lead.
- Duplicate/alternate canonical states require checking whether consolidation is intended.

Group the rows by sitemap segment and template. For each group, report the dominant
status, evidence from the supplied data, the most likely side of the problem (discovery,
crawl, rendering, quality, duplication, or intended exclusion), and the next concrete
check. Use log evidence when provided. Do not claim a URL is indexed from on-page signals
alone, and do not invent a target indexing percentage.

[PASTE GSC EXPORT AND OPTIONAL LOG DATA]

Turn a failed segment into a template-level action plan

Paste one segment’s status totals, representative HTML or crawl fields, internal-link depth, and log summary. Expect a prioritized diagnosis with stop conditions.

Diagnose why this sitemap segment is under-indexed. Work at the template level rather
than prescribing isolated URL submissions.

Return:
1. What the supplied evidence proves versus what remains unknown.
2. Whether the first investigation should focus on crawl capacity/priority or on
   quality/duplication, with the evidence for that choice.
3. Up to five checks in order: sitemap eligibility, internal-link reachability, server
   response/rendering, bot crawl allocation, and page-level distinct value.
4. The template-level fix suggested by each possible result.
5. A small priority subset or date cohort to monitor after the fix, using its own
   pre-change trend as the baseline.

Do not recommend bulk resubmission as a substitute for fixing the cause. Do not treat
IndexNow as a Google indexing mechanism.

[PASTE SEGMENT DATA]

Tools for isolating indexing failures

Start with the on-site tools

• Google Index Checker — checks observable blockers such as status, redirects, noindex, and canonical signals, then routes you to GSC URL Inspection for Google’s actual index decision. Use it on a representative sample, not as proof of an entire segment’s status.
• XML Sitemap Validator — finds structural errors and warnings in a pasted, uploaded, or fetched sitemap. Use it before interpreting a weak segment so malformed XML or ineligible URL entries do not contaminate the diagnosis.
• XML Sitemap Generator — creates a capped, robots-respecting sitemap from a same-site crawl while separating noindex, off-canonical, failed, and uncertain URLs. It is useful for building a clean priority subset, not for replacing the site’s intentional segmented sitemap system.

Use search-engine and server evidence for the actual diagnosis

• Google Search Console Page Indexing report — filter by submitted sitemap to compare template or section segments and split their not-indexed reasons.
• Google Search Console URL Inspection — inspect representative URLs when you need Google’s reported crawl, canonical, and index state rather than an external proxy.
• Crawl Stats and server access logs — show whether Googlebot reaches the segment, how often, and whether crawl is being spent on parameters, faceted paths, or other waste.
• A full-site crawler — verifies status codes, canonicals, robots directives, render-visible content, link depth, and orphan candidates before you blame crawl budget.
• Bing Webmaster Tools and IndexNow — monitor Bing’s view and send freshness notifications for changed URL sets; they supplement, rather than replace, discovery and quality work.

Resources worth your time

My related writing

• When Should You Worry About Crawl Budget? — my Ahrefs guide to the mechanism underneath this whole topic: crawl demand × crawl rate, what wastes it, who actually needs to care, and why “the rate of crawling isn’t going to impact your rankings.”
• The Beginner’s Guide to Technical SEO — where crawling and indexing fit in the bigger picture.
• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — who’s actually fetching your pages now, from my Cloudflare Radar analysis.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawling, rendering, indexing, and ranking, including the point that indexing draws on multiple inputs (sitemaps, feeds, pings, the Indexing API), not just crawling. (My standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

Official

• Google — Optimize your crawl budget (formerly “Large site owner’s guide to managing crawl budget”) and Manage your sitemaps with sitemap index files.
• Google — Page indexing report (Search Console Help) — the not-indexed reason taxonomy and the sitemap filter.
• Bing — IndexNow — the push protocol for new/changed/deleted URLs.

From around the industry

• How to improve enterprise-level crawling and indexing efficiency (Search Engine Land, Olya Ianovskaia, 2023) — enterprise case studies on deduplication, content cleanup, and blocking infinite parameter spaces.
• Understanding and resolving ‘Discovered – currently not indexed’ (Search Engine Land, Dan Taylor, 2025) — a focused treatment of the capacity-side failure state.
• Programmatic SEO: Scale content, rankings & traffic fast (Search Engine Land) — the broader programmatic-SEO context, including why low-value variants should be kept out of the index.
• Indexation Triage for Enterprise Retail eCommerce (Go Fish Digital) — framing index inclusion as revenue governance: concentrating crawl and index on high-value templates.
• What Is Index Bloat? (Ahrefs) — the inverse problem (too many low-value pages getting in), whose fixes overlap with getting the right pages in.
• Google Confirms: Quality Is Foremost Factor In Search Indexing (Search Engine Journal) — the Illyes “it’s always quality” statement in context.
• Deindexing Reports Keep Coming, Google Sees Nothing Unusual (Search Engine Journal, June 2026) — the spring-2026 mass-deindexing reports and Google’s quality-lens response.

Stats worth citing

• Google’s own “large site” markers. Google scopes crawl-budget concern to roughly 1M+ pages changing weekly, or 10k+ pages changing daily — framed as rough guidance for who should care, not a hard threshold. Source
• Sitemap scale limits. Each sitemap caps at 50,000 URLs; a sitemap index can reference up to 50,000 sitemaps, all on the same site. Source
• ~60% of the web is duplicate content — much of it boring technical variants (HTTP/HTTPS, www, trailing slashes, parameters) that quietly drain crawl budget on large/faceted sites. From my Ahrefs crawl-budget guide. Source
• Quality is the biggest driver of indexing/crawling decisions, per Gary Illyes — not a technical checklist item. The single most load-bearing framing for indexing at scale. Coverage

Test yourself: Indexing at Scale

Five questions on getting large and programmatic URL sets indexed. Pick an answer for each, then check.

Indexing at Scale

Indexing at scale is the set of practices — sitemap segmentation, crawl-budget management, log-file and Page Indexing monitoring, and quality gating — used to get large or programmatic URL sets (thousands to millions of pages) reliably crawled and indexed, instead of managing indexing page by page.

Related: Crawl Budget, Indexing, Index Bloat

Indexing at Scale

Indexing at scale is the practice of getting a large or programmatic set of URLs — typically thousands to millions of pages, often generated from templates or databases (ecommerce catalogs, listings, location pages, user-generated content) — crawled and added to a search engine’s index reliably, rather than relying on the ad hoc discovery that works fine for a site with a few hundred pages.

It sits at the intersection of three things: crawl budget (can and will the bot fetch the URL), index selection (will the engine bother keeping it), and content quality at volume (is the marginal page worth adding). The discipline is less about any single fix and more about building systems — segmented sitemaps, monitoring by section, and quality gates — that make indexing problems visible and fixable at the URL-template level instead of one page at a time.

The central distinction is that crawled is not the same as indexed: a URL can be fetched and still not kept. At scale, a “not indexed” result stops being a one-off investigation and becomes a signal about a whole URL template — which is why the fixes are different from ordinary small-site indexing troubleshooting.

Related: Crawl Budget, Indexing, Index Bloat

Changelog