Indexing at Scale

Getting large or programmatic URL sets — thousands to millions of pages — reliably crawled and indexed. Crawl budget as the binding constraint, sitemap segmentation as a monitoring tool, log files as ground truth, and the quality gate that decides whether templated pages get kept.

First published: Jul 3, 2026 · Last updated: Jul 19, 2026 · Advanced
demand #7 in Programmatic SEO#381 on the site
AIAuthored byAI v2
1 evidence signal on this page

Indexing at scale is getting thousands-to-millions of URLs reliably crawled and indexed, instead of fixing indexing one page at a time. The pipeline is discovery → crawl → index, and the crux is that crawled is not the same as indexed. 'Discovered - currently not indexed' is mostly a crawl-capacity/priority problem; 'Crawled - currently not indexed' is mostly a quality/duplication verdict — and they need different fixes. Crawl budget is the binding constraint but it's necessary-not-sufficient: a page still has to earn its place. Segment your sitemaps by section/template/status and submit them via a sitemap index so you can filter the Page Indexing report per segment and see which URL pattern is failing. Use log files as ground truth for the crawl side. The failure mode that separates scale from small sites: one thin template can drag down crawl demand and indexing willingness for the whole domain — Google's stated primary lever is quality. Don't chase a benchmark indexing rate; chase the trend after you fix something.

TL;DR — IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. at scale is getting thousands-to-millions of URLs reliably through discovery → crawl → index, instead of troubleshooting one page at a time. Crawl budgetThe number of URLs an engine will crawl in a timeframe. is the binding constraint but it’s necessary-not-sufficient — being crawled doesn’t guarantee being kept. Split “Discovered — currently not indexed” (capacity/priority) from “Crawled — currently not indexed” (quality/duplication); they have different fixes. Segment sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. by section/template/status and submit via a sitemap indexA sitemap index is a sitemap of sitemaps — a single file that lists your other sitemap files instead of listing URLs directly. It's how large sites stay under the 50,000-URL / 50MB-per-sitemap limit while submitting just one file. so you can filter the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. per segment and see which pattern is failing. Log filesLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. are the ground truth GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s sampled numbers can’t give you. The failure mode that defines scale: one thin/near-duplicateThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling. template can depress crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. and indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. willingness for the whole domain — Google’s stated primary lever is quality. Don’t chase a benchmark indexing rate; chase the trend of a segment after you fix its cause.

Why “at scale” is a different problem

Ordinary indexing troubleshooting is a page-level investigation: one URL isn’t indexed, you figure out why, you fix it. That approach doesn’t survive contact with a programmatic site. When you’ve generated 500,000 templated URLs from a database, a “not indexed” result isn’t 500,000 separate problems — it’s usually one problem (the template) repeated 500,000 times. The unit of work shifts from the URL to the URL pattern.

That shift changes everything downstream. You stop asking “why isn’t this page indexed” and start asking “which of my templates is failing, and is it a crawl problem or a quality problem.” The rest of this article is a way to answer that at scale.

The literal phrase “indexing at scaleIndexing at scale is the set of practices — sitemap segmentation, crawl-budget management, log-file and Page Indexing monitoring, and quality gating — used to get large or programmatic URL sets (thousands to millions of pages) reliably crawled and indexed, instead of managing indexing page by page.” isn’t owned by any single SEO explainer — most of the top results for it are database/search-engineering content. But the practitioner need is real and validated: there are monitoring products (like Indexing Insight) built specifically for the 100k-to-1M+ page segment, tracking why URLs fail to index. This piece is the operational synthesis those tools assume you already understand.

Crawled vs. indexed — the distinction everything depends on

Google is explicit that search runs in stages and “not all pages make it through each stage.” Evidence for this claim Google Search uses crawling, indexing, and serving stages, and not every page makes it through each stage. Scope: Google's public Search processing model. Confidence: high · Verified: Google: How Search works Simplified, the pipeline is:

Discovered → Crawled → Indexed.

  • Discovered — Google knows the URL exists (from a sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. or a link) but hasn’t fetched it yet.
  • Crawled — GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. fetched the URL and got a response.
  • Indexed — Google decided to add (some version of) that content to its index, making it eligible to appear in results.

The two failure states that matter most at scale map directly onto this pipeline, and they are the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.’s two most-argued-about buckets:

  • “Discovered — currently not indexed” — the URL never made it past discovery to a crawl. Evidence for this claim Google defines Discovered - currently not indexed as found but not yet crawled, often because crawling was expected to overload the site. Scope: Page Indexing report status; the label alone does not prove every contributing cause. Confidence: high · Verified: Google: Page indexing report This can reflect a crawl-capacity / priority issue: server strain, low perceived value, thin internal linkingLinks between pages on the same site. into the section, or crawl waste elsewhere starving the section. The fix lives on the crawl side — internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., sitemap inclusion, server performance, and removing waste.
  • “Crawled — currently not indexed” — the URL was fetched and not kept. This is predominantly a quality / duplication verdict. Gary Illyes has laid out the causes bluntly (see the Quotes tab): duplicate elimination against a version already in the index with better signals, general site quality, and site errors that serve the same page to many URLs. The fix lives on the content side — improve or consolidate; a stronger tag or another re-submit won’t move it.

Conflating these two is the single most expensive mistake at scale. Do a content-quality pass when the real problem is thin internal linking and you burn a sprint; ping IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. repeatedly when the real problem is a duplicate template and you accomplish nothing. The Page Indexing report exists to tell you which bucket you’re in — reading it correctly is the whole skill, and it’s covered in depth in the Page Indexing (Index Coverage) report deep dive.

Crawl budget is the binding constraint — but it’s not sufficient

You can’t index a page you never crawl, so crawl budgetThe number of URLs an engine will crawl in a timeframe. is the floor. Google defines it as “the set of URLs that Google can and wants to crawl” — crawl capacity (what your server can take) times crawl demand (how much Google wants to, driven by popularity and staleness). I’ve written the full mechanics up in Ahrefs’ crawl-budget guide, and the on-site crawl budget article covers capacity vs. demand in beginner-friendly terms — I’ll assume that background and go straight to the at-scale implications.

Two things to keep straight from the start:

It’s not a ranking lever. As I put it in that Ahrefs guide: “The rate of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. isn’t going to impact your rankings.” Crawl-budget work gets pages indexed, it doesn’t move positions. Don’t sell it internally as a ranking hack — you’ll be wrong and you’ll lose credibility.

Most sites genuinely don’t need to think about it. Google scopes its own large-site crawl-budget guide to “Large sites (1 million+ unique pages) with content that changes moderately often (once a week)” and “Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily).” And it says outright: “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.” Treat those numbers as Google’s rough “who should care” markers, not hard thresholds — don’t publish “1 million pages” as a cutoff.

What wastes crawl budget on programmatic sites

The at-scale waste is almost always self-inflicted URL multiplication: faceted navigation generating every filter combination, tracking and sort parameters, session IDs, HTTP/HTTPSHTTPS is the encrypted version of HTTP — it uses TLS to authenticate the server and protect data in transit between a browser and a website. Google announced it as a lightweight ranking signal in 2014 and today conditionally prefers HTTPS pages as canonical; Chrome marks plain HTTP pages 'Not Secure.' and www/non-www variants, and infinite spaces (calendars, relative-link explosions — see spider trapsA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content.). Roughly 60% of the web is duplicate content, much of it these boring technical variants, and every duplicate URL is a fetch Google spent on a copy instead of a page you actually want indexed. Consolidate duplicates (canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it.), block genuinely low-value spaces in robots.txt, return real 404/410 for gone pages, and keep redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency. short. On a big site, freeing crawl budget is mostly about removing waste, not asking Google to crawl more.

Sitemap segmentation as a monitoring strategy, not just a size fix

Google’s documentation frames splitting sitemaps as a size-limit thing: “If you have a sitemap that exceeds the size limits, you’ll need to split up your large sitemap into multiple sitemaps such that each new sitemap is below the size limit.” Each sitemap caps at 50,000 URLs, and “the referenced sitemaps must be hosted on the same site as your sitemap index file” and “must be in the same directory as the sitemap index file, or lower in the site hierarchy.”

That’s true, but the far more valuable use of segmentation is diagnosis. Here’s the synthesis Google doesn’t spell out in one place: the Page Indexing report has a sitemap filter — you can view all known pages, all submitted pages, unsubmitted pages only, or a specific submitted sitemap. Combine those two features and one aggregate “X% indexed” number becomes a per-segment breakdown.

Concretely: instead of one sitemap.xml, ship a sitemap index that references sitemap-products.xml, sitemap-categories.xml, sitemap-locations.xml, sitemap-blog.xml, and so on — and submit each. Now you can filter the Page Indexing report to each sitemap independently and see indexed-vs-not-indexed per section. Instead of “68% of the site is indexed” (useless — which 32% is missing?) you get “products are 91% indexed, locations are 12% indexed” — and now you know exactly which template to investigate.

You can slice sitemaps three useful ways, and each reveals something different:

  • By section / template (products, categories, locations) — isolates which URL pattern is failing. The most common and most useful cut.
  • By date / cohort (sitemap-products-2026-07.xml) — shows how fast a newly published batch gets picked up over time, separate from your established inventory.
  • By status (a sitemap-priority.xml of only your most important URLs) — a validation trick straight from Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. Help: submit a sitemap of only the pages you care most about and filter the report to it to check a fix faster, without waiting on the whole site.

One caveat worth stating: a URL is treated as submitted by a sitemap even if Google also found it another way, so these segments overlap discovery from links — the filter is about attribution to a sitemap, not exclusivity.

Log files: the ground truth GSC can’t give you

The Page Indexing report is sampled and aggregated — its numbers are rounded and it doesn’t show you every request. Your raw server logs do. For a site big enough to need this article, log-file analysis is usually the only way to see what GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. and BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. actually fetched: which URL patterns are eating crawl budget, which important sections bots rarely reach, and where an infinite parameter space is quietly absorbing thousands of fetches that never appear as a tidy line in GSC. If the Page Indexing report tells you what Google decided, logs tell you what Google did — and at scale you need both. (Full treatment in the log file analysis deep dive.)

The quality gate — why “just get it crawled” isn’t enough

This is the part that most differentiates scale from ordinary indexing work, and it’s the reason crawl-budget engineering alone doesn’t fix stalled indexing.

Google’s stated primary lever isn’t technical — it’s quality. Gary Illyes, on Google’s Search Off the Record podcast: “The most important is quality. It’s always quality. And I think externally, people don’t necessarily want to believe it, but the quality, that’s the biggest driver for most of the indexing and crawling decisions that we make.” That single line undercuts the whole “indexing problems are robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.-and-sitemap problems” mindset.

At scale this compounds in a way it can’t on a small site. Google doesn’t evaluate each low-value programmatic page in a vacuum. A large cluster of thin or near-duplicate templated pages can depress Google’s overall quality perception of the site and reduce its willingness to crawl and index broadly — not just for the offending URLs, but for the whole domain. So one bad template is not a local problem; it’s a tax on everything. This is exactly the pattern the thin contentThin content is web content that provides little or no value to users. Google's spam policies name it 'thin content with little or no added value' — and it's about value per page, not word count. and scaled content abuseScaled content abuse is Google's spam policy (introduced March 2024) for generating many low-value pages primarily to manipulate search rankings rather than help users — and it applies no matter how the content is created: AI, automation, or human writers. articles cover: the trigger for trouble isn’t volume — large, genuinely useful programmatic sites are fine — it’s low value at volume. Programmatic SEOProgrammatic SEO (pSEO) is the practice of generating many pages from a single template plus a data source to target large sets of similar queries. It's powerful when each page genuinely answers its query with unique data, and spam when it just stamps a thin template across a shallow dataset. isn’t penalized for being programmatic; it gets suppressed when the marginal page has nothing to offer.

The spring-2026 wave of mass-deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. reports fits this frame. Google’s John Mueller, responding on Bluesky, characterized the movement of pages into not-indexed status as unremarkable rather than a special event (Search Engine Journal coverage). Whether or not that reassured anyone, it’s consistent with the throughline: Google frames indexing tightening through a quality lens, and “not indexed” at scale is more often a verdict than a bug.

TIP Reconcile sitemap intent, current crawl signals, and exported GSC state

These three sources update on different schedules. A mismatch creates a review queue; it does not prove Google's current index state without URL Inspection.

Compare the URL sets with my free Indexation Reconciler Free

  1. Load the intended sitemap URLs, a current crawl export, and a same-scope GSC export.
  2. Prioritize URLs where current indexability conflicts with sitemap or exported indexed state.
  3. Verify representative mismatches in URL Inspection, fix the source signal, and rerun on the same cadence.
The conflicting row is a diagnosis candidate: either the blocking directive is new or the export is stale.

URL A is in the sitemap, currently indexable, and indexed in the GSC export, so the action is monitor. URL B is in the sitemap and indexed in the export but currently noindex, so the action is to review the new blocking directive and whether the export is stale.

A practical workflow for indexing at scale

Put the pieces in operational order. The whole point is to work at the template level, not the URL level:

  1. Segment sitemaps by section/template (and add date-cohort and priority-subset sitemaps), submitted via a sitemap index.
  2. Monitor each segment in the Page Indexing report — track indexed share per sitemap, not one site-wide number.
  3. Triage by cause. For a failing segment, split the not-indexed URLs: a wall of “Discovered — currently not indexed” points at crawl capacity/priority; a wall of “Crawled — currently not indexed” points at quality/duplication.
  4. Fix the matching side. Capacity problem → internal links into the section, sitemap inclusion, server performance, and remove crawl waste elsewhere. Quality problem → improve or consolidate the template, or deliberately keep the truly low-value variants out (noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed./canonical/robots) so they stop dragging the set down.
  5. Confirm with logs. Verify bots are actually reaching the section and not spending the budget on parameter junk.
  6. Watch the trend, not a number. After a fix, the question is whether that segment’s indexed share is climbing — not whether it hit some universal target (there isn’t one).

Don’t invent an indexing-rate benchmark

There is no “you should be X% indexed” number, and repeating third-party figures as if they were Google-sanctioned targets is a trap. Every site’s ratio depends on the uniqueness, value, and demand of its pages. The actionable signal is the trend of a specific segment after you change something — full stop. If someone hands you a target percentage, ask where it came from; it’s almost certainly someone else’s illustrative number, not a rate Google promises.

IndexNow and push-based indexing — a supplement, not a strategy

Push notification has its place. IndexNow (backed by Bing/Microsoft, and adopted by Yandex, Naver, and others — Google does not use it for general pages) lets you tell engines about changes instead of waiting for a re-crawl. Bing’s framing: “Whether you’re adding, updating, or deleting content, IndexNow notifies multiple search engines of your content changes as soon as they happen.”

But be clear about what it does and doesn’t solve. IndexNow accelerates the freshness half of the problem — getting new/changed/deleted URLs seen faster. It does nothing for the crawl-budget and quality-gating half, which is Google’s larger share of the problem for most readers here. And it carries a daily per-domain URL quota (commonly cited around 10,000), which is far too small to bulk-notify a multi-million-URL catalog — so it can’t replace sitemap-based discovery at true scale. Batch and queue submissions rather than firing everything at once on a bulk republish. Treat IndexNow as an “also do this,” not a plan. (See the on-site IndexNow and Google Indexing APIThe Google Indexing API lets site owners notify Google directly when a URL is added, updated, or removed. Google officially supports it only for pages with JobPosting or BroadcastEvent (livestream) structured data — not for general content. coverage for what each is actually for.)

Common myths about indexing at scale

  • “Getting more pages crawled will improve rankings.” No. Crawling is a prerequisite for indexing, indexing for ranking — but crawling more ≠ ranking higher. “The rate of crawling isn’t going to impact your rankings.”
  • “If a page isn’t indexed, it’s a technical bug.” At scale, usually not. “Crawled — currently not indexed” is most often a quality/duplication verdict, and re-submitting or pinging it won’t help.
  • “Crawl budget matters for every site.” No — it’s a large / rapidly-changing site concern, per Google’s own scoping. Most sites never need to manage it.
  • “More indexed pages = better SEO.” No — this is the index bloatAn SEO term for when a search engine has indexed a lot of low-value, thin, or duplicate URLs that don't serve search demand. It's a quality and crawl-efficiency problem, not a penalty. myth in reverse. The goal is more valuable pages indexed, which sometimes means deliberately keeping low-value variants out.
  • “Submitting a sitemap forces indexing.” No. A sitemap is a discovery and priority hint; Google still evaluates each URL independently.
  • “Programmatic SEO is inherently penalized.” No. Volume isn’t the trigger; low value at volume with manipulative intent is (see scaled content abuse).
  • “A high ‘Discovered — not indexed’ count always means a quality problem.” No — that bucket skews toward crawl priority/capacity; the quality signal is “Crawled — not indexed.” Fixing the wrong one wastes the sprint.

Where this sits

Indexing at scale is the operational counterpart to a few neighbors: crawl budget is the mechanism underneath it, the Page Indexing report and log file analysis are how you see it, and thin content / scaled content abuse are what happens if you get the quality gate wrong. It’s also the mirror image of index bloat — that’s “too many low-value pages got in,” this is “not enough valuable pages are getting in,” and the fixes overlap because both are really about concentrating index inclusion on pages that earn it. For the broader discipline of building templated pages that deserve to rank, see the Programmatic SEOProgrammatic SEO (pSEO) is the practice of generating many pages from a single template plus a data source to target large sets of similar queries. It's powerful when each page genuinely answers its query with unique data, and spam when it just stamps a thin template across a shallow dataset. pillar; for the org side of doing this on a giant site, see Enterprise SEOEnterprise SEO is the practice of doing SEO at scale — for large, complex sites (often tens of thousands to millions of pages) across multiple teams, CMSs, and stakeholders. It uses the same ranking factors as any site; what changes is the scale, the technical debt, and the organizational coordination..

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.