Discovery

TL;DR — Discovery is how a search engine finds out your page exists — the step before it ever downloads the page. Engines mostly find pages by following links and reading sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing.. If nothing links to a page and it’s not in a sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., it may never get foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore.. Submitting a page helps it get discovered, but it doesn’t guarantee it’ll be indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..

What discovery is

Before a search engine can show your page, three things have to happen in order:

1. Discover — the engine learns your URL exists.
2. Crawl — the engine downloads the page.
3. Index — the engine stores it so it can show up in results.

Discovery is step zero of that pipeline. Google even has a name for it: URL discovery. A page that’s never discovered can’t be crawled, indexed, or ranked — the engine simply doesn’t know it’s there.

A quick note on the model: discover → crawl → index is this hub’s teaching breakdown, because it’s the clearest way to reason about a stuck page. Google’s own official stage list is three stages — crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., serving — and it defines “URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched.” as the first part of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not as a separate fourth stage. Same sequence of events either way; we’re just naming the find step explicitly because it’s useful to talk about on its own.

How search engines find your pages

There are two big ways:

• Following links. When a botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. reads a page it already knows, it grabs the links on it and adds those URLs to its list. A category page linking to a new blog post is the classic example — that’s how the new post gets found. Links from other sites (backlinks) work the same way.
• Sitemaps. An XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags. is a list of your URLs that you hand straight to search engines, so they don’t have to find everything through links. RSSAn RSS or Atom feed is an XML file listing a site's most recently published or updated URLs. Search engines accept it as a sitemap-style discovery signal for fresh content — not a replacement for a full XML sitemap. and Atom feeds do a similar job for your most recent pages.

There are also “push” options where you actively ping an engine that a page changed (like IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. for Bing) — more on those in the Advanced version — but links and sitemaps do most of the work.

Orphan pages: the thing that breaks discovery

An orphan page is a page with no internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. pointing to it. Because links are the main way engines find pages, an orphan can be hard or impossible to discover. The fix isn’t a clever submission trick — it’s linking to the page from somewhere relevant on your own site.

The thing most people get wrong

Submitting a page doesn’t mean it gets indexed. Adding a URL to your sitemap or hitting “Request Indexing” in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. helps the engine discover it — but crawling and indexing still aren’t guaranteed. If you’ve ever seen “Discovered – currently not indexed” in Search Console, that’s exactly this: Google knows the URL exists, it just hasn’t gotten around to crawling it yet.

Want the deeper version — the full pull-vs-push channel list, why Google doesn’t use IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it., and how to fix “Discovered – currently not indexedA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal.”? Switch to the Advanced tab.

TL;DR — Discovery is this hub’s name for the first move in search (discover → crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve): the engine has to learn a URL exists before it fetches anything. Google’s own official model nests this inside its three-stage crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. stage rather than naming a separate fourth stage — same sequence, different vocabulary. URLs are discovered by pull (internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., backlinks, sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., RSSAn RSS or Atom feed is an XML file listing a site's most recently published or updated URLs. Search engines accept it as a sitemap-style discovery signal for fresh content — not a replacement for a full XML sitemap./Atom) and push (IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it., the Indexing APIThe Google Indexing API lets site owners notify Google directly when a URL is added, updated, or removed. Google officially supports it only for pages with JobPosting or BroadcastEvent (livestream) structured data — not for general content., sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. lastmod, WebSubWebSub (formerly PubSubHubbub) is a push notification protocol for RSS/Atom feeds that lets a site instantly broadcast new or updated content to subscribing search engines instead of waiting to be re-crawled., URL Inspection). Google calls the find step “URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched.,” and it’s distinct from crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — which is why “Discovered – currently not indexedA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal.” means found-but-not-yet-crawled, a state with several possible causes rather than one guaranteed diagnosis. Links and sitemaps are the two routes Google names directly, orphan pagesAn orphan page is a page on your site that no other page links to internally. Because crawlers discover pages by following links, an orphan page is effectively invisible to search engines unless it's in an XML sitemap or linked from an external site. are the canonical failure, IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. is Bing/others not Google, the Indexing API is scope-limited to JobPosting/BroadcastEvent, and submitting a URL never guarantees indexing.

What URL discovery is

Discovery is the find step. Before a page can be crawled, indexed, or served, the engine has to know it exists and add it to its list of known pages. Google names this explicitly: “This process is called ‘URL discovery’.” In Google’s own words, “Some pages are known because Google has already visited them. Other pages are discovered when Google extracts a link from a known page to a new page… Still other pages are discovered when you submit a list of pages (a sitemap) for Google to crawl.”

That single passage is the whole hub in miniature: pages are found through links (internal and external) and through sitemaps. Everything else is a variation on those two ideas — or a way to push a notification instead of waiting to be pulled.

A note on the stage model. This hub treats discover → crawl → index → serve as four distinct, named steps, because separating “found” from “fetched” from “stored” is the clearest way to diagnose a stuck page. Google’s own official model names three stages — crawling, indexing, serving — and defines URL discovery as the opening move within crawling, not as its own fourth stage. The sequence of events is identical either way; this hub just gives the find step its own name and its own section because it behaves differently enough (and gets misdiagnosed often enough) to deserve one.

Discovery vs crawling vs indexing

Keep these three ideas separate and most discovery confusion disappears — using our four-step teaching breakdown of Google’s three official stages:

• Discovery — the engine learns a URL exists. Nothing has been fetched yet.
• Crawling — the engine downloads the discovered URL (and renders it). That’s the CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. hub’s job.
• Indexing — the engine analyzes and stores the crawled page so it can be served.

Google frames the back half of this as three stages: “Google Search works in three stages, and not all pages make it through each stage” — crawling, indexing, and serving. Discovery is the front door of stage one. And none of these steps is a ranking factor on its own; they’re prerequisites to being eligible to rank. A URL that’s never discovered is never crawled, indexed, or served — full stop.

How search engines discover URLs: pull vs push

The cleanest mental model is pull vs push. Pull is the engine finding URLs on its own; push is you notifying it.

Pull — the engine finds it on its own

Internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. and backlinks. Google: “Other pages are discovered when Google extracts a link from a known page to a new page: for example, a hub page, such as a category page, links to a new blog post.” Google’s own docs also say a properly linked site can usually get most of its pages discovered this way, with new sites that have few external links and large sites with unlinked pages at greater risk of gaps — links aren’t the only route in, but they’re the one Google names first and the one that scales automatically as you publish. That’s why internal linkingLinks between pages on the same site. is one of the highest-leverage discovery levers you have, and why orphan pages struggle (more below). Backlinks from other sites work identically — an external link from a page the engine already knows surfaces your new URL.

Sitemaps. Google: “A sitemap helps search engines discover URLs on your site, but it doesn’t guarantee that all the items in your sitemap will be crawled and indexed.” That caveat is the whole point — a sitemap is a discovery aid, not an indexing guarantee. The full mechanics live in the nested sitemaps deep dives (XML sitemap, sitemap indexA sitemap index is a sitemap of sitemaps — a single file that lists your other sitemap files instead of listing URLs directly. It's how large sites stay under the 50,000-URL / 50MB-per-sitemap limit while submitting just one file., image sitemapAn image sitemap is a sitemap (or an extension added to an existing sitemap) that lists the images on your pages using Google's image namespace, helping Google discover images it might otherwise miss., video sitemapA video sitemap is an XML sitemap (or mRSS feed) that uses the video:video extension to tell Google about videos hosted on your pages — the landing page, a thumbnail, a title, a description, and a link to the video file or player. It helps Google discover and surface video content, but it doesn't guarantee indexing.).

RSS / Atom feeds. Google accepts feeds as a sitemap format — “Google accepts RSS 2.0 and Atom 1.0 feeds.” The catch is that a feed only surfaces your recently changed URLs, so it complements a full sitemap rather than replacing it.

Push — you notify the engine something changed

WebSubWebSub (formerly PubSubHubbub) is a push notification protocol for RSS/Atom feeds that lets a site instantly broadcast new or updated content to subscribing search engines instead of waiting to be re-crawled. (PubSubHubbub). For RSS/Atom feeds, Google supports WebSub: “If you use Atom or RSS, you can use WebSub to broadcast your changes to search engines, including Google.” Instead of waiting to be re-pulled, your feed broadcasts the change.

Sitemap lastmod. A freshness signal that tells engines a URL changed and may be worth re-crawling. Bing leans on it harder than Google does — Bing has said the lastmod field remains a key signal that helps it prioritize URLs for recrawling and reindexing. (Detail lives in the sitemaps cluster.)

Google Indexing APIThe Google Indexing API lets site owners notify Google directly when a URL is added, updated, or removed. Google officially supports it only for pages with JobPosting or BroadcastEvent (livestream) structured data — not for general content. — scope-limited. This is the one that gets misused most. The Indexing API can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject. It is not a general “submit any URL to Google” endpoint, no matter how often it’s pitched that way. If your page isn’t a job posting or a live-stream broadcast event, the Indexing API isn’t your tool.

IndexNow — Bing and others, not Google. IndexNow is a push protocol: it notifies enabled search engines the instant a URL is added, updated, or deleted, and per its own FAQ, “Search engines adopting the IndexNow protocol agree that submitted URLs will be automatically shared with all other participating search engines.” As of this writing (per indexnow.org’s current documentation, checked July 2026), the participating engines are Microsoft Bing, Yandex, Naver, Seznam.cz, and Yep — check the live IndexNow docs for the current list, since it can change. Google is not on that list. There’s no Google help page that says “we don’t support IndexNow” — Google’s non-participation is inferred from its absence from the current partner list plus Googlers describing it as something they only tested, not from a first-party Google confirmation, so treat it as the best available reading rather than an official statement. And regardless of engine: notifying IndexNow only tells it a URL changed — each participating engine still decides independently whether and when to crawl and index it, the same “notification isn’t a guarantee” rule that applies to sitemaps and every other discovery channel. For Google, you fall back to links, sitemaps, and URL Inspection.

Manual submission — URL Inspection. For one-off Google submission: “To request a crawl of individual URLs, use the URL Inspection toolA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version..” But don’t expect repeats to force speed — Google is explicit that “there’s a quota for submitting individual URLs and requesting a recrawl multiple times for the same URL won’t get it crawled any faster.” Bing has its own URL submission channel (up to 10,000 URLs/day), separate from IndexNow.

How Google and Bing each describe discovery

Both engines frame discovery as the front of the pipeline. Bing puts it directly in its definition of crawling — Fabrice Canel: “Crawling is the process by which bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. discovers new and updated documents or content to be added to Bing’s searchable index.”

The scale is worth sitting with, even if the exact figure moves over time. Fabrice Canel has described Bing discovering on the order of tens of billions of normalized URLs it had never seen before, per day — he put it as ”12s of billions … never seen before” in an August 2022 post, a number I haven’t independently re-verified against a current primary source, so treat it as a directional sense of scale rather than a precise, current daily count. Whatever the exact figure is today, discovery operates as a firehose, and the engines deprioritize aggressively — which is the backdrop for “Discovered – currently not indexed” below.

Orphan pages: when discovery fails

An orphan page has no internal links pointing to it. Because links are one of the two discovery routes Google names directly (alongside sitemaps), an orphan can only be found via a sitemap, an external link, a redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., or a canonical/hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others. reference — and if none of those exist, it may never be discovered at all.

The fix is internal linking, not push tricks. Link the page from a relevant hub/category page — exactly Google’s “category page links to a new blog post” example. Putting an orphan in your XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags. helps Google discover it, but it doesn’t replace the internal-link signal that tells the engine the page matters. Orphans frequently show up as “Discovered – currently not indexed” for precisely this reason: Google found the sitemap entry but deprioritized crawling a page nothing links to.

”Discovered – currently not indexed” in Search Console

What’s certain about this status is the stage boundary — this hub’s job is to fix that boundary in your head, not to fully diagnose it (the dedicated deep diveA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal. does that). Google’s description: the page was found by Google but not crawled yet — Google wanted to crawl the URL but expected it would overload the site, so it rescheduled the crawl, which is why the last-crawl date is empty.

Contrast it with “Crawled – currently not indexedA Google Search Console Page Indexing status meaning Googlebot fetched the page but Google decided not to index it — usually a content- or site-quality signal, not a technical error.,” which is a different, downstream problem: the page was crawled but not indexed, and it may or may not be indexed in the future. The stage boundary is the lesson most third-party write-ups miss:

• Discovered – currently not indexed = Google knows the URL but hasn’t fetched it. A discovery/crawl-scheduling state (empty last-crawl date).
• Crawled – currently not indexed = Google fetched it but chose not to index it. An indexing/quality state.

What causes “Discovered – currently not indexed” — work through it in this order, not as a single confirmed cause:

1. Crawl scheduling / perceived server-load deprioritization — Google’s own stated reason for the status itself.
2. The page is orphaned or only weakly internally linked, lowering how much Google wants to spend crawl effort on it.
3. Site-wide quality or authority signals dragging on overall crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. — a site-level pattern, not proof of a defect on this one page.
4. A new or low-authority site with little crawl demand yet.

None of these is the universal cause — Google documents the crawl-scheduling explanation directly but doesn’t publish a single deterministic reason a given URL sits in this state, so treat the list above as a hypothesis ladder to work through, not a diagnosis to assume. It also doesn’t mean something is broken: John Mueller has said “it’s completely normal that we don’t index everything off of the website,” with the advice being to reconsider overall site quality rather than hunt for a per-page technical issue, and Gary Illyes has pointed out that the vast majority of websites don’t need to think about crawl budgetThe number of URLs an engine will crawl in a timeframe. at all. My own practitioner shorthand, from my indexing guide at Ahrefs (worth a re-read before treating it as gospel, since I haven’t re-verified the live page in this pass): Discovered means Google knows the URL but hasn’t crawled it; Crawled means it was fetched but not indexed and usually points to a quality issue — and the fixes overlap. Make the content unique, valuable, and intent-matched; clear any stray noindex/robots/canonical blockers; keep the server fast and stable; build a logical hierarchy with strong internal linking; then use URL Inspection to request a re-crawl and monitor. For the full breakdown of crawl capacity vs. crawl demand and a step-by-step fix path, see the dedicated “Discovered – currently not indexed” articleA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal..

How to help discovery

In rough order of leverage:

1. Strengthen internal links to the page — kill the orphan. This is the highest- leverage lever, because links are one of the two discovery routes Google names directly.
2. Put it in a clean XML sitemap (canonical, indexable URLs; accurate lastmod). Submit the sitemap in Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. and Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility..
3. Raise overall site quality so crawl demand rises — this is what moves “Discovered – currently not indexed” pages.
4. Use URL Inspection → Request Indexing for a single important URL (remember the quota; repeats don’t speed it up).
5. For Bing and other participating engines, use IndexNow to push changed URLs instantly. For Google, there’s no equivalent push for general pages — rely on links + sitemaps + URL Inspection.

Where to go next

This hub is the map for how a URL becomes known. The deep dives nested under it cover the mechanics of the sitemap side:

Sitemaps

• Sitemaps — the overview: what a sitemap is, the formats Google accepts, and how submitting one fits into discovery (aid, not guarantee).
  • XML sitemap — the standard format, its fields, size limits, and lastmod done right.
  • Sitemap index — how to split and reference multiple sitemaps when you outgrow one file.
  • Image sitemap — surfacing images for discovery in image search.
  • Video sitemap — surfacing video content and its metadata.

For what happens after a URL is discovered — fetching, the crawl scheduler, renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., and crawl budget — see the CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. hub. Related push topics like IndexNow and the Google Indexing API also live near the crawling side. For the broader picture, see How Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank..

Every nested topic above is its own deep dive under this hub — they’re in the sidebar too.

AI summary

A condensed take on the Advanced version:

• Discovery = the find step in search (discover → crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve). The engine has to learn a URL exists before anything is fetched. Google calls it “URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched.” and nests it inside its official three-stage model (crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., serving) rather than naming it a separate fourth stage.
• It’s distinct from crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and indexing. Discovered = known. Crawled = fetched. Indexed = stored. None of the three is a ranking factor on its own.
• Pull channels (engine finds it): internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. and backlinks (the link-based route Google names directly), XML sitemapsAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags. (the other route Google names directly), RSSAn RSS or Atom feed is an XML file listing a site's most recently published or updated URLs. Search engines accept it as a sitemap-style discovery signal for fresh content — not a replacement for a full XML sitemap./Atom feeds (recent URLs only).
• Push channels (you notify): WebSubWebSub (formerly PubSubHubbub) is a push notification protocol for RSS/Atom feeds that lets a site instantly broadcast new or updated content to subscribing search engines instead of waiting to be re-crawled., sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. lastmod, the Google Indexing API (JobPosting/BroadcastEvent only — not general submission), IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. (Bing/Yandex/Naver/Seznam/Yep — not Google), and URL Inspection “Request Indexing” (quota-limited).
• SitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. aid discovery, not indexing — submitting a URL never guarantees a crawl or an index.
• Orphan pagesAn orphan page is a page on your site that no other page links to internally. Because crawlers discover pages by following links, an orphan page is effectively invisible to search engines unless it's in an XML sitemap or linked from an external site. (no internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them.) are the canonical discovery failure; fix with internal links, not push tricks.
• “Discovered – currently not indexedA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal.” = Google knows the URL but hasn’t crawled it (a scheduling/priority state with several possible contributing causes, not one guaranteed diagnosis), distinct from the quality-driven “Crawled – currently not indexed.”
• Help discovery with internal links, a clean sitemap, higher site quality, URL Inspection, and IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. for Bing.

Official documentation

Primary-source documentation from the search engines.

Google

• In-Depth Guide to How Google Search Works — defines “URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched.” and the discover → crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve pipeline.
• Sitemaps overview — how sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. help discovery (and why they don’t guarantee indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.).
• Build and submit a sitemap — RSSAn RSS or Atom feed is an XML file listing a site's most recently published or updated URLs. Search engines accept it as a sitemap-style discovery signal for fresh content — not a replacement for a full XML sitemap./Atom feed support and WebSubWebSub (formerly PubSubHubbub) is a push notification protocol for RSS/Atom feeds that lets a site instantly broadcast new or updated content to subscribing search engines instead of waiting to be re-crawled. for pushing feed changes.
• Indexing API quickstart — the scope limit: JobPosting and BroadcastEvent pages only.
• Ask Google to recrawl your URLs — URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. submission and the quota note.
• Page Indexing report — the “Discovered – currently not indexedA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal.” and “Crawled – currently not indexedA Google Search Console Page Indexing status meaning Googlebot fetched the page but Google decided not to index it — usually a content- or site-quality signal, not a technical error.” definitions.

Bing / Microsoft

• bingbot Series: Maximizing Crawl Efficiency — Bing’s definition of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. as a discovery process.
• Submit up to 10,000 URLs/day to Bing — Bing’s own URL submission channel.
• Keeping Content Discoverable with Sitemaps in AI-Powered Search — sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. + IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. as the discovery foundation, and lastmod as a recrawlCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. signal.
• IndexNow / indexnow.org — the push protocol for instantly signaling changed URLs.

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what discovery is

• “Google must constantly look for new and updated pages and add them to its list of known pages. This process is called ‘URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched.’.” — Google Search Central docs. Jump to quote
• “Other pages are discovered when Google extracts a link from a known page to a new page: for example, a hub page, such as a category page, links to a new blog post.” Jump to quote
• “Google Search works in three stages, and not all pages make it through each stage.” Jump to quote

Google — sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., feeds, and push

• “A sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. helps search engines discover URLs on your site, but it doesn’t guarantee that all the items in your sitemap will be crawled and indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..” — Google sitemaps overview. Jump to quote
• “Google accepts RSSAn RSS or Atom feed is an XML file listing a site's most recently published or updated URLs. Search engines accept it as a sitemap-style discovery signal for fresh content — not a replacement for a full XML sitemap. 2.0 and Atom 1.0 feeds.” Jump to quote
• “If you use Atom or RSS, you can use WebSubWebSub (formerly PubSubHubbub) is a push notification protocol for RSS/Atom feeds that lets a site instantly broadcast new or updated content to subscribing search engines instead of waiting to be re-crawled. to broadcast your changes to search engines, including Google.” Jump to quote
• “To request a crawl of individual URLs, use the URL Inspection toolA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version..” Jump to quote
• “There’s a quota for submitting individual URLs and requesting a recrawlCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. multiple times for the same URL won’t get it crawled any faster.” Jump to quote

Google — not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. everything is normal

• “It’s completely normal that we don’t index everything off of the website.” — John Mueller, Google. Read the coverage

Bing / Microsoft

• “CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is the process by which bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. discovers new and updated documents or content to be added to Bing’s searchable index.” — Fabrice Canel, Microsoft Bing. Jump to quote
• “Instead of Bing continually monitoring RSS and similar feeds or frequently crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. websites to check for new pages, discover content changes and/or new outbound links, websites will notify Bing directly about relevant URLs changing on their website.” — Microsoft Bing, on Bing’s URL submission. Jump to quote

IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it.

• “Search engines adopting the IndexNow protocol agree that submitted URLs will be automatically shared with all other participating search engines.” — indexnow.org documentation. Jump to quote

Discovery checklist

A quick pass to confirm search engines can actually find what matters:

• Every important page is linked from somewhere crawlable — no orphans.
• Important pages sit close to the homepage / main nav (the more linked, the easier to discover).
• An XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags. lists only canonical, indexable URLs, with accurate lastmod, and is submitted in Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. and Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility..
• If you publish frequently, an RSSAn RSS or Atom feed is an XML file listing a site's most recently published or updated URLs. Search engines accept it as a sitemap-style discovery signal for fresh content — not a replacement for a full XML sitemap./Atom feed surfaces recent URLs (and WebSubWebSub (formerly PubSubHubbub) is a push notification protocol for RSS/Atom feeds that lets a site instantly broadcast new or updated content to subscribing search engines instead of waiting to be re-crawled. broadcasts changes).
• You’re not relying on the Google Indexing APIThe Google Indexing API lets site owners notify Google directly when a URL is added, updated, or removed. Google officially supports it only for pages with JobPosting or BroadcastEvent (livestream) structured data — not for general content. for general pages — it’s only for JobPosting / BroadcastEvent.
• IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. is wired up for Bing and other participating engines (it does nothing for Google).
• For a single urgent URL, you use URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. → Request IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. (and don’t spam it — quota applies).
• Pages you want found aren’t accidentally blocked by robots.txt or buried behind click-only navigation instead of real <a href> links.
• You’ve checked the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. for “Discovered – currently not indexed” and treated it as a linking/quality signal, not a per-page bug.

The mental models

1. Discovery = pull + push. Pull is the engine finding URLs on its own (internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., backlinks, sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., RSSAn RSS or Atom feed is an XML file listing a site's most recently published or updated URLs. Search engines accept it as a sitemap-style discovery signal for fresh content — not a replacement for a full XML sitemap./Atom). Push is you notifying it (IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it., the Indexing APIThe Google Indexing API lets site owners notify Google directly when a URL is added, updated, or removed. Google officially supports it only for pages with JobPosting or BroadcastEvent (livestream) structured data — not for general content., lastmod, WebSubWebSub (formerly PubSubHubbub) is a push notification protocol for RSS/Atom feeds that lets a site instantly broadcast new or updated content to subscribing search engines instead of waiting to be re-crawled., URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version.). When a page isn’t being foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore., ask: which channel should be carrying it — and is anything actually linking to it?

2. Orphan pagesAn orphan page is a page on your site that no other page links to internally. Because crawlers discover pages by following links, an orphan page is effectively invisible to search engines unless it's in an XML sitemap or linked from an external site. can’t be discovered. Links are one of the two routes Google names directly, so a page nothing links to has almost nothing carrying it into the engine’s known-pages list. A sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. entry helps it get discovered but doesn’t replace the internal linkAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. that says the page matters. Fix discovery failures with links first.

3. Discovery ≠ crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. ≠ indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Three separate stages: Discovered (the engine knows the URL) → Crawled (it fetched the URL) → Indexed (it stored the URL). None is a ranking factor on its own — they’re prerequisites. Locate which stage a page is failing at before you change anything.

4. “Discovered – currently not indexedA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal.” = found, not yet (or never) crawled. It’s a discovery/scheduling state: Google knows the URL but hasn’t fetched it (empty last-crawl date). That’s different from “Crawled – currently not indexedA Google Search Console Page Indexing status meaning Googlebot fetched the page but Google decided not to index it — usually a content- or site-quality signal, not a technical error.,” which is a downstream quality/indexing state. The fix for Discovered is usually internal links + site quality + a clean sitemap, not repeated re-submission.

5. Submission ≠ indexing. Adding a URL to a sitemap, pinging IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it., or hitting Request Indexing only helps discovery. The engine still decides whether to crawl and index. Never promise a client that “submitting it” gets it indexed.

Discovery channels — cheat sheet

PULL — the engine finds it on its own

PUSH — you notify the engine something changed

Two myths to keep straight

• IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. = Bing/Yandex/others, NOT Google. There’s no Google IndexNow endpoint; for Google use links + sitemaps + URL Inspection.
• IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. API = Google, JobPosting/BroadcastEvent only. It is not a general “submit any URL to Google” tool.

Common discovery issues

Symptom-first fixes for the discovery problems that actually show up in Search Console and site audits.

”Discovered – currently not indexed” in the Page Indexing report

Symptom: A page sits in the “Discovered – currently not indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.” bucket, with an empty last-crawl date, sometimes for weeks.

Likely cause(s): Google knows the URL exists (from a link or a sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing.) but hasn’t scheduled a crawl for it yet — usually because of low crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side.: the page is orphaned or weakly linked, or site-wide quality/authority signals are keeping overall crawl demand low.

Fix + check: Add internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. to the page from a relevant hub/category page, confirm it’s in a clean XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags. with an accurate lastmod, and raise site-wide quality if the site as a whole has low crawl demand. Confirm the fix by re-checking the same URL in URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. after a few days — a populated last-crawl date means the state has moved (to indexed, or to “Crawled – currently not indexed,” which is a different problem).

A new page never shows up anywhere in Search Console

Symptom: A URL you know exists doesn’t appear in the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. at all — not discovered, not crawled, nothing.

Likely cause(s): It’s an orphan pageAn orphan page is a page on your site that no other page links to internally. Because crawlers discover pages by following links, an orphan page is effectively invisible to search engines unless it's in an XML sitemap or linked from an external site. with no internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. pointing to it, it was never added to the sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., and no external site linksSitelinks are extra links from the same domain that Google clusters together under a single search result, usually for branded or navigational queries. They're generated entirely algorithmically — there's no way to add, edit, or guarantee them. to it either.

Fix + check: Link to it from a relevant page on your own site (the single biggest lever), add it to your XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags., and submit the URL through URL Inspection → Request IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. for a one-off nudge. Check by searching site:yourdomain.com/the-url — no results confirms it’s still undiscovered.

IndexNow pings aren’t doing anything for Google traffic

Symptom: IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. is wired up, changed URLs are being pinged, but Google rankings/traffic for those URLs don’t move any faster.

Likely cause(s): Google does not participate in IndexNow — only Bing, Yandex, Naver, Seznam.cz, and Yep do. Pinging IndexNow has no effect on Google discovery.

Fix + check: Keep IndexNow for the engines that support it, and rely on links, sitemaps, and URL Inspection for Google. Confirm by checking whether the affected URLs show movement in Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. (where IndexNow should help) versus Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. (where it won’t).

The Indexing API call gets rejected or does nothing

Symptom: A call to Google’s Indexing API returns an error, or the URL never gets crawled faster despite a successful-looking call.

Likely cause(s): The Indexing API only accepts pages with JobPosting or BroadcastEvent (in a VideoObject) structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding. — it’s not a general “submit any URL” endpoint, no matter how it gets pitched.

Fix + check: Confirm the page actually carries JobPosting or BroadcastEvent markup before troubleshooting the API call itself. If it’s any other page type, switch to internal links, a sitemap, and URL Inspection instead.

Playbook: a new page isn’t showing up in search

A linear runbook for the most common discovery complaint — “I published it, why can’t I find it in Google?”

1. Confirm it’s actually a discovery problem, not an indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. problem. Run the URL through URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. in Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.. If the report shows “Discovered – currently not indexedA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal.,” you’re in the right place — keep going. If it shows “Crawled – currently not indexedA Google Search Console Page Indexing status meaning Googlebot fetched the page but Google decided not to index it — usually a content- or site-quality signal, not a technical error.,” this is a downstream quality issue, not a discovery issue — stop here and work the content/quality angle instead.

2. Check whether the page is an orphan. Crawl the site (or check your internal-link report) for any live <a href> links pointing to the URL. If nothing links to it, that’s a strong candidate for the root cause — links are one of the two discovery routes Google names directly.

3. If it’s an orphan, link to it from a relevant page. Add a real, crawlable link from a hub or category page that’s already indexed — exactly Google’s own example of a category page linking to a new blog post. Don’t substitute a sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. entry for this step; a sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. helps discovery but doesn’t carry the same weight as an internal linkAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them..

4. Check the sitemap regardless. Confirm the URL is listed in your XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags. with the canonical form and an accurate lastmod, and that the sitemap itself is submitted in Google Search Console and Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility..

5. If you see “Discovered – currently not indexed” and the page is already well linked, look at site-wide quality. Google and Bing both deprioritize crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. for sites with weaker overall quality signals — this is a site-level issue, not a bug on this one page. Improving overall site quality is what actually moves pages out of this state at scale.

6. For a single urgent URL, request indexing manually. Use URL Inspection → Request IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. once. Remember there’s a quota, and repeating the request doesn’t get it crawled any faster.

7. If the target engine is Bing (or Yandex, Naver, Seznam, Yep), push with IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. instead of waiting. IndexNow gives those engines an instant notification. It does nothing for Google — don’t expect it to move the Google needle.

8. Re-check after a few days, not a few hours. Discovery and crawl scheduling operate on a queue. Come back to URL Inspection later rather than repeatedly resubmitting.

Tools for checking and fixing discovery

• XML Sitemap Generator — build a clean sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. of canonical, indexable URLs to hand straight to search engines.
• Sitemap Validator — check an existing sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. for malformed entries, non-canonical URLsHow search engines pick one canonical URL among duplicates and consolidate signals onto it., or other issues that weaken it as a discovery aid.
• Robots.txt Tester — confirm robots.txt isn’t accidentally blocking crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. access to pages you want discovered.
• Canonical Checker — verify canonical tagsA rel=\"canonical\" annotation — in the HTML <head> or an HTTP Link header — that tells search engines which URL is the preferred version of duplicate or near-duplicate content. aren’t quietly pointing discovery and indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. signals away from the page you want foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore..
• Scout Site Audit Free — crawl your own site to surface orphan pagesAn orphan page is a page on your site that no other page links to internally. Because crawlers discover pages by following links, an orphan page is effectively invisible to search engines unless it's in an XML sitemap or linked from an external site. and weak internal-linking patterns before they turn into “Discovered – currently not indexedA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal..”
• Log File Analyzer — check server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. to see whether GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. and BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. are actually requesting the URLs you expect them to discover and crawl.
• Google Search Console CSV Analyzer — work with Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. and coverage data at scale instead of clicking through individual URLs.
• Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. — URL Inspection — check whether a specific URL is discovered, crawled, and indexed, and request a one-off crawl.
• Google Search Console — Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. — see how many URLs are stuck in “Discovered – currently not indexed” versus “Crawled – currently not indexedA Google Search Console Page Indexing status meaning Googlebot fetched the page but Google decided not to index it — usually a content- or site-quality signal, not a technical error..”
• Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. — Bing’s equivalent of URL Inspection and the Page Indexing report, plus its own URL submission channel and IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. status.
• IndexNow (indexnow.org) — ping Bing, Yandex, Naver, Seznam.cz, and Yep the moment a URL changes.

Resources worth your time

My related writing

• Crawled – currently not indexed: 7 ways to fix it — the GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. status that sits right after discovery, and how the fixes overlap with “Discovered – currently not indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..”
• Googlebot: What Is It & How Does It Work? — where GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. collects URLs from (pages, sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., RSS feedsAn RSS or Atom feed is an XML file listing a site's most recently published or updated URLs. Search engines accept it as a sitemap-style discovery signal for fresh content — not a replacement for a full XML sitemap., Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance., the Indexing APIThe Google Indexing API lets site owners notify Google directly when a URL is added, updated, or removed. Google officially supports it only for pages with JobPosting or BroadcastEvent (livestream) structured data — not for general content.) and how it reprocesses pages to find more links.
• IndexNow: now in Ahrefs and powering Yep — what IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. does, which engines participate, and why Google sits it out.
• The Beginner’s Guide to Technical SEO — where discovery fits in the bigger picture.

My speaking

• How Search Works (SlideShare) — my walkthrough of discovery, crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and ranking, including the discovery sources (links, XML sitemapsAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags., GSC requests, the Indexing API, RSS feedsAn RSS or Atom feed is an XML file listing a site's most recently published or updated URLs. Search engines accept it as a sitemap-style discovery signal for fresh content — not a replacement for a full XML sitemap., WebSubWebSub (formerly PubSubHubbub) is a push notification protocol for RSS/Atom feeds that lets a site instantly broadcast new or updated content to subscribing search engines instead of waiting to be re-crawled.) and the crawl-demand vs crawl-rate model that explains why a page sits “Discovered – currently not indexedA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal..” (My standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• r/TechSEO — the community for discovery, crawl, and index debugging.
• Google’s Crawling December series — official, and the best concentrated set of crawl/discovery explainers.
• Understanding and resolving “Discovered – currently not indexed” (Search Engine Land) — causes and remediation, including the Gary Illyes crawl-budget framing.
• It’s Normal for Pages of a Site to Not Be Indexed (Search Engine Journal) — covers John Mueller’s “completely normal” statement on Google not indexing everything, and the quality-over-submission advice.
• Search Engine Roundtable — Crawled, Currently Not Indexed Is A Quality Issue — Mueller on the downstream “Crawled – currently not indexedA Google Search Console Page Indexing status meaning Googlebot fetched the page but Google decided not to index it — usually a content- or site-quality signal, not a technical error.” status being site-wide quality, not a per-page bug.
• Bing Lets Webmasters Submit 10,000 URLs Per Day (Search Engine Land) — background on Bing’s own URL submission channel, distinct from IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it..
• Search Engine Crawling — How It Works (Lumar) — crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. mechanics, link-following, and sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. as a discovery foundation.

Podcasts

• Search Off the Record (Google Search Relations) — How GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. crawls the web. Gary Illyes and Martin Splitt on how GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. discovers and fetches the web: unified crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. infrastructure, conditional requestsA conditional request lets a crawler (or browser) ask a server 'has this changed since I last fetched it?' — using If-Modified-Since (checked against your Last-Modified header) and/or If-None-Match (checked against your ETag). If nothing changed, the server replies 304 Not Modified with no body, so the crawler reuses its existing copy instead of re-downloading the page., and how discovery feeds the scheduler. Listen

Videos

• Google Search Central (YouTube) — the How Google Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank. series and Martin Splitt’s crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor./renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. explainers, which cover where URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched. sits in the pipeline. Channel

Test yourself: Discovery

Five questions on how search engines find URLs, based on the hub above.

URL discovery

URL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched.

Related: Crawling, Sitemap, IndexNow

URL discovery

URL discovery is the very first thing that has to happen for a page to appear in search: the engine has to learn the URL exists. It’s Google’s own name for this step — before anything is crawled, indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., or ranked, a URL has to be discovered and added to the engine’s list of known pages.

Discovery happens two ways. Pull is the engine finding URLs on its own: it extracts links from pages it already knows (internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. and backlinks), and it reads the sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. and RSSAn RSS or Atom feed is an XML file listing a site's most recently published or updated URLs. Search engines accept it as a sitemap-style discovery signal for fresh content — not a replacement for a full XML sitemap./Atom feeds you publish — links and sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. are the two routes Google names directly, and links do especially heavy lifting on a well-linked site, which is why a page that nothing links to — an orphan pageAn orphan page is a page on your site that no other page links to internally. Because crawlers discover pages by following links, an orphan page is effectively invisible to search engines unless it's in an XML sitemap or linked from an external site. — struggles to get discovered at all. Push is you telling the engine directly that something changed: IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. (used by Bing, Yandex, and others — not Google, per IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it.’s current documentation), Google’s Indexing APIThe Google Indexing API lets site owners notify Google directly when a URL is added, updated, or removed. Google officially supports it only for pages with JobPosting or BroadcastEvent (livestream) structured data — not for general content. (officially only for JobPosting and BroadcastEvent pages), sitemap lastmod, WebSubWebSub (formerly PubSubHubbub) is a push notification protocol for RSS/Atom feeds that lets a site instantly broadcast new or updated content to subscribing search engines instead of waiting to be re-crawled. for feeds, and Google’s URL Inspection “Request IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..”

Discovery is distinct from crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and indexing. Discovery = the engine knows the URL. Crawling = the engine fetches it. Indexing = the engine analyzes and stores it. That’s why Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance.’s “Discovered – currently not indexedA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal.” status means Google knows about the URL but hasn’t crawled it yet — a discovery/scheduling state with several possible contributing causes (crawl scheduling, weak internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., site-wide crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side.), not a single deterministic content problem. Submitting a URL or a sitemap aids discovery; it never guarantees a crawl or an index.

Related: Crawling, Sitemap, IndexNow

Changelog