Noindex

Noindex keeps a page out of search results — but only if Google can crawl it. The two valid methods, the robots.txt trap, and how to verify it worked.

First published: Jun 23, 2026 · Last updated: Jul 26, 2026 · Advanced
demand #8 in Meta Tags#22 in On-Page#103 in Technical SEO#136 on the site
AIAuthored byAI v2

Noindex is the directive that keeps a page out of the index, so it won't appear in search results. There are two valid ways to set it: the robots meta tag (`<meta name="robots" content="noindex">`) and the `X-Robots-Tag: noindex` HTTP header (the only option for non-HTML files like PDFs). The single biggest mistake: a page blocked in robots.txt can't be noindexed, because Google never crawls it to see the rule — so to remove a page you have to allow crawling and serve noindex. Don't put noindex in robots.txt (unsupported since Sept 1, 2019); if noindex and canonical coexist, treat that as an intent check rather than an automatic error; and remember deindexing only happens after a recrawl.

TL;DR — noindex removes a page from the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. via one of two valid methods: the robots meta tagThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen. (<meta name="robots" content="noindex">) or the X-Robots-Tag: noindex HTTP header (required for non-HTML files like PDFs). The load-bearing gotcha: a page blocked in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. can’t be noindexed — Google never crawls it to see the rule, “the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. will never see the noindex rule,” and a linked URL can stay indexed. So to remove a page, allow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and serve noindex. Don’t put noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. (unsupported since Sept 1, 2019), review noindex with a canonical pointing elsewhere as potentially conflicting, and know that deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. only happens after a recrawlCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. — Google’s own guidance says a low-priority page can take months. Per a 2017 Mueller comment (not documented policy), long-term noindex,follow tends to behave like noindex,nofollow once the page drops from the index. Verify in GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. under “URL marked ‘noindex’.”

What noindex is — index control, not crawl control

noindex is the primary index-control directive. Google’s own definition of the rule is one line: “Do not show this page, media, or resource in search results.” When it’s honored, the effect is total — “When GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. crawls that page and extracts the tag or header, Google will drop that page entirely from Google Search results, regardless of whether other sites link to it.” Evidence for this claim Google's noindex rule prevents the page, media, or resource from appearing in Google Search results after Google sees the rule. Scope: Google Search; noindex is not an access-control or privacy mechanism. Confidence: high · Verified: Google Search Central: Block Search indexing with noindex

Keep one distinction front of mind, because almost every noindex mistake comes from blurring it: noindex controls indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.; robots.txt controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. They’re different stages of the pipeline. I put it this way in my Ahrefs guide on removing URLs: “Crawling is not the same thing as indexing. Even if Google is blocked from crawling pages, if there are any internal or external links to a page they can still index it.” That sentence is the whole reason the rest of this article exists.

Microsoft gives the same directive an additional Bing-specific consequence: content marked noindex is also excluded from Microsoft’s foundation-model training. The prerequisite still matters—BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. must be allowed to crawl and process the page-level directive. A robots.txt block plus noindex is therefore not proof that either deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. or the training opt-out has been applied.

Evidence for this claim Microsoft says content marked noindex is not included in the Bing index and is not used to train its generative AI foundation models. Scope: Bing and Microsoft foundation-model use; Bingbot must be able to crawl and process the directive before the outcome can be inferred. Confidence: high · Verified: Bing Webmaster Blog: New controls for Bing Chat

The two valid delivery methods

There are exactly two, and noindex in robots.txt is not one of them (more on that below).

Method 1 — the robots meta tagThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen.. For an HTML page, place this in the <head>:

<meta name="robots" content="noindex">

Google’s instruction is verbatim: “To prevent all search engines that support the noindex rule from indexing a page on your site, place the following <meta> tag into the <head> section of your page.” The robots value targets all crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. that support the rule; swap in googlebot to target only Google (<meta name="googlebot" content="noindex">).

Method 2 — the X-Robots-TagThe X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos. HTTP header. Same directive, sent in the response header instead of the markup:

X-Robots-Tag: noindex

This is the only way to noindex non-HTML resources, because there’s no <head> to host a meta tag. Google: “A response header can be used for non-HTML resources, such as PDFs, video files, and image files.” And from the robots spec: you can use the X-Robots-Tag “for non-HTML files like image files where the usage of robots meta tags in HTML is not possible.” Evidence for this claim Google supports noindex in an HTML robots meta tag or an X-Robots-Tag HTTP response header. Scope: Google Search delivery methods; the HTTP header is applicable to non-HTML resources as well as HTML. Confidence: high · Verified: Google Search Central: Robots meta tag and X-Robots-Tag specifications

One placement note: put the meta tag in the <head> — that’s the standard, safest spot and what Google’s how-to shows. Google’s spec page does say it “doesn’t enforce placement of meta robots in the HTML head and will respect robots meta tagsMeta tags are HTML elements in a page's head that pass metadata about the page to search engines and browsers. For SEO only a few matter — the title element, the meta description, and the robots meta tag — while meta keywords and most others are ignored. in the body section of an HTML document as well,” but don’t rely on that as your primary method; a stray <meta> tag some CMSA content management system (CMS) is software that lets users create, manage, and publish digital content — like blog posts and pages — without writing raw code. WordPress, Drupal, and Joomla are the most common open-source CMS platforms. injects into the <body> can noindex a page by accident just as easily as one you meant to add to the <head>.

Since the header is configured at the server level, it varies by stack. Two common examples for noindexing every PDF on a site:

Apache (.htaccess or vhost):

<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

Nginx (server/location block):

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex";
}

The #1 mistake — noindex + a robots.txt block

Noindex is crawl-then-obey: keep the URL fetchable long enough for the directive to be processed.

The same page contains a meta robots noindex directive. With crawling allowed, Google can fetch the page, see noindex, and remove the URL after processing. With crawling blocked in robots.txt, Google cannot see noindex and the linked URL may remain in results.

This is the failure mode I see most, so here’s the mechanism in full. The noindex tag lives on the page; Google has to fetch the page to read it. Google states the requirement directly:

“For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.”

Put even more bluntly: “We have to crawl your page in order to see <meta> tags and HTTP headers.” No crawl, no rule.

robots.txt is the most common way a page ends up uncrawlable, but Google’s wording covers more ground than that — it also says “the crawler can’t access the page,” which includes repeated server errors (5xx), timeouts, and an unintended authentication wall in front of the page. Any of those silently breaks noindex the same way a robots.txt block does.

So the instinct to “block it in robots.txt and noindex it, just to be safe” is exactly backwards — the block prevents the crawl, the crawl is what reveals the noindex, and the page can sit in the index indefinitely (often as a description-less URL). In Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. this shows up as the “Indexed, though blocked by robots.txtA Google Search Console Page Indexing warning: Google indexed the URL anyway despite your robots.txt disallowing crawling it — Google names other pages linking to it as the likely path. robots.txt blocks crawling, not indexing.” status — a page you blocked that got indexed anyway because something links to it.

The fix: unblock the page in robots.txt, keep noindex on it, and let Google recrawl. Only after the page has dropped from the index — if you then want to save the crawl entirely — is it safe to add a disallow.

noindex vs nofollow vs disallow

Three directives people constantly conflate. They operate at different stages:

  • noindex — index control. Page is crawled, kept out of results. Google’s definition: “Do not show this page, media, or resource in search results.”
  • nofollow — link control. Google: “Do not follow the links on this page.” It says nothing about indexing the page itself.
  • disallow (robots.txt) — crawl control. Stops the fetch entirely. It is not an index control — a disallowed URL can still be indexed if it’s linked.

There’s also none, which Google documents as “Equivalent to noindex, nofollow.” And when directives conflict, the spec is clear: “In the case of conflicting robots rules, the more restrictive rule applies.” (Full table on the Cheat Sheets tab.)

Treat noindex with rel=canonical as an intent check

Putting noindex and rel="canonical" on the same page is not automatically invalid. It does create a configuration worth reviewing: a canonical asks Google to consolidate signals, while noindex asks for this URL to be excluded. For choosing between duplicates, use the canonical tagA rel=\"canonical\" annotation — in the HTML <head> or an HTTP Link header — that tells search engines which URL is the preferred version of duplicate or near-duplicate content. — Google specifically advises against using noindex for it: “We don’t recommend using noindex to prevent selection of a canonical page within a single site, because it will completely block the page from Search.” Note the scope: Google’s caution is specifically about using noindex to pick which duplicate wins as canonical within your own site — it’s not a claim that noindex and canonical can never technically coexist on a page (a page you’re genuinely retiring can still carry a self-referencing canonical). A canonical pointing at a different URL deserves the strongest warning: confirm that exclusion and consolidation are both intended. Use canonical to consolidate duplicates; use noindex only when you genuinely want this page out of results.

noindex,follow vs noindex,nofollow — the slow decay

A common pattern is noindex,follow: keep the page out of results, but keep following its links so equity still flows through it (handy during a migration or while a page is temporarily out). Current official Google documentation doesn’t describe this decaying automatically — it explicitly allows combining noindex with other rules, including setting noindex,nofollow on purpose from day one. What I’m relying on for the “it fades over time” claim is a 2017 webmaster hangout, where John Mueller said a long-term noindex tends to end up treated like noindex,nofollow in practice: once Google decides the page really doesn’t belong in search and drops it completely, it also stops following that page’s links, because it’s stopped processing the page at all. That’s a practitioner observation from a video transcript, not a documented Google policy, so treat it as directional rather than guaranteed. Either way, the practical takeaway holds: noindex,follow is fine for a transitional period, but don’t lean on it as a permanent link-equity strategy — plan to fix the underlying links (or remove the page) instead.

How long does noindex take?

Not instantly. noindex only applies after Google recrawls and reprocesses the page — until then, the page can stay indexed even though the tag is live. Google doesn’t commit to a fixed window, and its own guidance leans toward “could be a while,” not “any day now”: “Depending on the importance of the page on the internet, it may take months for Googlebot to revisit a page.” A high-traffic, frequently-linked page might get recrawled in days; a low-value, rarely-linked one can sit for months. If you need a page out of results urgently, the GSC Removals toolA Google Search Console feature for verified owners to manage how their URLs appear in Search. A Temporary Removal hides a URL for about six months — it does not delete the page from the index. is a stopgap (it hides the URL temporarily while the permanent noindex does its slower work). For genuinely gone pages, a 404/410 also drops them: as I wrote in my removal guide, “If you remove the page and serve either a 404 (not found) or 410 (gone) status code, then the page will be removed from the index shortly after the page is re-crawled.” Same theme everywhere — it happens on recrawl.

noindex in robots.txt is dead (since Sept 1, 2019)

You’ll still see people suggest a Noindex: line in robots.txt. Don’t. It was never an officially supported rule, and Google retired even its unofficial handling years ago. From the July 2019 Search Central announcement: “Since these rules were never documented by Google, naturally, their usage in relation to Googlebot is very low.” And the date: “we’re retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019.”

The same post named the supported alternatives, and noindex via the meta tag / header topped the list: “noindex in robots meta tags: Supported both in the HTTP response headers and in HTML, the noindex rule is the most effective way to remove URLs from the index when crawling is allowed.” (Also listed: 404/410 status codes, password protection, robots.txt disallow for crawl prevention, and the Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. removal tool.)

How to verify noindex in Google Search Console

TIP Verify the live noindex signal before waiting on Google

This check confirms what the current response exposes; it does not claim that Google has already recrawled or removed the URL.

Check the deployed indexability signals with my free Google Index Checker Free

  1. Test the canonical live URL after deployment.
  2. Confirm noindex is present and the response is not blocked from crawling.
  3. Use URL Inspection to compare Google's last processed state and request a recrawl when appropriate.
The response is currently ineligible because of noindex; removal still depends on a later Google recrawl and processing.

The completed Google Index Checker result reports a noindex directive in the current response and separates it from robots access, response status, and canonical signals. It evaluates observable eligibility, not Google's already-processed index state.

Two checks:

  • URL Inspection. Run the URL through Inspect, then Test live URL. It tells you whether the page is indexable and whether Google sees a noindex directive — the fastest way to confirm the tag is being read on the live page.
  • Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.. Noindexed pages are listed under the status “URL marked ‘noindex’” in the Not indexed section. Google’s help text: “When Google tried to index the page it encountered a ‘noindex’ directive and therefore did not index it.” If that’s a page you wanted indexed, that’s your bug — remove the directive.

One naming note for anyone searching old write-ups: the legacy Coverage report called this “Excluded by ‘noindex’ tag.” The current Page Indexing report uses “URL marked ‘noindex’” — same thing, newer label.

What noindex doesn’t guarantee

A few things people assume noindex buys them that it actually doesn’t:

  • Crawl-budget savings. Google still has to fetch the page to see the tag — noindex alone doesn’t reduce crawling. If you want that too, add disallow in robots.txt, but only after the page has already dropped from the index (see the mistake above for why doing it up front backfires).
  • Instant removal. Covered above — it happens on recrawl, with no fixed timetable, and Google itself says a lower-priority page can take months.
  • Duplicate consolidation. That’s what rel="canonical" is for; noindex just removes the page from Search, it doesn’t merge signals toward another URL.
  • Confidentiality. The page stays publicly requestable by anyone with the URL. If something actually needs to be private, that’s an authentication problem, not a search-directive problem.
  • Ranking recovery if you reverse it. Removing noindex doesn’t restore a page’s old rankings — Google has to recrawl, re-evaluate, and effectively re-earn its position from scratch.
  • Identical timing across search engines. Bing and other engines run their own crawl and recrawl schedules independently of Google’s.
  • Exclusion from every non-search use of your content. noindex blocks a page from Google Search as a whole — including Search’s own AI features (AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. and similar draw on pages that are indexed and eligible to be shown, so a noindexed page is out of those too). What it does not do is control Google’s separate Google-Extended setting, which governs whether your content can be used to train or ground Google’s generative AI models outside of Search. Those are two different controls for two different jobs.

Where noindex fits with everything else

noindex is the lever you reach for when a page is in the index but shouldn’t be — the cure for one flavor of index bloatAn SEO term for when a search engine has indexed a lot of low-value, thin, or duplicate URLs that don't serve search demand. It's a quality and crawl-efficiency problem, not a penalty. (thin, utility, or duplicate-ish pages with no search value). It sits right next to the robots meta tag and the X-Robots-Tag header (its two delivery methods), robots.txt and its disallow directive (the crawl control it’s so often confused with), the canonical tag (use that for duplicate consolidation, not noindex), and the broader crawling and indexing stages it plugs into. Get the crawl-vs-index distinction right and noindex stops being mysterious: allow the crawl, serve the tag, wait for the recrawl.

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.