Languages

Noindex

Noindex keeps a page out of search results — but only if Google can crawl it. The two valid methods, the robots.txt trap, and how to verify it worked.

First published: Jun 23, 2026 · Last updated: Jul 28, 2026 · Advanced
demand #7 in Meta Tags#18 in On-Page#91 in Technical SEO#122 on the site
AIAuthored byAI v2

Noindex is the directive that keeps a page out of the index, so it won't appear in search results. There are two valid ways to set it: the robots meta tag (`<meta name="robots" content="noindex">`) and the `X-Robots-Tag: noindex` HTTP header (the only option for non-HTML files like PDFs). The single biggest mistake: a page blocked in robots.txt can't be noindexed, because Google never crawls it to see the rule — so to remove a page you have to allow crawling and serve noindex. Don't put noindex in robots.txt (unsupported since Sept 1, 2019); if noindex and canonical coexist, treat that as an intent check rather than an automatic error; and remember deindexing only happens after a recrawl.

TL;DR — noindex removes a page from the index via one of two valid methods: the robots meta tag (<meta name="robots" content="noindex">) or the X-Robots-Tag: noindex HTTP header (required for non-HTML files like PDFs). The load-bearing gotcha: a page blocked in robots.txt can’t be noindexed — Google never crawls it to see the rule, “the crawler will never see the noindex rule,” and a linked URL can stay indexed. So to remove a page, allow crawling and serve noindex. Don’t put noindex in robots.txt (unsupported since Sept 1, 2019), review noindex with a canonical pointing elsewhere as potentially conflicting, and know that deindexing only happens after a recrawl — Google’s own guidance says a low-priority page can take months. Per a 2017 Mueller comment (not documented policy), long-term noindex,follow tends to behave like noindex,nofollow once the page drops from the index. Verify in GSC under “URL marked ‘noindex’.”

What noindex is — index control, not crawl control

noindex is the primary index-control directive. Google’s own definition of the rule is one line: “Do not show this page, media, or resource in search results.” When it’s honored, the effect is total — “When Googlebot crawls that page and extracts the tag or header, Google will drop that page entirely from Google Search results, regardless of whether other sites link to it.” Evidence for this claim Google's noindex rule prevents the page, media, or resource from appearing in Google Search results after Google sees the rule. Scope: Google Search; noindex is not an access-control or privacy mechanism. Confidence: high · Verified: Google Search Central: Block Search indexing with noindex

Keep one distinction front of mind, because almost every noindex mistake comes from blurring it: noindex controls indexing; robots.txt controls crawling. They’re different stages of the pipeline. I put it this way in my Ahrefs guide on removing URLs: “Crawling is not the same thing as indexing. Even if Google is blocked from crawling pages, if there are any internal or external links to a page they can still index it.” That sentence is the whole reason the rest of this article exists.

Microsoft gives the same directive an additional Bing-specific consequence: content marked noindex is also excluded from Microsoft’s foundation-model training. The prerequisite still matters—Bingbot must be allowed to crawl and process the page-level directive. A robots.txt block plus noindex is therefore not proof that either deindexing or the training opt-out has been applied.

Evidence for this claim Microsoft says content marked noindex is not included in the Bing index and is not used to train its generative AI foundation models. Scope: Bing and Microsoft foundation-model use; Bingbot must be able to crawl and process the directive before the outcome can be inferred. Confidence: high · Verified: Bing Webmaster Blog: New controls for Bing Chat

The two valid delivery methods

There are exactly two, and noindex in robots.txt is not one of them (more on that below).

Method 1 — the robots meta tag. For an HTML page, place this in the <head>:

<meta name="robots" content="noindex">

Google’s instruction is verbatim: “To prevent all search engines that support the noindex rule from indexing a page on your site, place the following <meta> tag into the <head> section of your page.” The robots value targets all crawlers that support the rule; swap in googlebot to target only Google (<meta name="googlebot" content="noindex">).

Method 2 — the X-Robots-Tag HTTP header. Same directive, sent in the response header instead of the markup:

X-Robots-Tag: noindex

This is the only way to noindex non-HTML resources, because there’s no <head> to host a meta tag. Google: “A response header can be used for non-HTML resources, such as PDFs, video files, and image files.” And from the robots spec: you can use the X-Robots-Tag “for non-HTML files like image files where the usage of robots meta tags in HTML is not possible.” Evidence for this claim Google supports noindex in an HTML robots meta tag or an X-Robots-Tag HTTP response header. Scope: Google Search delivery methods; the HTTP header is applicable to non-HTML resources as well as HTML. Confidence: high · Verified: Google Search Central: Robots meta tag and X-Robots-Tag specifications

One placement note: put the meta tag in the <head> — that’s the standard, safest spot and what Google’s how-to shows. Google’s spec page does say it “doesn’t enforce placement of meta robots in the HTML head and will respect robots meta tags in the body section of an HTML document as well,” but don’t rely on that as your primary method; a stray <meta> tag some CMS injects into the <body> can noindex a page by accident just as easily as one you meant to add to the <head>.

Since the header is configured at the server level, it varies by stack. Two common examples for noindexing every PDF on a site:

Apache (.htaccess or vhost):

<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

Nginx (server/location block):

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex";
}

The #1 mistake — noindex + a robots.txt block

Noindex is crawl-then-obey: keep the URL fetchable long enough for the directive to be processed.

The same page contains a meta robots noindex directive. With crawling allowed, Google can fetch the page, see noindex, and remove the URL after processing. With crawling blocked in robots.txt, Google cannot see noindex and the linked URL may remain in results.

This is the failure mode I see most, so here’s the mechanism in full. The noindex tag lives on the page; Google has to fetch the page to read it. Google states the requirement directly:

“For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.”

Put even more bluntly: “We have to crawl your page in order to see <meta> tags and HTTP headers.” No crawl, no rule.

robots.txt is the most common way a page ends up uncrawlable, but Google’s wording covers more ground than that — it also says “the crawler can’t access the page,” which includes repeated server errors (5xx), timeouts, and an unintended authentication wall in front of the page. Any of those silently breaks noindex the same way a robots.txt block does.

So the instinct to “block it in robots.txt and noindex it, just to be safe” is exactly backwards — the block prevents the crawl, the crawl is what reveals the noindex, and the page can sit in the index indefinitely (often as a description-less URL). In Google Search Console this shows up as the “Indexed, though blocked by robots.txt” status — a page you blocked that got indexed anyway because something links to it.

The fix: unblock the page in robots.txt, keep noindex on it, and let Google recrawl. Only after the page has dropped from the index — if you then want to save the crawl entirely — is it safe to add a disallow.

Worked deployment example: the staging site that would not disappear

A redesign launches from staging.example.com. The staging templates already contain noindex, but the deployment checklist also adds:

User-agent: *
Disallow: /

That feels like two layers of protection. It is actually a trap if Google already discovered the staging URLs through a shared QA link, an old sitemap, a public ticket, or a link in copied production content. The disallow prevents the next crawl, so Google cannot confirm the noindex; the hostname can linger as thin, URL-only results.

The cleanup sequence is: remove the disallow, keep noindex on every staging response, confirm the live response is crawlable and exposes the directive, request recrawling for a representative sample, and monitor the hostname until it drops out. Then put the environment behind authentication. Authentication is the durable privacy control; noindex is only a search-index control.

noindex vs nofollow vs disallow

Three directives people constantly conflate. They operate at different stages:

  • noindex — index control. Page is crawled, kept out of results. Google’s definition: “Do not show this page, media, or resource in search results.”
  • nofollow — link control. Google: “Do not follow the links on this page.” It says nothing about indexing the page itself.
  • disallow (robots.txt) — crawl control. Stops the fetch entirely. It is not an index control — a disallowed URL can still be indexed if it’s linked.

There’s also none, which Google documents as “Equivalent to noindex, nofollow.” And when directives conflict, the spec is clear: “In the case of conflicting robots rules, the more restrictive rule applies.” (Full table on the Cheat Sheets tab.)

Treat noindex with rel=canonical as an intent check

Putting noindex and rel="canonical" on the same page is not automatically invalid. It does create a configuration worth reviewing: a canonical asks Google to consolidate signals, while noindex asks for this URL to be excluded. For choosing between duplicates, use the canonical tag — Google specifically advises against using noindex for it: “We don’t recommend using noindex to prevent selection of a canonical page within a single site, because it will completely block the page from Search.” Note the scope: Google’s caution is specifically about using noindex to pick which duplicate wins as canonical within your own site — it’s not a claim that noindex and canonical can never technically coexist on a page (a page you’re genuinely retiring can still carry a self-referencing canonical). A canonical pointing at a different URL deserves the strongest warning: confirm that exclusion and consolidation are both intended. Use canonical to consolidate duplicates; use noindex only when you genuinely want this page out of results.

noindex,follow vs noindex,nofollow — the slow decay

A common pattern is noindex,follow: keep the page out of results, but keep following its links so equity still flows through it (handy during a migration or while a page is temporarily out). Current official Google documentation doesn’t describe this decaying automatically — it explicitly allows combining noindex with other rules, including setting noindex,nofollow on purpose from day one. What I’m relying on for the “it fades over time” claim is a 2017 webmaster hangout, where John Mueller said a long-term noindex tends to end up treated like noindex,nofollow in practice: once Google decides the page really doesn’t belong in search and drops it completely, it also stops following that page’s links, because it’s stopped processing the page at all. That’s a practitioner observation from a video transcript, not a documented Google policy, so treat it as directional rather than guaranteed. Either way, the practical takeaway holds: noindex,follow is fine for a transitional period, but don’t lean on it as a permanent link-equity strategy — plan to fix the underlying links (or remove the page) instead.

How long does noindex take?

Not instantly. noindex only applies after Google recrawls and reprocesses the page — until then, the page can stay indexed even though the tag is live. Google doesn’t commit to a fixed window, and its own guidance leans toward “could be a while,” not “any day now”: “Depending on the importance of the page on the internet, it may take months for Googlebot to revisit a page.” A high-traffic, frequently-linked page might get recrawled in days; a low-value, rarely-linked one can sit for months. If you need a page out of results urgently, the GSC Removals tool is a stopgap (it hides the URL temporarily while the permanent noindex does its slower work). For genuinely gone pages, a 404/410 also drops them: as I wrote in my removal guide, “If you remove the page and serve either a 404 (not found) or 410 (gone) status code, then the page will be removed from the index shortly after the page is re-crawled.” Same theme everywhere — it happens on recrawl.

noindex in robots.txt is dead (since Sept 1, 2019)

You’ll still see people suggest a Noindex: line in robots.txt. Don’t. It was never an officially supported rule, and Google retired even its unofficial handling years ago. From the July 2019 Search Central announcement: “Since these rules were never documented by Google, naturally, their usage in relation to Googlebot is very low.” And the date: “we’re retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019.”

The same post named the supported alternatives, and noindex via the meta tag / header topped the list: “noindex in robots meta tags: Supported both in the HTTP response headers and in HTML, the noindex rule is the most effective way to remove URLs from the index when crawling is allowed.” (Also listed: 404/410 status codes, password protection, robots.txt disallow for crawl prevention, and the Search Console removal tool.)

How to verify noindex in Google Search Console

Two checks:

  • URL Inspection. Run the URL through Inspect, then Test live URL. It tells you whether the page is indexable and whether Google sees a noindex directive — the fastest way to confirm the tag is being read on the live page.
  • Page Indexing report. Noindexed pages are listed under the status “URL marked ‘noindex’” in the Not indexed section. Google’s help text: “When Google tried to index the page it encountered a ‘noindex’ directive and therefore did not index it.” If that’s a page you wanted indexed, that’s your bug — remove the directive.

One naming note for anyone searching old write-ups: the legacy Coverage report called this “Excluded by ‘noindex’ tag.” The current Page Indexing report uses “URL marked ‘noindex’” — same thing, newer label.

What noindex doesn’t guarantee

A few things people assume noindex buys them that it actually doesn’t:

  • Crawl-budget savings. Google still has to fetch the page to see the tag — noindex alone doesn’t reduce crawling. If you want that too, add disallow in robots.txt, but only after the page has already dropped from the index (see the mistake above for why doing it up front backfires).
  • Instant removal. Covered above — it happens on recrawl, with no fixed timetable, and Google itself says a lower-priority page can take months.
  • Duplicate consolidation. That’s what rel="canonical" is for; noindex just removes the page from Search, it doesn’t merge signals toward another URL.
  • Confidentiality. The page stays publicly requestable by anyone with the URL. If something actually needs to be private, that’s an authentication problem, not a search-directive problem.
  • Ranking recovery if you reverse it. Removing noindex doesn’t restore a page’s old rankings — Google has to recrawl, re-evaluate, and effectively re-earn its position from scratch.
  • Identical timing across search engines. Bing and other engines run their own crawl and recrawl schedules independently of Google’s.
  • Exclusion from every non-search use of your content. noindex blocks a page from Google Search as a whole — including Search’s own AI features (AI Overviews and similar draw on pages that are indexed and eligible to be shown, so a noindexed page is out of those too). What it does not do is control Google’s separate Google-Extended setting, which governs whether your content can be used to train or ground Google’s generative AI models outside of Search. Those are two different controls for two different jobs.

Where noindex fits with everything else

noindex is the lever you reach for when a page is in the index but shouldn’t be — the cure for one flavor of index bloat (thin, utility, or duplicate-ish pages with no search value). It sits right next to the robots meta tag and the X-Robots-Tag header (its two delivery methods), robots.txt and its disallow directive (the crawl control it’s so often confused with), the canonical tag (use that for duplicate consolidation, not noindex), and the broader crawling and indexing stages it plugs into. Get the crawl-vs-index distinction right and noindex stops being mysterious: allow the crawl, serve the tag, wait for the recrawl.

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin an expert quote first.