Why can a robots.txt-blocked URL remain indexed even when its HTML contains noindex?

The crawler cannot fetch the page to discover the noindex rule

Noindex only works on home pages

Robots.txt always overrides noindex by indexing the URL

Noindex is only supported in HTTP headers

Meta Robots Tag

TL;DR — The robots meta tag is a line of HTML you put in a page’s <head> to tell search engines how to handle that one page — most often <meta name="robots" content="noindex"> to keep it out of search. The catch that trips everyone up: Google has to be able to crawl the page to read the tag. If you also block the page in robots.txt, Google never sees the tag, and the page can stay in search. With no tag at all, the default is “index it and follow the links.”

What the robots meta tag is

The robots meta tag is a small HTML element that sits in the <head> of a page:

<meta name="robots" content="noindex">

It tells search engines how to treat this specific page — whether to show it in results, whether to follow the links on it, whether to show a snippet, and so on. The name="robots" part means “all search engines that read this tag.” You can swap in a crawler’s name, like name="googlebot", to talk to just one engine.

If there’s no robots meta tag on a page, the default is index, follow — show it in search and follow its links. So you only need the tag when you want to change that default. Evidence for this claim For Google, the default robots meta behavior is index, follow when no restrictive rule is present. Scope: Google-supported robots meta rules; other crawlers publish their own support and defaults. Confidence: high · Verified: 2026-07-14 Google Search Central: Robots meta tag specifications

The one thing to get right: don’t block the page you’re trying to noindex

This is the mistake I see most often. People want a page out of Google, so they do both things at once:

Add noindex to the page, and
Block the page in robots.txt.

That second step defeats the first. Blocking a URL in robots.txt tells Google “don’t even fetch this page.” So Google never downloads it, never reads the noindex, and the page can stay in the index. The fix is to leave the page crawlable and let Google read the noindex. Evidence for this claim Google can read and follow page-level robots rules only when it is allowed to access the page. Scope: Google-supported robots meta and X-Robots-Tag rules; robots.txt blocking can prevent rule discovery. Confidence: high · Verified: 2026-07-14 Google Search Central: Robots meta tag specifications

Think of it as two different jobs:

robots.txt controls crawling — whether bots fetch the page at all.
The robots meta tag controls indexing and serving — what happens once the page is fetched.

They’re not interchangeable, and you don’t want both on the same URL when your goal is to remove it from search.

Noindex is crawl-then-obey: keep the URL crawlable long enough for the search engine to fetch and process the directive. Source: Google Search Central

The same page contains a meta robots noindex directive. In the first path, crawling is allowed, so the crawler fetches the page, reads noindex, and can remove the URL from results after processing. In the second path, robots.txt blocks crawling, so the crawler cannot fetch the page or see noindex, and the URL may remain in results. Crawl control and index control are separate jobs.

The common directives

A few you’ll actually use:

noindex — keep this page out of search results.
nofollow — don’t follow the links on this page. (Different from putting rel="nofollow" on a single link — this one applies to every link on the page.)
none — shorthand for noindex, nofollow.
nosnippet — don’t show a text snippet for this page in results.

You can combine them with a comma: <meta name="robots" content="noindex, nofollow">.

A few honest gotchas

noindex doesn’t make a page private. The page is still public and crawlable — it just won’t show in search. For real privacy, use a login.
noindex doesn’t save crawl budget. Google still has to fetch the page to see the tag.
For a PDF or an image, you can’t add a <meta> tag — there’s no HTML <head>. That’s what the X-Robots-Tag HTTP header is for (more on that in the Advanced version).
There’s no fixed timeline for a noindexed page to drop out of Google. Google says it can take months for a lower-priority page to get recrawled and processed — don’t promise a client “2-4 weeks.”

Want the full directive list, how Google resolves conflicting rules, the googlebot-vs-robots edge case, and the X-Robots-Tag header in detail? Switch to the Advanced tab.

Test yourself: robots meta directives

Choose the control by outcome

Which robots control should I use?

Robots controls that fail their intended job

Combining noindex with a robots.txt block. The block prevents the crawler from seeing the removal directive.
Using disallow as a deindexing guarantee. It stops fetching, but a linked URL can still be indexed without page content.
Assuming a body-placed robots meta tag is silently ignored. Google says it will respect a robots meta tag in the body, so a template that injects it there still works for Google — but stick to <head> placement anyway, since it’s the only placement other crawlers and validators are guaranteed to expect.
Using a crawler-name token Google doesn’t read (e.g. name="bingbot") and assuming it also scopes the rule for Google. Google recognizes only googlebot and googlebot-news; any other name value is ignored by Google entirely.
Using an HTML meta tag for a PDF. Serve X-Robots-Tag in the HTTP response for non-HTML files.
Treating nofollow as noindex. Link handling does not remove the current page from results.
Publishing generic and crawler-specific tags without resolving the sum. Audit every robots, googlebot, and header directive together; the restrictive rule can survive in a second location.
Using robots rules for secrecy. Anyone can request a public URL or read robots.txt; confidential content needs authentication.

Deploy a noindex safely

Confirmed removal from search—not crawl reduction, canonical consolidation, or access control—is the intended outcome.
Used <meta name="robots" content="noindex"> for HTML or an X-Robots-Tag: noindex response header for a non-HTML resource.
Kept the URL crawlable and accessible to the target crawler.
Removed contradictory template, CMS, CDN, and crawler-specific directives.
Checked the rendered head and live response, not only the source template.
Tested both the canonical URL and meaningful variants or redirects.
Recorded the affected URL set and deployment time for later comparison.
Verified the rule in URL Inspection and monitored the Page Indexing report after recrawl.
Used authentication instead if the content must be private.

Inspect meta, headers, and crawl access

Replace the URL in this shell audit:

url="https://example.com/page/"
curl -sSI "$url" | grep -iE '^(HTTP/|x-robots-tag:|location:)'
curl -sS "$url" | grep -oiE '<meta[^>]+name=["'"'](robots|googlebot)["'"'][^>]*>'
curl -sS "https://example.com/robots.txt"

In DevTools Console, inventory every parsed robots tag rather than stopping at the first match:

console.table([...document.querySelectorAll('meta[name]')]
  .filter((meta) => /^(robots|googlebot|bingbot)$/i.test(meta.name))
  .map((meta) => ({ crawler: meta.name, content: meta.content })));

Headers are not visible in the DOM, so compare the console result with the actual response. Also follow redirects: a directive on an intermediate response does not prove the destination serves the same rule.

Patrick's relevant free tools

Google Index Checker — Check one URL’s observable indexability blockers, or reconcile sitemap, crawl, and supplied Search Console evidence across a URL set before verifying Google’s actual state in URL Inspection.
Google SERP Simulator — Interactive Google result preview: type a title, description, and URL, compare controlled-font desktop + mobile truncation guidance with simulated query bolding, copy the finished tags, and share a filled-in preview.
Meta Description Generator — Generate five meta-description options with AI, then compare their measured widths with approximate desktop and mobile SERP-preview guidance. Variants are scored for the keyword, a call to action, and length; you also get a matching title tag and a copy-ready tag block. Quick, paste-content, fetch-URL, and small bulk (≤10) modes. Falls back to editable templates when AI is off.

Test each layer separately

HTTP Header Checker — inspect status, redirects, and every live X-Robots-Tag header, including for PDFs.
Robots.txt Tester — check whether the target crawler is allowed to fetch the URL and therefore able to discover a robots directive.

Use both when diagnosing a stubborn indexed URL. A correct noindex response is not actionable if robots.txt prevents the fetch, and an allowed crawl does not prove the page actually serves noindex.

Prove the directive works after deployment

Test 1 — Live delivery

Hypothesis: Every intended URL serves one effective noindex rule.
Method: Sample the URL set, follow redirects, and inspect rendered meta plus response headers.
Pass condition: The final response is crawlable and exposes noindex to the intended crawler with no conflicting delivery path.
Fail condition: A redirect, CDN header, or crawler-specific tag changes it.
Next action: Fix the responsible layer and repeat the same sample.

Test 2 — Search-engine processing

Hypothesis: Google can crawl the page and has processed the removal rule.
Method: Run URL Inspection’s live test, then monitor the Page Indexing report after recrawl.
Pass condition: The live test detects noindex and the URL becomes not indexed for that reason.
Fail condition: The URL is blocked, the directive is absent, or the old index state persists without a new crawl.
Next action: Restore crawl access or request recrawl; do not add a disallow.

Measure intent, not a universal benchmark

Intended-noindex coverage

URLs serving the intended noindex ÷ URLs in the approved noindex set

Build the denominator from your own removal inventory, then compare crawls before and after deployment. The target is complete coverage of that approved set—not an industry percentage.

Unintended index-control conflicts

Track counts of intended-noindex URLs that are still indexed, robots.txt-blocked, or serving contradictory meta/header rules. Segment by template, CDN rule, and file type so one implementation fault does not hide inside a sitewide total.

Use Google Search Console’s Page Indexing report and URL Inspection as processing evidence, while recognizing that deindexing depends on recrawl. Preserve the pre-change count and deployment date; otherwise a falling total has no reliable baseline.

TL;DR — <meta name="robots" content="…"> in the <head> controls how a single page is indexed and served; with no tag the default is index, follow. It’s crawl-then-obey: “these settings can be read and followed only if crawlers are allowed to access the pages” — so a robots.txt-blocked URL is never fetched and its noindex is never seen (I have first-party data on the flip side of this). Conflicting rules resolve to the most restrictive; across a googlebot tag and the generic robots tag, Googlebot takes the sum of the negative rules. The tag is HTML-only — use the X-Robots-Tag header for non-HTML and at scale. Google reads only two crawler-named tokens — googlebot and googlebot-news — and ignores every other value, including other engines’ tokens like bingbot.

What it is, where it goes, and the default

The robots meta tag lets you, in Google’s words, “use a granular, page-specific approach to controlling how an individual HTML page should be indexed and served to users in Google Search results.” The conventional, portable place for it is the <head>:

<meta name="robots" content="noindex, nofollow">

Head placement is the authoring convention every engine expects, but Google is explicit that it isn’t a hard requirement for Google Search specifically: “Google Search doesn’t enforce placement of meta robots in the HTML head and will respect robots meta tags in the body section of an HTML document as well.” Treat that as tolerance for Google, not portable advice — still author it in the <head> so every crawler and validator that expects standard placement reads it correctly.

The name attribute is the audience, and this is where most guides overstate Google’s support. name="robots" addresses every crawler that reads the tag. Beyond that, Google supports exactly two crawler-named tokens, and ignores every other value: “Google supports two user agent tokens in the robots meta tag; other values are ignored: googlebot for all text results, and googlebot-news for news results.” A tag named name="bingbot" isn’t a documented Google control — Bing reads its own token on its own terms, but Google skips any name value it doesn’t recognize. Both the name and content attributes are case-insensitive to Google, and so are X-Robots-Tag header names and values.

When no robots meta tag is present, the default is index, follow (the all rule, which Google notes “has no effect if explicitly listed”). Evidence for this claim For Google, the default robots meta behavior is index, follow when no restrictive rule is present. Scope: Google-supported robots meta rules; other crawlers publish their own support and defaults. Confidence: high · Verified: 2026-07-14 Google Search Central: Robots meta tag specifications You only need the tag to change that default.

You combine rules two ways: comma-separated in one tag (noindex, nofollow) or as multiple <meta> tags. Google: you can “create a multi-rule instruction by combining robots meta tag rules with commas or by using multiple meta tags.”

The rule that breaks everything: Google must crawl the page to see the tag

This is the whole article. The robots meta tag is crawl-then-obey. Google has to fetch the page to read the tag — so anything that stops the fetch stops the tag from ever being applied. Straight from the spec:

“Keep in mind that these settings can be read and followed only if crawlers are allowed to access the pages that include these settings.”

And the consequence, spelled out:

“If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored.”

So the classic mistake — Disallow in robots.txt plus noindex on the same URL — silently defeats the noindex. Google never crawls the page, never sees the tag, and the URL can linger in the index (often as a bare, snippet-less result if something links to it). Google’s companion “Block Search Indexing” doc says the same thing in plainer language: for the noindex rule to work, the page “must not be blocked by a robots.txt file… If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results.”

I’ve watched the flip side of this mechanism happen with real data. In my experiment The Story of Blocking 2 High-Ranking Pages With Robots.txt, I deliberately blocked two of our ranking pages in robots.txt. Because Google could no longer crawl them, it couldn’t refresh anything about them — and the pages mostly kept ranking: “We lost a position here or there and all of the featured snippets for the pages.” My takeaway: “Accidentally blocking pages (that Google already ranks) from being crawled using robots.txt probably isn’t going to have much impact on your rankings, and they will likely still show in the search results.” That’s the same coin as the noindex problem — a blocked URL is frozen. Block ≠ remove. If you actually want a page gone, you need a crawlable noindex, which is the entire point of this tag.

This is the line I’ve drawn publicly on where each tool belongs. Asked whether Google should add noindex support to robots.txt, I said: “Google was clear they want robots.txt for crawl control only.” Crawling is robots.txt’s job; indexing is the meta tag’s (or the header’s). They don’t overlap, and the noindex directive in robots.txt was never officially supported — Google dropped parsing of it on September 1, 2019.

Every robots meta directive (the reference)

Google’s supported values, with the verbatim descriptions from the spec:

Indexing

all — “There are no restrictions for indexing or serving. This rule is the default value and has no effect if explicitly listed.”
noindex — “Do not show this page, media, or resource in search results.”
none — “Equivalent to noindex, nofollow.”
indexifembedded — “Google is allowed to index the content of a page if it’s embedded in another page through iframes or similar HTML tags, in spite of a noindex rule.” (The one directive that overrides a noindex, for embedded content.)

Links

nofollow — “Do not follow the links on this page.” This is page-level — different scope from a per-link rel="nofollow", which applies to one link.

Serving and snippets

nosnippet — “Do not show a text snippet or video preview in the search results for this page.” This scope is broader than the classic text snippet: Google says it “applies to all forms of search results (at Google: web search, Google Images, Discover, AI Overviews, AI Mode) and will also prevent the content from being used as a direct input for AI Overviews and AI Mode.”
max-snippet:[number] — “Use a maximum of [number] characters as a textual snippet for this search result.” Same broadened scope as nosnippet: it “applies to all forms of search results (such as Google web search, Google Images, Discover, Assistant, AI Overviews, AI Mode) and will also limit how much of the content may be used as a direct input for AI Overviews and AI Mode.” That’s a direct-input eligibility control for Google’s own AI search features — it is not a general AI-training opt-out. Keeping your content out of model training (e.g. Google-Extended) or out of Search’s separate generative-AI property-level control in Search Console are different systems with different scopes; don’t treat nosnippet/max-snippet as covering either.
max-image-preview:[setting] — “Set the maximum size of an image preview for this page in search results.” Settings: none, standard, or large (“A larger image preview, up to the width of the viewport, may be shown.”).
max-video-preview:[number] — “Use a maximum of [number] seconds as a video snippet for videos on this page in search results.”
notranslate — “Don’t offer translation of this page in search results.”
noimageindex — “Do not index images on this page.”
unavailable_after:[date/time] — “Do not show this page in search results after the specified date/time.”

Historical — no longer active Google controls

A few directives that still circulate in older guides are ones Google says it no longer uses. Don’t add these expecting them to do anything:

noarchive — “The noarchive rule is no longer used by Google Search to control whether a cached link is shown in search results, as the cached link feature no longer exists.”
nocache (a synonym some engines used for noarchive) — “The nocache rule isn’t used by Google Search.”
nositelinkssearchbox — “The nositelinkssearchbox rule is no longer used by Google Search to control whether the sitelink search box is shown for a given page, as the feature no longer exists.”

Paragraph-level (not in the meta tag)

There’s one sub-page control: the data-nosnippet attribute. Google: you can “designate textual parts of an HTML page not to be used as a snippet… on span, div, and section elements.” Everything in the meta tag is page-wide; nosnippet / data-nosnippet is how you keep one passage out of the snippet without touching the rest.

Combining directives and resolving conflicts

Two rules govern what happens when directives collide.

1. The more restrictive rule wins. “In the case of conflicting robots rules, the more restrictive rule applies. For example, if a page has both max-snippet:50 and nosnippet rules, the nosnippet rule will apply.” nosnippet is stricter than a 50-character cap, so nosnippet is what you get.

2. googlebot vs robots — the sum of the negative rules. This is the one most guides get wrong. A googlebot-named tag does not simply replace the generic robots tag — for the overlap, Googlebot takes the union of the restrictions. Google: “For situations where multiple crawlers are specified along with different rules, the search engine will use the sum of the negative rules.” Their worked example:

<meta name="robots" content="nofollow">
<meta name="googlebot" content="noindex">

“The page containing these meta tags will be interpreted as having a noindex, nofollow rule when crawled by Googlebot.” The nofollow from robots plus the noindex from googlebot add up to noindex, nofollow for Googlebot. (Where a googlebot tag and a robots tag set the same directive differently, the crawler-named one is the one that applies to that crawler.)

Meta robots tag vs X-Robots-Tag (the HTTP header)

The robots meta tag is HTML-only — it needs a <head>. For anything that isn’t HTML, you use the X-Robots-Tag, which delivers the exact same rule vocabulary in the HTTP response header. Google: “The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag.” And the reason it exists: “You can use the X-Robots-Tag for non-HTML files like image files where the usage of robots meta tags in HTML is not possible.”

So:

PDF, image, or other non-HTML? You can’t add a <meta> tag — use the header, e.g. X-Robots-Tag: noindex.
Whole directories or patterns? The header is set at the server/CDN level, so it scales to entire paths in one config rule.
One engine? The header can target a crawler too: X-Robots-Tag: googlebot: noindex, nofollow, and multiple X-Robots-Tag headers can be combined in one response.

Same rules, two delivery mechanisms: the meta tag for HTML pages, the header for everything else and for scale.

Which directives Bing and other engines support

Don’t assume the directive set is universal — it isn’t. Bing supports the core indexing and serving rules — noindex, nofollow, noarchive (with nocache as its synonym), and nosnippet — and it honors the X-Robots-Tag for non-HTML resources. But Bing does not support the none shorthand, so for cross-engine safety, write noindex, nofollow out explicitly rather than relying on none. The snippet- and preview-control family — max-snippet, max-image-preview, max-video-preview — along with noimageindex, notranslate, indexifembedded, and unavailable_after, is effectively Google-only. When in doubt, spell directives out and treat the max-* controls as Google features.

Common mistakes (and the fixes)

Disallow + noindex on the same URL. The noindex is never seen. Fix: leave the page crawlable; keep just the noindex.
noindex plus a rel=canonical pointing elsewhere. Conflicting signals — you’re telling Google both “drop this page” and “consolidate it into another one.” Pick one. (More in canonicalization.)
A staging-wide noindex shipped to production. Catastrophic, sitewide deindex. Check before launch.
A noindex injected only by client-side JavaScript. Google has to render the page to see it, and if the rendered HTML differs from what you expect, behavior differs too. Prefer the tag in the raw HTML or the header. (See rendering.)
Expecting Bing to honor Google-only directives (none, the max-* family).
Expecting a robots directive to do a job it doesn’t own. A noindex or nosnippet rule doesn’t by itself guarantee crawl-budget savings, secrecy, a ranking change, identical behavior across search engines, a specific removal timeline, or exclusion from every AI/search surface — each of those outcomes belongs to a different control (authentication for secrecy, robots.txt for crawl, each engine’s own docs for parity, Search Console or Google-Extended for AI-specific scopes). Google gives no fixed timeframe for when a noindexed page actually drops out — it depends on recrawl priority and “may take months” for a lower-importance page.

For where this sits in the bigger picture: robots.txt and crawling are the crawl-control side; noindex and indexing are the index-control side; and nosnippet / data-nosnippet, max-snippet, and max-image-preview are the serving controls you reach for when you want a page indexed but want to shape how it appears. The X-Robots-Tag is this same tag’s HTTP-header equivalent for non-HTML.

AI summary

A condensed take on the Advanced version:

The robots meta tag — <meta name="robots" content="…"> in the <head> — controls how a single HTML page is indexed and served. With no tag, the default is index, follow.
It’s crawl-then-obey. Google must crawl the page to read the tag: “these settings can be read and followed only if crawlers are allowed to access the pages.” So a robots.txt-blocked URL never gets its noindex seen — the classic mistake. Patrick’s blocked-pages experiment proves the mechanism: blocked pages stayed indexed and mostly kept ranking. Block ≠ remove.
robots.txt = crawl control; the meta tag (or X-Robots-Tag) = index/serve control. Not interchangeable. noindex in robots.txt was dropped Sept 1, 2019.
Directives: indexing (noindex, none, indexifembedded), links (nofollow, page-wide), serving (nosnippet, max-snippet, max-image-preview, max-video-preview, notranslate, noimageindex, unavailable_after), plus the paragraph-level data-nosnippet attribute. noarchive, nocache, and nositelinkssearchbox are historical — Google says it no longer uses them.
nosnippet/max-snippet also gate AI Overviews and AI Mode — Google says they control whether the page’s content can be used as a direct input for those features, not just the classic text snippet. That’s not a general AI-training opt-out; Google-Extended and Search’s separate generative-AI property control are different systems.
Conflicts resolve to the most restrictive (nosnippet beats max-snippet:50). Across a googlebot tag and the generic robots tag, Googlebot takes the sum of the negative rules — robots: nofollow + googlebot: noindex ⇒ noindex, nofollow. Google recognizes only the googlebot/googlebot-news name tokens; other values (like bingbot) are ignored by Google.
Meta tag is conventionally <head>-only (Google also tolerates body placement, but that’s Google-specific, not portable); the X-Robots-Tag carries the same rules in the HTTP header for PDFs, images, non-HTML, and whole directories.
Cross-engine: Bing supports noindex/nofollow/noarchive(nocache)/ nosnippet but not none; the max-* family is effectively Google-only — write directives out explicitly.
No guarantees: a robots directive alone doesn’t promise crawl-budget savings, secrecy, a ranking change, cross-engine parity, or a fixed removal timeline — Google gives no set timeframe for noindex to take effect.

Official documentation

Primary-source documentation from the search engines.

Google

Robots meta tag, data-nosnippet, and X-Robots-Tag specifications — the authoritative spec: every directive, combining rules, conflict resolution, the googlebot-vs-robots union, and the X-Robots-Tag header. Start here.
Block Search Indexing with noindex — how noindex works, the two delivery mechanisms (meta tag and header), and the crawl dependency in plain language.
Introduction to robots.txt — the crawl-control counterpart, so you don’t confuse the two jobs.
Crawling and Indexing — the hub for robots, sitemaps, canonicalization, and crawl controls.

Bing / Microsoft

Robots meta tags and attributes that Bing supports — Bing’s supported directives (confirm the exact list on the live page; it’s JavaScript-rendered).

Reference

MDN — <meta name="robots"> — cross-engine directive reference, including which engines use noarchive/nocache.

Quotes from the source

On-the-record statements from Google. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what the tag is and the default

“The robots meta tag lets you use a granular, page-specific approach to controlling how an individual HTML page should be indexed and served to users in Google Search results.” — Google Search Central docs. Jump to quote

Google — the crawl-then-obey dependency

“Keep in mind that these settings can be read and followed only if crawlers are allowed to access the pages that include these settings.” Jump to quote
“If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored.” Jump to quote
“For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.” — Google, “Block Search Indexing with noindex.” Jump to quote

Google — the directives (verbatim descriptions)

“Do not show this page, media, or resource in search results.” — noindex. Jump to quote
“Equivalent to noindex, nofollow.” — none. Jump to quote
“Do not show a text snippet or video preview in the search results for this page.” — nosnippet. Jump to quote
“Google is allowed to index the content of a page if it’s embedded in another page through iframes or similar HTML tags, in spite of a noindex rule.” — indexifembedded. Jump to quote
“A larger image preview, up to the width of the viewport, may be shown.” — max-image-preview:large. Jump to quote
“Do not index images on this page.” — noimageindex. Jump to quote
“You can designate textual parts of an HTML page not to be used as a snippet.” — data-nosnippet. Jump to quote

Google — combining and resolving conflicts

“You can create a multi-rule instruction by combining robots meta tag rules with commas or by using multiple meta tags.” Jump to quote
“In the case of conflicting robots rules, the more restrictive rule applies. For example, if a page has both max-snippet:50 and nosnippet rules, the nosnippet rule will apply.” Jump to quote
“For situations where multiple crawlers are specified along with different rules, the search engine will use the sum of the negative rules.” Jump to quote

Google — placement, crawler tokens, and case sensitivity

“Google Search doesn’t enforce placement of meta robots in the HTML head and will respect robots meta tags in the body section of an HTML document as well.” Jump to quote
“Google supports two user agent tokens in the robots meta tag; other values are ignored.” Jump to quote
“Both the name and the content attributes are case-insensitive.” Jump to quote

Google — noarchive and other historical directives

“The noarchive rule is no longer used by Google Search to control whether a cached link is shown in search results, as the cached link feature no longer exists.” Jump to quote
“The nocache rule isn’t used by Google Search.” Jump to quote

Google — nosnippet and max-snippet reach AI Overviews and AI Mode

“[nosnippet] applies to all forms of search results (at Google: web search, Google Images, Discover, AI Overviews, AI Mode) and will also prevent the content from being used as a direct input for AI Overviews and AI Mode.” Jump to quote
“[max-snippet] applies to all forms of search results (such as Google web search, Google Images, Discover, Assistant, AI Overviews, AI Mode) and will also limit how much of the content may be used as a direct input for AI Overviews and AI Mode.” Jump to quote

Google — no fixed timeframe for noindex to take effect

“If a page is still appearing in results, it’s probably because we haven’t crawled the page since you added the noindex rule. Depending on the importance of the page on the internet, it may take months for Googlebot to revisit a page.” — Google, “Block Search Indexing with noindex.” Jump to quote

Google — X-Robots-Tag

“The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag.” Jump to quote
“You can use the X-Robots-Tag for non-HTML files like image files where the usage of robots meta tags in HTML is not possible.” Jump to quote

Patrick Stox — robots.txt is for crawl control only

“Google was clear they want robots.txt for crawl control only. The biggest downside will probably be all the people who accidentally take their entire site out of the index.” — Patrick Stox, on whether Google should add noindex to robots.txt, in Search Engine Land. Jump to quote

Patrick Stox — blocking ranking pages with robots.txt (the crawl-vs-index proof)

“Accidentally blocking pages (that Google already ranks) from being crawled using robots.txt probably isn’t going to have much impact on your rankings, and they will likely still show in the search results.” Jump to quote
“We lost a position here or there and all of the featured snippets for the pages.” Jump to quote

Note on relayed sources: John Mueller’s “use noindex, not robots.txt” line is reported by Search Engine Roundtable, whose page resists automated fetching; and Bing’s supported-directive list (Bing Webmaster Tools help) is JavaScript-rendered. Those points are paraphrased here without quotation marks and should be confirmed against the live pages before being treated as verbatim.

Every robots meta directive — cheat sheet

The full set of Google-supported content values, what each does, and the default.

Directive	What it does	Default?
`all`	No restrictions on indexing or serving — the implicit default	Yes (when no tag)
`index`	Allow the page in search results (the default; rarely written)	Implicit
`noindex`	Keep this page/media/resource out of search results	No
`follow`	Follow the links on this page (the default; rarely written)	Implicit
`nofollow`	Don’t follow any links on this page (page-wide)	No
`none`	Shorthand for `noindex, nofollow` (not supported by Bing)	No
`nosnippet`	Don’t show a text snippet or video preview; also blocks the page as a direct input for AI Overviews/AI Mode	No
`max-snippet:[n]`	Cap the text snippet at `[n]` characters (`0` = none, `-1` = no limit); also caps how much can feed AI Overviews/AI Mode directly	No
`max-image-preview:[setting]`	Cap image-preview size: `none` / `standard` / `large`	No
`max-video-preview:[n]`	Cap video preview at `[n]` seconds (`0` = none, `-1` = no limit)	No
`notranslate`	Don’t offer a translation of this page in results	No
`noimageindex`	Don’t index the images on this page	No
`unavailable_after:[date/time]`	Drop the page from results after the given date/time	No
`indexifembedded`	Allow indexing of content embedded via iframe even with `noindex`	No

Historical — Google no longer uses these:

Directive	Status
`noarchive`	No longer used — the cached-link feature it controlled no longer exists
`nocache`	Not used by Google Search (some engines treated it as a `noarchive` synonym)
`nositelinkssearchbox`	No longer used — the sitelinks search box feature it controlled no longer exists

Paragraph-level (an HTML attribute, not a content value):

Attribute	What it does	Where
`data-nosnippet`	Keep a specific passage out of the snippet	On `span`, `div`, `section`

The syntax

<!-- one tag, comma-separated -->
<meta name="robots" content="noindex, nofollow">

<!-- target one engine -->
<meta name="googlebot" content="noindex">

<!-- the HTTP-header equivalent, for non-HTML / at scale -->
X-Robots-Tag: noindex
X-Robots-Tag: googlebot: noindex, nofollow

Fast facts

No tag at all → default index, follow.
none = noindex, nofollow — but Bing doesn’t support none; write it out.
max-*, noimageindex, notranslate, indexifembedded, unavailable_after are effectively Google-only.
The tag is HTML-only; the X-Robots-Tag header carries the same rules for PDFs, images, and whole directories.
Google reads exactly two crawler-name tokens — googlebot and googlebot-news — and ignores everything else, including other engines’ tokens like bingbot.
Head placement is the portable convention, but Google specifically also respects a robots meta tag placed in the <body>.
name/content and X-Robots-Tag values are case-insensitive to Google.

The mental models

1. Crawl-then-obey — the tag only works if the page is fetchable. Google has to crawl the page to read the tag. Anything that blocks the fetch (robots.txt disallow, auth, a server error) means the tag is never seen. So before you trust a noindex, confirm the URL is crawlable. The corollary: never Disallow a URL you’re trying to noindex.

2. Three tools, three jobs — don’t mix them up.

robots.txt = crawl control (whether bots fetch the page).
Robots meta tag / X-Robots-Tag = index & serve control (what happens once it’s fetched).
Authentication = secrecy (a noindex page is still public). Match the job to the tool. The most common failure is using robots.txt to try to deindex — that’s the meta tag’s job.

3. The decision rule for removing a page. Want it out of search? Leave it crawlable and add noindex — and don’t also block it in robots.txt, and don’t also canonical it to a different URL. Want bots to skip a space entirely (and you don’t care about indexing)? robots.txt disallow. The two are not interchangeable.

4. Conflict resolution — most restrictive wins. When rules collide, the stricter one applies (nosnippet beats max-snippet:50). Don’t try to out-clever this with combinations; assume the tightest rule is the one that takes effect.

5. googlebot vs robots — sum of the negatives, not override. For the overlap, Googlebot adds up the restrictions from the generic robots tag and the googlebot-named tag rather than picking one. robots: nofollow + googlebot: noindex ⇒ Googlebot gets noindex, nofollow. (A crawler-named tag does take precedence over the generic tag where they set the same directive differently — but the union is the rule to remember.)

6. HTML page → meta tag; everything else → header. If it has a <head>, use the <meta> tag. If it’s a PDF, an image, any non-HTML file, or you need to cover a whole directory, use the X-Robots-Tag header — same rule vocabulary, different delivery.

Robots Meta Tag

The robots meta tag is an HTML element in a page's head — <meta name="robots" content="noindex"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen.

Robots Meta Tag

The robots meta tag is an HTML element conventionally placed in the <head> of a page — <meta name="robots" content="noindex"> — that tells search engines how to index and serve that specific page. Google also tolerates the tag appearing in the page body, but that’s Google-specific behavior, not portable advice; keep it in <head>. The name can be robots (every crawler that reads the tag), or — for Google specifically — one of exactly two crawler-named tokens, googlebot or googlebot-news; Google ignores any other name value, including other engines’ tokens like bingbot. Its HTTP-header twin, the X-Robots-Tag, carries the exact same rules (case-insensitively) in the response header, so it works for non-HTML files (PDFs, images) where you can’t add a <head> tag.

The single most important thing about it is that it’s crawl-then-obey. Google has to be able to crawl the page to see the tag — so if the URL is blocked in robots.txt, Google never fetches it, never sees the noindex, and the page can stay indexed (Google gives no fixed timeframe for a noindexed page to drop out even when it is seen — it can take months on lower-priority pages). As Google puts it, “these settings can be read and followed only if crawlers are allowed to access the pages that include these settings.” That makes robots.txt and the robots meta tag two different jobs: robots.txt controls crawling, the meta tag controls indexing and serving. With no tag present, the default is index, follow.

The supported directives cover indexing (noindex, none, indexifembedded), link handling (nofollow), and serving and snippets (nosnippet, max-snippet, max-image-preview, max-video-preview, notranslate, noimageindex, unavailable_after) — nosnippet and max-snippet also gate whether the page’s content can be used as a direct input for AI Overviews and AI Mode, not just the classic text snippet. noarchive, nocache, and nositelinkssearchbox are directives Google says it no longer uses. When rules conflict, the more restrictive one wins — and for a crawler-named tag against the generic robots tag, Googlebot takes the sum of the negative rules, not a simple override.

Build-time retrieval analysis plus live signals for this exact article. The automatic chunk report includes a deterministic readiness score and is ready without a model download.

How this article chunks build baseline · local controls

A structure-aware retrieval preview using the shared Chunk Tester engine. The default lens is shown immediately; switch scopes to see what happens when every authored lens is treated as one source document.

Target size 300 tokens Overlap 15% Retrieval query

71readiness score / 100

14chunks

~308median tokens

86%standalone

6flagged chunks

Advanced lens only — the version readers see when this page opens.

Baseline generated at build time with a 300-token target and 15% overlap. Controls re-run the same deterministic engine in this browser.

Matching retrieval chunks

Ranked lexically for the page title, Trends, and Ngram terms. Enter a short query to test a different retrieval intent.

chunk 11 ~400 tok Combining directives and resolving conflicts

**2. `googlebot` vs `robots` — the sum of the negative rules.** This is the one most guides get wrong. A `googlebot`-named tag does **not** simply replace the generic `robots` tag — for the overlap, Googlebot takes the *…

chunk 1 ~285 tok Default visible lens

Meta Robots Tag **TL;DR —** ` ` in the ` ` controls how a single page is **indexed and served**; with no tag the default is `index, follow`. It's **crawl-then-obey**: "these settings can be read and followed only if craw…

chunk 2 ~310 tok What it is, where it goes, and the default

chunk 3 ~447 tok What it is, where it goes, and the default

The `name` attribute is the audience, and this is where most guides overstate Google's support. `name="robots"` addresses every crawler that reads the tag. Beyond that, **Google supports exactly two crawler-named tokens,…

chunk 6 ~508 tok The rule that breaks everything: Google must crawl the page to see the tag

I've watched the flip side of this mechanism happen with real data. In my experiment *The Story of Blocking 2 High-Ranking Pages With Robots.txt*, I deliberately blocked two of our ranking pages in `robots.txt`. Because …

Flagged retrieval chunks

chunk 3 ~447 tok What it is, where it goes, and the default

~447 tokens — over 1.3× the 300-token target; a retriever may truncate it.

The `name` attribute is the audience, and this is where most guides overstate Google's support. `name="robots"` addresses every crawler that reads the tag. Beyond that, **Google supports exactly two crawler-named tokens,…

chunk 4 ~200 tok The rule that breaks everything: Google must crawl the page to see the tag

Opens with "so" — refers back to text that may be in another chunk, so it doesn't stand alone.

*"If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored."* So the classic mistake — `Disallow` in **robots…

chunk 6 ~508 tok The rule that breaks everything: Google must crawl the page to see the tag

~508 tokens — over 1.3× the 300-token target; a retriever may truncate it.
Opens with "this" — refers back to text that may be in another chunk, so it doesn't stand alone.

I've watched the flip side of this mechanism happen with real data. In my experiment *The Story of Blocking 2 High-Ranking Pages With Robots.txt*, I deliberately blocked two of our ranking pages in `robots.txt`. Because …

chunk 8 ~306 tok Every robots meta directive (the reference)

A list is split across a chunk boundary — half its rows land in a different chunk.

• **`max-snippet:[number]`** — *"Use a maximum of [number] characters as a textual snippet for this search result."* Same broadened scope as `nosnippet`: it *"applies to all forms of search results (such as Google web se…

chunk 9 ~264 tok Every robots meta directive (the reference)

A list is split across a chunk boundary — half its rows land in a different chunk.

video snippet for videos on this page in search results."* • **`notranslate`** — *"Don't offer translation of this page in search results."* • **`noimageindex`** — *"Do not index images on this page."* • **`unavailable_a…

Showing 5 of 6 flagged chunks.

Methodology and limits

The readiness score starts with the percentage of standalone chunks and subtracts up to 35 points for the share carrying structural warnings. Structure-aware Markdown blocks use the selected 100–800-token target and 0–40% overlap. Tokens are estimated at roughly four English characters each. “Standalone” means a chunk has heading context and does not open with a context-dependent continuation.

Matching uses the shared lexical BM25 and term-cosine ranker. It is not an embedding or a replica of Google, an answer engine, or any vendor retriever. These chunk controls keep the article text in this browser and make no network request. The build-time baseline remains the reproducible report for this page.

Test your own content in Chunk Tester →

77readiness score / 100

35chunks

~324median tokens

89%standalone

12flagged chunks

All 15 authored lenses in this article source, with lens boundaries retained as headings.

Baseline generated at build time with a 300-token target and 15% overlap. Controls re-run the same deterministic engine in this browser.

Matching retrieval chunks

Ranked lexically for the page title, Trends, and Ngram terms. Enter a short query to test a different retrieval intent.

chunk 19 ~400 tok Combining directives and resolving conflicts

**2. `googlebot` vs `robots` — the sum of the negative rules.** This is the one most guides get wrong. A `googlebot`-named tag does **not** simply replace the generic `robots` tag — for the overlap, Googlebot takes the *…

chunk 1 ~324 tok Whole article

Meta Robots Tag Beginner lens **TL;DR —** The robots meta tag is a line of HTML you put in a page's ` ` to tell search engines how to handle *that one page* — most often ` ` to keep it out of search. The catch that trips…

chunk 6 ~312 tok Inspect meta, headers, and crawl access

In DevTools Console, inventory every parsed robots tag rather than stopping at the first match: console.table([...document.querySelectorAll('meta[name]')] .filter((meta) => /^(robots|googlebot|bingbot)$/i.test(meta.name)…

chunk 9 ~401 tok Advanced lens

**TL;DR —** ` ` in the ` ` controls how a single page is **indexed and served**; with no tag the default is `index, follow`. It's **crawl-then-obey**: "these settings can be read and followed only if crawlers are allowed…

chunk 25 ~338 tok AI summary

• **No guarantees:** a robots directive alone doesn't promise crawl-budget savings, secrecy, a ranking change, cross-engine parity, or a fixed removal timeline — Google gives no set timeframe for `noindex` to take effect…

Flagged retrieval chunks

chunk 9 ~401 tok Advanced lens

~401 tokens — over 1.3× the 300-token target; a retriever may truncate it.

**TL;DR —** ` ` in the ` ` controls how a single page is **indexed and served**; with no tag the default is `index, follow`. It's **crawl-then-obey**: "these settings can be read and followed only if crawlers are allowed…

chunk 10 ~406 tok What it is, where it goes, and the default

~406 tokens — over 1.3× the 300-token target; a retriever may truncate it.

Head placement is the authoring convention every engine expects, but Google is explicit that it isn't a hard requirement for Google Search specifically: *"Google Search doesn't enforce placement of meta robots in the HTM…

chunk 11 ~220 tok What it is, where it goes, and the default

Opens with "this" — refers back to text that may be in another chunk, so it doesn't stand alone.

You combine rules two ways: comma-separated in one tag (`noindex, nofollow`) or as multiple ` ` tags. Google: you can *"create a multi-rule instruction by combining robots `meta` tag rules with commas or by using multipl…

chunk 12 ~200 tok The rule that breaks everything: Google must crawl the page to see the tag

Opens with "so" — refers back to text that may be in another chunk, so it doesn't stand alone.

*"If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored."* So the classic mistake — `Disallow` in **robots…

chunk 14 ~508 tok The rule that breaks everything: Google must crawl the page to see the tag

~508 tokens — over 1.3× the 300-token target; a retriever may truncate it.
Opens with "this" — refers back to text that may be in another chunk, so it doesn't stand alone.

I've watched the flip side of this mechanism happen with real data. In my experiment *The Story of Blocking 2 High-Ranking Pages With Robots.txt*, I deliberately blocked two of our ranking pages in `robots.txt`. Because …

Showing 5 of 12 flagged chunks.

Methodology and limits

The readiness score starts with the percentage of standalone chunks and subtracts up to 35 points for the share carrying structural warnings. Structure-aware Markdown blocks use the selected 100–800-token target and 0–40% overlap. Tokens are estimated at roughly four English characters each. “Standalone” means a chunk has heading context and does not open with a context-dependent continuation.

Matching uses the shared lexical BM25 and term-cosine ranker. It is not an embedding or a replica of Google, an answer engine, or any vendor retriever. These chunk controls keep the article text in this browser and make no network request. The build-time baseline remains the reproducible report for this page.

Test your own content in Chunk Tester →

Search Console

Loading…

GA4 traffic (28d)

Loading…

Cloudflare traffic (7d)

Loading…

CrUX field data (28d, phone)

Loading…

Google NLP entities

Loading…

Changelog

Updated Jul 18, 2026.

Editorial summary and recorded change details.

Summary

Corrected the crawler-token, placement, and directive-lifecycle claims against Google's live robots-meta-tag docs, and added the AI Overviews/AI Mode scope of nosnippet and max-snippet.

Change details

Corrected the crawler-name token claim: Google recognizes only googlebot and googlebot-news in the meta name attribute and ignores all other values, including other engines' tokens like bingbot (previously implied bingbot was also a Google-read token).
Added Google's explicit tolerance for robots meta tags placed in the page body, while keeping head placement as the portable recommendation (previously implied head-only).
Reclassified noarchive, nocache, and nositelinkssearchbox as historical directives Google no longer uses, instead of listing noarchive as a current serving control.
Added that nosnippet and max-snippet also govern whether a page's content can be used as a direct input for AI Overviews and AI Mode, distinct from the separate Google-Extended and Search generative-AI property-level controls.
Added that name/content attributes and X-Robots-Tag values are case-insensitive to Google, and that Google gives no fixed timeframe for a noindex to take effect.

Full comparison unavailable — no prior snapshot was archived for this revision.

Meta Robots Tag

What the robots meta tag is

The one thing to get right: don’t block the page you’re trying to noindex

The common directives

A few honest gotchas

Test yourself: robots meta directives

Choose the control by outcome

Robots controls that fail their intended job

Deploy a noindex safely

Inspect meta, headers, and crawl access

Patrick's relevant free tools

Test each layer separately

Prove the directive works after deployment

Measure intent, not a universal benchmark

What it is, where it goes, and the default

The rule that breaks everything: Google must crawl the page to see the tag

Every robots meta directive (the reference)

Combining directives and resolving conflicts

Meta robots tag vs X-Robots-Tag (the HTTP header)

Which directives Bing and other engines support

Common mistakes (and the fixes)

AI summary

Official documentation

Quotes from the source

Every robots meta directive — cheat sheet

The mental models

Robots Meta Tag

Robots Meta Tag

How this article chunks build baseline · local controls

Matching retrieval chunks

Flagged retrieval chunks

Matching retrieval chunks

Flagged retrieval chunks

Google Trends Robots meta tag · Noindex

Books Ngram robots meta tag · meta robots

Changelog

Updated Jul 18, 2026.

Summary

Change details

People also ask