Meta Robots Tag

The robots meta tag controls how a page is indexed and served — every directive, the crawl-then-obey rule, conflict resolution, and meta tag vs X-Robots-Tag.

First published: Jun 23, 2026 · Last updated: Jul 18, 2026 · Advanced
demand #4 in Meta Tags#9 in On-Page#53 in Technical SEO#76 on the site
AIAuthored byAI v2
1 evidence signal on this page

The robots meta tag — <meta name="robots" content="noindex"> in the <head> — tells search engines how to index and serve a single page. The rule that breaks everything: it's crawl-then-obey, so a page blocked in robots.txt is never fetched and its noindex is never seen. With no tag, the default is index, follow. Conflicting rules resolve to the most restrictive; for a googlebot-named tag against the generic robots tag, Googlebot takes the sum of the negative rules. The tag is HTML-only — use the X-Robots-Tag header for PDFs, images, and other non-HTML.

TL;DR — <meta name="robots" content="…"> in the <head> controls how a single page is indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. and served; with no tag the default is index, follow. It’s crawl-then-obey: “these settings can be read and followed only if crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. are allowed to access the pages” — so a robots.txt-blocked URL is never fetched and its noindex is never seen (I have first-party data on the flip side of this). Conflicting rules resolve to the most restrictive; across a googlebot tag and the generic robots tag, GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. takes the sum of the negative rules. The tag is HTML-only — use the X-Robots-TagThe X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos. header for non-HTML and at scale. Google reads only two crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-named tokensA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. — googlebot and googlebot-news — and ignores every other value, including other engines’ tokens like bingbot.

What it is, where it goes, and the default

The robots meta tagThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen. lets you, in Google’s words, “use a granular, page-specific approach to controlling how an individual HTML page should be indexed and served to users in Google Search results.” The conventional, portable place for it is the <head>:

<meta name="robots" content="noindex, nofollow">

Head placement is the authoring convention every engine expects, but Google is explicit that it isn’t a hard requirement for Google Search specifically: “Google Search doesn’t enforce placement of meta robots in the HTML head and will respect robots meta tagsMeta tags are HTML elements in a page's head that pass metadata about the page to search engines and browsers. For SEO only a few matter — the title element, the meta description, and the robots meta tag — while meta keywords and most others are ignored. in the body section of an HTML document as well.” Treat that as tolerance for Google, not portable advice — still author it in the <head> so every crawler and validator that expects standard placement reads it correctly.

The name attribute is the audience, and this is where most guides overstate Google’s support. name="robots" addresses every crawler that reads the tag. Beyond that, Google supports exactly two crawler-named tokens, and ignores every other value: “Google supports two user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. tokens in the robots meta tag; other values are ignored: googlebot for all text results, and googlebot-news for news results.” A tag named name="bingbot" isn’t a documented Google control — Bing reads its own token on its own terms, but Google skips any name value it doesn’t recognize. Both the name and content attributes are case-insensitive to Google, and so are X-Robots-Tag header names and values.

When no robots meta tag is present, the default is index, follow (the all rule, which Google notes “has no effect if explicitly listed”). Evidence for this claim For Google, the default robots meta behavior is index, follow when no restrictive rule is present. Scope: Google-supported robots meta rules; other crawlers publish their own support and defaults. Confidence: high · Verified: Google Search Central: Robots meta tag specifications You only need the tag to change that default.

You combine rules two ways: comma-separated in one tag (noindex, nofollow) or as multiple <meta> tags. Google: you can “create a multi-rule instruction by combining robots meta tag rules with commas or by using multiple meta tags.”

The rule that breaks everything: Google must crawl the page to see the tag

This is the whole article. The robots meta tag is crawl-then-obey. Google has to fetch the page to read the tag — so anything that stops the fetch stops the tag from ever being applied. Straight from the spec:

“Keep in mind that these settings can be read and followed only if crawlers are allowed to access the pages that include these settings.”

Evidence for this claim Google can read and follow page-level robots rules only when it is allowed to access the page. Scope: Google-supported robots meta and X-Robots-Tag rules; robots.txt blocking can prevent rule discovery. Confidence: high · Verified: Google Search Central: Robots meta tag specifications

And the consequence, spelled out:

“If a page is disallowed from crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. through the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file, then any information about indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. or serving rules will not be found and will therefore be ignored.”

So the classic mistake — Disallow in robots.txt plus noindex on the same URL — silently defeats the noindex. Google never crawls the page, never sees the tag, and the URL can linger in the index (often as a bare, snippet-less result if something links to it). Google’s companion “Block Search Indexing” doc says the same thing in plainer language: for the noindex rule to work, the page “must not be blocked by a robots.txt file… If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results.”

TIP Check both places a noindex can hide

To inspect the final page response for HTML robots directives and an X-Robots-Tag before you wait for recrawling, run the On-Page SEO Checker Free

  1. Run the public final URL after deployment so redirects do not hide the directive you actually serve.
  2. Review the raw HTML robots meta value and the final-response X-Robots-Tag; either surface can contribute noindex.
  3. Compare an affected URL with a known-good control, then use Search Console separately to confirm Google’s observed indexing state after recrawl.

I’ve watched the flip side of this mechanism happen with real data. In my experiment The Story of Blocking 2 High-Ranking Pages With Robots.txt, I deliberately blocked two of our ranking pages in robots.txt. Because Google could no longer crawl them, it couldn’t refresh anything about them — and the pages mostly kept ranking: “We lost a position here or there and all of the featured snippets for the pages.” My takeaway: “Accidentally blocking pages (that Google already ranks) from being crawled using robots.txt probably isn’t going to have much impact on your rankings, and they will likely still show in the search results.” That’s the same coin as the noindex problem — a blocked URL is frozen. Block ≠ remove. If you actually want a page gone, you need a crawlable noindex, which is the entire point of this tag.

This is the line I’ve drawn publicly on where each tool belongs. Asked whether Google should add noindex support to robots.txt, I said: “Google was clear they want robots.txt for crawl control only.” CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is robots.txt’s job; indexing is the meta tag’s (or the header’s). They don’t overlap, and the noindex directive in robots.txt was never officially supported — Google dropped parsing of it on September 1, 2019.

Every robots meta directive (the reference)

Google’s supported values, with the verbatim descriptions from the spec:

Indexing

  • all — “There are no restrictions for indexing or serving. This rule is the default value and has no effect if explicitly listed.”
  • noindex — “Do not show this page, media, or resource in search results.”
  • none — “Equivalent to noindex, nofollow.”
  • indexifembedded — “Google is allowed to index the content of a page if it’s embedded in another page through iframesHTML element that displays one webpage inside another — how embeds work. or similar HTML tags, in spite of a noindex rule.” (The one directive that overrides a noindex, for embedded content.)

Links

  • nofollow — “Do not follow the links on this page.” This is page-level — different scope from a per-link rel="nofollow", which applies to one link.

Serving and snippets

  • nosnippet — “Do not show a text snippet or video preview in the search results for this page.” This scope is broader than the classic text snippet: Google says it “applies to all forms of search results (at Google: web search, Google Images, Discover, AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index., AI Mode) and will also prevent the content from being used as a direct input for AI Overviews and AI Mode.”
  • max-snippet:[number] — “Use a maximum of [number] characters as a textual snippet for this search result.” Same broadened scope as nosnippet: it “applies to all forms of search results (such as Google web search, Google Images, Discover, Assistant, AI Overviews, AI Mode) and will also limit how much of the content may be used as a direct input for AI Overviews and AI Mode.” That’s a direct-input eligibility control for Google’s own AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. features — it is not a general AI-training opt-out. Keeping your content out of model training (e.g. Google-Extended) or out of Search’s separate generative-AI property-level control in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. are different systems with different scopes; don’t treat nosnippet/max-snippet as covering either.
  • max-image-preview:[setting] — “Set the maximum size of an image preview for this page in search results.” Settings: none, standard, or large (“A larger image preview, up to the width of the viewport, may be shown.”).
  • max-video-preview:[number] — “Use a maximum of [number] seconds as a video snippet for videos on this page in search results.”
  • notranslate — “Don’t offer translation of this page in search results.”
  • noimageindex — “Do not index images on this page.”
  • unavailable_after:[date/time] — “Do not show this page in search results after the specified date/time.”

Historical — no longer active Google controls

A few directives that still circulate in older guides are ones Google says it no longer uses. Don’t add these expecting them to do anything:

  • noarchive — “The noarchive rule is no longer used by Google Search to control whether a cached link is shown in search results, as the cached link feature no longer exists.”
  • nocache (a synonym some engines used for noarchive) — “The nocache rule isn’t used by Google Search.”
  • nositelinkssearchbox — “The nositelinkssearchbox rule is no longer used by Google Search to control whether the sitelink search box is shown for a given page, as the feature no longer exists.”

Paragraph-level (not in the meta tag)

There’s one sub-page control: the data-nosnippet attribute. Google: you can “designate textual parts of an HTML page not to be used as a snippet… on span, div, and section elements.” Everything in the meta tag is page-wide; nosnippet / data-nosnippet is how you keep one passage out of the snippet without touching the rest.

Combining directives and resolving conflicts

Two rules govern what happens when directives collide.

1. The more restrictive rule wins. “In the case of conflicting robots rules, the more restrictive rule applies. For example, if a page has both max-snippet:50 and nosnippet rules, the nosnippet rule will apply.” nosnippet is stricter than a 50-character cap, so nosnippet is what you get.

2. googlebot vs robots — the sum of the negative rules. This is the one most guides get wrong. A googlebot-named tag does not simply replace the generic robots tag — for the overlap, GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. takes the union of the restrictions. Google: “For situations where multiple crawlers are specified along with different rules, the search engine will use the sum of the negative rules.” Their worked example:

<meta name="robots" content="nofollow">
<meta name="googlebot" content="noindex">

“The page containing these meta tagsMeta tags are HTML elements in a page's head that pass metadata about the page to search engines and browsers. For SEO only a few matter — the title element, the meta description, and the robots meta tag — while meta keywords and most others are ignored. will be interpreted as having a noindex, nofollow rule when crawled by Googlebot.” The nofollow from robots plus the noindex from googlebot add up to noindex, nofollow for Googlebot. (Where a googlebot tag and a robots tag set the same directive differently, the crawler-named one is the one that applies to that crawler.)

Meta robots tag vs X-Robots-Tag (the HTTP header)

The robots meta tag is HTML-only — it needs a <head>. For anything that isn’t HTML, you use the X-Robots-TagThe X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos., which delivers the exact same rule vocabulary in the HTTP response header. Google: “The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag.” And the reason it exists: “You can use the X-Robots-Tag for non-HTML files like image files where the usage of robots meta tags in HTML is not possible.”

So:

  • PDF, image, or other non-HTML? You can’t add a <meta> tag — use the header, e.g. X-Robots-Tag: noindex.
  • Whole directories or patterns? The header is set at the server/CDN level, so it scales to entire paths in one config rule.
  • One engine? The header can target a crawler too: X-Robots-Tag: googlebot: noindex, nofollow, and multiple X-Robots-Tag headers can be combined in one response.

Same rules, two delivery mechanisms: the meta tag for HTML pages, the header for everything else and for scale.

Which directives Bing and other engines support

Don’t assume the directive set is universal — it isn’t. Bing supports the core indexing and serving rules — noindex, nofollow, noarchive (with nocache as its synonym), and nosnippet — and it honors the X-Robots-Tag for non-HTML resources. But Bing does not support the none shorthand, so for cross-engine safety, write noindex, nofollow out explicitly rather than relying on none. The snippet- and preview-control family — max-snippet, max-image-preview, max-video-preview — along with noimageindex, notranslate, indexifembedded, and unavailable_after, is effectively Google-only. When in doubt, spell directives out and treat the max-* controls as Google features.

Common mistakes (and the fixes)

  • Disallow + noindex on the same URL. The noindex is never seen. Fix: leave the page crawlable; keep just the noindex.
  • noindex plus a rel=canonical pointing elsewhere. Conflicting signals — you’re telling Google both “drop this page” and “consolidate it into another one.” Pick one. (More in canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it..)
  • A staging-wide noindex shipped to production. Catastrophic, sitewide deindex. Check before launch.
  • A noindex injected only by client-side JavaScript. Google has to render the page to see it, and if the rendered HTML differs from what you expect, behavior differs too. Prefer the tag in the raw HTML or the header. (See renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM..)
  • Expecting Bing to honor Google-only directives (none, the max-* family).
  • Expecting a robots directive to do a job it doesn’t own. A noindex or nosnippet rule doesn’t by itself guarantee crawl-budget savings, secrecy, a ranking change, identical behavior across search engines, a specific removal timeline, or exclusion from every AI/search surface — each of those outcomes belongs to a different control (authentication for secrecy, robots.txt for crawl, each engine’s own docs for parity, Search Console or Google-Extended for AI-specific scopes). Google gives no fixed timeframe for when a noindexed page actually drops out — it depends on recrawl priority and “may take months” for a lower-importance page.

For where this sits in the bigger picture: robots.txt and crawling are the crawl-control side; noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. and indexing are the index-control side; and nosnippet / data-nosnippet, max-snippet, and max-image-preview are the serving controls you reach for when you want a page indexed but want to shape how it appears. The X-Robots-Tag is this same tag’s HTTP-header equivalent for non-HTML.

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.