X-Robots-Tag

TL;DR — The X-Robots-TagThe X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos. is a way to tell search engines “don’t indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. this file” — but instead of putting a tag in the page’s HTML, it’s sent in the server’s response. That matters because some files (PDFs, images, videos) have no HTML to put a tag in. So when you need to keep a PDF out of Google, the X-Robots-Tag is how you do it.
Evidence for this claim Google supports robots directives in the X-Robots-Tag HTTP response header, including for non-HTML resources such as PDFs. Scope: Google crawling and indexing controls delivered by HTTP header. Confidence: high · Verified: 2026-07-14 Google Search Central: X-Robots-Tag Evidence for this claim Google must be allowed to crawl a resource to discover and apply its X-Robots-Tag noindex rule. Scope: A robots.txt block can prevent Google from seeing the response header. Confidence: high · Verified: 2026-07-14 Google Search Central: Combining crawling and indexing rules

What it is

You’ve probably seen the robots meta tagThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen. — a little snippet in a page’s HTML that says <meta name="robots" content="noindex" /> to keep that page out of search. The X-Robots-Tag does the exact same job, but it travels in the HTTP response header the server sends back, not in the page’s HTML.

Why would you need that? Because some things on your site aren’t HTML pages. A PDF has no <head> section. Neither does an image or a video file. There’s nowhere to put a meta tag. So if you want to keep a PDF out of Google’s index, you can’t use the meta tag — you use the X-Robots-Tag header instead.

When you’d reach for it

You have a PDF (a price list, an old whitepaper, a gated download that leaked) showing up in search and you want it gone.
You want to noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. a whole class of files at once — every PDF on the site — with one rule, instead of editing them one by one.
You’re dealing with images or videos you don’t want indexed on their own.

For a normal HTML page, you’d just use the meta tag — it’s easier. The X-Robots-Tag is for the cases where the meta tag can’t reach.

The one mistake everyone makes

Here’s the trap, and it catches people constantly. They want a PDF gone, so they do two things: they block it in robots.txt and they add an X-Robots-Tag noindex. Belt and suspenders, right?

Wrong. Blocking it in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. stops Google from ever fetching the file — so Google never sees the noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. header you carefully set. The directive gets ignored, and the PDF can stay in search. The rule is: to noindex a file, you have to let Google crawl it so it can read the header. Don’t block it.

How to check it

You (or a developer) can confirm the header is live with a single command: curl -I https://example.com/file.pdf. That prints the response headers without downloading the file, and you look for the X-Robots-Tag line. There’s a copy-paste version in the Scripts tab.

Want the actual server config for Apache, Nginx, and Cloudflare — plus how to target just GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. and verify the live header? Switch to the Advanced tab.

TL;DR — The X-Robots-TagThe X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos. is the HTTP-header delivery of the same directive vocabulary as the robots meta tagThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen. — “Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag.” It exists for one big reason: non-HTML resources (PDFs, images, videos) have no <head>, so the header is the supported way to noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. them. You set it server-side (Apache <FilesMatch>, Nginx add_header, or a Cloudflare Transform Rule / Worker), so one rule covers a whole file type. Optionally target a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. with X-Robots-Tag: googlebot: noindex; with no user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. it applies to all of them. The spine caveat, shared with the meta tag: the file must be crawlable — a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. block hides the header, so the noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. is “not foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. and… therefore ignored.” Verify with curl -I, and watch for CDN-vs-origin header drift.
Evidence for this claim Google supports robots directives in the X-Robots-Tag HTTP response header, including for non-HTML resources such as PDFs. Scope: Google crawling and indexing controls delivered by HTTP header. Confidence: high · Verified: 2026-07-14 Google Search Central: X-Robots-Tag Evidence for this claim Google must be allowed to crawl a resource to discover and apply its X-Robots-Tag noindex rule. Scope: A robots.txt block can prevent Google from seeing the response header. Confidence: high · Verified: 2026-07-14 Google Search Central: Combining crawling and indexing rules

What the X-Robots-Tag actually is

The X-Robots-Tag is the HTTP response header version of the robots <meta> tag. That’s the whole concept. Google is explicit that the two are interchangeable in vocabulary: “The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag.” A response carrying it looks like this:

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noindex
(…)

So if you already understand the robots meta tagThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen., you already understand the directives here — noindex, nofollow, nosnippet, and the rest behave identically for Google. The only thing that changes is the delivery mechanism: header, not HTML.

That equivalence is a Google-documented contract, not a universal web standard — the X-Robots-Tag is a de facto header with no IETF specification backing it. Google says so directly on the same doc: “It is possible that these rules may not be treated the same by all other search engines.” Treat “same directives as the meta tag” as true for Google (and, per the Other Engines section below, largely true for Bing’s core set) — not as a guarantee for every crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., including AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls., that might read this header.

Why it exists — non-HTML files

The defining use case is files that have no HTML <head> to drop a meta tag into. Google’s guidance is direct: “To block indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. of non-HTML resources, such as PDF files, video files, or image files, use the X-Robots-Tag response header instead.” That’s the headline. A PDF, an image, or a video can’t carry a robots meta tag — there’s nowhere to put it — so the header is the supported route to noindex them.

There’s a second, quieter reason: it’s set server-side, so one rule can apply a directive to an entire class of files (every .pdf on the site) or site-wide, without you editing each resource. That’s powerful, and — as we’ll see — also where the foot-guns live.

Worked example — noindexing a single PDF: say https://example.com/whitepapers/old-report.pdf is showing up in search and you want it gone. Two conditions both have to hold: (1) the resource must stay crawlable — no Disallow on it in robots.txt — and (2) the server response for that exact URL must carry X-Robots-Tag: noindex. Set the rule (Apache/Nginx/ Cloudflare examples below), confirm with curl -I that the live response actually returns the header (see the verification section), then wait for Google to recrawl the URL. Skip either condition and the PDF stays indexed: skip crawlabilityCrawlability is how well search engine crawlers can discover, access, and fetch a site's pages. A crawlability issue is any technical condition — blocked access, broken links, server failures, or bloated URL inventory — that stops pages from reaching the index. and Google never reads the header; skip the header and there’s nothing telling Google to drop it.

Do not turn “non-HTML” into “wrong file type.” Google publishes a broad list of indexable file types, including PDF, common office formats, text, XML, and several source-code formats. A crawler should preserve three separate observations: the URL extension, the declared Content-Type, and the bytes actually returned. A mismatch is a delivery warning; a legitimate supported document is not an HTML-page failure and should have HTML-only checks marked not applicable. Use X-Robots-Tag or an HTTP Link canonical when those controls are needed for the non-HTML resource.

Size limits also need an engine-and-basis label. Google’s March 2026 crawler update documents a 2 MB per-URL cutoff for GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. (Search), a 64 MB PDF cutoff, and a 15 MB default for unspecified Google crawler clients. Those are decimal MB resource limits, not interchangeable with a tool’s decoded-body MiB safety cap. See Google’s current crawler limits before turning a product limit into an SEO diagnosis.

Meta tag vs X-Robots-Tag — when to use which

The decision is simple once you frame it around the <head>:

HTML page you control the markup of → use the robots meta tag. It’s easier to set, easier to read, and per-page.
Non-HTML file (PDF, image, video, feed) → use the X-Robots-Tag header. There’s no other option; these have no <head>.
You’d rather control it server-side / site-wide, or you can’t easily touch each page’s HTML (CDN, app layer) → the header is the cleaner lever even for HTML.

They’re not rivals and one isn’t “stronger” — same directives, different reach.

The directives you can use

The X-Robots-Tag accepts the same directive vocabulary as the meta tag. The full supported set:

noindex — don’t index this resource.
nofollow — don’t follow links from it.
none — shorthand for noindex, nofollow.
all — no restrictions (the default).
nosnippet — no text snippet or video preview in results (the page-level cousin of data-nosnippet, which scopes it to part of a page).
max-snippet: [number] — cap the snippet length.
max-image-preview: [none | standard | large] — cap the preview image size.
max-video-preview: [number] — cap the video preview length in seconds.
noarchive — no cached link.
noimageindex — don’t index images on the page.
notranslate — don’t offer translation in results.
indexifembedded — don’t index this resource on its own, but do allow indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. where it’s embedded (e.g. in an iframeHTML element that displays one webpage inside another — how embeds work.). Pair it with noindex.
unavailable_after: [date/time] — drop the URL from results after a date. The date must be in a widely adopted format (RFC 822, RFC 850, or ISO 8601), e.g.:

X-Robots-Tag: unavailable_after: 25 Jun 2010 15:00:00 PST

Presenting the table once is deliberate: it’s the same list shared by the meta tag, which is the cleanest way to think about the relationship between the two.

Targeting a specific crawler

You can optionally prefix the directive with a user-agent token: Google notes the header “may optionally specify a user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. before the rules,” and that “Rules specified without a user agent are valid for all crawlers. The HTTP header, the user agent name, and the specified values are not case sensitive.” So this is valid syntax to noindex only for Google:

X-Robots-Tag: googlebot: noindex

And you can stack lines for different bots. (One note on attribution: Google’s own documented example of per-crawler targeting uses X-Robots-Tag: googlebot: nofollow paired with X-Robots-Tag: otherbot: noindex, nofollow — so googlebot: noindex is correct, supported syntax, just not the verbatim line in Google’s sample.)

Multiple rules and conflicting directives

You’re not limited to one directive per response. Google documents this directly: “Multiple rules may be combined in a comma-separated list or in separate meta tagsMeta tags are HTML elements in a page's head that pass metadata about the page to search engines and browsers. For SEO only a few matter — the title element, the meta description, and the robots meta tag — while meta keywords and most others are ignored.. These rules are case-insensitive.” — the same is true for the header: send X-Robots-Tag: noindex, nofollow as one comma-separated line, or repeat the header field (X-Robots-Tag: noindex on one line, X-Robots-Tag: nofollow on another). Both are valid; the server config examples below use the comma-separated form.

When two rules would conflict, Google resolves it in favor of the tighter restriction: “In the case of conflicting robots rules, the more restrictive rule applies. For example, if a page has both max-snippet:50 and nosnippet rules, the nosnippet rule will apply.” That’s Google’s documented behavior — the same “not treated the same by all other search engines” caveat above applies to conflict resolution too, so don’t assume every crawler picks the stricter rule the same way.

How to set it (server config)

These are known-good, battle-tested patterns I’d reach for — treat every snippet below as an environment-specific example, not a portable recipe: inheritance rules, route matching, redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., response status codes, and any CDN or application layer in front of the server can all change what actually reaches the crawler. Confirm the delivered header on your own stack before shipping, not just the config file. Configs and snippets are in the Scripts tab, copy-paste ready, for Apache (<FilesMatch> + mod_headers), Nginx (location

add_header), and Cloudflare (a Transform Rule or a Worker). The short version: one rule, matched on the file extension, sets X-Robots-Tag: noindex on every file of that type. To target a single crawler, the user-agent token goes inside the header value (Header set X-Robots-Tag "googlebot: noindex"), not as a separate mechanism.

The #1 mistake — don’t block the file in robots.txt

This is the spine of the whole topic, so I’ll be loud about it. People who want a PDF gone often do two things at once: block it in robots.txt and set X-Robots-Tag: noindex. That defeats itself. Google has to crawl the file to read the header — and the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. block stops the crawl.

Google’s own wording: “robots meta tags and X-Robots-Tag HTTP headers are discovered when a URL is crawled. If a page is disallowed from crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored.” And the corollary: “If indexing or serving rules must be followed, the URLs containing those rules cannot be disallowed from crawling.”

Restated for noindex specifically: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results.” This is the same crawling-vs-indexing distinction that runs through robots.txt and the meta tag: robots.txt controls crawling; the X-Robots-Tag controls indexing; and a crawl block hides the indexing rule. To deindex a PDF: allow crawling, serve X-Robots-Tag: noindex, and wait for the re-crawl.

(Note the flip side too: a robots.txt Disallow on its own doesn’t deindex a file either — a blocked PDF can still be indexed if other pages link to it, just without a snippet. Blocking and noindexing solve different problems.)

How to verify it with curl

Don’t trust that a config “should” work — check the live header. curl -I (or curl -sI … | grep -i x-robots-tag) prints the response headers without downloading the body:

curl -sI https://example.com/file.pdf | grep -i x-robots-tag

You can also spoof the bot view to see what Googlebot would get (curl -A "Googlebot" -sI …). Most guides point you at a browser extension or Screaming Frog for this; curl is faster and scriptable, and full commands are in the Scripts tab.

One caveat on -I: it sends a HEAD request, and a server or CDN rule that only fires on GET won’t show up in a HEAD response. Treat curl -I as a fast first check, not the final word — confirm the same header on a real curl GET request (curl -sD - -o /dev/null https://example.com/file.pdf | grep -i x-robots-tag), and check it after following redirects and on the public CDN-fronted URL, not just origin.

TIP Inspect the header Googlebot receives, not the rule you intended

X-Robots-Tag is an HTTP-level signal, so verification has to happen on the public response. A CDN can add, strip, cache, or vary it after the origin configuration looks correct.

Use my HTTP Header Checker to follow the redirect chain, switch user agents, and inspect the exact final X-Robots-Tag value without working through raw curl output. HTTP Header Checker Free

Check the public file URL with the search-crawler user agent and inspect every redirect hop.
Confirm the final response carries the intended directive and does not vary unexpectedly at the CDN.
Keep the resource crawlable so the crawler can actually discover the noindex header.

Watch for CDN-vs-origin drift

Because the header is set server- and CDN-side, the value Googlebot receives can differ from what your origin emits — an edge rule can add, strip, or cache a header. The real-world failure mode here is the “Noindex detected in X-Robots-Tag” message in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance.: a CDN or edge config serving an X-Robots-Tag: noindex — even briefly or by mistake — gets logged by Google as a noindex signal and can deindex pages you never meant to touch. The lesson: always curl -I the public URL behind the CDN, not just origin, and treat edge rules with the same care as origin config.

Other engines

Bing supports the X-Robots-Tag header and broadly the same core directives — noindex, nofollow, noarchive/nocache (Bing uses nocache as a synonym for noarchive), nosnippet, and the preview-limit directives. If you need Bing’s exact supported list, confirm it on Bing’s live help page — their documentation is JavaScript-rendered and worth eyeballing directly before relying on a specific directive.

Where this sits

The X-Robots-Tag is the non-HTML half of on-page indexing control: the robots meta tag for pages, this header for everything else. It leans on the same noindex, nosnippet / max-snippet / max-image-preview vocabulary, and it depends on crawling being allowed — which ties it straight back to robots.txt and the broader indexing story. If you came here from the crawling side, the takeaway is the caveat; if you came from the meta-tag side, the takeaway is that the same directives reach PDFs and media through the header.

AI summary

A condensed take on the Advanced version:

What it is: the X-Robots-TagThe X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos. is the HTTP response header version of the robots meta tagThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen. — “Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag.” Same directives, different delivery — for Google specifically; Google itself notes other search engines may not treat the rules the same way.
Why it exists: non-HTML files (PDFs, images, videos) have no <head>, so the header is the supported way to noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. them — “use the X-Robots-Tag response header instead.” Requires the resource to stay crawlable, not just the header set.
Directives: the full meta-tag vocabulary — noindex, nofollow, none, all, nosnippet, max-snippet, max-image-preview, max-video-preview, noarchive, noimageindex, notranslate, indexifembedded, unavailable_after.
Multiple rules: combine directives in a comma-separated list or repeated header lines; Google resolves conflicts by applying the more restrictive rule.
Set it server-side: Apache <FilesMatch> + mod_headers, Nginx add_header, or a Cloudflare Transform Rule / Worker — one rule per file type. Treat every snippet as environment-specific (inheritance, redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., status codes, and CDN/app layers can all change what’s actually delivered) — for example, Nginx stops inheriting a parent’s add_header once a child location block sets its own.
Per-crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.: X-Robots-Tag: googlebot: noindex; no user-agent tokenA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. = all crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.; names/values are case-insensitive.
The #1 mistake: blocking the file in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. hides the header, so the noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. is “not foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. and… therefore ignored.” To deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist.: allow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. + serve noindex.
Verify: curl -I the public URL as a fast check, but confirm on a real GET request too (-I sends HEAD, which can miss GET-only rules); spoof with -A "Googlebot"; watch for CDN-vs-origin drift (the “Noindex detected in X-Robots-Tag” GSC failure).
Bing: supports it too, with nocache ≈ noarchive (confirm the live list).

Official documentation

Primary-source documentation for the X-Robots-TagThe X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos. and the indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. rules it carries.

Google

Robots meta tag, data-nosnippet, and X-Robots-Tag specifications — the canonical spec: the X-Robots-Tag section, the full directive table, per-crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. targeting, and the crawl-must-be-allowed caveat.
Block Search Indexing with noindex — the two ways to implement noindex (meta tag and HTTP header) and the crawlabilityCrawlability is how well search engine crawlers can discover, access, and fetch a site's pages. A crawlability issue is any technical condition — blocked access, broken links, server failures, or bloated URL inventory — that stops pages from reaching the index. dependency.
Introduction to robots.txt — what robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. does (controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.) — the other half of why a block hides the header.

Bing / Microsoft

Robots meta tags and attributes that Bing supports — Bing’s supported directives, including nocache (≈ noarchive). The page is JavaScript-rendered; confirm the exact list in a browser. (URL updated 2026-07-18 — Bing 301-redirected the old slug here.)

Quotes from the source

On-the-record statements from Google’s documentation. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what the X-Robots-TagThe X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos. is

“The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag.” — Google Search Central docs. Jump to quote

Google — why it exists (non-HTML files)

“To block indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. of non-HTML resources, such as PDF files, video files, or image files, use the X-Robots-Tag response header instead.” Jump to quote

Google — per-crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. targeting

“may optionally specify a user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. before the rules” Jump to quote
“Rules specified without a user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. are valid for all crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. The HTTP header, the user agent name, and the specified values are not case sensitive.” Jump to quote

Google — the crawl-must-be-allowed caveat (the spine)

“…robots meta tagsThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen. and X-Robots-Tag HTTP headers are discovered when a URL is crawled. If a page is disallowed from crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. through the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file, then any information about indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. or serving rules will not be foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. and will therefore be ignored.” Jump to quote
“For the noindex rule to be effective, the page or resource must not be blocked by a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.” — Google Search Central docs. Jump to quote

Google — two ways to implement noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed.

“There are two ways to implement noindex: as a <meta> tag and as an HTTP response header.” Jump to quote

Note: Bing’s “Which robots metatags does Bing support?” help page renders via JavaScript and resists automated checking, so the Bing directive list (and the nocache ≈ noarchive synonym) is corroborated via secondary coverage — confirm it against the live page before treating it as final. John Mueller’s commentary on the robots.txt-block-hides-the-header failure mode, and the “Noindex detected in X-Robots-Tag” CDN incident class, are paraphrased from industry coverage here, not quoted, for the same reason — verify exact wording in a browser before quoting.

X-Robots-Tag — cheat sheet

Meta tag vs X-Robots-TagThe X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos. — which to use

Situation	Use	Why
HTML page you control the markup of	Robots meta tagThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen.	Easiest, per-page, lives in the `<head>`
PDF, image, video, feed (no `<head>`)	X-Robots-Tag header	No HTML to put a meta tag in
Whole file type or site-wide rule	X-Robots-Tag header	One server rule covers them all
Can’t easily edit each page’s HTML	X-Robots-Tag header	Set it at the server / CDN layer

The two controls don’t overlap — keep them straight

Control	Stops crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.?	Stops indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.?	Use it for
`robots.txt` disallow	Yes	No	Keeping botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. out of low-value spaces
`X-Robots-Tag: noindex`	No (must be crawlable)	Yes	Removing a file from the index

Block a file in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. and add X-Robots-Tag: noindex and you cancel yourself out — Google never crawls the file to read the header. Pick one job per URL.

The “same directives, either delivery method” rule above is Google’s documented contract. It’s a reasonable working assumption for most major crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. (Bing’s core set overlaps), but it isn’t a guaranteed cross-engine standard — verify directly with any crawler/vendor you specifically depend on.

Per-crawler syntax

Goal	Header
NoindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. for everyone	`X-Robots-Tag: noindex`
NoindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. for Google only	`X-Robots-Tag: googlebot: noindex`
Different rules per bot	`X-Robots-Tag: googlebot: noindex` `X-Robots-Tag: bingbot: nofollow`
Timed removal	`X-Robots-Tag: unavailable_after: 25 Jun 2010 15:00:00 PST`
Index only when embedded	`X-Robots-Tag: noindex, indexifembedded`

No user-agent tokenA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. = applies to all crawlers. Header name, user-agent name, and values are all case-insensitive.

Verification, in one line

curl -sI https://example.com/file.pdf | grep -i x-robots-tag
Spoof GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.: curl -A "Googlebot" -sI https://example.com/file.pdf
Always check the public (CDN-fronted) URL, not just origin.

X-Robots-Tag implementation checklist

Choose the response types and URL patterns that need a header-level directive.
Keep affected resources crawlable so crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. can read the header.
Apply directives at one known layer: origin, application, web server, CDN, or edge.
Test successful, redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., error, cached, and uncached responses separately.
Check crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-specific prefixes only when a real crawler-specific policy is intended.
Document ownership and a rollback before applying a broad path or file-type rule.

Playbook: an X-Robots-Tag rollout affects the wrong URLs

Pause the broad rule or edge deployment.
Capture headers from affected and unaffected samples, including cache status and redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't..
Identify the owning layer and exact path, MIME-type, or route match.
Narrow the condition and remove duplicate directives from other layers.
Purge relevant caches, then retest the full sample matrix.
Exit when intended resources carry one coherent policy and ordinary pages do not inherit it.

Common X-Robots-Tag issues

The origin sends the header but the public URL does not

Cause: CDN or proxy behavior strips, caches, or overrides it. Fix: inspect each hop and configure the public edge response as the source of truth.

`noindex` never takes effect

Cause: robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. blocks the resource, so the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. cannot read its response header. Fix: allow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and retain the noindex response.

HTML pages inherit a PDF-only rule

Cause: the server match is too broad or uses the wrong MIME/path condition. Fix: narrow the rule and test multiple file and route types before redeploying.

Patrick's relevant free tools

Google Index Checker — Check one URL’s observable indexability blockers, or reconcile sitemap, crawl, and supplied Search Console evidence across a URL set before verifying Google’s actual state in URL Inspection.
Google SERP Simulator — Interactive Google result preview: type a title, description, and URL, compare controlled-font desktop + mobile truncation guidance with simulated query bolding, copy the finished tags, and share a filled-in preview.
Meta Description Generator — Generate five meta-description options with AI, then compare their measured widths with approximate desktop and mobile SERP-preview guidance. Variants are scored for the keyword, a call to action, and length; you also get a matching title tag and a copy-ready tag block. Quick, paste-content, fetch-URL, and small bulk (≤10) modes. Falls back to editable templates when AI is off.

Tools for header-level robots controls

HTTP Header Checker — inspect the live public X-Robots-TagThe X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos. response.
Robots.txt Tester — confirm the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. can fetch a resource whose header it must read.
Redirect Chain Mapper — see which response in a redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. path carries the directive.
Staging vs. Production SEO Diff — catch environment-specific header drift before release.

Validation tests

Test the live header matrix

Test to run — request representative file types, paths, statuses, and cached/uncached responses with the HTTP Header Checker or curl -I. Expected result — only intended responses carry the exact directive. Failure interpretation — a match rule, cache, or proxy layer is too broad or inconsistent. Monitoring window — immediate. Rollback trigger — revert if indexable HTML or unrelated resources inherit noindex.

Test crawlability plus noindex

Test to run — verify robots access and then fetch the response header as the target crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. Expected result — crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is allowed and X-Robots-Tag: noindex is visible. Failure interpretation — robots rules or crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-specific header syntax prevents the policy from being read. Monitoring window — immediate technically; indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. removal waits for recrawlCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial.. Rollback trigger — restore access immediately if a broad robots change blocks required resources.

Test yourself: X-Robots-Tag

Server config to noindex PDFs

These are known-good config patterns I’d reach for — solid starting points, but confirm them against your own stack (and the live curl -I check at the bottom) before relying on them. They are not Google quotes.

Apache (`.htaccess` or vhost — needs `mod_headers`)

NoindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. every PDF:

<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

Multiple file types at once:

<FilesMatch "\.(pdf|docx?|xlsx?|pptx?)$">
  Header set X-Robots-Tag "noindex, noarchive, nosnippet"
</FilesMatch>

Notes: Apache needs mod_headers enabled. Header set overwrites any existing value — use Header append if another rule may already set the header. To target one crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., put the tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. in the value: Header set X-Robots-Tag "googlebot: noindex".

Nginx (inside a `server {}` or `location` block)

NoindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. every PDF:

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

Multiple types:

location ~* \.(pdf|docx?|xlsx?|pptx?)$ {
  add_header X-Robots-Tag "noindex, noarchive, nosnippet";
}

Notes: the directive has to live in an active server/location block, and Nginx needs a reload (nginx -s reload) to pick it up. Inheritance trap: Nginx only inherits add_header directives from a parent block (server {}) into a child block (location {}) when the child has no add_header of its own. The moment a location block adds any header — even an unrelated one, like a CORS or cache-controlCaching stores a copy of a page or resource — in a browser, a CDN edge node, or a search crawler's own cache — so it can be served again without regenerating or re-downloading it. It isn't a direct ranking factor, but it feeds page speed and crawl efficiency. header — it silently stops inheriting every add_header set above it, including your X-Robots-Tag. If that happens, repeat the X-Robots-Tag line inside the child block; don’t assume it carried over.

Cloudflare — two options

Option 1 — Transform Rule (no code). Rules → Transform Rules → Modify Response Header → set X-Robots-Tag = noindex, nofollow when http.request.uri.path ends with .pdf.

Option 2 — Worker on a route matching *.pdf:

export default {
  async fetch(request, env, ctx) {
    const res = await fetch(request);
    const out = new Response(res.body, res);
    out.headers.set("X-Robots-Tag", "noindex, nofollow");
    return out;
  }
};

Either way, a CDN sitting in front of your origin can add, strip, or cache this header — which is exactly why you verify against the public URL below, not origin.

Verify the live header with curl

curl -I prints the response headers without downloading the body. Check the public URL (the one Google fetches), and optionally spoof the bot view.

macOS / Linux

# Print all response headers
curl -I https://example.com/file.pdf

# Just the X-Robots-Tag line
curl -sI https://example.com/file.pdf | grep -i x-robots-tag

# See what Googlebot would receive (spoof the user agent)
curl -A "Googlebot" -sI https://example.com/file.pdf | grep -i x-robots-tag

# Confirm on a real GET request too — -I sends HEAD, and a rule scoped to
# GET-only responses won't show up in a HEAD check
curl -sD - -o /dev/null https://example.com/file.pdf | grep -i x-robots-tag

Windows (PowerShell)

# Inspect the response headers
(Invoke-WebRequest -Method Head -Uri "https://example.com/file.pdf").Headers["X-Robots-Tag"]

# Spoof the Googlebot user agent
(Invoke-WebRequest -Method Head -Uri "https://example.com/file.pdf" `
  -UserAgent "Googlebot").Headers["X-Robots-Tag"]

If the header is missing, the most common causes are: it’s set on origin but stripped/overridden at the CDN; mod_headers isn’t enabled (Apache); the config wasn’t reloaded (Nginx); an Nginx location block redefined add_header and silently stopped inheriting the parent’s X-Robots-Tag line; the rule only fires on GET and you tested with -I (HEAD); or the file is blocked in robots.txt, so Google never fetches it to see the header in the first place.

X-Robots-Tag

The X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos.

X-Robots-Tag

The X-Robots-Tag is an HTTP response header that carries the same indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. and serving directives as the robots <meta> tag — noindex, nofollow, nosnippet, noarchive, max-snippet, max-image-preview, max-video-preview, unavailable_after, noimageindex, indexifembedded, notranslate, none, and all. Anything you can put in a robots meta tagThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen., you can send as an X-Robots-Tag header instead — that equivalence is Google’s documented contract; Google itself notes other search engines may not treat the rules the same way. You can also combine multiple directives in a comma-separated list or repeated header lines; Google resolves conflicting rules by applying the more restrictive one.

The reason it exists is simple: non-HTML files have no <head> to drop a meta tag into. A PDF, an image, or a video can’t carry a robots meta tagThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen., so when you need to noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. one of those, the header is the supported route. Because it’s set at the server level (Apache, Nginx, or a CDN like Cloudflare), a single rule can apply a directive to a whole class of files — every .pdf on the site, say — or even site-wide. You can optionally target one crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. by prefixing the directive with a user-agent tokenA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., e.g. X-Robots-Tag: googlebot: noindex; with no tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory., the rule applies to every crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index..

It shares one hard caveat with the meta tag: the URL has to be crawlable. If you block the file in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., Google never fetches it, never sees the header, and the noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. is silently ignored. So the header controls indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., not crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. — crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.’s job, and the two have to cooperate. Verify a live header with curl -I against the public URL, and watch for a CDN serving a different header than your origin.

Build-time retrieval analysis plus live signals for this exact article. The automatic chunk report includes a deterministic readiness score and is ready without a model download.

How this article chunks build baseline · local controls

A structure-aware retrieval preview using the shared Chunk Tester engine. The default lens is shown immediately; switch scopes to see what happens when every authored lens is treated as one source document.

Target size 300 tokens Overlap 15% Retrieval query

69readiness score / 100

12chunks

~346median tokens

92%standalone

8flagged chunks

Advanced lens only — the version readers see when this page opens.

Baseline generated at build time with a 300-token target and 15% overlap. Controls re-run the same deterministic engine in this browser.

Matching retrieval chunks

Ranked lexically for the page title, Trends, and Ngram terms. Enter a short query to test a different retrieval intent.

chunk 11 ~337 tok How to verify it with curl

curl -sI https://example.com/file.pdf | grep -i x-robots-tag You can also spoof the bot view to see what Googlebot would get (`curl -A "Googlebot" -sI …`). Most guides point you at a browser extension or Screaming Frog f…

chunk 10 ~400 tok The #1 mistake — don't block the file in robots.txt

Google's own wording: *"robots meta tags and X-Robots-Tag HTTP headers are discovered when a URL is crawled. If a page is disallowed from crawling through the robots.txt file, then any information about indexing or servi…

chunk 3 ~349 tok Why it exists — non-HTML files

The defining use case is files that have no HTML ` ` to drop a meta tag into. Google's guidance is direct: *"To block indexing of non-HTML resources, such as PDF files, video files, or image files, use the `X-Robots-Tag`…

chunk 4 ~472 tok Why it exists — non-HTML files

**Worked example — noindexing a single PDF:** say `https://example.com/whitepapers/old-report.pdf` is showing up in search and you want it gone. Two conditions both have to hold: (1) the resource must stay **crawlable** …

chunk 2 ~427 tok What the X-Robots-Tag actually is

The X-Robots-Tag is the **HTTP response header** version of the robots ` ` tag. That's the whole concept. Google is explicit that the two are interchangeable in vocabulary: *"The `X-Robots-Tag` can be used as an element …

Methodology and limits

The readiness score starts with the percentage of standalone chunks and subtracts up to 35 points for the share carrying structural warnings. Structure-aware Markdown blocks use the selected 100–800-token target and 0–40% overlap. Tokens are estimated at roughly four English characters each. “Standalone” means a chunk has heading context and does not open with a context-dependent continuation.

Matching uses the shared lexical BM25 and term-cosine ranker. It is not an embedding or a replica of Google, an answer engine, or any vendor retriever. These chunk controls keep the article text in this browser and make no network request. The build-time baseline remains the reproducible report for this page.

Test your own content in Chunk Tester →

76readiness score / 100

27chunks

~325median tokens

93%standalone

13flagged chunks

All 13 authored lenses in this article source, with lens boundaries retained as headings.

Baseline generated at build time with a 300-token target and 15% overlap. Controls re-run the same deterministic engine in this browser.

Matching retrieval chunks

Ranked lexically for the page title, Trends, and Ngram terms. Enter a short query to test a different retrieval intent.

chunk 24 ~393 tok Test crawlability plus noindex

**Test to run —** verify robots access and then fetch the response header as the target crawler. **Expected result —** crawling is allowed and `X-Robots-Tag: noindex` is visible. **Failure interpretation —** robots rules…

chunk 2 ~303 tok When you'd reach for it

showing up in search and you want it gone. • You want to **noindex** a whole class of files at once — every PDF on the site — with one rule, instead of editing them one by one. • You're dealing with images or videos you …

chunk 25 ~303 tok Nginx (inside a `server {}` or `location` block)

Multiple types: location ~* \.(pdf|docx?|xlsx?|pptx?)$ { add_header X-Robots-Tag "noindex, noarchive, nosnippet"; } Notes: the directive has to live in an active `server`/`location` block, and Nginx needs a reload (`ngin…

chunk 21 ~282 tok X-Robots-Tag — cheat sheet

chunk 13 ~337 tok How to verify it with curl

curl -sI https://example.com/file.pdf | grep -i x-robots-tag You can also spoof the bot view to see what Googlebot would get (`curl -A "Googlebot" -sI …`). Most guides point you at a browser extension or Screaming Frog f…

Methodology and limits

The readiness score starts with the percentage of standalone chunks and subtracts up to 35 points for the share carrying structural warnings. Structure-aware Markdown blocks use the selected 100–800-token target and 0–40% overlap. Tokens are estimated at roughly four English characters each. “Standalone” means a chunk has heading context and does not open with a context-dependent continuation.

Matching uses the shared lexical BM25 and term-cosine ranker. It is not an embedding or a replica of Google, an answer engine, or any vendor retriever. These chunk controls keep the article text in this browser and make no network request. The build-time baseline remains the reproducible report for this page.

Test your own content in Chunk Tester →

Search Console

Loading…

GA4 traffic (28d)

Loading…

Cloudflare traffic (7d)

Loading…

CrUX field data (28d, phone)

Loading…

Google NLP entities

Loading…

Changelog

Updated Jul 18, 2026.

Editorial summary and recorded change details.

Summary

Scoped the meta-tag/X-Robots-Tag directive equivalence to Google specifically (Google's own doc notes other search engines may not treat the rules the same way), added Google's documented multiple-rules/conflict-resolution behavior, tightened server-config and curl verification guidance (Nginx add_header inheritance trap, GET vs HEAD), and added a worked non-HTML noindex example.

Change details

Added a new 'Multiple rules and conflicting directives' section covering comma-separated/repeated X-Robots-Tag headers and Google's more-restrictive-rule-wins conflict resolution, sourced from a fresh re-fetch of Google's robots-meta-tag doc.
Scoped the 'same directives as the meta tag' equivalence explicitly to Google in the Advanced lens, cheat-sheet, and glossary, quoting Google's own caveat that other search engines may not treat the rules the same way.
Documented the Nginx add_header non-inheritance trap (a child location block that sets its own add_header stops inheriting the parent's X-Robots-Tag) in the Scripts lens and troubleshooting causes.
Added a GET-vs-HEAD caveat to curl verification guidance in both the Advanced and Scripts lenses, since curl -I sends HEAD and can miss GET-only header rules.
Added a concrete PDF noindex worked example stating the fetchability requirement inline, and updated the Bing citation URL to the page's current 301 target.

Full comparison unavailable — no prior snapshot was archived for this revision.

X-Robots-Tag

What it is

When you’d reach for it

The one mistake everyone makes

How to check it

What the X-Robots-Tag actually is

Why it exists — non-HTML files

Meta tag vs X-Robots-Tag — when to use which

The directives you can use

Targeting a specific crawler

Multiple rules and conflicting directives

How to set it (server config)

The #1 mistake — don’t block the file in robots.txt

How to verify it with curl

Watch for CDN-vs-origin drift

Other engines

Where this sits

AI summary

Official documentation

Quotes from the source

X-Robots-Tag — cheat sheet

X-Robots-Tag implementation checklist

Playbook: an X-Robots-Tag rollout affects the wrong URLs

Common X-Robots-Tag issues

The origin sends the header but the public URL does not

noindex never takes effect

HTML pages inherit a PDF-only rule

Patrick's relevant free tools

Tools for header-level robots controls

Validation tests

Test the live header matrix

Test crawlability plus noindex

Test yourself: X-Robots-Tag

Server config to noindex PDFs

Apache (.htaccess or vhost — needs mod_headers)

Nginx (inside a server {} or location block)

Cloudflare — two options

Verify the live header with curl

X-Robots-Tag

X-Robots-Tag

How this article chunks build baseline · local controls

Matching retrieval chunks

Flagged retrieval chunks

Matching retrieval chunks

Flagged retrieval chunks

Google Trends X-Robots-Tag · noindex

Books Ngram x-robots-tag · noindex pdf

Changelog

Updated Jul 18, 2026.

Summary

Change details

People also ask

`noindex` never takes effect

Apache (`.htaccess` or vhost — needs `mod_headers`)

Nginx (inside a `server {}` or `location` block)