Robots.txt

TL;DR — Robots.txt is a text file at the root of your site that tells crawlers which URLs they’re allowed to request. The one thing everyone gets wrong: it controls crawling, not indexing. Blocking a page does not remove it from Google — a blocked page can still show up in search (just without a description). If you want a page out of Google, that’s a different tool (noindex), and you must not block it in robots.txt or Google never sees the instruction.

What a robots.txt file is

When a search engine wants to crawl your site, the first thing it looks for is a file at https://yourdomain.com/robots.txt. That file is a list of instructions telling crawlers which parts of your site they may and may not request. It’s just plain text, and you can read any site’s robots.txt by typing /robots.txt after the domain.

A minimal one looks like this:

Tap to expand

The chart summarizes which AI user-agent names Cloudflare observed in robots.txt files during the selected four-week period.

User-agent: *
Disallow: /admin/

Sitemap: https://yourdomain.com/sitemap.xml

That says: to every crawler (*), don’t request anything under /admin/, and by the way, here’s where my sitemap is.

The one thing to understand: blocking ≠ removing

This is the single most common mistake in all of SEO, so I’ll say it loudly: robots.txt stops a page from being crawled, not from being indexed. Evidence for this claim A robots.txt rule controls crawling rather than guaranteeing removal from Google Search; a URL can still appear when Google cannot crawl it. Scope: Google Search crawler behavior. Other crawlers can interpret robots.txt differently. Confidence: high · Verified: 2026-07-14 Google: Introduction to robots.txt

Those sound like the same thing. They aren’t. Crawling is the engine fetching and reading your page. Indexing is the engine deciding to store it and show it in results. Robots.txt only blocks the first step.

So if you block a page in robots.txt, Google won’t fetch it — but if other pages link to that URL, Google can still index the URL itself and show it in search. It just shows up as a bare link with no description underneath, because Google was never allowed to read the page to write one.

If your actual goal is “get this page out of Google,” robots.txt is the wrong tool. You want noindex (a meta tag on the page or an HTTP header). And here’s the trap: a noindex only works if Google can crawl the page to see it. If you block the page in robots.txt and add a noindex, Google never crawls it, never sees the noindex, and the page can linger in the index anyway. Pick one job: block crawling or block indexing, not both on the same URL.

The right tool for each job

• “Don’t waste crawling on this.” → robots.txt Disallow.
• “Keep this out of Google’s index.” → noindex (and don’t block it).
• “Nobody should be able to see this at all.” → password protection / login. Robots.txt is public — anyone can read it — so listing /secret/ in it just tells the world the folder exists.

A few rules that trip people up

• One file per host. https://example.com, https://www.example.com, and https://blog.example.com each need their own robots.txt. They don’t share.
• It lives at the root. It has to be at /robots.txt, all lowercase. A file in a subfolder does nothing.
• Don’t block your CSS and JavaScript. Google needs those files to see your page the way a visitor does. Blocking them can hurt you.

Want the full syntax reference — wildcards, the Allow/Disallow matching rules, how Google handles a 404 vs a server error on your robots.txt, crawl-delay, and how to manage AI crawlers — switch to the Advanced tab.

TL;DR — Robots.txt is a plain-text file at the root of each host (/robots.txt, lowercase) that implements the Robots Exclusion Protocol (RFC 9309). It controls crawling, not indexing — a disallowed URL can still be indexed without a snippet if linked elsewhere; to deindex use noindex on a page that is not blocked. Google supports only user-agent, allow, disallow, and sitemap; noindex/nofollow/crawl-delay were dropped Sept 1, 2019. Scope is one host+protocol+port. Matching uses the most-specific (longest) rule, least-restrictive on ties; * and $ are the wildcards; paths are case-sensitive. Google caps the file at 500 KiB, caches ~24h, treats 4xx (except 429) as no-restrictions, and on a 5xx stalls crawling for ~12h then falls back to the last good copy for ~30 days. Don’t block render-critical CSS/JS, and don’t treat it as access control — the file is public.

What it is and where it lives

Robots.txt implements the Robots Exclusion Protocol, created by Martijn Koster in 1994 and finally standardized in 2022 as RFC 9309 — co-authored by Google’s Gary Illyes, Henner Zeller, Lizzi Sassman, and Koster himself. The standard’s own wording: “This document specifies and extends the ‘Robots Exclusion Protocol’ method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers.”

A few facts that catch people out:

• It must be at the root, lowercase. RFC 9309 is explicit: “The rules MUST be accessible in a file named ‘/robots.txt’ (all lowercase) in the top-level path of the service.” Google adds that the URL itself is case-sensitive, like any URL.
• Scope is one host + protocol + port. Google: “The rules listed in the robots.txt file apply only to the host, protocol, and port number where the robots.txt file is hosted.” So https://example.com, https://www.example.com, https://blog.example.com, and http://example.com each need their own file. Subdomains and protocols don’t share one.
• Supported protocols for Google are HTTP, HTTPS, and FTP.

The misconception that defines this topic: crawling vs indexing

If you take one thing from this page, take this: robots.txt controls crawling, not indexing. Blocking a URL is not the same as removing it from Google. Evidence for this claim A robots.txt rule controls crawling rather than guaranteeing removal from Google Search; a URL can still appear when Google cannot crawl it. Scope: Google Search crawler behavior. Other crawlers can interpret robots.txt differently. Confidence: high · Verified: 2026-07-14 Google: Introduction to robots.txt

Google’s own intro doc says it plainly: robots.txt “is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.” And on what actually happens to a blocked URL: “While Google won’t crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web.” The result is the familiar snippet-less listing: “its URL can still appear in search results, but the search result won’t have a description.”

The spec restates the same nuance for the disallow rule itself: “Google can’t index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippet.”

Why you must NOT block a page you want to noindex

This is the trap that quietly breaks deindexing efforts. A noindex only works if Google can crawl the page to read it. Google’s block-indexing doc spells out the dependency: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.” Evidence for this claim Google must be able to crawl a URL to see a noindex rule; blocking the URL in robots.txt can prevent the rule from being observed. Scope: Google Search indexing controls for HTML meta robots and X-Robots-Tag rules. Confidence: high · Verified: 2026-07-14 Google: Block indexing with noindex

So if your goal is to get a page out of the index, John Mueller’s guidance is the cleanest way to remember it: when you want to unindex pages, you should not block Google with robots.txt, but rather use noindex.

The lived proof: I blocked two of our own high-ranking pages

I don’t have to argue this from theory. In my experiment blocking two high-ranking Ahrefs pages, I deliberately blocked them in robots.txt and tracked what happened. The pages stayed indexed and kept ranking — they didn’t vanish. What we lost was the freshness Google gets from re-crawling: “We lost a position here or there and all of the featured snippets for the pages.” Traffic dropped, but less than I expected: “Both pages lost some traffic. But it didn’t result in much change to our traffic estimate like I was expecting.”

My takeaway from the data: “Accidentally blocking pages (that Google already ranks) from being crawled using robots.txt probably isn’t going to have much impact on your rankings, and they will likely still show in the search results.” And the blunt version: “Don’t block pages you want indexed. It hurts. Not as bad as you might think it does—but it still hurts.”

The flip side is reassurance: when Search Console flags “Indexed, though blocked by robots.txt” for a utility URL — cart, filter, parameter junk — it’s usually a non-issue. As Mueller put it about add-to-cart URLs, blocking them is fine, and even if they get “indexed,” it’s unlikely they’ll be shown in search unless someone runs a very specific query for those URLs, which real users don’t do. Distinguish the scary-sounding warning from an actual problem: it only matters if the blocked URL is a page you actually wanted crawled and indexed.

The syntax (the reference)

A robots.txt is a set of groups. Each group starts with one or more User-agent lines naming which crawler(s) it applies to, followed by the rules for them.

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

User-agent: Googlebot
Disallow: /no-google/

Sitemap: https://example.com/sitemap.xml

User-agent and groups. A crawler obeys exactly one group — the one with the most specific user-agent that matches it — and ignores the rest. Google: “Google’s crawlers determine the correct group of rules by finding in the robots.txt file the group with the most specific user agent that matches the crawler’s user agent. Other groups are ignored.” And: “Only one group is valid for a particular crawler.” (Bing behaves the same way — more on that below.)

That also means a specific group doesn’t get topped up with the wildcard group’s rules — it’s used on its own, not merged with User-agent: *. Google’s spec is explicit that “user agent specific groups and global groups (*) are not combined.” So if you write a User-agent: googlebot-news group, it has to be self-contained: anything you still want it to obey from the * group has to be repeated inside it, or Googlebot-News simply won’t see those rules at all.

Disallow and Allow. Disallow lists paths a crawler must not request; Allow carves exceptions back out. The disallow rule “specifies paths that must not be accessed by the crawlers identified by the user-agent line the disallow rule is grouped with.” The allow rule “specifies paths that may be accessed by the designated crawlers. When no path is specified, the rule is ignored.”

The matching rule (most guides get this wrong). When two rules conflict, the most specific one wins, and “most specific” means longest path: “When matching robots.txt rules to URLs, crawlers use the most specific rule based on the length of the rule path. In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule.” So on a genuine tie, the least restrictive rule wins — Allow beats Disallow. RFC 9309 frames it as the “Longest Match”: “The following example shows that in the case of two rules, the longest one is used for matching.” Evidence for this claim Google resolves matching robots.txt rules by path specificity and uses the least restrictive rule when equally specific rules conflict. Scope: Google crawler interpretation of robots.txt rules; other crawlers may implement different extensions. Confidence: high · Verified: 2026-07-14 Google: Robots.txt interpretation

Worked example:

User-agent: *
Allow: /folder/page
Disallow: /folder/

The URL /folder/page matches both rules. Allow: /folder/page (12 chars) is longer than Disallow: /folder/ (8 chars), so the longer, more specific Allow wins and the page is crawlable.

Wildcards * and $. Google: “* designates 0 or more instances of any valid character. $ designates the end of the URL.” So Disallow: /*.pdf$ blocks every URL ending in .pdf, and Disallow: /*? blocks every URL containing a query string. Matching is prefix-based: Disallow: /fish matches /fish, /fish.html, and /fish/salmon.html, but not /Fish (case-sensitive) or /catfish (it’s a prefix, not a substring).

Case sensitivity (the subtle one). Field and user-agent names are case-insensitive; path values are case-sensitive. Google: “Both the user-agent field name and its value are case-insensitive,” but “The field name (disallow) is case-insensitive, but its value is case-sensitive,” and “The path value must start with / to designate the root and the value is case-sensitive.” So Disallow: /Folder/ does not block /folder/.

Sitemap. The Sitemap: directive takes a full absolute URL and is independent of groups — it can sit anywhere in the file.

Comments. Anything after # is ignored: “To include comments, precede your comment with the # character.”

noindex, nofollow, and crawl-delay are NOT robots.txt directives

This is a persistent myth. As of September 1, 2019, Google retired support for unsupported, undocumented rules — including noindex, nofollow, and crawl-delay. Google’s announcement focused on rules unsupported by the internet draft, such as crawl-delay, nofollow, and noindex, noting they were never documented by Google, and said Google was retiring all code that handles unsupported and unpublished rules (such as noindex) on that date. The supported field list is short, and the spec calls out the exclusion directly: Google supports user-agent, allow, disallow, and sitemap, and “other fields such as crawl-delay aren’t supported.”

If you relied on noindex in robots.txt, the alternatives are a noindex meta tag or X-Robots-Tag header, 404/410 status codes, password protection, a Disallow, or the Search Console removal tool.

How Google handles robots.txt under the hood

• Size limit: 500 KiB. “Google enforces a robots.txt file size limit of 500 kibibytes (KiB). Content which is after the maximum file size is ignored.” RFC 9309 aligns: “The parsing limit MUST be at least 500 kibibytes [KiB].”
• Caching: ~24 hours. “Google generally caches the contents of robots.txt file for up to 24 hours, but may cache it longer in situations where refreshing the cached version isn’t possible.” So a change isn’t necessarily picked up instantly. Evidence for this claim Google generally caches robots.txt for up to 24 hours and changes crawling behavior according to the HTTP status returned for the file. Scope: Google crawler handling of robots.txt fetches, including documented 4xx, 5xx, and redirect behavior. Confidence: high · Verified: 2026-07-14 Google: Robots.txt file handling
• Status codes matter site-wide. This is the part most guides skip:
  • 4xx (except 429) → no restrictions. “Google’s crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn’t exist. This means that Google assumes that there are no crawl restrictions.” A 404 on /robots.txt means “crawl everything.” (Don’t use 401/403 to throttle crawling.)
  • 5xx / unreachable → dangerous. “For the first 12 hours, Google stops crawling the site but keeps trying to fetch the robots.txt file. If Google can’t fetch a new version, for the next 30 days Google will use the last good version, while still trying to fetch a new version.” So a server error on /robots.txt can effectively disallow your whole site for the first ~12 hours, then run on the last cached copy for ~30 days. A persistently erroring robots.txt is a site-wide crawl risk. And if it’s still broken after those 30 days: “If the errors are still not fixed after 30 days: If the site is generally available to Google, Google will behave as if there is no robots.txt file (but still keep checking for a new version).” In other words, a robots.txt that never recovers doesn’t stay disallowed forever — Google eventually falls back to crawling with no restrictions, the same as a 404.
  • 3xx → Google follows at least five redirect hops, then treats it as a 404.

robots.txt in Bing, Yandex, and beyond

The grouping and syntax are essentially shared, but two divergences matter:

• crawl-delay. Google ignores it, Bing still honors it, and Yandex dropped it in 2018 — Yandex’s own documentation states that “From February 22, 2018, Yandex doesn’t take into account the Crawl-delay directive,” pointing you to the site crawl rate setting in Yandex Webmaster instead. Bing is explicit that “The robots.txt file is the only valid place to set a crawl-delay directive for MSNBot,” and that the directive “accepts only positive, whole numbers as values… the higher the value, the more throttled down the crawl rate will be.” Note Bing treats the value as a relative throttle, not literally N seconds.
• The bingbot-section gotcha. Just like Google’s “only one group per crawler” rule, if you create a User-agent: bingbot section, Bing applies only that section and ignores the User-agent: * defaults (crawl-delay excepted). So a bingbot-specific group must repeat every directive you still want enforced.
• Amazon’s cache and failure behavior. Amazon says its crawlers may use a robots.txt copy cached within the previous 30 days. If they cannot fetch the file, they behave as though it does not exist. A checker can report the copy it fetched, but it cannot prove which cached version Amazon used—or that Amazon observed the same failure as the checker. Evidence for this claim Amazon says its crawlers may use a robots.txt copy cached within the previous 30 days and behave as though the file does not exist when they cannot fetch it. Scope: Amazon crawler behavior only; a checker result cannot establish which cached copy Amazon used or whether Amazon observed the same fetch failure. Confidence: high · Verified: 2026-07-19 Amazon: Amazonbot

Managing AI crawlers with robots.txt

Robots.txt is currently the main lever for managing AI crawlers, and they obey the same group/user-agent syntax. The catch: these are separate tokens, so blocking one doesn’t block the others.

• OpenAI runs several distinct bots, and the controls for each are independent — allowing one doesn’t allow the others, and blocking one doesn’t block the others. GPTBot crawls content for training OpenAI’s models; OAI-SearchBot surfaces sites in ChatGPT’s search features; OAI-AdsBot checks the safety of pages submitted as ads (its data isn’t used for training). Block training with User-agent: GPTBot / Disallow: / — that alone won’t stop the search or ads bots. ChatGPT-User is different again: it fires for actions a person triggers inside ChatGPT or a Custom GPT, not automatic crawling, and OpenAI says “robots.txt rules may not apply” to it — so don’t count on a Disallow to keep it out. If you do change what OAI-SearchBot can crawl, OpenAI notes it can take about 24 hours for the update to reach their search systems.
• Google-Extended controls Gemini/Vertex training and is separate from Googlebot.
• Others worth naming: CCBot (Common Crawl), ClaudeBot (Anthropic), PerplexityBot, and Bytespider.

The hard caveat: compliance is voluntary. Robots.txt requests; it doesn’t enforce. Well-behaved crawlers obey it; scrapers can and do ignore it. If you truly need to keep something away from a bot, that’s an authentication/blocking problem, not a robots.txt one.

Common mistakes (and the fixes)

The file is 200, but it is not actually a usable robots file. Status alone is not enough. Capture the response Content-Type and first bytes: a CDN/custom-error template can return HTML at /robots.txt with 200, which must be a warning rather than an “allow all” pass. Google documents robots.txt as UTF-8 plain text and may ignore invalid characters. A single UTF-8 BOM at the beginning is tolerated, but a second BOM, a BOM in the middle, UTF-16 bytes, NULs, or invisible/control characters can alter the first token or invalidate a line. Show the byte offset and affected line; do not silently normalize the file before telling the user what the crawler received. Apply Google’s effective 500 KiB parsing limit before calculating allow/disallow results, while still reporting the discarded tail.

• Blocking a page you also want deindexed. Block + noindex means Google never crawls it to see the noindex. Use noindex without the block.
• Using robots.txt to deindex. Wrong tool entirely — that’s noindex’s job.
• Blocking render-critical CSS/JS. Google needs those assets to see the page as a user does; Google’s own sample robots.txt explicitly re-allows .css/.js so Googlebot can crawl them.
• Trying to hide sensitive data. RFC 9309 is blunt: “The Robots Exclusion Protocol is not a substitute for valid content security measures. Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable.” Disallowing /secret-admin/ literally advertises it. Use auth.
• A stray Disallow: /. This blocks the entire site for the named crawler — the classic staging leftover that takes a site out of Google.
• Ignoring the response code on /robots.txt. A 5xx can stall crawling site-wide; treat the file’s availability as production-critical.

For the broader pipeline this sits inside — discovery, the crawl scheduler, rendering, and how crawling differs from indexing — see the crawling hub. The sibling topics (crawl budget, and how Google handles sitemaps) each go deeper on one piece of this.

AI summary

A condensed take on the Advanced version:

• Robots.txt = a plain-text file at the root of each host (/robots.txt, lowercase) implementing the Robots Exclusion Protocol (RFC 9309, 2022; origin 1994). The URL is case-sensitive.
• It controls crawling, not indexing. This is the #1 myth. A disallowed URL can still be indexed — shown as a bare URL with no description — if it’s linked from elsewhere. To keep a page out of the index, use noindex, and the page must not be blocked in robots.txt or Google never crawls it to see the noindex.
• Patrick’s experiment proved it: blocking two high-ranking pages didn’t deindex them — they kept ranking but lost featured snippets and descriptions. “Don’t block pages you want indexed. It hurts.”
• Right tool for each job: robots.txt = crawl control; noindex = index control; password/auth = secrecy (robots.txt is public).
• Supported fields: user-agent, allow, disallow, sitemap. Not supported by Google: noindex, nofollow, crawl-delay (retired Sept 1, 2019).
• Scope is one host + protocol + port — every subdomain/protocol needs its own file.
• Matching: most-specific (longest path) rule wins; least-restrictive (Allow) on ties. Wildcards * and $; path values are case-sensitive. A specific group isn’t merged with the * group — it stands alone.
• Under the hood: 500 KiB size limit; ~24h caching; 4xx (except 429) → no restrictions; 5xx/unreachable → ~12h crawl stop, then last good copy for ~30 days, then no restrictions if still broken; 3xx → follows ≥5 hops then treats as 404.
• Other engines: Bing honors crawl-delay; Google ignores it and Yandex stopped honoring it in February 2018 (use Yandex Webmaster’s crawl rate setting). A bingbot section makes Bing ignore the * defaults.
• AI crawlers (GPTBot, OAI-SearchBot, OAI-AdsBot, ChatGPT-User, Google-Extended, CCBot, ClaudeBot, PerplexityBot, Bytespider) are separate tokens with independent controls — block one, the others still crawl. ChatGPT-User is user-triggered, not automatic, so robots.txt may not apply to it at all; compliance from crawlers generally is voluntary.
• Don’t block render-critical CSS/JS, don’t use it to hide sensitive data (it’s public), and watch for a stray Disallow: /.

Official documentation

Primary-source documentation from the search engines and the standard itself.

Google

• Introduction to robots.txt — the beginner doc, including the crawl-vs-index caveat. Start here.
• Create and submit a robots.txt file / How Google interprets the robots.txt specification — the full reference: supported fields, grouping, matching, wildcards, size limit, caching, and status-code handling.
• Block Search Indexing with noindex — why a noindex page must NOT be blocked in robots.txt.
• A note on unsupported rules in robots.txt (July 2019) — the announcement that noindex, nofollow, and crawl-delay support ends Sept 1, 2019.

The standard

• RFC 9309 — Robots Exclusion Protocol — the IETF standard (Koster, Illyes, Zeller, Sassman; Sept 2022), including the /robots.txt location requirement, longest-match rule, 500 KiB parsing limit, and the security-considerations note.

Bing / Microsoft

• Bingbot guidance — current Bing Webmaster guidance documents crawl-delay values from 1–20 seconds.
• How to Create a robots.txt File (Bing Webmaster Tools help) — Bing’s robots.txt guidance and tester.

Quotes from the source

On-the-record statements from Google, Bing, and the RFC. Each link is a deep link that jumps to the quoted passage on the source page.

Google — crawling, not indexing

• “This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.” — Google Search Central docs. Jump to quote
• “While Google won’t crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web.” Jump to quote
• “If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result won’t have a description.” Jump to quote

Google — why a noindex page must stay crawlable

• “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.” — Google Search Central docs. Jump to quote

Google — location, scope, and the spec

• “The rules listed in the robots.txt file apply only to the host, protocol, and port number where the robots.txt file is hosted.” Jump to quote
• “Google can’t index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippet.” Jump to quote
• “When matching robots.txt rules to URLs, crawlers use the most specific rule based on the length of the rule path. In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule.” Jump to quote
• “* designates 0 or more instances of any valid character. $ designates the end of the URL.” Jump to quote
• “The field name (disallow) is case-insensitive, but its value is case-sensitive.” Jump to quote

Google — size, caching, and status codes

• “Google enforces a robots.txt file size limit of 500 kibibytes (KiB). Content which is after the maximum file size is ignored.” Jump to quote
• “Google generally caches the contents of robots.txt file for up to 24 hours, but may cache it longer in situations where refreshing the cached version isn’t possible.” Jump to quote
• “Google’s crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn’t exist. This means that Google assumes that there are no crawl restrictions.” Jump to quote
• “For the first 12 hours, Google stops crawling the site but keeps trying to fetch the robots.txt file. If Google can’t fetch a new version, for the next 30 days Google will use the last good version, while still trying to fetch a new version.” Jump to quote
• “If the errors are still not fixed after 30 days: If the site is generally available to Google, Google will behave as if there is no robots.txt file (but still keep checking for a new version).” Jump to quote
• “User agent specific groups and global groups (*) are not combined.” Jump to quote

RFC 9309 — the standard

• “This document specifies and extends the ‘Robots Exclusion Protocol’ method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers.” — RFC 9309 (Koster, Illyes, Zeller, Sassman). Jump to quote
• “The rules MUST be accessible in a file named ‘/robots.txt’ (all lowercase) in the top-level path of the service.” Jump to quote
• “The Robots Exclusion Protocol is not a substitute for valid content security measures. Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable.” Jump to quote

Bing — crawl-delay

• Bingbot guidance documents a crawl-delay range of 1–20 seconds. This is Bing-specific guidance, not an extension that Google processes.

Patrick Stox — the blocking experiment (my own Ahrefs experiment, reviewed by Joshua Hardwick)

• “We lost a position here or there and all of the featured snippets for the pages.” Jump to quote
• “Both pages lost some traffic. But it didn’t result in much change to our traffic estimate like I was expecting.” Jump to quote
• “Don’t block pages you want indexed. It hurts. Not as bad as you might think it does—but it still hurts.” Jump to quote

Robots.txt — syntax cheat sheet

Directives (Google-supported)

Special characters

Matching rules

• Most-specific (longest path) rule wins.
• On a tie, least-restrictive wins — Allow beats Disallow.
• Path values are case-sensitive (/Folder/ ≠ /folder/); field and user-agent names are case-insensitive.
• A crawler obeys exactly one group — the most specific user-agent match.

Not supported by Google (retired Sept 1, 2019): noindex, nofollow, crawl-delay. Bing still honors crawl-delay (as a relative throttle, not literal seconds); Yandex stopped honoring it on Feb 22, 2018 — set the crawl rate in Yandex Webmaster instead.

Under the hood (Google)

• Size limit: 500 KiB (rest ignored). Caching: ~24h.
• 4xx (except 429) → no restrictions. 5xx/unreachable → ~12h crawl stop, then last good copy ~30 days, then no restrictions if still broken. 3xx → ≥5 hops then treated as 404.
• A specific User-agent group is never merged with the * group.

Common recipes

# Block a folder for everyone
User-agent: *
Disallow: /admin/

# Block all PDFs
User-agent: *
Disallow: /*.pdf$

# Block AI training crawler (won't stop search bots)
User-agent: GPTBot
Disallow: /

# Point to the sitemap
Sitemap: https://example.com/sitemap.xml

Robots.txt audit checklist

• The file exists at exactly https://yourdomain.com/robots.txt (root, lowercase) and returns a 200.
• Every host and protocol has its own file (www, non-www, subdomains, http vs https).
• No stray Disallow: / blocking the whole site (the classic staging leftover).
• You are not blocking any page you also want to noindex (Google must crawl it to see the noindex).
• You are not using robots.txt to deindex pages — that’s noindex’s job.
• Render-critical CSS/JS is not blocked.
• No noindex, nofollow, or crawl-delay lines aimed at Google (unsupported since Sept 1, 2019).
• No sensitive paths listed — the file is public and advertises whatever it names; use authentication instead.
• Wildcard rules (*, $) tested against real URLs; remember path values are case-sensitive.
• If you have a User-agent: bingbot (or any bot-specific) section, it repeats every directive you still want enforced — bot-specific groups ignore the * defaults.
• Sitemap: directive present with an absolute URL.
• File is under 500 KiB.
• Validated in the Search Console robots.txt report; spot-checked any “Indexed, though blocked by robots.txt” URLs to confirm they’re genuinely junk and not pages you wanted crawled.

The mental models

1. Crawl control, not index control. Robots.txt only governs whether a crawler may fetch a URL. It says nothing about whether the URL gets indexed. Internalize this and most robots.txt mistakes disappear. Blocking a URL ≠ removing it from Google.

2. Most-specific rule wins; least-restrictive breaks ties. When two rules touch the same URL, the one with the longer path wins. On a genuine tie, Allow beats Disallow. This single rule explains nearly every “why is this URL crawlable / blocked?” head-scratcher.

3. Block vs noindex — a decision tree.

• Want it out of the index? → noindex, and make sure it’s not blocked in robots.txt (so Google can crawl it to see the noindex).
• Want to stop wasting crawl on it but don’t care if the bare URL shows? → Disallow in robots.txt.
• Want it genuinely private? → authentication. Robots.txt is public and never hides anything.

Never block and noindex the same URL — they cancel out, and the page can linger in the index.

4. The file’s availability is production-critical. The HTTP response code on /robots.txt has site-wide consequences: a 4xx means “no restrictions,” but a 5xx can stop Google crawling your whole site for ~12 hours. Treat /robots.txt like any other critical endpoint.

Fetch and audit robots.txt across hosts

Robots.txt is scoped per host+protocol+port, so it’s easy to forget one variant exists. This pulls every variant of a domain and flags any that error or 404.

macOS / Linux

for url in "https://example.com/robots.txt" "https://www.example.com/robots.txt" "http://example.com/robots.txt"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" "$url")
  echo "$code  $url"
done

Windows (PowerShell)

$urls = "https://example.com/robots.txt","https://www.example.com/robots.txt","http://example.com/robots.txt"
foreach ($u in $urls) {
  $r = Invoke-WebRequest -Uri $u -UseBasicParsing -SkipHttpErrorCheck
  Write-Host "$($r.StatusCode)  $u"
}

A 200 on every line is what you want. A 4xx means Google treats that host+protocol as unrestricted (fine if intentional); a 5xx is the dangerous one — it can stall crawling for that host.

Regex to pull every Disallow/Allow path

Useful for diffing a robots.txt before/after an edit, or scripting a check against a list of URLs you care about.

import re

robots_txt = open("robots.txt").read()

# Captures the directive (Disallow/Allow) and its path value
pattern = re.compile(r'^(Disallow|Allow):\s*(\S*)', re.IGNORECASE | re.MULTILINE)

for directive, path in pattern.findall(robots_txt):
    print(f"{directive}: {path or '(empty — matches nothing)'}")

• (Disallow|Allow) — captures which directive fired.
• \s*(\S*) — captures the path value, allowing for an empty Disallow: (which Google treats as “disallow nothing”).
• re.IGNORECASE — field names are case-insensitive even though path values aren’t (don’t lowercase the paths themselves when you extend this).

Chrome DevTools Console snippet

Paste into the Console panel (F12 → Console) on any page to fetch and print that host’s robots.txt without leaving the tab:

fetch(new URL('/robots.txt', location.origin))
  .then(r => r.text())
  .then(t => console.log(t))
  .catch(e => console.error('No robots.txt or fetch blocked:', e));

Bookmarklet: jump to the current site’s robots.txt

Save as a bookmark with this as the URL (drag it to your bookmarks bar, or add it manually and paste the code into the “URL” field), then click it on any page:

javascript:(function(){var u=new URL('/robots.txt',location.origin).href;location.href=u;})();

Tools for testing and validating robots.txt

• Google Search Console — robots.txt report (Settings) — Google’s current surface for seeing the robots.txt Google fetched, its status, and any parsing issues. (The old standalone robots.txt tester was retired; testing now lives in this report.)
• GSC URL Inspection — confirm whether a specific URL is blocked from crawling and how Google currently sees it.
• GSC “Indexed, though blocked by robots.txt” (Pages report) — surfaces URLs Google indexed despite the block, so you can decide which are real problems.
• Bing Webmaster Tools — robots.txt tester / guidance — Bing’s equivalent for validating your file against bingbot.
• Ahrefs Site Audit / Screaming Frog SEO Spider — crawl your own site to see which URLs are blocked, surface accidentally-disallowed pages, and test that your rules match the URLs you intend.
• Any browser — the simplest check of all: load yourdomain.com/robots.txt and read it. Remember everyone else can do the same.

Incident: you shipped Disallow: / to production

This is the classic staging leftover — a robots.txt built to block a staging environment gets deployed to the live site, and User-agent: * / Disallow: / blocks the whole thing. Here’s the ordered recovery.

Step 1 — confirm the blast radius. Fetch https://yourdomain.com/robots.txt right now (browser or curl -I). If you see a bare Disallow: / under User-agent: * with no narrower Allow: carving anything back out, every crawler that respects robots.txt is currently blocked from your entire site. Continue to Step 2.

Step 2 — fix the file and confirm the deploy. Replace it with the intended rules (or a minimal User-agent: * / Allow: / if you’re not sure what belongs) and deploy. Re-fetch the live URL — don’t trust your deploy dashboard, trust the actual response. If it still shows the old Disallow: /, you have a caching/CDN problem, not a code problem — purge the cache for that one path before moving on.

Step 3 — check how long the site was blocked. Google caches robots.txt for up to ~24 hours, so a fix that just went live won’t be picked up instantly. Open Search Console’s robots.txt report to see when Google last fetched the file and whether it’s already seen the corrected version. If it hasn’t yet, request it be re-fetched isn’t a manual option — you wait, or use URL Inspection on your most important pages and use “Request Indexing” once the file is confirmed fixed.

Step 4 — check for actual damage.

• If the block lasted hours, expect little to no visible impact — Google doesn’t drop pages from the index just because one crawl was blocked.
• If it lasted days to weeks, check Search Console’s Pages report for a jump in “Blocked by robots.txt” and check whether previously-indexed pages moved to that status. My own experiment blocking two high-ranking pages found they stayed indexed and mostly kept ranking, but lost freshness signals and featured snippets — the damage compounds with time blocked, not instantly.
• If you see rankings/impressions actually falling in Search Console’s Performance report for the blocked window, that’s your confirmation the incident had real impact, not just a scare.

Step 5 — verify the fix took, don’t just assume it. Run the specific paths you care about through the Robots.txt Tester against the corrected file. If any come back blocked, you’re not done — go back to Step 2.

Concrete mistakes to avoid

Blocking a page in robots.txt that you also want noindexed. Why it’s wrong: a noindex only works if the crawler can read the page to see it. Block the page and Google never crawls it, never sees the noindex, and the page can linger in the index if anything links to it. Do instead: leave the page crawlable and put noindex on the page itself (meta tag or X-Robots-Tag header).

Using robots.txt as your deindexing tool. Why it’s wrong: robots.txt controls crawling, not indexing. Google’s own docs are explicit that it “is not a mechanism for keeping a web page out of Google.” A disallowed URL can still show up in results, just without a description. Do instead: use noindex (page not blocked) to actually remove a page from the index.

Blocking render-critical CSS or JavaScript. Why it’s wrong: Google renders your pages with a real browser to see them the way a user does. Block the assets that build the layout or inject content and Google may see a broken or empty page. Do instead: leave /assets/, /static/, or wherever your CSS/JS lives crawlable — Google’s own sample robots.txt explicitly re-allows these.

Listing sensitive paths in robots.txt to “hide” them. Why it’s wrong: the file is public — anyone can read it — so Disallow: /admin-panel/ just tells the world that folder exists. RFC 9309 is blunt that the protocol “is not a substitute for valid content security measures.” Do instead: use authentication or a password wall for anything that actually needs to stay private.

Trailing-slash confusion in a Disallow path. Why it’s wrong: Disallow: /folder (no trailing slash) blocks /folder, /folder/, /folder-name/, and /folder.html — anything that starts with that string, because matching is prefix-based, not directory-based. Do instead: add the trailing slash (Disallow: /folder/) when you only mean the directory, and test the exact prefix behavior against your real URLs before shipping.

Assuming rules are case-insensitive. Why it’s wrong: field names are case-insensitive but path values are case-sensitive — Disallow: /Folder/ does nothing to /folder/. Do instead: match the exact case your URLs actually use, and don’t assume a rule “should” work because it reads correctly to a human.

Common issues

”I blocked this page, but it’s still showing up in Google”

• Symptom: the URL still appears in search results (usually with no description) even though it’s disallowed in robots.txt.
• Likely cause: the URL is linked from elsewhere on the web (or your own site), so Google indexed the bare URL without ever crawling its content. Blocking crawling was never going to remove it — that’s not what robots.txt does.
• Fix: if you want it gone, add noindex to the page and make sure it’s not blocked in robots.txt (temporarily allow crawling if it currently is). Confirm the fix with URL Inspection in Search Console once Google has re-crawled it.

”Search Console says ‘Indexed, though blocked by robots.txt’”

• Symptom: the Pages report in GSC flags this exact status for one or more URLs.
• Likely cause: usually harmless — it’s common for utility URLs (cart, filtered/faceted variants, session parameters) that got linked somewhere and indexed as a bare URL despite the block.
• Fix: check whether the flagged URLs are pages you actually wanted indexed. If they’re junk parameter URLs, this status is expected and not worth chasing. If a page you care about shows up here, that’s the “blocked + noindex” trap — go crawlable + noindex instead.

”I edited robots.txt, but the change doesn’t seem to be live”

• Symptom: you deployed a fix, but a tool or Google still seems to be reading the old rules.
• Likely cause: Google caches robots.txt for up to ~24 hours, so a change isn’t picked up instantly. Separately, a CDN or reverse proxy in front of your site can be serving a stale cached copy of the file itself.
• Fix: first confirm the live file actually changed — fetch yourdomain.com/robots.txt directly (not through a cache) or run it through the Robots.txt Tester. If the live file is correct, the delay is Google’s cache and just needs time; check the robots.txt report in Search Console to see when it last fetched.

”The tester says a path is allowed, but Google still isn’t crawling it”

• Symptom: robots.txt clearly permits the URL, but server logs or Search Console show no recent crawl activity for it.
• Likely cause: robots.txt allowing a URL isn’t a crawl guarantee — it’s necessary but not sufficient. The page might be orphaned (nothing links to it), missing from the sitemap, or simply low priority in Google’s crawl scheduling.
• Fix: confirm the page is discoverable (internal links, sitemap inclusion) rather than re-checking robots.txt again — the file isn’t the bottleneck here.

”My Disallow rule doesn’t seem to match the URLs I expect”

• Symptom: a rule that looks correct isn’t blocking (or is blocking too much of) the URLs you intended.
• Likely cause: case sensitivity (/Folder/ ≠ /folder/) or prefix matching catching more than expected (Disallow: /folder also matches /folder-name/).
• Fix: re-check the exact case of your live URLs, and add a trailing slash if you meant only the directory. Test the specific paths in the Robots.txt Tester before trusting the rule.

”Bing stopped following my default rules after I added a bingbot section”

• Symptom: rules under User-agent: * that used to apply to Bing seem to be ignored after you added a User-agent: bingbot group.
• Likely cause: Bing (like Google) applies only the most specific matching group to a crawler. A bingbot-specific group makes Bing ignore the * defaults entirely (crawl-delay excepted).
• Fix: repeat every directive you still want enforced inside the bingbot group — it has to be self-contained.

Validation tests

Run these after you edit robots.txt, before you consider the change done.

Test 1: the file itself is live and correct

• Test to run: curl -I https://yourdomain.com/robots.txt (or open the URL in a browser), and separately run it through the Robots.txt Tester.
• Expected result: HTTP/1.1 200 and the tester shows the exact rules you intended to ship.
• Failure interpretation: a non-200 status means the file isn’t reachable as expected (check deploy/routing); correct-looking rules that don’t match what you edited usually mean a cache (CDN or Google’s ~24h cache) is serving a stale copy.
• Monitoring window: immediate for the status code and content; allow up to ~24 hours before assuming Google has picked up the change.
• Rollback trigger: the file 404s or 5xxs unexpectedly, or shows rules you didn’t intend to ship.

Test 2: the specific path you changed behaves as intended

• Test to run: paste the exact URL you meant to allow or block into the Robots.txt Tester and check the verdict for Googlebot (and Bingbot if relevant).
• Expected result: the tool reports “allowed” for a path you meant to open up, or “blocked” for a path you meant to close off.
• Failure interpretation: the opposite verdict usually means a longer/more-specific rule elsewhere in the file is overriding the one you just added — check for a competing Allow/Disallow on a shorter or overlapping path.
• Monitoring window: immediate — this is a rules-matching check, not a live-crawl check.
• Rollback trigger: the tested path returns the wrong verdict after you’ve confirmed the live file matches what you edited.

Test 3: a page you unblocked actually gets crawled

• Test to run: Search Console URL Inspection on the specific URL, looking at “Crawl allowed?” and the last crawl date.
• Expected result: “Crawl allowed: Yes,” and a crawl date after your robots.txt change.
• Failure interpretation: “Crawl allowed: No” after you believe you unblocked it means Google is still reading cached rules, or another rule in the file still matches the path. No new crawl date yet doesn’t necessarily mean failure — Google may not have gotten to it yet.
• Monitoring window: 2–4 weeks for Google to actually re-crawl and reflect the change, longer on large or low-priority sites.
• Rollback trigger: “Crawl allowed” flips back to “No” on a later check with no robots.txt change on your end — investigate a CDN/cache issue.

Test 4: you didn’t accidentally block something else

• Test to run: crawl your site with Ahrefs Site Audit or Screaming Frog SEO Spider after the change and compare the “blocked by robots.txt” URL list before vs. after.
• Expected result: only the URLs you intended to newly block or unblock moved between lists — nothing else changed.
• Failure interpretation: an unexpected page showing up as newly blocked usually means a wildcard or prefix rule matched more broadly than intended.
• Monitoring window: immediate — run this crawl right after deploying.
• Rollback trigger: any page you rely on for organic traffic appears newly blocked.

Ready-to-copy prompts

Paste your actual robots.txt content (and, for the second prompt, the list of URLs you care about) after each prompt.

Audit a robots.txt file for conflicts and risky rules

I'm going to paste a robots.txt file. Read it as a technical SEO would and
flag:
1. Any Disallow rule that conflicts with an Allow rule on the same or an
   overlapping path (tell me which one wins under longest-match rules).
2. Any bare "Disallow: /" under a User-agent group — that blocks everything
   for that crawler.
3. Any rule that would block commonly render-critical paths (CSS, JS, fonts,
   images used for layout).
4. Any use of noindex, nofollow, or crawl-delay under a Google-facing
   User-agent — these are not supported by Google.
5. Anything that looks like it's trying to hide a sensitive path (remember
   this file is public).

Here is the file:
<paste robots.txt content>

Check whether specific URLs are safely crawlable and indexable

I want the following URLs to be crawlable AND indexable by Google. Given this
robots.txt file, tell me for each URL whether it would be blocked from
crawling, and separately remind me that robots.txt says nothing about
indexing — a URL can be blocked from crawling but still indexed if it's
linked elsewhere, and a URL can be crawlable but still noindexed via a meta
tag or header this file can't show you.

Robots.txt:
<paste robots.txt content>

URLs:
<paste list of URLs>

Resources worth your time

My related writing

• The Story of Blocking 2 High-Ranking Pages With Robots.txt — my own experiment: I deliberately blocked two high-ranking Ahrefs pages and tracked what actually happened to rankings and traffic. The proof that blocking ≠ deindexing.
• The Beginner’s Guide to Technical SEO — where robots.txt fits in the crawl/index foundation.
• Enterprise SEO Strategies — managing robots.txt at scale on large sites.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawl → render → index, the pipeline robots.txt gates at the crawl step. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• Robots.txt and SEO: Everything You Need to Know — the canonical Ahrefs guide, by Joshua Hardwick (not me) — a thorough syntax-and-recipes reference.
• When Unindexing Pages From Google Search Use Noindex, Not Robots.txt — Search Engine Roundtable coverage of John Mueller’s guidance: don’t block with robots.txt when you want pages out of the index, use noindex instead.
• Google Explains Why URLs Blocked By Robots.txt Can Still Be Indexed — Search Engine Journal coverage of Mueller’s “add-to-cart” example: blocked URLs can still be indexed, and “Indexed, though blocked by robots.txt” is often harmless for utility pages.
• Robots.txt Tip from Bing: Include All Relevant Directives If You Have a Bingbot Section — Search Engine Land on the bingbot-section gotcha: a Bingbot-specific group makes Bing ignore the wildcard defaults, so you must repeat every directive you still want enforced.
• Overview of OpenAI Crawlers (GPTBot, OAI-SearchBot, ChatGPT-User) — OpenAI’s official list of their crawler user-agent tokens, with robots.txt examples for blocking training vs. search vs. user-triggered fetches separately.
• r/TechSEO — the community for crawl/index and “Indexed, though blocked by robots.txt” debugging.

Test yourself: Robots.txt

Five quick questions on what robots.txt controls (and what it does not). Pick an answer for each, then check.

Robots.txt

A plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.

Related: Crawling, Crawl Budget

Robots.txt

Robots.txt is a plain-text file served from the root of a host — exactly https://example.com/robots.txt — that tells crawlers which URLs they may and may not request. It implements the Robots Exclusion Protocol, originally created by Martijn Koster in 1994 and standardized as RFC 9309 in 2022.

The single most important thing to understand is that robots.txt is a crawl-control file, not an index-control file. Blocking a URL stops Google from fetching it, but the URL can still be indexed — shown as a bare URL with no description — if other pages link to it. To keep a page out of the index you need noindex, and the page must not be blocked in robots.txt, or Google never crawls it to see the noindex.

The rules apply only to the exact host, protocol, and port the file is served from. Each subdomain and protocol needs its own file.

Related: Crawling, Crawl Budget

Changelog