Robots.txt

TL;DR — Robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. is a text file at the root of your site that tells crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. which URLs they’re allowed to request. The one thing everyone gets wrong: it controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Blocking a page does not remove it from Google — a blocked page can still show up in search (just without a description). If you want a page out of Google, that’s a different tool (noindex), and you must not block it in robots.txt or Google never sees the instruction.

What a robots.txt file is

When a search engine wants to crawl your site, the first thing it looks for is a file at https://yourdomain.com/robots.txt. That file is a list of instructions telling crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. which parts of your site they may and may not request. It’s just plain text, and you can read any site’s robots.txt by typing /robots.txt after the domain.

A minimal one looks like this:

User-agent: *
Disallow: /admin/

Sitemap: https://yourdomain.com/sitemap.xml

That says: to every crawler (*), don’t request anything under /admin/, and by the way, here’s where my sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. is.

The one thing to understand: blocking ≠ removing

This is the single most common mistake in all of SEO, so I’ll say it loudly: robots.txt stops a page from being crawled, not from being indexed. Evidence for this claim A robots.txt rule controls crawling rather than guaranteeing removal from Google Search; a URL can still appear when Google cannot crawl it. Scope: Google Search crawler behavior. Other crawlers can interpret robots.txt differently. Confidence: high · Verified: 2026-07-14 Google: Introduction to robots.txt

Those sound like the same thing. They aren’t. CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is the engine fetching and reading your page. IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. is the engine deciding to store it and show it in results. Robots.txt only blocks the first step.

So if you block a page in robots.txt, Google won’t fetch it — but if other pages link to that URL, Google can still index the URL itself and show it in search. It just shows up as a bare link with no description underneath, because Google was never allowed to read the page to write one.

If your actual goal is “get this page out of Google,” robots.txt is the wrong tool. You want noindex (a meta tag on the page or an HTTP header). And here’s the trap: a noindex only works if Google can crawl the page to see it. If you block the page in robots.txt and add a noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed., Google never crawls it, never sees the noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed., and the page can linger in the index anyway. Pick one job: block crawling or block indexing, not both on the same URL.

The right tool for each job

• “Don’t waste crawling on this.” → robots.txt Disallow.
• “Keep this out of Google’s index.” → noindex (and don’t block it).
• “Nobody should be able to see this at all.” → password protection / login. Robots.txt is public — anyone can read it — so listing /secret/ in it just tells the world the folder exists.

A few rules that trip people up

• One file per host. https://example.com, https://www.example.com, and https://blog.example.com each need their own robots.txt. They don’t share.
• It lives at the root. It has to be at /robots.txt, all lowercase. A file in a subfolder does nothing.
• Don’t block your CSS and JavaScript. Google needs those files to see your page the way a visitor does. Blocking them can hurt you.

Want the full syntax reference — wildcards, the Allow/Disallow matching rules, how Google handles a 404 vs a server error on your robots.txt, crawl-delayCrawl-delay is a non-standard robots.txt directive that asks bots to wait between successive page fetches to ease server load. Google has ignored it since September 1, 2019 and Yandex dropped it on February 22, 2018; Bing and many other crawlers still honor it, but Bing interprets the value differently than the plain reading., and how to manage AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. — switch to the Advanced tab.

TL;DR — Robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. is a plain-text file at the root of each host (/robots.txt, lowercase) that implements the Robots Exclusion Protocol (RFC 9309). It controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — a disallowed URL can still be indexed without a snippet if linked elsewhere; to deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. use noindex on a page that is not blocked. Google supports only user-agent, allow, disallow, and sitemap; noindex/nofollow/crawl-delay were dropped Sept 1, 2019. Scope is one host+protocol+port. Matching uses the most-specific (longest) rule, least-restrictive on ties; * and $ are the wildcards; paths are case-sensitive. Google caps the file at 500 KiB, caches ~24h, treats 4xx (except 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content.) as no-restrictions, and on a 5xx stalls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. for ~12h then falls back to the last good copy for ~30 days. Don’t block render-critical CSSCritical CSS is a performance technique that extracts the styles needed to render a chosen above-the-fold view, inlines them in the <head>, and defers the rest of the stylesheet. It speeds up first paint but is an advanced, optional optimization — Google says most sites don't need it — and carries real production risks around caching, CSP, and page state./JS, and don’t treat it as access control — the file is public.

What it is and where it lives

Robots.txt implements the Robots Exclusion Protocol, created by Martijn Koster in 1994 and finally standardized in 2022 as RFC 9309 — co-authored by Google’s Gary Illyes, Henner Zeller, Lizzi Sassman, and Koster himself. The standard’s own wording: “This document specifies and extends the ‘Robots Exclusion Protocol’ method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index..”

A few facts that catch people out:

• It must be at the root, lowercase. RFC 9309 is explicit: “The rules MUST be accessible in a file named ‘/robots.txt’ (all lowercase) in the top-level path of the service.” Google adds that the URL itself is case-sensitive, like any URL.
• Scope is one host + protocol + port. Google: “The rules listed in the robots.txt file apply only to the host, protocol, and port number where the robots.txt file is hosted.” So https://example.com, https://www.example.com, https://blog.example.com, and http://example.com each need their own file. Subdomains and protocols don’t share one.
• Supported protocols for Google are HTTP, HTTPSHTTPS is the encrypted version of HTTP — it uses TLS to authenticate the server and protect data in transit between a browser and a website. Google announced it as a lightweight ranking signal in 2014 and today conditionally prefers HTTPS pages as canonical; Chrome marks plain HTTP pages 'Not Secure.', and FTP.

The misconception that defines this topic: crawling vs indexing

If you take one thing from this page, take this: robots.txt controls crawling, not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Blocking a URL is not the same as removing it from Google. Evidence for this claim A robots.txt rule controls crawling rather than guaranteeing removal from Google Search; a URL can still appear when Google cannot crawl it. Scope: Google Search crawler behavior. Other crawlers can interpret robots.txt differently. Confidence: high · Verified: 2026-07-14 Google: Introduction to robots.txt

Google’s own intro doc says it plainly: robots.txt “is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.” And on what actually happens to a blocked URL: “While Google won’t crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web.” The result is the familiar snippet-less listing: “its URL can still appear in search results, but the search result won’t have a description.”

The spec restates the same nuance for the disallow rule itself: “Google can’t index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippet.”

Why you must NOT block a page you want to noindex

This is the trap that quietly breaks deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. efforts. A noindex only works if Google can crawl the page to read it. Google’s block-indexing doc spells out the dependency: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.” Evidence for this claim Google must be able to crawl a URL to see a noindex rule; blocking the URL in robots.txt can prevent the rule from being observed. Scope: Google Search indexing controls for HTML meta robots and X-Robots-Tag rules. Confidence: high · Verified: 2026-07-14 Google: Block indexing with noindex

So if your goal is to get a page out of the index, John Mueller’s guidance is the cleanest way to remember it: when you want to unindex pages, you should not block Google with robots.txt, but rather use noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed..

The lived proof: I blocked two of our own high-ranking pages

I don’t have to argue this from theory. In my experiment blocking two high-ranking Ahrefs pages, I deliberately blocked them in robots.txt and tracked what happened. The pages stayed indexed and kept ranking — they didn’t vanish. What we lost was the freshness Google gets from re-crawling: “We lost a position here or there and all of the featured snippets for the pages.” Traffic dropped, but less than I expected: “Both pages lost some traffic. But it didn’t result in much change to our traffic estimate like I was expecting.”

My takeaway from the data: “Accidentally blocking pages (that Google already ranks) from being crawled using robots.txt probably isn’t going to have much impact on your rankings, and they will likely still show in the search results.” And the blunt version: “Don’t block pages you want indexed. It hurts. Not as bad as you might think it does—but it still hurts.”

The flip side is reassurance: when Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. flags “Indexed, though blocked by robots.txt” for a utility URL — cart, filter, parameter junk — it’s usually a non-issue. As Mueller put it about add-to-cart URLs, blocking them is fine, and even if they get “indexed,” it’s unlikely they’ll be shown in search unless someone runs a very specific query for those URLs, which real users don’t do. Distinguish the scary-sounding warning from an actual problem: it only matters if the blocked URL is a page you actually wanted crawled and indexed.

The syntax (the reference)

A robots.txt is a set of groups. Each group starts with one or more User-agent lines naming which crawler(s) it applies to, followed by the rules for them.

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

User-agent: Googlebot
Disallow: /no-google/

Sitemap: https://example.com/sitemap.xml

User-agent and groups. A crawler obeys exactly one group — the one with the most specific user-agent that matches it — and ignores the rest. Google: “Google’s crawlers determine the correct group of rules by finding in the robots.txt file the group with the most specific user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. that matches the crawler’s user agent. Other groups are ignored.” And: “Only one group is valid for a particular crawler.” (Bing behaves the same way — more on that below.)

That also means a specific group doesn’t get topped up with the wildcard group’s rules — it’s used on its own, not merged with User-agent: *. Google’s spec is explicit that “user agent specific groups and global groups (*) are not combined.” So if you write a User-agent: googlebot-news group, it has to be self-contained: anything you still want it to obey from the * group has to be repeated inside it, or GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.-News simply won’t see those rules at all.

Disallow and Allow. Disallow lists paths a crawler must not request; Allow carves exceptions back out. The disallow rule “specifies paths that must not be accessed by the crawlers identified by the user-agent line the disallow rule is grouped with.” The allow rule “specifies paths that may be accessed by the designated crawlers. When no path is specified, the rule is ignored.”

The matching rule (most guides get this wrong). When two rules conflict, the most specific one wins, and “most specific” means longest path: “When matching robots.txt rules to URLs, crawlers use the most specific rule based on the length of the rule path. In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule.” So on a genuine tie, the least restrictive rule wins — Allow beats Disallow. RFC 9309 frames it as the “Longest Match”: “The following example shows that in the case of two rules, the longest one is used for matching.” Evidence for this claim Google resolves matching robots.txt rules by path specificity and uses the least restrictive rule when equally specific rules conflict. Scope: Google crawler interpretation of robots.txt rules; other crawlers may implement different extensions. Confidence: high · Verified: 2026-07-14 Google: Robots.txt interpretation

Worked example:

User-agent: *
Allow: /folder/page
Disallow: /folder/

The URL /folder/page matches both rules. Allow: /folder/page (12 chars) is longer than Disallow: /folder/ (8 chars), so the longer, more specific Allow wins and the page is crawlable.

Wildcards * and $. Google: “* designates 0 or more instances of any valid character. $ designates the end of the URL.” So Disallow: /*.pdf$ blocks every URL ending in .pdf, and Disallow: /*? blocks every URL containing a query string. Matching is prefix-based: Disallow: /fish matches /fish, /fish.html, and /fish/salmon.html, but not /Fish (case-sensitive) or /catfish (it’s a prefix, not a substring).

Case sensitivity (the subtle one). Field and user-agent names are case-insensitive; path values are case-sensitive. Google: “Both the user-agent field name and its value are case-insensitive,” but “The field name (disallow) is case-insensitive, but its value is case-sensitive,” and “The path value must start with / to designate the root and the value is case-sensitive.” So Disallow: /Folder/ does not block /folder/.

SitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing.. The Sitemap: directive takes a full absolute URL and is independent of groups — it can sit anywhere in the file.

Comments. Anything after # is ignored: “To include comments, precede your comment with the # character.”

noindex, nofollow, and crawl-delay are NOT robots.txt directives

This is a persistent myth. As of September 1, 2019, Google retired support for unsupported, undocumented rules — including noindex, nofollow, and crawl-delay. Google’s announcement focused on rules unsupported by the internet draft, such as crawl-delay, nofollowrel=\"nofollow\" is a value of the HTML link rel attribute that tells search engines you don't vouch for a linked page and don't want to pass ranking signals to it. Since 2019–2020 Google treats it as a hint, not a directive — and it does not reliably block crawling or indexing., and noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed., noting they were never documented by Google, and said Google was retiring all code that handles unsupported and unpublished rules (such as noindex) on that date. The supported field list is short, and the spec calls out the exclusion directly: Google supports user-agent, allow, disallow, and sitemap, and “other fields such as crawl-delay aren’t supported.”

If you relied on noindex in robots.txt, the alternatives are a noindex meta tag or X-Robots-TagThe X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos. header, 404/410 status codes, password protection, a Disallow, or the Search Console removal tool.

How Google handles robots.txt under the hood

• Size limit: 500 KiB. “Google enforces a robots.txt file size limit of 500 kibibytes (KiB). Content which is after the maximum file size is ignored.” RFC 9309 aligns: “The parsing limit MUST be at least 500 kibibytes [KiB].”
• Caching: ~24 hours. “Google generally caches the contents of robots.txt file for up to 24 hours, but may cache it longer in situations where refreshing the cached version isn’t possible.” So a change isn’t necessarily picked up instantly. Evidence for this claim Google generally caches robots.txt for up to 24 hours and changes crawling behavior according to the HTTP status returned for the file. Scope: Google crawler handling of robots.txt fetches, including documented 4xx, 5xx, and redirect behavior. Confidence: high · Verified: 2026-07-14 Google: Robots.txt file handling
• Status codes matter site-wide. This is the part most guides skip:
  • 4xx (except 429) → no restrictions. “Google’s crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn’t exist. This means that Google assumes that there are no crawl restrictions.” A 404 on /robots.txt means “crawl everything.” (Don’t use 401/403 to throttle crawling.)
  • 5xx / unreachable → dangerous. “For the first 12 hours, Google stops crawling the site but keeps trying to fetch the robots.txt file. If Google can’t fetch a new version, for the next 30 days Google will use the last good version, while still trying to fetch a new version.” So a server error on /robots.txt can effectively disallow your whole site for the first ~12 hours, then run on the last cached copy for ~30 days. A persistently erroring robots.txt is a site-wide crawl risk. And if it’s still broken after those 30 days: “If the errors are still not fixed after 30 days: If the site is generally available to Google, Google will behave as if there is no robots.txt file (but still keep checking for a new version).” In other words, a robots.txt that never recovers doesn’t stay disallowed forever — Google eventually falls back to crawling with no restrictions, the same as a 404.
  • 3xx → Google follows at least five redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. hops, then treats it as a 404.

robots.txt in Bing, Yandex, and beyond

The grouping and syntax are essentially shared, but two divergences matter:

• crawl-delay. Google ignores it, Bing still honors it, and Yandex dropped it in 2018 — Yandex’s own documentation states that “From February 22, 2018, Yandex doesn’t take into account the Crawl-delay directive,” pointing you to the site crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. setting in Yandex WebmasterYandex Webmaster (webmaster.yandex.com) is Yandex's free search console — the Yandex equivalent of Google Search Console — for verifying a site, monitoring indexing and queries, requesting reindexing, running diagnostics, and setting a site region. It's essential for any site targeting Russia and the wider CIS, where Yandex, not Google, is the dominant search engine. instead. Bing is explicit that “The robots.txt file is the only valid place to set a crawl-delay directive for MSNBot,” and that the directive “accepts only positive, whole numbers as values… the higher the value, the more throttled down the crawl rate will be.” Note Bing treats the value as a relative throttle, not literally N seconds.
• The bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.-section gotcha. Just like Google’s “only one group per crawler” rule, if you create a User-agent: bingbot section, Bing applies only that section and ignores the User-agent: * defaults (crawl-delay excepted). So a bingbot-specific group must repeat every directive you still want enforced.
• Amazon’s cache and failure behavior. Amazon says its crawlers may use a robots.txt copy cached within the previous 30 days. If they cannot fetch the file, they behave as though it does not exist. A checker can report the copy it fetched, but it cannot prove which cached version Amazon used—or that Amazon observed the same failure as the checker. Evidence for this claim Amazon says its crawlers may use a robots.txt copy cached within the previous 30 days and behave as though the file does not exist when they cannot fetch it. Scope: Amazon crawler behavior only; a checker result cannot establish which cached copy Amazon used or whether Amazon observed the same fetch failure. Confidence: high · Verified: 2026-07-19 Amazon: Amazonbot

Managing AI crawlers with robots.txt

Robots.txt is currently the main lever for managing AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls., and they obey the same group/user-agent syntax. The catch: these are separate tokens, so blocking one doesn’t block the others.

• OpenAI runs several distinct bots, and the controls for each are independent — allowing one doesn’t allow the others, and blocking one doesn’t block the others. GPTBot crawls content for training OpenAI’s models; OAI-SearchBot surfaces sites in ChatGPT’s search features; OAI-AdsBot checks the safety of pages submitted as ads (its data isn’t used for training). Block training with User-agent: GPTBot / Disallow: / — that alone won’t stop the search or ads bots. ChatGPT-User is different again: it fires for actions a person triggers inside ChatGPT or a Custom GPT, not automatic crawling, and OpenAI says “robots.txt rules may not apply” to it — so don’t count on a Disallow to keep it out. If you do change what OAI-SearchBot can crawl, OpenAI notes it can take about 24 hours for the update to reach their search systems.
• Google-Extended controls Gemini/Vertex training and is separate from Googlebot.
• Others worth naming: CCBot (Common Crawl), ClaudeBot (Anthropic), PerplexityBot, and Bytespider.

The hard caveat: compliance is voluntary. Robots.txt requests; it doesn’t enforce. Well-behaved crawlers obey it; scrapers can and do ignore it. If you truly need to keep something away from a bot, that’s an authentication/blocking problem, not a robots.txt one.

Common mistakes (and the fixes)

The file is 200, but it is not actually a usable robots file. Status alone is not enough. Capture the response Content-Type and first bytes: a CDN/custom-error template can return HTML at /robots.txt with 200, which must be a warning rather than an “allow all” pass. Google documents robots.txt as UTF-8 plain text and may ignore invalid characters. A single UTF-8 BOM at the beginning is tolerated, but a second BOM, a BOM in the middle, UTF-16 bytes, NULs, or invisible/control characters can alter the first token or invalidate a line. Show the byte offset and affected line; do not silently normalize the file before telling the user what the crawler received. Apply Google’s effective 500 KiB parsing limit before calculating allow/disallow results, while still reporting the discarded tail.

• Blocking a page you also want deindexed. Block + noindex means Google never crawls it to see the noindex. Use noindex without the block.
• Using robots.txt to deindex. Wrong tool entirely — that’s noindex’s job.
• Blocking render-critical CSSCritical CSS is a performance technique that extracts the styles needed to render a chosen above-the-fold view, inlines them in the <head>, and defers the rest of the stylesheet. It speeds up first paint but is an advanced, optional optimization — Google says most sites don't need it — and carries real production risks around caching, CSP, and page state./JS. Google needs those assets to see the page as a user does; Google’s own sample robots.txt explicitly re-allows .css/.js so Googlebot can crawl them.
• Trying to hide sensitive data. RFC 9309 is blunt: “The Robots Exclusion Protocol is not a substitute for valid content security measures. Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable.” Disallowing /secret-admin/ literally advertises it. Use auth.
• A stray Disallow: /. This blocks the entire site for the named crawler — the classic staging leftover that takes a site out of Google.
• Ignoring the response code on /robots.txt. A 5xx can stall crawling site-wide; treat the file’s availability as production-critical.

For the broader pipeline this sits inside — discovery, the crawl scheduler, renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., and how crawling differs from indexing — see the crawling hub. The sibling topics (crawl budgetThe number of URLs an engine will crawl in a timeframe., and how Google handles sitemaps) each go deeper on one piece of this.

TIP Test the decision and lint the whole file

Catch dangerous rules, parser mistakes, and crawler-specific gotchas before they reach production with my free Robots.txt Tester Free

1. Fetch the live file or paste the exact candidate file you plan to ship.
2. Test real crawler and URL combinations, then work through every file-health error and warning.
3. Publish and rerun the deployed file; an allowed result still does not mean the URL is indexed.

Tap to expand

The focused finding reads: Error, line 1. This rule appears before any User-agent line, so no crawler will ever apply it.

AI summary

A condensed take on the Advanced version:

• Robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. = a plain-text file at the root of each host (/robots.txt, lowercase) implementing the Robots Exclusion Protocol (RFC 9309, 2022; origin 1994). The URL is case-sensitive.
• It controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. This is the #1 myth. A disallowed URL can still be indexed — shown as a bare URL with no description — if it’s linked from elsewhere. To keep a page out of the index, use noindex, and the page must not be blocked in robots.txt or Google never crawls it to see the noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed..
• Patrick’s experiment proved it: blocking two high-ranking pages didn’t deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. them — they kept ranking but lost featured snippets and descriptions. “Don’t block pages you want indexed. It hurts.”
• Right tool for each job: robots.txt = crawl control; noindex = index control; password/auth = secrecy (robots.txt is public).
• Supported fields: user-agent, allow, disallow, sitemap. Not supported by Google: noindex, nofollow, crawl-delay (retired Sept 1, 2019).
• Scope is one host + protocol + port — every subdomain/protocol needs its own file.
• Matching: most-specific (longest path) rule wins; least-restrictive (Allow) on ties. Wildcards * and $; path values are case-sensitive. A specific group isn’t merged with the * group — it stands alone.
• Under the hood: 500 KiB size limit; ~24h cachingCaching stores a copy of a page or resource — in a browser, a CDN edge node, or a search crawler's own cache — so it can be served again without regenerating or re-downloading it. It isn't a direct ranking factor, but it feeds page speed and crawl efficiency.; 4xx (except 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content.) → no restrictions; 5xx/unreachable → ~12h crawl stop, then last good copy for ~30 days, then no restrictions if still broken; 3xx → follows ≥5 hops then treats as 404.
• Other engines: Bing honors crawl-delay; Google ignores it and Yandex stopped honoring it in February 2018 (use Yandex WebmasterYandex Webmaster (webmaster.yandex.com) is Yandex's free search console — the Yandex equivalent of Google Search Console — for verifying a site, monitoring indexing and queries, requesting reindexing, running diagnostics, and setting a site region. It's essential for any site targeting Russia and the wider CIS, where Yandex, not Google, is the dominant search engine.’s crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. setting). A bingbot section makes Bing ignore the * defaults.
• AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. (GPTBot, OAI-SearchBot, OAI-AdsBot, ChatGPT-User, Google-Extended, CCBot, ClaudeBot, PerplexityBot, Bytespider) are separate tokens with independent controls — block one, the others still crawl. ChatGPT-User is user-triggered, not automatic, so robots.txt may not apply to it at all; compliance from crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. generally is voluntary.
• Don’t block render-critical CSSCritical CSS is a performance technique that extracts the styles needed to render a chosen above-the-fold view, inlines them in the <head>, and defers the rest of the stylesheet. It speeds up first paint but is an advanced, optional optimization — Google says most sites don't need it — and carries real production risks around caching, CSP, and page state./JS, don’t use it to hide sensitive data (it’s public), and watch for a stray Disallow: /.

Official documentation

Primary-source documentation from the search engines and the standard itself.

Google

• Introduction to robots.txt — the beginner doc, including the crawl-vs-indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. caveat. Start here.
• Create and submit a robots.txt file / How Google interprets the robots.txt specification — the full reference: supported fields, grouping, matching, wildcards, size limit, cachingCaching stores a copy of a page or resource — in a browser, a CDN edge node, or a search crawler's own cache — so it can be served again without regenerating or re-downloading it. It isn't a direct ranking factor, but it feeds page speed and crawl efficiency., and status-code handling.
• Block Search Indexing with noindex — why a noindex page must NOT be blocked in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere..
• A note on unsupported rules in robots.txt (July 2019) — the announcement that noindex, nofollow, and crawl-delay support ends Sept 1, 2019.

The standard

• RFC 9309 — Robots Exclusion Protocol — the IETF standard (Koster, Illyes, Zeller, Sassman; Sept 2022), including the /robots.txt location requirement, longest-match rule, 500 KiB parsing limit, and the security-considerations note.

Bing / Microsoft

• Bingbot guidance — current Bing WebmasterMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. guidance documents crawl-delay values from 1–20 seconds.
• How to Create a robots.txt File (Bing Webmaster Tools help) — Bing’s robots.txt guidance and tester.

Quotes from the source

On-the-record statements from Google, Bing, and the RFC. Each link is a deep link that jumps to the quoted passage on the source page.

Google — crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.

• “This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. with noindex or password-protect the page.” — Google Search Central docs. Jump to quote
• “While Google won’t crawl or index the content blocked by a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file, we might still find and index a disallowed URL if it is linked from other places on the web.” Jump to quote
• “If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result won’t have a description.” Jump to quote

Google — why a noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. page must stay crawlable

• “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. If the page is blocked by a robots.txt file or the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.” — Google Search Central docs. Jump to quote

Google — location, scope, and the spec

• “The rules listed in the robots.txt file apply only to the host, protocol, and port number where the robots.txt file is hosted.” Jump to quote
• “Google can’t index the content of pages which are disallowed for crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., but it may still index the URL and show it in search results without a snippet.” Jump to quote
• “When matching robots.txt rules to URLs, crawlers use the most specific rule based on the length of the rule path. In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule.” Jump to quote
• “* designates 0 or more instances of any valid character. $ designates the end of the URL.” Jump to quote
• “The field name (disallow) is case-insensitive, but its value is case-sensitive.” Jump to quote

Google — size, cachingCaching stores a copy of a page or resource — in a browser, a CDN edge node, or a search crawler's own cache — so it can be served again without regenerating or re-downloading it. It isn't a direct ranking factor, but it feeds page speed and crawl efficiency., and status codes

• “Google enforces a robots.txt file size limit of 500 kibibytes (KiB). Content which is after the maximum file size is ignored.” Jump to quote
• “Google generally caches the contents of robots.txt file for up to 24 hours, but may cache it longer in situations where refreshing the cached version isn’t possible.” Jump to quote
• “Google’s crawlers treat all 4xx errorsThe 4xx (client error) and 5xx (server error) HTTP status codes a server returns instead of a successful 2xx — and how Google treats each class very differently., except 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content., as if a valid robots.txt file didn’t exist. This means that Google assumes that there are no crawl restrictions.” Jump to quote
• “For the first 12 hours, Google stops crawling the site but keeps trying to fetch the robots.txt file. If Google can’t fetch a new version, for the next 30 days Google will use the last good version, while still trying to fetch a new version.” Jump to quote
• “If the errors are still not fixed after 30 days: If the site is generally available to Google, Google will behave as if there is no robots.txt file (but still keep checking for a new version).” Jump to quote
• “User agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. specific groups and global groups (*) are not combined.” Jump to quote

RFC 9309 — the standard

• “This document specifies and extends the ‘Robots Exclusion Protocol’ method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers.” — RFC 9309 (Koster, Illyes, Zeller, Sassman). Jump to quote
• “The rules MUST be accessible in a file named ‘/robots.txt’ (all lowercase) in the top-level path of the service.” Jump to quote
• “The Robots Exclusion Protocol is not a substitute for valid content security measures. Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable.” Jump to quote

Bing — crawl-delay

• Bingbot guidance documents a crawl-delay range of 1–20 seconds. This is Bing-specific guidance, not an extension that Google processes.

Patrick Stox — the blocking experiment (my own Ahrefs experiment, reviewed by Joshua Hardwick)

• “We lost a position here or there and all of the featured snippets for the pages.” Jump to quote
• “Both pages lost some traffic. But it didn’t result in much change to our traffic estimate like I was expecting.” Jump to quote
• “Don’t block pages you want indexed. It hurts. Not as bad as you might think it does—but it still hurts.” Jump to quote

Robots.txt — syntax cheat sheet

Directives (Google-supported)

Special characters

Matching rules

• Most-specific (longest path) rule wins.
• On a tie, least-restrictive wins — Allow beats Disallow.
• Path values are case-sensitive (/Folder/ ≠ /folder/); field and user-agent names are case-insensitive.
• A crawler obeys exactly one group — the most specific user-agent match.

Not supported by Google (retired Sept 1, 2019): noindex, nofollow, crawl-delay. Bing still honors crawl-delay (as a relative throttle, not literal seconds); Yandex stopped honoring it on Feb 22, 2018 — set the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. in Yandex WebmasterYandex Webmaster (webmaster.yandex.com) is Yandex's free search console — the Yandex equivalent of Google Search Console — for verifying a site, monitoring indexing and queries, requesting reindexing, running diagnostics, and setting a site region. It's essential for any site targeting Russia and the wider CIS, where Yandex, not Google, is the dominant search engine. instead.

Under the hood (Google)

• Size limit: 500 KiB (rest ignored). CachingCaching stores a copy of a page or resource — in a browser, a CDN edge node, or a search crawler's own cache — so it can be served again without regenerating or re-downloading it. It isn't a direct ranking factor, but it feeds page speed and crawl efficiency.: ~24h.
• 4xx (except 429429 Too Many Requests is an HTTP client-error status code that means a client sent too many requests in a given time window — a mechanism called rate limiting. Unlike other 4xx codes, Google treats 429 as a server-overload signal and slows crawling instead of removing content.) → no restrictions. 5xx/unreachable → ~12h crawl stop, then last good copy ~30 days, then no restrictions if still broken. 3xx → ≥5 hops then treated as 404.
• A specific User-agent group is never merged with the * group.

Common recipes

# Block a folder for everyone
User-agent: *
Disallow: /admin/

# Block all PDFs
User-agent: *
Disallow: /*.pdf$

# Block AI training crawler (won't stop search bots)
User-agent: GPTBot
Disallow: /

# Point to the sitemap
Sitemap: https://example.com/sitemap.xml

Robots.txt audit checklist

• The file exists at exactly https://yourdomain.com/robots.txt (root, lowercase) and returns a 200.
• Every host and protocol has its own file (www, non-www, subdomains, http vs https).
• No stray Disallow: / blocking the whole site (the classic staging leftover).
• You are not blocking any page you also want to noindex (Google must crawl it to see the noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed.).
• You are not using robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. to deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. pages — that’s noindex’s job.
• Render-critical CSSCritical CSS is a performance technique that extracts the styles needed to render a chosen above-the-fold view, inlines them in the <head>, and defers the rest of the stylesheet. It speeds up first paint but is an advanced, optional optimization — Google says most sites don't need it — and carries real production risks around caching, CSP, and page state./JS is not blocked.
• No noindex, nofollow, or crawl-delay lines aimed at Google (unsupported since Sept 1, 2019).
• No sensitive paths listed — the file is public and advertises whatever it names; use authentication instead.
• Wildcard rules (*, $) tested against real URLs; remember path values are case-sensitive.
• If you have a User-agent: bingbot (or any botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-specific) section, it repeats every directive you still want enforced — bot-specific groups ignore the * defaults.
• Sitemap: directive present with an absolute URL.
• File is under 500 KiB.
• Validated in the Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. robots.txt report; spot-checked any “IndexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., though blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” URLs to confirm they’re genuinely junk and not pages you wanted crawled.

The mental models

1. Crawl control, not indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. control. Robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. only governs whether a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. may fetch a URL. It says nothing about whether the URL gets indexed. Internalize this and most robots.txt mistakes disappear. Blocking a URL ≠ removing it from Google.

2. Most-specific rule wins; least-restrictive breaks ties. When two rules touch the same URL, the one with the longer path wins. On a genuine tie, Allow beats Disallow. This single rule explains nearly every “why is this URL crawlable / blocked?” head-scratcher.

3. Block vs noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. — a decision tree.

• Want it out of the index? → noindex, and make sure it’s not blocked in robots.txt (so Google can crawl it to see the noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed.).
• Want to stop wasting crawl on it but don’t care if the bare URL shows? → Disallow in robots.txt.
• Want it genuinely private? → authentication. Robots.txt is public and never hides anything.

Never block and noindex the same URL — they cancel out, and the page can linger in the index.

4. The file’s availability is production-critical. The HTTP response code on /robots.txt has site-wide consequences: a 4xx means “no restrictions,” but a 5xx can stop Google crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. your whole site for ~12 hours. Treat /robots.txt like any other critical endpoint.

Fetch and audit robots.txt across hosts

Robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. is scoped per host+protocol+port, so it’s easy to forget one variant exists. This pulls every variant of a domain and flags any that error or 404.

macOS / Linux

for url in "https://example.com/robots.txt" "https://www.example.com/robots.txt" "http://example.com/robots.txt"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" "$url")
  echo "$code  $url"
done

Windows (PowerShell)

$urls = "https://example.com/robots.txt","https://www.example.com/robots.txt","http://example.com/robots.txt"
foreach ($u in $urls) {
  $r = Invoke-WebRequest -Uri $u -UseBasicParsing -SkipHttpErrorCheck
  Write-Host "$($r.StatusCode)  $u"
}

A 200 on every line is what you want. A 4xx means Google treats that host+protocol as unrestricted (fine if intentional); a 5xx is the dangerous one — it can stall crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. for that host.

Regex to pull every Disallow/Allow path

Useful for diffing a robots.txt before/after an edit, or scripting a check against a list of URLs you care about.

import re

robots_txt = open("robots.txt").read()

# Captures the directive (Disallow/Allow) and its path value
pattern = re.compile(r'^(Disallow|Allow):\s*(\S*)', re.IGNORECASE | re.MULTILINE)

for directive, path in pattern.findall(robots_txt):
    print(f"{directive}: {path or '(empty — matches nothing)'}")

• (Disallow|Allow) — captures which directive fired.
• \s*(\S*) — captures the path value, allowing for an empty Disallow: (which Google treats as “disallow nothing”).
• re.IGNORECASE — field names are case-insensitive even though path values aren’t (don’t lowercase the paths themselves when you extend this).

Chrome DevTools Console snippet

Paste into the Console panel (F12 → Console) on any page to fetch and print that host’s robots.txt without leaving the tab:

fetch(new URL('/robots.txt', location.origin))
  .then(r => r.text())
  .then(t => console.log(t))
  .catch(e => console.error('No robots.txt or fetch blocked:', e));

Bookmarklet: jump to the current site’s robots.txt

Save as a bookmark with this as the URL (drag it to your bookmarks bar, or add it manually and paste the code into the “URL” field), then click it on any page:

javascript:(function(){var u=new URL('/robots.txt',location.origin).href;location.href=u;})();

Tools for testing and validating robots.txt

• Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. — robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. report (Settings) — Google’s current surface for seeing the robots.txt Google fetched, its status, and any parsing issues. (The old standalone robots.txt tester was retired; testing now lives in this report.)
• GSC URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. — confirm whether a specific URL is blocked from crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and how Google currently sees it.
• GSC “IndexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., though blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” (Pages report) — surfaces URLs Google indexed despite the block, so you can decide which are real problems.
• Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. — robots.txt tester / guidance — Bing’s equivalent for validating your file against bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share..
• Ahrefs Site Audit / Screaming Frog SEO SpiderA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — crawl your own site to see which URLs are blocked, surface accidentally-disallowed pages, and test that your rules match the URLs you intend.
• Any browser — the simplest check of all: load yourdomain.com/robots.txt and read it. Remember everyone else can do the same.

Incident: you shipped Disallow: / to production

This is the classic staging leftover — a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. built to block a staging environment gets deployed to the live site, and User-agent: * / Disallow: / blocks the whole thing. Here’s the ordered recovery.

Step 1 — confirm the blast radius. Fetch https://yourdomain.com/robots.txt right now (browser or curl -I). If you see a bare Disallow: / under User-agent: * with no narrower Allow: carving anything back out, every crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. that respects robots.txt is currently blocked from your entire site. Continue to Step 2.

Step 2 — fix the file and confirm the deploy. Replace it with the intended rules (or a minimal User-agent: * / Allow: / if you’re not sure what belongs) and deploy. Re-fetch the live URL — don’t trust your deploy dashboard, trust the actual response. If it still shows the old Disallow: /, you have a cachingCaching stores a copy of a page or resource — in a browser, a CDN edge node, or a search crawler's own cache — so it can be served again without regenerating or re-downloading it. It isn't a direct ranking factor, but it feeds page speed and crawl efficiency./CDN problem, not a code problem — purge the cache for that one path before moving on.

Step 3 — check how long the site was blocked. Google caches robots.txt for up to ~24 hours, so a fix that just went live won’t be picked up instantly. Open Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s robots.txt report to see when Google last fetched the file and whether it’s already seen the corrected version. If it hasn’t yet, request it be re-fetched isn’t a manual option — you wait, or use URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. on your most important pages and use “Request Indexing” once the file is confirmed fixed.

Step 4 — check for actual damage.

• If the block lasted hours, expect little to no visible impact — Google doesn’t drop pages from the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. just because one crawl was blocked.
• If it lasted days to weeks, check Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance.’s Pages report for a jump in “Blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” and check whether previously-indexed pages moved to that status. My own experiment blocking two high-ranking pages foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. they stayed indexed and mostly kept ranking, but lost freshness signals and featured snippets — the damage compounds with time blocked, not instantly.
• If you see rankings/impressions actually falling in Search Console’s Performance reportThe Google Search Console report that shows how your site actually performed in Google Search, built from real impressions and clicks. It reports four metrics — clicks, impressions, average CTR, and average position — and keeps the most recent 16 months of data. for the blocked window, that’s your confirmation the incident had real impact, not just a scare.

Step 5 — verify the fix took, don’t just assume it. Run the specific paths you care about through the Robots.txt Tester against the corrected file. If any come back blocked, you’re not done — go back to Step 2.

Concrete mistakes to avoid

Blocking a page in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. that you also want noindexed. Why it’s wrong: a noindex only works if the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. can read the page to see it. Block the page and Google never crawls it, never sees the noindex, and the page can linger in the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. if anything links to it. Do instead: leave the page crawlable and put noindex on the page itself (meta tag or X-Robots-Tag headerThe X-Robots-Tag is an HTTP response header that carries the same indexing and serving directives as the robots meta tag (noindex, nofollow, nosnippet, and the rest). Because it lives in the header rather than the HTML, it's how you control indexing for non-HTML files like PDFs, images, and videos.).

Using robots.txt as your deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. tool. Why it’s wrong: robots.txt controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Google’s own docs are explicit that it “is not a mechanism for keeping a web page out of Google.” A disallowed URL can still show up in results, just without a description. Do instead: use noindex (page not blocked) to actually remove a page from the index.

Blocking render-critical CSSCritical CSS is a performance technique that extracts the styles needed to render a chosen above-the-fold view, inlines them in the <head>, and defers the rest of the stylesheet. It speeds up first paint but is an advanced, optional optimization — Google says most sites don't need it — and carries real production risks around caching, CSP, and page state. or JavaScript. Why it’s wrong: Google renders your pages with a real browser to see them the way a user does. Block the assets that build the layout or inject content and Google may see a broken or empty page. Do instead: leave /assets/, /static/, or wherever your CSS/JS lives crawlable — Google’s own sample robots.txt explicitly re-allows these.

Listing sensitive paths in robots.txt to “hide” them. Why it’s wrong: the file is public — anyone can read it — so Disallow: /admin-panel/ just tells the world that folder exists. RFC 9309 is blunt that the protocol “is not a substitute for valid content security measures.” Do instead: use authentication or a password wall for anything that actually needs to stay private.

Trailing-slash confusion in a Disallow path. Why it’s wrong: Disallow: /folder (no trailing slashA trailing slash is the forward slash (/) at the end of a URL — example.com/page/ versus example.com/page. Except at the bare root domain, the two versions are different URLs to search engines, so you pick one format and enforce it.) blocks /folder, /folder/, /folder-name/, and /folder.html — anything that starts with that string, because matching is prefix-based, not directory-based. Do instead: add the trailing slashA trailing slash is the forward slash (/) at the end of a URL — example.com/page/ versus example.com/page. Except at the bare root domain, the two versions are different URLs to search engines, so you pick one format and enforce it. (Disallow: /folder/) when you only mean the directory, and test the exact prefix behavior against your real URLs before shipping.

Assuming rules are case-insensitive. Why it’s wrong: field names are case-insensitive but path values are case-sensitive — Disallow: /Folder/ does nothing to /folder/. Do instead: match the exact case your URLs actually use, and don’t assume a rule “should” work because it reads correctly to a human.

Common issues

”I blocked this page, but it’s still showing up in Google”

• Symptom: the URL still appears in search results (usually with no description) even though it’s disallowed in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere..
• Likely cause: the URL is linked from elsewhere on the web (or your own site), so Google indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. the bare URL without ever crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. its content. Blocking crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. was never going to remove it — that’s not what robots.txt does.
• Fix: if you want it gone, add noindex to the page and make sure it’s not blocked in robots.txt (temporarily allow crawling if it currently is). Confirm the fix with URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. in Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. once Google has re-crawled it.

”Search Console says ‘Indexed, though blocked by robots.txt’”

• Symptom: the Pages report in GSC flags this exact status for one or more URLs.
• Likely cause: usually harmless — it’s common for utility URLs (cart, filtered/faceted variants, session parameters) that got linked somewhere and indexed as a bare URL despite the block.
• Fix: check whether the flagged URLs are pages you actually wanted indexed. If they’re junk parameter URLs, this status is expected and not worth chasing. If a page you care about shows up here, that’s the “blocked + noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed.” trap — go crawlable + noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. instead.

”I edited robots.txt, but the change doesn’t seem to be live”

• Symptom: you deployed a fix, but a tool or Google still seems to be reading the old rules.
• Likely cause: Google caches robots.txt for up to ~24 hours, so a change isn’t picked up instantly. Separately, a CDN or reverse proxy in front of your site can be serving a stale cached copy of the file itself.
• Fix: first confirm the live file actually changed — fetch yourdomain.com/robots.txt directly (not through a cache) or run it through the Robots.txt Tester. If the live file is correct, the delay is Google’s cache and just needs time; check the robots.txt report in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. to see when it last fetched.

”The tester says a path is allowed, but Google still isn’t crawling it”

• Symptom: robots.txt clearly permits the URL, but server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. or Search Console show no recent crawl activity for it.
• Likely cause: robots.txt allowing a URL isn’t a crawl guarantee — it’s necessary but not sufficient. The page might be orphaned (nothing links to it), missing from the sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., or simply low priority in Google’s crawl scheduling.
• Fix: confirm the page is discoverable (internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. inclusion) rather than re-checking robots.txt again — the file isn’t the bottleneck here.

”My Disallow rule doesn’t seem to match the URLs I expect”

• Symptom: a rule that looks correct isn’t blocking (or is blocking too much of) the URLs you intended.
• Likely cause: case sensitivity (/Folder/ ≠ /folder/) or prefix matching catching more than expected (Disallow: /folder also matches /folder-name/).
• Fix: re-check the exact case of your live URLs, and add a trailing slashA trailing slash is the forward slash (/) at the end of a URL — example.com/page/ versus example.com/page. Except at the bare root domain, the two versions are different URLs to search engines, so you pick one format and enforce it. if you meant only the directory. Test the specific paths in the Robots.txt Tester before trusting the rule.

”Bing stopped following my default rules after I added a bingbot section”

• Symptom: rules under User-agent: * that used to apply to Bing seem to be ignored after you added a User-agent: bingbot group.
• Likely cause: Bing (like Google) applies only the most specific matching group to a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. A bingbot-specific group makes Bing ignore the * defaults entirely (crawl-delay excepted).
• Fix: repeat every directive you still want enforced inside the bingbot group — it has to be self-contained.

Validation tests

Run these after you edit robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., before you consider the change done.

Test 1: the file itself is live and correct

• Test to run: curl -I https://yourdomain.com/robots.txt (or open the URL in a browser), and separately run it through the Robots.txt Tester.
• Expected result: HTTP/1.1 200 and the tester shows the exact rules you intended to ship.
• Failure interpretation: a non-200 status means the file isn’t reachable as expected (check deploy/routing); correct-looking rules that don’t match what you edited usually mean a cache (CDN or Google’s ~24h cache) is serving a stale copy.
• Monitoring window: immediate for the status code and content; allow up to ~24 hours before assuming Google has picked up the change.
• Rollback trigger: the file 404s or 5xxs unexpectedly, or shows rules you didn’t intend to ship.

Test 2: the specific path you changed behaves as intended

• Test to run: paste the exact URL you meant to allow or block into the Robots.txt Tester and check the verdict for GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. (and BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. if relevant).
• Expected result: the tool reports “allowed” for a path you meant to open up, or “blocked” for a path you meant to close off.
• Failure interpretation: the opposite verdict usually means a longer/more-specific rule elsewhere in the file is overriding the one you just added — check for a competing Allow/Disallow on a shorter or overlapping path.
• Monitoring window: immediate — this is a rules-matching check, not a live-crawl check.
• Rollback trigger: the tested path returns the wrong verdict after you’ve confirmed the live file matches what you edited.

Test 3: a page you unblocked actually gets crawled

• Test to run: Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. on the specific URL, looking at “Crawl allowed?” and the last crawl date.
• Expected result: “Crawl allowed: Yes,” and a crawl date after your robots.txt change.
• Failure interpretation: “Crawl allowed: No” after you believe you unblocked it means Google is still reading cached rules, or another rule in the file still matches the path. No new crawl date yet doesn’t necessarily mean failure — Google may not have gotten to it yet.
• Monitoring window: 2–4 weeks for Google to actually re-crawl and reflect the change, longer on large or low-priority sites.
• Rollback trigger: “Crawl allowed” flips back to “No” on a later check with no robots.txt change on your end — investigate a CDN/cache issue.

Test 4: you didn’t accidentally block something else

• Test to run: crawl your site with Ahrefs Site Audit or Screaming Frog SEO SpiderA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. after the change and compare the “blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” URL list before vs. after.
• Expected result: only the URLs you intended to newly block or unblock moved between lists — nothing else changed.
• Failure interpretation: an unexpected page showing up as newly blocked usually means a wildcard or prefix rule matched more broadly than intended.
• Monitoring window: immediate — run this crawl right after deploying.
• Rollback trigger: any page you rely on for organic trafficVisitors from unpaid search results — it compounds without ad spend. appears newly blocked.

Ready-to-copy prompts

Paste your actual robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. content (and, for the second prompt, the list of URLs you care about) after each prompt.

Audit a robots.txt file for conflicts and risky rules

I'm going to paste a robots.txt file. Read it as a technical SEO would and
flag:
1. Any Disallow rule that conflicts with an Allow rule on the same or an
   overlapping path (tell me which one wins under longest-match rules).
2. Any bare "Disallow: /" under a User-agent group — that blocks everything
   for that crawler.
3. Any rule that would block commonly render-critical paths (CSS, JS, fonts,
   images used for layout).
4. Any use of noindex, nofollow, or crawl-delay under a Google-facing
   User-agent — these are not supported by Google.
5. Anything that looks like it's trying to hide a sensitive path (remember
   this file is public).

Here is the file:
<paste robots.txt content>

Check whether specific URLs are safely crawlable and indexable

I want the following URLs to be crawlable AND indexable by Google. Given this
robots.txt file, tell me for each URL whether it would be blocked from
crawling, and separately remind me that robots.txt says nothing about
indexing — a URL can be blocked from crawling but still indexed if it's
linked elsewhere, and a URL can be crawlable but still noindexed via a meta
tag or header this file can't show you.

Robots.txt:
<paste robots.txt content>

URLs:
<paste list of URLs>

Resources worth your time

My related writing

• The Story of Blocking 2 High-Ranking Pages With Robots.txt — my own experiment: I deliberately blocked two high-ranking Ahrefs pages and tracked what actually happened to rankings and traffic. The proof that blocking ≠ deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist..
• The Beginner’s Guide to Technical SEO — where robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. fits in the crawl/indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. foundation.
• Enterprise SEO Strategies — managing robots.txt at scale on large sites.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawl → render → index, the pipeline robots.txt gates at the crawl step. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• Robots.txt and SEO: Everything You Need to Know — the canonical Ahrefs guide, by Joshua Hardwick (not me) — a thorough syntax-and-recipes reference.
• When Unindexing Pages From Google Search Use Noindex, Not Robots.txt — Search Engine Roundtable coverage of John Mueller’s guidance: don’t block with robots.txt when you want pages out of the index, use noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. instead.
• Google Explains Why URLs Blocked By Robots.txt Can Still Be Indexed — Search Engine Journal coverage of Mueller’s “add-to-cart” example: blocked URLs can still be indexed, and “Indexed, though blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing.” is often harmless for utility pages.
• Robots.txt Tip from Bing: Include All Relevant Directives If You Have a Bingbot Section — Search Engine Land on the bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.-section gotcha: a BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.-specific group makes Bing ignore the wildcard defaults, so you must repeat every directive you still want enforced.
• Overview of OpenAI Crawlers (GPTBot, OAI-SearchBot, ChatGPT-User) — OpenAI’s official list of their crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. user-agent tokensA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., with robots.txt examples for blocking training vs. search vs. user-triggered fetches separately.
• r/TechSEO — the community for crawl/index and “Indexed, though blocked by robots.txtA Google Search Console Page Indexing warning: Google indexed the URL anyway despite your robots.txt disallowing crawling it — Google names other pages linking to it as the likely path. robots.txt blocks crawling, not indexing.” debugging.

Test yourself: Robots.txt

Five quick questions on what robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. controls (and what it does not). Pick an answer for each, then check.

Robots.txt

A plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.

Related: Crawling, Crawl Budget

Robots.txt

Robots.txt is a plain-text file served from the root of a host — exactly https://example.com/robots.txt — that tells crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. which URLs they may and may not request. It implements the Robots Exclusion Protocol, originally created by Martijn Koster in 1994 and standardized as RFC 9309 in 2022.

The single most important thing to understand is that robots.txt is a crawl-control file, not an indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.-control file. Blocking a URL stops Google from fetching it, but the URL can still be indexed — shown as a bare URL with no description — if other pages link to it. To keep a page out of the index you need noindex, and the page must not be blocked in robots.txt, or Google never crawls it to see the noindex.

The rules apply only to the exact host, protocol, and port the file is served from. Each subdomain and protocol needs its own file.

Related: Crawling, Crawl Budget

Changelog