robots.txt Tester

How it works

The matcher is ported from Google's open-source robots.txt parser, so it decides allow/block the way Googlebot actually does — not with a naive "first match wins" rule that most testers use. That means longest-match-wins (the most specific Allow/Disallow path takes precedence regardless of order), * and $ wildcards, percent-encoding normalisation, and sub-agent fallback (a bot with no group of its own falls back to User-agent: *).

Crawl-delay is crawler-specific, not part of this Google matcher. Google does not process it. Bing documents 1–20 second values. This tool reports Google-style Allow/Disallow decisions and never pretends a Crawl-delay line changes Googlebot’s result.

All of that runs in your browser. The server is only touched to fetch a live robots.txt or sitemap, because browsers can't request them cross-origin. It also models robots.txt status-code semantics: most 4xx responses are treated as "no robots.txt," but 429 is a temporary failure like a 5xx or timeout. Temporary failures can initially pause crawling; Google may also reuse its last known good file, which a point-in-time tester cannot observe.

Features

• Google-accurate matching: longest-match-wins, wildcards, sub-agent fallback, status-code rules.
• Blocked/allowed matrix across as many URLs and bots as you like.
• Click any verdict to see the exact matching line highlighted in the file.
• One-click AI crawler audit for the major LLM and AI-search bots.
• Built-in linter that flags common robots.txt mistakes.
• Fetch by domain, upload a URL list, or pull URLs from a sitemap — including a separate live robots.txt for each entered subdomain (up to 10 origins per run).
• Privacy: matching is fully client-side — pasted files never leave your machine.

Common mistakes it catches

• Assuming rule order matters — it doesn't for Google; the most specific path wins.
• Blocking a resource (CSS/JS) that the page needs to render, then wondering why rankings drop.
• Using robots.txt to keep a page out of the index — disallow stops crawling, not indexing; use NoindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. for that.
• A stray Disallow: / under User-agent: * that quietly blocks the whole site.

Limitations

This checks Google-style robots.txt matching only — it tells you whether a bot is allowed to crawl a URL, not whether the page is indexable (a page can be crawlable and still carry a noindex). Fetch-by-domain reads the robots.txt exactly as served; if a site returns different robots.txt to different user-agents or regions, you'll see the version your fetch received. A robots.txt file is scoped to its origin, so fetch mode requests each entered subdomain’s own file. Paste mode deliberately uses the one pasted draft for every URL. Non-Google crawler behavior is not inferred here; consult that crawler’s own documentation before relying on system-specific extensions.

Frequently asked questions