Draft an explicit crawler policy without hand-typing bot tokens
Select whole crawler categories or individual user-agent tokens and review the separate generated groups before publishing.
robots.txt Generator
Free, no signup. Set a simple default policy, choose the bot groups to block, then test representative URLs against the generated file before publishing it.
Build your file
Select crawler groups to block. Search crawlers remain allowed unless you explicitly select them.
This non-standard directive is emitted only for Bingbot. Google does not process it; Bing documents 1–20 second values. It is omitted if you block Bingbot.
Runs entirely in your browser — nothing you paste is uploaded or stored. Everything here, including matching and copy output, runs in your browser. Nothing is uploaded. Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.
Generated robots.txt
Blocking ≠ deindexing. A Disallow asks compliant crawlers not to fetch a URL; it does not remove a URL already known to a search engine.
Live match preview
Need a full bot × URL matrix? Open robots.txt Tester →How to use it
- Choose which bot categories should not crawl the site.
- Add your absolute sitemap URL if you have one.
- Paste a few important URLs and a bot name to confirm the intended result.
- Copy the file to the site root as
/robots.txt.
How it works
The output uses separate user-agent groups and Disallow: / rules, so the intention is easy to audit. It uses the same Google-style matcher as the robots.txt Tester, including longest-match-wins behavior.
What this generator does not do
robots.txt is not security, and it is not a deindexing tool. Do not list sensitive paths expecting them to stay secret; the file is public. Use access controls for private material and noindex for index removal.
Frequently asked questions
Does blocking a bot deindex my pages?
No. robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not whether a URL remains indexed. Use a noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. directive on a crawlable page when you need a deindexing signalDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist..
Will every crawler obey this file?
No. robots.txt is a public, voluntary crawl preference, not access control. Use authentication, a WAF, or rate limiting to protect private content.
Why do I need a Sitemap line?
It gives crawlers an explicit absolute URL for your XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags.. It is optional, but usually useful for discoverability.
Site passport Local context for this saved site
Local data
Saved targets, named lists, and recent check summaries remain only in this browser.
Rate this tool
Feature requests for Robots Txt Generator
Upvote what you want most. New ideas can be submitted from the floating Feedback menu; requests appear here once approved, and the most-wanted rise to the top.
You won't be emailed about that request anymore.
Loading…
➕ Request a feature
New requests are reviewed before they appear here.
Where this tool helps
Common use cases
Keep search crawling separate from AI training and assistant access
Block only the intended categories while leaving search crawlers allowed unless they are explicitly selected.
Test representative URLs against the draft
Preview Google-style longest-match results for a supplied user-agent and URL sample before copying the file.
Add a sitemap and an optional Bingbot-specific delay
Emit an absolute Sitemap line and a documented 1–20 second Bingbot crawl-delay only when Bingbot is not blocked.
Create a reviewable publication handoff
Copy the draft, verify a fuller bot-by-URL matrix in robots.txt Tester, and keep deindexing, security, rate limiting, and crawler compliance decisions separate.
Watch the full workflow
robots.txt Generator walkthrough
Read the transcript
robots.txt Generator
Hi, this is the robots dot TXT Generator. It helps you create the small public file that tells compliant crawlers which parts of a website they may request. I’ll build a fictional policy, test it with sample pages, and explain what the file can and cannot do.
Step 1
A robots dot TXT file is a public set of crawl instructions placed at the root of a website. This generator helps you build that file without memorizing crawler names or rule formatting, then lets you test sample pages before publishing.
Step 2
Use it to create a starting policy, separate search crawlers from AI training or assistant tools, test important paths, add a sitemap, or prepare a file for review. It is especially useful when several crawler names would be easy to mistype.
Step 3
Before you select anything, the generated file allows the whole site. An asterisk means every crawler, and the rule “Allow: slash” means every path. The output updates automatically when you change the builder above it.
Step 4
The first cards group search crawlers and AI training crawlers. You can select an entire category or only specific names inside it. Additional cards cover assistants and archives. Review every choice because different sites may want different access policies.
Step 5
Let’s use the button labeled “Try an example.” It selects several training and archive entries, adds a fictional sitemap, and supplies two sample pages. The choices are demonstration data, not a recommendation that every site should block the same crawlers.
Step 6
The output keeps the default allow rule, then creates separate blocks for the selected crawler names. “Disallow: slash” means those named crawlers are asked not to request any path. The sitemap address appears at the bottom as a discovery hint.
Step 7
The live preview starts with “Googlebot.” Because Googlebot was not selected for blocking, both sample pages show “Allowed.” This preview checks the text rules locally; it does not contact Google or fetch either page.
Step 8
Now change the user agent to “GPTBot.” A user agent is the name a crawler presents to the site. Both rows switch to “Blocked” because the generated GPTBot group contains a rule that covers every path.
Step 9
Each preview row shows the verdict, the matching line, and the page you supplied. “Blocked” means this rule set asks that named crawler not to fetch the page. It does not prove the crawler will obey the request.
Step 10
If Bingbot needs slower requests, enter a whole number from one through twenty in “Bingbot crawl-delay.” Here I’ll use four seconds. The generator adds a separate Bingbot section and clearly labels this as an optional, non-standard instruction.
Step 11
The delay is written only for Bingbot. Google does not process this instruction, so the page links to current documentation instead of pretending it works everywhere. Leave the field blank unless you have a specific reason to use it.
Step 12
When the draft looks right, use “Copy robots.txt.” For a larger review, “Open in robots.txt Tester” provides a fuller crawler-by-page matrix. I like this handoff because the quick generator and deeper test stay separate.
Step 13
The output is editable, so here’s a more precise example. One rule blocks the private folder, while a second allows a public guide inside it. As soon as the text changes, the preview recalculates both sample pages.
Step 14
The private report is blocked, but the public guide is allowed. That happens because the more specific, longer matching rule wins. This is the most useful preview check: a broad block can still contain a carefully chosen public exception.
Step 15
This warning matters: blocking is not deindexing. Disallow asks a compliant crawler not to fetch a page, but the address can still appear in search results. Use a crawlable noindex instruction when removal from an index is the goal.
Step 16
Everything runs in the browser. The generator formats the selected groups, then compares your sample pages with the draft using Google-style matching rules. No crawler is contacted, no live robots file is read, and nothing is uploaded.
Step 17
Robots dot TXT is public and voluntary. It cannot protect passwords, customer records, private files, or any other sensitive content. Use authentication and access controls for privacy, and remember that some crawlers may ignore the file entirely.
Step 18
Before publishing, test your important public pages, blocked folders, and exceptions. Copy the final text to slash robots dot TXT at the correct site root, open that address in a browser to confirm it is served, and review crawler logs after launch.
Build it. Test it. Publish it carefully.
That’s the complete workflow. Review every selected crawler, test your important public and blocked paths, then publish the file at slash robots dot TXT. Use noindex when you need index removal, and use real access controls for anything private.
Common issues & how to fix them
- Warning Generated policy blocks a selected AI crawler Fix: Remove the selected crawler’s Disallow rule, or narrow it to only the paths that should remain unavailable to that AI crawler.
- Warning Generated policy blocks a search crawler Fix: Remove or narrow the Googlebot/Bingbot Disallow rule so pages intended for organic search are crawlable.
- Warning Sensitive path is not disallowed Fix: Add explicit Disallow rules for non-public admin, staging, cart, or internal-search paths, while protecting truly sensitive data with authentication rather than robots.txt.
- Information Generated file has no sitemap declaration Fix: Add an absolute Sitemap: https://…/sitemap.xml declaration pointing to the canonical, publicly fetchable sitemap or sitemap index.
- Warning Generated groups contain conflicting rules Fix: Combine rules for the same user-agent into one group and resolve overlapping Allow/Disallow paths to express one intentional policy.