robots.txt Generator

Free, no signup. Set a simple default policy, choose the bot groups to block, then test representative URLs against the generated file before publishing it.

Built by Patrick Stox

Build your file

Select crawler groups to block. Search crawlers remain allowed unless you explicitly select them.

This non-standard directive is emitted only for Bingbot. Google does not process it; Bing documents 1–20 second values. It is omitted if you block Bingbot.

Open in robots.txt Tester

Runs entirely in your browser — nothing you paste is uploaded or stored. Everything here, including matching and copy output, runs in your browser. Nothing is uploaded. Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.

Generated robots.txt

Blocking ≠ deindexing. A Disallow asks compliant crawlers not to fetch a URL; it does not remove a URL already known to a search engine.

Live match preview

Need a full bot × URL matrix? Open robots.txt Tester →

How to use it

  1. Choose which bot categories should not crawl the site.
  2. Add your absolute sitemap URL if you have one.
  3. Paste a few important URLs and a bot name to confirm the intended result.
  4. Copy the file to the site root as /robots.txt.

How it works

The output uses separate user-agent groups and Disallow: / rules, so the intention is easy to audit. It uses the same Google-style matcher as the robots.txt Tester, including longest-match-wins behavior.

What this generator does not do

robots.txt is not security, and it is not a deindexing tool. Do not list sensitive paths expecting them to stay secret; the file is public. Use access controls for private material and noindex for index removal.

Frequently asked questions

Does blocking a bot deindex my pages?

No. robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not whether a URL remains indexed. Use a noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. directive on a crawlable page when you need a deindexing signalDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist..

Will every crawler obey this file?

No. robots.txt is a public, voluntary crawl preference, not access control. Use authentication, a WAF, or rate limiting to protect private content.

Why do I need a Sitemap line?

It gives crawlers an explicit absolute URL for your XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags.. It is optional, but usually useful for discoverability.

Local data

Saved targets, named lists, and recent check summaries remain only in this browser.

Next steprobots.txt Tester — verify it with a direct check.

Explore more
ToolAI-Crawler Access CheckerToolLog File AnalyzerToolGooglebot VerifierArticleUser AgentArticleAI CrawlersArticlellms.txt

Feature requests for Robots Txt Generator

Upvote what you want most. New ideas can be submitted from the floating Feedback menu; requests appear here once approved, and the most-wanted rise to the top.

Loading…

➕ Request a feature

New requests are reviewed before they appear here.

Feedback
Report a bug

Found something broken in Robots Txt Generator? Let us know what happened — this goes straight to a private triage queue, not a public list.

What will be sent 
 No tool inputs, uploads, pasted source, complete results, query parameters, or URL fragments are attached automatically. You can edit or remove the selected passage above. Browser and anti-abuse metadata is processed for spam prevention. 
  
 
  
 
 
  
 
 
             
 
Common issues & how to fix them
 
 
 
 
 
robots.txt Generator · one finding
 
Generated policy blocks a selected AI crawler
 
Recommended action: Remove the selected crawler’s Disallow rule, or narrow it to only the paths that should remain unavailable to that AI crawler.
 
Fix principle: Crawler access is a policy decision, not a score optimization. Identify the exact current user-agent token from the operator, decide whether that crawler may fetch each path class, and test the complete robots.txt group selection so wildcard and specific rules do not create an accidental outcome.
 
 
 
 Implementation for your setup 
 
Confirm first: A named AI crawler is blocked or partially blocked contrary to the site owner's documented policy. Wildcard and crawler-specific groups produce a result different from the author's intention. Generated robots.txt output blocks search crawlers, exposes sensitive paths, or introduces contradictory group definitions.
 
Do not apply when: The owner has an explicit allow/block decision for each crawler class and path class, using operator-documented current tokens. A standards-conformant parser selects the intended group and result for representative URLs, with no reliance on rule order or unsupported directives.
 
Likely control points: server · edge-cdn · cms-app · managed-vendor
 
   
Choose the layer that owns the final response.
 
 
 
Generic HTTP / HTML
 
Status: supported · Location: The single robots.txt source served from the origin root
 
Steps
  1. Document the business policy first: which crawler operators and product uses are allowed, partially allowed, or blocked, and which path classes contain public versus sensitive material.
  2. Confirm each current crawler token from that operator's official documentation; do not infer policy from a display name or stale third-party list.
  3. Consolidate duplicate groups for the same token and make the wildcard group independently safe. Specific and wildcard groups may not combine the way the author expects.
  4. Use only supported allow/disallow path rules for access control, keep sensitive systems protected by authentication rather than robots.txt, and preserve sitemap declarations as absolute URLs.
  5. Test representative public, blocked, and borderline paths for every named token before publishing.
  
Verify on this platform
 
  • Fetch the production robots.txt and confirm it is reachable, parseable, and the exact reviewed revision.
  • Evaluate representative URLs with the exact crawler token and a wildcard token using a standards-conformant parser.
  • Re-run AI-Crawler Access Checker and manually confirm any intentional blocks remain described as policy, not defects.
 
Rollback
  1. Retain the prior robots.txt revision.
  2. If the new file accidentally blocks important search or service crawlers, restore the prior revision immediately and re-author the affected groups offline.
 
Caveats
  • robots.txt is publicly readable and is not an access-control system for confidential data.
  • Allowing a crawler does not guarantee crawling, training, citation, ranking, or referral traffic.
  • Different operators and products may use different tokens; a generic “AI bots” rule can overreach.
 
Official sources
 
 
Platform-managed / ask the vendor
 
Status: platform-managed · Location: Vendor robots.txt generator or edge policy
 
Implementation is controlled by the platform or vendor.
  
Verify on this platform
 
  • Ask the vendor how custom and generated groups are merged and whether updates can overwrite owner rules.
  • Fetch and test the final public robots.txt rather than relying on the admin preview.
  
Caveats
  • Escalate when the vendor cannot preserve an explicit crawler policy or export the active file.
 
Official sources
 
 
 
 
 How to verify 
 
  1. Record the owner-approved policy and operator-documented current user-agent tokens.
  2. Test each representative path against the exact specific and wildcard group selection.
  3. Fetch the final public file after deployment and rerun the checker.
  4. Keep sensitive content behind authentication regardless of the robots result.
 
Escalate when
 
Escalate when the operator token or intended product use cannot be confirmed, generated and custom robots rules conflict, or the requested policy could expose confidential content.
 
 
 
 
 
 
 
robots.txt Generator · one finding
 
Generated policy blocks a search crawler
 
Recommended action: Remove or narrow the Googlebot/Bingbot Disallow rule so pages intended for organic search are crawlable.
 
Fix principle: Crawler access is a policy decision, not a score optimization. Identify the exact current user-agent token from the operator, decide whether that crawler may fetch each path class, and test the complete robots.txt group selection so wildcard and specific rules do not create an accidental outcome.
 
 
 
 Implementation for your setup 
 
Confirm first: A named AI crawler is blocked or partially blocked contrary to the site owner's documented policy. Wildcard and crawler-specific groups produce a result different from the author's intention. Generated robots.txt output blocks search crawlers, exposes sensitive paths, or introduces contradictory group definitions.
 
Do not apply when: The owner has an explicit allow/block decision for each crawler class and path class, using operator-documented current tokens. A standards-conformant parser selects the intended group and result for representative URLs, with no reliance on rule order or unsupported directives.
 
Likely control points: server · edge-cdn · cms-app · managed-vendor
 
   
Choose the layer that owns the final response.
 
 
 
Generic HTTP / HTML
 
Status: supported · Location: The single robots.txt source served from the origin root
 
Steps
  1. Document the business policy first: which crawler operators and product uses are allowed, partially allowed, or blocked, and which path classes contain public versus sensitive material.
  2. Confirm each current crawler token from that operator's official documentation; do not infer policy from a display name or stale third-party list.
  3. Consolidate duplicate groups for the same token and make the wildcard group independently safe. Specific and wildcard groups may not combine the way the author expects.
  4. Use only supported allow/disallow path rules for access control, keep sensitive systems protected by authentication rather than robots.txt, and preserve sitemap declarations as absolute URLs.
  5. Test representative public, blocked, and borderline paths for every named token before publishing.
  
Verify on this platform
 
  • Fetch the production robots.txt and confirm it is reachable, parseable, and the exact reviewed revision.
  • Evaluate representative URLs with the exact crawler token and a wildcard token using a standards-conformant parser.
  • Re-run AI-Crawler Access Checker and manually confirm any intentional blocks remain described as policy, not defects.
 
Rollback
  1. Retain the prior robots.txt revision.
  2. If the new file accidentally blocks important search or service crawlers, restore the prior revision immediately and re-author the affected groups offline.
 
Caveats
  • robots.txt is publicly readable and is not an access-control system for confidential data.
  • Allowing a crawler does not guarantee crawling, training, citation, ranking, or referral traffic.
  • Different operators and products may use different tokens; a generic “AI bots” rule can overreach.
 
Official sources
 
 
Platform-managed / ask the vendor
 
Status: platform-managed · Location: Vendor robots.txt generator or edge policy
 
Implementation is controlled by the platform or vendor.
  
Verify on this platform
 
  • Ask the vendor how custom and generated groups are merged and whether updates can overwrite owner rules.
  • Fetch and test the final public robots.txt rather than relying on the admin preview.
  
Caveats
  • Escalate when the vendor cannot preserve an explicit crawler policy or export the active file.
 
Official sources
 
 
 
 
 How to verify 
 
  1. Record the owner-approved policy and operator-documented current user-agent tokens.
  2. Test each representative path against the exact specific and wildcard group selection.
  3. Fetch the final public file after deployment and rerun the checker.
  4. Keep sensitive content behind authentication regardless of the robots result.
 
Escalate when
 
Escalate when the operator token or intended product use cannot be confirmed, generated and custom robots rules conflict, or the requested policy could expose confidential content.
 
 
 
 
 
 
 
robots.txt Generator · one finding
 
Sensitive path is not disallowed
 
Recommended action: Add explicit Disallow rules for non-public admin, staging, cart, or internal-search paths, while protecting truly sensitive data with authentication rather than robots.txt.
 
Fix principle: Crawler access is a policy decision, not a score optimization. Identify the exact current user-agent token from the operator, decide whether that crawler may fetch each path class, and test the complete robots.txt group selection so wildcard and specific rules do not create an accidental outcome.
 
 
 
 Implementation for your setup 
 
Confirm first: A named AI crawler is blocked or partially blocked contrary to the site owner's documented policy. Wildcard and crawler-specific groups produce a result different from the author's intention. Generated robots.txt output blocks search crawlers, exposes sensitive paths, or introduces contradictory group definitions.
 
Do not apply when: The owner has an explicit allow/block decision for each crawler class and path class, using operator-documented current tokens. A standards-conformant parser selects the intended group and result for representative URLs, with no reliance on rule order or unsupported directives.
 
Likely control points: server · edge-cdn · cms-app · managed-vendor
 
   
Choose the layer that owns the final response.
 
 
 
Generic HTTP / HTML
 
Status: supported · Location: The single robots.txt source served from the origin root
 
Steps
  1. Document the business policy first: which crawler operators and product uses are allowed, partially allowed, or blocked, and which path classes contain public versus sensitive material.
  2. Confirm each current crawler token from that operator's official documentation; do not infer policy from a display name or stale third-party list.
  3. Consolidate duplicate groups for the same token and make the wildcard group independently safe. Specific and wildcard groups may not combine the way the author expects.
  4. Use only supported allow/disallow path rules for access control, keep sensitive systems protected by authentication rather than robots.txt, and preserve sitemap declarations as absolute URLs.
  5. Test representative public, blocked, and borderline paths for every named token before publishing.
  
Verify on this platform
 
  • Fetch the production robots.txt and confirm it is reachable, parseable, and the exact reviewed revision.
  • Evaluate representative URLs with the exact crawler token and a wildcard token using a standards-conformant parser.
  • Re-run AI-Crawler Access Checker and manually confirm any intentional blocks remain described as policy, not defects.
 
Rollback
  1. Retain the prior robots.txt revision.
  2. If the new file accidentally blocks important search or service crawlers, restore the prior revision immediately and re-author the affected groups offline.
 
Caveats
  • robots.txt is publicly readable and is not an access-control system for confidential data.
  • Allowing a crawler does not guarantee crawling, training, citation, ranking, or referral traffic.
  • Different operators and products may use different tokens; a generic “AI bots” rule can overreach.
 
Official sources
 
 
Platform-managed / ask the vendor
 
Status: platform-managed · Location: Vendor robots.txt generator or edge policy
 
Implementation is controlled by the platform or vendor.
  
Verify on this platform
 
  • Ask the vendor how custom and generated groups are merged and whether updates can overwrite owner rules.
  • Fetch and test the final public robots.txt rather than relying on the admin preview.
  
Caveats
  • Escalate when the vendor cannot preserve an explicit crawler policy or export the active file.
 
Official sources
 
 
 
 
 How to verify 
 
  1. Record the owner-approved policy and operator-documented current user-agent tokens.
  2. Test each representative path against the exact specific and wildcard group selection.
  3. Fetch the final public file after deployment and rerun the checker.
  4. Keep sensitive content behind authentication regardless of the robots result.
 
Escalate when
 
Escalate when the operator token or intended product use cannot be confirmed, generated and custom robots rules conflict, or the requested policy could expose confidential content.
 
 
 
 
 
 
 
robots.txt Generator · one finding
 
Generated groups contain conflicting rules
 
Recommended action: Combine rules for the same user-agent into one group and resolve overlapping Allow/Disallow paths to express one intentional policy.
 
Fix principle: Crawler access is a policy decision, not a score optimization. Identify the exact current user-agent token from the operator, decide whether that crawler may fetch each path class, and test the complete robots.txt group selection so wildcard and specific rules do not create an accidental outcome.
 
 
 
 Implementation for your setup 
 
Confirm first: A named AI crawler is blocked or partially blocked contrary to the site owner's documented policy. Wildcard and crawler-specific groups produce a result different from the author's intention. Generated robots.txt output blocks search crawlers, exposes sensitive paths, or introduces contradictory group definitions.
 
Do not apply when: The owner has an explicit allow/block decision for each crawler class and path class, using operator-documented current tokens. A standards-conformant parser selects the intended group and result for representative URLs, with no reliance on rule order or unsupported directives.
 
Likely control points: server · edge-cdn · cms-app · managed-vendor
 
   
Choose the layer that owns the final response.
 
 
 
Generic HTTP / HTML
 
Status: supported · Location: The single robots.txt source served from the origin root
 
Steps
  1. Document the business policy first: which crawler operators and product uses are allowed, partially allowed, or blocked, and which path classes contain public versus sensitive material.
  2. Confirm each current crawler token from that operator's official documentation; do not infer policy from a display name or stale third-party list.
  3. Consolidate duplicate groups for the same token and make the wildcard group independently safe. Specific and wildcard groups may not combine the way the author expects.
  4. Use only supported allow/disallow path rules for access control, keep sensitive systems protected by authentication rather than robots.txt, and preserve sitemap declarations as absolute URLs.
  5. Test representative public, blocked, and borderline paths for every named token before publishing.
  
Verify on this platform
 
  • Fetch the production robots.txt and confirm it is reachable, parseable, and the exact reviewed revision.
  • Evaluate representative URLs with the exact crawler token and a wildcard token using a standards-conformant parser.
  • Re-run AI-Crawler Access Checker and manually confirm any intentional blocks remain described as policy, not defects.
 
Rollback
  1. Retain the prior robots.txt revision.
  2. If the new file accidentally blocks important search or service crawlers, restore the prior revision immediately and re-author the affected groups offline.
 
Caveats
  • robots.txt is publicly readable and is not an access-control system for confidential data.
  • Allowing a crawler does not guarantee crawling, training, citation, ranking, or referral traffic.
  • Different operators and products may use different tokens; a generic “AI bots” rule can overreach.
 
Official sources
 
 
Platform-managed / ask the vendor
 
Status: platform-managed · Location: Vendor robots.txt generator or edge policy
 
Implementation is controlled by the platform or vendor.
  
Verify on this platform
 
  • Ask the vendor how custom and generated groups are merged and whether updates can overwrite owner rules.
  • Fetch and test the final public robots.txt rather than relying on the admin preview.
  
Caveats
  • Escalate when the vendor cannot preserve an explicit crawler policy or export the active file.
 
Official sources
 
 
 
 
 How to verify 
 
  1. Record the owner-approved policy and operator-documented current user-agent tokens.
  2. Test each representative path against the exact specific and wildcard group selection.
  3. Fetch the final public file after deployment and rerun the checker.
  4. Keep sensitive content behind authentication regardless of the robots result.
 
Escalate when
 
Escalate when the operator token or intended product use cannot be confirmed, generated and custom robots rules conflict, or the requested policy could expose confidential content.
 
 
 
 
  
 
 
     
  •  Information Generated file has no sitemap declaration Fix: Add an absolute Sitemap: https://…/sitemap.xml declaration pointing to the canonical, publicly fetchable sitemap or sitemap index. 
    •