robots.txt Generator
Free, no signup. Set a simple default policy, choose the bot groups to block, then test representative URLs against the generated file before publishing it.
Build your file
Select crawler groups to block. Search crawlers remain allowed unless you explicitly select them.
This non-standard directive is emitted only for Bingbot. Google does not process it; Bing documents 1–20 second values. It is omitted if you block Bingbot.
Runs entirely in your browser — nothing you paste is uploaded or stored. Everything here, including matching and copy output, runs in your browser. Nothing is uploaded. Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.
Generated robots.txt
Blocking ≠ deindexing. A Disallow asks compliant crawlers not to fetch a URL; it does not remove a URL already known to a search engine.
Live match preview
Need a full bot × URL matrix? Open robots.txt Tester →How to use it
- Choose which bot categories should not crawl the site.
- Add your absolute sitemap URL if you have one.
- Paste a few important URLs and a bot name to confirm the intended result.
- Copy the file to the site root as
/robots.txt.
How it works
The output uses separate user-agent groups and Disallow: / rules, so the intention is easy to audit. It uses the same Google-style matcher as the robots.txt Tester, including longest-match-wins behavior.
What this generator does not do
robots.txt is not security, and it is not a deindexing tool. Do not list sensitive paths expecting them to stay secret; the file is public. Use access controls for private material and noindex for index removal.
Frequently asked questions
Does blocking a bot deindex my pages?
No. robots.txt controls crawling, not whether a URL remains indexed. Use a noindex directive on a crawlable page when you need a deindexing signal.
Will every crawler obey this file?
No. robots.txt is a public, voluntary crawl preference, not access control. Use authentication, a WAF, or rate limiting to protect private content.
Why do I need a Sitemap line?
It gives crawlers an explicit absolute URL for your XML sitemap. It is optional, but usually useful for discoverability.
Site passport Local context for this saved site
Local data
Saved targets, named lists, and recent check summaries remain only in this browser.
Rate this tool
Feature requests for Robots Txt Generator
Upvote what you want most. New ideas can be submitted from the floating Feedback menu; requests appear here once approved, and the most-wanted rise to the top.
You won't be emailed about that request anymore.
Loading…
➕ Request a feature
New requests are reviewed before they appear here.
Informazioni sullo strumento
Crea un robots.txt pronto da copiare con preset per crawler di ricerca, training AI, assistenti e archivi, un crawl-delay Bingbot documentato e un matcher in stile Google.
Funzionalità
- Preset raggruppati per crawler di ricerca, AI, assistenti e archivi.
- Crawl-delay Bingbot opzionale con spiegazione.
- Matcher per testare un bot contro URL di esempio.
- Output copiabile e controlli locali nel browser.
Come funziona
Seleziona i gruppi da bloccare, aggiungi eventuale crawl-delay e genera il file. Prova poi user-agent e URL di esempio con il matcher prima di pubblicare.
Limitazioni
- robots.txt controlla il crawling, non garantisce la rimozione dall’indice.
- Le regole generate vanno verificate con il comportamento del tuo server e dei crawler.
Domande frequenti
robots.txt impedisce che un URL resti indicizzato?
No. Controlla il crawling, non lo stato dell’indice; usa un segnale noindex su una pagina che resta scansionabile per deindicizzare.
Quali crawler posso bloccare?
Puoi scegliere gruppi di ricerca, training AI, assistenti e archivi; i crawler di ricerca restano consentiti finché non li selezioni.
Il crawl-delay è rispettato da tutti?
No. Il campo documentato per Bingbot non è una direttiva universale.
Il tool pubblica il file?
No. Genera testo da copiare; devi installarlo e verificarlo sul tuo server.