{ robots.txt }
Yes, this is a valid robots.txt. The same bytes you're reading as a webpage parse cleanly as crawl directives — the server just labels it text/html for browsers and text/plain for crawlers. Don't try this at home. (Test it with my robots.txt Tester, whose matcher is ported from Google's open-source parser. Then try it at home.)
# _____ _____ ____ _ _ _ _ ___ ____ _ _ # |_ _| ____/ ___| | | | \ | |_ _/ ___| / \ | | # | | | _|| | | |_| | \| || | | / _ \ | | # | | | |__| |___| _ | |\ || | |___ / ___ \| |___ # |_| |_____\____|_| |_|_| \_|___\____/_/ \_\_____| # ____ _____ ___ # / ___|| ____/ _ \ # \___ \| _|| | | | # ___) | |__| |_| | # |____/|_____\___/ # # The technical SEO's technical SEO — patrickstox.com User-agent: * Allow: / Content-Signal: search=yes, ai-input=yes, ai-train=yes Disallow: /admin/ Disallow: /preview/ Disallow: /api/ # Honeypot: nothing lives here. Fetching it anyway gets you on a list. Disallow: /api/trap/ # Gateway redirects (Slack invite etc.) — nothing to crawl or index. Disallow: /go/ Sitemap: https://patrickstox.com/sitemap-index.xml Sitemap: https://patrickstox.com/media-sitemap.xml
How it works: crawlers parse robots.txt line by line and ignore anything that isn't a directive — all the HTML wrapped around the rules reads as noise to them. Every directive above sits on its own clean line inside a single User-agent: * group, and the response is byte-identical no matter who asks. No cloaking, just content negotiation on the label.