47 of 49 SEO tool sites served a robots.txt file
A live benchmark of robots.txt availability, sitemap declarations, and sitemap-shaped responses across 49 SEO software sites.
2 evidence signals on this page
- Original research or data studyThis article is explicitly classified as a study.
- Linked source dataDownload the dated JSON snapshot
On September 13, 2026, 47 of 49 fixed-cohort SEO software sites returned HTTP 200 for /robots.txt. Among those 47 successful fetches, 34 declared at least one sitemap and 13 did not. All 34 declarers returned at least one sitemap-shaped response from the first three declared endpoints; 7 of the 13 non-declarers returned a sitemap-shaped response at the conventional /sitemap.xml fallback. A shaped response is not proof that every URL in the sitemap is valid or indexable.
I checked the root robots.txt and a bounded set of sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. endpoints for 49 public SEO software sites on September 13, 2026.
Key findings
- 47 of 49 sites returned HTTP 200 for
/robots.txt. One returned 403 and one returned 404. - Among the 47 readable files, 34 declared at least one sitemap and 13 did not. The other two declaration states are unknown, not zero.
- All 34 declarers returned at least one HTTP-200, sitemap-shaped response among the first three declared endpoints tested.
- Of the 13 sites with no declaration, 7 returned a sitemap-shaped response at the conventional
/sitemap.xmlfallback.
Robots.txt and sitemap discovery across 49 SEO tool sites
47 robots.txt files were readable. Of those, 34 declared a sitemap, 13 did not, and 2 sites had an unknown declaration state because robots.txt was unavailable.
View chart data
| Measure | Origins |
|---|---|
| Sitemap declared | 34/49 |
| No declaration | 13/49 |
| Declaration unknown | 2/49 |
Live endpoint observations collected September 13, 2026. Declared and conventional fallback sitemaps are separate states.
Methodology: The collector fetched one root robots.txt per fixed-cohort origin. It tested at most three declared sitemap URLs; only when a readable robots.txt declared none did it test /sitemap.xml as a labeled fallback. Public robots.txt and sitemap endpoints.
What this does and does not show
This measures live endpoint availability for an editorially selected cohort of SEO software sites. It is not a representative sample of the web. “Sitemap-shaped” means the response was HTTP 200 and contained a urlset or sitemapindex root marker. It does not validate every listed URL, canonical, status code, or lastmod value.
Download the dated JSON snapshot.
Build-time retrieval analysis plus live signals for this exact article. The automatic chunk report includes a deterministic readiness score and is ready without a model download.