Robots.txt and sitemap benchmark
47 of 49 sites served a readable robots.txt file; 34 of those 47 declared at least one sitemap.
Read the studyDated, reproducible snapshots of the technical SEO ecosystem. Every study includes its method, limitations, and downloadable source data.
47 of 49 sites served a readable robots.txt file; 34 of those 47 declared at least one sitemap.
Read the studyAcross 47 readable files, only ByteSpider had an explicit root-wide disallow, on one site.
Read the studyOne Common Crawl index contained 46 of 49 exact SEO software homepages in the fixed cohort.
Read the studyThe eight-plugin cohort spanned rounded active-install bands from 10,000 to 10 million.
Read the studyAmong 12 matched entities, 11 had an official website, 10 inception data, and 9 headquarters data.
Read the studyAll 10 packages were available and none of their latest versions carried an npm deprecation notice.
Read the studyEight of nine reviewed repositories were not archived; seven had received a push since May 2026.
Read the studyAll seven snapshots were collected September 13, 2026.