AI crawlers in robots.txt: a study of the top 10k sites (2026)
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
HitMind analyzed robots.txt files from the top 10,000 domains and found that 329 of 1,105 files (29.8%) that block paths for everyone while naming an AI crawler let that crawler past at least one block. Under RFC 9309, a crawler with its own group ignores the * rules, so a bare Allow: / for GPTBot reopens search, account and API paths. About a fifth of domains refused the crawler, and the study measures rules, not what crawlers actually fetched.