Which big sites block AI training crawlers but let AI search in
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
A Show HN study checked robots.txt for 945 well-known websites against 14 AI crawlers as of October 2026. About 26% of the 746 readable sites block at least one crawler, and 9.2% block AI training while leaving AI search open.