Block training, keep answers? What 13,938 sites ask of AI bots
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Trakis analysed Common Crawl's September 2026 copies of robots.txt files for 13,938 readable sites and found that 17.1% of sites blocking OpenAI's GPTBot also block its ChatGPT search crawler, OAI-SearchBot.