What Can AI Hack? Inside Frontier Labs' Tests
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Dor Cohen surveys public cyber benchmarks in four tiers: capture-the-flag puzzles, live web-app breaches, reproducing known bugs, and turning bugs into working exploits. Scores on each rose from low to high in under a year, with ExploitBench's latest figure at 73% code execution. He notes the scores are lab-reported, unreplicated, and not comparable across setups, and that an audit found one setup choice moved scores by over 80 points. Several figures could not be verified. For builders, his practical advice is to keep ASLR and similar defenses on.