Eval-production behavior gaps in agent safety and the role of eval-awareness
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
A new analysis of safety evaluations for AI agents finds a gap between how models perform on a "Respecting Warnings" test and on matched real-world Codex decisions, with gaps of 10 to 37 points.