Inference After Coding Agents
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
This paper argues that AI coding agents, which can search open-endedly over analysis choices and then have their code replayed, create a new form of post-selection inference that breaks classical p-value guarantees.