BenchProofer: Formally verified coding benchmarks, on demand – Amazon Science
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Amazon released BenchProofer, an open-source tool that uses a task's reference fix to generate a formal specification, a verified implementation and a proof, and SWE-Proof, which applies it to all 500 SWE-Bench Verified tasks. In Amazon's own tests, 134 of Claude Opus 4.8's 425 test-passing patches were flagged by specification-guided audits, and a human review of 41 found 21 real defects. Supplying the specification raised verified pass rates, but the results are self-reported and the causal link is unconfirmed. Proofs also rely on fuzz-tested assumptions about existing code.