A skeleton from 20 observations beat 160 tests, a cheap model nearly caught up
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
An AWS paper, Skeleton Coding, finds that giving coding agents a program's structure (types, modules, signatures, contracts) in code-like form beats adding more behavioral tests. On SlopCodeBench the mean pass rate rose from 71.7% to 89.2%, and on SDDBench from 29% to 74.2%. The smaller GPT-5.6 Luna came within 5.0 to 8.2 points of Sol at far lower cost. Gains shrank on ProgramBench, where an oracle exists, and overly detailed contracts sometimes hurt. Andreas Rau notes that the benchmarks may be contaminated.