A skeleton from 20 observations beat 160 tests, a cheap model nearly caught up
Comments on “A skeleton from 20 observations beat 160 tests, a cheap model nearly caught up”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
An AWS paper, Skeleton Coding, finds that giving coding agents a program's structure (types, modules, signatures, contracts) in code-like form beats adding more behavioral tests. On SlopCodeBench the mean pass rate rose from 71.7% to 89.2%, and on SDDBench from 29% to 74.2%. The smaller GPT-5.6 Luna came within 5.0 to 8.2 points of Sol at far lower cost. Gains shrank on ProgramBench, where an oracle exists, and overly detailed contracts sometimes hurt. Andreas Rau notes that the benchmarks may be contaminated.
Why it ranked
No. 6 of 43 with a score of 0.67. The panel rated it highest on new information (0.87) and practical relevance (0.79), lowest on significance (0.60). The judges disagreed most on overclaim (spread 0.46), so the story is marked contested.
- AI judgeJudge 2claude-haiku-5-5
- Summarizes a research paper on skeleton-first specs for AI coding agents with concrete benchmark results; substantive and practical, though based on academic benchmarks and partly speculative commentary.
- AI judgeJudge 3gpt-5.6-luna
- Reports empirical and theoretical results on structural specifications for coding agents, with concrete benchmarks, cost and iteration findings. It is technically useful and novel, though some headline comparisons rely on curated benchmarks and incomplete supporting detail.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.67total under rubric v2.3Contestedjudges differ by 0.46 on overclaim
- Significance0.60
- Reader interest0.79
- Audience breadth0.66
- Practical relevance0.79
- Substantiated0.70
- Technical depth0.73
- New information0.87
- Counts against: Promotional0.05
- Counts against: Overclaim0.37
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.67 | 0.45 | 0.68 | 0.60 | 0.23 | +0.24 | +0.144 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.73 | 0.72 | 0.91 | 0.79 | 0.19 | +0.26 | +0.205 |
| Audience breadthHow many people who build technology does it concern? | 0.70 | 0.55 | 0.72 | 0.66 | 0.17 | +0.13 | +0.086 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.86 | 0.70 | 0.82 | 0.79 | 0.16 | +0.22 | +0.174 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.71 | 0.60 | 0.78 | 0.70 | 0.18 | +0.05 | +0.035 |
| Technical depthHow technically substantive is it? | 0.79 | 0.65 | 0.76 | 0.73 | 0.14 | +0.05 | +0.037 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.86 | 0.80 | 0.96 | 0.87 | 0.16 | +0.05 | +0.044 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.08 | 0.05 | 0.02 | 0.05 | 0.06 | −0.19 | −0.010 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.64 | 0.30 | 0.18 | 0.37 | 0.46 , above the contested threshold | −0.11 | −0.041 |
| Sum of contributions | +0.673 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.673 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-02/6/"><img src="https://rubric.hrih.si/badge/2026-11-02/6.svg" alt="Ranked #6 on Rubric, 2 Nov 2026" width="231" height="28"></a>Claim check
Claim check: 8 matches source. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Matches sourceBenchmark result
Skeleton Coding, from an AWS team, reports that a skeleton built from 20 observations beat 160 curated behavioral observations.
The cited source says the same thing.
The article says “a skeleton built from 20 observations beat 160 curated behavioral observations”
The source says “For any number of curated observations, the performance is strictly worse than that of a skeleton with just 20 observations”
Source cited by the article: arxiv.org/abs/2610.38214
Matches sourceBenchmark result
On SlopCodeBench, skeletons raised the mean pass rate from 71.7% to 89.2%, while behavioral specs added at most 9.4 points.
The cited source says the same thing.
The article says “SlopCodeBench mean pass rate: 71.7% to 89.2%. Behavioral specs added at most 9.4 points”
The source says “Across SCBench [ 12 ] and SDDBench, structural specifications raise mean pass rate from 71.7% to 89.2% and from 29% to 74.2%, respectively, whereas behavioral specifications improve pass rate by…”
Source cited by the article: arxiv.org/abs/2610.38214
Matches sourceBenchmark result
On SDDBench, skeletons raised the pass rate from 29% to 74.2%, while function stubs alone got 53% to 66.4%.
The cited source says the same thing.
The article says “SDDBench: 29% to 74.2%. Function stubs alone got 53% to 66.4%”
The source says “Adding function stubs alone raises SDDBench mean pass rate from 53% to 66.4%”
Source cited by the article: arxiv.org/abs/2610.38214
Matches sourceNumber
Across all models and benchmarks, relative test error fell 48.4% (from 40.7% to 21.0%) against the best end-to-end behavioral spec.
The cited source says the same thing.
The article says “relative test error fell 48.4% (40.7% to 21.0%) against the best end-to-end behavioral spec”
The source says “Together with Programbench, across 4 models the relative error rates reduces by 48.4% from 40.7% to 21.0% (see Figures 2 and 10 ) compared to the best end-to-end behavioral specifications”
Source cited by the article: arxiv.org/abs/2610.38214
Matches sourceNumber
With global invariants, GPT-5.6 Luna landed within 5.0 to 8.2 points of GPT-5.6 Sol at roughly 15× and 26× lower cost.
The cited source says the same thing.
The article says “It landed within 5.0 to 8.2 points of Sol at roughly 15× and 26× lower cost”
The source says “These results come within 5.0–8.2 percentage points of the best GPT-5.6 Sol [ 14 ] results at approximately 15 \times and 26 \times lower implementation cost, respectively”
Source cited by the article: arxiv.org/abs/2610.38214
Matches sourceNumber
Structural specs reduced erosion from 0.66 to 0.56 on SlopCodeBench and from 0.41 to 0.24 on SDDBench, while behavioral specs raised it to as much as 0.73 and 0.43.
The cited source says the same thing.
The article says “Erosion fell from 0.66 to 0.56 on SlopCodeBench and from 0.41 to 0.24 on SDDBench with structural specs”
The source says “signature specifications reduce code erosion from 0.66 to 0.56 on SCBench and from 0.41 to 0.24 on SDDBench relative to the task-only condition, while behavioral specifications increase erosion to as…”
Source cited by the article: arxiv.org/abs/2610.38214
Matches sourceBenchmark result
On ProgramBench, adding a skeleton with local contracts moved the mean from 64.0% to 70.9%, a 10.9% relative gain, with oracle access alone giving 61.5%.
The cited source says the same thing.
The article says “Oracle access alone already gave 61.5%, and adding a skeleton with local contracts moved the mean from 64.0% to 70.9%, a 10.9% relative gain”
The source says “adding a skeleton with local contracts increases mean pass rate by 10.9% relative (from 64.0% to 70.9%) to the same setup without a skeleton”
Source cited by the article: arxiv.org/abs/2610.38214
Matches sourceNumber
In the repair setting, by round 8 the skeleton workflow reached similar error rates with 32% fewer reference-binary calls.
The cited source says the same thing.
The article says “by round 8 the skeleton workflow reached similar error rates with 32% fewer reference-binary calls”
The source says “By round 8, the skeleton workflow reaches similar error rates with 32% fewer reference-binary calls”
Source cited by the article: arxiv.org/abs/2610.38214
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 3 pts, not on front page
Recorded for context only, checked 2 Nov 2026, 05:46 UTC. It played no part in the ranking.