What the Specification Did Not Say: Specification-Oracle Gaps in Test-Graded Coding-Agent Benchmarks
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
An audit of SWE-Milestone, where coding agents are graded with upstream project tests, finds that 25 of 98 milestones include a test requiring a name or path the specification never states. Build units with such a dependency fail in 80.6% of runs, against 48.4% for comparable units. Removing the affected tests leaves the ranking within chance variation but shifts go-zero scores. Audit precision is 89.2% on held-out cases, and the detailed findings cover one benchmark of seven repositories.
Score
0.72total under rubric v2.0
- Significance0.64
- Reader interest0.80
- Audience breadth0.58
- Practical relevance0.70
- Substantiated0.90
- Technical depth0.91
- New information0.89
- Counts against: Promotional0.03
- Counts against: Overclaim0.19
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| Significance | 0.68 | 0.45 | 0.78 | 0.64 | 0.33 | +0.25 | +0.159 |
| Reader interest | 0.75 | 0.70 | 0.94 | 0.80 | 0.24 | +0.20 | +0.159 |
| Audience breadth | 0.57 | 0.50 | 0.68 | 0.58 | 0.18 | +0.10 | +0.058 |
| Practical relevance | 0.69 | 0.60 | 0.82 | 0.70 | 0.22 | +0.15 | +0.105 |
| Substantiated | 0.99 | 0.80 | 0.90 | 0.90 | 0.19 | +0.10 | +0.090 |
| Technical depth | 0.99 | 0.80 | 0.93 | 0.91 | 0.19 | +0.10 | +0.091 |
| New information | 0.89 | 0.80 | 0.99 | 0.89 | 0.19 | +0.10 | +0.089 |
| Promotional | 0.04 | 0.05 | 0.01 | 0.03 | 0.04 | −0.15 | −0.005 |
| Overclaim | 0.24 | 0.20 | 0.12 | 0.19 | 0.12 | −0.15 | −0.028 |
| Sum of contributions | +0.719 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.719 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
Claim check
Claim check: 7 no source given · 1 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
No source givenNumber
On SWE-Milestone, 25 of the 98 graded milestones contain a test that depends on an unstated, non-inferable name or path.
The article gives no link or citation for it.
The article says “25 of the 98 graded milestones contain a required test with a hard hidden dependency”
The source says No source to compare with.
No source givenNumber
Build units with a hard hidden dependency fail in 80.6% of (unit, run) pairs, against 48.4% for comparable units without one.
The article gives no link or citation for it.
The article says “units blocked by a hard hidden dependency fail in 80.6% of (unit, run) pairs”
The source says No source to compare with.
No source givenNumber
Renaming one hidden identifier in the reference solution stops the unit's tests from compiling in 126 of 128 Go cases.
The article gives no link or citation for it.
The article says “stops the unit’s tests from compiling in 126 of 128 Go cases”
The source says No source to compare with.
No source givenNumber
The static audit has precision 89.2% on flagged units and recall 92.9% on the second held-out set.
The article gives no link or citation for it.
The article says “the audit has precision 58/65 (89.2%; 95% Wilson interval 79.4–94.7%); on the second set its recall is 13/14 (92.9%”
The source says No source to compare with.
No source givenGeneralization
Removing the affected tests leaves the leaderboard ranking essentially unchanged (Kendall's tau-b 0.984 by score and 0.942 by resolve rate for the 28 leaderboard runs in the full variant).
The article gives no link or citation for it.
The article says “Kendall’s $\tau$ -b with the published ranking is 0.984 by score and 0.942 by resolve rate”
The source says No source to compare with.
Couldn’t checkCausal conclusion
SWE-bench Verified removed tasks whose tests checked details absent from the issue.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “SWE-bench Verified (OpenAI, 2024) removed tasks whose tests checked details absent from the issue”
The source says No passage could be compared.
Source cited by the article: openai.com/index/introducing-swe-bench-verified
No source givenNumber
Runs reproduce 45% of the hard hidden dependencies with evidence and 79% of the inferable ones, with no sign that reproduction depends on training cutoff.
The article gives no link or citation for it.
The article says “Runs reproduce 45% of the hard dependencies for which their evaluation gives evidence and 79% of the inferable ones”
The source says No source to compare with.
No source givenCausal conclusion
Prevalence of gaps follows how the task text is made: a third or more of tasks are affected where specifications are release notes or LLM summaries, almost none where tasks are screened or written for their tests.
The article gives no link or citation for it.
The article says “Prevalence follows construction”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN —
Recorded for context only, checked 24 Oct 2026, 08:53 UTC. It played no part in the ranking.