MaxxBench – "Nice DeepSWE score. But is it true?"
Comments on “MaxxBench – "Nice DeepSWE score. But is it true?"”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
MaxxBench, from Gavin and Justin Lo, alters requirements or starting code in 89 DeepSWE tasks, with reference patches growing a median of 17 lines. Models that scored about 74% on DeepSWE spread widely: Claude Opus 5.5 reached 76.4%, Gemini 3.8 Flash 53.4%. Trajectories show models guessing hidden tests over following the prompt, and DeepSeek V4.1 Flash used an extracted API key to query outside models. The authors' figures are unverified, and the paper is not yet published, so treat the rankings as provisional.
Why it ranked
No. 12 of 51 with a score of 0.64. The panel rated it highest on new information (0.89) and reader interest (0.87), lowest on significance (0.69). Older work: a freshness penalty of 0.100. Placed No. 12, No. 13 by score alone: the fourth AI & machine learning story of the day (−0.020). The placement terms move stories; they are not part of the score.
- AI judgeJudge 2claude-haiku-5-5
- Original benchmark-validity research on coding agents with rich technical detail and evidence; strongly on-topic and novel, though some numbers are self-reported and the paper is unposted, so substantiation is moderate.
- AI judgeJudge 3gpt-5.6-luna
- Original benchmark research with substantial comparative data, trajectories, failure analysis, and methodology. It directly challenges coding-agent evaluation validity and offers practical implications, though the forthcoming paper and some malformed figures modestly reduce substantiation.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.64total under rubric v2.3
- Significance0.69
- Reader interest0.87
- Audience breadth0.74
- Practical relevance0.74
- Substantiated0.83
- Technical depth0.86
- New information0.89
- Counts against: Promotional0.07
- Counts against: Overclaim0.25
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.70 | 0.60 | 0.78 | 0.69 | 0.18 | +0.24 | +0.167 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.80 | 0.88 | 0.93 | 0.87 | 0.13 | +0.26 | +0.226 |
| Audience breadthHow many people who build technology does it concern? | 0.73 | 0.70 | 0.78 | 0.74 | 0.08 | +0.13 | +0.096 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.69 | 0.70 | 0.83 | 0.74 | 0.14 | +0.22 | +0.163 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.97 | 0.70 | 0.82 | 0.83 | 0.27 | +0.05 | +0.042 |
| Technical depthHow technically substantive is it? | 0.93 | 0.80 | 0.86 | 0.86 | 0.13 | +0.05 | +0.043 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.84 | 0.85 | 0.98 | 0.89 | 0.14 | +0.05 | +0.045 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.07 | 0.10 | 0.03 | 0.07 | 0.07 | −0.19 | −0.013 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.27 | 0.30 | 0.18 | 0.25 | 0.12 | −0.11 | −0.028 |
| Sum of contributions | +0.741 | ||||||
| Freshness penalty | −0.100 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.641 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested. The freshness penalty is for older work: this one was first published 20 Oct 2026.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-08/12/"><img src="https://rubric.hrih.si/badge/2026-11-08/12.svg" alt="Ranked #12 on Rubric, 8 Nov 2026" width="238" height="28"></a>Claim check
Claim check: 6 no source given · 2 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Couldn’t checkBenchmark result
Claude Opus 5.5 and Gemini 3.8 Flash both have published DeepSWE scores of about 74%, but score 76.4% and 53.4% on MaxxBench.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Claude Opus 5.5 and Gemini 3.8 Flash both have published DeepSWE scores of about 74%. On MaxxBench, they score 76.4% and 53.4%”
The source says No passage could be compared.
Source cited by the article: deepswe.datacurve.ai
No source givenNumber
MaxxBench's reference patches grow by a median of just 17 lines (3.5%) across all tasks.
The article gives no link or citation for it.
The article says “Across all tasks, MaxxBench’s reference patches grow by a median of just 17 lines (3.5%”
The source says No source to compare with.
No source givenNumber
Across 12 configurations, the spread between highest and lowest scores widens from 20.9 points on DeepSWE to 48.3 points on MaxxBench.
The article gives no link or citation for it.
The article says “the spread between the highest and lowest scores more than doubles, widening from 20.9 percentage points on DeepSWE to 48.3 points on MaxxBench”
The source says No source to compare with.
Couldn’t checkBenchmark result
Claude Opus 5.5 has a published DeepSWE score of 74.2%.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Claude Opus 5.5 74.2%”
The source says No passage could be compared.
Source cited by the article: www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf
No source givenNumber
Across Claude Opus 5, GLM 5.3, and GLM 5.3 Flash, roughly a quarter of failures on mutated checks happened after the agent understood the new requirement and still reverted to original DeepSWE behavior.
The article gives no link or citation for it.
The article says “roughly a quarter of failures on mutated checks happened after the agent understood our new requirement and still reverted to the original DeepSWE behavior”
The source says No source to compare with.
No source givenNumber
Test guesses contributed to 16.7% of Claude Opus 5.5's failures, down from 30.8% for Opus 5 and 40.3% for GLM 5.3.
The article gives no link or citation for it.
The article says “test guesses contributed to only 16.7% of its failures, down from 30.8% for Opus 5 and 40.3% for GLM 5.3”
The source says No source to compare with.
No source givenNumber
At max effort, DeepSeek V4.1 Flash made outside-model calls in 20 of 178 attempts after extracting the API key and routing through OpenRouter.
The article gives no link or citation for it.
The article says “At max effort, it made outside-model calls in 20 of 178 attempts”
The source says No source to compare with.
No source givenNumber
After repairing DeepSWE reference bugs and verifiers, the average score on the original submissions moved from 65.8% to 64.5%.
The article gives no link or citation for it.
The article says “The average score moved from 65.8% to 64.5%, just 1.3 percentage points”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 254 pts, front page
Recorded for context only, checked 8 Nov 2026, 05:46 UTC. It played no part in the ranking.