How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?
Comments on “How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
An EPFL and Apple study holds backbone, hardware and time budget fixed and finds that Malena, a single long session of a minimal coding agent, matches or beats four open-source MLE harnesses (MLEvolve, AiScientist, Arbor, ScienceFlow) on MLE-bench and NatureBench at every frontier backbone tested. Execution access is the largest effect; search strategy, delegation and parallelism add no significant gain. The exception is Gemma 4 31B, where MLEvolve leads by mean estimate. Most ablations use one backbone, and several cost and medal-rate figures lack cited sources.
Why it ranked
No. 10 of 62 with a score of 0.69. The panel rated it highest on substantiated (0.88) and new information (0.88), lowest on audience breadth (0.61).
Every answer behind this is in the table under Score.
Score
0.69total under rubric v2.2
- Significance0.68
- Reader interest0.80
- Audience breadth0.61
- Practical relevance0.71
- Substantiated0.88
- Technical depth0.86
- New information0.88
- Counts against: Promotional0.04
- Counts against: Overclaim0.38
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| Significance | 0.69 | 0.55 | 0.78 | 0.68 | 0.23 | +0.24 | +0.162 |
| Reader interest | 0.66 | 0.80 | 0.93 | 0.80 | 0.27 | +0.26 | +0.207 |
| Audience breadth | 0.58 | 0.60 | 0.66 | 0.61 | 0.08 | +0.13 | +0.080 |
| Practical relevance | 0.67 | 0.70 | 0.78 | 0.71 | 0.12 | +0.22 | +0.157 |
| Substantiated | 0.97 | 0.75 | 0.91 | 0.88 | 0.22 | +0.05 | +0.044 |
| Technical depth | 0.93 | 0.70 | 0.94 | 0.86 | 0.24 | +0.05 | +0.043 |
| New information | 0.92 | 0.85 | 0.86 | 0.88 | 0.07 | +0.05 | +0.044 |
| Promotional | 0.05 | 0.05 | 0.01 | 0.04 | 0.04 | −0.19 | −0.007 |
| Overclaim | 0.51 | 0.35 | 0.28 | 0.38 | 0.23 | −0.11 | −0.042 |
| Sum of contributions | +0.688 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.688 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-10-26/10/"><img src="https://rubric.hrih.si/badge/2026-10-26/10.svg" alt="Ranked #10 on Rubric, 26 Oct 2026" width="245" height="28"></a>Claim check
Claim check: 1 matches source · 7 no source given. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Matches sourceNumber
Across 35 releases of software-engineering harnesses, Sghaier et al. find no statistically significant improvement over a fixed backbone on SWE-bench.
The cited source says the same thing.
The article says “Across 35 releases of software-engineering harnesses, Sghaier et al. (2026) find no statistically significant improvement over a fixed backbone on SWE-bench”
The source says “there is no statistically significant improvement in SWE-bench benchmark score (i.e., resolve rates of bugs) across releases for a given fixed LLM version”
Source cited by the article: arxiv.org/abs/2607.03691
No source givenBenchmark result
Under matched conditions, Malena matches or beats every harness on every tested pair across 17 harness–backbone pairs on 30 MLE-bench tasks and 6 pairs on 40 NatureBench tasks.
The article gives no link or citation for it.
The article says “Malena matches or beats every harness on every tested pair”
The source says No source to compare with.
No source givenBenchmark result
With GLM 5.2, Malena earns a medal on 62.5% of the competitions, compared with 47.1% for the best external harness tested.
The article gives no link or citation for it.
The article says “With GLM 5.2, Malena earns a medal on 62.5% of the competitions, compared with 47.1% for the best external harness tested”
The source says No source to compare with.
No source givenGeneralization
Giving the model a coding agent environment is the largest effect measured, and no further intervention significantly improves performance.
The article gives no link or citation for it.
The article says “Giving the model a coding agent environment is by far the largest effect we measure; beyond it, no intervention significantly improves performance”
The source says No source to compare with.
No source givenNumber
Among search strategies, the largest gap is UCB1 ahead of Best-of-N by an average of 2.71 percentile points, 95% CI (-1.44,+6.60), not statistically significant.
The article gives no link or citation for it.
The article says “UCB1 ahead by an average of 2.71 percentile points (pp), 95% CI $(-1.44,+6.60)$”
The source says No source to compare with.
No source givenNumber
At GLM 5.2, Malena's modeled USD cost ($12.12) is about 6.2 times AiScientist's ($1.95).
The article gives no link or citation for it.
The article says “At GLM 5.2, Malena’s modeled USD cost ($12.12) is $\approx$ 6.2 $\times$ AiScientist’s ($1.95”
The source says No source to compare with.
No source givenFirst, only or fastest
Malena exhibits the lowest validation gap of all iterations, at 1.876pp.
The article gives no link or citation for it.
The article says “Malena exhibits the lowest validation gap of all iterations at 1.876pp”
The source says No source to compare with.
No source givenVersion
The OpenCode-based harness was built on OpenCode v1.15.6.
The article gives no link or citation for it.
The article says “We build our harness on the OpenCode v1.15.6 coding agent framework”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 2 pts, not on front page
Recorded for context only, checked 26 Oct 2026, 07:27 UTC. It played no part in the ranking.