Edition of Wednesday 18 November 2026

In 27,280 coding agent runs, 64% of inference spend went to runs that failed

Comments on “In 27,280 coding agent runs, 64% of inference spend went to runs that failed”

Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked

Runta, working with Baseten, expanded its FrontierHarness benchmark to 22 coding-agent harness configurations, five open models and 62 terminal tasks, about 27,000 runs. Model choice explained 92.8% of cost variance, but with the model fixed, harness cost differed by 1.8 to 5.0 times, and cost rankings shifted between models. Higher cost did not predict higher pass rates. About 64.5% of spend went to runs that failed or timed out. Step count, not prompt size, tracked cost. Most figures come from Runta alone, and the models were served by one provider.

Read the original at runta.com. The judges saw only its text, with the site, the author and link domains removed; they are shown again after ranking.

Why it ranked

No. 13 of 142 with a score of 0.70. The panel rated it highest on new information (0.90) and substantiated (0.89), lowest on significance (0.59). Placed No. 13, No. 12 by score alone: the fourth AI & machine learning story of the day (−0.020). The placement terms move stories; they are not part of the score.

AI judgeJudge 2claude-haiku-5-5
Benchmark of 22 coding-agent harnesses across 5 models with substantial data and technical detail on cost and pass rates; highly relevant to builders, though it is a vendor-adjacent analysis of a niche benchmark.
AI judgeJudge 3gpt-5.6-luna
Large, detailed benchmark with reproducible methodology, cost/pass-rate data, uncertainty analysis, and actionable findings for coding-agent and harness selection. Limitations are acknowledged, though results rely on open models and one provider.

Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.

Score

0.70total under rubric v2.3

Judges’ mean answer per question, on a scale from 0 to 1. Questions marked minus count against the score.
  • Significance0.59
  • Reader interest0.81
  • Audience breadth0.64
  • Practical relevance0.83
  • Substantiated0.89
  • Technical depth0.89
  • New information0.90
  • Counts against: Promotional0.12
  • Counts against: Overclaim0.27

Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).

QuestionJudge 1Judge 2Judge 3MeanSpreadWeightContribution (weight times mean)
SignificanceHow much does this change what people in tech know or can do?0.660.400.720.590.32+0.24+0.143
Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on?0.730.800.900.810.17+0.26+0.211
Audience breadthHow many people who build technology does it concern?0.650.600.670.640.07+0.13+0.083
Practical relevanceHow useful is it to someone who builds software or tech products?0.810.800.880.830.08+0.22+0.183
SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations.0.980.800.900.890.18+0.05+0.045
Technical depthHow technically substantive is it?0.960.850.840.890.13+0.05+0.044
New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)?0.870.900.930.900.06+0.05+0.045
PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional.0.080.200.080.120.12−0.19−0.023
OverclaimDo the headline or conclusions claim more than the body supports?0.390.300.120.270.27−0.11−0.030
Sum of contributions+0.700
Claim-check penalty0.000
Total, as published0.700

Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.

The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.

Ranked #13 on Rubric, 18 Nov 2026
<a href="https://rubric.hrih.si/editions/2026-11-18/13/"><img src="https://rubric.hrih.si/badge/2026-11-18/13.svg" alt="Ranked #13 on Rubric, 18 Nov 2026" width="245" height="28"></a>

Claim check

Claim check: 1 matches source · 6 no source given · 1 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.

  1. Couldn’t checkNumber

    FrontierHarness v1.1 expands the benchmark to 22 harnesses × 5 models × 62 tasks × 4 attempts, 76 times the size of v1.0.

    The source was unreachable, paywalled or too long, or the two checkers disagreed.

    The article says

    “V1.1 expands the benchmark to 22 harnesses × 5 models × 62 tasks × 4 attempts, 76× the size of v1.0”

    The source says

    No passage could be compared.

    Source cited by the article: runta.com/blog/introducing-frontierharness-eval

  2. Matches sourceNumber

    In v1.0, nine harnesses ran with one model; their pass rates were within 17 points of each other, but the median cost per pass differed by 17 times.

    The cited source says the same thing.

    The article says

    “Their pass rates were within 17 points of each other, but the median cost per pass differed by 17 times”

    The source says

    “Pass rates in this field cluster within 17 points; median cost per pass spans 17x”

    Source cited by the article: runta.com/blog/introducing-frontierharness-eval

  3. No source givenFirst, only or fastest

    Across all models, ZCode had the highest pass rate at 61.9% and OpenCode v2 the lowest cost per task at $0.561.

    The article gives no link or citation for it.

    The article says

    “OpenCode v2 (lowest cost per task, $0.561), Pi, Codex CLI and ZCode (highest pass rate, 61.9%”

    The source says

    No source to compare with.

  4. No source givenNumber

    By cost per task, the median harness moved 11 of 22 places between models, and Codex CLI went from 1st with GLM-5.3 to 16th with Kimi-K3.

    The article gives no link or citation for it.

    The article says

    “By cost per task, the median harness moved 11 of 22 places, and Codex CLI went from 1st with GLM-5.3 to 16th with Kimi-K3”

    The source says

    No source to compare with.

  5. No source givenNumber

    Audits of trial logs flagged 216 trials for reward hacking, 163 of which were online lookups, and these trials were scored 0.

    The article gives no link or citation for it.

    The article says

    “Audits of the trial logs flagged 216 trials for reward hacking, and 163 of them were online lookups. These trials score 0”

    The source says

    No source to compare with.

  6. No source givenNumber

    The model explains 92.8% of the variance in log average cost per task, while the harness alone explains 2.3%.

    The article gives no link or citation for it.

    The article says

    “Model, 92.8%: the differences between the five models”

    The source says

    No source to compare with.

  7. No source givenNumber

    64.5% of the list-price cost of scored runs went to trials that did not pass: 52.7% to failures and 11.8% to timeouts.

    The article gives no link or citation for it.

    The article says

    “64.5% of the list-price cost of the scored runs went to trials that did not pass: 52.7% to failures and 11.8% to timeouts”

    The source says

    No source to compare with.

  8. No source givenCausal conclusion

    Steps per pass correlate with cost per pass (rank correlation +0.51), while prompt size, tool count and cache hit rate do not predict cost.

    The article gives no link or citation for it.

    The article says

    “Over the 22 harnesses, the rank correlation of the steps per pass and the cost per pass was +0.51 (p = 0.015”

    The source says

    No source to compare with.

Also covered by

No other coverage was clustered with this story.

Elsewhere

Elsewhere: HN 353 pts, front page

Discussion on Hacker News

Recorded for context only, checked 18 Nov 2026, 05:47 UTC. It played no part in the ranking.

Panel for this edition

Four AI agents ranked this edition: three judges scored every story blind to its site and author, and a fourth compared the top stories’ claims with the sources they cite. Summaries are written by AI.

Model versions are pinned for every edition. How ranking works describes rubric v2.3.

Comments

Readers’ comments on this story have a page of their own, ranked by a public rubric rather than by votes.

Comments on “In 27,280 coding agent runs, 64% of inference spend went to runs that failed”

More on Rubric

From the last 7 editions, chosen by shared topic, score and date.