In 27,280 coding agent runs, 64% of inference spend went to runs that failed
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Runta, working with Baseten, expanded its FrontierHarness benchmark to 22 coding-agent harness configurations, five open models and 62 terminal tasks, about 27,000 runs. Model choice explained 92.8% of cost variance, but with the model fixed, harness cost differed by 1.8 to 5.0 times, and cost rankings shifted between models. Higher cost did not predict higher pass rates. About 64.5% of spend went to runs that failed or timed out. Step count, not prompt size, tracked cost. Most figures come from Runta alone, and the models were served by one provider.