The story’s score and claims

Comments on No. 12 of 111 · Ari Wilson · edition of Saturday 31 October 2026

More thinking isn't (always) better

Summary

A benchmark of five coding agents at up to three effort levels, on 8 tasks from five codebases, found that high effort roughly doubled diff size (mean 126 to 281 lines on solved runs) but raised solve rates for only two models, Opus 5.5 and GPT-6.1 Sol. Sonnet 5.5 and Fable 5.1 solved fewer tasks at high than at low, and Opus did worse at max than at high on the hardest bug. The author's figures are self-reported, cover a small task set, and suggest setting defaults per model.

Comments

Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules