Costs of longer agentic task lengths
Comments on “Costs of longer agentic task lengths”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
A blog post builds a turn-based Markov model of agent-assisted coding, in which longer autonomous turns raise the risk of wrong abstractions that must later be rolled back. With its default parameters, a taste-0.2 engineer peaks at 2.04× an expert's pace with 1.2-hour turns, and a taste-1 engineer at 3.11× with 3.2-hour turns. Quadrupling the time horizon adds only about 0.3×, and two parallel agents is optimal. These figures come from the author's own assumptions, not measurements, so they show a dynamic rather than a forecast.
Why it ranked
No. 10 of 127 with a score of 0.70. The panel rated it highest on new information (0.85) and reader interest (0.82), lowest on substantiated (0.51). Placed No. 10, No. 18 by score alone: the fourth AI & machine learning story of the day (−0.020). The placement terms move stories; they are not part of the score.
- AI judgeJudge 2claude-haiku-5-5
- Modeling-based analysis of agentic coding productivity with a simulation of task length and review; practically relevant to builders, but the model's assumptions are heavily stipulated and unvalidated, limiting substantiation.
- AI judgeJudge 3gpt-5.6-luna
- A technically detailed analytical model connects agent task horizons, abstraction drift, review, prompting, and swarm coordination to practical software productivity. Its conclusions are useful but depend heavily on simplifying assumptions and simulated parameters.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.70total under rubric v2.3
- Significance0.61
- Reader interest0.82
- Audience breadth0.78
- Practical relevance0.80
- Substantiated0.51
- Technical depth0.79
- New information0.85
- Counts against: Promotional0.04
- Counts against: Overclaim0.32
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.65 | 0.45 | 0.72 | 0.61 | 0.27 | +0.24 | +0.146 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.77 | 0.80 | 0.90 | 0.82 | 0.13 | +0.26 | +0.214 |
| Audience breadthHow many people who build technology does it concern? | 0.86 | 0.70 | 0.78 | 0.78 | 0.16 | +0.13 | +0.102 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.81 | 0.75 | 0.84 | 0.80 | 0.09 | +0.22 | +0.176 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.51 | 0.35 | 0.68 | 0.51 | 0.33 | +0.05 | +0.026 |
| Technical depthHow technically substantive is it? | 0.91 | 0.55 | 0.91 | 0.79 | 0.36 | +0.05 | +0.040 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.86 | 0.80 | 0.88 | 0.85 | 0.08 | +0.05 | +0.042 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.07 | 0.05 | 0.01 | 0.04 | 0.06 | −0.19 | −0.008 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.29 | 0.40 | 0.28 | 0.32 | 0.12 | −0.11 | −0.036 |
| Sum of contributions | +0.701 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.701 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-10-30/10/"><img src="https://rubric.hrih.si/badge/2026-10-30/10.svg" alt="Ranked #10 on Rubric, 30 Oct 2026" width="245" height="28"></a>Claim check
Claim check: 3 matches source · 5 no source given. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Matches sourceNumber
The claim that frontier model time horizons are doubling every 4 months is made by the cited source.
The cited source says the same thing.
The article says “the claim that time horizons at the frontier are doubling every 4 months would seem to imply a rate of productivity growth in the creation of software which is stratospheric”
The source says “The scenario’s supporting argument uses an approximately four-month doubling of AI task horizons from 2024 onward”
Source cited by the article: ai2027-tracker.com/predictions/metr-doubling
Matches sourceBenchmark result
On SlopCodeBench the latest models reach a maximum 16.3% pass rate, achieved by GPT-6 Astra XHigh, versus 14.8% for GPT-5.5 Sol in May.
The cited source says the same thing.
The article says “achieve a maximum 16.3% pass rate (achieved by GPT-6 Astra XHigh) compared to a 14.8% achieved by GPT-5.5 Sol in May”
The source says “This month, I ran the frontier models through the same campaigns, and the highest scorer was gpt-6 astra, hitting **16.3%”
Source cited by the article: github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-new-models-on-slop-code-bench.md
Matches sourceBenchmark result
Many frontier models in September scored lower on SlopCodeBench than GPT-5.5 Sol did in May.
The cited source says the same thing.
The article says “Indeed many of the frontier models in September scored lower than GPT-5.5 Sol did in May”
The source says “the frontier models we tested here had strict pass rates in the **12.2-16.3%** range”
Source cited by the article: github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-new-models-on-slop-code-bench.md
No source givenGeneralization
On SlopCodeBench there is a negative correlation between model thinking time and outcomes.
The article gives no link or citation for it.
The article says “There is even a negative correlation between thinking time and outcomes”
The source says No source to compare with.
No source givenNumber
In the author's model, a taste-0.2 engineer peaks at 2.04x an expert's pace with 1.2 h turns, and a taste-1 engineer peaks at 3.11x with 3.2 h turns.
The article gives no link or citation for it.
The article says “a taste-0.2 engineer peaks at 2.04× of an expert's pace with 1.2 h turns, delivering the 40 h project in 22 h”
The source says No source to compare with.
No source givenNumber
In the author's model, a 4x increase in H50 raises productivity for a high-taste developer only from about 3.0x to about 3.3x.
The article gives no link or citation for it.
The article says “a 4x increase in 𝐻50 corresponds to a productivity gain from ~3.0x to ~3.3x for a high taste developer”
The source says No source to compare with.
No source givenGeneralization
In the author's model, optimal productivity with default parameters is achieved with just two agents at a time, since the engineer's time saturates.
The article gives no link or citation for it.
The article says “with our default choices of parameters, optimal productivity is achieved with just two agents at a time”
The source says No source to compare with.
No source givenNumber
In the author's model, a high-taste engineer's careful review leaves 2-3 hours of latent rework versus 30-50 hours for an agent swarm.
The article gives no link or citation for it.
The article says “a high-taste engineer requiring 2-3 hours worth of rework surfaced from their careful review, compared to 30-50 hours for the swarm”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 1 pts, not on front page
Recorded for context only, checked 30 Oct 2026, 05:47 UTC. It played no part in the ranking.