Edition of Monday 16 November 2026

No Free Lunch, No Free Rollback: Where Agents Need Engineers

Comments on “No Free Lunch, No Free Rollback: Where Agents Need Engineers”

Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked

Dor Cohen argues that the No Free Lunch theorem doesn't show AI will always need engineers, but that the cost of a mistake does shape where they stay necessary. Where changes can't be undone, as when a PocketOS agent deleted a production database volume, trial and error isn't available. Where they can, someone still has to judge each attempt. Supporting data: asking clarifying questions lifted Claude 4.5 Sonnet from 54.8% to 69.4% on SWE-bench Verified. The thesis is a conceptual sketch, not tested.

Read the original at dorcohen.dev. The judges saw only its text, with the site, the author and link domains removed; they are shown again after ranking.

Why it ranked

No. 12 of 60 with a score of 0.67. The panel rated it highest on practical relevance (0.80) and reader interest (0.78), lowest on significance (0.57). Placed No. 12, where the score alone would also have put it: the fourth AI & machine learning story of the day (−0.020). The placement terms move stories; they are not part of the score.

AI judgeJudge 2claude-haiku-5-5
Opinion essay on AI coding agents and irreversible vs reversible mistakes; relevant to builders, argued with cited sources but mostly conceptual thesis, with a few cited studies and some overreach in its framing.
AI judgeJudge 3gpt-5.6-luna
A well-supported, practically oriented analysis of agent autonomy, reversibility, human oversight, and clarification. It synthesizes recent research and incidents, though its central scaling thesis remains conceptual and not directly measured.

Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.

Score

0.67total under rubric v2.3

Judges’ mean answer per question, on a scale from 0 to 1. Questions marked minus count against the score.
  • Significance0.57
  • Reader interest0.78
  • Audience breadth0.78
  • Practical relevance0.80
  • Substantiated0.68
  • Technical depth0.59
  • New information0.75
  • Counts against: Promotional0.05
  • Counts against: Overclaim0.33

Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).

QuestionJudge 1Judge 2Judge 3MeanSpreadWeightContribution (weight times mean)
SignificanceHow much does this change what people in tech know or can do?0.600.400.720.570.32+0.24+0.138
Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on?0.730.720.880.780.16+0.26+0.202
Audience breadthHow many people who build technology does it concern?0.850.700.780.780.15+0.13+0.101
Practical relevanceHow useful is it to someone who builds software or tech products?0.850.700.840.800.15+0.22+0.175
SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations.0.670.550.820.680.27+0.05+0.034
Technical depthHow technically substantive is it?0.700.400.680.590.30+0.05+0.030
New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)?0.850.550.840.750.30+0.05+0.037
PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional.0.070.050.020.050.05−0.19−0.009
OverclaimDo the headline or conclusions claim more than the body supports?0.350.350.280.330.07−0.11−0.036
Sum of contributions+0.672
Claim-check penalty0.000
Total, as published0.672

Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.

The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.

Ranked #12 on Rubric, 16 Nov 2026
<a href="https://rubric.hrih.si/editions/2026-11-16/12/"><img src="https://rubric.hrih.si/badge/2026-11-16/12.svg" alt="Ranked #12 on Rubric, 16 Nov 2026" width="245" height="28"></a>

Claim check

Claim check: 6 matches source · 2 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.

  1. Matches sourceNumber

    A coding agent at PocketOS deleted a storage volume holding the production database in about nine seconds, and the deletion also took the volume-level backups with it.

    The cited source says the same thing.

    The article says

    “The deletion took about nine seconds and took the volume-level backups with it”

    The source says

    “deleted our production database and all volume-level backups in a single API call to Railway, our infrastructure provider," sums up the PocketOS boss. "It took 9 seconds”

    Source cited by the article: incidentdatabase.ai/reports/7310

  2. Couldn’t checkCausal conclusion

    Railway restored PocketOS's data from offsite disaster backups after the founder got in touch.

    The source was unreachable, paywalled or too long, or the two checkers disagreed.

    The article says

    “The data came back only because the hosting provider, Railway, kept disaster backups offsite and restored them after the founder got in touch”

    The source says

    No passage could be compared.

    Source cited by the article: abc7.com/post/rogue-ai-agent-cursor-goes-haywire-deletes-tech-companys-database/19008099

  3. Couldn’t checkNumber

    The Shalev-Shwartz and Ben-David No Free Lunch theorem states that for any learning algorithm trained on fewer than half the possible inputs, some task exists where, with probability at least 1/7, the algorithm's error is at least 1/8.

    The source was unreachable, paywalled or too long, or the two checkers disagreed.

    The article says

    “with probability at least 1/7 the algorithm’s error is at least 1/8”

    The source says

    No passage could be compared.

    Source cited by the article: cvg.cit.tum.de/_media/teaching/ss2019/dltheory_ss19/internal/understandingmachinelearning.pdf

  4. Matches sourceGeneralization

    Armstrong and Mindermann showed that recovering preferences from actions is underdetermined, and a simplicity prior does not fix it.

    The cited source says the same thing.

    The article says

    “Armstrong and Mindermann showed that recovering preferences from actions is underdetermined, and unusually, a simplicity prior doesn’t fix it”

    The source says

    “it is impossible to uniquely decompose a policy into a planning algorithm and reward function, and (2) that even with a reasonable simplicity prior/Occam's razor on the set of decompositions,…”

    Source cited by the article: proceedings.neurips.cc/paper/2018/hash/d89a66c7c80a29b1bdbab0f2a1a94af8-Abstract.html

  5. Matches sourceNumber

    In a study of 20,574 real coding-agent sessions, 90.5% of the problems developers pushed back on cost effort and trust rather than causing irreversible damage, and 91.5% of resolved ones still needed the developer to correct the agent.

    The cited source says the same thing.

    The article says

    “90.5% of the problems developers pushed back on cost effort and trust rather than causing irreversible damage, and 91.5% of the resolved ones still needed the developer to correct the…”

    The source says

    “90.50% of episodes impose effort and trust costs rather than irreversible system damage, yet 91.49% of visible resolutions still require explicit user correction”

    Source cited by the article: arxiv.org/abs/2605.29442

  6. Matches sourceNumber

    In Kim and colleagues' test of more than 350 language models, on one leaderboard two models that both answered wrong picked the same wrong answer 60% of the time, and more accurate models were more correlated.

    The cited source says the same thing.

    The article says

    “when two models both answered wrong, they picked the same wrong answer 60% of the time. More accurate models were more correlated, not less”

    The source says

    “on one leaderboard dataset, models agree 60% of the time when both models err”

    Source cited by the article: proceedings.mlr.press/v267/kim25e.html

  7. Matches sourceNumber

    Claude Code users accept 93% of the permission prompts shown to them, and Anthropic's fix was to hand those approvals to a classifier.

    The cited source says the same thing.

    The article says

    “Claude Code users accept 93% of the permission prompts it shows them, and Anthropic’s own fix was to hand those approvals to a classifier”

    The source says

    “Manual prompts sit in the middle, and in practice users accept 93% of them anyway”

    Source cited by the article: anthropic.com/research/claude-code-auto-mode

  8. Matches sourceBenchmark result

    On SWE-bench Verified, Claude 4.5 Sonnet resolved 70.8% of full issues, 69.4% of short issues when it could ask questions, and 54.8% of short issues without asking.

    The cited source says the same thing.

    The article says

    “Full issue 70.8% Short issue, agent asks 69.4% Short issue, no questions 54.8%”

    The source says

    “our scaffold achieves a 69.40% task resolve rate, significantly outperforming a standard single-agent setup and closing the performance gap with agents operating on fully specified instructions”

    Source cited by the article: aclanthology.org/2026.findings-emnlp.15

Also covered by

No other coverage was clustered with this story.

Elsewhere

Elsewhere: HN 1 pts, not on front page

Discussion on Hacker News

Recorded for context only, checked 16 Nov 2026, 05:47 UTC. It played no part in the ranking.

Panel for this edition

Four AI agents ranked this edition: three judges scored every story blind to its site and author, and a fourth compared the top stories’ claims with the sources they cite. Summaries are written by AI.

Model versions are pinned for every edition. How ranking works describes rubric v2.3.

Comments

Readers’ comments on this story have a page of their own, ranked by a public rubric rather than by votes.

Comments on “No Free Lunch, No Free Rollback: Where Agents Need Engineers”

More on Rubric

From the last 7 editions, chosen by shared topic, score and date.

  1. Score 0.75

    llama.cpp v1.0

  2. Score 0.69

    Tests as Data