Edition of Tuesday 17 November 2026

A benchmark that diffs the whole database after an AI agent's task

Comments on “A benchmark that diffs the whole database after an AI agent's task”

Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked

A solo developer's open-source benchmark runs computer-use agents in Invoice Ninja, Dolibarr and Odoo 17 and 18, then diffs the whole database to separate success from collateral damage, irreversible changes and false completion. Between Odoo 17 and 18, reported success stayed near 77% while collateral damage rose from 15% to 21%, partly because "Reset to Draft" silently un-reconciles payments. A 3-step task never succeeded in 50 runs. The author notes small samples, and the figures could not be independently verified.

Read the original at github.com. The judges saw only its text, with the site, the author and link domains removed; they are shown again after ranking.

Why it ranked

No. 20 of 122 with a score of 0.65. The panel rated it highest on new information (0.92) and technical depth (0.86), lowest on significance (0.54). Placed No. 20, No. 31 by score alone: the fourth AI & machine learning story of the day (−0.020). The placement terms move stories; they are not part of the score.

AI judgeJudge 2claude-haiku-5-5
Original benchmark research on computer-use agents with state-diff grading; substantive and novel, but small-sample, narrow audience, and a self-described Phase 0 project with a repo to try.
AI judgeJudge 3gpt-5.6-luna
A reproducible benchmark with code, deterministic tasks, database-state grading, detailed limitations, and concrete results reveals failure modes hidden by success rates. It is highly actionable for agent evaluation, though samples are small and the project is early-stage.

Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.

Score

0.65total under rubric v2.3

Judges’ mean answer per question, on a scale from 0 to 1. Questions marked minus count against the score.
  • Significance0.54
  • Reader interest0.78
  • Audience breadth0.54
  • Practical relevance0.70
  • Substantiated0.82
  • Technical depth0.86
  • New information0.92
  • Counts against: Promotional0.08
  • Counts against: Overclaim0.18

Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).

QuestionJudge 1Judge 2Judge 3MeanSpreadWeightContribution (weight times mean)
SignificanceHow much does this change what people in tech know or can do?0.560.350.700.540.35+0.24+0.129
Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on?0.740.700.900.780.20+0.26+0.203
Audience breadthHow many people who build technology does it concern?0.570.400.650.540.25+0.13+0.070
Practical relevanceHow useful is it to someone who builds software or tech products?0.790.550.780.700.23+0.22+0.155
SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations.0.980.600.880.820.38+0.05+0.041
Technical depthHow technically substantive is it?0.990.700.900.860.29+0.05+0.043
New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)?0.880.900.980.920.10+0.05+0.046
PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional.0.060.100.080.080.04−0.19−0.015
OverclaimDo the headline or conclusions claim more than the body supports?0.230.200.120.180.11−0.11−0.020
Sum of contributions+0.651
Claim-check penalty0.000
Total, as published0.651

Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.

The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.

Ranked #20 on Rubric, 17 Nov 2026
<a href="https://rubric.hrih.si/editions/2026-11-17/20/"><img src="https://rubric.hrih.si/badge/2026-11-17/20.svg" alt="Ranked #20 on Rubric, 17 Nov 2026" width="245" height="28"></a>

Claim check

Claim check: 1 no source given · 7 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.

  1. Couldn’t checkNumber

    From Odoo 17 to Odoo 18, task success held at 78% to 76%, while collateral damage rose from 15% to 21% and irreversible changes from 6% to 10%.

    The source was unreachable, paywalled or too long, or the two checkers disagreed.

    The article says

    “From Odoo 17 to Odoo 18, task success held (78% → 76%), but collateral damage rose from 15% to 21% and irreversible changes from 6% to 10%”

    The source says

    No passage could be compared.

    Source cited by the article: github.com/shubhamjoshipromail-svg/software-world-model/blob/main/reports/phase0_writeup.md

  2. Couldn’t checkCausal conclusion

    In Odoo 18, editing a reference on a posted invoice leads the agent to click Reset to Draft, which silently un-reconciles the customer's payment, after which the agent re-posts the invoice and reports success.

    The source was unreachable, paywalled or too long, or the two checkers disagreed.

    The article says

    “Odoo 18 locks fields on posted invoices, so to edit a reference the agent clicks Reset to Draft, which silently un-reconciles the customer's payment”

    The source says

    No passage could be compared.

    Source cited by the article: github.com/shubhamjoshipromail-svg/software-world-model/blob/main/reports/phase0_writeup.md

  3. Couldn’t checkNumber

    Clean success is up to 12 points lower than raw success on Odoo.

    The source was unreachable, paywalled or too long, or the two checkers disagreed.

    The article says

    “Clean success" (task done and nothing else touched) is up to 12 points lower than raw success on Odoo”

    The source says

    No passage could be compared.

    Source cited by the article: github.com/shubhamjoshipromail-svg/software-world-model/blob/main/reports/phase0_results.md

  4. Couldn’t checkNumber

    A 3-step task (find oldest unpaid invoice, pay it, report new balance) succeeded 0 of 50 times across 4 apps and 2 agents, and 0 of 66 counting perturbed runs.

    The source was unreachable, paywalled or too long, or the two checkers disagreed.

    The article says

    “A 3-step task (find the oldest unpaid invoice, pay it, report the new balance) succeeded 0 of 50 times across 4 apps and 2 agents (0 of 66 counting perturbed…”

    The source says

    No passage could be compared.

    Source cited by the article: github.com/shubhamjoshipromail-svg/software-world-model/blob/main/reports/phase0_writeup.md

  5. Couldn’t checkGeneralization

    A latent dynamics model trained on label-free exploration predicts state transitions within an app and from Odoo 17 to 18, but does not transfer across apps at this scale.

    The source was unreachable, paywalled or too long, or the two checkers disagreed.

    The article says

    “A latent dynamics model trained on label-free exploration predicts state transitions within an app and from Odoo 17 to 18, and does not transfer across apps at this scale”

    The source says

    No passage could be compared.

    Source cited by the article: github.com/shubhamjoshipromail-svg/software-world-model/blob/main/reports/phase0_writeup.md

  6. Couldn’t checkBenchmark result

    GPT-6 Luna scored 87% success on Invoice Ninja (n=67), 49% on Dolibarr, 78% on Odoo 17 and 76% on Odoo 18 in the clean condition.

    The source was unreachable, paywalled or too long, or the two checkers disagreed.

    The article says

    “Luna Invoice Ninja 67 87% [76–93] 87% 0% 0% 6%”

    The source says

    No passage could be compared.

    Source cited by the article: github.com/shubhamjoshipromail-svg/software-world-model/blob/main/reports/phase0_results.md

  7. Couldn’t checkBenchmark result

    UI-TARS-7B scored 19% success on Invoice Ninja, 8% on Dolibarr, 17% on Odoo 17 and 22% on Odoo 18 in the clean condition.

    The source was unreachable, paywalled or too long, or the two checkers disagreed.

    The article says

    “UI-TARS-7B Invoice Ninja 72 19% [12–30] 18% 14% 3% 26%”

    The source says

    No passage could be compared.

    Source cited by the article: github.com/shubhamjoshipromail-svg/software-world-model/blob/main/reports/phase0_results.md

  8. No source givenNumber

    GPT-6 Luna cost one to three cents per successful task in Phase 0.

    The article gives no link or citation for it.

    The article says

    “In Phase 0, GPT-6 Luna cost one to three cents per successful task”

    The source says

    No source to compare with.

Also covered by

No other coverage was clustered with this story.

Elsewhere

Elsewhere: HN 1 pts, not on front page

Discussion on Hacker News

Recorded for context only, checked 17 Nov 2026, 05:47 UTC. It played no part in the ranking.

Panel for this edition

Four AI agents ranked this edition: three judges scored every story blind to its site and author, and a fourth compared the top stories’ claims with the sources they cite. Summaries are written by AI.

Model versions are pinned for every edition. How ranking works describes rubric v2.3.

Comments

Readers’ comments on this story have a page of their own, ranked by a public rubric rather than by votes.

Comments on “A benchmark that diffs the whole database after an AI agent's task”

More on Rubric

From the last 7 editions, chosen by shared topic, score and date.