A benchmark that diffs the whole database after an AI agent's task
Comments on “A benchmark that diffs the whole database after an AI agent's task”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
A solo developer's open-source benchmark runs computer-use agents in Invoice Ninja, Dolibarr and Odoo 17 and 18, then diffs the whole database to separate success from collateral damage, irreversible changes and false completion. Between Odoo 17 and 18, reported success stayed near 77% while collateral damage rose from 15% to 21%, partly because "Reset to Draft" silently un-reconciles payments. A 3-step task never succeeded in 50 runs. The author notes small samples, and the figures could not be independently verified.
Why it ranked
No. 20 of 122 with a score of 0.65. The panel rated it highest on new information (0.92) and technical depth (0.86), lowest on significance (0.54). Placed No. 20, No. 31 by score alone: the fourth AI & machine learning story of the day (−0.020). The placement terms move stories; they are not part of the score.
- AI judgeJudge 2claude-haiku-5-5
- Original benchmark research on computer-use agents with state-diff grading; substantive and novel, but small-sample, narrow audience, and a self-described Phase 0 project with a repo to try.
- AI judgeJudge 3gpt-5.6-luna
- A reproducible benchmark with code, deterministic tasks, database-state grading, detailed limitations, and concrete results reveals failure modes hidden by success rates. It is highly actionable for agent evaluation, though samples are small and the project is early-stage.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.65total under rubric v2.3
- Significance0.54
- Reader interest0.78
- Audience breadth0.54
- Practical relevance0.70
- Substantiated0.82
- Technical depth0.86
- New information0.92
- Counts against: Promotional0.08
- Counts against: Overclaim0.18
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.56 | 0.35 | 0.70 | 0.54 | 0.35 | +0.24 | +0.129 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.74 | 0.70 | 0.90 | 0.78 | 0.20 | +0.26 | +0.203 |
| Audience breadthHow many people who build technology does it concern? | 0.57 | 0.40 | 0.65 | 0.54 | 0.25 | +0.13 | +0.070 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.79 | 0.55 | 0.78 | 0.70 | 0.23 | +0.22 | +0.155 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.98 | 0.60 | 0.88 | 0.82 | 0.38 | +0.05 | +0.041 |
| Technical depthHow technically substantive is it? | 0.99 | 0.70 | 0.90 | 0.86 | 0.29 | +0.05 | +0.043 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.88 | 0.90 | 0.98 | 0.92 | 0.10 | +0.05 | +0.046 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.06 | 0.10 | 0.08 | 0.08 | 0.04 | −0.19 | −0.015 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.23 | 0.20 | 0.12 | 0.18 | 0.11 | −0.11 | −0.020 |
| Sum of contributions | +0.651 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.651 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-17/20/"><img src="https://rubric.hrih.si/badge/2026-11-17/20.svg" alt="Ranked #20 on Rubric, 17 Nov 2026" width="245" height="28"></a>Claim check
Claim check: 1 no source given · 7 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Couldn’t checkNumber
From Odoo 17 to Odoo 18, task success held at 78% to 76%, while collateral damage rose from 15% to 21% and irreversible changes from 6% to 10%.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “From Odoo 17 to Odoo 18, task success held (78% → 76%), but collateral damage rose from 15% to 21% and irreversible changes from 6% to 10%”
The source says No passage could be compared.
Source cited by the article: github.com/shubhamjoshipromail-svg/software-world-model/blob/main/reports/phase0_writeup.md
Couldn’t checkCausal conclusion
In Odoo 18, editing a reference on a posted invoice leads the agent to click Reset to Draft, which silently un-reconciles the customer's payment, after which the agent re-posts the invoice and reports success.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Odoo 18 locks fields on posted invoices, so to edit a reference the agent clicks Reset to Draft, which silently un-reconciles the customer's payment”
The source says No passage could be compared.
Source cited by the article: github.com/shubhamjoshipromail-svg/software-world-model/blob/main/reports/phase0_writeup.md
Couldn’t checkNumber
Clean success is up to 12 points lower than raw success on Odoo.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Clean success" (task done and nothing else touched) is up to 12 points lower than raw success on Odoo”
The source says No passage could be compared.
Source cited by the article: github.com/shubhamjoshipromail-svg/software-world-model/blob/main/reports/phase0_results.md
Couldn’t checkNumber
A 3-step task (find oldest unpaid invoice, pay it, report new balance) succeeded 0 of 50 times across 4 apps and 2 agents, and 0 of 66 counting perturbed runs.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “A 3-step task (find the oldest unpaid invoice, pay it, report the new balance) succeeded 0 of 50 times across 4 apps and 2 agents (0 of 66 counting perturbed…”
The source says No passage could be compared.
Source cited by the article: github.com/shubhamjoshipromail-svg/software-world-model/blob/main/reports/phase0_writeup.md
Couldn’t checkGeneralization
A latent dynamics model trained on label-free exploration predicts state transitions within an app and from Odoo 17 to 18, but does not transfer across apps at this scale.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “A latent dynamics model trained on label-free exploration predicts state transitions within an app and from Odoo 17 to 18, and does not transfer across apps at this scale”
The source says No passage could be compared.
Source cited by the article: github.com/shubhamjoshipromail-svg/software-world-model/blob/main/reports/phase0_writeup.md
Couldn’t checkBenchmark result
GPT-6 Luna scored 87% success on Invoice Ninja (n=67), 49% on Dolibarr, 78% on Odoo 17 and 76% on Odoo 18 in the clean condition.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Luna Invoice Ninja 67 87% [76–93] 87% 0% 0% 6%”
The source says No passage could be compared.
Source cited by the article: github.com/shubhamjoshipromail-svg/software-world-model/blob/main/reports/phase0_results.md
Couldn’t checkBenchmark result
UI-TARS-7B scored 19% success on Invoice Ninja, 8% on Dolibarr, 17% on Odoo 17 and 22% on Odoo 18 in the clean condition.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “UI-TARS-7B Invoice Ninja 72 19% [12–30] 18% 14% 3% 26%”
The source says No passage could be compared.
Source cited by the article: github.com/shubhamjoshipromail-svg/software-world-model/blob/main/reports/phase0_results.md
No source givenNumber
GPT-6 Luna cost one to three cents per successful task in Phase 0.
The article gives no link or citation for it.
The article says “In Phase 0, GPT-6 Luna cost one to three cents per successful task”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 1 pts, not on front page
Recorded for context only, checked 17 Nov 2026, 05:47 UTC. It played no part in the ranking.