A benchmark that diffs the whole database after an AI agent's task
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
A solo developer's open-source benchmark runs computer-use agents in Invoice Ninja, Dolibarr and Odoo 17 and 18, then diffs the whole database to separate success from collateral damage, irreversible changes and false completion. Between Odoo 17 and 18, reported success stayed near 77% while collateral damage rose from 15% to 21%, partly because "Reset to Draft" silently un-reconciles payments. A 3-step task never succeeded in 50 runs. The author notes small samples, and the figures could not be independently verified.