The story’s score and claims

Comments on No. 18 of 140 · contextflo.com · edition of Friday 20 November 2026

Company docs didn't help our AI agent. Four shortnotes doubled accuracy

Summary

Contextflo ran a Claude Haiku 5.5 agent on DABstep, the Adyen and Hugging Face payments benchmark, and found that company docs did not raise accuracy while four short rule notes did. On 42 held-out questions, tries right went from 40% (tables only) to 36% (docs) to 76% (docs plus notes), at roughly a quarter of the cost. One broad note broke a previously correct answer type. The results are self-reported, use one model and one dataset, and mix official and rebuilt scoring.

Comments

Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules