Company docs didn't help our AI agent. Four shortnotes doubled accuracy
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Contextflo ran a Claude Haiku 5.5 agent on DABstep, the Adyen and Hugging Face payments benchmark, and found that company docs did not raise accuracy while four short rule notes did. On 42 held-out questions, tries right went from 40% (tables only) to 36% (docs) to 76% (docs plus notes), at roughly a quarter of the cost. One broad note broke a previously correct answer type. The results are self-reported, use one model and one dataset, and mix official and rebuilt scoring.