Learnings from Auditing Public Evaluations

Researchers from Prime Intellect audited three popular LLM benchmarks (AutomationBench, OfficeQA Pro v2, and HLE-Diamond) and found grading bugs, ambiguous tasks, and web-based answer leakage that distort reported scores. They replaced prose tool schemas with JSON Schema, swapped brittle regex grading for a judge model, and removed web access.