Making Flawed Benchmarks Useful

Blatent researchers examine why open benchmark scoring can misjudge correct model answers, citing cases like PHYBench's algebraically equivalent expressions and QASPER's partial-credit penalties. They describe how exact-match and token-F1 metrics miss semantic equivalence, and propose checking exact-source extraction deterministically before using an LLM judge to assess correctness.