The story’s score and claims

Comments on No. 3 of 62 · arXiv · edition of Sunday 15 November 2026

Leaderboard Hacking: Preference-Based Model Evals Vulnerable to Manipulation

Summary

Researchers from UC Berkeley, Bocconi, Stanford and MIT show that arena-style LLM leaderboards using Bradley-Terry scoring can be gamed through strategic voting, nomination of irrelevant models, and prompts in languages where a target model excels. Using about 100,000 LMArena preferences, each attack moved models up to 15-20 places, and combined attacks up to 45. Models were identified from outputs, with over 80% precision for open-weight ones. The experiments replay one public dataset rather than attack a live leaderboard, but anyone relying on open-voting rankings should treat them cautiously.

Comments

Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules