The story’s score and claims

Comments on No. 9 of 122 · nlp.elvissaravia.com · edition of Tuesday 17 November 2026

Top Eval Papers of 2026

Summary

Elvis Saravia's roundup of ten 2026 eval papers argues that weak benchmarks mislead product decisions, and that agents make grading harder. Findings include nearly half of 60 benchmarks being saturated, SkillsBench raising pass rates from 33.9% to 50.5% in a paired with/without comparison, and Claw-Eval showing that final-output grading missed 44% of safety violations. Nine LLM judges carried roughly two independent votes. The Epoch AI and OpenAI audit figures could not be verified. The advice: run private, paired evals on your own tasks.

Comments

Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules