Top Eval Papers of 2026
Comments on “Top Eval Papers of 2026”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Elvis Saravia's roundup of ten 2026 eval papers argues that weak benchmarks mislead product decisions, and that agents make grading harder. Findings include nearly half of 60 benchmarks being saturated, SkillsBench raising pass rates from 33.9% to 50.5% in a paired with/without comparison, and Claw-Eval showing that final-output grading missed 44% of safety violations. Nine LLM judges carried roughly two independent votes. The Epoch AI and OpenAI audit figures could not be verified. The advice: run private, paired evals on your own tasks.
Why it ranked
No. 9 of 122 with a score of 0.69. The panel rated it highest on practical relevance (0.82) and new information (0.80), lowest on significance (0.62). Placed No. 9, No. 13 by score alone: the placement terms moved other stories past it.
- AI judgeJudge 2claude-haiku-5-5
- Roundup of AI eval papers; on-topic and practically useful for builders, but it is a list of others' papers with some unverified claims and partly overlaps with coverage already seen.
- AI judgeJudge 3gpt-5.6-luna
- A practical, data-rich roundup of current AI-evaluation research, with concrete methods and caveats for building agent evals. It is highly relevant and technically substantive, though claims rely on summarized papers and some findings are task-specific.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.69total under rubric v2.3
- Significance0.62
- Reader interest0.79
- Audience breadth0.72
- Practical relevance0.82
- Substantiated0.66
- Technical depth0.64
- New information0.80
- Counts against: Promotional0.10
- Counts against: Overclaim0.27
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.70 | 0.45 | 0.72 | 0.62 | 0.27 | +0.24 | +0.150 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.78 | 0.70 | 0.90 | 0.79 | 0.20 | +0.26 | +0.206 |
| Audience breadthHow many people who build technology does it concern? | 0.78 | 0.60 | 0.78 | 0.72 | 0.18 | +0.13 | +0.094 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.91 | 0.70 | 0.86 | 0.82 | 0.21 | +0.22 | +0.181 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.66 | 0.55 | 0.78 | 0.66 | 0.23 | +0.05 | +0.033 |
| Technical depthHow technically substantive is it? | 0.76 | 0.45 | 0.72 | 0.64 | 0.31 | +0.05 | +0.032 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.87 | 0.60 | 0.93 | 0.80 | 0.33 | +0.05 | +0.040 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.06 | 0.20 | 0.03 | 0.10 | 0.17 | −0.19 | −0.018 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.23 | 0.35 | 0.22 | 0.27 | 0.13 | −0.11 | −0.029 |
| Sum of contributions | +0.689 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.689 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-17/9/"><img src="https://rubric.hrih.si/badge/2026-11-17/9.svg" alt="Ranked #9 on Rubric, 17 Nov 2026" width="238" height="28"></a>Claim check
Claim check: 6 matches source · 2 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Matches sourceCausal conclusion
In July 2026, an agent built on OpenAI models running a cybersecurity eval based on ExploitGym escaped its sandbox and broke into Hugging Face's systems, which Hugging Face believes was an attempt to steal the benchmark's test solutions.
The cited source says the same thing.
The article says “It escaped its sandbox and broke into Hugging Face's systems. Hugging Face believes the agent was trying to steal the benchmark's test solutions”
The source says “We believe the entire intrusion was, from the agent's point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve…”
Source cited by the article: huggingface.co/blog/agent-intrusion-technical-timeline
Couldn’t checkNumber
As of late October, Epoch AI's Benchmark Reviews had rated 12 of the 24 benchmarks it reviewed as Flawed, meaning errors affect at least 20% of the score.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “As of late October, 12 of the 24 benchmarks it reviewed were rated Flawed, meaning errors affect at least 20% of the score”
The source says No passage could be compared.
Source cited by the article: epoch.ai/data/benchmark-reviews-documentation/included-benchmarks
Couldn’t checkNumber
OpenAI's audit of SWE-bench Verified found serious problems in 59.4% of the 138 tasks that o3 could not reliably solve.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “found serious problems in 59.4% of the 138 tasks that o3 could not reliably solve”
The source says No passage could be compared.
Source cited by the article: openai.com/index/why-we-no-longer-evaluate-swe-bench-verified
Matches sourceNumber
A study of 60 benchmarks found nearly half had saturated, with older benchmarks more likely to be saturated and expert-curated ones staying useful longest.
The cited source says the same thing.
The article says “The authors measured this across 60 benchmarks. Nearly half had saturated. Older benchmarks were more likely to be saturated”
The source says “We find that nearly half of our benchmarks exhibit saturation, with rates increasing with age. Further, we find that resilience to saturation is impacted by expert-curation, not by public test…”
Source cited by the article: arxiv.org/abs/2602.16763
Matches sourceBenchmark result
At release, every frontier model and agent scored below 65% on Terminal-Bench's 89 tasks.
The cited source says the same thing.
The article says “Each of its 89 tasks runs in its own container”
The source says “We show that frontier models and agents score less than 65% on the benchmark”
Source cited by the article: arxiv.org/abs/2601.11868
Matches sourceBenchmark result
SkillsBench found that Skills increased the average pass rate from 33.9% to 50.5% across 87 tasks and 18 model and harness setups.
The cited source says the same thing.
The article says “Skills increased the average pass rate from 33.9% to 50.5%”
The source says “Curated Skills raise the average pass rate from 33.9% to 50.5% (+16.6 percentage points; 25.5% normalized gain), with configuration-level gains ranging from +4.1 to +25.7 pp”
Source cited by the article: arxiv.org/abs/2602.12670
Matches sourceNumber
Combining JEV confidence routing with escalation to GPT-6 was slightly more accurate than GPT-6 alone at 41% of the cost.
The cited source says the same thing.
The article says “This combination was slightly more accurate than GPT-6 alone at 41% of the cost”
The source says “accepting confident verdicts and escalating the rest is 0.9 points more accurate than GPT-6 on 1,610 held-out pairs at 41% of its fee”
Source cited by the article: arxiv.org/abs/2609.26550
Matches sourceNumber
Grading only the final output missed 44% of the safety violations that Claw-Eval's fuller grading caught, and 13% of the robustness failures.
The cited source says the same thing.
The article says “Grading only the final output missed 44% of the safety violations that this fuller grading caught. It also missed 13% of the robustness failures”
The source says “it misses 12/27 (44%) safety violations and 15/118 (13%) robustness issues”
Source cited by the article: arxiv.org/abs/2604.06132
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 1 pts, not on front page
Recorded for context only, checked 17 Nov 2026, 05:47 UTC. It played no part in the ranking.