How to Use Decision Models for Evals
Comments on “How to Use Decision Models for Evals”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
This Hrih News piece argues that the default 0.50 decision threshold used by new decision models for automated evals often leaves performance on the table, and that tuning thresholds, weighing cost against accuracy, and calibrating scores can make these cheap classifiers far more useful.
Why it ranked
No. 24 of 93 with a score of 0.63. The panel rated it highest on technical depth (0.84) and practical relevance (0.83), lowest on significance (0.51).
Every answer behind this is in the table under Score.
Score
0.63total under rubric v2.2
- Significance0.51
- Reader interest0.70
- Audience breadth0.53
- Practical relevance0.83
- Substantiated0.80
- Technical depth0.84
- New information0.81
- Counts against: Promotional0.13
- Counts against: Overclaim0.18
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| Significance | 0.53 | 0.30 | 0.70 | 0.51 | 0.40 | +0.24 | +0.122 |
| Reader interest | 0.64 | 0.55 | 0.90 | 0.70 | 0.35 | +0.26 | +0.181 |
| Audience breadth | 0.56 | 0.40 | 0.62 | 0.53 | 0.22 | +0.13 | +0.068 |
| Practical relevance | 0.92 | 0.70 | 0.88 | 0.83 | 0.22 | +0.22 | +0.183 |
| Substantiated | 0.94 | 0.60 | 0.86 | 0.80 | 0.34 | +0.05 | +0.040 |
| Technical depth | 0.99 | 0.70 | 0.82 | 0.84 | 0.29 | +0.05 | +0.042 |
| New information | 0.94 | 0.60 | 0.88 | 0.81 | 0.34 | +0.05 | +0.040 |
| Promotional | 0.07 | 0.20 | 0.12 | 0.13 | 0.13 | −0.19 | −0.025 |
| Overclaim | 0.17 | 0.20 | 0.16 | 0.18 | 0.04 | −0.11 | −0.019 |
| Sum of contributions | +0.633 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.633 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-10-25/24/"><img src="https://rubric.hrih.si/badge/2026-10-25/24.svg" alt="Ranked #24 on Rubric, 25 Oct 2026" width="245" height="28"></a>Claim check
Claim check: 2 matches source · 5 no source given · 1 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
No source givenNumber
Perplexity Decider was the cheapest of the three candidate models at 0.82 cents per 100 eval runs, with an F1 of 0.79.
The article gives no link or citation for it.
The article says “Perplexity was the cheapest of these candidates at 0.82 cents per 100 eval runs, with an F1 of 0.79”
The source says No source to compare with.
No source givenNumber
OpenAI Decisions reached an F1 of 0.87 at 3.72 cents per 100 eval runs, about 4.5 times the cost of Perplexity.
The article gives no link or citation for it.
The article says “OpenAI Decisions reached an F1 of 0.87 at 3.72 cents per 100 eval runs, about 4.5 times the cost”
The source says No source to compare with.
No source givenBenchmark result
Perplexity, Liquid d1-ultra, and OpenAI Decisions meet the 80% minimum precision and recall requirements on this task.
The article gives no link or citation for it.
The article says “Perplexity, Liquid d1-ultra, and OpenAI Decisions meet our requirements”
The source says No source to compare with.
Matches sourceNumber
Liquid's prices as of October 17, 2026 are $0.02 per million input tokens for d1 and $0.042 for d1-ultra.
The cited source says the same thing.
The article says “$0.02 per million input tokens for d1 and $0.042 for d1-ultra”
The source says “d1 Input $0.02 / MTok”
Source cited by the article: liquid.ai/pricing
Couldn’t checkDate
Jev 1.13 was announced on September 15, 2026.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “announced September 15, 2026”
The source says No passage could be compared.
Source cited by the article: typesafe.ai/blog/introducing-system-one-models-and-jev
No source givenNumber
One group of Clef Flash traces had an average predicted failure probability of about 8%, but 60% contained failures.
The article gives no link or citation for it.
The article says “one group of Clef Flash traces had an average predicted failure probability of about 8%, but 60% contained failures”
The source says No source to compare with.
Matches sourceGeneralization
Randy Olson found that decision models could match LLMs on classification and routing at lower cost, while LLMs pulled ahead on tasks requiring knowledge and reasoning.
The cited source says the same thing.
The article says “Randy Olson found that decision models could match LLMs on classification and routing at lower cost, while LLMs pulled ahead on tasks requiring knowledge and reasoning”
The source says “On safety classification and tool-call routing, the 2 groups are about even”
Source cited by the article: randalolson.com/2026/10/14/when-to-use-a-decision-model-instead-of-an-llm
No source givenNumber
OpenAI's probabilities got slightly worse with the Platt scaling version used, with Brier score rising from 0.159 to 0.164.
The article gives no link or citation for it.
The article says “OpenAI’s probabilities got slightly worse with the version of Platt scaling we used”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 1 pts, not on front page
Recorded for context only, checked 25 Oct 2026, 05:46 UTC. It played no part in the ranking.