The story’s score and claims

Comments on No. 12 of 62 · anthropic.com · edition of Sunday 15 November 2026

Building Strong Safety Evaluations

Summary

Anthropic published guidance on building AI safety evals, borrowing preregistration and sample-size practices from psychology's replication reforms. Key points: score appropriate responses rather than refusal rates, pair harmful with benign prompts, use multi-turn simulations, validate LLM graders against human labels, and require at least 100 multi-turn or 1,000 single-turn items. Anthropic reports single-turn harmless rates of 94–98% across recent models, so those tests mostly serve as regression checks. Several figures come from Anthropic's own evals and have no cited source.

Comments

Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules