The story’s score and claims

Comments on No. 119 of 122 · Doug Turnbull's Blog · edition of Tuesday 17 November 2026

Prefer synthetic traffic over LLM as a judge

Summary

The author argues that using LLMs as judges for search relevance labels merely bolts AI onto old evaluation metrics like NDCG, which ignore how the user interface shapes results.

Comments

Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules