Prefer synthetic traffic over LLM as a judge
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
The author argues that using LLMs as judges for search relevance labels merely bolts AI onto old evaluation metrics like NDCG, which ignore how the user interface shapes results.