Search golden sets are anything but
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
A critique of standard search evaluation argues that golden sets built from BEIR-style judgment lists can mislead teams about real relevance. The author walks through NDCG using ESCI and WANDS examples, showing how a poor ideal ranking can make weak results look acceptable.