Can we trust LLMs to judge whether a research idea is novel?
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Ai2's new study, "Old Ideas, Novel Problems," tests whether LLM "novelty judges" can reliably assess whether research ideas are new. Using ICLR 2026 peer reviews to label human-authored ideas, the team found that small prompt changes can shift judge accuracy by more than 50 points on AI-generated ideas, that extra…