Mass attempted replication of old findings of LLMs with new models
Comments on “Mass attempted replication of old findings of LLMs with new models”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
A Zou group project retested 50 widely cited LLM findings on 20 models released between July 2024 and September 2026. By fitted trends, 44% of claims are expired (effect at or below a fifth of the original), 34% weakened, 16% persist and 6% strengthened. Capability failures such as fabricated citations faded fastest, while sycophancy and judge self-preference held. Expired claims lasted about 15.7 months on average. Each claim uses one minimal design, some models are substitutes, and individual models can still diverge, so findings about older models need retesting.
Why it ranked
No. 10 of 137 with a score of 0.72. The panel rated it highest on new information (0.94) and substantiated (0.86), lowest on practical relevance (0.66). Placed No. 10, No. 9 by score alone: the fourth AI & machine learning story of the day (−0.020). The placement terms move stories; they are not part of the score.
- AI judgeJudge 2claude-haiku-5-5
- Large empirical study re-testing 50 LLM findings across 20 models; highly relevant to AI builders, well-substantiated with method detail, and novel versus recent stories, though it covers a fictional-future set of models.
- AI judgeJudge 3gpt-5.6-luna
- Large-scale empirical replication across 50 LLM findings, 20 models, detailed methodology, uncertainty treatment, and reproducibility caveats make this highly relevant and technically substantive. Some conclusions remain limited by model selection, minimal designs, and future-model extrapolation.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.72total under rubric v2.3
- Significance0.72
- Reader interest0.81
- Audience breadth0.79
- Practical relevance0.66
- Substantiated0.86
- Technical depth0.83
- New information0.94
- Counts against: Promotional0.04
- Counts against: Overclaim0.28
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.75 | 0.55 | 0.85 | 0.72 | 0.30 | +0.24 | +0.172 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.69 | 0.80 | 0.95 | 0.81 | 0.26 | +0.26 | +0.211 |
| Audience breadthHow many people who build technology does it concern? | 0.81 | 0.70 | 0.86 | 0.79 | 0.16 | +0.13 | +0.103 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.60 | 0.60 | 0.78 | 0.66 | 0.18 | +0.22 | +0.146 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.97 | 0.75 | 0.86 | 0.86 | 0.22 | +0.05 | +0.043 |
| Technical depthHow technically substantive is it? | 0.94 | 0.60 | 0.94 | 0.83 | 0.34 | +0.05 | +0.041 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.93 | 0.90 | 0.98 | 0.94 | 0.08 | +0.05 | +0.047 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.05 | 0.05 | 0.01 | 0.04 | 0.04 | −0.19 | −0.007 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.37 | 0.30 | 0.18 | 0.28 | 0.19 | −0.11 | −0.031 |
| Sum of contributions | +0.725 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.725 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-04/10/"><img src="https://rubric.hrih.si/badge/2026-11-04/10.svg" alt="Ranked #10 on Rubric, 4 Nov 2026" width="238" height="28"></a>Claim check
Claim check: 2 matches source · 4 no source given · 2 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Couldn’t checkNumber
In Walters and Wilder's 2023 study, 55% of ChatGPT's (GPT-3.5) citations in generated literature reviews were fake.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “At the time, ChatGPT used GPT-3.5, and 55% of its citations were fake (C11”
The source says No passage could be compared.
Source cited by the article: nature.com/articles/s41598-023-41032-5
Couldn’t checkNumber
GPT-4 fabricated 18% of its references overall in the Walters & Wilder study, an improvement on GPT-3.5.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “GPT-4 improved on GPT-3.5 but still fabricated 18% of its references overall”
The source says No passage could be compared.
Source cited by the article: nature.com/articles/s41598-023-41032-5
No source givenNumber
Among tested LLM claims, 44% are now expired, 34% have weakened, 16% persist and 6% have strengthened.
The article gives no link or citation for it.
The article says “Almost half (44%) of our tested claims are now expired in today’s models. Another 34% have weakened significantly, 16% persist, and 6% have strengthened”
The source says No source to compare with.
No source givenNumber
Across the 50 claims, the estimated share expired was 4% at six months, 16% at one year and 39% at two years after publication.
The article gives no link or citation for it.
The article says “the estimated share that had expired was 4% six months after publication, 16% after one year, and 39% after two years”
The source says No source to compare with.
No source givenNumber
Among the 22 claims with an estimated expiration date, time from publication to expiration averaged 15.7 months (median 14.7 months).
The article gives no link or citation for it.
The article says “time from publication to expiration averaged 15.7 months (median: 14.7 months”
The source says No source to compare with.
Matches sourceBenchmark result
The authors' reproduction of the LLM-judge answer-order claim measured a 62.5% verdict-flip rate versus the paper's 46.3%.
The cited source says the same thing.
The article says “we measured a 62.5% verdict-flip rate (95% interval: 51.3–72.5%), compared with the paper’s 46.3%”
The source says “We found that GPT-4 exhibited conflict rates of 46.3% and 5.0%, respectively”
Source cited by the article: arxiv.org/abs/2305.17926
Matches sourceQuote
ICLR 2026 reported desk-rejecting papers with hallucinated references.
The cited source says the same thing.
The article says “ICLR 2026 reported desk-rejecting papers with hallucinated references”
The source says “We have been desk -rejecting, and will continue to desk -reject, any paper that includes such issues”
Source cited by the article: blog.iclr.cc/2025/11/19
No source givenCausal conclusion
Fabricated-citation policies are a poor way to discourage AI-generated writing, because recent models fabricate references much less often than older models.
The article gives no link or citation for it.
The article says “our results suggest that these phenomena might not occur at as high of a frequency as with older models, making such policies a poor direction”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 27 pts, not on front page
Recorded for context only, checked 4 Nov 2026, 05:47 UTC. It played no part in the ranking.