Ernie Evolve: automated research system for improving AI training and inference
Comments on “Ernie Evolve: automated research system for improving AI training and inference”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Baidu describes ERNIE Evolve, an automated research system that proposes directions, edits code, and keeps only changes that pass a hidden evaluator and a code reviewer. It reports beating public bests on NanoGPT Speedrun Track 1 (38.9 s versus 39.9 s) and NanoChat AutoResearch, and scoring 0.819 on SOL-ExecBench against a leading 0.827. On ERNIE-5.5-Flash it claims a 33.7% GPU throughput gain with determinism preserved. All figures are Baidu's own; the Kunlunxin gain largely came from enabling existing optimized paths, and the leaderboard submission is unconfirmed.
Why it ranked
No. 15 of 95 with a score of 0.66. The panel rated it highest on new information (0.92) and technical depth (0.83), lowest on audience breadth (0.62). Placed No. 15, No. 20 by score alone: the fourth AI & machine learning story of the day (−0.020). The placement terms move stories; they are not part of the score.
- AI judgeJudge 2claude-haiku-5-5
- Vendor announcement of an automated AI-research system with benchmark gains and detailed inference-optimization case studies; substantive technical detail but self-reported results and promotional framing limit substantiation.
- AI judgeJudge 3gpt-5.6-luna
- A detailed engineering account of an automated research loop, with reproducible benchmark comparisons and substantial real-world inference gains. It is highly relevant and technically deep, though internal results and broad RSI claims warrant some caution.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.66total under rubric v2.3
- Significance0.73
- Reader interest0.82
- Audience breadth0.62
- Practical relevance0.66
- Substantiated0.80
- Technical depth0.83
- New information0.92
- Counts against: Promotional0.19
- Counts against: Overclaim0.36
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.72 | 0.60 | 0.86 | 0.73 | 0.26 | +0.24 | +0.174 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.75 | 0.80 | 0.92 | 0.82 | 0.17 | +0.26 | +0.214 |
| Audience breadthHow many people who build technology does it concern? | 0.61 | 0.60 | 0.66 | 0.62 | 0.06 | +0.13 | +0.081 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.64 | 0.55 | 0.78 | 0.66 | 0.23 | +0.22 | +0.144 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.94 | 0.65 | 0.80 | 0.80 | 0.29 | +0.05 | +0.040 |
| Technical depthHow technically substantive is it? | 0.89 | 0.70 | 0.92 | 0.83 | 0.22 | +0.05 | +0.042 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.89 | 0.90 | 0.96 | 0.92 | 0.07 | +0.05 | +0.046 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.10 | 0.30 | 0.18 | 0.19 | 0.20 | −0.19 | −0.037 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.23 | 0.40 | 0.44 | 0.36 | 0.21 | −0.11 | −0.039 |
| Sum of contributions | +0.665 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.665 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-10/15/"><img src="https://rubric.hrih.si/badge/2026-11-10/15.svg" alt="Ranked #15 on Rubric, 10 Nov 2026" width="245" height="28"></a>Claim check
Claim check: 2 matches source · 5 no source given · 1 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Couldn’t checkBenchmark result
ERNIE Evolve reached a validation BPB of 0.8763 on NanoChat AutoResearch, better than the best public result of 0.8878 reported in Rekursiv's Auto²-Research report.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “The final version achieved a validation bits-per-byte (BPB) score of 0.8763”
The source says No passage could be compared.
Source cited by the article: rekursiv.ai/blog/autoautoresearch
Matches sourceBenchmark result
ERNIE Evolve reached the NanoGPT Speedrun Track 1 target loss in 38.9 seconds, versus the best public result of 39.9 s.
The cited source says the same thing.
The article says “The final version reached the target loss in 38.9 seconds, and repeated runs confirmed that it met the loss requirement”
The source says “Under 40 seconds on 8xH100 (the llm.c GPT-2 replication needed 45 minutes”
Source cited by the article: github.com/KellerJordan/modded-nanogpt/tree/master
No source givenBenchmark result
ERNIE Evolve reduced NanoGPT Speedrun Track 3 steps to the target loss to 2,590, versus the best public result of 2,690.
The article gives no link or citation for it.
The article says “Together, these changes reduced the number of steps to the target loss to 2,590”
The source says No source to compare with.
No source givenBenchmark result
ERNIE Evolve achieved an average SOL score of 0.819 on SOL-ExecBench, close to the leading score of 0.827.
The article gives no link or citation for it.
The article says “ERNIE Evolve achieved an average SOL score of 0.819, ahead of Recursive and Hyra-1.0 and close to the leading score of 0.827”
The source says No source to compare with.
Matches sourceNumber
NanoGPT Speedrun has had more than two years of community optimization with over 90 successive improvements.
The cited source says the same thing.
The article says “NanoGPT Speedrun has undergone more than two years of community optimization, with over 90 successive improvements”
The source says “92 | 0.665 minutes | Sampled softmax over the early stages, full FP8 MLP fwd+bwd, 84.6M-row hashed n-gram table, ANVIL optimizer stack”
Source cited by the article: github.com/KellerJordan/modded-nanogpt/tree/master
No source givenNumber
Across four rounds of research, ERNIE Evolve raised ERNIE-5.5-Flash generation speed on Kunlunxin P800 to 623.6% of the baseline while passing all correctness checks.
The article gives no link or citation for it.
The article says “Across four rounds of research, ERNIE Evolve raised generation speed to 623.6% of the baseline while passing all correctness checks”
The source says No source to compare with.
No source givenNumber
After roughly three days of research, ERNIE Evolve increased deterministic inference throughput by 33.7%, cut average time per output token by 26% and time to first token by 12%, with determinism preserved.
The article gives no link or citation for it.
The article says “After roughly three days of research, ERNIE Evolve increased throughput by 33.7%, reduced average time per output token by 26%, and reduced time to first token by 12%”
The source says No source to compare with.
No source givenGeneralization
ERNIE Evolve's inference changes passed production-grade validation and show that automated research can produce practical improvements in real AI development.
The article gives no link or citation for it.
The article says “The resulting changes passed production-grade validation and delivered substantial performance gains, showing that automated research can produce practical improvements in real AI development”
The source says No source to compare with.
Also covered by
Elsewhere
Elsewhere: HN 2 pts, not on front page
Recorded for context only, checked 10 Nov 2026, 05:47 UTC. It played no part in the ranking.