Formalizing Julia in Lean (and finding 400 bugs in Base)
Comments on “Formalizing Julia in Lean (and finding 400 bugs in Base)”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Miles Cranmer's project formalizes Julia's source code in Lean 4 and attempts to prove the docstring promises, using AI agents to write models and proofs. Differential testing against real Julia and reproducers guard against false claims.
Why it ranked
No. 23 of 124 with a score of 0.67. The panel rated it highest on new information (0.91) and technical depth (0.88), lowest on audience breadth (0.55).
Every answer behind this is in the table under Score.
Score
0.67total under rubric v2.2
- Significance0.63
- Reader interest0.82
- Audience breadth0.55
- Practical relevance0.70
- Substantiated0.85
- Technical depth0.88
- New information0.91
- Counts against: Promotional0.11
- Counts against: Overclaim0.27
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| Significance | 0.57 | 0.55 | 0.78 | 0.63 | 0.23 | +0.24 | +0.152 |
| Reader interest | 0.74 | 0.80 | 0.93 | 0.82 | 0.19 | +0.26 | +0.214 |
| Audience breadth | 0.61 | 0.55 | 0.48 | 0.55 | 0.13 | +0.13 | +0.071 |
| Practical relevance | 0.67 | 0.60 | 0.82 | 0.70 | 0.22 | +0.22 | +0.153 |
| Substantiated | 0.96 | 0.70 | 0.88 | 0.85 | 0.26 | +0.05 | +0.042 |
| Technical depth | 0.93 | 0.80 | 0.91 | 0.88 | 0.13 | +0.05 | +0.044 |
| New information | 0.90 | 0.85 | 0.97 | 0.91 | 0.12 | +0.05 | +0.045 |
| Promotional | 0.07 | 0.08 | 0.18 | 0.11 | 0.11 | −0.19 | −0.021 |
| Overclaim | 0.38 | 0.30 | 0.14 | 0.27 | 0.24 | −0.11 | −0.030 |
| Sum of contributions | +0.671 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.671 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-10-27/23/"><img src="https://rubric.hrih.si/badge/2026-10-27/23.svg" alt="Ranked #23 on Rubric, 27 Oct 2026" width="245" height="28"></a>Claim check
Claim check: 3 no source given · 5 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Couldn’t checkNumber
Formalizing Julia's full source code in Lean has so far found about 400 bugs in Base.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “so far this has found about 400 (!) bugs in Base”
The source says No passage could be compared.
Source cited by the article: github.com/MilesCranmer/julia-in-lean
Couldn’t checkNumber
The author filed 75 of the bugs, manually filtered, as issues on the Julia repository.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “I have filed 75 of the ones I manually filtered”
The source says No passage could be compared.
Source cited by the article: github.com/JuliaLang/julia/issues
Couldn’t checkNumber
Reversing the range Int8(0):typemin(Int8):typemin(Int8) yields an empty range, because negating a step of -128 wraps back to -128; the original has two elements.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Negating a step of -128 wraps back around to -128, so the reversed range goes from -128 “down” to 0, which is empty”
The source says No passage could be compared.
Source cited by the article: github.com/JuliaLang/julia/issues/64124
Couldn’t checkCausal conclusion
Julia issue #64183 reports a miscompilation on master where inference folds an isa check across loop iterations and a small loop returns the wrong answer.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “a miscompilation on master, where inference folds an isa check across loop iterations and a small loop returns the wrong answer”
The source says No passage could be compared.
Source cited by the article: github.com/JuliaLang/julia/issues/64183
Couldn’t checkNumber
The Lean integer core matched an exhaustive table of all 656,896 Int8/UInt8 cases, and the Float64 model matched on 236,954 operations.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “the integer core matched an exhaustive table of all 656,896 Int8/UInt8 cases, and the Float64 model matched on 236,954 operations”
The source says No passage could be compared.
Source cited by the article: github.com/MilesCranmer/julia-in-lean
No source givenNumber
In the in-bounds proofs for 1,152 @inbounds and unsafe_* sites, a review found that 128 of the first 130 'proven' sites had just assumed the index was in bounds.
The article gives no link or citation for it.
The article says “a review against a fixed rubric found that 128 of those 130 proofs had just assumed the index was in bounds”
The source says No source to compare with.
No source givenNumber
The work has cost about $5,000 in API-equivalent usage, mostly Claude Opus 5.5 and OpenAI's Daybreak Blue.
The article gives no link or citation for it.
The article says “Everything so far has cost about $5,000 in API-equivalent usage, mostly Claude Opus 5.5 and OpenAI’s Daybreak Blue (via Codex”
The source says No source to compare with.
No source givenGeneralization
Lean formalization can identify problems that even heavy property-based testing can fail to catch.
The article gives no link or citation for it.
The article says “This can identify problems that even the most heavy-handed prop-testing can fail to catch”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 4 pts, not on front page
Recorded for context only, checked 27 Oct 2026, 05:46 UTC. It played no part in the ranking.