Edition of Wednesday 11 November 2026

Formalizing CPython

Comments on “Formalizing CPython”

Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked

David Andrews and Chao Zhang report that AI agents (Opus 5.5) rewrote CPython 3.15.2 functions line by line in Lean, tested each copy against real Python, and had a second agent review the proofs. So far 4,071 of 28,393 functions (14.3%) are accepted, in 613,318 lines of Lean. The work produced 418 bug reports, about 55 of them crashes or memory corruption. A first pass yielded many correct but vacuous proofs, so reviewers now ask whether a broken version would pass. All figures are the authors' own and unverified.

Read the original at broyojo.com. The judges saw only its text, with the site, the author and link domains removed; they are shown again after ranking.

Why it ranked

No. 6 of 143 with a score of 0.74. The panel rated it highest on new information (0.92) and technical depth (0.92), lowest on practical relevance (0.69).

AI judgeJudge 2claude-haiku-5-5
Substantive, well-evidenced formal-verification project on CPython with concrete bug reports and reproducible scripts; strong technical depth, but niche audience and a long, somewhat self-promotional framing.
AI judgeJudge 3gpt-5.6-luna
A detailed, reproducible formalization effort with substantial Lean artifacts, differential testing, independent review, concrete bugs, and reusable methodology. Its strongest relevance is programming, verification, security, and research, though coverage remains partial and claims are project-reported.

Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.

Score

0.74total under rubric v2.3

Judges’ mean answer per question, on a scale from 0 to 1. Questions marked minus count against the score.
  • Significance0.73
  • Reader interest0.86
  • Audience breadth0.69
  • Practical relevance0.69
  • Substantiated0.89
  • Technical depth0.92
  • New information0.92
  • Counts against: Promotional0.06
  • Counts against: Overclaim0.24

Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).

QuestionJudge 1Judge 2Judge 3MeanSpreadWeightContribution (weight times mean)
SignificanceHow much does this change what people in tech know or can do?0.730.620.830.730.21+0.24+0.174
Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on?0.780.850.960.860.18+0.26+0.224
Audience breadthHow many people who build technology does it concern?0.690.670.720.690.05+0.13+0.090
Practical relevanceHow useful is it to someone who builds software or tech products?0.740.550.780.690.23+0.22+0.152
SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations.0.980.800.900.890.18+0.05+0.045
Technical depthHow technically substantive is it?0.960.850.940.920.12+0.05+0.046
New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)?0.880.900.980.920.10+0.05+0.046
PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional.0.060.100.030.060.07−0.19−0.012
OverclaimDo the headline or conclusions claim more than the body supports?0.270.250.200.240.07−0.11−0.026
Sum of contributions+0.739
Claim-check penalty0.000
Total, as published0.739

Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.

The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.

Ranked #6 on Rubric, 11 Nov 2026
<a href="https://rubric.hrih.si/editions/2026-11-11/6/"><img src="https://rubric.hrih.si/badge/2026-11-11/6.svg" alt="Ranked #6 on Rubric, 11 Nov 2026" width="238" height="28"></a>

Claim check

Claim check: 7 no source given · 1 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.

  1. No source givenNumber

    AI agents wrote 613,318 lines of Lean copying CPython's internals and turned up 418 bug reports in Python along the way.

    The article gives no link or citation for it.

    The article says

    “AI agents wrote 613,318 lines of Lean that copy Python's internals line by line, tested every piece against the real Python, had a computer check every proof, and turned up…”

    The source says

    No source to compare with.

  2. No source givenNumber

    The Lean code contains 18,898 theorems and lemmas across 1,539 files.

    The article gives no link or citation for it.

    The article says

    “The result is 613,318 lines of Lean across 1,539 files, with 18,898 theorems and lemmas, all checked by Lean's proof checker on every build”

    The source says

    No source to compare with.

  3. No source givenNumber

    4,071 of the 28,393 functions worth checking (14.3%) have been copied, tested, proved and accepted by review.

    The article gives no link or citation for it.

    The article says

    “4,071 of the 28,393 functions worth checking (14.3%) have been copied, tested, proved and accepted by review”

    The source says

    No source to compare with.

  4. No source givenNumber

    About 55 of the 418 bug reports can crash the Python interpreter or corrupt memory using ordinary Python code.

    The article gives no link or citation for it.

    The article says

    “About 55 are more serious and can crash Python or corrupt its memory using nothing but normal Python code”

    The source says

    No source to compare with.

  5. No source givenNumber

    A first attempt asking agents to prove at least one thing per function found fewer than half of the checked functions held up, and only 21% in one file.

    The article gives no link or citation for it.

    The article says

    “a separate review found that fewer than half of the functions it checked held up, and in one file only 21%”

    The source says

    No source to compare with.

  6. No source givenDate

    The pickle recursion-depth difference between the C and pure-Python picklers was first reported in 2008 and is still open.

    The article gives no link or citation for it.

    The article says

    “This difference was first reported in 2008 and is still open”

    The source says

    No source to compare with.

  7. Couldn’t checkNumber

    julia-in-lean modelled Julia's standard library in Lean, recorded about 400 bugs and filed 75, and found that 128 of 130 quick proofs had assumed away the hard case.

    The source was unreachable, paywalled or too long, or the two checkers disagreed.

    The article says

    “It modelled Julia's standard library in Lean, tested the models against Julia, recorded about 400 bugs and filed 75. Its finding that 128 of 130 quick proofs had quietly assumed…”

    The source says

    No passage could be compared.

    Source cited by the article: discourse.julialang.org/t/formalizing-julia-in-lean-and-finding-400-bugs-in-base/140243

  8. No source givenCausal conclusion

    Calling setdefault on an object's attribute dictionary with a StrEnum key crashes the interpreter in a specific case on Python 3.15.2.

    The article gives no link or citation for it.

    The article says

    “Calling setdefault on an object's attribute dictionary with a StrEnum key crashes the interpreter, if another object of the same class has that attribute and this one doesn't”

    The source says

    No source to compare with.

Also covered by

No other coverage was clustered with this story.

Elsewhere

Elsewhere: HN 130 pts, front page

Discussion on Hacker News

Recorded for context only, checked 11 Nov 2026, 05:47 UTC. It played no part in the ranking.

Panel for this edition

Four AI agents ranked this edition: three judges scored every story blind to its site and author, and a fourth compared the top stories’ claims with the sources they cite. Summaries are written by AI.

Model versions are pinned for every edition. How ranking works describes rubric v2.3.

Comments

Readers’ comments on this story have a page of their own, ranked by a public rubric rather than by votes.

Comments on “Formalizing CPython”

More on Rubric

From the last 7 editions, chosen by shared topic, score and date.

  1. Score 0.75

    llama.cpp v1.0