Bf16 might be freezing your weight updates
Comments on “Bf16 might be freezing your weight updates”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
A blog post by mweiss17 argues that fine-tuning Mistral-7B with bf16 weights and a plain AdamW optimizer, as in the released Quiet-STaR code, effectively freezes most weights.
Why it ranked
No. 39 of 107 with a score of 0.64. The panel rated it highest on technical depth (0.97) and substantiated (0.90), lowest on audience breadth (0.62). The claim check took 0.100 off. Placed No. 39, No. 24 by score alone: the tenth AI & machine learning story of the day (−0.080). The placement terms move stories; they are not part of the score.
- AI judgeJudge 2claude-haiku-5-5
- Detailed, well-evidenced technical investigation of bf16 rounding silently freezing weights in fine-tuning, with reproducible diagnostics and practical fixes. Strong practitioner relevance, though the audience is narrower than general tech news.
- AI judgeJudge 3gpt-5.6-luna
- Detailed experiments, code, checkpoint diffs, ablations, and diagnostics support a novel finding about bf16 AdamW silently inducing sparse fine-tuning. It is highly actionable for ML practitioners, though some broader conclusions rest on limited seeds and one recipe.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.64total under rubric v2.3
- Significance0.68
- Reader interest0.83
- Audience breadth0.62
- Practical relevance0.83
- Substantiated0.90
- Technical depth0.97
- New information0.89
- Counts against: Promotional0.03
- Counts against: Overclaim0.29
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.69 | 0.52 | 0.82 | 0.68 | 0.30 | +0.24 | +0.162 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.75 | 0.80 | 0.94 | 0.83 | 0.19 | +0.26 | +0.216 |
| Audience breadthHow many people who build technology does it concern? | 0.57 | 0.60 | 0.70 | 0.62 | 0.13 | +0.13 | +0.081 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.82 | 0.78 | 0.88 | 0.83 | 0.10 | +0.22 | +0.182 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.98 | 0.86 | 0.86 | 0.90 | 0.12 | +0.05 | +0.045 |
| Technical depthHow technically substantive is it? | 1.00 | 0.92 | 0.98 | 0.97 | 0.08 | +0.05 | +0.048 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.90 | 0.85 | 0.93 | 0.89 | 0.08 | +0.05 | +0.045 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.05 | 0.03 | 0.01 | 0.03 | 0.04 | −0.19 | −0.006 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.32 | 0.22 | 0.32 | 0.29 | 0.10 | −0.11 | −0.032 |
| Sum of contributions | +0.742 | ||||||
| Claim-check penalty | −0.100 | ||||||
| Total, as published | 0.642 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-03/39/"><img src="https://rubric.hrih.si/badge/2026-11-03/39.svg" alt="Ranked #39 on Rubric, 3 Nov 2026" width="238" height="28"></a>Claim check
Claim check: 2 matches source · 1 not in source · 3 no source given · 2 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Couldn’t checkCausal conclusion
Barron, Gnanapragasam & Tschannen show that standard cross-entropy computation in bf16 rounds the correct answer's gradient to exactly zero once the model is confident enough (margin about 5.5), and that this interacting with Adam produces the grokking "slingshot" loss spikes.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “They show that the standard way of computing cross-entropy rounds the correct answer's gradient to exactly zero once the model is confident enough”
The source says No passage could be compared.
Source cited by the article: arxiv.org/abs/2610.31642
No source givenNumber
The Quiet-STaR released checkpoint differs from Mistral-7B-v0.1 in 5.4% of its 7.24B weights, and above 5.12e-4 only 110 of about 6.2 billion weights changed.
The article gives no link or citation for it.
The article says “Above 5.12e-4, only 110 of about 6.2 billion weights changed, in any tensor, in any layer”
The source says No source to compare with.
Not in sourceNumber
The Quiet-STaR paper fine-tunes Mistral-7B at learning rate 1e-6, and its released training code uses bf16 weights, fused torch AdamW and no master copy.
The cited source does not contain it.
The article says “The paper fine-tunes Mistral-7B at learning rate 1e-6, and its released training code does exactly the thing above: bf16 weights, fused torch AdamW, no master copy”
The source says “This project is implemented by simply patching the base Mistral implementation in Huggingface `transformers` using a new `modeling_mistral.py` and a new `configuration_mistral.py` and otherwise applying standard `transformers` features (e.g. the…”
Source cited by the article: github.com/ezelikman/quiet-star
Matches sourceBenchmark result
PaFi fine-tunes only the 0.5% of weights with the smallest magnitude (plus normalization layers) using a fixed mask, and matches full fine-tuning on GLUE.
The cited source says the same thing.
The article says “fine-tunes only the 0.5% of weights with the smallest magnitude (plus the normalization layers), using a mask fixed before training, and matches full fine-tuning on GLUE”
The source says “Compared to existing sparse and adapter fine-tuning methods, our PaFi obtains the best result, achieving the same score (89.3) as Full FT with only updating 0.5% parameters”
Source cited by the article: arxiv.org/abs/2305.16742
Couldn’t checkBenchmark result
NanoAdam updates only the bottom-k weights by magnitude, beats AdamW on GLUE, nearly eliminates forgetting and tolerates larger learning rates.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “It beats AdamW on GLUE, nearly eliminates forgetting and, notably, tolerates larger learning rates”
The source says No passage could be compared.
Source cited by the article: arxiv.org/abs/2506.21374
No source givenBenchmark result
In the author's rerun, the bf16 recipe raised scores (CommonsenseQA 35.72 to 44.25), while both corrected optimizer versions broke the model.
The article gives no link or citation for it.
The article says “The bf16 run does what the paper says: both scores go up. Both corrected versions break”
The source says No source to compare with.
No source givenBenchmark result
At step 100, no correct-optimizer run beat bf16; the closest, at lr 1e-7, scored 41.99 on CommonsenseQA versus bf16's 44.25.
The article gives no link or citation for it.
The article says “At step 100, no correct-optimizer run beat bf16: the closest, at 1e-7, scored 41.99 on CommonsenseQA against bf16's 44.25”
The source says No source to compare with.
Matches sourceCausal conclusion
The RL community found that RL fine-tuning updates look sparse and later traced much of that sparsity to bf16 rounding.
The cited source says the same thing.
The article says “the RL community has noticed that RL fine-tuning updates look sparse (Mukherjee et al., NeurIPS 2025) and later traced much of that to bf16 rounding”
The source says “This analysis reconciles prior explanations of sparsity in RL fine-tuning ( Mukherjee et al., 2025 ; Shenfeld et al., 2025 ; Zhu et al., 2025 ). BF16 precision supplies the…”
Source cited by the article: arxiv.org/abs/2602.03839
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 1 pts, not on front page
Recorded for context only, checked 3 Nov 2026, 05:47 UTC. It played no part in the ranking.