Architecture Decisions for Marin 535B-A23B
Comments on “Architecture Decisions for Marin 535B-A23B”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Open Athena's Larry Dial explains the architecture choices behind Marin 535B-A23B, a mixture-of-experts model pretrained on 18T tokens on 768 GB200s. Scaling-ladder runs up to 9e21 FLOPs forecast a final loss near 2.04. Expert parallelism with token dropping was tested for stability: extending context from 4k to 65k raised dropping from 7% to 37%, with eval loss within 0.01 of a dropless baseline. Debugging led to logit z loss, plain Adam on the embedding and a learnable norm on the latent. Most figures come from the team's own ablations.
Why it ranked
No. 13 of 143 with a score of 0.71. The panel rated it highest on technical depth (0.97) and substantiated (0.93), lowest on audience breadth (0.60). Placed No. 13, where the score alone would also have put it: the fourth AI & machine learning story of the day (−0.020). The placement terms move stories; they are not part of the score.
- AI judgeJudge 2claude-haiku-5-5
- Detailed, well-evidenced engineering write-up of a large MoE pretraining run with ablations and reproducible methods; high technical depth, strong relevance to ML builders, limited breadth beyond AI infrastructure.
- AI judgeJudge 3gpt-5.6-luna
- A highly detailed first-person engineering account of training a 535B MoE, with architecture specifics, ablations, scaling-law evidence, hardware constraints, stability investigations, and reproducible experimental results. It is broadly valuable to frontier-model practitioners.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.71total under rubric v2.3
- Significance0.69
- Reader interest0.81
- Audience breadth0.60
- Practical relevance0.67
- Substantiated0.93
- Technical depth0.97
- New information0.91
- Counts against: Promotional0.05
- Counts against: Overclaim0.16
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.64 | 0.55 | 0.88 | 0.69 | 0.33 | +0.24 | +0.166 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.67 | 0.80 | 0.95 | 0.81 | 0.28 | +0.26 | +0.210 |
| Audience breadthHow many people who build technology does it concern? | 0.49 | 0.60 | 0.72 | 0.60 | 0.23 | +0.13 | +0.078 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.52 | 0.70 | 0.78 | 0.67 | 0.26 | +0.22 | +0.146 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.99 | 0.85 | 0.95 | 0.93 | 0.14 | +0.05 | +0.046 |
| Technical depthHow technically substantive is it? | 1.00 | 0.92 | 0.99 | 0.97 | 0.08 | +0.05 | +0.048 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.89 | 0.85 | 0.98 | 0.91 | 0.13 | +0.05 | +0.045 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.05 | 0.08 | 0.03 | 0.05 | 0.05 | −0.19 | −0.010 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.19 | 0.20 | 0.08 | 0.16 | 0.12 | −0.11 | −0.017 |
| Sum of contributions | +0.713 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.713 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-11/13/"><img src="https://rubric.hrih.si/badge/2026-11-11/13.svg" alt="Ranked #13 on Rubric, 11 Nov 2026" width="245" height="28"></a>Claim check
Claim check: 1 matches source · 2 no source given · 5 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
No source givenDate
Marin 535B-A23B finished pretraining on 18T tokens on November 5, 2026.
The article gives no link or citation for it.
The article says “Marin 535B-A23B finished pretraining on 18T tokens on November 5, 2026”
The source says No source to compare with.
No source givenNumber
Marin's two prior MoE runs (129B-A16B on 1T tokens and 67B-A2B Snowball on 10T tokens) both landed within 1 percent of their loss target with zero loss spikes.
The article gives no link or citation for it.
The article says “Both pre-training runs landed within 1 percent of their loss target with zero loss spikes”
The source says No source to compare with.
Matches sourceNumber
Moving to expert parallelism, adding 50% more routed weights, and moving to normalized latent MoE together gave roughly 1.3× compute efficiency, measured from a scaling ladder from 4e19 to 8e21 FLOPs.
The cited source says the same thing.
The article says “Moving to EP, adding 50% more routed weights, and moving to normalized latent MoE together gave roughly 1.3× compute efficiency”
The source says “Under an equal-throughput assumption, the loss differences corresponded to an estimated 18–35% compute-efficiency gain over the FSDP baselines”
Source cited by the article: openathena.ai/blog/expert-parallelism
Couldn’t checkNumber
Extending context from 4k to 65k raised EP token dropping from 7% to 37%, yet EP's eval loss landed within 0.01 of switching to dropless FSDP.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Token dropping jumped from 7% to 37%, yet EP's eval loss landed within 0.01 of switching to dropless FSDP”
The source says No passage could be compared.
Source cited by the article: github.com/marin-community/marin/issues/8062
Couldn’t checkNumber
Adding a learnable RMSNorm to the latent gave 1.25× compute efficiency compared to hyperball with no intervention.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Compared to hyperball with no intervention, this gave 1.25× compute efficiency”
The source says No passage could be compared.
Source cited by the article: github.com/marin-community/marin/issues/6822
Couldn’t checkBenchmark result
The hero run finished at a Paloma loss of 1.982 against the ladder's forecast of 2.039.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “finishing at 1.982 against a forecast of 2.039”
The source says No passage could be compared.
Source cited by the article: github.com/marin-community/marin/issues/8435
Couldn’t checkCausal conclusion
Applying the quantile-balancing bias to raw logits rather than after the sigmoid gave substantially faster balancing; the Kimi K3-style histogram ended at 3.4% token dropping and averaging at 4.9%, versus near 2% on raw logits.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “we see substantially faster balancing from logits in our experiments”
The source says No passage could be compared.
Source cited by the article: github.com/marin-community/marin/issues/8033
Couldn’t checkCausal conclusion
Without logit z loss, loss blows up in the overtrained regime because the RMSNorm gain on the activation entering the lm_head keeps growing, causing large logit growth.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Running without logit z loss on the lm_head caused loss blow up in the overtrained regime, so we kept it in the recipe”
The source says No passage could be compared.
Source cited by the article: github.com/marin-community/marin/issues/8227
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 3 pts, not on front page
Recorded for context only, checked 11 Nov 2026, 05:47 UTC. It played no part in the ranking.