Architecture Decisions for Marin 535B-A23B
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Open Athena's Larry Dial explains the architecture choices behind Marin 535B-A23B, a mixture-of-experts model pretrained on 18T tokens on 768 GB200s. Scaling-ladder runs up to 9e21 FLOPs forecast a final loss near 2.04. Expert parallelism with token dropping was tested for stability: extending context from 4k to 65k raised dropping from 7% to 37%, with eval loss within 0.01 of a dropless baseline. Debugging led to logit z loss, plain Adam on the embedding and a learnable norm on the latent. Most figures come from the team's own ablations.