How to Scale Your Model – Google DeepMind
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Google DeepMind's JAX team has published "How to Scale Your Model," a free online book that teaches roofline analysis for training and serving Transformers on TPUs, with a new section on GPUs. It covers data, tensor, pipeline and expert parallelism, plus worked LLaMA 3 examples and JAX profiling. Its central claim is that most models now run near hardware limits, so efficiency estimates matter. Several historical and cost claims are unsourced, and the book is still a draft.