Looped Transformers Explained
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Looped transformers reuse one shared block of layers several times per token, so a model executes more layers than it stores. NVIDIA's four-billion-parameter Dory model reportedly beats some larger models on average across twelve benchmarks with two or more loops.