Attention to Mamba: A Recipe for Cross-Architecture Distillation
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Researchers Abhinav Moudgil, Ningyuan Huang, and colleagues propose a two-stage recipe for distilling a Transformer into a Mamba-style state space model without keeping any attention blocks. The method first linearizes attention via a kernel trick, then distills that into Mamba.