Recursive Sparse Attention: Million-Token Extrapolation at Logarithmic Cost
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Haoai Lab's RecSA builds a tree over the KV cache so each query searches it level by level, keeping selection cost and entries read at O(log L). Unselected nodes stay in the softmax as summaries, and candidate-local RoPE keeps relative distances within the trained range. Starting from Qwen3-4B and training on 0.62B tokens at 32K, it scores 60.5 on RULER at 1M versus 40.0 for Dense + DCA, and decodes 3.8× faster at 524K on one H200. Results are the authors' own, on a 4B model; larger models are untested.