The story’s score and claims

Comments on No. 12 of 140 · haoailab.com · edition of Friday 20 November 2026

Recursive Sparse Attention: Million-Token Extrapolation at Logarithmic Cost

Summary

Haoai Lab's RecSA builds a tree over the KV cache so each query searches it level by level, keeping selection cost and entries read at O(log L). Unselected nodes stay in the softmax as summaries, and candidate-local RoPE keeps relative distances within the trained range. Starting from Qwen3-4B and training on 0.62B tokens at 32K, it scores 60.5 on RULER at 1M versus 40.0 for Dense + DCA, and decodes 3.8× faster at 524K on one H200. Results are the authors' own, on a 4B model; larger models are untested.

Comments

Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules