Efficient Streaming Language Models with Attention Sinks

Attention-sink research finds that evicting the first few tokens from a KV cache makes perplexity collapse, even when those tokens are only newlines. In a GPT-2 small replication on wikitext-2 with the cache capped at 256, perplexity rose from 10.7 to 3768 once token 0 was evicted, and keeping four sink tokens restored the baseline. The rolling-cache fix can be applied to existing models. It preserves stable streaming, not memory of evicted text. The replication covers one small model, and the claims cite no sources.