GAAC: Lossless KV Cache Compression for Faster LLM Inference

Researchers at the University of Notre Dame describe GAAC, a lossless KV cache codec that stores BF16 exponents as short offsets from calibrated per-channel bases and reconstructs them inside a fused attention kernel. It compresses the cache about 1.41× across five models and cuts resident KV memory by roughly 26%. On DGX Spark, 64K-context decode attention runs 1.19–1.27× faster than FlashAttention-2, and serving throughput rises 13–29% at equal KV memory. These are the authors' own measurements, and the Apple M4 speedups are smaller at short contexts.