Why it matters. Every deployed language model hits a memory wall: the key-value cache grows linearly with context length, and current solutions—summarization, token eviction, token merging—are catastrophically lossy at high compression ratios. Fast KV Compaction via Attention Matching proposes compressing context in latent space rather than token space, directly constructing compact keys and values that reproduce the original attention mechanism's behavior. Using closed-form linear algebra instead of gradient descent, Attention Matching achieves 50× compression in seconds with quality rivaling expensive end-to-end optimization methods—and dramatically outperforming summarization on information-dense tasks like medical records QA, where summarization performs no better than having no context at all.

MIT CSAIL. The paper comes from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL). It builds on Cartridges (GitHub), which demonstrated latent-space KV cache compaction but required GPU-hours of gradient-based optimization per context. The Attention Matching code is available on GitHub. The paper is on arXiv (HTML version). Related prior work on token eviction includes H2O, SnapKV, PyramidKV, and KVzip.

The Researchers. Adam Zweiger (MIT undergraduate, lead author, personal site), Xinghong Fu (MIT), Han Guo (MIT PhD student, personal site), and Yoon Kim (Associate Professor at MIT CSAIL, faculty page).

Key Technical Concepts. The core innovation is Attention Matching: instead of optimizing a language model's final output end-to-end (as Cartridges does via prefix-tuning), the method optimizes attention outputs and attention mass directly at each KV head. This decomposes into three sequential steps—key selection (via attention scoring or orthogonal matching pursuit), bias fitting via nonnegative least squares, and value fitting via ordinary least squares—each with efficient closed-form or near-closed-form solutions. Scalar biases added to attention logits allow compact keys to account for multiple original tokens' worth of attention mass, leveraging the softmax decomposition used by FlashAttention. The paper's most impactful finding is nonuniform head budgets: attention head sensitivity to compaction is stable across inputs, so a one-time per-model budget allocation (computed via greedy exchange) dominates all other design choices. This connects to findings in hybrid architectures like Gemma 3, where most heads naturally specialize to local context and only a few perform long-range retrieval.

Daily Tech Feed: From the Labs is available on Apple Podcasts, Spotify, and wherever fine podcasts are distributed. Visit us at pod.c457.org for all our shows. New episodes daily.