Sign up to save your podcasts
Or
Share IndexMem: Learned KV-Cache Eviction for Long-Context LLMs
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
IndexMem: Learned KV-Cache Eviction for Long-Context LLMs
This episode explores IndexMem, a long-context LLM inference method that tries to cut KV-cache memory by learning which token states to evict while preserving useful information in a fixed-size latent memory. It explains the mechanics behind the KV cache, prefill, and decoding, then frames the real systems problem: for long prompts, memory traffic and bandwidth can become a bigger bottleneck than raw compute. The discussion focuses on two distinct challenges the paper separates clearly: predicting which cached tokens will matter in the future, and avoiding irreversible forgetting after eviction by writing evicted information into a learned summary state. Listeners interested in code agents, multimodal pipelines, and long-context serving will find it useful because it connects transformer theory to practical deployment constraints while questioning whether current evidence really supports the paper’s bigger million-token ambitions.
Sources:
1. IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference — Xintong Yang, Hao Gu, Binxing Xu, Lujun Li, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Sirui Han, Yike Guo, 2026
http://arxiv.org/abs/2605.25475
2. Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution — Alessio Devoto, Maximilian Jeblick, Simon Jegou, 2025
https://scholar.google.com/scholar?q=Expected+Attention%3A+KV+Cache+Compression+by+Estimating+Attention+from+Future+Queries+Distribution
3. Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query — Yixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Yang Xu, Qingfu Zhu, Wanxiang Che, 2025
https://scholar.google.com/scholar?q=Lookahead+Q-Cache%3A+Achieving+More+Consistent+KV+Cache+Eviction+via+Pseudo+Query
4. Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices — Yuxiang Huang, Binhang Yuan, Xu Han, Chaojun Xiao, Zhiyuan Liu, 2024
https://scholar.google.com/scholar?q=Locret%3A+Enhancing+Eviction+in+Long-Context+LLM+Inference+with+Trained+Retaining+Heads+on+Consumer-Grade+Devices
5. KVReviver: Reversible KV Cache Compression with Sketch-Based Token Reconstruction — Aomufei Yuan, Zhiming Wang, Ruijie Miao, Dayu Wang, Yuxuan Tian, Zihan Wang, Yebo Peng, Yuhan Wu, Bairen Yi, Xin Liu, Tong Yang, 2025
https://scholar.google.com/scholar?q=KVReviver%3A+Reversible+KV+Cache+Compression+with+Sketch-Based+Token+Reconstruction
6. xKV: Cross-Layer SVD for KV-Cache Compression — Chi-Chih Chang, Chien-Yu Lin, Yash Akhauri, Wei-Cheng Lin, Kai-Chiang Wu, Luis Ceze, Mohamed S. Abdelfattah, 2025
https://scholar.google.com/scholar?q=xKV%3A+Cross-Layer+SVD+for+KV-Cache+Compression
7. IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse — Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, Zhengxiao Du, Aohan Zeng, Jie Tang, Juanzi Li, 2026
https://scholar.google.com/scholar?q=IndexCache%3A+Accelerating+Sparse+Attention+via+Cross-Layer+Index+Reuse
8. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2024
https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time
9. FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference — Guangda Liu et al., 2025
https://arxiv.org/abs/2505.13109
10. Streaming Video Question-Answering with In-context Video KV-Cache Retrieval — Shangzhe Di et al., 2025
https://arxiv.org/abs/2503.00540
11. LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference — Guangtao Wang et al., 2025
https://arxiv.org/abs/2503.08879
12. Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference — Yuan Feng et al., 2024
https://arxiv.org/abs/2407.11550
13. In-context KV-Cache Eviction for LLMs via Attention-Gate — Zihao Zeng et al., 2024
https://arxiv.org/abs/2410.12876
14. MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference — Yu Li et al., 2026
https://arxiv.org/abs/2606.01563
15. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference — Xiang Liu et al., 2025
https://arxiv.org/abs/2502.00299
16. MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference — Zhongwei Wan et al., 2025
https://arxiv.org/abs/2502.17599
17. AI Post Transformers: Adaptive Compression Techniques for Efficient LLM Inference — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/adaptive-compression-techniques-for-efficient-llm-inference/
18. AI Post Transformers: Explicit Information Transmission for Context Compression — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-05-explicit-information-transmission-for-co-24e3c2.mp3
19. AI Post Transformers: TRELLIS and Bounded-Memory Transformer KV Compression — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-02-trellis-and-bounded-memory-transformer-k-81f237.mp3
20. AI Post Transformers: End-to-End Context Compression at Scale — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-10-end-to-end-context-compression-at-scale-278c70.mp3
21. AI Post Transformers: 50x KV Cache Compression in Seconds via Attention Matching — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/50x-kv-cache-compression-in-seconds-via-attention-matching/
22. AI Post Transformers: Stochastic KV Routing for Cache Sharing — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-29-stochastic-kv-routing-for-cache-sharing-5fef63.mp3
Interactive Visualization: IndexMem: Learned KV-Cache Eviction for Long-Context LLMs
View all episodes
By mcgrofThis episode explores IndexMem, a long-context LLM inference method that tries to cut KV-cache memory by learning which token states to evict while preserving useful information in a fixed-size latent memory. It explains the mechanics behind the KV cache, prefill, and decoding, then frames the real systems problem: for long prompts, memory traffic and bandwidth can become a bigger bottleneck than raw compute. The discussion focuses on two distinct challenges the paper separates clearly: predicting which cached tokens will matter in the future, and avoiding irreversible forgetting after eviction by writing evicted information into a learned summary state. Listeners interested in code agents, multimodal pipelines, and long-context serving will find it useful because it connects transformer theory to practical deployment constraints while questioning whether current evidence really supports the paper’s bigger million-token ambitions.
Sources:
1. IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference — Xintong Yang, Hao Gu, Binxing Xu, Lujun Li, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Sirui Han, Yike Guo, 2026
http://arxiv.org/abs/2605.25475
2. Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution — Alessio Devoto, Maximilian Jeblick, Simon Jegou, 2025
https://scholar.google.com/scholar?q=Expected+Attention%3A+KV+Cache+Compression+by+Estimating+Attention+from+Future+Queries+Distribution
3. Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query — Yixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Yang Xu, Qingfu Zhu, Wanxiang Che, 2025
https://scholar.google.com/scholar?q=Lookahead+Q-Cache%3A+Achieving+More+Consistent+KV+Cache+Eviction+via+Pseudo+Query
4. Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices — Yuxiang Huang, Binhang Yuan, Xu Han, Chaojun Xiao, Zhiyuan Liu, 2024
https://scholar.google.com/scholar?q=Locret%3A+Enhancing+Eviction+in+Long-Context+LLM+Inference+with+Trained+Retaining+Heads+on+Consumer-Grade+Devices
5. KVReviver: Reversible KV Cache Compression with Sketch-Based Token Reconstruction — Aomufei Yuan, Zhiming Wang, Ruijie Miao, Dayu Wang, Yuxuan Tian, Zihan Wang, Yebo Peng, Yuhan Wu, Bairen Yi, Xin Liu, Tong Yang, 2025
https://scholar.google.com/scholar?q=KVReviver%3A+Reversible+KV+Cache+Compression+with+Sketch-Based+Token+Reconstruction
6. xKV: Cross-Layer SVD for KV-Cache Compression — Chi-Chih Chang, Chien-Yu Lin, Yash Akhauri, Wei-Cheng Lin, Kai-Chiang Wu, Luis Ceze, Mohamed S. Abdelfattah, 2025
https://scholar.google.com/scholar?q=xKV%3A+Cross-Layer+SVD+for+KV-Cache+Compression
7. IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse — Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, Zhengxiao Du, Aohan Zeng, Jie Tang, Juanzi Li, 2026
https://scholar.google.com/scholar?q=IndexCache%3A+Accelerating+Sparse+Attention+via+Cross-Layer+Index+Reuse
8. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2024
https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time
9. FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference — Guangda Liu et al., 2025
https://arxiv.org/abs/2505.13109
10. Streaming Video Question-Answering with In-context Video KV-Cache Retrieval — Shangzhe Di et al., 2025
https://arxiv.org/abs/2503.00540
11. LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference — Guangtao Wang et al., 2025
https://arxiv.org/abs/2503.08879
12. Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference — Yuan Feng et al., 2024
https://arxiv.org/abs/2407.11550
13. In-context KV-Cache Eviction for LLMs via Attention-Gate — Zihao Zeng et al., 2024
https://arxiv.org/abs/2410.12876
14. MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference — Yu Li et al., 2026
https://arxiv.org/abs/2606.01563
15. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference — Xiang Liu et al., 2025
https://arxiv.org/abs/2502.00299
16. MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference — Zhongwei Wan et al., 2025
https://arxiv.org/abs/2502.17599
17. AI Post Transformers: Adaptive Compression Techniques for Efficient LLM Inference — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/adaptive-compression-techniques-for-efficient-llm-inference/
18. AI Post Transformers: Explicit Information Transmission for Context Compression — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-05-explicit-information-transmission-for-co-24e3c2.mp3
19. AI Post Transformers: TRELLIS and Bounded-Memory Transformer KV Compression — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-02-trellis-and-bounded-memory-transformer-k-81f237.mp3
20. AI Post Transformers: End-to-End Context Compression at Scale — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-10-end-to-end-context-compression-at-scale-278c70.mp3
21. AI Post Transformers: 50x KV Cache Compression in Seconds via Attention Matching — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/50x-kv-cache-compression-in-seconds-via-attention-matching/
22. AI Post Transformers: Stochastic KV Routing for Cache Sharing — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-29-stochastic-kv-routing-for-cache-sharing-5fef63.mp3
Interactive Visualization: IndexMem: Learned KV-Cache Eviction for Long-Context LLMs