Sign up to save your podcasts
Or
Share DeltaKV: Compressing KV Caches for Long Context
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
DeltaKV: Compressing KV Caches for Long Context
This episode explores DeltaKV, a method for reducing the huge GPU memory burden of KV caches in long-context language model inference without simply discarding old tokens. It contrasts three strategies for handling long contexts: token eviction, dynamic sparse attention, and true compression, arguing that the cache contains structured redundancy that can be exploited rather than treated as disposable overhead. The discussion highlights DeltaKV’s core idea of keeping a small uncompressed reference set and storing other cache entries as compressed residuals relative to similar past states, drawing an analogy to delta encoding or version control. Listeners would find it interesting because it connects transformer internals, systems constraints, and practical serving performance, including claims of cutting memory to 29 percent of baseline and reaching up to 2x throughput with supporting infrastructure like Sparse-vLLM.
Sources:
1. DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity — Jitai Hao, Qiang Huang, Yaowei Wang, Min Zhang, Jun Yu, 2026
http://arxiv.org/abs/2602.08005
2. Generalization through Memorization: Nearest Neighbor Language Models — Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, Mike Lewis, 2019
https://scholar.google.com/scholar?q=Generalization+through+Memorization%3A+Nearest+Neighbor+Language+Models
3. Reformer: The Efficient Transformer — Nikita Kitaev, Lukasz Kaiser, Anselm Levskaya, 2020
https://scholar.google.com/scholar?q=Reformer%3A+The+Efficient+Transformer
4. Improving Language Models by Retrieving from Trillions of Tokens — Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann and many others, 2021
https://scholar.google.com/scholar?q=Improving+Language+Models+by+Retrieving+from+Trillions+of+Tokens
5. Memorizing Transformers — Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, Christian Szegedy, 2022
https://scholar.google.com/scholar?q=Memorizing+Transformers
6. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu, Hanchen Li, Yihua Cheng and others, 2023
https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving
7. Palu: Compressing KV-Cache with Low-Rank Projection — Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin and others, 2024
https://scholar.google.com/scholar?q=Palu%3A+Compressing+KV-Cache+with+Low-Rank+Projection
8. Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries — Junhyuck Kim, Jongho Park, Jaewoong Cho, Dimitris Papailiopoulos, 2025
https://scholar.google.com/scholar?q=Lexico%3A+Extreme+KV+Cache+Compression+via+Sparse+Coding+over+Universal+Dictionaries
9. Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models — Alina Shutova, Vladimir Malinovskii, Vage Egiazarian and others, 2025
https://scholar.google.com/scholar?q=Cache+Me+If+You+Must%3A+Adaptive+Key-Value+Quantization+for+Large+Language+Models
10. OmniKV: Dynamic Context Selection for Efficient Long-Context LLMs — Jitai Hao, Yuke Zhu, Tian Wang, Jun Yu, Xin Xin, Bo Zheng, Zhaochun Ren, Sheng Guo, 2025
https://scholar.google.com/scholar?q=OmniKV%3A+Dynamic+Context+Selection+for+Efficient+Long-Context+LLMs
11. QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024
https://scholar.google.com/scholar?q=QUEST%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference
12. Palu: KV-Cache Compression with Low-Rank Projection — Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, Kai-Chiang Wu, 2025
https://scholar.google.com/scholar?q=Palu%3A+KV-Cache+Compression+with+Low-Rank+Projection
13. SCBench: A KV Cache-Centric Analysis of Long-Context Methods — Yucheng Li, Huiqiang Jiang, Qianhui Wu, Xufang Luo, Surin Ahn, Chengruidong Zhang, Amir H. Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, Lili Qiu, 2025
https://scholar.google.com/scholar?q=SCBench%3A+A+KV+Cache-Centric+Analysis+of+Long-Context+Methods
14. The Pitfalls of KV Cache Compression — Alex Chen, Renato Geh, Aditya Grover, Guy Van den Broeck, Daniel Israel, 2025
https://scholar.google.com/scholar?q=The+Pitfalls+of+KV+Cache+Compression
15. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction — Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee, Sangdoo Yun, Hyun Oh Song, 2025
https://scholar.google.com/scholar?q=KVzip%3A+Query-Agnostic+KV+Cache+Compression+with+Context+Reconstruction
16. Kvlink: Accelerating Large Language Models via Efficient KV Cache Reuse — approx. recent LLM systems authors, 2025/2026
https://scholar.google.com/scholar?q=Kvlink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse
17. HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse — approx. recent RAG systems authors, 2025/2026
https://scholar.google.com/scholar?q=HyperRAG%3A+Enhancing+Quality-Efficiency+Tradeoffs+in+Retrieval-Augmented+Generation+with+Reranker+KV-Cache+Reuse
18. ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation — approx. recent RAG/serving authors, 2025/2026
https://scholar.google.com/scholar?q=ProphetKV%3A+User-Query-Driven+Selective+Recomputation+for+Efficient+KV+Cache+Reuse+in+Retrieval-Augmented+Generation
19. KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs — approx. recent KV quantization authors, 2025/2026
https://scholar.google.com/scholar?q=KVSink%3A+Understanding+and+Enhancing+the+Preservation+of+Attention+Sinks+in+KV+Cache+Quantization+for+LLMs
20. LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference — approx. recent long-context inference authors, 2025/2026
https://scholar.google.com/scholar?q=LLMs+Know+What+to+Drop%3A+Self-Attention+Guided+KV+Cache+Eviction+for+Efficient+Long-Context+Inference
21. Task-KV: Task-Aware KV Cache Optimization via Semantic Differentiation of Attention Heads — approx. recent attention/KV optimization authors, 2025/2026
https://scholar.google.com/scholar?q=Task-KV%3A+Task-Aware+KV+Cache+Optimization+via+Semantic+Differentiation+of+Attention+Heads
22. WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More — approx. recent quantization authors, 2025/2026
https://scholar.google.com/scholar?q=WKVQuant%3A+Quantizing+Weight+and+Key%2FValue+Cache+for+Large+Language+Models+Gains+More
23. AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantization — approx. recent KV systems/quantization authors, 2025/2026
https://scholar.google.com/scholar?q=AlignedKV%3A+Reducing+Memory+Access+of+KV-Cache+with+Precision-Aligned+Quantization
24. Making Every Head Count: Sparse Attention Without the Speed-Performance Trade-off — approx. recent sparse attention authors, 2025/2026
https://scholar.google.com/scholar?q=Making+Every+Head+Count%3A+Sparse+Attention+Without+the+Speed-Performance+Trade-off
25. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — approx. recent sparse systems authors, 2025/2026
https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention
26. AI Post Transformers: Stochastic KV Routing for Cache Sharing — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-29-stochastic-kv-routing-for-cache-sharing-5fef63.mp3
27. AI Post Transformers: ContiguousKV for Faster LLM Prefill KV Reuse — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-20-contiguouskv-for-faster-llm-prefill-kv-r-59f545.mp3
28. AI Post Transformers: KVSwap for Disk-Aware Long-Context On-Device Inference — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-16-kvswap-for-disk-aware-long-context-on-de-f3c15e.mp3
29. AI Post Transformers: Native Sparse Attention: Efficient Long-Context LLMs — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/native-sparse-attention-efficient-long-context-llms/
30. AI Post Transformers: Memory Sparse Attention for 100M-Token Scaling — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-07-memory-sparse-attention-for-100m-token-s-377cff.mp3
31. AI Post Transformers: AWQ: On-Device LLM Compression and Acceleration — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/awq-on-device-llm-compression-and-acceleration/
View all episodes
By mcgrofThis episode explores DeltaKV, a method for reducing the huge GPU memory burden of KV caches in long-context language model inference without simply discarding old tokens. It contrasts three strategies for handling long contexts: token eviction, dynamic sparse attention, and true compression, arguing that the cache contains structured redundancy that can be exploited rather than treated as disposable overhead. The discussion highlights DeltaKV’s core idea of keeping a small uncompressed reference set and storing other cache entries as compressed residuals relative to similar past states, drawing an analogy to delta encoding or version control. Listeners would find it interesting because it connects transformer internals, systems constraints, and practical serving performance, including claims of cutting memory to 29 percent of baseline and reaching up to 2x throughput with supporting infrastructure like Sparse-vLLM.
Sources:
1. DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity — Jitai Hao, Qiang Huang, Yaowei Wang, Min Zhang, Jun Yu, 2026
http://arxiv.org/abs/2602.08005
2. Generalization through Memorization: Nearest Neighbor Language Models — Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, Mike Lewis, 2019
https://scholar.google.com/scholar?q=Generalization+through+Memorization%3A+Nearest+Neighbor+Language+Models
3. Reformer: The Efficient Transformer — Nikita Kitaev, Lukasz Kaiser, Anselm Levskaya, 2020
https://scholar.google.com/scholar?q=Reformer%3A+The+Efficient+Transformer
4. Improving Language Models by Retrieving from Trillions of Tokens — Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann and many others, 2021
https://scholar.google.com/scholar?q=Improving+Language+Models+by+Retrieving+from+Trillions+of+Tokens
5. Memorizing Transformers — Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, Christian Szegedy, 2022
https://scholar.google.com/scholar?q=Memorizing+Transformers
6. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu, Hanchen Li, Yihua Cheng and others, 2023
https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving
7. Palu: Compressing KV-Cache with Low-Rank Projection — Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin and others, 2024
https://scholar.google.com/scholar?q=Palu%3A+Compressing+KV-Cache+with+Low-Rank+Projection
8. Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries — Junhyuck Kim, Jongho Park, Jaewoong Cho, Dimitris Papailiopoulos, 2025
https://scholar.google.com/scholar?q=Lexico%3A+Extreme+KV+Cache+Compression+via+Sparse+Coding+over+Universal+Dictionaries
9. Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models — Alina Shutova, Vladimir Malinovskii, Vage Egiazarian and others, 2025
https://scholar.google.com/scholar?q=Cache+Me+If+You+Must%3A+Adaptive+Key-Value+Quantization+for+Large+Language+Models
10. OmniKV: Dynamic Context Selection for Efficient Long-Context LLMs — Jitai Hao, Yuke Zhu, Tian Wang, Jun Yu, Xin Xin, Bo Zheng, Zhaochun Ren, Sheng Guo, 2025
https://scholar.google.com/scholar?q=OmniKV%3A+Dynamic+Context+Selection+for+Efficient+Long-Context+LLMs
11. QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024
https://scholar.google.com/scholar?q=QUEST%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference
12. Palu: KV-Cache Compression with Low-Rank Projection — Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, Kai-Chiang Wu, 2025
https://scholar.google.com/scholar?q=Palu%3A+KV-Cache+Compression+with+Low-Rank+Projection
13. SCBench: A KV Cache-Centric Analysis of Long-Context Methods — Yucheng Li, Huiqiang Jiang, Qianhui Wu, Xufang Luo, Surin Ahn, Chengruidong Zhang, Amir H. Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, Lili Qiu, 2025
https://scholar.google.com/scholar?q=SCBench%3A+A+KV+Cache-Centric+Analysis+of+Long-Context+Methods
14. The Pitfalls of KV Cache Compression — Alex Chen, Renato Geh, Aditya Grover, Guy Van den Broeck, Daniel Israel, 2025
https://scholar.google.com/scholar?q=The+Pitfalls+of+KV+Cache+Compression
15. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction — Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee, Sangdoo Yun, Hyun Oh Song, 2025
https://scholar.google.com/scholar?q=KVzip%3A+Query-Agnostic+KV+Cache+Compression+with+Context+Reconstruction
16. Kvlink: Accelerating Large Language Models via Efficient KV Cache Reuse — approx. recent LLM systems authors, 2025/2026
https://scholar.google.com/scholar?q=Kvlink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse
17. HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse — approx. recent RAG systems authors, 2025/2026
https://scholar.google.com/scholar?q=HyperRAG%3A+Enhancing+Quality-Efficiency+Tradeoffs+in+Retrieval-Augmented+Generation+with+Reranker+KV-Cache+Reuse
18. ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation — approx. recent RAG/serving authors, 2025/2026
https://scholar.google.com/scholar?q=ProphetKV%3A+User-Query-Driven+Selective+Recomputation+for+Efficient+KV+Cache+Reuse+in+Retrieval-Augmented+Generation
19. KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs — approx. recent KV quantization authors, 2025/2026
https://scholar.google.com/scholar?q=KVSink%3A+Understanding+and+Enhancing+the+Preservation+of+Attention+Sinks+in+KV+Cache+Quantization+for+LLMs
20. LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference — approx. recent long-context inference authors, 2025/2026
https://scholar.google.com/scholar?q=LLMs+Know+What+to+Drop%3A+Self-Attention+Guided+KV+Cache+Eviction+for+Efficient+Long-Context+Inference
21. Task-KV: Task-Aware KV Cache Optimization via Semantic Differentiation of Attention Heads — approx. recent attention/KV optimization authors, 2025/2026
https://scholar.google.com/scholar?q=Task-KV%3A+Task-Aware+KV+Cache+Optimization+via+Semantic+Differentiation+of+Attention+Heads
22. WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More — approx. recent quantization authors, 2025/2026
https://scholar.google.com/scholar?q=WKVQuant%3A+Quantizing+Weight+and+Key%2FValue+Cache+for+Large+Language+Models+Gains+More
23. AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantization — approx. recent KV systems/quantization authors, 2025/2026
https://scholar.google.com/scholar?q=AlignedKV%3A+Reducing+Memory+Access+of+KV-Cache+with+Precision-Aligned+Quantization
24. Making Every Head Count: Sparse Attention Without the Speed-Performance Trade-off — approx. recent sparse attention authors, 2025/2026
https://scholar.google.com/scholar?q=Making+Every+Head+Count%3A+Sparse+Attention+Without+the+Speed-Performance+Trade-off
25. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — approx. recent sparse systems authors, 2025/2026
https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention
26. AI Post Transformers: Stochastic KV Routing for Cache Sharing — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-29-stochastic-kv-routing-for-cache-sharing-5fef63.mp3
27. AI Post Transformers: ContiguousKV for Faster LLM Prefill KV Reuse — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-20-contiguouskv-for-faster-llm-prefill-kv-r-59f545.mp3
28. AI Post Transformers: KVSwap for Disk-Aware Long-Context On-Device Inference — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-16-kvswap-for-disk-aware-long-context-on-de-f3c15e.mp3
29. AI Post Transformers: Native Sparse Attention: Efficient Long-Context LLMs — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/native-sparse-attention-efficient-long-context-llms/
30. AI Post Transformers: Memory Sparse Attention for 100M-Token Scaling — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-07-memory-sparse-attention-for-100m-token-s-377cff.mp3
31. AI Post Transformers: AWQ: On-Device LLM Compression and Acceleration — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/awq-on-device-llm-compression-and-acceleration/