Sign up to save your podcasts
Or
Share From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation
This episode explores a systems paper on speeding up retrieval-augmented generation by reusing transformer KV cache state more intelligently, instead of recomputing long retrieved prompts from scratch on every request. It explains why RAG often improves grounding yet suffers from high time-to-first-token, especially when multiple retrieved chunks must be prefetched and encoded together. The discussion focuses on the paper’s central argument that naive chunk-level cache reuse breaks important cross-chunk interactions, and that the proposed FusionRAG Cache tries to preserve quality through offline chunk enrichment and selective online recomputation. A listener would find it interesting because it connects familiar RAG concepts to the real serving bottlenecks that determine whether enterprise assistants feel practical or painfully slow.
Sources:
1. From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation — Jiahao Wang, Weiyu Xie, Mingxing Zhang, Boxing Zhang, Jianwei Dong, Yuening Zhu, Chen Lin, Jinqi Tang, Yaochen Han, Zhiyuan Ai, Xianglin Chen, Yongwei Wu, Congfeng Jiang, 2026
http://arxiv.org/abs/2601.12904v1
2. REALM: Retrieval-Augmented Language Model Pre-Training — Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang, 2020
https://scholar.google.com/scholar?q=REALM%3A+Retrieval-Augmented+Language+Model+Pre-Training
3. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, Douwe Kiela, 2020
https://scholar.google.com/scholar?q=Retrieval-Augmented+Generation+for+Knowledge-Intensive+NLP+Tasks
4. Few-shot Learning with Retrieval Augmented Language Models — Gautier Izacard, Edouard Grave, Patrick Lewis, Benjamin Chintala, Tim Rocktaschel, Fabio Petroni, 2022
https://scholar.google.com/scholar?q=Few-shot+Learning+with+Retrieval+Augmented+Language+Models
5. Retrieval-Augmented Generation for Large Language Models: A Survey — Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, Haofen Wang, 2023
https://scholar.google.com/scholar?q=Retrieval-Augmented+Generation+for+Large+Language+Models%3A+A+Survey
6. APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding — Xinyu Yang, Tianqi Chen, Beidi Chen, 2025
https://scholar.google.com/scholar?q=APE%3A+Faster+and+Longer+Context-Augmented+Generation+via+Adaptive+Parallel+Encoding
7. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang, 2024
https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion
8. Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation — Shubham Agarwal, Sai Narayan Sundaresan, Subrata Mitra, Deb Mahapatra, Tong Yu, Shiv Saini, 2025
https://scholar.google.com/scholar?q=Cache-Craft%3A+Managing+Chunk-Caches+for+Efficient+Retrieval-Augmented+Generation
9. From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation — Jiahao Wang, Weiyu Xie, Mingxing Zhang, Boxing Zhang, Jianwei Dong, Yuening Zhu, Chen Lin, Jinqi Tang, Yaochen Han, Zhiyuan Ai, Xianglin Chen, Yongwei Wu, Congfeng Jiang, 2026
https://scholar.google.com/scholar?q=From+Prefix+Cache+to+Fusion+RAG+Cache%3A+Accelerating+LLM+Inference+in+Retrieval-Augmented+Generation
10. HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse — authors not reliably recoverable from the browsed snippets, 2025
https://scholar.google.com/scholar?q=HyperRAG%3A+Enhancing+Quality-Efficiency+Tradeoffs+in+Retrieval-Augmented+Generation+with+Reranker+KV-Cache+Reuse
11. Kvlink: Accelerating Large Language Models via Efficient KV Cache Reuse — approx. recent systems/LLM inference authors, 2025/2026
https://scholar.google.com/scholar?q=Kvlink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse
12. Sparse Attention across Multiple-context KV Cache — approx. recent long-context/RAG inference authors, 2025/2026
https://scholar.google.com/scholar?q=Sparse+Attention+across+Multiple-context+KV+Cache
13. CacheClip: Accelerating RAG with Effective KV Cache Reuse — approx. recent RAG systems authors, 2025/2026
https://scholar.google.com/scholar?q=CacheClip%3A+Accelerating+RAG+with+Effective+KV+Cache+Reuse
14. CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference — approx. recent long-context inference authors, 2025/2026
https://scholar.google.com/scholar?q=CHESS%3A+Context-aware+Hierarchical+Efficient+Semantic+Selection+for+Long-Context+LLM+Inference
15. BudgetMem: Learning Selective Memory Policies for Cost-Efficient Long-Context Processing in Language Models — approx. recent memory-policy learning authors, 2025/2026
https://scholar.google.com/scholar?q=BudgetMem%3A+Learning+Selective+Memory+Policies+for+Cost-Efficient+Long-Context+Processing+in+Language+Models
16. TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval — approx. recent RAG systems authors, 2025/2026
https://scholar.google.com/scholar?q=TeleRAG%3A+Efficient+Retrieval-Augmented+Generation+Inference+with+Lookahead+Retrieval
17. Understanding and Optimizing Multi-Stage AI Inference Pipelines — approx. recent systems authors, 2025/2026
https://scholar.google.com/scholar?q=Understanding+and+Optimizing+Multi-Stage+AI+Inference+Pipelines
18. AI Post Transformers: Episode: LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-21-lookaheadkv-fast-and-accurate-kv-c9d436.mp3
19. AI Post Transformers: CacheSlide: Position-Aware KV Cache Reuse for Agent LLMs — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-16-cacheslide-position-aware-kv-cache-reuse-cd59c7.mp3
20. AI Post Transformers: SolidAttention: Co-Designing Sparse Attention and SSD I/O — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-18-solidattention-co-designing-sparse-atten-5a8622.mp3
Interactive Visualization: Episode: From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation
View all episodes
By mcgrofThis episode explores a systems paper on speeding up retrieval-augmented generation by reusing transformer KV cache state more intelligently, instead of recomputing long retrieved prompts from scratch on every request. It explains why RAG often improves grounding yet suffers from high time-to-first-token, especially when multiple retrieved chunks must be prefetched and encoded together. The discussion focuses on the paper’s central argument that naive chunk-level cache reuse breaks important cross-chunk interactions, and that the proposed FusionRAG Cache tries to preserve quality through offline chunk enrichment and selective online recomputation. A listener would find it interesting because it connects familiar RAG concepts to the real serving bottlenecks that determine whether enterprise assistants feel practical or painfully slow.
Sources:
1. From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation — Jiahao Wang, Weiyu Xie, Mingxing Zhang, Boxing Zhang, Jianwei Dong, Yuening Zhu, Chen Lin, Jinqi Tang, Yaochen Han, Zhiyuan Ai, Xianglin Chen, Yongwei Wu, Congfeng Jiang, 2026
http://arxiv.org/abs/2601.12904v1
2. REALM: Retrieval-Augmented Language Model Pre-Training — Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang, 2020
https://scholar.google.com/scholar?q=REALM%3A+Retrieval-Augmented+Language+Model+Pre-Training
3. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, Douwe Kiela, 2020
https://scholar.google.com/scholar?q=Retrieval-Augmented+Generation+for+Knowledge-Intensive+NLP+Tasks
4. Few-shot Learning with Retrieval Augmented Language Models — Gautier Izacard, Edouard Grave, Patrick Lewis, Benjamin Chintala, Tim Rocktaschel, Fabio Petroni, 2022
https://scholar.google.com/scholar?q=Few-shot+Learning+with+Retrieval+Augmented+Language+Models
5. Retrieval-Augmented Generation for Large Language Models: A Survey — Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, Haofen Wang, 2023
https://scholar.google.com/scholar?q=Retrieval-Augmented+Generation+for+Large+Language+Models%3A+A+Survey
6. APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding — Xinyu Yang, Tianqi Chen, Beidi Chen, 2025
https://scholar.google.com/scholar?q=APE%3A+Faster+and+Longer+Context-Augmented+Generation+via+Adaptive+Parallel+Encoding
7. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang, 2024
https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion
8. Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation — Shubham Agarwal, Sai Narayan Sundaresan, Subrata Mitra, Deb Mahapatra, Tong Yu, Shiv Saini, 2025
https://scholar.google.com/scholar?q=Cache-Craft%3A+Managing+Chunk-Caches+for+Efficient+Retrieval-Augmented+Generation
9. From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation — Jiahao Wang, Weiyu Xie, Mingxing Zhang, Boxing Zhang, Jianwei Dong, Yuening Zhu, Chen Lin, Jinqi Tang, Yaochen Han, Zhiyuan Ai, Xianglin Chen, Yongwei Wu, Congfeng Jiang, 2026
https://scholar.google.com/scholar?q=From+Prefix+Cache+to+Fusion+RAG+Cache%3A+Accelerating+LLM+Inference+in+Retrieval-Augmented+Generation
10. HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse — authors not reliably recoverable from the browsed snippets, 2025
https://scholar.google.com/scholar?q=HyperRAG%3A+Enhancing+Quality-Efficiency+Tradeoffs+in+Retrieval-Augmented+Generation+with+Reranker+KV-Cache+Reuse
11. Kvlink: Accelerating Large Language Models via Efficient KV Cache Reuse — approx. recent systems/LLM inference authors, 2025/2026
https://scholar.google.com/scholar?q=Kvlink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse
12. Sparse Attention across Multiple-context KV Cache — approx. recent long-context/RAG inference authors, 2025/2026
https://scholar.google.com/scholar?q=Sparse+Attention+across+Multiple-context+KV+Cache
13. CacheClip: Accelerating RAG with Effective KV Cache Reuse — approx. recent RAG systems authors, 2025/2026
https://scholar.google.com/scholar?q=CacheClip%3A+Accelerating+RAG+with+Effective+KV+Cache+Reuse
14. CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference — approx. recent long-context inference authors, 2025/2026
https://scholar.google.com/scholar?q=CHESS%3A+Context-aware+Hierarchical+Efficient+Semantic+Selection+for+Long-Context+LLM+Inference
15. BudgetMem: Learning Selective Memory Policies for Cost-Efficient Long-Context Processing in Language Models — approx. recent memory-policy learning authors, 2025/2026
https://scholar.google.com/scholar?q=BudgetMem%3A+Learning+Selective+Memory+Policies+for+Cost-Efficient+Long-Context+Processing+in+Language+Models
16. TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval — approx. recent RAG systems authors, 2025/2026
https://scholar.google.com/scholar?q=TeleRAG%3A+Efficient+Retrieval-Augmented+Generation+Inference+with+Lookahead+Retrieval
17. Understanding and Optimizing Multi-Stage AI Inference Pipelines — approx. recent systems authors, 2025/2026
https://scholar.google.com/scholar?q=Understanding+and+Optimizing+Multi-Stage+AI+Inference+Pipelines
18. AI Post Transformers: Episode: LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-21-lookaheadkv-fast-and-accurate-kv-c9d436.mp3
19. AI Post Transformers: CacheSlide: Position-Aware KV Cache Reuse for Agent LLMs — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-16-cacheslide-position-aware-kv-cache-reuse-cd59c7.mp3
20. AI Post Transformers: SolidAttention: Co-Designing Sparse Attention and SSD I/O — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-18-solidattention-co-designing-sparse-atten-5a8622.mp3
Interactive Visualization: Episode: From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation