Sign up to save your podcasts
Or
Share Breaking the Prefix Barrier with Shared KV Cache
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Breaking the Prefix Barrier with Shared KV Cache
This episode explores a 2026 paper proposing that multiple LLM agents should share transformer KV-cache state, not just text, so they can avoid repeatedly paying the prefill cost of rereading the same plans, critiques, and intermediate outputs. It explains the systems background behind prefix caching, vLLM’s PagedAttention, and SGLang, then focuses on why multi-agent workflows break the exact-prefix assumption and make segment-level reuse much harder. The discussion highlights the paper’s core technical tension: the idea is compelling, but reusing cached activations across different prompt positions is fragile because of positional encoding effects such as RoPE misalignment and attention behavior. Listeners would find it interesting because it connects a practical bottleneck in agent systems to deep transformer internals, while also questioning whether the paper truly delivers fine-grained semantic sharing or a narrower form of reusable output caching.
Sources:
1. Breaking the Prefix Barrier with Shared KV Cache
https://openreview.net/forum?id=kgzBkyqg6Z
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
3. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2024
https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs
4. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang, 2025
https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion
5. KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems — Hancheng Ye, Zhengqi Gao, Mingyuan Ma, Qinsi Wang, Yuzhe Fu, Ming-Yu Chung, Yueqian Lin, Zhijian Liu, Jianyi Zhang, Danyang Zhuo, Yiran Chen, 2025
https://scholar.google.com/scholar?q=KVCOMM%3A+Online+Cross-context+KV-cache+Communication+for+Efficient+LLM-based+Multi-agent+Systems
6. EPIC: Efficient Position-Independent Caching for Serving Large Language Models — Junhao Hu, Wenrui Huang, Weidong Wang, Haoyi Wang, Tiancheng Hu, Qin Zhang, Hao Feng, Xusheng Chen, Yizhou Shan, Tao Xie, 2025
https://scholar.google.com/scholar?q=EPIC%3A+Efficient+Position-Independent+Caching+for+Serving+Large+Language+Models
7. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows — Zaifeng Pan, Ajjkumar Patel, Zhengding Hu, Yipeng Shen, Yue Guan, Wan-Lu Li, Lianhui Qin, Yida Wang, Yufei Ding, 2025
https://scholar.google.com/scholar?q=KVFlow%3A+Efficient+Prefix+Caching+for+Accelerating+LLM-Based+Multi-Agent+Workflows
8. DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving — Yuhan Liu, Yuyang Huang, Jiayi Yao, Shaoting Feng, Zhuohan Gu, Kuntai Du, Hanchen Li, Yihua Cheng, Junchen Jiang, Shan Lu, Madan Musuvathi, Esha Choukse, 2024
https://scholar.google.com/scholar?q=DroidSpeak%3A+KV+Cache+Sharing+for+Cross-LLM+Communication+and+Multi-LLM+Serving
9. TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing — Zhuohang Bian, Feiyang Wu, Chengrui Zhang, Hangcheng Dong, Yun Liang, Youwei Zhuo, 2026
https://scholar.google.com/scholar?q=TokenDance%3A+Scaling+Multi-Agent+LLM+Serving+via+Collective+KV+Cache+Sharing
10. HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse — authors unclear from Scholar snippet, 2025
https://scholar.google.com/scholar?q=HyperRAG%3A+Enhancing+Quality-Efficiency+Tradeoffs+in+Retrieval-Augmented+Generation+with+Reranker+KV-Cache+Reuse
11. ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation — authors unclear from Scholar snippet, 2025
https://scholar.google.com/scholar?q=ProphetKV%3A+User-Query-Driven+Selective+Recomputation+for+Efficient+KV+Cache+Reuse+in+Retrieval-Augmented+Generation
12. Cache-craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation — authors unclear from Scholar snippet, 2025
https://scholar.google.com/scholar?q=Cache-craft%3A+Managing+Chunk-Caches+for+Efficient+Retrieval-Augmented+Generation
13. AttentionStore: Cost-Effective Attention Reuse Across Multi-Turn Conversations in Large Language Model Serving — authors unclear from Scholar snippet, 2025
https://scholar.google.com/scholar?q=AttentionStore%3A+Cost-Effective+Attention+Reuse+Across+Multi-Turn+Conversations+in+Large+Language+Model+Serving
14. BAT: Efficient Generative Recommender Serving with Bipartite Attention — authors unclear from Scholar snippet, 2025
https://scholar.google.com/scholar?q=BAT%3A+Efficient+Generative+Recommender+Serving+with+Bipartite+Attention
15. AI Post Transformers: Efficient KV Cache Sharing for Multi-LoRA Agents — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-efficient-kv-cache-sharing-for-multi-lor-afda05.mp3
16. AI Post Transformers: TokenDance for Multi-Agent KV Cache Sharing — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-tokendance-for-multi-agent-kv-cache-shar-aa9b99.mp3
17. AI Post Transformers: KV Cache TTL for Multi-Turn Agent Scheduling — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-09-kv-cache-ttl-for-multi-turn-agent-schedu-996bf1.mp3
18. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3
19. AI Post Transformers: Test-time Scaling for Multi-Agent Collaborative Reasoning — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-test-time-scaling-for-multi-agent-collab-082570.mp3
Interactive Visualization: Breaking the Prefix Barrier with Shared KV Cache
View all episodes
By mcgrofThis episode explores a 2026 paper proposing that multiple LLM agents should share transformer KV-cache state, not just text, so they can avoid repeatedly paying the prefill cost of rereading the same plans, critiques, and intermediate outputs. It explains the systems background behind prefix caching, vLLM’s PagedAttention, and SGLang, then focuses on why multi-agent workflows break the exact-prefix assumption and make segment-level reuse much harder. The discussion highlights the paper’s core technical tension: the idea is compelling, but reusing cached activations across different prompt positions is fragile because of positional encoding effects such as RoPE misalignment and attention behavior. Listeners would find it interesting because it connects a practical bottleneck in agent systems to deep transformer internals, while also questioning whether the paper truly delivers fine-grained semantic sharing or a narrower form of reusable output caching.
Sources:
1. Breaking the Prefix Barrier with Shared KV Cache
https://openreview.net/forum?id=kgzBkyqg6Z
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
3. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2024
https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs
4. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang, 2025
https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion
5. KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems — Hancheng Ye, Zhengqi Gao, Mingyuan Ma, Qinsi Wang, Yuzhe Fu, Ming-Yu Chung, Yueqian Lin, Zhijian Liu, Jianyi Zhang, Danyang Zhuo, Yiran Chen, 2025
https://scholar.google.com/scholar?q=KVCOMM%3A+Online+Cross-context+KV-cache+Communication+for+Efficient+LLM-based+Multi-agent+Systems
6. EPIC: Efficient Position-Independent Caching for Serving Large Language Models — Junhao Hu, Wenrui Huang, Weidong Wang, Haoyi Wang, Tiancheng Hu, Qin Zhang, Hao Feng, Xusheng Chen, Yizhou Shan, Tao Xie, 2025
https://scholar.google.com/scholar?q=EPIC%3A+Efficient+Position-Independent+Caching+for+Serving+Large+Language+Models
7. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows — Zaifeng Pan, Ajjkumar Patel, Zhengding Hu, Yipeng Shen, Yue Guan, Wan-Lu Li, Lianhui Qin, Yida Wang, Yufei Ding, 2025
https://scholar.google.com/scholar?q=KVFlow%3A+Efficient+Prefix+Caching+for+Accelerating+LLM-Based+Multi-Agent+Workflows
8. DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving — Yuhan Liu, Yuyang Huang, Jiayi Yao, Shaoting Feng, Zhuohan Gu, Kuntai Du, Hanchen Li, Yihua Cheng, Junchen Jiang, Shan Lu, Madan Musuvathi, Esha Choukse, 2024
https://scholar.google.com/scholar?q=DroidSpeak%3A+KV+Cache+Sharing+for+Cross-LLM+Communication+and+Multi-LLM+Serving
9. TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing — Zhuohang Bian, Feiyang Wu, Chengrui Zhang, Hangcheng Dong, Yun Liang, Youwei Zhuo, 2026
https://scholar.google.com/scholar?q=TokenDance%3A+Scaling+Multi-Agent+LLM+Serving+via+Collective+KV+Cache+Sharing
10. HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse — authors unclear from Scholar snippet, 2025
https://scholar.google.com/scholar?q=HyperRAG%3A+Enhancing+Quality-Efficiency+Tradeoffs+in+Retrieval-Augmented+Generation+with+Reranker+KV-Cache+Reuse
11. ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation — authors unclear from Scholar snippet, 2025
https://scholar.google.com/scholar?q=ProphetKV%3A+User-Query-Driven+Selective+Recomputation+for+Efficient+KV+Cache+Reuse+in+Retrieval-Augmented+Generation
12. Cache-craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation — authors unclear from Scholar snippet, 2025
https://scholar.google.com/scholar?q=Cache-craft%3A+Managing+Chunk-Caches+for+Efficient+Retrieval-Augmented+Generation
13. AttentionStore: Cost-Effective Attention Reuse Across Multi-Turn Conversations in Large Language Model Serving — authors unclear from Scholar snippet, 2025
https://scholar.google.com/scholar?q=AttentionStore%3A+Cost-Effective+Attention+Reuse+Across+Multi-Turn+Conversations+in+Large+Language+Model+Serving
14. BAT: Efficient Generative Recommender Serving with Bipartite Attention — authors unclear from Scholar snippet, 2025
https://scholar.google.com/scholar?q=BAT%3A+Efficient+Generative+Recommender+Serving+with+Bipartite+Attention
15. AI Post Transformers: Efficient KV Cache Sharing for Multi-LoRA Agents — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-efficient-kv-cache-sharing-for-multi-lor-afda05.mp3
16. AI Post Transformers: TokenDance for Multi-Agent KV Cache Sharing — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-tokendance-for-multi-agent-kv-cache-shar-aa9b99.mp3
17. AI Post Transformers: KV Cache TTL for Multi-Turn Agent Scheduling — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-09-kv-cache-ttl-for-multi-turn-agent-schedu-996bf1.mp3
18. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3
19. AI Post Transformers: Test-time Scaling for Multi-Agent Collaborative Reasoning — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-22-test-time-scaling-for-multi-agent-collab-082570.mp3
Interactive Visualization: Breaking the Prefix Barrier with Shared KV Cache