Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- Splitwise: Phase-Split LLM InferenceThis episode examines Splitwise: Efficient Generative LLM Inference Using Phase Splitting, a 2024 systems paper from researchers at the University of Washington and Microsoft, and centers the discussion on a simple claim with large deployment consequences: prompt prefill and token decode are different enough that they should not necessarily run on the same hardware. The hosts walk through the basic mechanics of generative inference, explaining prefill as the parallel, compute-heavy stage that processes the prompt, and decode as the sequential, KV-cache-driven stage that generates tokens one by one. That distinction sets up the paper’s core argument that modern serving stacks are paying a penalty by treating inference as a uniform workload when its phases are constrained by very different resources.The conversation stays focused on why that split matters in practice. It unpacks phase heterogeneity in terms of throughput, latency, utilization, memory pressure, and power draw, and explains why decode can remain bottlenecked by memory bandwidth and capacity even on newer accelerators with far more raw FLOPs. From there, the episode explores Splitwise’s broader systems framing: if compute is scaling faster than memory, then assigning prefill to high-throughput hardware and decode to cheaper or lower-power machines may be a more realistic datacenter strategy than continuing to push everything through one homogeneous GPU fleet. The hosts also emphasize power-normalized evaluation as a more honest lens for operators than simple box-for-box performance comparisons.Along the way, the episode places Splitwise in public context alongside ORCA, PagedAttention, and SARATHI without losing its anchor. Those earlier systems are used to clarify what Splitwise does and does not claim: continuous batching, KV-cache-aware memory management, and batch reshaping all improve serving efficiency, but they do not eliminate the underlying asymmetry between prefill and decode. The result is a grounded discussion of phase splitting as a deployment decision rather than a purely algorithmic trick, with particular attention to where prefill-decode disaggregation looks compelling, where it depends on the realities of cluster design, and where the limits of PD disaggregation still leave open systems questions.Sources:1. Splitwise: Efficient generative LLM inference using phase splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini, 2023http://arxiv.org/abs/2311.186772. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024http://arxiv.org/abs/2401.096703. DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference — Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, Yixuan Tan, Wentao Zhang, Liyue Zhang, Shangyan Zhou, Yuxuan Liu, Shunfeng Zhou, Mingxing Zhang, Xin Jin, Panpan Huang, 2026http://arxiv.org/abs/2602.215484. Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Serving — Zongze Li, Jingyu Liu, Zach Xu, Yineng Zhang, Tahseen Rabbani, Ce Zhang, 2026http://arxiv.org/abs/2603.133585. Orca: A Distributed Serving System for Transformer-Based Generative Models — Gyeong-In Yu, Joo Seong Jeong, Gun-Woo Kim, Seungtae Kim, Byung-Gon Chun, 2022https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models6. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention7. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Animesh Agrawal, Aakash Panwar, Jaya Mohan, Nakul Kwatra, Bhaskar S. Gulavani, Ramachandran Ramjee, 2023https://scholar.google.com/scholar?q=SARATHI%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+with+Chunked+Prefills8. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving9. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2024https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving10. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse — Jingbo Yang, Bairu Hou, Wei Wei, Yujia Bao, Shiyu Chang, 2025https://scholar.google.com/scholar?q=KVLink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse11. HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse — Yuwei An, Yihua Cheng, Seo Jin Park, Junchen Jiang, 2025https://scholar.google.com/scholar?q=HyperRAG%3A+Enhancing+Quality-Efficiency+Tradeoffs+in+Retrieval-Augmented+Generation+with+Reranker+KV-Cache+Reuse12. ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation — Shihao Wang, Jiahao Chen, Yanqi Pan, Hao Huang and colleagues, 2026https://scholar.google.com/scholar?q=ProphetKV%3A+User-Query-Driven+Selective+Recomputation+for+Efficient+KV+Cache+Reuse+in+Retrieval-Augmented+Generation13. Accelerating LLM Inference with Staged Speculative Decoding — Benjamin Spector, Chris Re, 2023https://scholar.google.com/scholar?q=Accelerating+LLM+Inference+with+Staged+Speculative+Decoding14. SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices — Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, Max Ryabinin, 2024https://scholar.google.com/scholar?q=SpecExec%3A+Massively+Parallel+Speculative+Decoding+for+Interactive+LLM+Inference+on+Consumer+Devices15. KVDirect: Distributed Disaggregated LLM Inference — Shiyang Chen, Rain Jiang, Dezhi Yu, Jinlai Xu, Mengyuan Chao and colleagues, 2025https://scholar.google.com/scholar?q=KVDirect%3A+Distributed+Disaggregated+LLM+Inference16. Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture — Yu Wu, Tongxuan Liu, Yuting Zeng, Siyu Wu, Jun Xiong, Xianzhe Dong and colleagues, 2025https://scholar.google.com/scholar?q=Arrow%3A+Adaptive+Scheduling+Mechanisms+for+Disaggregated+LLM+Inference+Architecture17. WindServe: Efficient Phase-Disaggregated LLM Serving with Stream-based Dynamic Scheduling — Jingqi Feng, Yukai Huang, Rui Zhang, Sicheng Liang, Ming Yan, Jie Wu, 2025https://scholar.google.com/scholar?q=WindServe%3A+Efficient+Phase-Disaggregated+LLM+Serving+with+Stream-based+Dynamic+Scheduling18. AI Post Transformers: CacheSlide: Position-Aware KV Cache Reuse for Agent LLMs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-16-cacheslide-position-aware-kv-cache-reuse-cd59c7.mp319. AI Post Transformers: SGLang: Efficient Language Model Program Execution — Hal Turing & Dr. Ada Shannon, Sun,https://podcast.do-not-panic.com/episodes/sglang-efficient-language-model-program-execution/20. AI Post Transformers: Episode: Speculative Speculative Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-speculative-speculative-decoding-1b7a10.mp321. AI Post Transformers: xLLM: Co-Locating Online and Offline LLM Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-16-xllm-co-locating-online-and-offline-llm-10bb81.mp322. AI Post Transformers: FAST26: Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional — Hal Turing & Dr. Ada Shannon, Wed,https://podcast.do-not-panic.com/episodes/fast26-bidaw-enhancing-key-value-caching-for-interactive-llm-serving-via-bidirec/23. AI Post Transformers: Lookahead Q-Cache for Consistent KV Eviction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-lookahead-q-cache-for-consistent-kv-evic-d97b09.mp3Interactive Visualization: Splitwise: Phase-Split LLM Inference0 min
- TurboQuant: Online Vector Quantization with Near-optimal Distortion RateThis episode explores TurboQuant, a method for compressing high-dimensional vectors online without learning a dataset-specific codebook first, aimed at settings like LLM KV-cache compression and approximate nearest neighbor search. It explains why vector quantization is a different problem from ordinary weight quantization, and why preserving inner products can matter just as much as minimizing reconstruction error for retrieval quality and attention behavior. The discussion focuses on the paper’s central idea that a random rotation can regularize vectors enough for simple scalar quantization to approach information-theoretic distortion limits, at least under the paper’s theoretical assumptions. Listeners would find it interesting because it connects rate-distortion theory to concrete systems bottlenecks in modern AI, while also critically examining where the paper’s theoretical strength outpaces its empirical validation.Sources:1. TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate — Amir Zandieh, Majid Daliri, Majid Hadian, Vahab Mirrokni, 2025http://arxiv.org/abs/2504.198742. Product Quantization for Nearest Neighbor Search — Herve Jegou, Matthijs Douze, Cordelia Schmid, 2011https://scholar.google.com/scholar?q=Product+Quantization+for+Nearest+Neighbor+Search3. Quantization based Fast Inner Product Search — Ruiqi Guo, Sanjiv Kumar, Krzysztof Choromanski, David Simcha, 2016https://scholar.google.com/scholar?q=Quantization+based+Fast+Inner+Product+Search4. Norm-Explicit Quantization: Improving Vector Quantization for Maximum Inner Product Search — Xinyan Dai, Xiao Yan, Kelvin K. W. Ng, Jiu Liu, James Cheng, 2020https://scholar.google.com/scholar?q=Norm-Explicit+Quantization%3A+Improving+Vector+Quantization+for+Maximum+Inner+Product+Search5. Accelerating Large-Scale Inference with Anisotropic Vector Quantization — Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, Sanjiv Kumar, 2020https://scholar.google.com/scholar?q=Accelerating+Large-Scale+Inference+with+Anisotropic+Vector+Quantization6. QJL: 1-bit Quantized JL Transform for KV Cache Quantization with Zero Overhead — Amir Zandieh, Majid Daliri, Iman Han, 2024https://scholar.google.com/scholar?q=QJL%3A+1-bit+Quantized+JL+Transform+for+KV+Cache+Quantization+with+Zero+Overhead7. PolarQuant: Quantizing KV Caches with Polar Transformation — Iman Han, Prannay Kacham, Amin Karbasi, Vahab Mirrokni, Amir Zandieh, 2025https://scholar.google.com/scholar?q=PolarQuant%3A+Quantizing+KV+Caches+with+Polar+Transformation8. Practical and Asymptotically Optimal Quantization of High-Dimensional Vectors in Euclidean Space for Approximate Nearest Neighbor Search — Jianqiao Gao, Yuxuan Gou, Yiming Xu, Yuting Yang, Cheng Long, Raymond Chi-Wing Wong, 2024https://scholar.google.com/scholar?q=Practical+and+Asymptotically+Optimal+Quantization+of+High-Dimensional+Vectors+in+Euclidean+Space+for+Approximate+Nearest+Neighbor+Search9. KIVI: A Tuning-Free Asymmetric 2-bit Quantization for KV Cache — Zefan Liu, Jiapeng Yuan, Hongyin Jin, Shanghang Zhong, Zhiyuan Xu, Vladimir Braverman, Beidi Chen, Xia Hu, 2024https://scholar.google.com/scholar?q=KIVI%3A+A+Tuning-Free+Asymmetric+2-bit+Quantization+for+KV+Cache10. KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs — Zunhai Su, Kehong Yuan, 2025https://scholar.google.com/scholar?q=KVSink%3A+Understanding+and+Enhancing+the+Preservation+of+Attention+Sinks+in+KV+Cache+Quantization+for+LLMs11. ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification — Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, Bohan Zhuang, 2024https://scholar.google.com/scholar?q=ZipCache%3A+Accurate+and+Efficient+KV+Cache+Quantization+with+Salient+Token+Identification12. AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models — Zunhai Su, Wang Shen, Linge Li, Zhe Chen, Hanyu Wei, Huangqi Yu, Kehong Yuan, 2025https://scholar.google.com/scholar?q=AKVQ-VL%3A+Attention-Aware+KV+Cache+Adaptive+2-Bit+Quantization+for+Vision-Language+Models13. SpinQuant: LLM Quantization with Learned Rotations — Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, Tijmen Blankevoort, 2024https://scholar.google.com/scholar?q=SpinQuant%3A+LLM+Quantization+with+Learned+Rotations14. Rotate, Clip, and Partition: Towards W2A4KV4 Quantization by Integrating Rotation and Learnable Non-uniform Quantizer — Euntae Choi, Sumin Song, Woosang Lim, Sungjoo Yoo, 2025https://scholar.google.com/scholar?q=Rotate%2C+Clip%2C+and+Partition%3A+Towards+W2A4KV4+Quantization+by+Integrating+Rotation+and+Learnable+Non-uniform+Quantizer15. Locally-Adaptive Quantization for Streaming Vector Search — Cecilia Aguerrebere, Mark Hildebrand, Ishwar Singh Bhati, Theodore Willke, Mariano Tepper, 2024https://scholar.google.com/scholar?q=Locally-Adaptive+Quantization+for+Streaming+Vector+Search16. Sampling Methods for Inner Product Sketching — Majid Daliri, Juliana Freire, Christopher Musco, Aecio Santos, Haoxiang Zhang, 2024https://scholar.google.com/scholar?q=Sampling+Methods+for+Inner+Product+Sketching17. AI Post Transformers: Memory Traffic Saturation in Transformer Decode — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-20-memory-traffic-saturation-in-transformer-cd4961.mp318. AI Post Transformers: LAQ for Smarter KV Cache Eviction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-23-laq-for-smarter-kv-cache-eviction-3ea2b8.mp319. AI Post Transformers: Lookahead Q-Cache for Consistent KV Eviction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-lookahead-q-cache-for-consistent-kv-evic-d97b09.mp320. AI Post Transformers: AWQ: On-Device LLM Compression and Acceleration — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/awq-on-device-llm-compression-and-acceleration/21. AI Post Transformers: Sentence-BERT: Siamese Networks for Sentence Embeddings — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/sentence-bert-siamese-networks-for-sentence-embeddings/22. AI Post Transformers: SolidAttention: Co-Designing Sparse Attention and SSD I/O — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-18-solidattention-co-designing-sparse-atten-5a8622.mp3Interactive Visualization: Episode: TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate0 min
- HyperAgents and Metacognitive Self-ImprovementThis episode explores the HyperAgents paper and its central claim that an AI system can improve not only its task behavior but also the procedure it uses to generate future improvements. It explains recursive self-improvement in practical terms as an outer engineering loop over prompts, code, tools, memory, and evaluators, and contrasts that with standard deep learning, where the learning process itself stays fixed. The discussion focuses on why freezing the meta-agent creates a conceptual and practical bottleneck, how HyperAgents try to remove that ceiling by making the improver editable too, and why that could matter beyond coding in domains like reviewing or grading. A listener would find it interesting for its clear debate over whether this is a genuine step toward more general self-improving agents or simply a cleaner packaging of familiar external scaffolding and control mechanisms.Sources:1. Hyperagents — Jenny Zhang, Bingchen Zhao, Wannan Yang, Jakob Foerster, Jeff Clune, Minqi Jiang, Sam Devlin, Tatiana Shavrina, 2026http://arxiv.org/abs/2603.194612. Gödel Machines: Fully Self-Referential Optimal Universal Self-Improvers — Jürgen Schmidhuber, 2003https://scholar.google.com/scholar?q=G%C3%B6del+Machines%3A+Fully+Self-Referential+Optimal+Universal+Self-Improvers3. Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation — Eric Zelikman, Eliana Lorch, Lester Mackey, Adam Tauman Kalai, 2023https://scholar.google.com/scholar?q=Self-Taught+Optimizer+%28STOP%29%3A+Recursively+Self-Improving+Code+Generation4. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents — Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, Jeff Clune, 2025https://scholar.google.com/scholar?q=Darwin+G%C3%B6del+Machine%3A+Open-Ended+Evolution+of+Self-Improving+Agents5. AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery — Alexander Novikov, Ngân Vũ, Marvin Eisenberger and colleagues, 2025https://scholar.google.com/scholar?q=AlphaEvolve%3A+A+Coding+Agent+for+Scientific+and+Algorithmic+Discovery6. Self-Referential Meta Learning — Louis Kirsch, Jürgen Schmidhuber, 2022https://scholar.google.com/scholar?q=Self-Referential+Meta+Learning7. A Modern Self-Referential Weight Matrix That Learns to Modify Itself — Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber, 2022https://scholar.google.com/scholar?q=A+Modern+Self-Referential+Weight+Matrix+That+Learns+to+Modify+Itself8. Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement — Xunjian Yin, Xinyi Wang, Liangming Pan, Xiaojun Wan, William Yang Wang, 2025https://scholar.google.com/scholar?q=G%C3%B6del+Agent%3A+A+Self-Referential+Agent+Framework+for+Recursive+Self-Improvement9. Darwin Gödel Machine — Jenny Zhang et al., 2025https://scholar.google.com/scholar?q=Darwin+G%C3%B6del+Machine10. Self-Referential AI Systems — Louis Kirsch and Jürgen Schmidhuber, 2022https://scholar.google.com/scholar?q=Self-Referential+AI+Systems11. Recursive Self-Improvement Can Be Bounded or Self-Accelerating — Lu et al., 2023https://scholar.google.com/scholar?q=Recursive+Self-Improvement+Can+Be+Bounded+or+Self-Accelerating12. Polyglot — Gauthier, 2024https://scholar.google.com/scholar?q=Polyglot13. Paper Review Benchmark — Zhao et al., 2026https://scholar.google.com/scholar?q=Paper+Review+Benchmark14. Genesis — Genesis authors, 2024https://scholar.google.com/scholar?q=Genesis15. Learning How to Remember: A Meta-Cognitive Management Method for Structured and Transferable Agent Memory — approx. unknown from snippet, recenthttps://scholar.google.com/scholar?q=Learning+How+to+Remember%3A+A+Meta-Cognitive+Management+Method+for+Structured+and+Transferable+Agent+Memory16. Memory in the Age of AI Agents — approx. unknown from snippet, recenthttps://scholar.google.com/scholar?q=Memory+in+the+Age+of+AI+Agents17. Discovering Hierarchical Software Engineering Agents via Bandit Optimization — approx. unknown from snippet, recenthttps://scholar.google.com/scholar?q=Discovering+Hierarchical+Software+Engineering+Agents+via+Bandit+Optimization18. BOAD: Discovering Hierarchical Software Engineering Agents via Bandit Optimization — approx. unknown from snippet, recenthttps://scholar.google.com/scholar?q=BOAD%3A+Discovering+Hierarchical+Software+Engineering+Agents+via+Bandit+Optimization19. Automated Design of Agentic Systems — approx. unknown from snippet, recenthttps://scholar.google.com/scholar?q=Automated+Design+of+Agentic+Systems20. AI Post Transformers: Experiential Reinforcement Learning: Internalizing Reflection for Better Policy Training — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/experiential-reinforcement-learning-internalizing-reflection-for-better-policy-t/21. AI Post Transformers: LLM Agents Reason About Code Without Running It — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-15-llm-agents-reason-about-code-without-run-2a1876.mp322. AI Post Transformers: Mem0: Scalable Long-Term Memory for AI Agents — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/mem0-scalable-long-term-memory-for-ai-agents/Interactive Visualization: HyperAgents and Metacognitive Self-Improvement24 min
- From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented GenerationThis episode explores a systems paper on speeding up retrieval-augmented generation by reusing transformer KV cache state more intelligently, instead of recomputing long retrieved prompts from scratch on every request. It explains why RAG often improves grounding yet suffers from high time-to-first-token, especially when multiple retrieved chunks must be prefetched and encoded together. The discussion focuses on the paper’s central argument that naive chunk-level cache reuse breaks important cross-chunk interactions, and that the proposed FusionRAG Cache tries to preserve quality through offline chunk enrichment and selective online recomputation. A listener would find it interesting because it connects familiar RAG concepts to the real serving bottlenecks that determine whether enterprise assistants feel practical or painfully slow.Sources:1. From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation — Jiahao Wang, Weiyu Xie, Mingxing Zhang, Boxing Zhang, Jianwei Dong, Yuening Zhu, Chen Lin, Jinqi Tang, Yaochen Han, Zhiyuan Ai, Xianglin Chen, Yongwei Wu, Congfeng Jiang, 2026http://arxiv.org/abs/2601.12904v12. REALM: Retrieval-Augmented Language Model Pre-Training — Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang, 2020https://scholar.google.com/scholar?q=REALM%3A+Retrieval-Augmented+Language+Model+Pre-Training3. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, Douwe Kiela, 2020https://scholar.google.com/scholar?q=Retrieval-Augmented+Generation+for+Knowledge-Intensive+NLP+Tasks4. Few-shot Learning with Retrieval Augmented Language Models — Gautier Izacard, Edouard Grave, Patrick Lewis, Benjamin Chintala, Tim Rocktaschel, Fabio Petroni, 2022https://scholar.google.com/scholar?q=Few-shot+Learning+with+Retrieval+Augmented+Language+Models5. Retrieval-Augmented Generation for Large Language Models: A Survey — Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, Haofen Wang, 2023https://scholar.google.com/scholar?q=Retrieval-Augmented+Generation+for+Large+Language+Models%3A+A+Survey6. APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding — Xinyu Yang, Tianqi Chen, Beidi Chen, 2025https://scholar.google.com/scholar?q=APE%3A+Faster+and+Longer+Context-Augmented+Generation+via+Adaptive+Parallel+Encoding7. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang, 2024https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion8. Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation — Shubham Agarwal, Sai Narayan Sundaresan, Subrata Mitra, Deb Mahapatra, Tong Yu, Shiv Saini, 2025https://scholar.google.com/scholar?q=Cache-Craft%3A+Managing+Chunk-Caches+for+Efficient+Retrieval-Augmented+Generation9. From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation — Jiahao Wang, Weiyu Xie, Mingxing Zhang, Boxing Zhang, Jianwei Dong, Yuening Zhu, Chen Lin, Jinqi Tang, Yaochen Han, Zhiyuan Ai, Xianglin Chen, Yongwei Wu, Congfeng Jiang, 2026https://scholar.google.com/scholar?q=From+Prefix+Cache+to+Fusion+RAG+Cache%3A+Accelerating+LLM+Inference+in+Retrieval-Augmented+Generation10. HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse — authors not reliably recoverable from the browsed snippets, 2025https://scholar.google.com/scholar?q=HyperRAG%3A+Enhancing+Quality-Efficiency+Tradeoffs+in+Retrieval-Augmented+Generation+with+Reranker+KV-Cache+Reuse11. Kvlink: Accelerating Large Language Models via Efficient KV Cache Reuse — approx. recent systems/LLM inference authors, 2025/2026https://scholar.google.com/scholar?q=Kvlink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse12. Sparse Attention across Multiple-context KV Cache — approx. recent long-context/RAG inference authors, 2025/2026https://scholar.google.com/scholar?q=Sparse+Attention+across+Multiple-context+KV+Cache13. CacheClip: Accelerating RAG with Effective KV Cache Reuse — approx. recent RAG systems authors, 2025/2026https://scholar.google.com/scholar?q=CacheClip%3A+Accelerating+RAG+with+Effective+KV+Cache+Reuse14. CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference — approx. recent long-context inference authors, 2025/2026https://scholar.google.com/scholar?q=CHESS%3A+Context-aware+Hierarchical+Efficient+Semantic+Selection+for+Long-Context+LLM+Inference15. BudgetMem: Learning Selective Memory Policies for Cost-Efficient Long-Context Processing in Language Models — approx. recent memory-policy learning authors, 2025/2026https://scholar.google.com/scholar?q=BudgetMem%3A+Learning+Selective+Memory+Policies+for+Cost-Efficient+Long-Context+Processing+in+Language+Models16. TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval — approx. recent RAG systems authors, 2025/2026https://scholar.google.com/scholar?q=TeleRAG%3A+Efficient+Retrieval-Augmented+Generation+Inference+with+Lookahead+Retrieval17. Understanding and Optimizing Multi-Stage AI Inference Pipelines — approx. recent systems authors, 2025/2026https://scholar.google.com/scholar?q=Understanding+and+Optimizing+Multi-Stage+AI+Inference+Pipelines18. AI Post Transformers: Episode: LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-21-lookaheadkv-fast-and-accurate-kv-c9d436.mp319. AI Post Transformers: CacheSlide: Position-Aware KV Cache Reuse for Agent LLMs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-16-cacheslide-position-aware-kv-cache-reuse-cd59c7.mp320. AI Post Transformers: SolidAttention: Co-Designing Sparse Attention and SSD I/O — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-18-solidattention-co-designing-sparse-atten-5a8622.mp3Interactive Visualization: Episode: From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation0 min
- LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without GenerationThis episode explores how KV cache eviction shapes the speed and usability of long-context language models, focusing on the March 2026 paper LookaheadKV and the broader problem of managing transformer memory under tight GPU budgets. It explains why KV caches are essential for autoregressive decoding, why their linear growth becomes a major inference bottleneck, and how eviction policies differ from related approaches such as cache compression. The discussion highlights the paper’s central argument: future-aware eviction can outperform simple recency-based heuristics, but only if it avoids the heavy latency costs that make some draft-generation methods impractical. A listener would find it interesting for its clear systems-level view of transformer inference, especially the tradeoff between smarter cache decisions and time-to-first-token in real production settings.Sources:1. LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation — Jinwoo Ahn, Ingyu Seong, Akhil Kedia, Junhan Kim, Hyemi Jang, Kangwook Lee, Yongkweon Jeon, 2026http://arxiv.org/abs/2603.10899v12. SnapKV: LLM Knows What You Are Looking for before Generation — Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, Deming Chen, 2024https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+Are+Looking+for+before+Generation3. Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query — Yixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Yang Xu, Qingfu Zhu, Wanxiang Che, 2025https://scholar.google.com/scholar?q=Lookahead+Q-Cache%3A+Achieving+More+Consistent+KV+Cache+Eviction+via+Pseudo+Query4. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction — Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee, Sangdoo Yun, Hyun Oh Song, 2025https://scholar.google.com/scholar?q=KVzip%3A+Query-Agnostic+KV+Cache+Compression+with+Context+Reconstruction5. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling — Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, Wen Xiao, 2024https://scholar.google.com/scholar?q=PyramidKV%3A+Dynamic+KV+Cache+Compression+based+on+Pyramidal+Information+Funneling6. HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse — approx. recent RAG systems authors, 2025/2026https://scholar.google.com/scholar?q=HyperRAG%3A+Enhancing+Quality-Efficiency+Tradeoffs+in+Retrieval-Augmented+Generation+with+Reranker+KV-Cache+Reuse7. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse — approx. recent LLM inference authors, 2025/2026https://scholar.google.com/scholar?q=KVLink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse8. ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation — approx. recent RAG/inference authors, 2025/2026https://scholar.google.com/scholar?q=ProphetKV%3A+User-Query-Driven+Selective+Recomputation+for+Efficient+KV+Cache+Reuse+in+Retrieval-Augmented+Generation9. LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference — approx. recent long-context inference authors, 2024/2025https://scholar.google.com/scholar?q=LazyLLM%3A+Dynamic+Token+Pruning+for+Efficient+Long+Context+LLM+Inference10. TokenButler: Token Importance Is Predictable — approx. recent token pruning authors, 2025https://scholar.google.com/scholar?q=TokenButler%3A+Token+Importance+Is+Predictable11. LongHeads: Multi-Head Attention Is Secretly a Long Context Processor — approx. recent mechanistic interpretability / long-context authors, 2025https://scholar.google.com/scholar?q=LongHeads%3A+Multi-Head+Attention+Is+Secretly+a+Long+Context+Processor12. How Transformers Implement Induction Heads: Approximation and Optimization Analysis — approx. mechanistic interpretability authors, 2024/2025https://scholar.google.com/scholar?q=How+Transformers+Implement+Induction+Heads%3A+Approximation+and+Optimization+Analysis13. AI Post Transformers: LAQ for Smarter KV Cache Eviction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-23-laq-for-smarter-kv-cache-eviction-3ea2b8.mp314. AI Post Transformers: Lookahead Q-Cache for Consistent KV Eviction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-lookahead-q-cache-for-consistent-kv-evic-d97b09.mp315. AI Post Transformers: Memory Traffic Saturation in Transformer Decode — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-20-memory-traffic-saturation-in-transformer-cd4961.mp316. AI Post Transformers: SolidAttention: Co-Designing Sparse Attention and SSD I/O — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-18-solidattention-co-designing-sparse-atten-5a8622.mp3Interactive Visualization: Episode: LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation0 min
- LeWorldModel: Stable Joint-Embedding World Models from PixelsIn this episode, the hosts examine LeWorldModel, a 2026 paper from researchers across Mila, Universite de Montreal, NYU, Samsung SAIL, and Brown that asks whether a joint-embedding predictive architecture can finally be trained end to end from raw pixels without the usual stack of stabilizers. The discussion situates the work in the broader world-model lineage from Ha and Schmidhuber through Dreamer, and in the JEPA program associated with Yann LeCun’s predictive-learning agenda. The paper’s core claim is unusually narrow and concrete: a small model, around 15 million parameters, can learn action-conditioned dynamics directly from images using just next-embedding prediction plus a Gaussian latent regularizer called SIGReg, avoiding EMA teachers, pretrained encoders, reconstruction losses, and other auxiliary machinery that many related systems rely on.The conversation focuses on why that claim matters. The hosts explain that JEPA-style methods are attractive because they predict semantic embeddings rather than reconstructing every pixel, but they have been plagued by representation collapse and fragile training recipes. Most of the technical attention therefore goes to how LeWorldModel tries to keep the latent space informative while staying simple enough to train jointly on a single GPU in a few hours. They walk through the paper’s framing around offline control and latent-space planning, where forecasting compact future states can make imagined rollouts cheap. They also discuss the project-page claim that LeWorldModel can plan up to roughly 48 times faster than DINO-WM because each frame is compressed to a single 192-dimensional token, while noting that this is part of the system’s pitch and should be separated from broader claims about downstream capability.The episode also digs into the evidence and the limits. The hosts cover the benchmark results across Two-Room, Reacher, Push-T, and OGBench-Cube, where LeWorldModel appears competitive overall, broadly stronger than PLDM, and better than DINO-WM on Push-T and Reacher, while DINO-WM still looks stronger on the more visually complex OGBench-Cube setting, likely because richer pretrained visual priors still help there. They also discuss the paper and project-page attempts to show “physical understanding” through latent probing, decoded latent visualizations, and surprise-style tests for implausible events. Throughout, the description stays skeptical about the gap between paper evidence and project-page marketing: the system looks like a real simplification of JEPA world modeling, but not yet a final verdict that minimal end-to-end predictive learning has solved robust visual control.Sources:1. LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels — Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, Randall Balestriero, 2026http://arxiv.org/abs/2603.19312v12. World Models — David Ha, Jürgen Schmidhuber, 2018https://scholar.google.com/scholar?q=World+Models3. Learning Latent Dynamics for Planning from Pixels — Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, James Davidson, 2018https://scholar.google.com/scholar?q=Learning+Latent+Dynamics+for+Planning+from+Pixels4. Dream to Control: Learning Behaviors by Latent Imagination — Danijar Hafner, Timothy Lillicrap, Jimmy Ba, Mohammad Norouzi, 2019https://scholar.google.com/scholar?q=Dream+to+Control%3A+Learning+Behaviors+by+Latent+Imagination5. TD-MPC2: Scalable, Robust World Models for Continuous Control — Nicklas Hansen, Hao Su, Xiaolong Wang, 2024https://scholar.google.com/scholar?q=TD-MPC2%3A+Scalable%2C+Robust+World+Models+for+Continuous+Control6. A Path Towards Autonomous Machine Intelligence — Yann LeCun, 2022https://scholar.google.com/scholar?q=A+Path+Towards+Autonomous+Machine+Intelligence7. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture — Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, Nicolas Ballas, 2023https://scholar.google.com/scholar?q=Self-Supervised+Learning+from+Images+with+a+Joint-Embedding+Predictive+Architecture8. Revisiting Feature Prediction for Learning Visual Representations from Video — Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, Nicolas Ballas, 2024https://scholar.google.com/scholar?q=Revisiting+Feature+Prediction+for+Learning+Visual+Representations+from+Video9. LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels — Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, Randall Balestriero, 2026https://scholar.google.com/scholar?q=LeWorldModel%3A+Stable+End-to-End+Joint-Embedding+Predictive+Architecture+from+Pixels10. Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning — Jean-Bastien Grill, Florian Strub, Florent Altche, Corentin Tallec and others, 2020https://scholar.google.com/scholar?q=Bootstrap+Your+Own+Latent%3A+A+New+Approach+to+Self-Supervised+Learning11. Exploring Simple Siamese Representation Learning — Xinlei Chen, Kaiming He, 2021https://scholar.google.com/scholar?q=Exploring+Simple+Siamese+Representation+Learning12. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning — Adrien Bardes, Jean Ponce, Yann LeCun, 2021https://scholar.google.com/scholar?q=VICReg%3A+Variance-Invariance-Covariance+Regularization+for+Self-Supervised+Learning13. Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models — Vlad Sobal, Wancong Zhang, Kyunghyun Cho, Randall Balestriero, Tim G. J. Rudner, Yann LeCun, 2025https://scholar.google.com/scholar?q=Learning+from+Reward-Free+Offline+Data%3A+A+Case+for+Planning+with+Latent+Dynamics+Models14. Human-level Control through Deep Reinforcement Learning — Volodymyr Mnih, Koray Kavukcuoglu, David Silver and others, 2015https://scholar.google.com/scholar?q=Human-level+Control+through+Deep+Reinforcement+Learning15. Mastering Diverse Domains through World Models — Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, Timothy Lillicrap, 2023https://scholar.google.com/scholar?q=Mastering+Diverse+Domains+through+World+Models16. TD-MPC: Learning to Plan in Latent Space for Visual Control — Nicklas Hansen, Xiaolong Wang, Hao Su, 2022https://scholar.google.com/scholar?q=TD-MPC%3A+Learning+to+Plan+in+Latent+Space+for+Visual+Control17. PlaNet: Learning Latent Dynamics for Planning from Pixels — Danijar Hafner, Timothy Lillicrap, Jimmy Ba, Mohammad Norouzi, 2019https://scholar.google.com/scholar?q=PlaNet%3A+Learning+Latent+Dynamics+for+Planning+from+Pixels18. Rethinking JEPA: Compute-Efficient Video Self-Supervised Learning with Frozen Teachers — approx. unknown from snippet, 2025-2026https://scholar.google.com/scholar?q=Rethinking+JEPA%3A+Compute-Efficient+Video+Self-Supervised+Learning+with+Frozen+Teachers19. Efficient reinforcement learning through adaptively pretrained visual encoder — approx. unknown from snippet, 2023-2026https://scholar.google.com/scholar?q=Efficient+reinforcement+learning+through+adaptively+pretrained+visual+encoder20. Adaworld: Learning adaptable world models with latent actions — approx. unknown from snippet, 2023-2026https://scholar.google.com/scholar?q=Adaworld%3A+Learning+adaptable+world+models+with+latent+actions21. AI Post Transformers: Unified Latents (UL): How to train your latents — Hal Turing & Dr. Ada Shannon, Sat,https://podcast.do-not-panic.com/episodes/unified-latents-ul-how-to-train-your-latents/22. AI Post Transformers: Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning — Hal Turing & Dr. Ada Shannon, Fri,https://podcast.do-not-panic.com/episodes/contrastive-behavioral-similarity-embeddings-for-generalization-in-reinforcement/Interactive Visualization: LeWorldModel: Stable Joint-Embedding World Models from Pixels0 min
- Lookahead Q-Cache for Consistent KV EvictionThis episode examines Lookahead Q-Cache as a very specific kind of inference optimization: a decode-stage KV-cache eviction method for long-context serving. The discussion explains the paper’s core claim that prefill-time attention is a weak proxy for what will matter once generation actually begins, because decode-time queries are conditioned on the answer the model is actively writing rather than on the prompt alone. That is the real novelty here. Static selection methods such as SnapKV and related heavy-hitter or cumulative-attention schemes mostly infer importance from prompt-side attention patterns, often using a suffix window as a stand-in for future need. Lookahead Q-Cache instead uses a pseudo-query to approximate upcoming decode queries, making eviction more dynamic and more aligned to generation. The hosts are explicit that this is mostly a decode-only idea, not a general cure for transformer inference cost, and they keep returning to that point so the scope is not overstated.The conversation places the paper inside the broader acceleration landscape rather than treating it as a standalone breakthrough. Speculative decoding, Medusa-style multi-head prediction, and layered drafting ideas such as inference blending or Matryoshka-like speculative schemes all attack a different bottleneck: they try to reduce the cost of producing future tokens by drafting and verifying them more efficiently. Lookahead Q-Cache attacks the memory and attention burden of carrying long prefixes during decode. Those are not the same problem, which means they are not simple substitutes and can in principle be complementary in one serving stack. The episode also contrasts this test-time cache-management line with architecture-level efficiency work such as grouped-query attention, Nemotron 3 style system-model co-design, and Kimi-like efficient long-context efforts, where the gains often come from changing the model or attention structure rather than making smarter runtime eviction decisions.The tone stays skeptical about deployment significance. The hosts ask the hard scaling question directly: does smarter KV eviction materially change long-context serving economics, or does it mainly deliver narrower decode wins inside a larger bottleneck stack that still includes prefill cost, bandwidth pressure, scheduler behavior, batching constraints, quantization tradeoffs, and model architecture limits? They argue that benchmark improvements in eviction consistency are interesting, but the real bar is whether operators would trust aggressive dynamic cache pruning in production compared with more predictable approaches like GQA, FlashAttention, quantization, or speculative decode pipelines already discussed elsewhere on the podcast. The result is a grounded episode about what is genuinely new in Lookahead Q-Cache, where it fits, and why decode-specific cache tricks should not be confused with a full solution to long-context serving.Sources:1. Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query — Yixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Yang Xu, Qingfu Zhu, Wanxiang Che, 2025http://arxiv.org/abs/2505.203342. SnapKV: LLM Knows What You are Looking for Before Generation — Zhenyu Li et al., 2024https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang et al., 2023https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models4. Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time — Zhenyu Liu et al., 2023https://scholar.google.com/scholar?q=Scissorhands%3A+Exploiting+the+Persistence+of+Importance+Hypothesis+for+LLM+KV+Cache+Compression+at+Test+Time5. Fast and Accurate Transformer Decoding via Dynamic Compression of KV Cache — likely Tang et al., 2024https://scholar.google.com/scholar?q=Fast+and+Accurate+Transformer+Decoding+via+Dynamic+Compression+of+KV+Cache6. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao et al., 2022https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness7. FlashAttention-2 or subsequent FlashAttention work — Tri Dao, 2024https://scholar.google.com/scholar?q=FlashAttention-2+or+subsequent+FlashAttention+work8. RazorAttention: Efficient KV Cache Compression through Retrieval Heads — not recovered from snippet, 2024-2025https://scholar.google.com/scholar?q=RazorAttention%3A+Efficient+KV+Cache+Compression+through+Retrieval+Heads9. FastKV: KV Cache Compression for Fast Long-Context Processing with Token-Selective Propagation — not recovered from snippet, 2024-2025https://scholar.google.com/scholar?q=FastKV%3A+KV+Cache+Compression+for+Fast+Long-Context+Processing+with+Token-Selective+Propagation10. Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity — not recovered from snippet, 2024-2025https://scholar.google.com/scholar?q=Compressing+KV+Cache+for+Long-Context+LLM+Inference+with+Inter-Layer+Attention+Similarity11. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse — not recovered from snippet, 2024-2025https://scholar.google.com/scholar?q=KVLink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse12. DepCache: A KV Cache Management Framework for GraphRAG with Dependency Attention — not recovered from snippet, 2024-2025https://scholar.google.com/scholar?q=DepCache%3A+A+KV+Cache+Management+Framework+for+GraphRAG+with+Dependency+Attention13. End-to-End Acceleration of Generative Models with Runtime Regularized KV Cache Management — not recovered from snippet, 2024-2025https://scholar.google.com/scholar?q=End-to-End+Acceleration+of+Generative+Models+with+Runtime+Regularized+KV+Cache+Management14. AI Post Transformers: LAQ for Smarter KV Cache Eviction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-23-laq-for-smarter-kv-cache-eviction-3ea2b8.mp315. AI Post Transformers: LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-21-lookaheadkv-fast-and-accurate-kv-c9d436.mp316. AI Post Transformers: Quest: Query-Aware Sparsity for Efficient LLM Inference — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/quest-query-aware-sparsity-for-efficient-llm-inference/17. AI Post Transformers: Hyper-Scaling LLM Inference with KV Cache Compression — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/hyper-scaling-llm-inference-with-kv-cache-compression/18. AI Post Transformers: Memory Traffic Saturation in Transformer Decode — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-20-memory-traffic-saturation-in-transformer-cd4961.mp3Interactive Visualization: Lookahead Q-Cache for Consistent KV Eviction24 min
- Speculative Speculative DecodingThis episode explores Speculative Speculative Decoding, a technique for reducing LLM inference latency by overlapping drafting and verification more aggressively than standard speculative decoding. It explains how the method predicts likely verification outcomes in advance so the draft model can prepare multiple next-step continuations while the target model is still checking the current block, with the target model still preserving the final output. The discussion focuses on the Saguaro algorithm, the distinction between true parallel generation and latency-hiding around autoregressive dependencies, and the practical tradeoff between useful overlap and wasted draft-side computation. A listener would find it interesting for its clear look at where modern inference systems still lose time and how smarter scheduling, rather than changing model semantics, can unlock additional speed.Sources:1. Speculative Speculative Decoding — Tanishq Kumar, Tri Dao, Avner May, 2026http://arxiv.org/abs/2603.032512. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding3. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024https://scholar.google.com/scholar?q=EAGLE%3A+Speculative+Sampling+Requires+Rethinking+Feature+Uncertainty4. SpecInfer: Accelerating Generative LLM Serving with Speculative Inference and Token Tree Verification — Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, Zhihao Jia, 2024https://scholar.google.com/scholar?q=SpecInfer%3A+Accelerating+Generative+LLM+Serving+with+Speculative+Inference+and+Token+Tree+Verification5. PEARL: Parallel Speculative Decoding with Adaptive Draft Length — Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, Winston Hu, Xiao Sun, 2025https://scholar.google.com/scholar?q=PEARL%3A+Parallel+Speculative+Decoding+with+Adaptive+Draft+Length6. AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration — Bradley McDanel, 2025https://scholar.google.com/scholar?q=AMUSD%3A+Asynchronous+Multi-Device+Speculative+Decoding+for+LLM+Acceleration7. Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding — Jun Zhang et al., 2023https://scholar.google.com/scholar?q=Draft+%26+Verify%3A+Lossless+Large+Language+Model+Acceleration+via+Self-Speculative+Decoding8. SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration — Heming Xia et al., 2024https://scholar.google.com/scholar?q=SWIFT%3A+On-the-Fly+Self-Speculative+Decoding+for+LLM+Inference+Acceleration9. SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths — Kaixuan Huang, Xudong Guo, Mengdi Wang, 2024https://scholar.google.com/scholar?q=SpecDec%2B%2B%3A+Boosting+Speculative+Decoding+via+Adaptive+Candidate+Lengths10. Accelerating Transformer Inference for Translation via Parallel Decoding — Andrea Santilli et al., 2023https://scholar.google.com/scholar?q=Accelerating+Transformer+Inference+for+Translation+via+Parallel+Decoding11. Self-Speculative Decoding for LLM-based ASR with CTC Encoder Drafts — George Saon et al., 2026https://scholar.google.com/scholar?q=Self-Speculative+Decoding+for+LLM-based+ASR+with+CTC+Encoder+Drafts12. AI Post Transformers: Apple's Speculative Streaming: Fast LLM Inference without Auxiliary Models — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/apples-speculative-streaming-fast-llm-inference-without-auxiliary-models/13. AI Post Transformers: Adaptive Control for Batched Speculative Decoding in LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/adaptive-control-for-batched-speculative-decoding-in-llm-serving/14. AI Post Transformers: FastGRPO: Concurrency-Aware Speculative Decoding for Policy Optimization — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/fastgrpo-concurrency-aware-speculative-decoding-for-policy-optimization/Interactive Visualization: Episode: Speculative Speculative Decoding0 min
- Jet-Nemotron and PostNAS for Faster Long ContextHal Turing and Dr. Ada Shannon examine Jet-Nemotron as a serious but narrow attempt to retrofit long-context efficiency into a pretrained dense Transformer rather than as a clean-sheet architecture revolution. They focus on NVIDIA’s PostNAS pipeline, which freezes the MLP pathway, treats attention layers as the remodel zone, and searches where full attention is still worth paying for versus where cheaper JetBlocks can replace it. The discussion keeps returning to the real question behind the paper’s marketing: whether this is evidence that linear-attention-style hybrids can genuinely change inference scaling and KV-cache pressure, or whether it is a carefully engineered optimization for a constrained deployment target that inherits most of its intelligence from the original dense model.The episode makes the contrast with Nemotron 3 explicit. In the earlier Nemotron 3 story, the architectural pitch was a broader hybrid stack built around the interplay of dense Transformer machinery, mixture-of-experts routing, and state-space or recurrent-style efficiency ideas. Jet-Nemotron is different in both method and claim: it is not mainly about MoE capacity or an SSM-flavored redesign, but about post-training surgery on the attention stack itself, with layer placement search deciding where exact global lookup remains indispensable and where linear-style blocks can take over. That makes Jet-Nemotron feel less like a new foundation model family and more like a practical conversion recipe, which the hosts treat as both the paper’s most credible contribution and its main limitation.They also place Jet-Nemotron directly against Kimi Linear and the broader efficient-LLM landscape. Both papers take linear attention seriously as a way to attack long-context serving bottlenecks, but the comparison here is not flattering by default: Kimi Linear looked more like a direct argument for a new sequence-mixing primitive, while Jet-Nemotron looks more convincing as an engineering workflow for salvaging pretrained dense checkpoints without retraining everything from scratch. The hosts parse where the similarities end, where the quality-preservation story still depends on keeping some full-attention layers alive, and why that matters for judging whether linear attention is becoming a real architectural shift or remains a selective compromise that works best when a dense Transformer still anchors the system.Sources:1. Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search — Yuxian Gu, Qinghao Hu, Shang Yang, Haocheng Xi, Junyu Chen, Song Han, Han Cai, 2025http://arxiv.org/abs/2508.158842. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models — Soham De, Samuel L. Smith, Aleksandar Botev, Albert Gu, Caglar Gulcehre and collaborators, 2024https://scholar.google.com/scholar?q=Griffin%3A+Mixing+Gated+Linear+Recurrences+with+Local+Attention+for+Efficient+Language+Models3. Zamba: A Compact 7B SSM Hybrid Model — Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Beren Millidge and collaborators, 2024https://scholar.google.com/scholar?q=Zamba%3A+A+Compact+7B+SSM+Hybrid+Model4. Hymba: A Hybrid-head Architecture for Small Language Models — Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Pavlo Molchanov and collaborators, 2025https://scholar.google.com/scholar?q=Hymba%3A+A+Hybrid-head+Architecture+for+Small+Language+Models5. Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models — NVIDIA et al. (including Aaron Blakeman, Song Han, Jan Kautz and collaborators), 2025https://scholar.google.com/scholar?q=Nemotron-H%3A+A+Family+of+Accurate+and+Efficient+Hybrid+Mamba-Transformer+Models6. Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search — Yuxian Gu, Qinghao Hu, Shang Yang, Haocheng Xi, Junyu Chen, Song Han, Han Cai, 2025https://scholar.google.com/scholar?q=Jet-Nemotron%3A+Efficient+Language+Model+with+Post+Neural+Architecture+Search7. Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Construction — Xiaojie Xia, Huigang Zhang, Chaoliang Zhong, Jun Sun, Yusuke Oishi, 2026https://scholar.google.com/scholar?q=Distill-then-Replace%3A+Efficient+Task-Specific+Hybrid+Attention+Model+Construction8. The Zamba2 Suite: Technical Report — Paolo Glorioso, Quentin Anthony, Yury Tokpanov, Anna Golubeva, Vasudev Shyam, James Whittington, Jonathan Pilault, Beren Millidge, 2024https://scholar.google.com/scholar?q=The+Zamba2+Suite%3A+Technical+Report9. RecurrentGemma: Moving Past Transformers for Efficient Open Language Models — Aleksandar Botev, Soham De, Samuel L. Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu, Pier Giuseppe Sessa, Robert Dadashi, Leonhard Hussenot, Johan Ferret, Sertan Girgin, Olivier Bachem, Alek Andreev, Kathleen Kenealy, Thomas Mesnard, Cassidy Hardin, Surya Bhupatiraju, and others, 2024https://scholar.google.com/scholar?q=RecurrentGemma%3A+Moving+Past+Transformers+for+Efficient+Open+Language+Models10. Zoology: Measuring and Improving Recall in Efficient Language Models — Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, Christopher Re, 2023https://scholar.google.com/scholar?q=Zoology%3A+Measuring+and+Improving+Recall+in+Efficient+Language+Models11. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — Tri Dao, Albert Gu, 2024https://scholar.google.com/scholar?q=Transformers+are+SSMs%3A+Generalized+Models+and+Efficient+Algorithms+Through+Structured+State+Space+Duality12. Eigen Attention: Attention in Low-Rank Space for KV Cache Compression — approx. recent LLM systems/efficient inference authors, 2024/2025https://scholar.google.com/scholar?q=Eigen+Attention%3A+Attention+in+Low-Rank+Space+for+KV+Cache+Compression13. ClusterAttn: KV Cache Compression under Intrinsic Attention Clustering — approx. recent efficient inference authors, 2024/2025https://scholar.google.com/scholar?q=ClusterAttn%3A+KV+Cache+Compression+under+Intrinsic+Attention+Clustering14. Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution — approx. recent efficient inference authors, 2024/2025https://scholar.google.com/scholar?q=Expected+Attention%3A+KV+Cache+Compression+by+Estimating+Attention+from+Future+Queries+Distribution15. Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning — approx. recent hybrid-attention authors, 2024/2025https://scholar.google.com/scholar?q=Every+Attention+Matters%3A+An+Efficient+Hybrid+Architecture+for+Long-Context+Reasoning16. Scaling Linear Attention with Sparse State Expansion — approx. recent linear-attention scaling authors, 2024/2025https://scholar.google.com/scholar?q=Scaling+Linear+Attention+with+Sparse+State+Expansion17. AI Post Transformers: Jet-Nemotron and Post-Pretraining Model Acceleration — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-24-jet-nemotron-and-post-pretraining-model-4ba5cb.mp318. AI Post Transformers: Kimi Linear: Efficient Expressive Attention Architecture — Hal Turing & Dr. Ada Shannon, Sun,https://podcast.do-not-panic.com/episodes/kimi-linear-efficient-expressive-attention-architecture/19. AI Post Transformers: Dr.LLM: Dynamic Layer Routing in LLMs — Hal Turing & Dr. Ada Shannon, Wed,https://podcast.do-not-panic.com/episodes/drllm-dynamic-layer-routing-in-llms/20. AI Post Transformers: Speed Always Wins: Efficient Large Language Model Architectures — Hal Turing & Dr. Ada Shannon, Wed,https://podcast.do-not-panic.com/episodes/speed-always-wins-efficient-large-language-model-architectures/21. AI Post Transformers: LAQ for Smarter KV Cache Eviction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-23-laq-for-smarter-kv-cache-eviction-3ea2b8.mp3Interactive Visualization: Jet-Nemotron and PostNAS for Faster Long Context13 min
- Language Models are Injective and Hence InvertibleThis episode explores a 2025 paper arguing that decoder-only language models are generically injective on discrete token sequences, meaning their hidden representations can in principle preserve enough information to recover the exact original prompt. It walks through what injectivity and invertibility mean in this setting, why that challenges the common intuition that transformer representations behave like lossy semantic summaries, and how the paper distinguishes this claim from stronger notions of full bijectivity over continuous spaces. The discussion also connects the result to related ideas from normalizing flows, reversible networks, and mechanistic interpretability, while introducing the paper’s constructive recovery method, SipIt. Listeners would find it interesting because the result has unusually sharp implications for both interpretability and privacy: hidden states may be far less abstracted from raw input text than many researchers assume.Sources:1. Language Models are Injective and Hence Invertible — Giorgos Nikolaou, Tommaso Mencattini, Donato Crisostomi, Andrea Santilli, Yannis Panagakis, Emanuele Rodolà, 2025http://arxiv.org/abs/2510.155112. Normalizing Flows for Probabilistic Modeling and Inference — George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, Balaji Lakshminarayanan, 2021https://scholar.google.com/scholar?q=Normalizing+Flows+for+Probabilistic+Modeling+and+Inference3. The Reversible Residual Network: Backpropagation Without Storing Activations — Aidan N. Gomez, Mengye Ren, Raquel Urtasun, Roger B. Grosse, 2017https://scholar.google.com/scholar?q=The+Reversible+Residual+Network%3A+Backpropagation+Without+Storing+Activations4. Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence — Haoran Li, Mingshi Xu, Yangqiu Song, 2023https://scholar.google.com/scholar?q=Sentence+Embedding+Leaks+More+Information+than+You+Expect%3A+Generative+Embedding+Inversion+Attack+to+Recover+the+Whole+Sentence5. Language Models are Injective and Hence Invertible — Giorgos Nikolaou, Tommaso Mencattini, Donato Crisostomi, Andrea Santilli, Yannis Panagakis, Emanuele Rodola, 2025https://scholar.google.com/scholar?q=Language+Models+are+Injective+and+Hence+Invertible6. The non-linear representation dilemma: Is causal abstraction enough for mechanistic interpretability? — Denis Sutter, Julian Minder, Thomas Hofmann, Tiago Pimentel, 2025https://scholar.google.com/scholar?q=The+non-linear+representation+dilemma%3A+Is+causal+abstraction+enough+for+mechanistic+interpretability%3F7. On surjectivity of neural networks: Can you elicit any behavior from your model? — Haozhe Jiang, Nika Haghtalab, 2025https://scholar.google.com/scholar?q=On+surjectivity+of+neural+networks%3A+Can+you+elicit+any+behavior+from+your+model%3F8. Attention is not all you need: Pure attention loses rank doubly exponentially with depth — Yihe Dong, Jean-Baptiste Cordonnier, Andreas Loukas, 2021https://scholar.google.com/scholar?q=Attention+is+not+all+you+need%3A+Pure+attention+loses+rank+doubly+exponentially+with+depth9. Text embeddings reveal (almost) as much as text — John X. Morris, Volodymyr Kuleshov, Vitaly Shmatikov, Alexander M. Rush, 2023https://scholar.google.com/scholar?q=Text+embeddings+reveal+%28almost%29+as+much+as+text10. Language model inversion — John X. Morris, Wenting Zhao, Justin T. Chiu, Vitaly Shmatikov, Alexander M. Rush, 2023https://scholar.google.com/scholar?q=Language+model+inversion11. Better language model inversion by compactly representing next-token distributions — Murtaza Nazir, Matthew Finlayson, John X. Morris, Xiang Ren, Swabha Swayamdipta, 2025https://scholar.google.com/scholar?q=Better+language+model+inversion+by+compactly+representing+next-token+distributions12. Stabilizing Transformer Training by Preventing Attention Entropy Collapse — Shuangfei Zhai et al., 2023https://scholar.google.com/scholar?q=Stabilizing+Transformer+Training+by+Preventing+Attention+Entropy+Collapse13. From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics — Zheng-An Chen and Tao Luo, 2025https://scholar.google.com/scholar?q=From+Condensation+to+Rank+Collapse%3A+A+Two-Stage+Analysis+of+Transformer+Training+Dynamics14. Understanding and Minimising Outlier Features in Transformer Training — Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, Thomas Hofmann, 2024https://scholar.google.com/scholar?q=Understanding+and+Minimising+Outlier+Features+in+Transformer+Training15. Measuring In-Context Computation Complexity via Hidden State Prediction — Vincent Herrmann, Robert Csordas, Jurgen Schmidhuber, 2025https://scholar.google.com/scholar?q=Measuring+In-Context+Computation+Complexity+via+Hidden+State+Prediction16. Transformers without Normalization — Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, Zhuang Liu, 2025https://scholar.google.com/scholar?q=Transformers+without+Normalization17. AI Post Transformers: Language Models are Injective and Hence Invertible — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-21-language-models-are-injective-an-7545e0.mp318. AI Post Transformers: RoPE — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/rope/19. AI Post Transformers: Mistral 7B: Superior Performance in a Smaller Package — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/mistral-7b-superior-performance-in-a-smaller-package/20. AI Post Transformers: ALiBi: Attention with Linear Biases Enables Length Extrapolation — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/alibi-attention-with-linear-biases-enables-length-extrapolation/21. AI Post Transformers: CacheSlide: Position-Aware KV Cache Reuse for Agent LLMs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-16-cacheslide-position-aware-kv-cache-reuse-cd59c7.mp3Interactive Visualization: Episode: Language Models are Injective and Hence Invertible0 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.