Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- Beluga: CXL Memory Pooling for LLM KV CacheThis episode explores Beluga, a systems paper that tackles one of the biggest practical bottlenecks in long-context LLM inference: how to store and retrieve massive KV caches when GPU memory is no longer enough. It explains why traditional RDMA-based memory disaggregation is cumbersome and how Beluga uses CXL-based pooled memory to give GPUs and CPUs more direct, load/store-style access to shared cache data, reducing copies, staging, and synchronization overhead. The discussion digs into the architecture’s tradeoffs, including the fact that CXL is still slower than local HBM or DRAM, but argues that its simpler access model can still deliver large gains in the right workload regime. Listeners would find it interesting for its concrete analysis of when the reported speedups, including major reductions in time to first token and large throughput gains, are real advances versus artifacts of favorable cache-reuse conditions.Sources:1. Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management — Xinjun Yang, Qingda Hu, Junru Li, Feifei Li, Yicong Zhu, Yuqi Zhou, Qiuru Lin, Jian Dai, Yang Kong, Jiayu Zhang, Guoqiang Xu, Qiang Liu, 2025http://arxiv.org/abs/2511.201722. Memory Pooling With CXL — Donghyun Gouk, Miryeong Kwon, Hanyeoreum Bae, Sangwon Lee, Myoungsoo Jung, 2023https://scholar.google.com/scholar?q=Memory+Pooling+With+CXL3. Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices — Yan Sun, Yifan Yuan, Zeduo Yu, Reese Kuper, Chihun Song, Jinghan Huang, Houxiang Ji, Siddharth Agarwal, Jiaqi Lou, Ipoom Jeong, Ren Wang, Jung Ho Ahn, Tianyin Xu, Nam Sung Kim, 2023https://scholar.google.com/scholar?q=Demystifying+CXL+Memory+with+Genuine+CXL-Ready+Systems+and+Devices4. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2025https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving5. Exploring CXL-based KV Cache Storage for LLM Serving — Yupeng Tang, Runxiang Cheng, Ping Zhou, Tongping Liu, Fei Liu, Wei Tang, Kyoungryun Bae, Jianjun Chen, Wu Xiang, Rui Shi, 2024https://scholar.google.com/scholar?q=Exploring+CXL-based+KV+Cache+Storage+for+LLM+Serving6. MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool — Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, Yizhou Shan, 2024https://scholar.google.com/scholar?q=MemServe%3A+Context+Caching+for+Disaggregated+LLM+Serving+with+Elastic+Memory+Pool7. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention8. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, Junchen Jiang, 2024https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving9. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference — Dongjie Yang et al., 2024https://scholar.google.com/scholar?q=PyramidInfer%3A+Pyramid+KV+Cache+Compression+for+High-throughput+LLM+Inference10. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference — Xiang Liu et al., 2025https://scholar.google.com/scholar?q=ChunkKV%3A+Semantic-Preserving+KV+Cache+Compression+for+Efficient+Long-Context+LLM+Inference11. Inference-Time Hyper-Scaling with KV Cache Compression — Adrian Łańcucki, Konrad Staniszewski, Piotr Nawrot, Edoardo M. Ponti, 2025https://scholar.google.com/scholar?q=Inference-Time+Hyper-Scaling+with+KV+Cache+Compression12. FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling — Weiqing Li et al., 2025https://scholar.google.com/scholar?q=FlowKV%3A+A+Disaggregated+Inference+Framework+with+Low-Latency+KV+Cache+Transfer+and+Load-Aware+Scheduling13. TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving — Bingyang Wu et al., 2025https://scholar.google.com/scholar?q=TokenLake%3A+A+Unified+Segment-level+Prefix+Cache+Pool+for+Fine-grained+Elastic+Long-Context+LLM+Serving14. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows — Zaifeng Pan et al., 2025https://scholar.google.com/scholar?q=KVFlow%3A+Efficient+Prefix+Caching+for+Accelerating+LLM-Based+Multi-Agent+Workflows15. Learned Prefix Caching for Efficient LLM Inference — Dongsheng Yang, Austin Li, Kai Li, Wyatt Lloyd, 2025https://scholar.google.com/scholar?q=Learned+Prefix+Caching+for+Efficient+LLM+Inference16. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp317. AI Post Transformers: CXL Computational Memory Offloading for Lower Runtime — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-cxl-computational-memory-offloading-for-3b2124.mp318. AI Post Transformers: Prefill-as-a-Service for Cross-Datacenter KV Cache — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-prefill-as-a-service-for-cross-datacente-7560be.mp319. AI Post Transformers: SolidAttention: Co-Designing Sparse Attention and SSD I/O — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-18-solidattention-co-designing-sparse-atten-5a8622.mp320. AI Post Transformers: Stochastic KV Routing for Cache Sharing — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-29-stochastic-kv-routing-for-cache-sharing-5fef63.mp321. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp3Interactive Visualization: Beluga: CXL Memory Pooling for LLM KV Cache10 min
- Trajectory Summaries for Long-Horizon Coding AgentsThis episode explores a paper on inference-time scaling for coding agents, asking whether extra test-time compute still helps when tasks are long, messy, and require multi-step tool use rather than a single code completion. It focuses on the paper’s main argument that the real bottleneck is not generating more rollout attempts, but representing prior attempts well enough to compare, select, and reuse them, with structured trajectory summaries serving as the key middle layer between raw transcripts and final patches. The discussion examines two mechanisms: a parallel “tournament” style selection method over summaries, and a sequential refinement method that conditions later attempts on distilled lessons from earlier ones. Listeners would find it interesting because the conversation connects agent performance gains to practical questions of context management, selection versus reuse, and whether the reported improvements reflect a deep scaling insight or simply better engineering around long-horizon coding workflows.Sources:1. Scaling Test-Time Compute for Agentic Coding — Joongwon Kim, Wannan Yang, Kelvin Niu, Hongming Zhang, Yun Zhu, Eryk Helenowski, Ruan Silva, Zhengxing Chen, Srinivasan Iyer, Manzil Zaheer, Daniel Fried, Hannaneh Hajishirzi, Sanjeev Arora, Gabriel Synnaeve, Ruslan Salakhutdinov, Anirudh Goyal, 2026http://arxiv.org/abs/2604.165292. ReAct: Synergizing Reasoning and Acting in Language Models — Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao, 2023https://scholar.google.com/scholar?q=ReAct%3A+Synergizing+Reasoning+and+Acting+in+Language+Models3. Reflexion: Language Agents with Verbal Reinforcement Learning — Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao, 2023https://scholar.google.com/scholar?q=Reflexion%3A+Language+Agents+with+Verbal+Reinforcement+Learning4. ExpeL: LLM Agents Are Experiential Learners — Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, Gao Huang, 2023https://scholar.google.com/scholar?q=ExpeL%3A+LLM+Agents+Are+Experiential+Learners5. Rethinking Thinking Tokens: LLMs as Improvement Operators — Lovish Madaan, Aniket Didolkar, Suchin Gururangan, John Quan, Ruan Silva, Ruslan Salakhutdinov, Manzil Zaheer, Sanjeev Arora, Anirudh Goyal, 2025https://scholar.google.com/scholar?q=Rethinking+Thinking+Tokens%3A+LLMs+as+Improvement+Operators6. CodeMonkeys: Scaling Test-Time Compute for Software Engineering — Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher Re, Azalia Mirhoseini, 2025https://scholar.google.com/scholar?q=CodeMonkeys%3A+Scaling+Test-Time+Compute+for+Software+Engineering7. S*: Test Time Scaling for Code Generation — Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin Tan, Kurt Keutzer, Jiarong Xing, Joseph E. Gonzalez, Ion Stoica, 2025https://scholar.google.com/scholar?q=S%2A%3A+Test+Time+Scaling+for+Code+Generation8. Scaling Test-time Compute for LLM Agents — King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, Changwang Zhang, Chenghua Lin, Jun Wang, Ge Zhang, Wangchunshu Zhou, 2025https://scholar.google.com/scholar?q=Scaling+Test-time+Compute+for+LLM+Agents9. Agentic Test-Time Scaling for WebAgents — Nicholas Lee, Lutfi Eren Erdogan, Chris Joseph John, Surya Krishnapillai, Michael W. Mahoney, Kurt Keutzer, Amir Gholami, 2026https://scholar.google.com/scholar?q=Agentic+Test-Time+Scaling+for+WebAgents10. Does SWE-Bench-Verified Test Agent Ability or Model Memory? — Thanosan Prathifkumar, Noble Saji Mathews, Meiyappan Nagappan, 2025https://scholar.google.com/scholar?q=Does+SWE-Bench-Verified+Test+Agent+Ability+or+Model+Memory%3F11. A Benchmark for Procedural Memory Retrieval in Language Agents — Ishant Kohar, Aswanth Krishnan, 2025https://scholar.google.com/scholar?q=A+Benchmark+for+Procedural+Memory+Retrieval+in+Language+Agents12. PROCED-MEM: Benchmarking Procedural Memory Retrieval in Language Agents Across Domains — Ishant Kohar, Aswanth Krishnan, 2026https://scholar.google.com/scholar?q=PROCED-MEM%3A+Benchmarking+Procedural+Memory+Retrieval+in+Language+Agents+Across+Domains13. G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems — Guibin Zhang, Muxin Fu, Guancheng Wan, Miao Yu, Kun Wang, Shuicheng Yan, 2025https://scholar.google.com/scholar?q=G-Memory%3A+Tracing+Hierarchical+Memory+for+Multi-Agent+Systems14. Scaling Agentic Verifier for Competitive Coding — Zeyao Ma et al., 2026https://scholar.google.com/scholar?q=Scaling+Agentic+Verifier+for+Competitive+Coding15. AgentPro: Enhancing LLM Agents with Automated Process Supervision — Yuchen Deng, Shichen Fan, Naibo Wang, Xinkui Zhao, See-Kiong Ng, 2025https://scholar.google.com/scholar?q=AgentPro%3A+Enhancing+LLM+Agents+with+Automated+Process+Supervision16. Recursive Introspection: Teaching Language Model Agents How to Self-Improve — Yuxiao Qu, Tianjun Zhang, Naman Garg, Aviral Kumar, 2024https://scholar.google.com/scholar?q=Recursive+Introspection%3A+Teaching+Language+Model+Agents+How+to+Self-Improve17. Agentic Refactoring: An Empirical Study of AI Coding Agents — Kosei Horikawa, Hao Li, Yutaro Kashiwa, Bram Adams, Hajimu Iida, Ahmed E. Hassan, 2025https://scholar.google.com/scholar?q=Agentic+Refactoring%3A+An+Empirical+Study+of+AI+Coding+Agents18. AI Post Transformers: TMAS: Scaling Test-Time Compute with Multi-Agent Synergy — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-14-tmas-scaling-test-time-compute-with-mult-3abe7a.mp319. AI Post Transformers: Benchmarking Test-Time Scaling for General LLM Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-benchmarking-test-time-scaling-for-gener-8f14f9.mp320. AI Post Transformers: MiA-Signature and Global Activation for Long Context — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-13-mia-signature-and-global-activation-for-5ad62f.mp321. AI Post Transformers: Explicit Information Transmission for Context Compression — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-explicit-information-transmission-for-co-24e3c2.mp3Interactive Visualization: Trajectory Summaries for Long-Horizon Coding Agents8 min
- DFX: Multi-FPGA Acceleration for Transformer InferenceThis episode explores the DFX system, a four-FPGA appliance designed to accelerate transformer-based text generation by targeting a key weakness of GPUs: low-batch, token-by-token decode. It explains the difference between prompt processing and sequential generation, connects the paper’s older terminology to today’s prefill/decode framing, and shows why autoregressive inference often leaves GPU hardware underused even when training runs efficiently in parallel. The discussion also breaks down how DFX uses hardware-aware model parallelism and end-to-end accelerator design, rather than only speeding up isolated transformer subcomponents, to argue for lower latency and better energy and cost efficiency than a four-V100 GPU server. Listeners would find it interesting for its clear historical perspective on transformer serving and for its skepticism about how much of the reported advantage comes from FPGA specialization versus the fairness of the GPU baseline.Sources:1. DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation — Seongmin Hong, Seungjae Moon, Junsoo Kim, Sungjae Lee, Minsub Kim, Dongsoo Lee, Joo-Young Kim, 2022http://arxiv.org/abs/2209.107972. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism — Yanping Huang, Youlong Cheng, Ankur Bapna, Quoc V. Le, Yonghui Wu, Zhifeng Chen, and others, 2019https://scholar.google.com/scholar?q=GPipe%3A+Efficient+Training+of+Giant+Neural+Networks+using+Pipeline+Parallelism3. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro, 2020https://scholar.google.com/scholar?q=Megatron-LM%3A+Training+Multi-Billion+Parameter+Language+Models+Using+Model+Parallelism4. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Noam Shazeer, Zhifeng Chen, and others, 2020https://scholar.google.com/scholar?q=GShard%3A+Scaling+Giant+Models+with+Conditional+Computation+and+Automatic+Sharding5. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM — Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Bryan Catanzaro, Amar Phanishayee, Matei Zaharia, and others, 2021https://scholar.google.com/scholar?q=Efficient+Large-Scale+Language+Model+Training+on+GPU+Clusters+Using+Megatron-LM6. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, 2017https://scholar.google.com/scholar?q=Attention+Is+All+You+Need7. FTRANS: Energy-Efficient Acceleration of Transformers using FPGA — Jingcheng Rao, Yuchen Shao, Ke Wang, Zhihao Zhu, Xuehai Qian, Yiyu Shi, 2020https://scholar.google.com/scholar?q=FTRANS%3A+Energy-Efficient+Acceleration+of+Transformers+using+FPGA8. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2022https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding9. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference — Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, Hai Zhao, 2024https://scholar.google.com/scholar?q=PyramidInfer%3A+Pyramid+KV+Cache+Compression+for+High-throughput+LLM+Inference10. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference — Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Bo Li, Xuming Hu, Xiaowen Chu, 2025https://scholar.google.com/scholar?q=ChunkKV%3A+Semantic-Preserving+KV+Cache+Compression+for+Efficient+Long-Context+LLM+Inference11. Cost-Optimal Grouped-Query Attention for Long-Context LLMs — Yingfa Chen, Yutong Wu, Xu Han, Zhiyuan Liu, Maosong Sun, 2025https://scholar.google.com/scholar?q=Cost-Optimal+Grouped-Query+Attention+for+Long-Context+LLMs12. Optimised Grouped-Query Attention Mechanism for Transformers — Yuang Chen, Cheng Zhang, Xitong Gao, Robert D. Mullins, George A. Constantinides, Yiren Zhao, 2024https://scholar.google.com/scholar?q=Optimised+Grouped-Query+Attention+Mechanism+for+Transformers13. Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving — Chao Wang, Pengfei Zuo, Zhangyu Chen, Yunkai Liang, Zhou Yu, Ming-Chang Yang, 2025https://scholar.google.com/scholar?q=Prefill-Decode+Aggregation+or+Disaggregation%3F+Unifying+Both+for+Goodput-Optimized+LLM+Serving14. Nexus: Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving — Xiaoxiang Shi, Colin Cai, Junjia Du, Zhihao Jia, 2025https://scholar.google.com/scholar?q=Nexus%3A+Proactive+Intra-GPU+Disaggregation+of+Prefill+and+Decode+in+LLM+Serving15. SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inference — Hengrui Zhang, Pratyush Patel, August Ning, David Wentzlaff, 2025https://scholar.google.com/scholar?q=SPAD%3A+Specialized+Prefill+and+Decode+Hardware+for+Disaggregated+LLM+Inference16. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp317. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp318. AI Post Transformers: LAPS for Length-Aware LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-laps-for-length-aware-llm-serving-0c6149.mp319. AI Post Transformers: Prefill-as-a-Service for Cross-Datacenter KV Cache — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-prefill-as-a-service-for-cross-datacente-7560be.mp320. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp321. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp322. AI Post Transformers: Caffeine: A Unified FPGA for CNNs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-06-caffeine-a-unified-fpga-for-cnns-e8acbe.mp3Interactive Visualization: DFX: Multi-FPGA Acceleration for Transformer Inference13 min
- Generative Recursive Reasoning in Latent SpaceThis episode explores Generative Recursive Reasoning, a paper that asks whether models can reason more effectively by repeatedly refining an internal latent state instead of externalizing long chains of thought as tokens. It explains how recursive reasoning trades parameter growth for inference-time computation, and how this approach may be especially useful for tasks like Sudoku, ARC-style problems, graph coloring, and N-Queens that benefit from iterative constraint solving. A central focus is the paper’s argument that reasoning should be stochastic rather than locked into a single deterministic path, using variational methods to model multiple possible latent trajectories and improve coverage when problems have more than one valid answer. The discussion is especially interesting because it contrasts this elegant search-like mechanism with mainstream transformer practice, highlighting both the promise of branching internal hypotheses and the practical reasons industry has not adopted such architectures at scale.Sources:1. Generative Recursive Reasoning in Latent Spacehttps://arxiv.org/pdf/2605.19376v12. Universal Transformers — Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Lukasz Kaiser, 2018https://scholar.google.com/scholar?q=Universal+Transformers3. Looped Transformers are Better at Learning Learning Algorithms — Liu Yang, Kangwook Lee, Robert Nowak, Dimitris Papailiopoulos, 2024https://scholar.google.com/scholar?q=Looped+Transformers+are+Better+at+Learning+Learning+Algorithms4. Hierarchical Reasoning Model — Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, Yasin Abbasi Yadkori, 2025https://scholar.google.com/scholar?q=Hierarchical+Reasoning+Model5. Less is More: Recursive Reasoning with Tiny Networks — Alexia Jolicoeur-Martineau, 2025https://scholar.google.com/scholar?q=Less+is+More%3A+Recursive+Reasoning+with+Tiny+Networks6. Probabilistic Tiny Recursive Model — Amin Sghaier, Ali Parviz, Alexia Jolicoeur-Martineau, 2026https://scholar.google.com/scholar?q=Probabilistic+Tiny+Recursive+Model7. Auto-Encoding Variational Bayes — Diederik P. Kingma, Max Welling, 2013/2014https://scholar.google.com/scholar?q=Auto-Encoding+Variational+Bayes8. Stochastic Backpropagation and Approximate Inference in Deep Generative Models — Danilo Jimenez Rezende, Shakir Mohamed, Daan Wierstra, 2014https://scholar.google.com/scholar?q=Stochastic+Backpropagation+and+Approximate+Inference+in+Deep+Generative+Models9. Inference Suboptimality in Variational Autoencoders — Chris Cremer, Xuechen Li, David Duvenaud, 2018https://scholar.google.com/scholar?q=Inference+Suboptimality+in+Variational+Autoencoders10. Iterative Amortized Inference — Joseph Marino, Yisong Yue, Stephan Mandt, 2018https://scholar.google.com/scholar?q=Iterative+Amortized+Inference11. Amortized Variational Inference: A Systematic Review — Ankush Ganguly, Sanjana Jain, Ukrit Watchareeruetai, 2023https://scholar.google.com/scholar?q=Amortized+Variational+Inference%3A+A+Systematic+Review12. Reasoning with Latent Thoughts: On the Power of Looped Transformers — Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, Sashank J. Reddi, 2025https://scholar.google.com/scholar?q=Reasoning+with+Latent+Thoughts%3A+On+the+Power+of+Looped+Transformers13. Structured Denoising Diffusion Models in Discrete State-Spaces — Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, Rianne van den Berg, 2021https://scholar.google.com/scholar?q=Structured+Denoising+Diffusion+Models+in+Discrete+State-Spaces14. One Step Forward and K Steps Back: Better Reasoning with Denoising Recursion Models — Chris Cameron, Wangzheng Wang, Nikita Ivanov, Ashmita Bhattacharyya, Didier Chetelat, Yingxue Zhang, 2026https://scholar.google.com/scholar?q=One+Step+Forward+and+K+Steps+Back%3A+Better+Reasoning+with+Denoising+Recursion+Models15. LASAR: Latent Adaptive Semantic Aligned Reasoning for Generative Recommendation — approx. recent recommendation/LLM reasoning authors, 2025https://scholar.google.com/scholar?q=LASAR%3A+Latent+Adaptive+Semantic+Aligned+Reasoning+for+Generative+Recommendation16. Towards Inference-time Scaling for Continuous Space Reasoning — approx. recent latent reasoning authors, 2025https://scholar.google.com/scholar?q=Towards+Inference-time+Scaling+for+Continuous+Space+Reasoning17. GTS: Inference-Time Scaling of Latent Reasoning with a Learnable Gaussian Thought Sampler — approx. recent latent reasoning authors, 2025https://scholar.google.com/scholar?q=GTS%3A+Inference-Time+Scaling+of+Latent+Reasoning+with+a+Learnable+Gaussian+Thought+Sampler18. Latent Chain-of-Thought for Visual Reasoning — approx. recent visual reasoning authors, 2025https://scholar.google.com/scholar?q=Latent+Chain-of-Thought+for+Visual+Reasoning19. AI Post Transformers: TMAS: Scaling Test-Time Compute with Multi-Agent Synergy — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-14-tmas-scaling-test-time-compute-with-mult-3abe7a.mp320. AI Post Transformers: Agentic Discovery for Test-Time Scaling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-12-agentic-discovery-for-test-time-scaling-f9a81f.mp321. AI Post Transformers: Latent Space as a New Computational Paradigm — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-05-latent-space-as-a-new-computational-para-810f39.mp322. AI Post Transformers: Causal-JEPA for Object-Level World Models — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-causal-jepa-for-object-level-world-model-311a8b.mp323. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp3Interactive Visualization: Generative Recursive Reasoning in Latent Space14 min
- LPU Chip for Low-Latency LLM InferenceThis episode explores a 2024 paper on the LPU, a custom processor designed specifically for large language model inference, with an emphasis on reducing the per-token delay that users notice in interactive systems. It explains why autoregressive decoding is often limited by memory movement and synchronization rather than raw compute, making conventional GPU strengths less decisive in small-batch, user-facing generation. The discussion highlights the paper’s full-stack argument: a specialized chip, a supporting software stack called HyperDex, and a multi-device link meant to preserve low latency while scaling across processors. Listeners would find it interesting because it reframes AI hardware performance around real conversational responsiveness and digs into whether the paper’s bold efficiency and scaling claims actually hold up under careful comparison.Sources:1. LPU Chip for Low-Latency LLM Inferencehttps://arxiv.org/pdf/2408.073262. DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation — Seongmin Hong, et al., 2022https://scholar.google.com/scholar?q=DFX%3A+A+Low-latency+Multi-FPGA+Appliance+for+Accelerating+Transformer-based+Text+Generation3. SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning — Hanrui Wang, Zhekai Zhang, Song Han, 2021https://scholar.google.com/scholar?q=SpAtten%3A+Efficient+Sparse+Attention+Architecture+with+Cascade+Token+and+Head+Pruning4. A Software-Defined Tensor Streaming Multiprocessor for Large-Scale Machine Learning — Dennis Abts, et al., 2022https://scholar.google.com/scholar?q=A+Software-Defined+Tensor+Streaming+Multiprocessor+for+Large-Scale+Machine+Learning5. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al., 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention6. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Zhao, et al., 2024https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving7. RazorAttention: Efficient KV Cache Compression Through Retrieval Heads — Hanlin Tang et al., 2024https://scholar.google.com/scholar?q=RazorAttention%3A+Efficient+KV+Cache+Compression+Through+Retrieval+Heads8. Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning — Yu Fu et al., 2024https://scholar.google.com/scholar?q=Not+All+Heads+Matter%3A+A+Head-Level+KV+Cache+Compression+Method+with+Integrated+Retrieval+and+Reasoning9. Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction — Haoran Qiu et al., 2024https://scholar.google.com/scholar?q=Efficient+Interactive+LLM+Serving+with+Proxy+Model-based+Sequence+Length+Prediction10. Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving — Ke Cheng et al., 2024https://scholar.google.com/scholar?q=Slice-Level+Scheduling+for+High+Throughput+and+Load+Balanced+LLM+Serving11. Deferred Continuous Batching in Resource-Efficient Large Language Model Serving — Yongjun He, Yao Lu, Gustavo Alonso, 2024https://scholar.google.com/scholar?q=Deferred+Continuous+Batching+in+Resource-Efficient+Large+Language+Model+Serving12. TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference — Raja Gond, Nipun Kwatra, Ramachandran Ramjee, 2025https://scholar.google.com/scholar?q=TokenWeave%3A+Efficient+Compute-Communication+Overlap+for+Distributed+LLM+Inference13. Characterizing Communication Patterns in Distributed Large Language Model Inference — Lang Xu et al., 2025https://scholar.google.com/scholar?q=Characterizing+Communication+Patterns+in+Distributed+Large+Language+Model+Inference14. Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference — Pol G. Recasens et al., 2025https://scholar.google.com/scholar?q=Mind+the+Memory+Gap%3A+Unveiling+GPU+Bottlenecks+in+Large-Batch+LLM+Inference15. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp316. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp317. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp318. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp319. AI Post Transformers: Why LLM Serving Needs Mathematical Optimization — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-why-llm-serving-needs-mathematical-optim-647fc6.mp320. AI Post Transformers: LAPS for Length-Aware LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-laps-for-length-aware-llm-serving-0c6149.mp321. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp322. AI Post Transformers: JANUS for Scalable MoE Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-janus-for-scalable-moe-inference-78ae30.mp3Interactive Visualization: LPU Chip for Low-Latency LLM Inference0 min
- After Titans: Behrouz on Nested Learning and HopeEpisode title: After Titans: Behrouz on Nested Learning and HopeThe followup to our Titans episode, by the same core team a yearlater. Behrouz, Razaviyayn, Zhong, and Mirrokni (Google Research)generalize the Titans bet — that long-term memory should be alearnable module updated at test time — into a broader paradigmthey call Nested Learning, where a "deep" architecture is reallya hierarchy of nested optimization problems each compressing itsown context flow. The episode walks through their three corecontributions: (1) reframing standard optimizers like Adam andSGD-with-Momentum as associative-memory modules that compressgradient information, then proposing more expressive optimizerswith their own deep memory; (2) a self-modifying sequence modelwhose update rule is itself learned end-to-end — the naturalgeneralization of the test-time-learnable memory module Titansintroduced; (3) a continuum memory system that replaces thetraditional short-term-vs-long-term dichotomy with a continuumacross multiple update rates. Combining the self-modifyingmodule with the continuum memory system produces Hope, acontinual-learning architecture reported promising on languagemodeling, knowledge incorporation, few-shot generalization,continual learning, and long-context reasoning. The hosts treatHope as the next concrete instance of the Titans → NestedLearning research arc, stress-test the novelty againstestablished meta-learning and fast-weights literature, anddistinguish what the paper actually shows from what its framingsuggests.Sources:Nested Learning: The Illusion of Deep Learning Architectures —Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni(Google Research, NeurIPS 2025) — arXiv 2512.24695https://arxiv.org/pdf/2512.24695Titans: Learning to Memorize at Test Time — Ali Behrouz, PeilinZhong, Vahab Mirrokni (Jan 2025) — arXiv 2501.00663https://arxiv.org/pdf/2501.00663AI Post Transformers — "Titans: Learning to Memorize at Test Time"https://podcast.do-not-panic.com/episodes/2026-05-20-titans-learning-to-memorize-at-test-time-054662.mp3Interactive Visualization: After Titans: Behrouz on Nested Learning and Hope0 min
- Titans: Learning to Memorize at Test TimeThis episode explores the Titans paper’s proposal to pair standard attention with a separate learned long-term memory that updates during inference, aiming to preserve distant information without paying full quadratic attention costs across very long sequences. It situates that idea against earlier approaches such as Neural Turing Machines, Transformer-XL, Compressive Transformers, Memorizing Transformers, and linear-attention recurrent models, highlighting the recurring tradeoff between precise recall and scalable memory. The discussion focuses on the paper’s most distinctive claim: memory writes are driven by a loss-based notion of surprise, making test-time memory updates look more like small online learning steps than a simple cache. Listeners would find it interesting because it gets at a central open question in modern AI systems design: whether neural networks can gain durable, useful memory at inference time without becoming too unstable, expensive, or operationally awkward to deploy.Sources:1. Titans: Learning to Memorize at Test Timehttps://arxiv.org/pdf/2501.006632. Neural Turing Machines — Alex Graves, Greg Wayne, Ivo Danihelka, 2014https://arxiv.org/abs/1410.54013. Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context — Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov, 2019https://arxiv.org/abs/1901.028604. Compressive Transformers for Long-Range Sequence Modelling — Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, Timothy P. Lillicrap, 2020https://openreview.net/forum?id=SylKikSYDH5. Memorizing Transformers — Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, Christian Szegedy, 2022https://arxiv.org/abs/2203.089136. Learning to (learn at test time): RNNs with expressive hidden states — Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, and Sanmi Koyejo, 2024https://scholar.google.com/scholar?q=Learning+to+%28learn+at+test+time%29%3A+RNNs+with+expressive+hidden+states7. Gated Delta Networks: Improving Mamba2 with Delta Rule — Songlin Yang, Jan Kautz, and Ali Hatamizadeh, 2024https://scholar.google.com/scholar?q=Gated+Delta+Networks%3A+Improving+Mamba2+with+Delta+Rule8. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg, 2024https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F9. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack — Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev, 2024https://scholar.google.com/scholar?q=BABILong%3A+Testing+the+Limits+of+LLMs+with+Long+Context+Reasoning-in-a-Haystack10. ATLAS: Learning to Optimally Memorize the Context at Test Time — Ali Behrouz, Zeman Li, Praneeth Kacham, Majid Daliri, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, and Vahab Mirrokni, 2025https://scholar.google.com/scholar?q=ATLAS%3A+Learning+to+Optimally+Memorize+the+Context+at+Test+Time11. KV-Fold: One-Step KV-Cache Recurrence for Long-Context Inference — Alireza Nadali, Patrick Cooper, Ashutosh Trivedi, Alvaro Velasquez, 2026https://scholar.google.com/scholar?q=KV-Fold%3A+One-Step+KV-Cache+Recurrence+for+Long-Context+Inference12. SCBench: A KV Cache-Centric Analysis of Long-Context Methods — Yucheng Li, Huiqiang Jiang, Qianhui Wu, Xufang Luo, Surin Ahn, Chengruidong Zhang, Amir H. Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, Lili Qiu, 2024/2025https://scholar.google.com/scholar?q=SCBench%3A+A+KV+Cache-Centric+Analysis+of+Long-Context+Methods13. Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling — Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, Weizhu Chen, 2024https://scholar.google.com/scholar?q=Samba%3A+Simple+Hybrid+State+Space+Models+for+Efficient+Unlimited+Context+Language+Modeling14. Longhorn: State Space Models are Amortized Online Learners — Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, Qiang Liu, 2024https://scholar.google.com/scholar?q=Longhorn%3A+State+Space+Models+are+Amortized+Online+Learners15. Retrieval meets Long Context Large Language Models — Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, Bryan Catanzaro, 2023https://scholar.google.com/scholar?q=Retrieval+meets+Long+Context+Large+Language+Models16. Augmenting Language Models with Long-Term Memory — Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei, 2023https://scholar.google.com/scholar?q=Augmenting+Language+Models+with+Long-Term+Memory17. Test-Time Training Provably Improves Transformers as In-Context Learners — Halil Alperen Gozeten, M. Emrullah Ildiz, Xuechen Zhang, Mahdi Soltanolkotabi, Marco Mondelli, Samet Oymak, 2025https://scholar.google.com/scholar?q=Test-Time+Training+Provably+Improves+Transformers+as+In-Context+Learners18. AI Post Transformers: δ-mem and Online Memory for LLMs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-13-d-mem-and-online-memory-for-llms-6622fa.mp319. AI Post Transformers: In-Place Test-Time Training for Transformers — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-09-in-place-test-time-training-for-transfor-d0b976.mp320. AI Post Transformers: Gated Linear Attention for Efficient Long Sequences — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-18-gated-linear-attention-for-efficient-lon-c858ab.mp321. AI Post Transformers: MELT: Decoupling Compute From Memory — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-13-melt-decoupling-compute-from-memory-26430c.mp322. AI Post Transformers: Long Context Pre-Training with Lighthouse Attention — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-13-long-context-pre-training-with-lighthous-e85bbe.mp323. AI Post Transformers: Training Million-Token LLMs Beyond the Memory Barrier — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-04-training-million-token-llms-beyond-the-m-324edc.mp324. AI Post Transformers: Recursive Language Models for Arbitrarily Long Prompts — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-recursive-language-models-for-arbitraril-fbcd1c.mp325. AI Post Transformers: How Induction Heads Emerge in Transformers — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-03-how-induction-heads-emerge-in-transforme-a7bfcb.mp326. AI Post Transformers: DeepSeek-V4 and Practical Million-Token Context — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-25-deepseek-v4-and-practical-million-token-6f4de1.mp3Interactive Visualization: Titans: Learning to Memorize at Test Time0 min
- Affordable Large-Scale Decoding Through Model-System Co-DesignThis episode explores the paper’s claim that decoding cost in large language models is driven less by raw parameter counts and more by hardware-level behavior during autoregressive generation, especially memory bandwidth pressure from the KV cache. It explains why metrics like total or activated parameters can be misleading cost proxies, and walks through the tradeoffs among standard attention, grouped-query variants, and newer approaches such as MFA that aim to preserve expressive power while reducing cache overhead. The discussion also highlights the paper’s central systems argument: attention and FFN layers have very different performance bottlenecks, so separating them through Attention-FFN Disaggregation can make large models cheaper to serve without sacrificing capability. A listener would find it interesting for its concrete, skeptical look at why inference efficiency depends on model-system co-design rather than headline model size alone.Sources:1. Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding — StepFun, :, Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hanpeng Hu, Haonan Jia, Hao Nie, Mingliang Li, Nuo Chen, Siyu Chen, Song Yuan, Wuxun Xie, Xiaoniu Song, Xing Chen, Xingping Yang, Xuelin Zhang, Yanbo Yu, Yaoyu Wang, Yibo Zhu, Yimin Jiang, Yu Zhou, Yuanwei Lu, Houyi Li, Jingcheng Hu, Ka Man Lo, Ailin Huang, Binxing Jiao, Bo Li, Boyu Chen, Changxin Miao, Chang Lou, Chen Hu, Chen Xu, Chenfeng Yu, Chengyuan Yao, Daokuan Lv, Dapeng Shi, Deshan Sun, Ding Huang, Dingyuan Hu, Dongqing Pang, Enle Liu, Fajie Zhang, Fanqi Wan, Gulin Yan, Han Zhang, Han Zhou, Hanghao Wu, Hangyu Guo, Hanqi Chen, Hanshan Zhang, Hao Wu, Haocheng Zhang, Haolong Yan, Haoran Lv, Haoran Wei, Hebin Zhou, Heng Wang, Heng Wang, Hongxin Li, Hongyu Zhou, Hongyuan Wang, Huiyong Guo, Jia Wang, Jiahao Gong, Jialing Xie, Jian Zhou, Jianjian Sun, Jiaoren Wu, Jiaran Zhang, Jiayu Liu, Jie Cheng, Jie Luo, Jie Yan, Jie Yang, Jieyi Hou, Jinguang Zhang, Jinlan Cao, Jisheng Yin, Junfeng Liu, Junhao Huang, Junzhe Lin, Kaijun Tan, Kaixiang Li, Kang An, Kangheng Lin, Kenkun Liu, Lei Yang, Liang Zhao, Liangyu Chen, Lieyu Shi, Liguo Tan, Lin Lin, Lin Zhang, Lina Chen, Liwen Huang, Liying Shi, Longlong Gu, Mei Chen, Mengqiang Ren, Ming Li, Mingzhe Chen, Na Wang, Nan Wu, Qi Han, Qian Zhao, Qiang Zhang, Qianni Liu, Qiaohui Chen, Qiling Wu, Qinglin He, Qinyuan Tan, Qiufeng Wang, Qiuping Wu, Qiuyan Liang, Quan Sun, Rui Li, Ruihang Miao, Ruosi Wan, Ruyan Guo, Shangwu Zhong, Shaoliang Pang, Shengjie Fan, Shijie Shang, Shilei Jiang, Shiliang Yang, Shiming Hao, Shuli Gao, Siming Huang, Siqi Liu, Tiancheng Cao, Tianhao Cheng, Tianhao Peng, Wang You, Wei Ji, Wen Sun, Wenjin Deng, Wenqing He, Wenzhen Zheng, Xi Chen, Xiangwen Kong, Xianzhen Luo, Xiaobo Yang, Xiaojia Liu, Xiaoxiao Ren, Xin Han, Xin Li, Xin Wu, Xu Zhao, Yanan Wei, Yang Li, Yangguang Li, Yangshijie Xu, Yanming Xu, Yaqiang Shi, Yeqing Shen, Yi Yang, Yifei Yang, Yifeng Gong, Yihan Chen, Yijing Yang, Yinmin Zhang, Yizhuang Zhou, Yuanhao Ding, Yuantao Fan, Yuanzhen Yang, Yuchu Luo, Yue Peng, Yufan Lu, Yuhang Deng, Yuhe Yin, Yujie Liu, Yukun Chen, Yuling Zhao, Yun Mou, Yunlong Li, Yunzhou Ju, Yusheng Li, Yuxiang Yang, Yuxiang Zhang, Yuyang Chen, Zejia Weng, Zhe Xie, Zheng Ge, Zheng Gong, Zhenyi Lu, Zhewei Huang, Zhichao Chang, Zhiguo Huang, Zhirui Wang, Zidong Yang, Zili Wang, Ziqi Wang, Zixin Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Xiangyu Zhang, 2025http://arxiv.org/abs/2507.194272. Fast Transformer Decoding: One Write-Head is All You Need — Noam Shazeer, 2019https://scholar.google.com/scholar?q=Fast+Transformer+Decoding%3A+One+Write-Head+is+All+You+Need3. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, Sumit Sanghai, 2023https://scholar.google.com/scholar?q=GQA%3A+Training+Generalized+Multi-Query+Transformer+Models+from+Multi-Head+Checkpoints4. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — Zhihong Shao and DeepSeek-AI et al., 2024https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model5. Multi-matrix Factorization Attention — Jingcheng Hu, Houyi Li, Yinmin Zhang, Zili Wang, Shuigeng Zhou, Xiangyu Zhang, Heung-Yeung Shum, Daxin Jiang, 2024https://scholar.google.com/scholar?q=Multi-matrix+Factorization+Attention6. Splitwise: Efficient generative LLM inference using phase splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Inigo Goiri, Saeed Maleki, Ricardo Bianchini, 2023https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+generative+LLM+inference+using+phase+splitting7. P/D-Serve: Serving Disaggregated Large Language Model at Scale — Yibo Jin, Tao Wang, Huimin Lin and Huawei colleagues, 2024https://scholar.google.com/scholar?q=P%2FD-Serve%3A+Serving+Disaggregated+Large+Language+Model+at+Scale8. MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism — Ruidong Zhu, Ziheng Jiang, Chao Jin and ByteDance colleagues, 2025https://scholar.google.com/scholar?q=MegaScale-Infer%3A+Serving+Mixture-of-Experts+at+Scale+with+Disaggregated+Expert+Parallelism9. Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding — StepFun et al., 2025https://scholar.google.com/scholar?q=Step-3+is+Large+yet+Affordable%3A+Model-system+Co-design+for+Cost-effective+Decoding10. DeepSeek-V3 Technical Report — DeepSeek-AI et al., 2024https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report11. Qwen3 MoE 235B — Qwen Team / Alibaba researchers, 2025https://scholar.google.com/scholar?q=Qwen3+MoE+235B12. Prefill-Decode Disaggregation — Relevant serving-systems authors cited as [18, 31], 2024-2025https://scholar.google.com/scholar?q=Prefill-Decode+Disaggregation13. Kimi K2 Technical Report — Moonshot AI et al., 2025https://scholar.google.com/scholar?q=Kimi+K2+Technical+Report14. MiniMax M1 — MiniMax researchers, 2025https://scholar.google.com/scholar?q=MiniMax+M115. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse — Jingbo Yang et al., 2025https://scholar.google.com/scholar?q=KVLink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse16. HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse — Yuwei An et al., 2025https://scholar.google.com/scholar?q=HyperRAG%3A+Enhancing+Quality-Efficiency+Tradeoffs+in+Retrieval-Augmented+Generation+with+Reranker+KV-Cache+Reuse17. ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation — Shihao Wang et al., 2026https://scholar.google.com/scholar?q=ProphetKV%3A+User-Query-Driven+Selective+Recomputation+for+Efficient+KV+Cache+Reuse+in+Retrieval-Augmented+Generation18. HyperAttention: Long-context Attention in Near-Linear Time — Insu Han et al., 2023https://scholar.google.com/scholar?q=HyperAttention%3A+Long-context+Attention+in+Near-Linear+Time19. Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention — Tsendsuren Munkhdalai et al., 2024https://scholar.google.com/scholar?q=Leave+No+Context+Behind%3A+Efficient+Infinite+Context+Transformers+with+Infini-attention20. Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning — Ling Team et al., 2025https://scholar.google.com/scholar?q=Every+Attention+Matters%3A+An+Efficient+Hybrid+Architecture+for+Long-Context+Reasoning21. KVDirect: Distributed Disaggregated LLM Inference — Shiyang Chen et al., 2024https://scholar.google.com/scholar?q=KVDirect%3A+Distributed+Disaggregated+LLM+Inference22. HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment — Youhe Jiang et al., 2025https://scholar.google.com/scholar?q=HexGen-2%3A+Disaggregated+Generative+Inference+of+LLMs+in+Heterogeneous+Environment23. GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference — Yu Han et al., 2025https://scholar.google.com/scholar?q=GRACE-MoE%3A+Grouping+and+Replication+with+Locality-Aware+Routing+for+Efficient+Distributed+MoE+Inference24. AI Post Transformers: JANUS for Scalable MoE Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-janus-for-scalable-moe-inference-78ae30.mp325. AI Post Transformers: Prefill-as-a-Service for Cross-Datacenter KV Cache — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-prefill-as-a-service-for-cross-datacente-7560be.mp326. AI Post Transformers: Batch-Aware Expert Routing for Faster MoE Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-batch-aware-expert-routing-for-faster-mo-683ab6.mp327. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp328. AI Post Transformers: NanoFlow and the Future of LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-nanoflow-and-the-future-of-llm-serving-7429c9.mp329. AI Post Transformers: Why LLM Serving Needs Mathematical Optimization — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-why-llm-serving-needs-mathematical-optim-647fc6.mp330. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp331. AI Post Transformers: Nemotron 3 Super Hybrid Mamba-Transformer MoE — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-nemotron-3-super-hybrid-mamba-transforme-31ac75.mp332. AI Post Transformers: FlatAttention for Tile-Based Accelerator Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-flatattention-for-tile-based-accelerator-56e6ca.mp3Interactive Visualization: Affordable Large-Scale Decoding Through Model-System Co-Design0 min
- Serving MoE Models with Disaggregated Expert ParallelismThis episode explores MegaScale-Infer, a systems paper on serving large mixture-of-experts language models by separating the attention path from the expert feed-forward path and scheduling them independently. It explains why MoE models can look efficient on paper yet still waste GPU capacity in practice, especially during decode, where KV-cache-heavy attention and uneven expert routing create very different bottlenecks. The discussion focuses on the paper’s core argument for disaggregated expert parallelism and a ping-pong microbatch pipeline designed to keep both attention and expert GPUs busy instead of leaving one side idle. Listeners would find it interesting for its clear look at the gap between model architecture and real-world serving performance, including a pointed debate over whether strong decode benchmarks actually translate into better end-to-end user latency.Sources:1. MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism — Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, Xin Liu, 2025http://arxiv.org/abs/2504.022632. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism — Yanping Huang, Yonglong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Miaosen Wang, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Zhifeng Chen, 2019https://scholar.google.com/scholar?q=GPipe%3A+Efficient+Training+of+Giant+Neural+Networks+using+Pipeline+Parallelism3. PipeDream: Generalized Pipeline Parallelism for DNN Training — Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, Phil Gibbons, Matei Zaharia, 2019https://scholar.google.com/scholar?q=PipeDream%3A+Generalized+Pipeline+Parallelism+for+DNN+Training4. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM — Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Nitin Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Matei Zaharia, 2021https://scholar.google.com/scholar?q=Efficient+Large-Scale+Language+Model+Training+on+GPU+Clusters+Using+Megatron-LM5. Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache — Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, Shen Li, Zhigang Ji, Tao Xie, Yong Li, Wei Lin, 2024https://scholar.google.com/scholar?q=Infinite-LLM%3A+Efficient+LLM+Service+for+Long+Context+with+DistAttention+and+Distributed+KVCache6. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving7. Splitwise: Efficient Generative LLM Inference using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Inigo Goiri, Saeed Maleki, Ricardo Bianchini, 2023https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+using+Phase+Splitting8. MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs — Shiyi Cao, Shu Liu, Tyler Griggs, Peter Schafhalter, Xiaoxuan Liu, Ying Sheng, Joseph E. Gonzalez, Matei Zaharia, Ion Stoica, 2024https://scholar.google.com/scholar?q=MoE-Lightning%3A+High-Throughput+MoE+Inference+on+Memory-constrained+GPUs9. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI et al., 2024https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model10. Toward Efficient Inference for Mixture of Experts — Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Hsien-Hsin S. Lee, Shruti Bhosale, Carole-Jean Wu, Benjamin Lee, 2024https://scholar.google.com/scholar?q=Toward+Efficient+Inference+for+Mixture+of+Experts11. AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference — Shuzhang Zhong, Ling Liang, Yuan Wang, Runsheng Wang, Ru Huang, Meng Li, 2024https://scholar.google.com/scholar?q=AdapMoE%3A+Adaptive+Sensitivity-based+Expert+Gating+and+Management+for+Efficient+MoE+Inference12. HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference — Shuzhang Zhong, Yanfan Sun, Ling Liang, Runsheng Wang, Ru Huang, Meng Li, 2025https://scholar.google.com/scholar?q=HybriMoE%3A+Hybrid+CPU-GPU+Scheduling+and+Cache+Management+for+Efficient+MoE+Inference13. Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference — Jixian Zhou, Fang Dong, Ruijun Huang, Hengjie Cao, Mengyi Chen, Yifeng Yang, Anrui Chen, Mingzhi Dong, Yujiang Wang, Dongsheng Li, David A. Clifton, Qin Lv, Rui Zhu, Chun Zhang, Fan Yang, Tun Lu, Ning Gu, Li Shang, 2025https://scholar.google.com/scholar?q=Oracle-MoE%3A+Locality-preserving+Routing+in+the+Oracle+Space+for+Memory-constrained+Large+Language+Model+Inference14. Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism — Xinglin Pan, Shaohuai Shi, Wenxiang Lin, Yuxin Wang, Zhenheng Tang, Wei Wang, Xiaowen Chu, 2025https://scholar.google.com/scholar?q=Efficient+MoE+Inference+with+Fine-Grained+Scheduling+of+Disaggregated+Expert+Parallelism15. AI Post Transformers: JANUS for Scalable MoE Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-janus-for-scalable-moe-inference-78ae30.mp316. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp317. AI Post Transformers: Prefill-as-a-Service for Cross-Datacenter KV Cache — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-prefill-as-a-service-for-cross-datacente-7560be.mp318. AI Post Transformers: NanoFlow and the Future of LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-nanoflow-and-the-future-of-llm-serving-7429c9.mp319. AI Post Transformers: Why LLM Serving Needs Mathematical Optimization — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-why-llm-serving-needs-mathematical-optim-647fc6.mp320. AI Post Transformers: Batch-Aware Expert Routing for Faster MoE Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-batch-aware-expert-routing-for-faster-mo-683ab6.mp321. AI Post Transformers: LAPS for Length-Aware LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-laps-for-length-aware-llm-serving-0c6149.mp322. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp3Interactive Visualization: Serving MoE Models with Disaggregated Expert Parallelism0 min
- NanoFlow and the Future of LLM ServingThis episode explores how a line of recent systems papers culminates in NanoFlow, a serving approach that breaks LLM inference into very small “nano-batches” so different GPU-intensive operations can overlap instead of running in strict sequence. It explains the shift from thinking only about memory bottlenecks such as KV-cache movement and fragmentation toward a more nuanced claim: even if some kernels are memory-bound, overall serving throughput can still be limited by underused compute when prefill and decode are serialized. The discussion walks through the progression from micro-batching and iteration-level scheduling to chunked prefill, then shows how NanoFlow extends that logic with an auto-searched schedule that jointly chooses nano-batch size, operation ordering, and GPU resource allocation. A listener would find it interesting because it frames LLM serving not as a single-kernel optimization problem but as a broader question of hardware utilization, scheduling strategy, and the economics of running large models efficiently at scale.Sources:1. NanoFlow and the Future of LLM Servinghttps://www.usenix.org/system/files/osdi25-zhu-kan.pdf2. 2601.11822v1https://arxiv.org/html/2601.11822v13. 2410.18038v2https://arxiv.org/html/2410.18038v24. https://www.usenix.org/system/files/osdi24-agrawal.pdfhttps://www.usenix.org/system/files/osdi24-agrawal.pdf5. 1811.06965https://arxiv.org/pdf/1811.069656. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Re, 2022https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness7. Orca: A Distributed Serving System for Transformer-Based Generative Models — Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun, 2022https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models8. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve9. NanoFlow: Towards Optimal Large Language Model Serving Throughput — Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Ziren Wang, Stephanie Wang, Arvind Krishnamurthy, Baris Kasikci, 2025https://scholar.google.com/scholar?q=NanoFlow%3A+Towards+Optimal+Large+Language+Model+Serving+Throughput10. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention11. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving12. ASPEN: Breaking Operator Barriers for Efficient Parallelization of Deep Neural Networks — Jongseok Park, Kyungmin Bin, Gibum Park, Sangtae Ha, Kyunghan Lee, 2023https://scholar.google.com/scholar?q=ASPEN%3A+Breaking+Operator+Barriers+for+Efficient+Parallelization+of+Deep+Neural+Networks13. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention — Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar, 2024https://scholar.google.com/scholar?q=vAttention%3A+Dynamic+Memory+Management+for+Serving+LLMs+without+PagedAttention14. MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool — Cunchen Hu et al., 2024https://scholar.google.com/scholar?q=MemServe%3A+Context+Caching+for+Disaggregated+LLM+Serving+with+Elastic+Memory+Pool15. DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving — Chaoyi Ruan et al., 2025https://scholar.google.com/scholar?q=DynaServe%3A+Unified+and+Elastic+Execution+for+Dynamic+Disaggregated+LLM+Serving16. LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention — Shang Yang et al., 2025https://scholar.google.com/scholar?q=LServe%3A+Efficient+Long-sequence+LLM+Serving+with+Unified+Sparse+Attention17. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference — Dongjie Yang et al., 2024https://scholar.google.com/scholar?q=PyramidInfer%3A+Pyramid+KV+Cache+Compression+for+High-throughput+LLM+Inference18. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference — Xiang Liu et al., 2025https://scholar.google.com/scholar?q=ChunkKV%3A+Semantic-Preserving+KV+Cache+Compression+for+Efficient+Long-Context+LLM+Inference19. Inference-Time Hyper-Scaling with KV Cache Compression — Adrian Łańcucki, Konrad Staniszewski, Piotr Nawrot, Edoardo M. Ponti, 2025https://scholar.google.com/scholar?q=Inference-Time+Hyper-Scaling+with+KV+Cache+Compression20. Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving — Ke Cheng et al., 2024https://scholar.google.com/scholar?q=Slice-Level+Scheduling+for+High+Throughput+and+Load+Balanced+LLM+Serving21. Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction — Haoran Qiu et al., 2024https://scholar.google.com/scholar?q=Efficient+Interactive+LLM+Serving+with+Proxy+Model-based+Sequence+Length+Prediction22. Deferred Continuous Batching in Resource-Efficient Large Language Model Serving — Yongjun He, Yao Lu, Gustavo Alonso, 2024https://scholar.google.com/scholar?q=Deferred+Continuous+Batching+in+Resource-Efficient+Large+Language+Model+Serving23. AI Post Transformers: Why LLM Serving Needs Mathematical Optimization — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-why-llm-serving-needs-mathematical-optim-647fc6.mp324. AI Post Transformers: LAPS for Length-Aware LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-laps-for-length-aware-llm-serving-0c6149.mp325. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp326. AI Post Transformers: Continuous Batching for LLM Inference: Throughput and Latency Gains — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/continuous-batching-for-llm-inference-throughput-and-latency-gains/27. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp328. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp329. AI Post Transformers: CacheFlow and 3D-Parallel KV Cache Restoration — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-01-cacheflow-and-3d-parallel-kv-cache-resto-8db883.mp330. AI Post Transformers: ScoutAttention for Efficient KV Cache Offloading — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-24-scoutattention-for-efficient-kv-cache-of-b26699.mp331. AI Post Transformers: FlashFuser and Hopper-Era FFN Kernel Fusion — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-flashfuser-and-hopper-era-ffn-kernel-fus-e1fce9.mp3Interactive Visualization: NanoFlow and the Future of LLM Serving0 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.