Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- RAGEN-2: Reasoning Collapse in Agentic RLThis episode explores the RAGEN-2 paper’s claim that agentic reinforcement learning can produce reasoning traces that look active and diverse while losing real dependence on the input. It explains the paper’s central distinction between ordinary entropy, which measures diversity within a single prompt, and template collapse, where traces across many different prompts become generic variations of the same pattern. The discussion also covers the proposed mutual-information-style monitoring approach, which rescoring traces against other prompts to test whether reasoning remains identifiable to its source, and links the failure mode to weak reward signal, PPO-style regularization, and sparse long-horizon feedback. Listeners would find it interesting because it reframes a core question in reasoning RL: not whether an agent looks busy, but whether its reasoning is still actually about the problem in front of it.Sources:1. RAGEN-2: Reasoning Collapse in Agentic RL — Zihan Wang, Chi Gui, Xing Jin, Qineng Wang, Licheng Liu, Kangrui Wang, Shiqi Chen, Linjie Li, Zhengyuan Yang, Pingyue Zhang, Yiping Lu, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, Manling Li, 2026http://arxiv.org/abs/2604.062682. MINE: Mutual Information Neural Estimation — Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, R. Devon Hjelm, 2018https://scholar.google.com/scholar?q=MINE%3A+Mutual+Information+Neural+Estimation3. Representation Learning with Contrastive Predictive Coding — Aaron van den Oord, Yazhe Li, Oriol Vinyals, 2018https://scholar.google.com/scholar?q=Representation+Learning+with+Contrastive+Predictive+Coding4. Learning deep representations by mutual information estimation and maximization — R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Phil Bachman, Adam Trischler, Yoshua Bengio, 2019https://scholar.google.com/scholar?q=Learning+deep+representations+by+mutual+information+estimation+and+maximization5. RAGEN-2: Reasoning Collapse in Agentic RL — Zihan Wang, Chi Gui, Xing Jin, Qineng Wang, Manling Li, et al., 2026https://scholar.google.com/scholar?q=RAGEN-2%3A+Reasoning+Collapse+in+Agentic+RL6. Proximal Policy Optimization Algorithms — John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017https://scholar.google.com/scholar?q=Proximal+Policy+Optimization+Algorithms7. Signal-to-Noise Ratio Analysis of Policy Gradient Algorithms — John W. Roberts, Russ Tedrake, 2008https://scholar.google.com/scholar?q=Signal-to-Noise+Ratio+Analysis+of+Policy+Gradient+Algorithms8. Understanding the Impact of Entropy on Policy Optimization — Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, Dale Schuurmans, 2019https://scholar.google.com/scholar?q=Understanding+the+Impact+of+Entropy+on+Policy+Optimization9. Prioritized Experience Replay — Tom Schaul, John Quan, Ioannis Antonoglou, David Silver, 2016https://scholar.google.com/scholar?q=Prioritized+Experience+Replay10. DAPO: An Open-Source LLM Reinforcement Learning System at Scale — Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, et al., 2025https://scholar.google.com/scholar?q=DAPO%3A+An+Open-Source+LLM+Reinforcement+Learning+System+at+Scale11. No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping — Thanh-Long V. Le, Myeongho Jeon, Kim Vu, Viet Lai, Eunho Yang, 2025https://scholar.google.com/scholar?q=No+Prompt+Left+Behind%3A+Exploiting+Zero-Variance+Prompts+in+LLM+Reinforcement+Learning+via+Entropy-Guided+Advantage+Shaping12. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning — Zihan Wang et al., 2025https://scholar.google.com/scholar?q=RAGEN%3A+Understanding+Self-Evolution+in+LLM+Agents+via+Multi-Turn+Reinforcement+Learning13. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models — Ganqu Cui et al., 2025https://scholar.google.com/scholar?q=The+Entropy+Mechanism+of+Reinforcement+Learning+for+Reasoning+Language+Models14. ASTER: Agentic Scaling with Tool-integrated Extended Reasoning — Xuqin Zhang, Quan He, Zhenrui Zheng, Zongzhang Zhang, Xu He, and Dong Li, 2026https://scholar.google.com/scholar?q=ASTER%3A+Agentic+Scaling+with+Tool-integrated+Extended+Reasoning15. Demystifying Reinforcement Learning in Agentic Reasoning — Zhaochen Yu, Ling Yang, Jiaru Zou, Shuicheng Yan, and Mengdi Wang, 2025https://scholar.google.com/scholar?q=Demystifying+Reinforcement+Learning+in+Agentic+Reasoning16. The Price of Format: Diversity Collapse in LLMs — Longfei Yun, Chenyang An, Zilong Wang, Letian Peng, and Jingbo Shang, 2025https://scholar.google.com/scholar?q=The+Price+of+Format%3A+Diversity+Collapse+in+LLMs17. Revisiting Entropy in Reinforcement Learning for Large Reasoning Models — Renren Jin et al., 2025https://scholar.google.com/scholar?q=Revisiting+Entropy+in+Reinforcement+Learning+for+Large+Reasoning+Models18. ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models — Song Yu and Li Li, 2026https://scholar.google.com/scholar?q=ERPO%3A+Token-Level+Entropy-Regulated+Policy+Optimization+for+Large+Reasoning+Models19. Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning — Chen Qian et al., 2025https://scholar.google.com/scholar?q=Demystifying+Reasoning+Dynamics+with+Mutual+Information%3A+Thinking+Tokens+are+Information+Peaks+in+LLM+Reasoning20. MITS: Enhanced Tree Search Reasoning for LLMs via Pointwise Mutual Information — Jiaxi Li et al., 2025https://scholar.google.com/scholar?q=MITS%3A+Enhanced+Tree+Search+Reasoning+for+LLMs+via+Pointwise+Mutual+Information21. On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning — Yifan Zhang et al., 2025https://scholar.google.com/scholar?q=On+the+Design+of+KL-Regularized+Policy+Gradient+Algorithms+for+LLM+Reasoning22. Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization — Kezhao Liu et al., 2025https://scholar.google.com/scholar?q=Rethinking+KL+Regularization+in+RLHF%3A+From+Value+Estimation+to+Gradient+Optimization23. What Makes a Reward Model a Good Teacher? An Optimization Perspective — Noam Razin et al., 2025https://scholar.google.com/scholar?q=What+Makes+a+Reward+Model+a+Good+Teacher%3F+An+Optimization+Perspective24. Accelerating RLHF Training with Reward Variance Increase — Zonglin Yang et al., 2025https://scholar.google.com/scholar?q=Accelerating+RLHF+Training+with+Reward+Variance+Increase25. Efficient RLVR Training via Weighted Mutual Information Data Selection — Xinyu Zhou et al., 2026https://scholar.google.com/scholar?q=Efficient+RLVR+Training+via+Weighted+Mutual+Information+Data+Selection26. AI Post Transformers: Stabilizing Efficient Reasoning with Step-Level Advantage Selection — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-29-stabilizing-efficient-reasoning-with-ste-1e589d.mp327. AI Post Transformers: Reasoning Theater and Unfaithful Chain-of-Thought — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-reasoning-theater-and-unfaithful-chain-o-a4507e.mp328. AI Post Transformers: Benchmarking Test-Time Scaling for General LLM Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-benchmarking-test-time-scaling-for-gener-8f14f9.mp329. AI Post Transformers: Can Models Learn from Long Context? — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-can-models-learn-from-long-context-77533e.mp30 min
- Latent Reasoning with Normalizing FlowsThis episode explores Latent Reasoning with Normalizing Flows, a paper that asks whether a standard left-to-right transformer can do its intermediate reasoning in continuous latent states instead of spelling every step out as text. It explains how the method uses a frozen VAE during training to compress written rationales into short latent sequences, then uses shallow normalizing flows so the same autoregressive backbone can predict both latent thought slots and normal answer tokens while preserving exact likelihoods, sampling, and KV-cache-friendly decoding. The discussion highlights matched coding results on Qwen3-8B-Base, where the reported benchmark average rises from 55.8 for the base model to 68.8 for NF-CoT Unified and 70.1 after latent-space reinforcement learning, with strong pass@k gains that suggest better exploration of multiple solution paths. Listeners would find it interesting because it frames latent reasoning as a practical alternative to verbose chain-of-thought, while also noting the current evidence is still narrow, centered on one post-trained coding model and not uniformly better than diffusion baselines on every benchmark.Sources:1. Latent Reasoning with Normalizing Flows — Guancheng Tu, Xiangjun Fu, Suhao Yu, Yao Tang, Haoqiang Kang, Lianhui Qin, Yizhe Zhang, Jiatao Gu, 2026http://arxiv.org/abs/2606.064472. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Jason Wei, Xuezhi Wang, Denny Zhou, Quoc Le, et al., 2022https://scholar.google.com/scholar?q=Chain-of-Thought+Prompting+Elicits+Reasoning+in+Large+Language+Models3. Training Large Language Models to Reason in a Continuous Latent Space — Shibo Hao, Sainbayar Sukhbaatar, Zhiting Hu, Jason Weston, Yuandong Tian, et al., 2024 preprint; COLM 2025https://scholar.google.com/scholar?q=Training+Large+Language+Models+to+Reason+in+a+Continuous+Latent+Space4. Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning — Xinghao Chen, Anhao Zhao, Xiaoyu Shen, et al., 2025https://scholar.google.com/scholar?q=Reasoning+Beyond+Language%3A+A+Comprehensive+Survey+on+Latent+Chain-of-Thought+Reasoning5. LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning — Haoqiang Kang, Yizhe Zhang, Navdeep Jaitly, Yi-An Ma, Lianhui Qin, et al., 2025https://scholar.google.com/scholar?q=LaDiR%3A+Latent+Diffusion+Enhances+LLMs+for+Text+Reasoning6. CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation — Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, Yulan He, 2025https://scholar.google.com/scholar?q=CODI%3A+Compressing+Chain-of-Thought+into+Continuous+Space+via+Self-Distillation7. Normalizing Flows are Capable Generative Models — Shuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Angel Bautista, Navdeep Jaitly, Josh Susskind, 2024https://scholar.google.com/scholar?q=Normalizing+Flows+are+Capable+Generative+Models8. Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought — Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, Yuandong Tian, 2025https://scholar.google.com/scholar?q=Reasoning+by+Superposition%3A+A+Theoretical+Perspective+on+Chain+of+Continuous+Thought9. Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structure — Zirui Li, Xuefeng Bai, Kehai Chen, Yizhi Li, Jian Yang, Chenghua Lin, Min Zhang, 2026https://scholar.google.com/scholar?q=Dynamics+Within+Latent+Chain-of-Thought%3A+An+Empirical+Study+of+Causal+Structure10. Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning — Jingcheng Deng, Zihao Wei, Liang Pang, Junhong Wu, Shicheng Xu, Zenghao Duan, Huawei Shen, 2026https://scholar.google.com/scholar?q=Latent-GRPO%3A+Group+Relative+Policy+Optimization+for+Latent+Reasoning11. Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervision — Dawei Zhu et al., 2025https://arxiv.org/abs/2502.2079012. Supervised Chain of Thought — Xiang Zhang and Dujian Ding, 2024https://arxiv.org/abs/2410.1419813. Large language models can learn and generalize steganographic chain-of-thought under process supervision — Joey Skaf et al., 2025https://arxiv.org/abs/2506.0192614. Hybrid Latent Reasoning via Reinforcement Learning — Zhenrui Yue et al., 2025https://arxiv.org/abs/2505.1845415. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping et al., 2025https://arxiv.org/abs/2502.0517116. R-KV: Redundancy-aware KV Cache Compression for Reasoning Models — Zefan Cai et al., 2025https://arxiv.org/abs/2505.2413317. Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning — Yu Fu et al., 2024https://arxiv.org/abs/2410.1925818. AI Post Transformers: Generative Recursive Reasoning in Latent Space — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-21-generative-recursive-reasoning-in-latent-a9371d.mp319. AI Post Transformers: MELT: Decoupling Compute From Memory — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-13-melt-decoupling-compute-from-memory-26430c.mp320. AI Post Transformers: Reasoning Theater and Unfaithful Chain-of-Thought — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-reasoning-theater-and-unfaithful-chain-o-a4507e.mp321. AI Post Transformers: Gradient Descent at Inference Time for LLM Reasoning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-10-gradient-descent-at-inference-time-for-l-20617d.mp322. AI Post Transformers: Explicit Information Transmission for Context Compression — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-explicit-information-transmission-for-co-24e3c2.mp323. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3Interactive Visualization: Latent Reasoning with Normalizing Flows11 min
- EMO: Emergent Modularity in Sparse Language ModelsThis episode explores EMO, a Mixture-of-Experts language model that tries to turn token-level sparsity into real modularity by letting coherent expert groups emerge from document structure during pretraining. It explains why standard MoEs are not automatically deployable as smaller task-specific slices, then walks through EMO’s main idea: each document routes tokens through a learned document-specific pool of experts rather than the full expert set, with global load balancing to keep training stable. The discussion highlights that EMO was trained at substantial scale and reportedly matches a conventional MoE as a full model, while retaining surprisingly strong performance when only a fraction of experts are kept in memory for a given task. A listener would find it interesting because it connects a concrete systems problem, serving large models under tight memory budgets, to a plausible path toward more reusable, domain-specialized LLM components.Sources:1. EMO: Emergent Modularity in Sparse Language Modelshttps://allenai.org/papers/emo2. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean, 2017https://scholar.google.com/scholar?q=Outrageously+Large+Neural+Networks%3A+The+Sparsely-Gated+Mixture-of-Experts+Layer3. A Review of Sparse Expert Models in Deep Learning — William Fedus, Jeff Dean, Barret Zoph, 2022https://scholar.google.com/scholar?q=A+Review+of+Sparse+Expert+Models+in+Deep+Learning4. ST-MoE: Designing Stable and Transferable Sparse Expert Models — Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, William Fedus, 2022https://scholar.google.com/scholar?q=ST-MoE%3A+Designing+Stable+and+Transferable+Sparse+Expert+Models5. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Damai Dai, Chengqi Deng, Chenggang Zhao, Huazuo Gao, Deli Chen, Jiashi Li, Chong Ruan, Zhifang Sui, Wenfeng Liang, 2024https://scholar.google.com/scholar?q=DeepSeekMoE%3A+Towards+Ultimate+Expert+Specialization+in+Mixture-of-Experts+Language+Models6. Neural Module Networks — Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein, 2015https://scholar.google.com/scholar?q=Neural+Module+Networks7. AdapterFusion: Non-Destructive Task Composition for Transfer Learning — Jonas Pfeiffer, Aishwarya Kamath, Andreas Rucklé, Kyunghyun Cho, Iryna Gurevych, 2021https://scholar.google.com/scholar?q=AdapterFusion%3A+Non-Destructive+Task+Composition+for+Transfer+Learning8. Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners — Shashank Gupta, Subhabrata Mukherjee, Krishan Subudhi, Eduardo Gonzalez, Damien Jose, Ahmed H. Awadallah, Jianfeng Gao, 2022https://scholar.google.com/scholar?q=Sparsely+Activated+Mixture-of-Experts+are+Robust+Multi-Task+Learners9. GLaM: Efficient Scaling of Language Models with Mixture-of-Experts — Nan Du, Yanping Huang, Andrew Dai, Sebastian Goodman, Orhan Firat, Quoc Le, Yonghui Wu, Zhifeng Chen, Claire Cui, et al., 2021https://scholar.google.com/scholar?q=GLaM%3A+Efficient+Scaling+of+Language+Models+with+Mixture-of-Experts10. OLMoE: Open Mixture-of-Experts Language Models — Niklas Muennighoff et al., 2024https://scholar.google.com/scholar?q=OLMoE%3A+Open+Mixture-of-Experts+Language+Models11. Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM — Sainbayar Sukhbaatar et al., 2024https://scholar.google.com/scholar?q=Branch-Train-MiX%3A+Mixing+Expert+LLMs+into+a+Mixture-of-Experts+LLM12. FlexOlmo: Open Language Models for Flexible Data Use — Weijia Shi et al., 2025https://scholar.google.com/scholar?q=FlexOlmo%3A+Open+Language+Models+for+Flexible+Data+Use13. Emergent Modularity in Pre-trained Transformers — Zhengyan Zhang et al., 2023https://scholar.google.com/scholar?q=Emergent+Modularity+in+Pre-trained+Transformers14. Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations — Zican Dong et al., 2025https://scholar.google.com/scholar?q=Domain-Specific+Pruning+of+Large+Mixture-of-Experts+Models+with+Few-shot+Demonstrations15. Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models — Xudong Lu et al., 2024https://arxiv.org/abs/2402.1480016. Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs — Enshu Liu et al., 2024https://arxiv.org/abs/2407.0094517. The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise — Xi Wang, Soufiane Hayou, Eric Nalisnick, 2026https://arxiv.org/abs/2604.0978018. Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation — Junzhuo Li et al., 2025https://arxiv.org/abs/2509.1688219. AI Post Transformers: Batch-Aware Expert Routing for Faster MoE Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-batch-aware-expert-routing-for-faster-mo-683ab6.mp320. AI Post Transformers: JANUS for Scalable MoE Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-janus-for-scalable-moe-inference-78ae30.mp321. AI Post Transformers: Nemotron 3 Super Hybrid Mamba-Transformer MoE — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-nemotron-3-super-hybrid-mamba-transforme-31ac75.mp312 min
- When AI Builds Itself and Recursive Self-ImprovementThis episode explores Marina Favaro et al.’s 2026 paper on whether AI is beginning to accelerate frontier AI development enough to hint at recursive self-improvement, laying out the ladder from chatbots and coding agents to systems that materially help build their own successors. It distinguishes progress on long-horizon, fixed-goal engineering tasks from the harder problem of genuine research judgment, using public evidence such as METR, CORE-Bench, and RE-Bench to argue that AI is clearly getting better at sustained execution but has not yet shown strong scientific taste or reliable autonomy. It then digs into Anthropic’s internal evidence, including claims that by May 2026 Claude was responsible for over 80% of merged production code, open-ended coding-task success had risen sharply, and fixed-goal research engineering tasks improved from roughly 3x to 52x over a year, while the speakers repeatedly stress that code volume and self-reported productivity overstate true impact. Listeners would find it interesting because the discussion ties concrete benchmark results and lab productivity numbers to the bigger question of whether faster AI development also compresses the timelines for safety, governance, and the arrival of more capable successor systems.Sources:1. When AI Builds Itself and Recursive Self-Improvementhttps://www.anthropic.com/institute/recursive-self-improvement2. Measuring AI Ability to Complete Long Tasks — Thomas Kwa, Ben West, Joel Becker, Amy Deng, et al., 2025https://scholar.google.com/scholar?q=Measuring+AI+Ability+to+Complete+Long+Tasks3. RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts — Hjalmar Wijk, Tao Roa Lin, Joel Becker, Sami Jawhar, et al., 2025https://scholar.google.com/scholar?q=RE-Bench%3A+Evaluating+Frontier+AI+R%26D+Capabilities+of+Language+Model+Agents+against+Human+Experts4. CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark — Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, Arvind Narayanan, 2024https://scholar.google.com/scholar?q=CORE-Bench%3A+Fostering+the+Credibility+of+Published+Research+Through+a+Computational+Reproducibility+Agent+Benchmark5. Automated Weak-to-Strong Researcher — Jiaxin Wen, Liang Qiu, Joe Benton, Jan Hendrik Kirchner, Jan Leike, 2026https://scholar.google.com/scholar?q=Automated+Weak-to-Strong+Researcher6. PaperBench: Evaluating AI's Ability to Replicate AI Research — Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, et al., 2025https://scholar.google.com/scholar?q=PaperBench%3A+Evaluating+AI%27s+Ability+to+Replicate+AI+Research7. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity — Joel Becker, Nate Rush, Elizabeth Barnes, David Rein, 2025https://scholar.google.com/scholar?q=Measuring+the+Impact+of+Early-2025+AI+on+Experienced+Open-Source+Developer+Productivity8. SWE-bench Goes Live! — Linghao Zhang et al., 2025https://scholar.google.com/scholar?q=SWE-bench+Goes+Live%219. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving — Daoguang Zan et al., 2025https://scholar.google.com/scholar?q=Multi-SWE-bench%3A+A+Multilingual+Benchmark+for+Issue+Resolving10. SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? — John Yang et al., 2024https://scholar.google.com/scholar?q=SWE-bench+Multimodal%3A+Do+AI+Systems+Generalize+to+Visual+Software+Domains%3F11. Can AI Conduct Autonomous Scientific Research? Case Studies on Two Real-World Tasks — S. Agrawal et al., 2026https://scholar.google.com/scholar?q=Can+AI+Conduct+Autonomous+Scientific+Research%3F+Case+Studies+on+Two+Real-World+Tasks12. Collapse of Self-trained Language Models — David Herel and Tomas Mikolov, 2024https://scholar.google.com/scholar?q=Collapse+of+Self-trained+Language+Models13. Exploring Automation Bias in Human-AI Collaboration: A Review and Implications for Explainable AI — G. Romeo and D. Conti, 2025https://scholar.google.com/scholar?q=Exploring+Automation+Bias+in+Human-AI+Collaboration%3A+A+Review+and+Implications+for+Explainable+AI14. AI Post Transformers: ASI-Evolve for Data, Architectures, and RL — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-05-asi-evolve-for-data-architectures-and-rl-197b2b.mp315. AI Post Transformers: Trajectory Summaries for Long-Horizon Coding Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-24-trajectory-summaries-for-long-horizon-co-0194be.mp316. AI Post Transformers: TMAS: Scaling Test-Time Compute with Multi-Agent Synergy — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-14-tmas-scaling-test-time-compute-with-mult-3abe7a.mp317. AI Post Transformers: Air Force One, Jensen Huang, and Anthropic's 2028 Memo — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-2028-scenarios-for-global-ai-leadership-d0ec29.mp3Interactive Visualization: When AI Builds Itself and Recursive Self-Improvement0 min
- Mooncake for KV Cache-Centric LLM ServingThis episode explores Mooncake, a production LLM serving architecture that treats KV cache reuse and movement as the central challenge in long-context chat, not just raw GPU compute. It explains why prefill and decode stress hardware in different ways, how metrics like time to first token and time between tokens drive system design, and why separating those phases helps meet real latency targets. The discussion walks through Mooncake’s cache-first scheduler, tiered KV storage across GPU memory, CPU DRAM, and SSD, and its use of RDMA, chunked prefill, and layer-wise overlap to start decoding sooner while reusing existing state. It also argues that Mooncake’s interest lies less in a single breakthrough than in how it combines prefix-aware routing, overload-aware early rejection, and cross-node KV reuse into a practical serving stack for large-scale chat systems.Sources:1. Mooncake for KV Cache-Centric LLM Servinghttps://arxiv.org/pdf/2407.000792. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, et al., 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention3. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Ying Sheng, et al., 2023https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs4. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving5. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2024https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving6. Overload Control for Scaling WeChat Microservices — Hao Zhou, Ming Chen, Qian Lin, Yong Wang, Xiaobin She, Sifan Liu, Rui Gu, Beng Chin Ooi, Junfeng Yang, 2018https://scholar.google.com/scholar?q=Overload+Control+for+Scaling+WeChat+Microservices7. Overload Control for microsecond-scale RPCs with Breakwater — Inho Cho, Ahmed Saeed, Joshua Fried, Seo Jin Park, Mohammad Alizadeh, Adam Belay, 2020https://scholar.google.com/scholar?q=Overload+Control+for+microsecond-scale+RPCs+with+Breakwater8. SCORPIO: Serving the Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference — Yinghao Tang, Tingfeng Lan, Xiuqi Huang, Hui Lu, Wei Chen, 2025https://scholar.google.com/scholar?q=SCORPIO%3A+Serving+the+Right+Requests+at+the+Right+Time+for+Heterogeneous+SLOs+in+LLM+Inference9. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Inigo Goiri, Aashaka Shah, Saeed Maleki, Ricardo Bianchini, 2023https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting10. AttentionStore: Cost-Effective Attention Reuse Across Multi-Turn Conversations in Large Language Model Serving — Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, Pengfei Zuo, 2024https://scholar.google.com/scholar?q=AttentionStore%3A+Cost-Effective+Attention+Reuse+Across+Multi-Turn+Conversations+in+Large+Language+Model+Serving11. Preble: Efficient Distributed Prompt Scheduling for LLM Serving — Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, Yiying Zhang, 2024https://scholar.google.com/scholar?q=Preble%3A+Efficient+Distributed+Prompt+Scheduling+for+LLM+Serving12. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang, 2024https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion13. P/D-Serve: Serving Disaggregated Large Language Model at Scale — Yibo Jin et al., 2024https://scholar.google.com/scholar?q=P%2FD-Serve%3A+Serving+Disaggregated+Large+Language+Model+at+Scale14. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference — Aditya K. Kamath et al., 2025https://scholar.google.com/scholar?q=POD-Attention%3A+Unlocking+Full+Prefill-Decode+Overlap+for+Faster+LLM+Inference15. Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models — Siyan Zhao et al., 2024https://scholar.google.com/scholar?q=Prepacking%3A+A+Simple+Method+for+Fast+Prefilling+and+Increased+Throughput+in+Large+Language+Models16. Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference — Donghyeon Joo et al., 2025https://scholar.google.com/scholar?q=Mustafar%3A+Promoting+Unstructured+Sparsity+for+KV+Cache+Pruning+in+LLM+Inference17. SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning — Huanxuan Liao et al., 2025/2026https://scholar.google.com/scholar?q=SparK%3A+Query-Aware+Unstructured+Sparsity+with+Recoverable+KV+Cache+Channel+Pruning18. More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression — Jiebin Zhang et al., 2024https://scholar.google.com/scholar?q=More+Tokens%2C+Lower+Precision%3A+Towards+the+Optimal+Token-Precision+Trade-off+in+KV+Cache+Compression19. KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse — Huan Yang et al., 2025https://scholar.google.com/scholar?q=KVShare%3A+An+LLM+Service+System+with+Efficient+and+Effective+Multi-Tenant+KV+Cache+Reuse20. CacheSolidarity: Preventing Prefix Caching Side Channels in Multi-tenant LLM Serving Systems — Panagiotis Georgios Pennas et al., 2026https://scholar.google.com/scholar?q=CacheSolidarity%3A+Preventing+Prefix+Caching+Side+Channels+in+Multi-tenant+LLM+Serving+Systems21. AI Post Transformers: Prefill-as-a-Service for Cross-Datacenter KV Cache — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-prefill-as-a-service-for-cross-datacente-7560be.mp322. AI Post Transformers: Characterizing LLM KV Cache Workloads in Production — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/characterizing-llm-kv-cache-workloads-in-production/23. AI Post Transformers: ContiguousKV for Faster LLM Prefill KV Reuse — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-20-contiguouskv-for-faster-llm-prefill-kv-r-59f545.mp324. AI Post Transformers: CacheFlow and 3D-Parallel KV Cache Restoration — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-01-cacheflow-and-3d-parallel-kv-cache-resto-8db883.mp325. AI Post Transformers: Beluga: CXL Memory Pooling for LLM KV Cache — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-27-beluga-cxl-memory-pooling-for-llm-kv-cac-b6142f.mp326. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3Interactive Visualization: Mooncake for KV Cache-Centric LLM Serving0 min
- Technical AGI Safety and Security FrameworkThis episode explores DeepMind’s paper on technical AGI safety and security, focusing on how labs might prevent severe, humanity-scale harm before highly capable systems are deployed. It breaks down the paper’s core distinctions between misuse and misalignment, explains what the authors mean by Exceptional AGI and the no-human-ceiling assumption, and examines dangerous capability evaluations in areas like cyber, biology, persuasion, and self-proliferation. The discussion highlights the paper’s main argument that safety measures such as refusal training, jailbreak hardening, access controls, monitoring, anomaly detection, and model-weight security only matter if they are explicitly tied to capability thresholds that trigger real deployment restrictions. Listeners would find it interesting because it turns abstract AGI risk debates into a concrete governance and engineering framework for deciding when a model is too dangerous to release under normal conditions.Sources:1. Technical AGI Safety and Security Frameworkhttps://arxiv.org/pdf/2504.018492. The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation — Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, et al., 2018https://scholar.google.com/scholar?q=The+Malicious+Use+of+Artificial+Intelligence%3A+Forecasting%2C+Prevention%2C+and+Mitigation3. Model evaluation for extreme risks — Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, et al., 2023https://scholar.google.com/scholar?q=Model+evaluation+for+extreme+risks4. Frontier AI Regulation: Managing Emerging Risks to Public Safety — Markus Anderljung, Joslyn Barnhart, Anton Korinek, Jade Leung, et al., 2023https://scholar.google.com/scholar?q=Frontier+AI+Regulation%3A+Managing+Emerging+Risks+to+Public+Safety5. Evaluating Frontier Models for Dangerous Capabilities — Mary Phuong, Matthew Aitchison, Elliot Catt, Victoria Krakovna, et al., 2024https://scholar.google.com/scholar?q=Evaluating+Frontier+Models+for+Dangerous+Capabilities6. Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS) — Richard Hawkins, Colin Paterson, Chiara Picardi, Ibrahim Habli, et al., 2021https://scholar.google.com/scholar?q=Guidance+on+the+Assurance+of+Machine+Learning+in+Autonomous+Systems+%28AMLAS%297. Safety Cases: How to Justify the Safety of Advanced AI Systems — Joshua Clymer, Nick Gabrieli, David Krueger, Thomas Larsen, 2024https://scholar.google.com/scholar?q=Safety+Cases%3A+How+to+Justify+the+Safety+of+Advanced+AI+Systems8. Safety case template for frontier AI: A cyber inability argument — Arthur Goemans, Marie Davidsen Buhl, Jonas Schuett, Geoffrey Irving, et al., 2024https://scholar.google.com/scholar?q=Safety+case+template+for+frontier+AI%3A+A+cyber+inability+argument9. The BIG Argument for AI Safety Cases — Ibrahim Habli, Richard Hawkins, Colin Paterson, Mark Sujan, et al., 2025https://scholar.google.com/scholar?q=The+BIG+Argument+for+AI+Safety+Cases10. Safety cases: Justifying the safety of advanced AI systems — J. Clymer, N. Gabrieli, D. Krueger, and T. Larsen, 2024https://scholar.google.com/scholar?q=Safety+cases%3A+Justifying+the+safety+of+advanced+AI+systems11. AI control: Improving safety despite intentional subversion — R. Greenblatt, B. Shlegeris, K. Sachan, and F. Roger, 2024https://scholar.google.com/scholar?q=AI+control%3A+Improving+safety+despite+intentional+subversion12. Alignment faking in large language models — R. Greenblatt, C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Belonax, J. Chen, D. Duvenaud, et al., 2024https://scholar.google.com/scholar?q=Alignment+faking+in+large+language+models13. Towards evaluations-based safety cases for AI scheming — M. Balesni, M. Hobbhahn, D. Lindner, A. Meinke, T. Korbak, J. Clymer, B. Shlegeris, J. Scheurer, C. Stix, R. Shah, et al., 2024https://scholar.google.com/scholar?q=Towards+evaluations-based+safety+cases+for+AI+scheming14. Generative AI misuse: A taxonomy of tactics and insights from real-world data — N. Marchal, R. Xu, R. Elasmar, I. Gabriel, B. Goldberg, and W. Isaac, 2024https://scholar.google.com/scholar?q=Generative+AI+misuse%3A+A+taxonomy+of+tactics+and+insights+from+real-world+data15. Stress-Testing Capability Elicitation With Password-Locked Models — Ryan Greenblatt et al., 2024https://scholar.google.com/scholar?q=Stress-Testing+Capability+Elicitation+With+Password-Locked+Models16. Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models — Cameron Tice et al., 2024https://scholar.google.com/scholar?q=Noise+Injection+Reveals+Hidden+Capabilities+of+Sandbagging+Language+Models17. Benchmarking Misuse Mitigation Against Covert Adversaries — Davis Brown et al., 2025https://scholar.google.com/scholar?q=Benchmarking+Misuse+Mitigation+Against+Covert+Adversaries18. On scalable oversight with weak LLMs judging strong LLMs — Zachary Kenton et al., 2024https://scholar.google.com/scholar?q=On+scalable+oversight+with+weak+LLMs+judging+strong+LLMs19. Scaling Laws For Scalable Oversight — Joshua Engels et al., 2025https://scholar.google.com/scholar?q=Scaling+Laws+For+Scalable+Oversight20. Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs — Kyle O'Brien et al., 2025https://scholar.google.com/scholar?q=Deep+Ignorance%3A+Filtering+Pretraining+Data+Builds+Tamper-Resistant+Safeguards+into+Open-Weight+LLMs21. AI Post Transformers: When LLM Judges Become Coin Flips — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-when-llm-judges-become-coin-flips-8b43ef.mp322. AI Post Transformers: Reasoning Theater and Unfaithful Chain-of-Thought — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-reasoning-theater-and-unfaithful-chain-o-a4507e.mp323. AI Post Transformers: Split Personality Training Reveals Latent Knowledge — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-08-split-personality-training-reveals-laten-c84616.mp324. AI Post Transformers: Trajectory Summaries for Long-Horizon Coding Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-24-trajectory-summaries-for-long-horizon-co-0194be.mp313 min
- VeriCache: Lossless LLM Inference from Lossy KV CachesThis episode explores VeriCache, a systems paper that asks whether a large language model can draft tokens using a compressed, lossy KV cache and then verify them against the full cache to recover exactly the same greedy-decoding output. It explains why KV cache size has become a central bottleneck in long-context inference, unpacking token dropping, KV quantization, prefix caching, and the speculative decoding ideas that VeriCache turns into a closed-loop draft-and-verify scheme. The discussion argues that fluency is not enough for real deployments because even a single wrong token can break code, structured JSON, or tool calls, so exact token-by-token agreement is the real standard for safe acceleration. Listeners get a clear picture of where serving stacks such as vLLM, Hugging Face TGI, and NVIDIA already use neighboring optimizations, and why VeriCache’s specific lossless-verification approach is both technically appealing and operationally difficult.Sources:1. VeriCache: Turning Lossy KV Cache into Lossless LLM Inference — Jiayi Yao, Samuel Shen, Kuntai Du, Shaoting Feng, Dongjoo Seo, Rui Zhang, Yuyang Huang, Yuhan Liu, Shan Lu, Junchen Jiang, 2026http://arxiv.org/abs/2605.176132. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023https://proceedings.mlr.press/v202/leviathan23a.html3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Beidi Chen, et al., 2023https://proceedings.neurips.cc/paper_files/paper/2023/hash/6ceefa7b15572587b78ecfcebb2827f8-Abstract-Conference.html4. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache — Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Beidi Chen, Xia Hu, et al., 2024https://arxiv.org/abs/2402.027505. QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache — Rishabh Tiwari, Haocheng Xi, Aditya Tomar, Coleman Hooper, Kurt Keutzer, Amir Gholami, et al., 2025https://arxiv.org/abs/2502.104246. MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding — Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, et al., 2024https://scholar.google.com/scholar?q=MagicDec%3A+Breaking+the+Latency-Throughput+Tradeoff+for+Long+Context+Generation+with+Speculative+Decoding7. Accelerating Large-Scale Reasoning Model Inference: Self-Speculative Decoding with Sparse Attention — Yilong Zhao, Jiaming Tang, Kan Zhu, Zihao Ye, Chi-Chih Chang, Chaofan Lin, Jongseok Park, Guangxuan Xiao, Mohamed S. Abdelfattah, Mingyu Gao, Baris Kasikci, Song Han, and Ion Stoica, 2025https://scholar.google.com/scholar?q=Accelerating+Large-Scale+Reasoning+Model+Inference%3A+Self-Speculative+Decoding+with+Sparse+Attention8. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, and Junchen Jiang, 2024https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving9. ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching — Xingyu Xiang, Raj Joshi, Yuhan Liu, Jiayi Yao, Chenxingyu Zhao, Junchen Jiang, Yang Zhou, Eddie Kohler, and Minlan Yu, 2025https://scholar.google.com/scholar?q=ShadowServe%3A+Interference-Free+KV+Cache+Fetching+for+Distributed+Prefix+Caching10. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, et al., 2025https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference11. ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs — Jianlong Lei and Shashikant Ilager, 2026https://arxiv.org/abs/2603.0872712. Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs — Sayed Pedram Haeri Boroujeni et al., 2026https://arxiv.org/abs/2604.0472213. LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification — Penghui Yang et al., 2025https://arxiv.org/abs/2502.1742114. RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding — Guanzheng Chen et al., 2025https://arxiv.org/abs/2502.2033015. LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management — Yi Xiong et al., 2024https://arxiv.org/abs/2410.0042816. TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving — Bingyang Wu et al., 2025https://arxiv.org/abs/2508.1721917. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows — Zaifeng Pan et al., 2025https://arxiv.org/abs/2507.0740018. KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse — Huan Yang et al., 2025https://arxiv.org/abs/2503.1652519. KV Cache Offloading for Context-Intensive Tasks — Andrey Bocharnikov et al., 2026https://arxiv.org/abs/2604.0842620. AI Post Transformers: KVzip for Query-Agnostic KV Cache Compression — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-29-kvzip-for-query-agnostic-kv-cache-compre-72afe5.mp321. AI Post Transformers: PackKV Lossy Compression for KV Caches — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-04-packkv-lossy-compression-for-kv-caches-b37bce.mp322. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp323. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp324. AI Post Transformers: Stochastic KV Routing for Cache Sharing — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-29-stochastic-kv-routing-for-cache-sharing-5fef63.mp314 min
- SFMP Search-Free Mixed-Precision LLM QuantizationThis episode explores SFMP, a post-training, weight-only quantization method for large language models that targets awkward memory budgets like 3.25 or 3.75 bits per weight, where standard 3-bit or 4-bit schemes are a poor fit. It explains the core ideas behind mixed-precision quantization, including salience scoring with diagonal Fisher information, block-wise allocation, and why hardware constraints make many fine-grained methods difficult to deploy in practice. The discussion centers on SFMP’s main claim: instead of running an expensive search over bit allocations, it ranks blocks by importance and assigns only the two neighboring integer precisions, such as giving the most salient 25 percent of blocks 4-bit weights and the rest 3-bit for a 3.25 BPW target. Listeners would find it interesting because it frames quantization not as a narrow accuracy trick, but as a concrete engineering problem about trading off model quality, memory limits, and real serving-system compatibility.Sources:1. SFMP Search-Free Mixed-Precision LLM Quantizationhttps://arxiv.org/pdf/2602.010272. HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision — Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, Kurt Keutzer, 2019https://scholar.google.com/scholar?q=HAWQ%3A+Hessian+AWare+Quantization+of+Neural+Networks+with+Mixed-Precision3. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2023https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers4. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, Song Han, 2024https://scholar.google.com/scholar?q=AWQ%3A+Activation-aware+Weight+Quantization+for+LLM+Compression+and+Acceleration5. AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Models — Sangjun Lee, Seung-taek Woo, Jungyu Jin, Changhun Lee, Eunhyeok Park, 2025https://scholar.google.com/scholar?q=AMQ%3A+Enabling+AutoML+for+Mixed-precision+Weight-Only+Quantization+of+Large+Language+Models6. BitStack: Fine-Grained Size Control for Compressed Large Language Models in Variable Memory Environments — Xinghao Wang, Pengyu Wang, Bo Wang, Dong Zhang, Yunhua Zhou, Xipeng Qiu, 2024https://scholar.google.com/scholar?q=BitStack%3A+Fine-Grained+Size+Control+for+Compressed+Large+Language+Models+in+Variable+Memory+Environments7. SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models — Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, Xiaojuan Qi, 2024https://scholar.google.com/scholar?q=SliM-LLM%3A+Salience-Driven+Mixed-Precision+Quantization+for+Large+Language+Models8. FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs — Xilong Xie, Liang Wang, Limin Xiao, Meng Han, Lin Sun, Shuai Zheng, Xiangrong Xu, 2025https://scholar.google.com/scholar?q=FineQ%3A+Software-Hardware+Co-Design+for+Low-Bit+Fine-Grained+Mixed-Precision+Quantization+of+LLMs9. T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge — Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, Mao Yang, 2024https://scholar.google.com/scholar?q=T-MAC%3A+CPU+Renaissance+via+Table+Lookup+for+Low-Bit+LLM+Deployment+on+Edge10. MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design — Zhen Zheng et al., 2024https://arxiv.org/abs/2412.1459011. SpinQuant: LLM quantization with learned rotations — Zechun Liu et al., 2024https://arxiv.org/abs/2405.1640612. DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization — Yuantian Shao et al., 2025https://arxiv.org/abs/2511.0406313. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization — Coleman Hooper et al., 2024https://arxiv.org/abs/2401.1807914. QAQ: Quality Adaptive Quantization for LLM KV Cache — Shichen Dong et al., 2024https://arxiv.org/abs/2403.0464315. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp316. AI Post Transformers: PackKV Lossy Compression for KV Caches — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-04-packkv-lossy-compression-for-kv-caches-b37bce.mp317. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp318. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp314 min
- Harvest: Borrowing Peer GPU Memory for LLMsThis episode explores Harvest, a system for LLM inference that uses idle HBM on neighboring NVLink-connected GPUs as a temporary cache when a serving GPU runs out of local memory. It explains why LLM serving is often bottlenecked more by memory capacity and data movement than by raw compute, focusing on two concrete cases: growing KV caches during long-context decoding and the shifting expert weights used in mixture-of-experts models. A key argument is that peer GPU memory is only useful if it is revocable without breaking correctness, so Harvest treats borrowed memory as a best-effort cache backed by authoritative copies or reconstruction paths elsewhere. Listeners get specific performance results, including up to 5.65x lower KV-cache transfer latency than CPU offload, 7.5x to 9.5x faster expert transfers over NVLink, and roughly 1.5x to 2.0x throughput gains on models such as Qwen and Phi-3.5.Sources:1. Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inference — Nikhil Gopal, Kostis Kaffes, 2026http://arxiv.org/abs/2602.003282. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention3. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, Ce Zhang, 2023https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU4. Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models — Keisuke Kamahori, Tian Tang, Yile Gu, Kan Zhu, Baris Kasikci, 2025https://scholar.google.com/scholar?q=Fiddler%3A+CPU-GPU+Orchestration+for+Fast+Inference+of+Mixture-of-Experts+Models5. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, et al., 2025https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving6. AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains — Abhishek Vijaya Kumar, Gianni Antichi, Rachee Singh, 2025https://scholar.google.com/scholar?q=AQUA%3A+Network-Accelerated+Memory+Offloading+for+LLMs+in+Scale-Up+GPU+Domains7. Accurate Expert Predictions in MoE Inference via Cross-Layer Gate — Zhiyuan Fang, Hong Huang, Yiming Lyu, Jiyang Chen, Yu Yu, Zexi Zheng, 2025https://scholar.google.com/scholar?q=Accurate+Expert+Predictions+in+MoE+Inference+via+Cross-Layer+Gate8. Characterization of Large Language Model Development in the Datacenter — Qinghao Hu et al., 2024https://scholar.google.com/scholar?q=Characterization+of+Large+Language+Model+Development+in+the+Datacenter9. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters — Qizhen Weng et al., 2022https://scholar.google.com/scholar?q=MLaaS+in+the+Wild%3A+Workload+Analysis+and+Scheduling+in+Large-Scale+Heterogeneous+GPU+Clusters10. MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool — Cunchen Hu et al., 2024https://arxiv.org/abs/2406.1756511. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows — Zaifeng Pan et al., 2025https://arxiv.org/abs/2507.0740012. Learned Prefix Caching for Efficient LLM Inference — Dongsheng Yang et al., 2025https://papers.neurips.cc/paper_files/paper/2025/hash/414f642a1ea9350006669774cba9bcd4-Abstract-Conference.html13. DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurance — Yuning Zhang et al., 2025https://arxiv.org/abs/2509.0737914. ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference — Zixu Shen et al., 2025https://arxiv.org/abs/2510.2673015. AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference — Shuzhang Zhong et al., 2024https://arxiv.org/abs/2408.1028416. HyperAttention: Long-context Attention in Near-Linear Time — Insu Han et al., 2023https://arxiv.org/abs/2310.0586917. Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory — Aydar Bulatov et al., 2024https://ojs.aaai.org/index.php/AAAI/article/download/29722/3123918. AI Post Transformers: ScoutAttention for Efficient KV Cache Offloading — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-24-scoutattention-for-efficient-kv-cache-of-b26699.mp319. AI Post Transformers: JANUS for Scalable MoE Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-janus-for-scalable-moe-inference-78ae30.mp320. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp321. AI Post Transformers: FlexGen: High-Throughput LLM Inference on a Single GPU — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/flexgen-high-throughput-llm-inference-on-a-single-gpu/22. AI Post Transformers: Stochastic KV Routing for Cache Sharing — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-29-stochastic-kv-routing-for-cache-sharing-5fef63.mp323. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp311 min
- Opaque Serial Depth and Chain-of-Thought LimitsThis episode explores a 2026 paper on whether language models can keep doing long, step-by-step reasoning internally or eventually need to expose some of that reasoning in visible chain-of-thought tokens. It explains the paper’s core idea of opaque serial depth, or a model’s hidden reasoning horizon, and argues that this is a better safety-relevant measure than raw model size, parameter count, or informal layer counting. The discussion connects that metric to circuit complexity, fixed-precision computation, and transformer internals, showing why models can perform huge amounts of parallel work in one pass yet still face structural limits on long private sequential reasoning. Listeners would find it interesting because it sharpens a major AI safety question: whether monitoring visible reasoning can meaningfully constrain powerful models, and where that hope may break down.Sources:1. Opaque Serial Depth and Chain-of-Thought Limitshttps://arxiv.org/pdf/2603.097862. Quantifying the Necessity of Chain of Thought through Opaque Serial Depth — Jonah Brown-Cohen, David Lindner, Rohin Shah, 2026https://scholar.google.com/scholar?q=Quantifying+the+Necessity+of+Chain+of+Thought+through+Opaque+Serial+Depth3. Chain of Thought Empowers Transformers to Solve Inherently Serial Problems — Zhiyuan Li, Hong Liu, Denny Zhou, Tengyu Ma, 2024https://scholar.google.com/scholar?q=Chain+of+Thought+Empowers+Transformers+to+Solve+Inherently+Serial+Problems4. Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety — Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Mark Chen, Rohin Shah, et al., 2025https://scholar.google.com/scholar?q=Chain+of+Thought+Monitorability%3A+A+New+and+Fragile+Opportunity+for+AI+Safety5. When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors — Scott Emmons, Erik Jenner, David K. Elson, Rif A. Saurous, Senthooran Rajamanoharan, Heng Chen, Irhum Shafkat, Rohin Shah, 2025https://scholar.google.com/scholar?q=When+Chain+of+Thought+is+Necessary%2C+Language+Models+Struggle+to+Evade+Monitors6. Saturated Transformers are Constant-Depth Threshold Circuits — William Merrill, Ashish Sabharwal, Noah A. Smith, 2022https://scholar.google.com/scholar?q=Saturated+Transformers+are+Constant-Depth+Threshold+Circuits7. The Parallelism Tradeoff: Limitations of Log-Precision Transformers — William Merrill, Ashish Sabharwal, 2023https://scholar.google.com/scholar?q=The+Parallelism+Tradeoff%3A+Limitations+of+Log-Precision+Transformers8. The Expressive Power of Transformers with Chain of Thought — William Merrill, Ashish Sabharwal, 2024https://scholar.google.com/scholar?q=The+Expressive+Power+of+Transformers+with+Chain+of+Thought9. Theoretical Limitations of Self-Attention in Neural Sequence Models — Michael Hahn, 2020https://scholar.google.com/scholar?q=Theoretical+Limitations+of+Self-Attention+in+Neural+Sequence+Models10. Continuous Chain of Thought Enables Parallel Exploration and Reasoning — Halil Alperen Gozeten, M. Emrullah Ildiz, Xuechen Zhang, Hrayr Harutyunyan, Ankit Singh Rawat, Samet Oymak, 2025https://scholar.google.com/scholar?q=Continuous+Chain+of+Thought+Enables+Parallel+Exploration+and+Reasoning11. Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought — Siddharth Boppana, Annabel Ma, Max Loeffler, Raphael Sarfati, Eric Bigelow, Atticus Geiger, Owen Lewis, Jack Merullo, 2026https://scholar.google.com/scholar?q=Reasoning+Theater%3A+Disentangling+Model+Beliefs+from+Chain-of-Thought12. Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps — Martin Tutek et al., 2025https://scholar.google.com/scholar?q=Measuring+Chain+of+Thought+Faithfulness+by+Unlearning+Reasoning+Steps13. Counterfactual Simulation Training for Chain-of-Thought Faithfulness — Peter Hase, Christopher Potts, 2026https://scholar.google.com/scholar?q=Counterfactual+Simulation+Training+for+Chain-of-Thought+Faithfulness14. Why Models Know But Don't Say: Chain-of-Thought Faithfulness Divergence Between Thinking Tokens and Answers in Open-Weight Reasoning Models — Richard J. Young, 2026https://scholar.google.com/scholar?q=Why+Models+Know+But+Don%27t+Say%3A+Chain-of-Thought+Faithfulness+Divergence+Between+Thinking+Tokens+and+Answers+in+Open-Weight+Reasoning+Models15. Reasoning with Latent Thoughts: On the Power of Looped Transformers — Nikunj Saunshi et al., 2025https://scholar.google.com/scholar?q=Reasoning+with+Latent+Thoughts%3A+On+the+Power+of+Looped+Transformers16. Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding — Haolin Chen et al., 2024https://scholar.google.com/scholar?q=Language+Models+are+Hidden+Reasoners%3A+Unlocking+Latent+Reasoning+Capabilities+via+Self-Rewarding17. Efficient Post-Training Refinement of Latent Reasoning in Large Language Models — Xinyuan Wang et al., 2025https://scholar.google.com/scholar?q=Efficient+Post-Training+Refinement+of+Latent+Reasoning+in+Large+Language+Models18. AI Post Transformers: Reasoning Theater and Unfaithful Chain-of-Thought — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-reasoning-theater-and-unfaithful-chain-o-a4507e.mp319. AI Post Transformers: Generative Recursive Reasoning in Latent Space — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-21-generative-recursive-reasoning-in-latent-a9371d.mp320. AI Post Transformers: How Models Detect Hidden Activation Steering — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-08-how-models-detect-hidden-activation-stee-577f73.mp321. AI Post Transformers: Latent Space as a New Computational Paradigm — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-05-latent-space-as-a-new-computational-para-810f39.mp322. AI Post Transformers: Neural Computers as Learned Latent Runtimes — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-11-neural-computers-as-learned-latent-runti-9fa282.mp3Interactive Visualization: Opaque Serial Depth and Chain-of-Thought Limits14 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.