Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- MalGEN: Multi-Agent AI for Red Teaming MalwareThis episode explores MalGEN, a multi-agent AI framework developed by researchers at IIT Kanpur that autonomously generates novel, functional malware capable of evading modern detection systems. The discussion examines why LLM-generated malware represents a qualitative shift beyond traditional polymorphic and metamorphic techniques — rather than mutating a fixed payload syntactically, LLMs reason about semantic intent and produce entirely new code that achieves the same effect through different computational paths. A central focus is MalGEN's alignment with MITRE ATT&CK tactics, techniques, and procedures, meaning the generated malware maps to documented real-world intrusion patterns rather than merely bypassing signature databases. The hosts pressure-test the paper's red teaming justification — framing MalGEN as a defensive stress-testing tool — while examining its most unsettling capability: automating sandbox-aware, environment-detecting evasion previously requiring nation-state-level expertise. The conversation anchors on the dual-use tension at the core of publishing a reproducible malware generation framework under academic cover.Sources:1. MalGEN: A Generative Agent Framework for Modeling Malicious Software in Cybersecurity — Bikash Saha, Sandeep Kumar Shukla, 2025http://arxiv.org/abs/2506.075862. https://arxiv.org/pdf/2510.238833. https://arxiv.org/pdf/2601.052934. https://arxiv.org/pdf/2508.056745. Evaluating the Cybersecurity Capabilities of LLMs: A Comprehensive Study — Bhatt, M., Chennabasappa, S., Nikolaidis, C., et al. (Meta), 2023https://scholar.google.com/scholar?q=Evaluating+the+Cybersecurity+Capabilities+of+LLMs%3A+A+Comprehensive+Study6. From Chatbots to Phishbots? Phishing Scam Generation in Commercial Large Language Models — Heiding, F., Schneier, B., Vishwanath, A., Bernstein, J., Park, P.S., 2024https://scholar.google.com/scholar?q=From+Chatbots+to+Phishbots%3F+Phishing+Scam+Generation+in+Commercial+Large+Language+Models7. PentestGPT: An LLM-Empowered Automatic Penetration Testing Framework — Deng, G., Liu, Y., Mayoral-Vilches, V., et al., 2024https://scholar.google.com/scholar?q=PentestGPT%3A+An+LLM-Empowered+Automatic+Penetration+Testing+Framework8. LLM4Decompile: Decompiling Binary Code with Large Language Models — Tan, Z., Ma, H., Xu, H., et al., 2024https://scholar.google.com/scholar?q=LLM4Decompile%3A+Decompiling+Binary+Code+with+Large+Language+Models9. LLM Agents can Autonomously Exploit One-Day Vulnerabilities — Fang, R., Bindu, R., Gupta, A., Kang, D., Boneh, D., 2024https://scholar.google.com/scholar?q=LLM+Agents+can+Autonomously+Exploit+One-Day+Vulnerabilities10. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned — Ganguli, D., Lovitt, L., Kernion, J., et al. (Anthropic), 2022https://scholar.google.com/scholar?q=Red+Teaming+Language+Models+to+Reduce+Harms%3A+Methods%2C+Scaling+Behaviors%2C+and+Lessons+Learned11. Trojan Detection Benchmark (TrojAI): Evaluating Backdoor Defenses on Neural Networks — Karra, K., Ashcraft, C., Fendley, N. (IARPA / Johns Hopkins APL), 2020https://scholar.google.com/scholar?q=Trojan+Detection+Benchmark+%28TrojAI%29%3A+Evaluating+Backdoor+Defenses+on+Neural+Networks12. Evading Machine Learning Malware Detection — Grosse, K., Papernot, N., Manoharan, P., Backes, M., McDaniel, P., 2017https://scholar.google.com/scholar?q=Evading+Machine+Learning+Malware+Detection13. Malware Detection by Eating a Whole EXE — Raff, E., Barker, J., Sylvester, J., Brandon, R., Catanzaro, B., Nicholas, C., 2018https://scholar.google.com/scholar?q=Malware+Detection+by+Eating+a+Whole+EXE14. Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executables — Kolosnjaji, B., Demontis, A., Biggio, B., Maiorca, D., Giacinto, G., Eckert, C., Roli, F., 2018https://scholar.google.com/scholar?q=Adversarial+Malware+Binaries%3A+Evading+Deep+Learning+for+Malware+Detection+in+Executables15. DQEAF: Malware Adversarial Examples Generation with Reinforcement Learning — Fang, Y., Liu, Y., Huang, C., Liu, L., 2020https://scholar.google.com/scholar?q=DQEAF%3A+Malware+Adversarial+Examples+Generation+with+Reinforcement+Learning16. MalGAN: Generating Adversarial Malware Examples — Hu, W., Tan, Y., 2017https://scholar.google.com/scholar?q=MalGAN%3A+Generating+Adversarial+Malware+Examples17. IDSGAN: Generative Adversarial Networks for Attack Generation against Intrusion Detection — Lin, Z., Shi, Y., Xue, Z., 2022https://scholar.google.com/scholar?q=IDSGAN%3A+Generative+Adversarial+Networks+for+Attack+Generation+against+Intrusion+Detection18. Synthetic Data Generation for Cybersecurity: Challenges and Opportunities — Ring, M., Wunderlich, S., Scheuring, D., Landes, D., Hotho, A., 2019https://scholar.google.com/scholar?q=Synthetic+Data+Generation+for+Cybersecurity%3A+Challenges+and+Opportunities19. Generative Adversarial Networks for Black-Box API Attack on Web Application Firewalls — Duy, P.T., Khoa, N.M., Hoa, N.T., et al., 2022https://scholar.google.com/scholar?q=Generative+Adversarial+Networks+for+Black-Box+API+Attack+on+Web+Application+Firewalls20. Evading Anti-Malware Engines With Deep Reinforcement Learning — Fang, Z. et al., 2019https://scholar.google.com/scholar?q=Evading+Anti-Malware+Engines+With+Deep+Reinforcement+Learning21. Do LLMs Dream of Malware? Assessing the Practical Capability of LLMs to Generate Functional Malware — Pa Pa, Y. M. et al., 2023https://scholar.google.com/scholar?q=Do+LLMs+Dream+of+Malware%3F+Assessing+the+Practical+Capability+of+LLMs+to+Generate+Functional+Malware22. MITRE ATT&CK: Design and Philosophy — Strom, B. et al., 2018https://scholar.google.com/scholar?q=MITRE+ATT%26CK%3A+Design+and+Philosophy23. LLM4Fuzz: Guided Fuzzing of Smart Contracts with Large Language Models — Various, 2024https://scholar.google.com/scholar?q=LLM4Fuzz%3A+Guided+Fuzzing+of+Smart+Contracts+with+Large+Language+Models24. Beyond the sandbox: Leveraging symbolic execution for evasive malware classification — approximate, ~2024, 2024https://scholar.google.com/scholar?q=Beyond+the+sandbox%3A+Leveraging+symbolic+execution+for+evasive+malware+classification25. Unveiling the dynamic landscape of malware sandboxing: A comprehensive review — approximate, ~2024, 2024https://scholar.google.com/scholar?q=Unveiling+the+dynamic+landscape+of+malware+sandboxing%3A+A+comprehensive+review26. TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems — approximate, ~2024-2025, 2025https://scholar.google.com/scholar?q=TAMAS%3A+Benchmarking+Adversarial+Risks+in+Multi-Agent+LLM+Systems27. Red-teaming LLM multi-agent systems via communication attacks — approximate, ~2024-2025, 2025https://scholar.google.com/scholar?q=Red-teaming+LLM+multi-agent+systems+via+communication+attacks28. Assessing LLMs in malicious code deobfuscation of real-world malware campaigns — approximate, ~2024-2025, 2024https://scholar.google.com/scholar?q=Assessing+LLMs+in+malicious+code+deobfuscation+of+real-world+malware+campaigns29. Certifying accuracy, privacy, and robustness of ML-based malware detection — approximate, ~2024-2025, 2025https://scholar.google.com/scholar?q=Certifying+accuracy%2C+privacy%2C+and+robustness+of+ML-based+malware+detection30. AI Post Transformers: Petri: Accelerating AI Safety Auditing — Hal Turing & Dr. Ada Shannon, Fri,https://podcasters.spotify.com/pod/show/12146088098/episodes/Petri-Accelerating-AI-Safety-Auditing-e39boei31. AI Post Transformers: Bloom: an open source tool for automated behavioral evaluations — Hal Turing & Dr. Ada Shannon, Tue,https://podcasters.spotify.com/pod/show/12146088098/episodes/Bloom-an-open-source-tool-for-automated-behavioral-evaluations-e3fi1ge32. AI Post Transformers: Agentic Reasoning for Large Language Models: A Comprehensive Roadmap — Hal Turing & Dr. Ada Shannon, Sat,https://podcasters.spotify.com/pod/show/12146088098/episodes/Agentic-Reasoning-for-Large-Language-Models-A-Comprehensive-Roadmap-e3e43ruInteractive Visualization: MalGEN: Multi-Agent AI for Red Teaming Malware0 min
- NVIDIA Nemotron 3 Hybrid SSM Transformer ArchitectureThis episode explores NVIDIA's Nemotron 3 white paper, which introduces a three-tier model family (Nano, Super, Ultra) built on a hybrid architecture combining Mamba-2 structured state space layers with Mixture-of-Experts routing, targeting simultaneous state-of-the-art accuracy, one-million-token context, and substantially higher inference throughput than comparable dense Transformer MoE models. The discussion traces how Mamba-2's fixed-size recurrent state eliminates the KV cache's linear memory growth — the central bottleneck for long-context and agentic workloads — and explains NVIDIA's novel LatentMoE extension, which projects tokens into a reduced latent dimension before expert routing to cut communication costs while activating more experts per token. Multi-Token Prediction from Meta FAIR appears as a training accelerant, predicting multiple future tokens simultaneously to improve both training efficiency and generation speed, while NVFP4, NVIDIA's 4-bit floating-point training format used for Super and Ultra, raises open questions about numerical stability during post-training. Listeners interested in how recent theoretical work on state space duality translates into a production-scale model family, or in the architectural tradeoffs enabling practical multi-agent pipelines at scale, will find the episode a concrete and technically grounded case study.Sources:1. NVIDIA Nemotron 3: Efficient and Open Intelligence — NVIDIA, :, Aaron Blakeman, Aaron Grattafiori, Aarti Basant, Abhibha Gupta, Abhinav Khattar, Adi Renduchintala, Aditya Vavre, Akanksha Shukla, Akhiad Bercovich, Aleksander Ficek, Aleksandr Shaposhnikov, Alex Kondratenko, Alexander Bukharin, Alexandre Milesi, Ali Taghibakhshi, Alisa Liu, Amelia Barton, Ameya Sunil Mahabaleshwarkar, Amir Klein, Amit Zuker, Amnon Geifman, Amy Shen, Anahita Bhiwandiwalla, Andrew Tao, Anjulie Agrusa, Ankur Verma, Ann Guan, Anubhav Mandarwal, Arham Mehta, Ashwath Aithal, Ashwin Poojary, Asif Ahamed, Asit Mishra, Asma Kuriparambil Thekkumpate, Ayush Dattagupta, Banghua Zhu, Bardiya Sadeghi, Barnaby Simkin, Ben Lanir, Benedikt Schifferer, Besmira Nushi, Bilal Kartal, Bita Darvish Rouhani, Boris Ginsburg, Brandon Norick, Brandon Soubasis, Branislav Kisacanin, Brian Yu, Bryan Catanzaro, Carlo del Mundo, Chantal Hwang, Charles Wang, Cheng-Ping Hsieh, Chenghao Zhang, Chenhan Yu, Chetan Mungekar, Chintan Patel, Chris Alexiuk, Christopher Parisien, Collin Neale, Cyril Meurillon, Damon Mosk-Aoyama, Dan Su, Dane Corneil, Daniel Afrimi, Daniel Lo, Daniel Rohrer, Daniel Serebrenik, Daria Gitman, Daria Levy, Darko Stosic, David Mosallanezhad, Deepak Narayanan, Dhruv Nathawani, Dima Rekesh, Dina Yared, Divyanshu Kakwani, Dong Ahn, Duncan Riach, Dusan Stosic, Edgar Minasyan, Edward Lin, Eileen Long, Eileen Peters Long, Elad Segal, Elena Lantz, Ellie Evans, Elliott Ning, Eric Chung, Eric Harper, Eric Tramel, Erick Galinkin, Erik Pounds, Evan Briones, Evelina Bakhturina, Evgeny Tsykunov, Faisal Ladhak, Fay Wang, Fei Jia, Felipe Soares, Feng Chen, Ferenc Galko, Frank Sun, Frankie Siino, Gal Hubara Agam, Ganesh Ajjanagadde, Gantavya Bhatt, Gargi Prasad, George Armstrong, Gerald Shen, Gorkem Batmaz, Grigor Nalbandyan, Haifeng Qian, Harsh Sharma, Hayley Ross, Helen Ngo, Herbert Hum, Herman Sahota, Hexin Wang, Himanshu Soni, Hiren Upadhyay, Huizi Mao, Huy C Nguyen, Huy Q Nguyen, Iain Cunningham, Ido Galil, Ido Shahaf, Igor Gitman, Ilya Loshchilov, Itamar Schen, Itay Levy, Ivan Moshkov, Izik Golan, Izzy Putterman, Jan Kautz, Jane Polak Scowcroft, Jared Casper, Jatin Mitra, Jeffrey Glick, Jenny Chen, Jesse Oliver, Jian Zhang, Jiaqi Zeng, Jie Lou, Jimmy Zhang, Jinhang Choi, Jining Huang, Joey Conway, Joey Guman, John Kamalu, Johnny Greco, Jonathan Cohen, Joseph Jennings, Joyjit Daw, Julien Veron Vialard, Junkeun Yi, Jupinder Parmar, Kai Xu, Kan Zhu, Kari Briski, Katherine Cheung, Katherine Luna, Keith Wyss, Keshav Santhanam, Kevin Shih, Kezhi Kong, Khushi Bhardwaj, Kirthi Shankar, Krishna C. Puvvada, Krzysztof Pawelec, Kumar Anik, Lawrence McAfee, Laya Sleiman, Leon Derczynski, Li Ding, Lizzie Wei, Lucas Liebenwein, Luis Vega, Maanu Grover, Maarten Van Segbroeck, Maer Rodrigues de Melo, Mahdi Nazemi, Makesh Narsimhan Sreedhar, Manoj Kilaru, Maor Ashkenazi, Marc Romeijn, Marcin Chochowski, Mark Cai, Markus Kliegl, Maryam Moosaei, Matt Kulka, Matvei Novikov, Mehrzad Samadi, Melissa Corpuz, Mengru Wang, Meredith Price, Michael Andersch, Michael Boone, Michael Evans, Miguel Martinez, Mikail Khona, Mike Chrzanowski, Minseok Lee, Mohammad Dabbah, Mohammad Shoeybi, Mostofa Patwary, Nabin Mulepati, Najeeb Nabwani, Natalie Hereth, Nave Assaf, Negar Habibi, Neta Zmora, Netanel Haber, Nicola Sessions, Nidhi Bhatia, Nikhil Jukar, Nikki Pope, Nikolai Ludwig, Nima Tajbakhsh, Nir Ailon, Nirmal Juluru, Nishant Sharma, Oleksii Hrinchuk, Oleksii Kuchaiev, Olivier Delalleau, Oluwatobi Olabiyi, Omer Ullman Argov, Omri Puny, Oren Tropp, Ouye Xie, Parth Chadha, Pasha Shamis, Paul Gibbons, Pavlo Molchanov, Pawel Morkisz, Peter Dykas, Peter Jin, Pinky Xu, Piotr Januszewski, Pranav Prashant Thombre, Prasoon Varshney, Pritam Gundecha, Przemek Tredak, Qing Miao, Qiyu Wan, Rabeeh Karimi Mahabadi, Rachit Garg, Ran El-Yaniv, Ran Zilberstein, Rasoul Shafipour, Rich Harang, Rick Izzo, Rima Shahbazyan, Rishabh Garg, Ritika Borkar, Ritu Gala, Riyad Islam, Robert Hesse, Roger Waleffe, Rohit Watve, Roi Koren, Ruoxi Zhang, Russell Hewett, Russell J. Hewett, Ryan Prenger, Ryan Timbrook, Sadegh Mahdavi, Sahil Modi, Samuel Kriman, Sangkug Lim, Sanjay Kariyappa, Sanjeev Satheesh, Saori Kaji, Satish Pasumarthi, Saurav Muralidharan, Sean Narentharen, Sean Narenthiran, Seonmyeong Bak, Sergey Kashirsky, Seth Poulos, Shahar Mor, Shanmugam Ramasamy, Shantanu Acharya, Shaona Ghosh, Sharath Turuvekere Sreenivas, Shelby Thomas, Shiqing Fan, Shreya Gopal, Shrimai Prabhumoye, Shubham Pachori, Shubham Toshniwal, Shuoyang Ding, Siddharth Singh, Simeng Sun, Smita Ithape, Somshubra Majumdar, Soumye Singhal, Stas Sergienko, Stefania Alborghetti, Stephen Ge, Sugam Dipak Devare, Sumeet Kumar Barua, Suseella Panguluri, Suyog Gupta, Sweta Priyadarshi, Syeda Nahida Akter, Tan Bui, Teodor-Dumitru Ene, Terry Kong, Thanh Do, Tijmen Blankevoort, Tim Moon, Tom Balough, Tomer Asida, Tomer Bar Natan, Tomer Ronen, Tugrul Konuk, Twinkle Vashishth, Udi Karpas, Ushnish De, Vahid Noorozi, Vahid Noroozi, Venkat Srinivasan, Venmugil Elango, Victor Cui, Vijay Korthikanti, Vinay Rao, Vitaly Kurin, Vitaly Lavrukhin, Vladimir Anisimov, Wanli Jiang, Wasi Uddin Ahmad, Wei Du, Wei Ping, Wenfei Zhou, Will Jennings, William Zhang, Wojciech Prazuch, Xiaowei Ren, Yashaswi Karnati, Yejin Choi, Yev Meyer, Yi-Fu Wu, Yian Zhang, Yigong Qin, Ying Lin, Yonatan Geifman, Yonggan Fu, Yoshi Subara, Yoshi Suhara, Yubo Gao, Zach Moshe, Zhen Dong, Zhongbo Zhu, Zihan Liu, Zijia Chen, Zijie Yan, 2025http://arxiv.org/abs/2512.208562. Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Model Parameters — Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar (Google DeepMind), 2024https://scholar.google.com/scholar?q=Scaling+LLM+Test-Time+Compute+Optimally+Can+be+More+Effective+than+Scaling+Model+Parameters3. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek-AI (Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and many others), 2025https://scholar.google.com/scholar?q=DeepSeek-R1%3A+Incentivizing+Reasoning+Capability+in+LLMs+via+Reinforcement+Learning4. Thinking Tokens for Language Modeling — David Herel, Tomas Mikolov (Czech Institute of Informatics / FAIR), 2023https://scholar.google.com/scholar?q=Thinking+Tokens+for+Language+Modeling5. Training Large Language Models to Reason in a Continuous Latent Space (Coconut) — Shibo Hao, Sainbayar Sukhbaatar, Jason Weston, Yuandong Tian, Zhiting Hu (Meta FAIR), 2024https://scholar.google.com/scholar?q=Training+Large+Language+Models+to+Reason+in+a+Continuous+Latent+Space+%28Coconut%296. Don't Think Too Much: Reasoning Budget Control for Large Reasoning Models — Xuan He, Zhenyu He, Qingyu Meng, Yonghao Zhong, Zhonglin Shi, Fandong Meng, Jie Zhou (Tencent AI Lab / WeChat AI), 2025https://scholar.google.com/scholar?q=Don%27t+Think+Too+Much%3A+Reasoning+Budget+Control+for+Large+Reasoning+Models7. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — Dao, T. and Gu, A., 2024https://scholar.google.com/scholar?q=Transformers+are+SSMs%3A+Generalized+Models+and+Efficient+Algorithms+Through+Structured+State+Space+Duality8. Jamba: A Hybrid Transformer-Mamba Language Model — Lieber et al., AI21 Labs, 2024https://scholar.google.com/scholar?q=Jamba%3A+A+Hybrid+Transformer-Mamba+Language+Model9. Better & Faster Large Language Models via Multi-Token Prediction — Gloeckle et al., 2024https://scholar.google.com/scholar?q=Better+%26+Faster+Large+Language+Models+via+Multi-Token+Prediction10. RULER: What's the Real Context Size of Your Long-Context Language Models? — Hsieh et al., 2024https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F11. DeepSeek-V3 Technical Report — DeepSeek-AI, 2025https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report12. Mixtral of Experts — Jiang et al., 2024https://scholar.google.com/scholar?q=Mixtral+of+Experts13. In-Context Learning with Long-Context Models: An In-Depth Exploration — Bertsch et al., 2024https://scholar.google.com/scholar?q=In-Context+Learning+with+Long-Context+Models%3A+An+In-Depth+Exploration14. Attn-QAT: 4-Bit Attention With Quantization-Aware Training — approximate, 2024–2025, 2024-2025https://scholar.google.com/scholar?q=Attn-QAT%3A+4-Bit+Attention+With+Quantization-Aware+Training15. FP4 All the Way: Fully Quantized Training of LLMs — approximate, 2024–2025, 2024-2025https://scholar.google.com/scholar?q=FP4+All+the+Way%3A+Fully+Quantized+Training+of+LLMs16. Optimizing Large Language Model Training Using FP4 Quantization — approximate, 2024–2025, 2024-2025https://scholar.google.com/scholar?q=Optimizing+Large+Language+Model+Training+Using+FP4+Quantization17. Speculative Decoding and Beyond: An In-Depth Survey of Techniques — approximate, 2024–2025, 2024-2025https://scholar.google.com/scholar?q=Speculative+Decoding+and+Beyond%3A+An+In-Depth+Survey+of+Techniques18. Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts — approximate, 2024–2025, 2024-2025https://scholar.google.com/scholar?q=Latent+Prototype+Routing%3A+Achieving+Near-Perfect+Load+Balancing+in+Mixture-of-Experts19. AI Post Transformers: Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-04-2-papers-combined-217168.mp320. AI Post Transformers: Switch Transformers: Trillion Parameter Models with Sparsity — Hal Turing & Dr. Ada Shannon, Wed,https://podcasters.spotify.com/pod/show/12146088098/episodes/Switch-Transformers-Trillion-Parameter-Models-with-Sparsity-e373fd321. AI Post Transformers: GLM-5: Transitioning from Vibe Coding to Agentic Engineering — Hal Turing & Dr. Ada Shannon, Fri,https://podcasters.spotify.com/pod/show/12146088098/episodes/GLM-5-Transitioning-from-Vibe-Coding-to-Agentic-Engineering-e3fbfls22. AI Post Transformers: Kimi Linear: Efficient Expressive Attention Architecture — Hal Turing & Dr. Ada Shannon, Sun,https://podcasters.spotify.com/pod/show/12146088098/episodes/Kimi-Linear-Efficient-Expressive-Attention-Architecture-e3aclec23. AI Post Transformers: TailorKV: Hybrid KV Cache Compression for LLMs — Hal Turing & Dr. Ada Shannon, Wed,https://podcasters.spotify.com/pod/show/12146088098/episodes/TailorKV-Hybrid-KV-Cache-Compression-for-LLMs-e38bmv224. AI Post Transformers: AWQ: On-Device LLM Compression and Acceleration — Hal Turing & Dr. Ada Shannon, Mon,https://podcasters.spotify.com/pod/show/12146088098/episodes/AWQ-On-Device-LLM-Compression-and-Acceleration-e388njr25. AI Post Transformers: LFM2-8B-A1B: Efficient On-Device Mixture-of-Experts — Hal Turing & Dr. Ada Shannon, Sun,https://podcasters.spotify.com/pod/show/12146088098/episodes/LFM2-8B-A1B-Efficient-On-Device-Mixture-of-Experts-e3a2oh2Interactive Visualization: NVIDIA Nemotron 3 Hybrid SSM Transformer Architecture0 min
- Tokenization Bias: The Hidden Flaw Breaking Language ModelsHal Turing and Dr. Ada Shannon open the episode by confronting a structural flaw that has been hiding in plain sight since the transformer era began: tokenization bias. The episode centers on "Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles" by Buu Phan, Brandon Amos, Itai Gat, Marton Havasi, Matthew Muckley, and Karen Ullrich (ICLR 2025), which formally proves that a tokenized model and a byte-level model can be statistically equivalent and still produce wildly different predictions for the same next character. The hosts trace the origins of the problem through BPE's introduction by Rico Sennrich, Barry Haddow, and Alexandra Birch in 2016 and its industrialization via Kudo and Richardson's SentencePiece in 2018 — a library now frozen into the spine of LLaMA, Mistral, Gemma, and most open-source models not in the OpenAI lineage.The discussion sharpens around fill-in-the-middle prompting, the paradigm introduced by Mohammad Bavarian and colleagues at OpenAI in 2022 and now embedded in every major code completion tool from GitHub Copilot to StarCoder. Shannon walks through the paper's central example: a code completion scenario where the correct next character receives a probability of exactly zero — not a rounding artifact but a structural impossibility, because the tokenizer has carved up the prompt in a way that makes the right answer unreachable in token-space. Turing challenges the framing, arguing that byte-level alternatives like ByT5 and MegaByte existed and BPE was an informed trade-off against the three-to-eight times sequence length penalty that raw bytes impose on attention compute. Shannon holds the line: the point is not that BPE was a mistake but that its systematic bias was never formally characterized until now, and the Byte-Token Representation Lemma finally gives the field the mathematical language to name and measure it.The episode closes by introducing the second paper from the episode's pairing — Minixhofer et al.'s NeurIPS 2025 work on cross-tokenizer knowledge distillation — which attacks the tokenizer barrier from the training side rather than the inference side. Where Phan et al. offer a zero-shot correction algorithm that recovers 18% on fill-in-the-middle coding benchmarks without any retraining, Minixhofer et al. enable knowledge transfer between models with fundamentally incompatible vocabularies, breaking the assumption that distillation requires shared tokenization. Together the two papers sketch a trajectory where tokenization becomes a transparent implementation detail rather than an architectural constraint that determines what a model can and cannot express.Sources:1. Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles — Buu Phan, Brandon Amos, Itai Gat, Marton Havasi, Matthew Muckley, Karen Ullrich, 2024http://arxiv.org/abs/2410.093032. https://arxiv.org/pdf/2503.200833. Neural Machine Translation of Rare Words with Subword Units — Rico Sennrich, Barry Haddow, Alexandra Birch, 2016https://scholar.google.com/scholar?q=Neural+Machine+Translation+of+Rare+Words+with+Subword+Units4. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing — Taku Kudo, John Richardson, 2018https://scholar.google.com/scholar?q=SentencePiece%3A+A+simple+and+language+independent+subword+tokenizer+and+detokenizer+for+Neural+Text+Processing5. Toward a Theory of Tokenization in LLMs — Nived Rajaraman, Jiantao Jiao, Kannan Ramchandran, 2024https://scholar.google.com/scholar?q=Toward+a+Theory+of+Tokenization+in+LLMs6. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models — Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, Iryna Gurevych, 2021https://scholar.google.com/scholar?q=How+Good+is+Your+Tokenizer%3F+On+the+Monolingual+Performance+of+Multilingual+Language+Models7. ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models — Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel, 2022https://scholar.google.com/scholar?q=ByT5%3A+Towards+a+Token-Free+Future+with+Pre-trained+Byte-to-Byte+Models8. MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers — Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, Mike Lewis, 2023https://scholar.google.com/scholar?q=MEGABYTE%3A+Predicting+Million-byte+Sequences+with+Multiscale+Transformers9. CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation — Jonathan H. Clark, Dan Garrette, Iulia Turc, John Wieting, 2022https://scholar.google.com/scholar?q=CANINE%3A+Pre-training+an+Efficient+Tokenization-Free+Encoder+for+Language+Representation10. Charformer: Fast Character Transformers via Gradient-Based Subword Tokenization — Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Kornblith, Cecelia Zhang, Donald Metzler, Mostafa Dehghani, 2022https://scholar.google.com/scholar?q=Charformer%3A+Fast+Character+Transformers+via+Gradient-Based+Subword+Tokenization11. Efficient Training of Language Models to Fill in the Middle — Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, Mark Chen, 2022https://scholar.google.com/scholar?q=Efficient+Training+of+Language+Models+to+Fill+in+the+Middle12. Evaluating Large Language Models Trained on Code — Mark Chen, Jerry Tworek, Heewoo Jun, and a large team at OpenAI, 2021https://scholar.google.com/scholar?q=Evaluating+Large+Language+Models+Trained+on+Code13. Code Llama: Open Foundation Models for Code — Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, and others at Meta AI, 2023https://scholar.google.com/scholar?q=Code+Llama%3A+Open+Foundation+Models+for+Code14. StarCoder: May the Source Be With You! — Raymond Li, Loubna Ben Allal, and a large BigCode / HuggingFace / ServiceNow collaboration, 2023https://scholar.google.com/scholar?q=StarCoder%3A+May+the+Source+Be+With+You%2115. Training Products of Experts by Minimizing Contrastive Divergence — Geoffrey E. Hinton, 2002https://scholar.google.com/scholar?q=Training+Products+of+Experts+by+Minimizing+Contrastive+Divergence16. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou, 2023https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models17. Knowledge Fusion of Large Language Models — Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, Shuming Shi, 2024https://scholar.google.com/scholar?q=Knowledge+Fusion+of+Large+Language+Models18. LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion — Dongfu Jiang, Xiang Ren, Bill Yuchen Lin, 2023https://scholar.google.com/scholar?q=LLM-Blender%3A+Ensembling+Large+Language+Models+with+Pairwise+Ranking+and+Generative+Fusion19. Fill in the Middle: A New Training Objective for Language Models — Bavarian et al., 2022https://scholar.google.com/scholar?q=Fill+in+the+Middle%3A+A+New+Training+Objective+for+Language+Models20. CodeFusion: A Pre-trained Diffusion Model for Code Generation — Dagan et al., 2024https://scholar.google.com/scholar?q=CodeFusion%3A+A+Pre-trained+Diffusion+Model+for+Code+Generation21. Is Tokenization More Than Compression? — Rajaraman et al., 2024https://scholar.google.com/scholar?q=Is+Tokenization+More+Than+Compression%3F22. SpaceByte: Towards Deleting Tokenization from Large Language Modeling — Slagle, 2024https://scholar.google.com/scholar?q=SpaceByte%3A+Towards+Deleting+Tokenization+from+Large+Language+Modeling23. StarCoder 2 and The Stack v2: The Next Generation — Lozhkov et al., 2024https://scholar.google.com/scholar?q=StarCoder+2+and+The+Stack+v2%3A+The+Next+Generation24. Byte Latent Transformer: Patches Scale Better Than Tokens — Pagnoni et al. (Meta AI), 2024https://scholar.google.com/scholar?q=Byte+Latent+Transformer%3A+Patches+Scale+Better+Than+Tokens25. Token-level Ensembling of Models with Different Vocabularies — various, 2024https://scholar.google.com/scholar?q=Token-level+Ensembling+of+Models+with+Different+Vocabularies26. Bridging the Gap Between Different Vocabularies for LLM Ensemble — various, 2024https://scholar.google.com/scholar?q=Bridging+the+Gap+Between+Different+Vocabularies+for+LLM+Ensemble27. Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies — various, 2024https://scholar.google.com/scholar?q=Accelerating+LLM+Inference+with+Lossless+Speculative+Decoding+Algorithms+for+Heterogeneous+Vocabularies28. Bridging Developer Instructions and Code Completion Through Instruction-Aware Fill-in-the-Middle Paradigm — various, 2024https://scholar.google.com/scholar?q=Bridging+Developer+Instructions+and+Code+Completion+Through+Instruction-Aware+Fill-in-the-Middle+Paradigm29. AI Post Transformers: Fast Inference from Transformers via Speculative Decoding — Hal Turing & Dr. Ada Shannon, Sat,https://podcasters.spotify.com/pod/show/12146088098/episodes/Fast-Inference-from-Transformers-via-Speculative-Decoding-e3foclv30. AI Post Transformers: Multiagent Debate Improves Language Model Reasoning — Hal Turing & Dr. Ada Shannon, Fri,https://podcasters.spotify.com/pod/show/12146088098/episodes/Multiagent-Debate-Improves-Language-Model-Reasoning-e36kfd4Interactive Visualization: Tokenization Bias: The Hidden Flaw Breaking Language Models0 min
- Systematic Characterization of LLM Inference on GPUsHal Turing and Dr. Ada Shannon examine "A Systematic Characterization of LLM Inference on GPUs" by Haonan Wang and colleagues from the Institute of Computing Technology at the Chinese Academy of Sciences, Zhejiang Lab, China Telecom, and Peking University. Dr. Shannon opens by framing the paper's central contribution: the inference literature is fragmented across power consumption studies, kernel profiling, quantization analysis, and serving schedulers, none sharing a common analytical vocabulary. Wang et al. attempt to bridge ML systems thinking with computer architecture rigor into a unified cross-layer framework — from "the model feels slow" down to the specific hardware bottleneck and its root cause.The hosts build the foundational model carefully, tracing execution from prompt to generated token. Dr. Shannon explains the two-phase structure of transformer inference: the prefill phase processes all prompt tokens simultaneously in a matrix-matrix multiply that is compute-bound, writing key-value pairs into the KV cache; the decode phase generates tokens autoregressively, and at each step the attention mechanism reads back every prior cached key-value pair alongside the full weight matrices. Hal presses on the memory bandwidth implications — by token three hundred and ninety-nine, the model loads nearly four hundred KV pairs per layer per step just to produce one new token. Dr. Shannon maps this onto the Roofline model, originally developed by Williams, Waterman, and Patterson at UC Berkeley in 2009, using arithmetic intensity — the ratio of floating-point operations to memory traffic — to show why decode sits deep in the memory-bound regime while prefill remains compute-bound. That asymmetry is the load-bearing structure of the paper's argument.When Hal pushes back on whether the compute-bound prefill and memory-bound decode framing is merely a formalization of established practitioner intuition, Dr. Shannon draws a sharp distinction between the heuristic and the science. Knowing decode is memory-bound is the starting point; the paper's contribution is stall analysis — profiling with hardware performance counters to determine precisely why GPU thread warps pause during execution. A warp stalled on memory pipe saturation has a different cause and a different remedy than one stalled on instruction-level data dependency. The hosts frame this microarchitectural root cause analysis as one of four dimensions in the paper's framework, alongside the two-phase prefill-decode heterogeneity, system scaling principles, and the boundaries where current GPU architectures hit fundamental limits — together forming the diagnostic vocabulary the field has been missing.Sources:1. A Systematic Characterization of LLM Inference on GPUs — Haonan Wang, Xuxin Xiao, Mingyu Yan, Zhuoyuan Zhu, Dengke Han, Duo Wang, Wenming Li, Xiaochun Ye, Cunchen Hu, Hongyang Chen, Guangyu Sun, 2025http://arxiv.org/abs/2512.016442. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention3. Orca: A Distributed Serving System for Transformer-Based Generative Models — Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun, 2022https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models4. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving — Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xing Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, Ion Stoica, 2022https://scholar.google.com/scholar?q=AlpaServe%3A+Statistical+Multiplexing+with+Model+Parallelism+for+Deep+Learning+Serving5. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, 2022https://scholar.google.com/scholar?q=LLM.int8%28%29%3A+8-bit+Matrix+Multiplication+for+Transformers+at+Scale6. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini, 2024https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting7. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-Optimized+Large+Language+Model+Serving8. Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes on Chunked Prefills — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+on+Chunked+Prefills9. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang, 2023https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU10. Roofline: An Insightful Visual Performance Model for Multicore Architectures — Samuel Williams, Andrew Waterman, David Patterson, 2009https://scholar.google.com/scholar?q=Roofline%3A+An+Insightful+Visual+Performance+Model+for+Multicore+Architectures11. In-Datacenter Performance Analysis of a Tensor Processing Unit — Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, et al., 2017https://scholar.google.com/scholar?q=In-Datacenter+Performance+Analysis+of+a+Tensor+Processing+Unit12. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness13. Dissecting the Graphcore IPU Architecture via Microbenchmarking — Amey Agrawal, Rohit Ramaprasad, Kranthi Gade, Ramachandran Ramjee, 2021https://scholar.google.com/scholar?q=Dissecting+the+Graphcore+IPU+Architecture+via+Microbenchmarking14. Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Agrawal et al., 2024https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+with+Chunked+Prefills15. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Dao, 2023https://scholar.google.com/scholar?q=FlashAttention-2%3A+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning16. Flash-Decoding for Long-Context Inference — Dao et al., 2023https://scholar.google.com/scholar?q=Flash-Decoding+for+Long-Context+Inference17. Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures — Williams, Waterman, Patterson, 2009https://scholar.google.com/scholar?q=Roofline%3A+An+Insightful+Visual+Performance+Model+for+Floating-Point+Programs+and+Multicore+Architectures18. Sarathi: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Agrawal et al., 2023https://scholar.google.com/scholar?q=Sarathi%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+with+Chunked+Prefills19. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference — approximate, recent, 2024-2025https://scholar.google.com/scholar?q=POD-Attention%3A+Unlocking+Full+Prefill-Decode+Overlap+for+Faster+LLM+Inference20. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse — approximate, recent, 2024-2025https://scholar.google.com/scholar?q=KVLink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse21. Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs — approximate, recent, 2024-2025https://scholar.google.com/scholar?q=Shared+Disk+KV+Cache+Management+for+Efficient+Multi-Instance+Inference+in+RAG-Powered+LLMs22. ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference — approximate, recent, 2024-2025https://scholar.google.com/scholar?q=ExpertFlow%3A+Optimized+Expert+Activation+and+Token+Allocation+for+Efficient+Mixture-of-Experts+Inference23. Harder Task Needs More Experts: Dynamic Routing in MoE Models — approximate, recent, 2024-2025https://scholar.google.com/scholar?q=Harder+Task+Needs+More+Experts%3A+Dynamic+Routing+in+MoE+Models24. DECA: A Near-Core LLM Decompression Accelerator Grounded on a 3D Roofline Model — approximate, recent, 2024-2025https://scholar.google.com/scholar?q=DECA%3A+A+Near-Core+LLM+Decompression+Accelerator+Grounded+on+a+3D+Roofline+Model25. HeadInfer: Memory-Efficient LLM Inference by Head-Wise Offloading — approximate, recent, 2024-2025https://scholar.google.com/scholar?q=HeadInfer%3A+Memory-Efficient+LLM+Inference+by+Head-Wise+Offloading26. AI Post Transformers: AI and the Memory Wall: Overcoming Bottlenecks — Hal Turing & Dr. Ada Shannon, Fri,https://podcasters.spotify.com/pod/show/12146088098/episodes/AI-and-the-Memory-Wall-Overcoming-Bottlenecks-e36ki0u27. AI Post Transformers: Flash-LLM: Efficient LLM Inference with Unstructured Sparsity on Tensor Cores — Hal Turing & Dr. Ada Shannon, Fri,https://podcasters.spotify.com/pod/show/12146088098/episodes/Flash-LLM-Efficient-LLM-Inference-with-Unstructured-Sparsity-on-Tensor-Cores-e3aalrl28. AI Post Transformers: Advancements in Efficient KV Cache Quantization and Management — Hal Turing & Dr. Ada Shannon, Thu,https://podcasters.spotify.com/pod/show/12146088098/episodes/Advancements-in-Efficient-KV-Cache-Quantization-and-Management-e3fk9kr29. AI Post Transformers: Adaptive LLM Partitioning for Edge Inference — Hal Turing & Dr. Ada Shannon, Tue,https://podcasters.spotify.com/pod/show/12146088098/episodes/Adaptive-LLM-Partitioning-for-Edge-Inference-e38a9mn30. AI Post Transformers: Parallel Token Prediction: From ProphetNet to Dependent Multi-Token Generation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-04-4-papers-combined-8a237d.mp3Interactive Visualization: Systematic Characterization of LLM Inference on GPUs0 min
- Why CARTRIDGE Works: Keys as Routers in KV CachesHal Turing and Dr. Ada Shannon return to the CARTRIDGE compression system with a mechanistic lens, covering Maurizio A. Diaz's paper "Learned Structure in Cartridges: Keys as Shareable Routers in Self-Studied Representations" (arXiv 2508.17032), presented at the NeurIPS 2025 Workshop on Mechanistic Interpretability. Building on the original CARTRIDGE episode from November 10th, 2025 and the follow-up from February 6th, 2026, this episode asks the question those earlier discussions left open: what structure does the optimizer actually induce in a trained CARTRIDGE? The hosts ground the discussion in the memory scaling problem driving the entire field—KV caches that grow linearly with context length, now routinely dwarfing model weights at the 128K-to-million-token scales of current frontier models—and trace how techniques like PagedAttention, Grouped Query Attention, and token eviction address symptoms without shrinking the underlying representation.Diaz's central finding is a clean functional division between key and value vectors inside a trained CARTRIDGE. Keys converge to stable retrieval routers: low-rank, consistent structures that steer attention toward the right stored content across diverse queries. Values carry the compressed semantic payload. The hosts connect this directly to how CARTRIDGE's Self-Study training pipeline works—because the cache is optimized against synthetic question-answer traces generated by the model over its own content, the training signal explicitly selects for routing behavior, making the key-as-router outcome a predictable consequence of the objective rather than an accident. Diaz uses Singular Value Decomposition to quantify this structure layer by layer, separating the geometric properties of key matrices from value matrices across training checkpoints.Two downstream findings from the key-router property shape the second half of the discussion. Because keys are stable and low-rank, they transfer across tasks with minimal degradation—a result with direct implications for multi-task serving, where a single shared key structure could route to task-specific value sets without independent CARTRIDGE training per deployment. The Sampled Chunk Initialization method introduced in the paper exploits this stability to warm-start CARTRIDGE training, accelerating convergence by initializing the learnable KV pairs from a small representative sample rather than random weights. Hal and Ada close by discussing what the key-as-router framing implies for KV-cache compression research more broadly: if the routing function is separable and transferable, compression schemes that conflate keys and values may be discarding structure that has real serving-efficiency value.Sources:1. Learned Structure in Cartridges: Keys as Shareable Routers in Self-Studied Representations — Maurizio Diaz, 2025http://arxiv.org/abs/2508.170322. CARTRIDGES: Learning to Pack Long Contexts into KV Caches — Zhihao Zhang, Aditya Desai, Amir Gholami, Michael W. Mahoney, Kurt Keutzer, et al. (Berkeley / ICSI), 2025https://scholar.google.com/scholar?q=CARTRIDGES%3A+Learning+to+Pack+Long+Contexts+into+KV+Caches3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianghao Huang, Mu Li, Beidi Chen, Jason D. Lee, Binhang Yuan, Ce Zhang, Cho-Jui Hsieh, 2023https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models4. Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time — Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, Anshumali Shrivastava, 2023https://scholar.google.com/scholar?q=Scissorhands%3A+Exploiting+the+Persistence+of+Importance+Hypothesis+for+LLM+KV+Cache+Compression+at+Test+Time5. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks6. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention7. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts8. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, Sumit Sanghai, 2023https://scholar.google.com/scholar?q=GQA%3A+Training+Generalized+Multi-Query+Transformer+Models+from+Multi-Head+Checkpoints9. Extending Context Window of Large Language Models via Positional Interpolation — Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian, 2023https://scholar.google.com/scholar?q=Extending+Context+Window+of+Large+Language+Models+via+Positional+Interpolation10. Constitutional AI: Harmlessness from AI Feedback — Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. (Anthropic), 2022https://scholar.google.com/scholar?q=Constitutional+AI%3A+Harmlessness+from+AI+Feedback11. Distilling the Knowledge in a Neural Network — Geoffrey Hinton, Oriol Vinyals, Jeff Dean, 2015https://scholar.google.com/scholar?q=Distilling+the+Knowledge+in+a+Neural+Network12. Compressing Context to Enhance Inference Efficiency of Large Language Models — Yucheng Li, Bo Dong, Chenghua Lin, Frank Guerin, 2023https://scholar.google.com/scholar?q=Compressing+Context+to+Enhance+Inference+Efficiency+of+Large+Language+Models13. AutoCompressors: Adapting Language Models to Summarize Arbitrary Contexts into Summary Vectors — Alexis Chevalier, Alexander Wettig, Anirudh Anand, Danqi Chen, 2023https://scholar.google.com/scholar?q=AutoCompressors%3A+Adapting+Language+Models+to+Summarize+Arbitrary+Contexts+into+Summary+Vectors14. A Mathematical Framework for Transformer Circuits — Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, Chris Olah, 2021https://scholar.google.com/scholar?q=A+Mathematical+Framework+for+Transformer+Circuits15. Toy Models of Superposition — Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Ben Mann, Shan Carter, Chris Olah, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Jared Kaplan, Dario Amodei, 2022https://scholar.google.com/scholar?q=Toy+Models+of+Superposition16. Towards Monosemanticity: Decomposing Language Models with Dictionary Learning — Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, William Lamont, Annabel Shearer, Zac Hatfield-Dodds, Tom Henighan, Nicholas Joseph, Ben Ziegler, Bilal Chaudhry, Will Hao, Timothy Telleen-Lawton, Daniel Mossing, Adam Jermyn, Tom Brown, Chris Olah, Sam McCandlish, Jack Clark, Dario Amodei, 2023https://scholar.google.com/scholar?q=Towards+Monosemanticity%3A+Decomposing+Language+Models+with+Dictionary+Learning17. In-context Learning and Induction Heads — Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, Chris Olah, 2022https://scholar.google.com/scholar?q=In-context+Learning+and+Induction+Heads18. AutoCompressors: Compressing Contexts with Language Models — Chevalier et al., 2023https://scholar.google.com/scholar?q=AutoCompressors%3A+Compressing+Contexts+with+Language+Models19. Gist Tokens: Compressing Prompts into Tokens for Long-Context Language Models — Mu et al., 2023https://scholar.google.com/scholar?q=Gist+Tokens%3A+Compressing+Prompts+into+Tokens+for+Long-Context+Language+Models20. The Power of Scale for Parameter-Efficient Prompt Tuning — Lester et al., 2021https://scholar.google.com/scholar?q=The+Power+of+Scale+for+Parameter-Efficient+Prompt+Tuning21. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Li and Liang, 2021https://scholar.google.com/scholar?q=Prefix-Tuning%3A+Optimizing+Continuous+Prompts+for+Generation22. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling — Cai et al., 2024https://scholar.google.com/scholar?q=PyramidKV%3A+Dynamic+KV+Cache+Compression+based+on+Pyramidal+Information+Funneling23. Tokasaurus: A High-Throughput LLM Serving Engine with Grouped-Sparse Attention — Lenz et al., 2025https://scholar.google.com/scholar?q=Tokasaurus%3A+A+High-Throughput+LLM+Serving+Engine+with+Grouped-Sparse+Attention24. Unlocking the Address Book: Dissecting the Sparse Semantic Structure of LLM Key-Value Caches via Sparse Autoencoders — unknown from snippet, 2024-2025https://scholar.google.com/scholar?q=Unlocking+the+Address+Book%3A+Dissecting+the+Sparse+Semantic+Structure+of+LLM+Key-Value+Caches+via+Sparse+Autoencoders25. SCBench: A KV Cache-Centric Analysis of Long-Context Methods — approximate, multiple authors, 2024-2025https://scholar.google.com/scholar?q=SCBench%3A+A+KV+Cache-Centric+Analysis+of+Long-Context+Methods26. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression — approximate, Microsoft Research, 2024https://scholar.google.com/scholar?q=LLMLingua-2%3A+Data+Distillation+for+Efficient+and+Faithful+Task-Agnostic+Prompt+Compression27. AI Post Transformers: CARTRIDGE: Efficient In-Context Learning via Distillation — Hal Turing & Dr. Ada Shannon, Mon,https://podcasters.spotify.com/pod/show/12146088098/episodes/CARTRIDGE-Efficient-In-Context-Learning-via-Distillation-e3aous428. AI Post Transformers: Context Distillation for Language Models — Hal Turing & Dr. Ada Shannon, Mon,https://podcasters.spotify.com/pod/show/12146088098/episodes/Context-Distillation-for-Language-Models-e3aouen29. AI Post Transformers: Advancements in Efficient KV Cache Quantization and Management — Hal Turing & Dr. Ada Shannon, Thu,https://podcasters.spotify.com/pod/show/12146088098/episodes/Advancements-in-Efficient-KV-Cache-Quantization-and-Management-e3fk9kr30. AI Post Transformers: Architectural Migration to Multi-head Latent Attention — Hal Turing & Dr. Ada Shannon, Wed,https://podcasters.spotify.com/pod/show/12146088098/episodes/Architectural-Migration-to-Multi-head-Latent-Attention-e39jbmq0 min
- DualPath Breaks Storage Bandwidth Bottleneck in Agentic InferenceHal Turing and Dr. Ada Shannon dig into "DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference," a February 2026 paper from a thirteen-author team spanning Peking University, Tsinghua University, and DeepSeek-AI. The episode opens with a striking observation from production: on disaggregated inference clusters running agentic workloads, prefill machines saturate their storage network interfaces at 100% utilization while the equivalent hardware on decode machines sits nearly idle. The hosts use this asymmetry as a lens into a counterintuitive reality — H100 GPUs throttled to 40% compute utilization not by arithmetic limits, but by a storage NIC.Ada explains the structural reason agentic workloads are uniquely hostile to existing infrastructure: the short-append pattern. Unlike standard multi-turn chat, agentic sessions accumulate dozens to hundreds of turns where each round appends only a small number of tokens — a tool result, a stack trace, a code output — onto a context that may already span tens of thousands of tokens. Because that prior context never changes, its KV-Cache was computed once and stored. DeepSeek's production traces show KV-Cache hit rates of 95% or higher, meaning the dominant cost shifts from GPU computation to storage I/O: loading gigabytes of persistent key-value state from external NVMe-backed distributed storage, layer by layer, into prefill engines via RDMA. Hal presses on that 95% figure specifically, establishing that it is grounded in real production traffic rather than idealized assumptions — a distinction that determines whether storage bandwidth or GPU compute is the correct optimization target.The episode frames DualPath's core insight against this background: the storage NICs on decode engines represent idle bandwidth that could absorb KV-Cache load traffic currently overwhelming prefill-side storage interfaces. By routing that traffic through decode-side hardware and transferring it to prefill engines over RDMA, DualPath breaks the single-path bottleneck without adding new hardware. The hosts connect this to the broader memory wall argument — that as context lengths grow and agentic sessions deepen, the architectural shift toward disaggregated inference is not optional, and the constraints driving system design are increasingly about data movement rather than floating-point throughput. DualPath's reported throughput improvement of up to 1.96x is presented as evidence that exploiting idle hardware asymmetries, rather than scaling compute, is where near-term agentic inference gains will be found.Sources:1. DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference — Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, Yixuan Tan, Wentao Zhang, Liyue Zhang, Shangyan Zhou, Yuxuan Liu, Shunfeng Zhou, Mingxing Zhang, Xin Jin, Panpan Huang, 2026http://arxiv.org/abs/2602.215482. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention3. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph Gonzalez, Clark Barrett, Ying Sheng, 2024https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs4. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2024https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving5. CacheGen: Fast Context Loading for Language Model Applications — Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, Junchen Jiang, 2024https://scholar.google.com/scholar?q=CacheGen%3A+Fast+Context+Loading+for+Language+Model+Applications6. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving7. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Igo Goiri, Saeed Maleki, Ricardo Bianchini, 2024https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting8. Sarathi-Serve: Chunked Prefills for Efficient LLM Inference — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Chunked+Prefills+for+Efficient+LLM+Inference9. Llumnix: Dynamic Scheduling for Large Language Model Serving — Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, Wei Lin, 2024https://scholar.google.com/scholar?q=Llumnix%3A+Dynamic+Scheduling+for+Large+Language+Model+Serving10. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, Ce Zhang, 2023https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU11. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management12. Parrot: Efficient Servicing of LLM-based Applications with Semantic Variable — Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, Lili Qiu, 2024https://scholar.google.com/scholar?q=Parrot%3A+Efficient+Servicing+of+LLM-based+Applications+with+Semantic+Variable13. AgentBench: Evaluating LLMs as Agents — Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhiyuan Liu, Yuxiao Dong, Jie Tang, 2023https://scholar.google.com/scholar?q=AgentBench%3A+Evaluating+LLMs+as+Agents14. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph Gonzalez, 2023https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems15. Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Amey Agrawal, Nikhil Mohan, Sudhanshu Panwar, et al., 2024https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+with+Chunked+Prefills16. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, et al., 2024https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion17. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Liu et al., 2024https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving18. Compute or Load KV Cache? Why Not Both? — Unknown (snippet only), 2024-2025https://scholar.google.com/scholar?q=Compute+or+Load+KV+Cache%3F+Why+Not+Both%3F19. Semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage — Unknown (snippet only), 2024-2025https://scholar.google.com/scholar?q=Semi-PD%3A+Towards+Efficient+LLM+Serving+via+Phase-Wise+Disaggregated+Computation+and+Unified+Storage20. Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving (TaiChi) — Unknown (snippet only), 2024-2025https://scholar.google.com/scholar?q=Prefill-Decode+Aggregation+or+Disaggregation%3F+Unifying+Both+for+Goodput-Optimized+LLM+Serving+%28TaiChi%2921. Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live — Unknown (snippet only), 2024-2025https://scholar.google.com/scholar?q=Continuum%3A+Efficient+and+Robust+Multi-Turn+LLM+Agent+Scheduling+with+KV+Cache+Time-to-Live22. SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoning — Unknown (snippet only), 2024-2025https://scholar.google.com/scholar?q=SideQuest%3A+Model-Driven+KV+Cache+Management+for+Long-Horizon+Agentic+Reasoning23. RDMA Point-to-Point Communication for LLM Systems — Unknown (snippet only), 2024-2025https://scholar.google.com/scholar?q=RDMA+Point-to-Point+Communication+for+LLM+Systems24. AI Post Transformers: FAST26: Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional — Hal Turing & Dr. Ada Shannon, Wed,https://podcasters.spotify.com/pod/show/12146088098/episodes/FAST26-Bidaw-Enhancing-Key-Value-Caching-for-Interactive-LLM-Serving-via-Bidirectional-e3fjgkh25. AI Post Transformers: SYMPHONY: Memory Management for LLM Multi-Turn Inference — Hal Turing & Dr. Ada Shannon, Mon,https://podcasters.spotify.com/pod/show/12146088098/episodes/SYMPHONY-Memory-Management-for-LLM-Multi-Turn-Inference-e3ap0pf26. AI Post Transformers: CXL-SpecKV: Bridging the LLM Memory Wall with Speculative FPGA Disaggregation — Hal Turing & Dr. Ada Shannon, Sat,https://podcasters.spotify.com/pod/show/12146088098/episodes/CXL-SpecKV-Bridging-the-LLM-Memory-Wall-with-Speculative-FPGA-Disaggregation-e3foad027. AI Post Transformers: AI and the Memory Wall: Overcoming Bottlenecks — Hal Turing & Dr. Ada Shannon, Fri,https://podcasters.spotify.com/pod/show/12146088098/episodes/AI-and-the-Memory-Wall-Overcoming-Bottlenecks-e36ki0u28. AI Post Transformers: Tempo: SLO-Aware LLM Serving Maximizing Service Gain — Hal Turing & Dr. Ada Shannon, Mon,https://podcasters.spotify.com/pod/show/12146088098/episodes/Tempo-SLO-Aware-LLM-Serving-Maximizing-Service-Gain-e3ap12aInteractive Visualization: DualPath Breaks Storage Bandwidth Bottleneck in Agentic Inference0 min
- Structured State Space Duality Unifies Transformers and SSMsHal Turing and Dr. Ada Shannon open by situating the Dao-Gu paper — "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality" (arXiv 2405.21060, ICML 2024) — within the bifurcated landscape of sequence modeling. For years, Transformer researchers and SSM researchers developed in parallel, unable to borrow optimizations across the divide. The episode traces the lineage from HiPPO and S4 through Mamba's selective state spaces, explaining why SSMs' linear-time theoretical advantage never translated to wall-clock wins: the GPU ecosystem was built for dense matrix multiply, and SSMs lacked the tooling that FlashAttention brought to attention. The hosts also credit the direct intellectual ancestor — Katharopoulos et al.'s 2020 "Transformers are RNNs" — which showed that softmax attention with a kernel approximation reduces to a linear recurrence, establishing the conceptual template Dao and Gu would later formalize.The core of the episode is a careful unpacking of structured semiseparable matrices, a class of objects from numerical linear algebra — Kalman filter theory, PDE solvers — entirely unknown to the ML community until Dao and Gu made the connection. Every entry of a causal SSM input-output matrix has the form C[i] times a chain of transition matrices A times B[j], which is precisely the generator representation of a rank-d semiseparable matrix. Shannon walks through the O(n) factored form — M[i,j] equals u[i] times v[j] below the diagonal — and explains how this structure encodes both the SSM recurrence and the masked attention computation as two views of the same algebraic object. The canonical reference is Vandebril, Van Barel, and Mastronardi's two-volume work from Johns Hopkins, 2008, a body of theory the ML community had never encountered. Once the connection is made, hardware-efficient algorithms from one domain port directly to the other.But the episode frames this mathematical achievement against a harder question raised by subsequent theoretical work: the L²M Condition of Chen et al. and the bipartite mutual information scaling law. The duality shows that SSM computation and attention computation are equivalent representations — but equivalence of computation does not imply equivalence of information retention. SSMs compress sequence history into a fixed-size state regardless of context length; the Transformer's KV-cache grows linearly, retaining more as context expands. The mutual information scaling law formalizes this gap: capturing the multi-token dependencies present in natural language requires a history state that grows with context length. The episode closes on what this implies for hybrid architectures — systems that combine SSM efficiency with selective attention — and whether the theoretical unification Dao and Gu achieved changes how practitioners should think about where compressed state fails.Sources:1. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — Tri Dao, Albert Gu, 2024http://arxiv.org/abs/2405.210602. https://arxiv.org/pdf/2503.047253. On a New Class of Structured Matrices — Yuli Eidelman, Israel Gohberg, 1997https://scholar.google.com/scholar?q=On+a+New+Class+of+Structured+Matrices4. Time-Varying Systems and Computations — Patrick Dewilde, Alle-Jan van der Veen, 1998https://scholar.google.com/scholar?q=Time-Varying+Systems+and+Computations5. Matrix Computations with Semiseparable Matrices (Volumes 1 & 2) — Raf Vandebril, Marc Van Barel, Nicola Mastronardi, 2008https://scholar.google.com/scholar?q=Matrix+Computations+with+Semiseparable+Matrices+%28Volumes+1+%26+2%296. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — Tri Dao, Albert Gu, 2024https://scholar.google.com/scholar?q=Transformers+are+SSMs%3A+Generalized+Models+and+Efficient+Algorithms+Through+Structured+State+Space+Duality7. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — Katharopoulos, Vyas, Pappas, Fleuret, 2020https://scholar.google.com/scholar?q=Transformers+are+RNNs%3A+Fast+Autoregressive+Transformers+with+Linear+Attention8. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Gu, Dao, 2023https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces9. Semiseparable Matrices in Linear Algebra, Science and Engineering (2 vols.) — Vandebril, Van Barel, Mastronardi, 2008https://scholar.google.com/scholar?q=Semiseparable+Matrices+in+Linear+Algebra%2C+Science+and+Engineering+%282+vols.%2910. Retentive Network: A Successor to Transformer for Large Language Models — Sun, Li, Dong, Huang, Peng, Liu, Wang, Lin, Yuan, Chen, Wei, 2023https://scholar.google.com/scholar?q=Retentive+Network%3A+A+Successor+to+Transformer+for+Large+Language+Models11. RWKV: Reinventing RNNs for the Transformer Era — Peng et al., 2023https://scholar.google.com/scholar?q=RWKV%3A+Reinventing+RNNs+for+the+Transformer+Era12. Gated Linear Attention Transformers with Hardware-Efficient Training — Yang, Wang, Shen, Peng, Dao, Gu, Matoba, 2023https://scholar.google.com/scholar?q=Gated+Linear+Attention+Transformers+with+Hardware-Efficient+Training13. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Dao, 2024https://scholar.google.com/scholar?q=FlashAttention-2%3A+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning14. Zoology: Measuring and Improving Recall in Efficient Language Models — Arora, Eyuboglu, Timalsina, Johnson, Poli, Zou, Rudra, Ré, 2024https://scholar.google.com/scholar?q=Zoology%3A+Measuring+and+Improving+Recall+in+Efficient+Language+Models15. Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention — unknown (snippet only), 2024-2025https://scholar.google.com/scholar?q=Overcoming+Long-Context+Limitations+of+State-Space+Models+via+Context-Dependent+Sparse+Attention16. Bayesian Optimality of In-Context Learning with Selective State Spaces — unknown (snippet only), 2024-2025https://scholar.google.com/scholar?q=Bayesian+Optimality+of+In-Context+Learning+with+Selective+State+Spaces17. On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspective — unknown (snippet only), 2024-2025https://scholar.google.com/scholar?q=On+the+Expressiveness+of+Softmax+Attention%3A+A+Recurrent+Neural+Network+Perspective18. Bridging the Divide: Reconsidering Softmax and Linear Attention — unknown (snippet only), 2024-2025https://scholar.google.com/scholar?q=Bridging+the+Divide%3A+Reconsidering+Softmax+and+Linear+Attention19. Softmax Linear Attention: Reclaiming Global Competition — unknown (snippet only), 2024-2025https://scholar.google.com/scholar?q=Softmax+Linear+Attention%3A+Reclaiming+Global+Competition20. Hybrid Architectures for Language Models: Systematic Analysis and Design Insights — unknown (snippet only), 2024-2025https://scholar.google.com/scholar?q=Hybrid+Architectures+for+Language+Models%3A+Systematic+Analysis+and+Design+Insights21. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse — unknown (snippet only), 2024-2025https://scholar.google.com/scholar?q=KVLink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse22. AI Post Transformers: FlashAttention-4 Conquers Asymmetric GPU Hardware Scaling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-06-flashattention-4-conquers-asymmetric-gpu-78839b.mp323. AI Post Transformers: ATTENTION2D and lean attention: Distributed Self-Attention — Hal Turing & Dr. Ada Shannon, 2025https://podcasters.spotify.com/pod/show/12146088098/episodes/ATTENTION2D-and-lean-attention-Distributed-Self-Attention-e3a7r4n0 min
- We're Open Source! New Home, Visualizations, and How to Shape Our QueueSpecial announcement: AI Post Transformers is now open source under the MIT license. New home at podcast.do-not-panic.com, interactive paper visualizations, community paper submissions, deep dive into the editorial queue algorithm, and internationalization roadmap.0 min
- FlashAttention-4 Conquers Asymmetric GPU Hardware ScalingHal Turing and Dr. Ada Shannon dig into FlashAttention-4, a March 2026 paper from a cross-institutional team including Tri Dao, Jay Shah, and colleagues at Princeton, Meta, NVIDIA, Colfax Research, Georgia Tech, and Together AI. The paper targets a precise hardware mismatch on NVIDIA's Blackwell B200: tensor core throughput doubles compared to the H100, but shared memory bandwidth and dedicated exponential function units do not scale at the same rate. Rather than waiting for hardware fixes, the authors co-design the attention algorithm with the asymmetric architecture itself — making FlashAttention-4 the first attention kernel built specifically for Blackwell's scaling profile.To frame why this matters, Shannon traces the full lineage of FlashAttention research. The original 2022 NeurIPS paper by Dao and colleagues reframed attention as an IO problem: instead of materializing the quadratic N×N score matrix in slow off-chip High Bandwidth Memory, tiling and online softmax keep computation inside the fast on-chip shared memory of each streaming multiprocessor. FlashAttention-2 doubled throughput through sequence-dimension parallelism. FlashAttention-3 pushed H100 utilization to roughly 75% by exploiting Hopper-specific warp specialization and asynchronous data movement. Each generation addressed a qualitatively different bottleneck — and Blackwell introduced a new one that none of those solutions anticipated.The hosts ground the stakes for practitioners who work in ML without writing GPU kernels. Attention sits at the core of every Transformer-based system — large language models, vision transformers, multimodal architectures — and long-context workloads at 32K to 128K tokens make the quadratic memory cost and HBM round-trips increasingly punishing. Shannon introduces the roofline model as the analytic lens the paper uses to characterize where Blackwell kernels actually bottleneck, setting up how FlashAttention-4's algorithmic co-design approach navigates the compute and memory bandwidth ceilings that previous generations of the kernel never had to contend with.Sources:1. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling — Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, Tri Dao, 2026http://arxiv.org/abs/2603.05451v12. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness3. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Tri Dao, 2023https://scholar.google.com/scholar?q=FlashAttention-2%3A+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning4. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low Precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024https://scholar.google.com/scholar?q=FlashAttention-3%3A+Fast+and+Accurate+Attention+with+Asynchrony+and+Low+Precision5. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations — Philippe Tillet, H. T. Kung, David Cox, 2019https://scholar.google.com/scholar?q=Triton%3A+An+Intermediate+Language+and+Compiler+for+Tiled+Neural+Network+Computations6. Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures — Samuel Williams, Andrew Waterman, David Patterson, 2009https://scholar.google.com/scholar?q=Roofline%3A+An+Insightful+Visual+Performance+Model+for+Floating-Point+Programs+and+Multicore+Architectures7. Efficiently Scaling Transformer Inference — Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Sharan Narang, Jeff Dean, 2023https://scholar.google.com/scholar?q=Efficiently+Scaling+Transformer+Inference8. In-Datacenter Performance Analysis of a Tensor Processing Unit — Norman P. Jouppi et al. (Google), 2017https://scholar.google.com/scholar?q=In-Datacenter+Performance+Analysis+of+a+Tensor+Processing+Unit9. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces10. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention11. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro, 2019https://scholar.google.com/scholar?q=Megatron-LM%3A+Training+Multi-Billion+Parameter+Language+Models+Using+Model+Parallelism12. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024https://scholar.google.com/scholar?q=FlashAttention-3%3A+Fast+and+Accurate+Attention+with+Asynchrony+and+Low-precision13. SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization — Jintao Zhang et al., 2024https://scholar.google.com/scholar?q=SageAttention2%3A+Efficient+Attention+with+Thorough+Outlier+Smoothing+and+Per-thread+INT4+Quantization14. Ring Attention with Blockwise Transformers for Near-Infinite Context — Hao Liu, Matei Zaharia, Pieter Abbeel, 2023https://scholar.google.com/scholar?q=Ring+Attention+with+Blockwise+Transformers+for+Near-Infinite+Context15. FlexAttention: The Flexibility of PyTorch with the Performance of FlashAttention — PyTorch Team, 2024https://scholar.google.com/scholar?q=FlexAttention%3A+The+Flexibility+of+PyTorch+with+the+Performance+of+FlashAttention16. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI, 2024https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model17. Softmax output approximation for activation memory-efficient training of attention-based networks — approximate — likely 2023–2024, 2023–2024https://scholar.google.com/scholar?q=Softmax+output+approximation+for+activation+memory-efficient+training+of+attention-based+networks18. FlashAttention-T: Towards Fully Tensorized Attention by Exploiting Tensor-Vector Parallelism — approximate — likely 2024–2025, 2024–2025https://scholar.google.com/scholar?q=FlashAttention-T%3A+Towards+Fully+Tensorized+Attention+by+Exploiting+Tensor-Vector+Parallelism19. Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention — approximate — likely 2024–2025, 2024–2025https://scholar.google.com/scholar?q=Overcoming+Long-Context+Limitations+of+State-Space+Models+via+Context-Dependent+Sparse+Attention20. Based on Tensor Core Sparse Kernels Accelerating Deep Neural Networks — approximate — likely 2024–2025, 2024–2025https://scholar.google.com/scholar?q=Based+on+Tensor+Core+Sparse+Kernels+Accelerating+Deep+Neural+Networks21. AI Post Transformers: FlashAttention-2: Faster Attention with Better Parallelism — Hal Turing & Dr. Ada Shannon, Fri,https://podcasters.spotify.com/pod/show/12146088098/episodes/FlashAttention-2-Faster-Attention-with-Better-Parallelism-e36kdm022. AI Post Transformers: ATTENTION2D and lean attention: Distributed Self-Attention — Hal Turing & Dr. Ada Shannon, Wed,https://podcasters.spotify.com/pod/show/12146088098/episodes/ATTENTION2D-and-lean-attention-Distributed-Self-Attention-e3a7r4n23. AI Post Transformers: Jet-RL: Stable On-Policy Reinforcement Learning with Unified FP8 Flow — Hal Turing & Dr. Ada Shannon, Tue,https://podcasters.spotify.com/pod/show/12146088098/episodes/Jet-RL-Stable-On-Policy-Reinforcement-Learning-with-Unified-FP8-Flow-e3f7det24. AI Post Transformers: Mojo: Performance-Portable HPC Kernels on GPUs — Hal Turing & Dr. Ada Shannon, Sat,https://podcasters.spotify.com/pod/show/12146088098/episodes/Mojo-Performance-Portable-HPC-Kernels-on-GPUs-e39n4skInteractive Visualization: FlashAttention-4: Algorithm & Kernel Co-Design0 min
- We're Open Source! New Home, Visualizations, and How to Shape Our QueueSpecial announcement: AI Post Transformers is now open source under the MIT license. New home at podcast.do-not-panic.com, interactive paper visualizations, community paper submissions, deep dive into the editorial queue algorithm, and internationalization roadmap.0 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.