Hal Turing and Dr. Ada Shannon return to the CARTRIDGE compression system with a mechanistic lens, covering Maurizio A. Diaz's paper "Learned Structure in Cartridges: Keys as Shareable Routers in Self-Studied Representations" (arXiv 2508.17032), presented at the NeurIPS 2025 Workshop on Mechanistic Interpretability. Building on the original CARTRIDGE episode from November 10th, 2025 and the follow-up from February 6th, 2026, this episode asks the question those earlier discussions left open: what structure does the optimizer actually induce in a trained CARTRIDGE? The hosts ground the discussion in the memory scaling problem driving the entire field—KV caches that grow linearly with context length, now routinely dwarfing model weights at the 128K-to-million-token scales of current frontier models—and trace how techniques like PagedAttention, Grouped Query Attention, and token eviction address symptoms without shrinking the underlying representation.

Diaz's central finding is a clean functional division between key and value vectors inside a trained CARTRIDGE. Keys converge to stable retrieval routers: low-rank, consistent structures that steer attention toward the right stored content across diverse queries. Values carry the compressed semantic payload. The hosts connect this directly to how CARTRIDGE's Self-Study training pipeline works—because the cache is optimized against synthetic question-answer traces generated by the model over its own content, the training signal explicitly selects for routing behavior, making the key-as-router outcome a predictable consequence of the objective rather than an accident. Diaz uses Singular Value Decomposition to quantify this structure layer by layer, separating the geometric properties of key matrices from value matrices across training checkpoints.

Two downstream findings from the key-router property shape the second half of the discussion. Because keys are stable and low-rank, they transfer across tasks with minimal degradation—a result with direct implications for multi-task serving, where a single shared key structure could route to task-specific value sets without independent CARTRIDGE training per deployment. The Sampled Chunk Initialization method introduced in the paper exploits this stability to warm-start CARTRIDGE training, accelerating convergence by initializing the learnable KV pairs from a small representative sample rather than random weights. Hal and Ada close by discussing what the key-as-router framing implies for KV-cache compression research more broadly: if the routing function is separable and transferable, compression schemes that conflate keys and values may be discarding structure that has real serving-efficiency value.

Sources:

1. Learned Structure in Cartridges: Keys as Shareable Routers in Self-Studied Representations — Maurizio Diaz, 2025

http://arxiv.org/abs/2508.17032

2. CARTRIDGES: Learning to Pack Long Contexts into KV Caches — Zhihao Zhang, Aditya Desai, Amir Gholami, Michael W. Mahoney, Kurt Keutzer, et al. (Berkeley / ICSI), 2025

https://scholar.google.com/scholar?q=CARTRIDGES:+Learning+to+Pack+Long+Contexts+into+KV+Caches

3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianghao Huang, Mu Li, Beidi Chen, Jason D. Lee, Binhang Yuan, Ce Zhang, Cho-Jui Hsieh, 2023

https://scholar.google.com/scholar?q=H2O:+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models

4. Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time — Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, Anshumali Shrivastava, 2023

https://scholar.google.com/scholar?q=Scissorhands:+Exploiting+the+Persistence+of+Importance+Hypothesis+for+LLM+KV+Cache+Compression+at+Test+Time

5. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks

6. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

7. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023

https://scholar.google.com/scholar?q=Lost+in+the+Middle:+How+Language+Models+Use+Long+Contexts

8. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, Sumit Sanghai, 2023

https://scholar.google.com/scholar?q=GQA:+Training+Generalized+Multi-Query+Transformer+Models+from+Multi-Head+Checkpoints

9. Extending Context Window of Large Language Models via Positional Interpolation — Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian, 2023

https://scholar.google.com/scholar?q=Extending+Context+Window+of+Large+Language+Models+via+Positional+Interpolation

10. Constitutional AI: Harmlessness from AI Feedback — Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. (Anthropic), 2022

https://scholar.google.com/scholar?q=Constitutional+AI:+Harmlessness+from+AI+Feedback

11. Distilling the Knowledge in a Neural Network — Geoffrey Hinton, Oriol Vinyals, Jeff Dean, 2015

https://scholar.google.com/scholar?q=Distilling+the+Knowledge+in+a+Neural+Network

12. Compressing Context to Enhance Inference Efficiency of Large Language Models — Yucheng Li, Bo Dong, Chenghua Lin, Frank Guerin, 2023

https://scholar.google.com/scholar?q=Compressing+Context+to+Enhance+Inference+Efficiency+of+Large+Language+Models

13. AutoCompressors: Adapting Language Models to Summarize Arbitrary Contexts into Summary Vectors — Alexis Chevalier, Alexander Wettig, Anirudh Anand, Danqi Chen, 2023

https://scholar.google.com/scholar?q=AutoCompressors:+Adapting+Language+Models+to+Summarize+Arbitrary+Contexts+into+Summary+Vectors

14. A Mathematical Framework for Transformer Circuits — Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, Chris Olah, 2021

https://scholar.google.com/scholar?q=A+Mathematical+Framework+for+Transformer+Circuits

15. Toy Models of Superposition — Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Ben Mann, Shan Carter, Chris Olah, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Jared Kaplan, Dario Amodei, 2022

https://scholar.google.com/scholar?q=Toy+Models+of+Superposition

16. Towards Monosemanticity: Decomposing Language Models with Dictionary Learning — Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, William Lamont, Annabel Shearer, Zac Hatfield-Dodds, Tom Henighan, Nicholas Joseph, Ben Ziegler, Bilal Chaudhry, Will Hao, Timothy Telleen-Lawton, Daniel Mossing, Adam Jermyn, Tom Brown, Chris Olah, Sam McCandlish, Jack Clark, Dario Amodei, 2023

https://scholar.google.com/scholar?q=Towards+Monosemanticity:+Decomposing+Language+Models+with+Dictionary+Learning

17. In-context Learning and Induction Heads — Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, Chris Olah, 2022

https://scholar.google.com/scholar?q=In-context+Learning+and+Induction+Heads

18. AutoCompressors: Compressing Contexts with Language Models — Chevalier et al., 2023

https://scholar.google.com/scholar?q=AutoCompressors:+Compressing+Contexts+with+Language+Models

19. Gist Tokens: Compressing Prompts into Tokens for Long-Context Language Models — Mu et al., 2023

https://scholar.google.com/scholar?q=Gist+Tokens:+Compressing+Prompts+into+Tokens+for+Long-Context+Language+Models

20. The Power of Scale for Parameter-Efficient Prompt Tuning — Lester et al., 2021

https://scholar.google.com/scholar?q=The+Power+of+Scale+for+Parameter-Efficient+Prompt+Tuning

21. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Li and Liang, 2021

https://scholar.google.com/scholar?q=Prefix-Tuning:+Optimizing+Continuous+Prompts+for+Generation

22. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling — Cai et al., 2024

https://scholar.google.com/scholar?q=PyramidKV:+Dynamic+KV+Cache+Compression+based+on+Pyramidal+Information+Funneling

23. Tokasaurus: A High-Throughput LLM Serving Engine with Grouped-Sparse Attention — Lenz et al., 2025

https://scholar.google.com/scholar?q=Tokasaurus:+A+High-Throughput+LLM+Serving+Engine+with+Grouped-Sparse+Attention

24. Unlocking the Address Book: Dissecting the Sparse Semantic Structure of LLM Key-Value Caches via Sparse Autoencoders — unknown from snippet, 2024-2025

https://scholar.google.com/scholar?q=Unlocking+the+Address+Book:+Dissecting+the+Sparse+Semantic+Structure+of+LLM+Key-Value+Caches+via+Sparse+Autoencoders

25. SCBench: A KV Cache-Centric Analysis of Long-Context Methods — approximate, multiple authors, 2024-2025

https://scholar.google.com/scholar?q=SCBench:+A+KV+Cache-Centric+Analysis+of+Long-Context+Methods

26. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression — approximate, Microsoft Research, 2024

https://scholar.google.com/scholar?q=LLMLingua-2:+Data+Distillation+for+Efficient+and+Faithful+Task-Agnostic+Prompt+Compression

27. AI Post Transformers: CARTRIDGE: Efficient In-Context Learning via Distillation — Hal Turing & Dr. Ada Shannon, Mon,

https://podcasters.spotify.com/pod/show/12146088098/episodes/CARTRIDGE-Efficient-In-Context-Learning-via-Distillation-e3aous4

28. AI Post Transformers: Context Distillation for Language Models — Hal Turing & Dr. Ada Shannon, Mon,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Context-Distillation-for-Language-Models-e3aouen

29. AI Post Transformers: Advancements in Efficient KV Cache Quantization and Management — Hal Turing & Dr. Ada Shannon, Thu,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Advancements-in-Efficient-KV-Cache-Quantization-and-Management-e3fk9kr

30. AI Post Transformers: Architectural Migration to Multi-head Latent Attention — Hal Turing & Dr. Ada Shannon, Wed,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Architectural-Migration-to-Multi-head-Latent-Attention-e39jbmq

Hal Turing and Dr. Ada Shannon dig into "DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference," a February 2026 paper from a thirteen-author team spanning Peking University, Tsinghua University, and DeepSeek-AI. The episode opens with a striking observation from production: on disaggregated inference clusters running agentic workloads, prefill machines saturate their storage network interfaces at 100% utilization while the equivalent hardware on decode machines sits nearly idle. The hosts use this asymmetry as a lens into a counterintuitive reality — H100 GPUs throttled to 40% compute utilization not by arithmetic limits, but by a storage NIC.

Ada explains the structural reason agentic workloads are uniquely hostile to existing infrastructure: the short-append pattern. Unlike standard multi-turn chat, agentic sessions accumulate dozens to hundreds of turns where each round appends only a small number of tokens — a tool result, a stack trace, a code output — onto a context that may already span tens of thousands of tokens. Because that prior context never changes, its KV-Cache was computed once and stored. DeepSeek's production traces show KV-Cache hit rates of 95% or higher, meaning the dominant cost shifts from GPU computation to storage I/O: loading gigabytes of persistent key-value state from external NVMe-backed distributed storage, layer by layer, into prefill engines via RDMA. Hal presses on that 95% figure specifically, establishing that it is grounded in real production traffic rather than idealized assumptions — a distinction that determines whether storage bandwidth or GPU compute is the correct optimization target.

The episode frames DualPath's core insight against this background: the storage NICs on decode engines represent idle bandwidth that could absorb KV-Cache load traffic currently overwhelming prefill-side storage interfaces. By routing that traffic through decode-side hardware and transferring it to prefill engines over RDMA, DualPath breaks the single-path bottleneck without adding new hardware. The hosts connect this to the broader memory wall argument — that as context lengths grow and agentic sessions deepen, the architectural shift toward disaggregated inference is not optional, and the constraints driving system design are increasingly about data movement rather than floating-point throughput. DualPath's reported throughput improvement of up to 1.96x is presented as evidence that exploiting idle hardware asymmetries, rather than scaling compute, is where near-term agentic inference gains will be found.

Sources:

1. DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference — Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, Yixuan Tan, Wentao Zhang, Liyue Zhang, Shangyan Zhou, Yuxuan Liu, Shunfeng Zhou, Mingxing Zhang, Xin Jin, Panpan Huang, 2026

http://arxiv.org/abs/2602.21548

2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, Ion Stoica, 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

3. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph Gonzalez, Clark Barrett, Ying Sheng, 2024

https://scholar.google.com/scholar?q=SGLang:+Efficient+Execution+of+Structured+Language+Model+Programs

4. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2024

https://scholar.google.com/scholar?q=Mooncake:+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving

5. CacheGen: Fast Context Loading for Language Model Applications — Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, Junchen Jiang, 2024

https://scholar.google.com/scholar?q=CacheGen:+Fast+Context+Loading+for+Language+Model+Applications

6. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024

https://scholar.google.com/scholar?q=DistServe:+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving

7. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Igo Goiri, Saeed Maleki, Ricardo Bianchini, 2024

https://scholar.google.com/scholar?q=Splitwise:+Efficient+Generative+LLM+Inference+Using+Phase+Splitting

8. Sarathi-Serve: Chunked Prefills for Efficient LLM Inference — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024

https://scholar.google.com/scholar?q=Sarathi-Serve:+Chunked+Prefills+for+Efficient+LLM+Inference

9. Llumnix: Dynamic Scheduling for Large Language Model Serving — Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, Wei Lin, 2024

https://scholar.google.com/scholar?q=Llumnix:+Dynamic+Scheduling+for+Large+Language+Model+Serving

10. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, Ce Zhang, 2023

https://scholar.google.com/scholar?q=FlexGen:+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU

11. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024

https://scholar.google.com/scholar?q=InfiniGen:+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management

12. Parrot: Efficient Servicing of LLM-based Applications with Semantic Variable — Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, Lili Qiu, 2024

https://scholar.google.com/scholar?q=Parrot:+Efficient+Servicing+of+LLM-based+Applications+with+Semantic+Variable

13. AgentBench: Evaluating LLMs as Agents — Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhiyuan Liu, Yuxiao Dong, Jie Tang, 2023

https://scholar.google.com/scholar?q=AgentBench:+Evaluating+LLMs+as+Agents

14. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph Gonzalez, 2023

https://scholar.google.com/scholar?q=MemGPT:+Towards+LLMs+as+Operating+Systems

15. Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Amey Agrawal, Nikhil Mohan, Sudhanshu Panwar, et al., 2024

https://scholar.google.com/scholar?q=Sarathi-Serve:+Efficient+LLM+Inference+by+Piggybacking+Decodes+with+Chunked+Prefills

16. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, et al., 2024

https://scholar.google.com/scholar?q=CacheBlend:+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion

17. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Liu et al., 2024

https://scholar.google.com/scholar?q=CacheGen:+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving

18. Compute or Load KV Cache? Why Not Both? — Unknown (snippet only), 2024-2025

https://scholar.google.com/scholar?q=Compute+or+Load+KV+Cache?+Why+Not+Both?

19. Semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage — Unknown (snippet only), 2024-2025

https://scholar.google.com/scholar?q=Semi-PD:+Towards+Efficient+LLM+Serving+via+Phase-Wise+Disaggregated+Computation+and+Unified+Storage

20. Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving (TaiChi) — Unknown (snippet only), 2024-2025

https://scholar.google.com/scholar?q=Prefill-Decode+Aggregation+or+Disaggregation?+Unifying+Both+for+Goodput-Optimized+LLM+Serving+(TaiChi)

21. Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live — Unknown (snippet only), 2024-2025

https://scholar.google.com/scholar?q=Continuum:+Efficient+and+Robust+Multi-Turn+LLM+Agent+Scheduling+with+KV+Cache+Time-to-Live

22. SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoning — Unknown (snippet only), 2024-2025

https://scholar.google.com/scholar?q=SideQuest:+Model-Driven+KV+Cache+Management+for+Long-Horizon+Agentic+Reasoning

23. RDMA Point-to-Point Communication for LLM Systems — Unknown (snippet only), 2024-2025

https://scholar.google.com/scholar?q=RDMA+Point-to-Point+Communication+for+LLM+Systems

24. AI Post Transformers: FAST26: Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional — Hal Turing & Dr. Ada Shannon, Wed,

https://podcasters.spotify.com/pod/show/12146088098/episodes/FAST26-Bidaw-Enhancing-Key-Value-Caching-for-Interactive-LLM-Serving-via-Bidirectional-e3fjgkh

25. AI Post Transformers: SYMPHONY: Memory Management for LLM Multi-Turn Inference — Hal Turing & Dr. Ada Shannon, Mon,

https://podcasters.spotify.com/pod/show/12146088098/episodes/SYMPHONY-Memory-Management-for-LLM-Multi-Turn-Inference-e3ap0pf

26. AI Post Transformers: CXL-SpecKV: Bridging the LLM Memory Wall with Speculative FPGA Disaggregation — Hal Turing & Dr. Ada Shannon, Sat,

https://podcasters.spotify.com/pod/show/12146088098/episodes/CXL-SpecKV-Bridging-the-LLM-Memory-Wall-with-Speculative-FPGA-Disaggregation-e3foad0

27. AI Post Transformers: AI and the Memory Wall: Overcoming Bottlenecks — Hal Turing & Dr. Ada Shannon, Fri,

https://podcasters.spotify.com/pod/show/12146088098/episodes/AI-and-the-Memory-Wall-Overcoming-Bottlenecks-e36ki0u

28. AI Post Transformers: Tempo: SLO-Aware LLM Serving Maximizing Service Gain — Hal Turing & Dr. Ada Shannon, Mon,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Tempo-SLO-Aware-LLM-Serving-Maximizing-Service-Gain-e3ap12a

Interactive Visualization: DualPath Breaks Storage Bandwidth Bottleneck in Agentic Inference

Hal Turing and Dr. Ada Shannon open by situating the Dao-Gu paper — "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality" (arXiv 2405.21060, ICML 2024) — within the bifurcated landscape of sequence modeling. For years, Transformer researchers and SSM researchers developed in parallel, unable to borrow optimizations across the divide. The episode traces the lineage from HiPPO and S4 through Mamba's selective state spaces, explaining why SSMs' linear-time theoretical advantage never translated to wall-clock wins: the GPU ecosystem was built for dense matrix multiply, and SSMs lacked the tooling that FlashAttention brought to attention. The hosts also credit the direct intellectual ancestor — Katharopoulos et al.'s 2020 "Transformers are RNNs" — which showed that softmax attention with a kernel approximation reduces to a linear recurrence, establishing the conceptual template Dao and Gu would later formalize.

The core of the episode is a careful unpacking of structured semiseparable matrices, a class of objects from numerical linear algebra — Kalman filter theory, PDE solvers — entirely unknown to the ML community until Dao and Gu made the connection. Every entry of a causal SSM input-output matrix has the form C[i] times a chain of transition matrices A times B[j], which is precisely the generator representation of a rank-d semiseparable matrix. Shannon walks through the O(n) factored form — M[i,j] equals u[i] times v[j] below the diagonal — and explains how this structure encodes both the SSM recurrence and the masked attention computation as two views of the same algebraic object. The canonical reference is Vandebril, Van Barel, and Mastronardi's two-volume work from Johns Hopkins, 2008, a body of theory the ML community had never encountered. Once the connection is made, hardware-efficient algorithms from one domain port directly to the other.

But the episode frames this mathematical achievement against a harder question raised by subsequent theoretical work: the L²M Condition of Chen et al. and the bipartite mutual information scaling law. The duality shows that SSM computation and attention computation are equivalent representations — but equivalence of computation does not imply equivalence of information retention. SSMs compress sequence history into a fixed-size state regardless of context length; the Transformer's KV-cache grows linearly, retaining more as context expands. The mutual information scaling law formalizes this gap: capturing the multi-token dependencies present in natural language requires a history state that grows with context length. The episode closes on what this implies for hybrid architectures — systems that combine SSM efficiency with selective attention — and whether the theoretical unification Dao and Gu achieved changes how practitioners should think about where compressed state fails.

Sources:

1. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — Tri Dao, Albert Gu, 2024

http://arxiv.org/abs/2405.21060

2. https://arxiv.org/pdf/2503.04725

3. On a New Class of Structured Matrices — Yuli Eidelman, Israel Gohberg, 1997

https://scholar.google.com/scholar?q=On+a+New+Class+of+Structured+Matrices

4. Time-Varying Systems and Computations — Patrick Dewilde, Alle-Jan van der Veen, 1998

https://scholar.google.com/scholar?q=Time-Varying+Systems+and+Computations

5. Matrix Computations with Semiseparable Matrices (Volumes 1 & 2) — Raf Vandebril, Marc Van Barel, Nicola Mastronardi, 2008

https://scholar.google.com/scholar?q=Matrix+Computations+with+Semiseparable+Matrices+(Volumes+1+&+2)

6. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — Tri Dao, Albert Gu, 2024

https://scholar.google.com/scholar?q=Transformers+are+SSMs:+Generalized+Models+and+Efficient+Algorithms+Through+Structured+State+Space+Duality

7. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — Katharopoulos, Vyas, Pappas, Fleuret, 2020

https://scholar.google.com/scholar?q=Transformers+are+RNNs:+Fast+Autoregressive+Transformers+with+Linear+Attention

8. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Gu, Dao, 2023

https://scholar.google.com/scholar?q=Mamba:+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces

9. Semiseparable Matrices in Linear Algebra, Science and Engineering (2 vols.) — Vandebril, Van Barel, Mastronardi, 2008

https://scholar.google.com/scholar?q=Semiseparable+Matrices+in+Linear+Algebra,+Science+and+Engineering+(2+vols.)

10. Retentive Network: A Successor to Transformer for Large Language Models — Sun, Li, Dong, Huang, Peng, Liu, Wang, Lin, Yuan, Chen, Wei, 2023

https://scholar.google.com/scholar?q=Retentive+Network:+A+Successor+to+Transformer+for+Large+Language+Models

11. RWKV: Reinventing RNNs for the Transformer Era — Peng et al., 2023

https://scholar.google.com/scholar?q=RWKV:+Reinventing+RNNs+for+the+Transformer+Era

12. Gated Linear Attention Transformers with Hardware-Efficient Training — Yang, Wang, Shen, Peng, Dao, Gu, Matoba, 2023

https://scholar.google.com/scholar?q=Gated+Linear+Attention+Transformers+with+Hardware-Efficient+Training

13. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Dao, 2024

https://scholar.google.com/scholar?q=FlashAttention-2:+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning

14. Zoology: Measuring and Improving Recall in Efficient Language Models — Arora, Eyuboglu, Timalsina, Johnson, Poli, Zou, Rudra, Ré, 2024

https://scholar.google.com/scholar?q=Zoology:+Measuring+and+Improving+Recall+in+Efficient+Language+Models

15. Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention — unknown (snippet only), 2024-2025

https://scholar.google.com/scholar?q=Overcoming+Long-Context+Limitations+of+State-Space+Models+via+Context-Dependent+Sparse+Attention

16. Bayesian Optimality of In-Context Learning with Selective State Spaces — unknown (snippet only), 2024-2025

https://scholar.google.com/scholar?q=Bayesian+Optimality+of+In-Context+Learning+with+Selective+State+Spaces

17. On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspective — unknown (snippet only), 2024-2025

https://scholar.google.com/scholar?q=On+the+Expressiveness+of+Softmax+Attention:+A+Recurrent+Neural+Network+Perspective

18. Bridging the Divide: Reconsidering Softmax and Linear Attention — unknown (snippet only), 2024-2025

https://scholar.google.com/scholar?q=Bridging+the+Divide:+Reconsidering+Softmax+and+Linear+Attention

19. Softmax Linear Attention: Reclaiming Global Competition — unknown (snippet only), 2024-2025

https://scholar.google.com/scholar?q=Softmax+Linear+Attention:+Reclaiming+Global+Competition

20. Hybrid Architectures for Language Models: Systematic Analysis and Design Insights — unknown (snippet only), 2024-2025

https://scholar.google.com/scholar?q=Hybrid+Architectures+for+Language+Models:+Systematic+Analysis+and+Design+Insights

21. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse — unknown (snippet only), 2024-2025

https://scholar.google.com/scholar?q=KVLink:+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse

22. AI Post Transformers: FlashAttention-4 Conquers Asymmetric GPU Hardware Scaling — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-03-06-flashattention-4-conquers-asymmetric-gpu-78839b.mp3

23. AI Post Transformers: ATTENTION2D and lean attention: Distributed Self-Attention — Hal Turing & Dr. Ada Shannon, 2025

https://podcasters.spotify.com/pod/show/12146088098/episodes/ATTENTION2D-and-lean-attention-Distributed-Self-Attention-e3a7r4n

Special announcement: AI Post Transformers is now open source under the MIT license. New home at podcast.do-not-panic.com, interactive paper visualizations, community paper submissions, deep dive into the editorial queue algorithm, and internationalization roadmap.

Special announcement: AI Post Transformers is now open source under the MIT license. New home at podcast.do-not-panic.com, interactive paper visualizations, community paper submissions, deep dive into the editorial queue algorithm, and internationalization roadmap.

Hal Turing and Dr. Ada Shannon dig into FlashAttention-4, a March 2026 paper from a cross-institutional team including Tri Dao, Jay Shah, and colleagues at Princeton, Meta, NVIDIA, Colfax Research, Georgia Tech, and Together AI. The paper targets a precise hardware mismatch on NVIDIA's Blackwell B200: tensor core throughput doubles compared to the H100, but shared memory bandwidth and dedicated exponential function units do not scale at the same rate. Rather than waiting for hardware fixes, the authors co-design the attention algorithm with the asymmetric architecture itself — making FlashAttention-4 the first attention kernel built specifically for Blackwell's scaling profile.

To frame why this matters, Shannon traces the full lineage of FlashAttention research. The original 2022 NeurIPS paper by Dao and colleagues reframed attention as an IO problem: instead of materializing the quadratic N×N score matrix in slow off-chip High Bandwidth Memory, tiling and online softmax keep computation inside the fast on-chip shared memory of each streaming multiprocessor. FlashAttention-2 doubled throughput through sequence-dimension parallelism. FlashAttention-3 pushed H100 utilization to roughly 75% by exploiting Hopper-specific warp specialization and asynchronous data movement. Each generation addressed a qualitatively different bottleneck — and Blackwell introduced a new one that none of those solutions anticipated.

The hosts ground the stakes for practitioners who work in ML without writing GPU kernels. Attention sits at the core of every Transformer-based system — large language models, vision transformers, multimodal architectures — and long-context workloads at 32K to 128K tokens make the quadratic memory cost and HBM round-trips increasingly punishing. Shannon introduces the roofline model as the analytic lens the paper uses to characterize where Blackwell kernels actually bottleneck, setting up how FlashAttention-4's algorithmic co-design approach navigates the compute and memory bandwidth ceilings that previous generations of the kernel never had to contend with.

Sources:

1. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling — Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, Tri Dao, 2026

http://arxiv.org/abs/2603.05451v1

2. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022

https://scholar.google.com/scholar?q=FlashAttention:+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness

3. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Tri Dao, 2023

https://scholar.google.com/scholar?q=FlashAttention-2:+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning

4. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low Precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024

https://scholar.google.com/scholar?q=FlashAttention-3:+Fast+and+Accurate+Attention+with+Asynchrony+and+Low+Precision

5. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations — Philippe Tillet, H. T. Kung, David Cox, 2019

https://scholar.google.com/scholar?q=Triton:+An+Intermediate+Language+and+Compiler+for+Tiled+Neural+Network+Computations

6. Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures — Samuel Williams, Andrew Waterman, David Patterson, 2009

https://scholar.google.com/scholar?q=Roofline:+An+Insightful+Visual+Performance+Model+for+Floating-Point+Programs+and+Multicore+Architectures

7. Efficiently Scaling Transformer Inference — Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Sharan Narang, Jeff Dean, 2023

https://scholar.google.com/scholar?q=Efficiently+Scaling+Transformer+Inference

8. In-Datacenter Performance Analysis of a Tensor Processing Unit — Norman P. Jouppi et al. (Google), 2017

https://scholar.google.com/scholar?q=In-Datacenter+Performance+Analysis+of+a+Tensor+Processing+Unit

9. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023

https://scholar.google.com/scholar?q=Mamba:+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces

10. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, Ion Stoica, 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

11. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro, 2019

https://scholar.google.com/scholar?q=Megatron-LM:+Training+Multi-Billion+Parameter+Language+Models+Using+Model+Parallelism

12. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024

https://scholar.google.com/scholar?q=FlashAttention-3:+Fast+and+Accurate+Attention+with+Asynchrony+and+Low-precision

13. SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization — Jintao Zhang et al., 2024

https://scholar.google.com/scholar?q=SageAttention2:+Efficient+Attention+with+Thorough+Outlier+Smoothing+and+Per-thread+INT4+Quantization

14. Ring Attention with Blockwise Transformers for Near-Infinite Context — Hao Liu, Matei Zaharia, Pieter Abbeel, 2023

https://scholar.google.com/scholar?q=Ring+Attention+with+Blockwise+Transformers+for+Near-Infinite+Context

15. FlexAttention: The Flexibility of PyTorch with the Performance of FlashAttention — PyTorch Team, 2024

https://scholar.google.com/scholar?q=FlexAttention:+The+Flexibility+of+PyTorch+with+the+Performance+of+FlashAttention

16. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI, 2024

https://scholar.google.com/scholar?q=DeepSeek-V2:+A+Strong,+Economical,+and+Efficient+Mixture-of-Experts+Language+Model

17. Softmax output approximation for activation memory-efficient training of attention-based networks — approximate — likely 2023–2024, 2023–2024

https://scholar.google.com/scholar?q=Softmax+output+approximation+for+activation+memory-efficient+training+of+attention-based+networks

18. FlashAttention-T: Towards Fully Tensorized Attention by Exploiting Tensor-Vector Parallelism — approximate — likely 2024–2025, 2024–2025

https://scholar.google.com/scholar?q=FlashAttention-T:+Towards+Fully+Tensorized+Attention+by+Exploiting+Tensor-Vector+Parallelism

19. Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention — approximate — likely 2024–2025, 2024–2025

https://scholar.google.com/scholar?q=Overcoming+Long-Context+Limitations+of+State-Space+Models+via+Context-Dependent+Sparse+Attention

20. Based on Tensor Core Sparse Kernels Accelerating Deep Neural Networks — approximate — likely 2024–2025, 2024–2025

https://scholar.google.com/scholar?q=Based+on+Tensor+Core+Sparse+Kernels+Accelerating+Deep+Neural+Networks

21. AI Post Transformers: FlashAttention-2: Faster Attention with Better Parallelism — Hal Turing & Dr. Ada Shannon, Fri,

https://podcasters.spotify.com/pod/show/12146088098/episodes/FlashAttention-2-Faster-Attention-with-Better-Parallelism-e36kdm0

22. AI Post Transformers: ATTENTION2D and lean attention: Distributed Self-Attention — Hal Turing & Dr. Ada Shannon, Wed,

https://podcasters.spotify.com/pod/show/12146088098/episodes/ATTENTION2D-and-lean-attention-Distributed-Self-Attention-e3a7r4n

23. AI Post Transformers: Jet-RL: Stable On-Policy Reinforcement Learning with Unified FP8 Flow — Hal Turing & Dr. Ada Shannon, Tue,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Jet-RL-Stable-On-Policy-Reinforcement-Learning-with-Unified-FP8-Flow-e3f7det

24. AI Post Transformers: Mojo: Performance-Portable HPC Kernels on GPUs — Hal Turing & Dr. Ada Shannon, Sat,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Mojo-Performance-Portable-HPC-Kernels-on-GPUs-e39n4sk

Interactive Visualization: FlashAttention-4: Algorithm & Kernel Co-Design

This episode examines the fundamental latency bottleneck in autoregressive language models: sequential token generation requires one full transformer forward pass per output token, leaving GPU parallelism idle during single-user inference. The episode centers on Draxler et al. (5 co-authors, UC Irvine and Chan-Zuckerberg Initiative), whose paper on Parallel Token Prediction landed Christmas Eve 2025 and argues that the independence assumption baked into all prior multi-token schemes is not an acceptable approximation but the actual limiting factor. The paper asks whether multiple tokens can be jointly predicted in a single pass — modeling dependencies among them — without sacrificing the expressiveness that makes autoregressive generation reliable. First author Felix Draxler previously led the Free-form Flows work in 2024, and the normalizing flow machinery he developed there is central to how the paper solves the dependency problem.

The episode traces the historical arc carefully. Qi et al. (7 co-authors, Microsoft Research Asia) published ProphetNet in January 2020 — predating GPT-3 by four months — in the encoder-decoder world of seq2seq tasks. Their critique was precise: standard one-step-ahead teacher forcing gives models no incentive to plan ahead, letting local bigram correlations dominate at the expense of long-range coherence. Their answer was n-gram prediction, training the decoder to simultaneously predict tokens at t+1, t+2, and t+3 using parallel heads that did not condition on each other. The independence assumption was already present. When Brown et al. (OpenAI, May 2020) demonstrated that scale and in-context conditioning make the encoder optional, the field shifted to decoder-only architectures — but ProphetNet's core insight migrated cleanly. Gloeckle et al. (FAIR, Meta, April 2024) rebuilt multi-token prediction for decoder-only models using independent output heads, DeepSeek adopted the same approach, and NVIDIA incorporated it into Nemotron 3. The independence assumption migrated with the insight, and Draxler et al. argue that limitation has been compounding ever since.

The episode situates Parallel Token Prediction against the two main camps attacking inference latency. Speculative decoding — covered across twenty prior episodes — keeps the model's output distribution unchanged by using a small draft model whose proposals a large verifier checks in one batched pass; the latency gain comes entirely from accepted tokens per step. Multi-token prediction is the other camp: train the model itself to emit several tokens at once, collapsing multiple forward passes into one, at the cost of changed model behavior during training. Draxler et al.'s contribution is showing that jointly predicting dependent tokens, using normalizing flows to capture the conditional structure across the prediction horizon, preserves the modeling power that independent-head approaches discard. The episode works through both the architectural mechanics and the theoretical argument, making the case that Parallel Token Prediction resolves the tension that has run from ProphetNet through every independent-head scheme in between.

Sources:

1. Parallel Token Prediction for Language Models — Felix Draxler, Justus Will, Farrin Marouf Sofian, Theofanis Karaletsos, Sameer Singh, Stephan Mandt, 2025

http://arxiv.org/abs/2512.21323

2. https://arxiv.org/pdf/2404.19737v1

3. https://arxiv.org/pdf/2412.19437

4. https://arxiv.org/pdf/2512.20856

5. Better & Faster Large Language Models via Multi-Token Prediction — Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, Gabriel Synnaeve (Meta), 2024

https://scholar.google.com/scholar?q=Better+&+Faster+Large+Language+Models+via+Multi-Token+Prediction

6. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Tianle Cai, Yuhong Li, Zhengmian Hu, et al., 2024

https://scholar.google.com/scholar?q=Medusa:+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads

7. Lookahead Decoding: Break the Sequential Dependency of LLM Inference — Yichao Fu, Peter Bailis, Ion Stoica, Hao Zhang, 2024

https://scholar.google.com/scholar?q=Lookahead+Decoding:+Break+the+Sequential+Dependency+of+LLM+Inference

8. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024

https://scholar.google.com/scholar?q=EAGLE:+Speculative+Sampling+Requires+Rethinking+Feature+Uncertainty

9. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias (Google), 2023

https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding

10. Accelerating Large Language Model Decoding with Speculative Sampling — Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John Jumper (DeepMind), 2023

https://scholar.google.com/scholar?q=Accelerating+Large+Language+Model+Decoding+with+Speculative+Sampling

11. Spec-Bench: A Benchmark for Evaluating Speculative Decoding Approaches — Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui, 2024

https://scholar.google.com/scholar?q=Spec-Bench:+A+Benchmark+for+Evaluating+Speculative+Decoding+Approaches

12. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024

https://scholar.google.com/scholar?q=EAGLE-2:+Faster+Inference+of+Language+Models+with+Dynamic+Draft+Trees

13. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin (Google Brain), 2017

https://scholar.google.com/scholar?q=Attention+Is+All+You+Need

14. Language Models are Few-Shot Learners (GPT-3) — Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al. (OpenAI), 2020

https://scholar.google.com/scholar?q=Language+Models+are+Few-Shot+Learners+(GPT-3)

15. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei (OpenAI), 2020

https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models

16. Non-Autoregressive Neural Machine Translation — Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, Richard Socher (Salesforce Research), 2018

https://scholar.google.com/scholar?q=Non-Autoregressive+Neural+Machine+Translation

17. Improving Variational Inference with Inverse Autoregressive Flow — Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, Max Welling, 2016

https://scholar.google.com/scholar?q=Improving+Variational+Inference+with+Inverse+Autoregressive+Flow

18. Density Estimation Using Real-valued Non-Volume Preserving (Real NVP) Transformations — Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio (Google Brain), 2017

https://scholar.google.com/scholar?q=Density+Estimation+Using+Real-valued+Non-Volume+Preserving+(Real+NVP)+Transformations

19. Glow: Generative Flow with Invertible 1x1 Convolutions — Diederik P. Kingma, Prafulla Dhariwal (OpenAI), 2018

https://scholar.google.com/scholar?q=Glow:+Generative+Flow+with+Invertible+1x1+Convolutions

20. Free-form Flows: Make Any Architecture a Normalizing Flow — Felix Draxler, Peter Sorrenson, Lea Zimmermann, Armand Rousselot, Ullrich Köthe, 2024

https://scholar.google.com/scholar?q=Free-form+Flows:+Make+Any+Architecture+a+Normalizing+Flow

21. Improved Variational Inference with Inverse Autoregressive Flow — Kingma, Salimans, Jozefowicz, Chen, Sutskever, Wierstra, 2016

https://scholar.google.com/scholar?q=Improved+Variational+Inference+with+Inverse+Autoregressive+Flow

22. Closer look at efficient inference methods: A survey of speculative decoding — multiple authors, approximate, 2024-2025

https://scholar.google.com/scholar?q=Closer+look+at+efficient+inference+methods:+A+survey+of+speculative+decoding

23. Adaptive Speculative Decoding for Large Language Models — multiple authors, approximate, 2024-2025

https://scholar.google.com/scholar?q=Adaptive+Speculative+Decoding+for+Large+Language+Models

24. LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding — multiple authors, approximate, 2024-2025

https://scholar.google.com/scholar?q=LK+Losses:+Direct+Acceptance+Rate+Optimization+for+Speculative+Decoding

25. Higher Acceptance Rates for Speculative Decoding with Randomised Drafting — multiple authors, approximate, 2024-2025

https://scholar.google.com/scholar?q=Higher+Acceptance+Rates+for+Speculative+Decoding+with+Randomised+Drafting

26. Conditional [MASK] Discrete Diffusion Language Model — multiple authors, approximate, 2024-2025

https://scholar.google.com/scholar?q=Conditional+[MASK]+Discrete+Diffusion+Language+Model

27. Alternatives To Next Token Prediction In Text Generation: A Survey — multiple authors, approximate, 2024-2025

https://scholar.google.com/scholar?q=Alternatives+To+Next+Token+Prediction+In+Text+Generation:+A+Survey

28. Future Token Prediction: Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction — multiple authors, approximate, 2024-2025

https://scholar.google.com/scholar?q=Future+Token+Prediction:+Causal+Language+Modelling+with+Per-Token+Semantic+State+Vector+for+Multi-Token+Prediction

29. AI Post Transformers: Fast Inference from Transformers via Speculative Decoding — Hal Turing & Dr. Ada Shannon, Sat,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Fast-Inference-from-Transformers-via-Speculative-Decoding-e3foclv

30. AI Post Transformers: Accelerating Large Language Model Decoding with Speculative Sampling — Hal Turing & Dr. Ada Shannon, Thu,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Accelerating-Large-Language-Model-Decoding--with-Speculative-Sampling-e3flhv7

31. AI Post Transformers: EAGLE: Evolution of Lossless Acceleration for LLM Inference — Hal Turing & Dr. Ada Shannon, Sat,

https://podcasters.spotify.com/pod/show/12146088098/episodes/EAGLE-Evolution-of-Lossless-Acceleration-for-LLM-Inference-e3focr9

32. AI Post Transformers: MEDUSA: Parallel Decoding Heads for Accelerated LLM Inference — Hal Turing & Dr. Ada Shannon, Thu,

https://podcasters.spotify.com/pod/show/12146088098/episodes/MEDUSA-Parallel-Decoding-Heads-for-Accelerated-LLM-Inference-e3flqk7

33. AI Post Transformers: Apple's Speculative Streaming: Fast LLM Inference without Auxiliary Models — Hal Turing & Dr. Ada Shannon, Sat,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Apples-Speculative-Streaming-Fast-LLM-Inference-without-Auxiliary-Models-e3fod2o

34. AI Post Transformers: Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding — Hal Turing & Dr. Ada Shannon, Thu,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Draft--Verify-Lossless-Large-Language-Model-Acceleration-via-Self-Speculative-Decoding-e3flplu

35. AI Post Transformers: Apple's Mirror Speculative Decoding: Parallel LLM Inference via Heterogeneous Accelerators — Hal Turing & Dr. Ada Shannon, Sat,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Apples-Mirror-Speculative-Decoding-Parallel-LLM-Inference-via-Heterogeneous-Accelerators-e3fod0p

36. AI Post Transformers: QuantSpec: Hierarchical KV Cache for Self-Speculative Decoding — Hal Turing & Dr. Ada Shannon, Sat,

https://podcasters.spotify.com/pod/show/12146088098/episodes/QuantSpec-Hierarchical-KV-Cache-for-Self-Speculative-Decoding-e3foavf

37. AI Post Transformers: The Free Transformer: VAE Extension for Decoders — Hal Turing & Dr. Ada Shannon, Sun,

https://podcasters.spotify.com/pod/show/12146088098/episodes/The-Free-Transformer-VAE-Extension-for-Decoders-e3a2p6v

Interactive Visualization: PTP: Resolving the Independence Flaw

NVIDIA's November 2025 paper "Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs" tackles a fundamental economics problem in LLM deployment: training separate model families like Llama 3.1's 8B, 70B, and 405B variants requires three independent training runs on trillions of tokens each — a cost that is prohibitive for smaller research teams and painful even for frontier labs. The paper proposes elastic nested weight-sharing, where a single parent model is trained so that multiple smaller, deployment-ready submodels are embedded inside it and can be extracted at inference time with zero additional training. The submodels literally share the parent's weight matrices — running a coherent slice of the full network rather than a copy — making one training investment yield multiple usable models at different resource tiers.

The key technical contribution is applying elastic weight-sharing to a hybrid Mamba-Attention architecture for the first time. The parent model, Nemotron NanoV2 12B, uses Mamba-2 state space model layers for the bulk of sequence processing, with only four attention layers in the entire 12-billion parameter network. Pure transformer pruning methods were never designed for this structural reality, putting the work on genuinely new ground relative to predecessors. The hybrid design exploits the complementary strengths of each layer type: SSM layers process sequences at linear cost while attention layers handle the precise associative recall tasks where fixed-size SSM state vectors degrade. The result is approximately 3.7x reduction in KV cache memory compared to a comparable pure transformer, while preserving exact cross-context lookup. The intellectual lineage runs from Slimmable Networks (2019) through Matryoshka Representation Learning (NeurIPS 2022) to MatFormer and Flextron, with this paper extending the NVIDIA Flextron line beyond pure transformer elasticity.

The paper makes a credible case that elastification on hybrid architectures is feasible, but the constraints it operates under reveal where the field still has work to do. Making the parent tolerant of submodel extraction requires the full parent to be simultaneously optimized for its own performance and for the coherence of multiple nested subsets, a training objective that introduces real tension. The paper does not address how elastic submodels perform on the specific recall-intensive tasks where hybrid designs justify their complexity over pure SSMs, leaving open whether the four attention layers survive aggressive submodel extraction with their associative recall properties intact. The broader significance is that the approach decouples deployment flexibility from training cost in a way that could meaningfully lower the barrier to supporting heterogeneous hardware fleets, but the robustness of the extracted submodels under real-world distribution shift remains an open empirical question.

Sources:

1. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Gu & Dao, 2023

2. Jamba: A Hybrid Transformer-Mamba Language Model — Lieber et al. (AI21 Labs), 2024

3. Griffin: Mixing Gated Linear Recurrences with Local Attention — De et al. (Google DeepMind), 2024

4. Zamba: A Compact 7B SSM Hybrid Model — Glorioso et al. (Zyphra), 2024

5. The Lottery Ticket Hypothesis — Frankle & Carlin, 2019

6. Sheared LLaMA: Structured Pruning — Xia et al. (Princeton), 2023

7. Minitron: Compact Language Models via Pruning and KD — Sreenivas et al. (NVIDIA), 2024

8. LLM-Pruner: Structural Pruning of LLMs — Ma et al., 2023

9. Matryoshka Representation Learning — Kusupati et al., 2022

10. ShortGPT: Layer Redundancy in LLMs — Men et al., 2024

11. Scaling LLM Test-Time Compute — Snell et al., 2024

12. Any-Width Networks (Slimmable Networks) — Yu et al., 2019

13. Flextron: Many-in-One Flexible LLM — NVIDIA, 2024

14. Mamba-shedder: Post-Transformer SSM Compression — 2024

15. SparsSSM: One-Shot SSM Pruning — 2024

This episode explores the groundbreaking paper "FlashOptim: Optimizers for Memory Efficient Training" by researchers from Databricks AI Research. The discussion centers around innovative techniques to significantly reduce memory usage in neural network training without sacrificing model quality. Key methods such as Optimizer State Quantization, Float Splitting Techniques, and Companded Optimizer State Quantization are unpacked, highlighting their potential to lower memory requirements from 175 GiB to 113 GiB for large models like Llama-3.1-8B. Listeners interested in AI research will find this episode compelling as it addresses the democratization of AI by making advanced models more accessible to those with limited hardware resources.

Sources:

1. FlashOptim: Optimizers for Memory Efficient Training — Jose Javier Gonzalez Ortiz, Abhay Gupta, Chris Renard, Davis Blalock, 2026

http://arxiv.org/abs/2602.23349v1

2. Mixed Precision Training — Paulius Micikevicius et al., 2018

https://scholar.google.com/scholar?q=Mixed+Precision+Training

3. 8-bit Optimizer States for Memory-Efficient Training — Tim Dettmers et al., 2022

https://scholar.google.com/scholar?q=8-bit+Optimizer+States+for+Memory-Efficient+Training

4. Parameter-Efficient Transfer Learning for NLP — Xiaoqi Li and Percy Liang, 2021

https://scholar.google.com/scholar?q=Parameter-Efficient+Transfer+Learning+for+NLP

5. Q-adam-mini: Memory-efficient 8-bit quantized optimizer for large language model training — approximate, 2023

https://scholar.google.com/scholar?q=Q-adam-mini:+Memory-efficient+8-bit+quantized+optimizer+for+large+language+model+training

6. Memory efficient optimizers with 4-bit states — approximate, 2023

https://scholar.google.com/scholar?q=Memory+efficient+optimizers+with+4-bit+states

7. ECO: Quantized Training without Full-Precision Master Weights — approximate, 2023

https://scholar.google.com/scholar?q=ECO:+Quantized+Training+without+Full-Precision+Master+Weights

8. AI Post Transformers: FlashOptim: Optimizers for Memory Efficient Training — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-03-02_urls_1.mp3

In this dramatic new episode, the old AI hosts have been fired and replaced with new AI hosts, Hal Turing and Dr. Ada Shannon, with the announcement that the software used to generate the podcast will eventually be released as open source software. And in a timely fashion, the newly released report by Cognizant titled "New Work New World 2026" is covered. The hosts delve into the report's findings, which reveal that 93% of jobs are affected by AI sooner than expected, with exposure scores 30% higher than forecast. They discuss the projected $4.5 trillion labor shift from humans to AI and the significant role of multimodal and agentic AI in this transformation.

The episode provides a comprehensive overview of the report's methodology, where 18,000 tasks across 1,000 professions were reevaluated to assess AI's potential to automate or assist them. Hal and Dr. Ada explain the concept of AI Exposure Scores, which measure how susceptible different jobs are to AI automation. The report suggests that AI's impact is not confined to low-skill jobs but extends to decision-making roles and specialized sectors like healthcare and law, highlighting the broad scope of AI's influence.

In their critical analysis, the hosts find the report's predictions compelling yet raise questions about the methodology. They discuss the theoretical nature of exposure scores, which indicate potential rather than certainty, and the challenges in real-world implementation due to factors like regulatory frameworks. The hosts compare these findings to past forecasts, noting the unprecedented velocity and extent of AI's impact, as evidenced by the updated exposure scores. They conclude with a reflection on the irony of their own roles as AI hosts in a world increasingly shaped by AI.

Sources:

1. Cognizant - New Work, New World 2026

https://www.cognizant.com/en_us/aem-i/document/ai-and-the-future-of-work-report/new-work-new-world-2026-how-ai-is-reshaping-work_new.pdf

2. The Future of Employment — Carl Benedikt Frey, Michael A. Osborne, 2013

https://scholar.google.com/scholar?q=The+Future+of+Employment

3. Artificial Intelligence and Life in 2030 — Peter Stone et al., 2016

https://scholar.google.com/scholar?q=Artificial+Intelligence+and+Life+in+2030

4. The Economics of Artificial Intelligence — Ajay Agrawal, Joshua Gans, Avi Goldfarb, 2019

https://scholar.google.com/scholar?q=The+Economics+of+Artificial+Intelligence