Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- ASI-Evolve for Data, Architectures, and RLThis episode explores whether an agentic AI system can meaningfully improve AI itself across three hard parts of the pipeline: pretraining data curation, neural architecture search, and reinforcement learning algorithm design, using the paper ASI-Evolve as the focal point. It argues that this is a step beyond traditional AutoML, framing “AI-for-AI” as automating parts of the research loop itself—reading prior work, proposing changes, running experiments, interpreting noisy results, and deciding what to try next. The discussion highlights why this is difficult: real ML research involves expensive, delayed, and ambiguous feedback rather than clean benchmark-style signals, making claims of a unified framework especially significant and worth skepticism. Listeners would find it interesting for its clear breakdown of what makes autonomous AI research different from ordinary model assistance, and for its debate over whether recent systems are genuine progress toward automating frontier AI development or still mostly polished demos.Sources:1. ASI-Evolve: AI Accelerates AI — Weixian Xu, Tiantian Mi, Yixiu Liu, Yang Nan, Zhimeng Zhou, Lyumanshan Ye, Lin Zhang, Yu Qiao, Pengfei Liu, 2026http://arxiv.org/abs/2603.296402. AutoML: A Survey of the State-of-the-Art — Xin He, Kaiyong Zhao, Xiaowen Chu, 2021https://scholar.google.com/scholar?q=AutoML%3A+A+Survey+of+the+State-of-the-Art3. Large Language Models as Optimizers — Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou and others, 2024https://scholar.google.com/scholar?q=Large+Language+Models+as+Optimizers4. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery — Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster and others, 2024https://scholar.google.com/scholar?q=The+AI+Scientist%3A+Towards+Fully+Automated+Open-Ended+Scientific+Discovery5. AlphaEvolve — Novikov et al., 2025https://scholar.google.com/scholar?q=AlphaEvolve6. Neural Architecture Search with Reinforcement Learning — Barret Zoph, Quoc V. Le, 2017https://scholar.google.com/scholar?q=Neural+Architecture+Search+with+Reinforcement+Learning7. Regularized Evolution for Image Classifier Architecture Search — Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V. Le and others, 2019https://scholar.google.com/scholar?q=Regularized+Evolution+for+Image+Classifier+Architecture+Search8. DARTS: Differentiable Architecture Search — Hanxiao Liu, Karen Simonyan, Yiming Yang, 2019https://scholar.google.com/scholar?q=DARTS%3A+Differentiable+Architecture+Search9. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks — Mingxing Tan, Quoc V. Le, 2019https://scholar.google.com/scholar?q=EfficientNet%3A+Rethinking+Model+Scaling+for+Convolutional+Neural+Networks10. The Pile: An 800GB Dataset of Diverse Text for Language Modeling — Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster and others, 2020https://scholar.google.com/scholar?q=The+Pile%3A+An+800GB+Dataset+of+Diverse+Text+for+Language+Modeling11. What Language Model to Train if You Have One Million GPU Hours? — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford and others, 2022https://scholar.google.com/scholar?q=What+Language+Model+to+Train+if+You+Have+One+Million+GPU+Hours%3F12. FineWeb — Hugging Face researchers and collaborators, 2024https://scholar.google.com/scholar?q=FineWeb13. DCLM: DataComp for Language Models — DataComp-LM collaborators, 2024https://scholar.google.com/scholar?q=DCLM%3A+DataComp+for+Language+Models14. Discovering Reinforcement Learning Algorithms — Benjamin Eysenbach, Ruslan Salakhutdinov, Sergey Levine and others, 2021https://scholar.google.com/scholar?q=Discovering+Reinforcement+Learning+Algorithms15. Learned Optimizers that Scale and Generalize — Researchers from Google and collaborators, including Liyuan Liu, Andrew Dai, and others, 2022https://scholar.google.com/scholar?q=Learned+Optimizers+that+Scale+and+Generalize16. Proximal Policy Optimization Algorithms — John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017https://scholar.google.com/scholar?q=Proximal+Policy+Optimization+Algorithms17. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — DeepSeek-AI authors, 2024https://scholar.google.com/scholar?q=DeepSeekMath%3A+Pushing+the+Limits+of+Mathematical+Reasoning+in+Open+Language+Models18. AI Scientist — Lu et al., 2024https://scholar.google.com/scholar?q=AI+Scientist19. MLEvolve — Du et al., 2025https://scholar.google.com/scholar?q=MLEvolve20. GEPA — Unknown from excerpt, 2025https://scholar.google.com/scholar?q=GEPA21. OpenEvolve — Unknown from excerpt, 2025https://scholar.google.com/scholar?q=OpenEvolve22. DeltaNet — Yang et al., 2025https://scholar.google.com/scholar?q=DeltaNet23. Recent human-designed improvements over DeltaNet — Dao and Gu, 2024https://scholar.google.com/scholar?q=Recent+human-designed+improvements+over+DeltaNet24. GRPO — Guo et al., 2025https://scholar.google.com/scholar?q=GRPO25. MMLU — Hendrycks et al., 2021https://scholar.google.com/scholar?q=MMLU26. SciMaster — Chai et al., 2025https://scholar.google.com/scholar?q=SciMaster27. From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery — approx. recent survey, authors unclear from snippet, 2025https://scholar.google.com/scholar?q=From+AI+for+Science+to+Agentic+Science%3A+A+Survey+on+Autonomous+Scientific+Discovery28. DiscoveryWorld: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents — approx. authors unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=DiscoveryWorld%3A+A+Virtual+Environment+for+Developing+and+Evaluating+Automated+Scientific+Discovery+Agents29. AI, Agentic Models and Lab Automation for Scientific Discovery—the Beginning of scAInce — approx. authors unclear from snippet, 2025https://scholar.google.com/scholar?q=AI%2C+Agentic+Models+and+Lab+Automation+for+Scientific+Discovery%E2%80%94the+Beginning+of+scAInce30. SciAgents: Automating Scientific Discovery Through Bioinspired Multi-Agent Intelligent Graph Reasoning — approx. authors unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=SciAgents%3A+Automating+Scientific+Discovery+Through+Bioinspired+Multi-Agent+Intelligent+Graph+Reasoning31. Optimization Problem Solving Can Transition to Evolutionary Agentic Workflows — approx. authors unclear from snippet, 2025https://scholar.google.com/scholar?q=Optimization+Problem+Solving+Can+Transition+to+Evolutionary+Agentic+Workflows32. AVO: Agentic Variation Operators for Autonomous Evolutionary Search — approx. authors unclear from snippet, 2025https://scholar.google.com/scholar?q=AVO%3A+Agentic+Variation+Operators+for+Autonomous+Evolutionary+Search33. Toward Weight-level Self-improving Agents with Meta-knowledge Discovery — approx. authors unclear from snippet, 2025/2026https://scholar.google.com/scholar?q=Toward+Weight-level+Self-improving+Agents+with+Meta-knowledge+Discovery34. AI Post Transformers: Kimi Linear: Efficient Expressive Attention Architecture — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/kimi-linear-efficient-expressive-attention-architecture/35. AI Post Transformers: Training-Free GRPO: Policy Optimization via Context Space — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/training-free-grpo-policy-optimization-via-context-space/36. AI Post Transformers: Experiential Reinforcement Learning: Internalizing Reflection for Better Policy Training — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/experiential-reinforcement-learning-internalizing-reflection-for-better-policy-t/37. AI Post Transformers: Kosmos AI Scientist for Autonomous Discovery — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-kosmos-ai-scientist-for-autonomous-disco-311775.mp338. AI Post Transformers: HyperController: Fast, Stable Reinforcement Learning Hyperparameter Optimization — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/hypercontroller-fast-stable-reinforcement-learning-hyperparameter-optimization/39. AI Post Transformers: NeurIPS 2025: Reinforcement Learning for Reasoning in Large Language Models with One Training Example — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/neurips-2025-reinforcement-learning-for-reasoning-in-large-language-models-with/Interactive Visualization: ASI-Evolve for Data, Architectures, and RL0 min
- Latent Space as a New Computational ParadigmThis episode explores a major 2026 survey arguing that latent space in language models should be treated not as hidden plumbing, but as the primary substrate of computation—distinct from both human-readable token space and the latent spaces used in image generation. It traces the idea back to representation learning, transformers, and variational autoencoders, then explains how newer work reframes continuous internal states as a workspace for reasoning, planning, memory, and multimodal fusion rather than just intermediate features for next-token prediction. A central argument is that forcing every internal step into language is inefficient: text is useful for communication, but dense vector states may be better suited for compact, general-purpose computation and memory. Listeners interested in where AI systems may be headed will find it compelling because it offers a concrete framework for thinking about models that increasingly “think” in latent representations while using language mainly as an interface.Sources:1. The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook — Xinlei Yu, Zhangquan Chen, Yongbo He, Tianyu Fu, Cheng Yang, Chengming Xu, Yue Ma, Xiaobin Hu, Zhe Cao, Jie Xu, Guibin Zhang, Jiale Tao, Jiayi Zhang, Siyuan Ma, Kaituo Feng, Haojie Huang, Youxing Li, Ronghao Chen, Huacan Wang, Chenglin Wu, Zikun Su, Xiaogang Xu, Kelu Yao, Kun Wang, Chen Gao, Yue Liao, Ruqi Huang, Tao Jin, Cheng Tan, Jiangning Zhang, Wenqi Ren, Yanwei Fu, Yong Liu, Yu Wang, Xiangyu Yue, Yu-Gang Jiang, Shuicheng Yan, 2026http://arxiv.org/abs/2604.020292. Auto-Encoding Variational Bayes — Diederik P. Kingma, Max Welling, 2013https://scholar.google.com/scholar?q=Auto-Encoding+Variational+Bayes3. Representation Learning: A Review and New Perspectives — Yoshua Bengio, Aaron Courville, Pascal Vincent, 2013https://scholar.google.com/scholar?q=Representation+Learning%3A+A+Review+and+New+Perspectives4. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, 2017https://scholar.google.com/scholar?q=Attention+Is+All+You+Need5. On the Opportunities and Risks of Foundation Models — Rishi Bommasani and many coauthors, 2021https://scholar.google.com/scholar?q=On+the+Opportunities+and+Risks+of+Foundation+Models6. A Simple Framework for Contrastive Learning of Visual Representations — Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton, 2020https://scholar.google.com/scholar?q=A+Simple+Framework+for+Contrastive+Learning+of+Visual+Representations7. Learning Transferable Visual Models From Natural Language Supervision — Alec Radford and coauthors, 2021https://scholar.google.com/scholar?q=Learning+Transferable+Visual+Models+From+Natural+Language+Supervision8. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — Colin Raffel, Noam Shazeer, Adam Roberts and coauthors, 2019https://scholar.google.com/scholar?q=Exploring+the+Limits+of+Transfer+Learning+with+a+Unified+Text-to-Text+Transformer9. World Models — David Ha and Jürgen Schmidhuber, 2018https://scholar.google.com/scholar?q=World+Models10. DreamerV3: Mastering Diverse Domains through World Models — Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap, 2023https://scholar.google.com/scholar?q=DreamerV3%3A+Mastering+Diverse+Domains+through+World+Models11. Representation Engineering: A Top-Down Approach to AI Transparency — Andy Zou, Catherine Wong, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson, 2023https://scholar.google.com/scholar?q=Representation+Engineering%3A+A+Top-Down+Approach+to+AI+Transparency12. Chain-of-Thought Reasoning Without Prompting — Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa, 2022https://scholar.google.com/scholar?q=Chain-of-Thought+Reasoning+Without+Prompting13. The Geometry of Meaning in Word Embeddings — Felix Hill, Kyunghyun Cho, and Anna Korhonen, 2016https://scholar.google.com/scholar?q=The+Geometry+of+Meaning+in+Word+Embeddings14. Token Turing Machines — Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis, 2022https://scholar.google.com/scholar?q=Token+Turing+Machines15. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, and Vahab Mirrokni, 2025https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time16. Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study — approx. 2024 authors unclear from snippet, 2024https://scholar.google.com/scholar?q=Dissecting+Logical+Reasoning+in+LLMs%3A+A+Fine-Grained+Evaluation+and+Supervision+Study17. ThinkRouter: Efficient Reasoning via Routing Thinking between Latent and Discrete Spaces — approx. 2025 authors unclear from snippet, 2025https://scholar.google.com/scholar?q=ThinkRouter%3A+Efficient+Reasoning+via+Routing+Thinking+between+Latent+and+Discrete+Spaces18. A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models — approx. 2024/2025 survey authors unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=A+Survey+on+Parallel+Text+Generation%3A+From+Parallel+Decoding+to+Diffusion+Language+Models19. Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding — approx. 2024 authors unclear from snippet, 2024https://scholar.google.com/scholar?q=Skeleton-of-Thought%3A+Large+Language+Models+Can+Do+Parallel+Decoding20. PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition — approx. 2024/2025 authors unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=PaDeLLM-NER%3A+Parallel+Decoding+in+Large+Language+Models+for+Named+Entity+Recognition21. Translating Natural Language to Planning Goals with Large-Language Models — approx. 2024/2025 authors unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Translating+Natural+Language+to+Planning+Goals+with+Large-Language+Models22. AI Post Transformers: Episode: Language Models are Injective and Hence Invertible — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-21-language-models-are-injective-an-7545e0.mp323. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp324. AI Post Transformers: Reinforced Attention Learning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/reinforced-attention-learning/25. AI Post Transformers: Recursive Language Models for Arbitrarily Long Prompts — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-recursive-language-models-for-arbitraril-fbcd1c.mp326. AI Post Transformers: Process Reward Learning for LLM Reasoning Optimization — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/process-reward-learning-for-llm-reasoning-optimization/27. AI Post Transformers: Agentic AI and the Next Intelligence Explosion — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-28-agentic-ai-and-the-next-intelligence-exp-d06561.mp328. AI Post Transformers: Survey of Emerging Topics in AI and Robotics — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/survey-of-emerging-topics-in-ai-and-robotics/Interactive Visualization: Latent Space as a New Computational Paradigm0 min
- Neural Computers as Learned Latent RuntimesThis episode explores a provocative 2026 paper that proposes the “neural computer” as a new machine abstraction: a model whose hidden state serves as the runtime itself, unifying computation, memory, and interface I/O rather than merely predicting tokens or controlling external tools. The discussion contrasts this idea with earlier systems like Neural Turing Machines and with world models, arguing that the key novelty is treating latent state as the machine’s active execution substrate rather than as a helper for prediction. It also examines the paper’s early evidence—interface-conditioned video models for command lines and GUIs—while stressing the gap between the paper’s sweeping manifesto and what the prototype actually proves. Listeners interested in AI architectures will find it compelling for its mix of big conceptual ambition, historical context, and a sharp critique of why learned latent systems struggle with the exactness and reliability that real computation demands.Sources:1. Neural Computers — Mingchen Zhuge, Changsheng Zhao, Haozhe Liu, Zijian Zhou, Shuming Liu, Wenyi Wang, Ernie Chang, Gael Le Lan, Junjie Fei, Wenxuan Zhang, Yasheng Sun, Zhipeng Cai, Zechun Liu, Yunyang Xiong, Yining Yang, Yuandong Tian, Yangyang Shi, Vikas Chandra, Jürgen Schmidhuber, 2026http://arxiv.org/abs/2604.064252. Neural Computers — Mingchen Zhuge, Changsheng Zhao, Haozhe Liu, Zijian Zhou, Shuming Liu, Wenyi Wang, Ernie Chang, Gael Le Lan, Junjie Fei, Wenxuan Zhang, Yasheng Sun, Zhipeng Cai, Zechun Liu, Yunyang Xiong, Yining Yang, Yuandong Tian, Yangyang Shi, Vikas Chandra, Jürgen Schmidhuber, 2026https://scholar.google.com/scholar?q=Neural+Computers3. Neural Turing Machines — Alex Graves, Greg Wayne, Ivo Danihelka, 2014https://scholar.google.com/scholar?q=Neural+Turing+Machines4. Hybrid Computing Using a Neural Network with Dynamic External Memory — Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, Demis Hassabis, 2016https://scholar.google.com/scholar?q=Hybrid+Computing+Using+a+Neural+Network+with+Dynamic+External+Memory5. World Models — David Ha, Jürgen Schmidhuber, 2018https://scholar.google.com/scholar?q=World+Models6. Genie: Generative Interactive Environments — not specified in provided excerpt, 2024https://scholar.google.com/scholar?q=Genie%3A+Generative+Interactive+Environments7. Veo 3.1 — Google, 2025https://scholar.google.com/scholar?q=Veo+3.18. Sora 2 — OpenAI, 2025https://scholar.google.com/scholar?q=Sora+29. Understanding World or Predicting Future? A Comprehensive Survey of World Models — unknown / survey authors unclear from snippet, recent, likely 2024-2025https://scholar.google.com/scholar?q=Understanding+World+or+Predicting+Future%3F+A+Comprehensive+Survey+of+World+Models10. Learning to Model the World: A Survey of World Models in Artificial Intelligence — unknown / survey authors unclear from snippet, recent, likely 2024-2025https://scholar.google.com/scholar?q=Learning+to+Model+the+World%3A+A+Survey+of+World+Models+in+Artificial+Intelligence11. Critiques of World Models — unknown / exact authors unclear from snippet, recent, likely 2024-2025https://scholar.google.com/scholar?q=Critiques+of+World+Models12. iVideoGPT: Interactive VideoGPTs Are Scalable World Models — approx. Yan et al. / exact authors unclear, recent, likely 2025https://scholar.google.com/scholar?q=iVideoGPT%3A+Interactive+VideoGPTs+Are+Scalable+World+Models13. CompilerDream: Learning a Compiler World Model for General Code Optimization — approximate; exact authors unclear from snippet, recent, likely 2024-2025https://scholar.google.com/scholar?q=CompilerDream%3A+Learning+a+Compiler+World+Model+for+General+Code+Optimization14. Emergent Representations of Program Semantics in Language Models Trained on Programs — approximate; exact authors unclear from snippet, recent, likely 2024-2025https://scholar.google.com/scholar?q=Emergent+Representations+of+Program+Semantics+in+Language+Models+Trained+on+Programs15. Confidence-Based Interactable Neural-Symbolic Visual Question Answering — approximate; exact authors unclear from snippet, recenthttps://scholar.google.com/scholar?q=Confidence-Based+Interactable+Neural-Symbolic+Visual+Question+Answering16. AI Reasoning in Deep Learning Era: From Symbolic AI to Neural-Symbolic AI — approximate; exact authors unclear from snippet, recenthttps://scholar.google.com/scholar?q=AI+Reasoning+in+Deep+Learning+Era%3A+From+Symbolic+AI+to+Neural-Symbolic+AI17. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp318. AI Post Transformers: AI Agent Traps and Prompt Injection — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-02-ai-agent-traps-and-prompt-injection-7ce4ba.mp319. AI Post Transformers: Memory Sparse Attention for 100M-Token Scaling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-07-memory-sparse-attention-for-100m-token-s-377cff.mp320. AI Post Transformers: Real Context Size and Context Rot — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-07-real-context-size-and-context-rot-56cbb4.mp3Interactive Visualization: Neural Computers as Learned Latent Runtimes0 min
- In-Place Test-Time Training for TransformersThis episode explores a paper on “in-place” test-time training for autoregressive transformer LLMs, asking whether a standard model can update some of its own weights during inference without requiring a new architecture. It explains how test-time training differs from in-context learning by storing temporary information in fast-changing parameters rather than only in tokens or KV cache, and argues that the paper’s main contribution is to reuse an existing transformer MLP projection and train it with a next-token-prediction-aligned objective instead of a generic self-supervised loss. The discussion also situates the work within earlier test-time training and long-sequence modeling research, highlighting why prior approaches struggled to fit mainstream LLM serving stacks. Listeners would find it interesting for its clear look at a possible path toward models that keep adapting after deployment, along with a skeptical examination of whether that promise is truly practical as a “drop-in” enhancement.Sources:1. In-Place Test-Time Training — Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Di He, Wenhao Huang, Tianle Cai, 2026http://arxiv.org/abs/2604.061692. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts — Yu Sun, Xiaolong Wang, Ziwei Liu, John Miller, Alexei A. Efros, Moritz Hardt, 2020https://scholar.google.com/scholar?q=Test-Time+Training+with+Self-Supervision+for+Generalization+under+Distribution+Shifts3. TTT Layers: Online Throughput-Optimized Training for Long Sequence Modeling — Michael A. Ahn, Zhiqing Sun, et al., 2024https://scholar.google.com/scholar?q=TTT+Layers%3A+Online+Throughput-Optimized+Training+for+Long+Sequence+Modeling4. Learning to (Learn at Test Time): RNNs with Expressive Hidden States — various authors in the fast-weights / online adaptation literature; often contextualized through the TTT and expressive-state sequence modeling thread, 2024https://scholar.google.com/scholar?q=Learning+to+%28Learn+at+Test+Time%29%3A+RNNs+with+Expressive+Hidden+States5. In-Place Test-Time Training — Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Di He, Wenhao Huang, Tianle Cai, 2026https://scholar.google.com/scholar?q=In-Place+Test-Time+Training6. Overcoming Catastrophic Forgetting in Neural Networks — James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, et al., 2017https://scholar.google.com/scholar?q=Overcoming+Catastrophic+Forgetting+in+Neural+Networks7. Continual Learning with Deep Generative Replay — Hanul Shin, Jung Kwon Lee, Jaehong Kim, Jiwon Kim, 2017https://scholar.google.com/scholar?q=Continual+Learning+with+Deep+Generative+Replay8. Dark Experience for General Continual Learning: a Strong, Simple Baseline — Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, Mohamed Elhoseiny, 2020https://scholar.google.com/scholar?q=Dark+Experience+for+General+Continual+Learning%3A+a+Strong%2C+Simple+Baseline9. Continual Learning in Neural Networks: An Overview — German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, Stefan Wermter, 2019https://scholar.google.com/scholar?q=Continual+Learning+in+Neural+Networks%3A+An+Overview10. The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2024https://scholar.google.com/scholar?q=The+Hedgehog+%26+the+Porcupine%3A+Expressive+Linear+Attentions+with+Softmax+Mimicry11. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces12. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — Tri Dao, Albert Gu, 2024https://scholar.google.com/scholar?q=Transformers+are+SSMs%3A+Generalized+Models+and+Efficient+Algorithms+Through+Structured+State+Space+Duality13. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2025https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time14. LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu, Yelong Shen, Phillip Wallis, et al., 2021https://scholar.google.com/scholar?q=LoRA%3A+Low-Rank+Adaptation+of+Large+Language+Models15. Memorizing Transformers — Jack W. Rae, Sebastian Borgeaud, Trevor Cai, et al., 2021https://scholar.google.com/scholar?q=Memorizing+Transformers16. Compute or Load KV Cache? Why Not Both? — approx. recent systems paper, authors unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Compute+or+Load+KV+Cache%3F+Why+Not+Both%3F17. AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving — approx. recent systems paper, authors unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=AdaptCache%3A+KV+Cache+Native+Storage+Hierarchy+for+Low-Delay+and+High-Quality+Language+Model+Serving18. DynamicKV: Task-aware Adaptive KV Cache Compression for Long Context LLMs — approx. recent systems paper, authors unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=DynamicKV%3A+Task-aware+Adaptive+KV+Cache+Compression+for+Long+Context+LLMs19. Streaming Lifelong Learning with Any-Time Inference — approx. recent continual-learning paper, authors unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Streaming+Lifelong+Learning+with+Any-Time+Inference20. Enabling Real-Time Inference in Online Continual Learning via Device-Cloud Collaboration — approx. recent continual-learning/systems paper, authors unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Enabling+Real-Time+Inference+in+Online+Continual+Learning+via+Device-Cloud+Collaboration21. Test-Time Training on Nearest Neighbors for Large Language Models — approx. authors unclear from snippet, 2023/2024https://scholar.google.com/scholar?q=Test-Time+Training+on+Nearest+Neighbors+for+Large+Language+Models22. Test-Time Learning for Large Language Models — approx. survey or position paper, authors unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Test-Time+Learning+for+Large+Language+Models23. AI Post Transformers: NVIDIA: TTT-E2E: Unlocking Long-Context Learning via End-to-End Test-Time Training — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/nvidia-ttt-e2e-unlocking-long-context-learning-via-end-to-end-test-time-training/24. AI Post Transformers: Jet-Nemotron and PostNAS for Faster Long Context — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-24-jet-nemotron-and-postnas-for-faster-long-436381.mp325. AI Post Transformers: Memory Sparse Attention for 100M-Token Scaling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-07-memory-sparse-attention-for-100m-token-s-377cff.mp326. AI Post Transformers: Native Sparse Attention: Efficient Long-Context LLMs — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/native-sparse-attention-efficient-long-context-llms/27. AI Post Transformers: Kimi Linear: Efficient Expressive Attention Architecture — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/kimi-linear-efficient-expressive-attention-architecture/28. AI Post Transformers: NeurIPS 2025: Self-Adapting Language Models — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/neurips-2025-self-adapting-language-models/29. AI Post Transformers: MetaClaw: Just Talk and Continual Agent Adaptation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-31-metaclaw-meta-learning-agents-in-the-wil-ab324c.mp3Interactive Visualization: In-Place Test-Time Training for Transformers37 min
- Computation-Bandwidth-Memory Trade-offs for AI InfrastructureThis episode explores a systems paper that argues AI infrastructure should treat computation, interconnect bandwidth, and memory as a single joint design space rather than three separate bottlenecks. It explains the paper’s “AI Trinity” framework and walks through the main trade-offs: using extra compute to reduce communication, using networked or disaggregated memory to ease local memory limits, and using caching or stored intermediates to avoid recomputation. The discussion connects that framing to real AI practice, from distributed training bottlenecked by all-reduce bandwidth to inference constrained by KV-cache memory, while grounding it in broader ideas like scaling laws, the “Bitter Lesson,” FlashAttention’s IO-aware design, and the roofline model. A listener would find it interesting because it translates familiar pains—GPU memory ceilings, gradient traffic, and hardware inefficiency—into a clearer systems-level way of thinking about how modern AI actually scales.Sources:1. Computation-Bandwidth-Memory Trade-offs: A Unified Paradigm for AI Infrastructure — Yuankai Fan, Qizhen Weng, Xuelong Li, 2025http://arxiv.org/abs/2601.115772. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He, 2020https://scholar.google.com/scholar?q=ZeRO%3A+Memory+Optimizations+Toward+Training+Trillion+Parameter+Models3. Training Deep Nets with Sublinear Memory Cost — Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin, 2016https://scholar.google.com/scholar?q=Training+Deep+Nets+with+Sublinear+Memory+Cost4. vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design — Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, Stephen W. Keckler, 2016https://scholar.google.com/scholar?q=vDNN%3A+Virtualized+Deep+Neural+Networks+for+Scalable%2C+Memory-Efficient+Neural+Network+Design5. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro, 2019https://scholar.google.com/scholar?q=Megatron-LM%3A+Training+Multi-Billion+Parameter+Language+Models+Using+Model+Parallelism6. Reducing Activation Recomputation in Large Transformer Models — Benjamin L. Kirby, Jackson Kernion, et al., 2024https://scholar.google.com/scholar?q=Reducing+Activation+Recomputation+in+Large+Transformer+Models7. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness8. Split Computing for Mobile Deep Inference: Survey and Research Directions — Yiping Kang, Johan Hauswald, et al. / related survey literature, 2020https://scholar.google.com/scholar?q=Split+Computing+for+Mobile+Deep+Inference%3A+Survey+and+Research+Directions9. Learning-Based Video Compression — Oren Rippel, Lubomir Bourdev, 2017https://scholar.google.com/scholar?q=Learning-Based+Video+Compression10. Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training — Yujun Lin, Song Han, Huizi Mao, Yu Wang, William J. Dally, 2018https://scholar.google.com/scholar?q=Deep+Gradient+Compression%3A+Reducing+the+Communication+Bandwidth+for+Distributed+Training11. Accelerating Diffusion Models with Cache-Based or Feature Reuse Methods (e.g., DeepCache / related 2024 diffusion caching work) — Various, 2024https://scholar.google.com/scholar?q=Accelerating+Diffusion+Models+with+Cache-Based+or+Feature+Reuse+Methods+%28e.g.%2C+DeepCache+%2F+related+2024+diffusion+caching+work%2912. RazorAttention: Efficient KV Cache Compression through Retrieval Heads — approx. recent LLM systems/serving authors, 2024/2025https://scholar.google.com/scholar?q=RazorAttention%3A+Efficient+KV+Cache+Compression+through+Retrieval+Heads13. StreamKV: Streaming Video Question-Answering with Segment-Based KV Cache Retrieval and Compression — approx. recent multimodal/LLM authors, 2024/2025https://scholar.google.com/scholar?q=StreamKV%3A+Streaming+Video+Question-Answering+with+Segment-Based+KV+Cache+Retrieval+and+Compression14. Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning — approx. recent LLM inference authors, 2024/2025https://scholar.google.com/scholar?q=Not+All+Heads+Matter%3A+A+Head-Level+KV+Cache+Compression+Method+with+Integrated+Retrieval+and+Reasoning15. In-Network Aggregation with Transport Transparency for Distributed Training — approx. systems/networking authors, 2023/2024https://scholar.google.com/scholar?q=In-Network+Aggregation+with+Transport+Transparency+for+Distributed+Training16. GRID: Gradient Routing with In-Network Aggregation for Distributed Training — approx. systems/networking authors, 2024/2025https://scholar.google.com/scholar?q=GRID%3A+Gradient+Routing+with+In-Network+Aggregation+for+Distributed+Training17. InArt: In-Network Aggregation with Route Selection for Accelerating Distributed Training — approx. systems/networking authors, 2024/2025https://scholar.google.com/scholar?q=InArt%3A+In-Network+Aggregation+with+Route+Selection+for+Accelerating+Distributed+Training18. PrivyNAS: Privacy-Aware Neural Architecture Search for Split Computing in Edge-Cloud Systems — approx. edge AI / NAS authors, 2024/2025https://scholar.google.com/scholar?q=PrivyNAS%3A+Privacy-Aware+Neural+Architecture+Search+for+Split+Computing+in+Edge-Cloud+Systems19. Advancements and Challenges in Privacy-Preserving Split Learning: Experimental Findings and Future Directions — approx. survey/review authors, 2024/2025https://scholar.google.com/scholar?q=Advancements+and+Challenges+in+Privacy-Preserving+Split+Learning%3A+Experimental+Findings+and+Future+Directions20. Lightweight User-Personalization Method for Closed Split Computing — approx. split-computing authors, 2024/2025https://scholar.google.com/scholar?q=Lightweight+User-Personalization+Method+for+Closed+Split+Computing21. AI Post Transformers: FlatAttention for Tile-Based Accelerator Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-flatattention-for-tile-based-accelerator-56e6ca.mp322. AI Post Transformers: CXL Computational Memory Offloading for Lower Runtime — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-cxl-computational-memory-offloading-for-3b2124.mp323. AI Post Transformers: FAST26: Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/fast26-bidaw-enhancing-key-value-caching-for-interactive-llm-serving-via-bidirec/24. AI Post Transformers: Episode: LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-lookaheadkv-fast-and-accurate-kv-9cfc9f.mp325. AI Post Transformers: Memory Sparse Attention for 100M-Token Scaling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-07-memory-sparse-attention-for-100m-token-s-377cff.mp326. AI Post Transformers: Accelerating LLM Cold Starts with Programmable Page Cache — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-17-accelerating-llm-cold-starts-with-progra-0912d1.mp327. AI Post Transformers: Paris: Decentralized Open-Weight Diffusion Model — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/paris-decentralized-open-weight-diffusion-model/Interactive Visualization: Computation-Bandwidth-Memory Trade-offs for AI Infrastructure0 min
- DRAM-Free In-Flash Computing for LLM InferenceThis episode explores a 2025 arXiv paper proposing “KVNAND,” an on-device LLM inference system that stores both model weights and the attention KV cache in compute-enabled 3D NAND flash to reduce or eliminate reliance on external DRAM. The discussion explains why decode-time generation is often bottlenecked by memory movement rather than raw compute, and argues that the KV cache—not just model weights—has become a major systems problem for long-context inference. It also examines whether the paper’s “DRAM-free” claim is technically convincing, especially given how KV cache costs vary across attention designs like MHA, GQA, and MQA. A listener would find it interesting for its concrete look at hardware-software tradeoffs in local LLM deployment and its skepticism about whether flashy architectural claims hold up under realistic workloads.Sources:1. KVNAND: Efficient On-Device Large Language Model Inference Using DRAM-Free In-Flash Computing — Lishuo Deng, Shaojie Xu, Jinwu Chen, Changwei Yan, Jiajie Wang, Zhe Jiang, Weiwei Shan, 2025http://arxiv.org/abs/2512.036082. A Survey of Processing-in-Memory: Techniques, Applications, and Challenges — Seyed H. N. Fatemi Langroudi and others, 2024https://scholar.google.com/scholar?q=A+Survey+of+Processing-in-Memory%3A+Techniques%2C+Applications%2C+and+Challenges3. Computational Storage: Where Are We Today? — Keith Townsend, Nils Bjerregaard, Javier Gonzalez and others, 2022https://scholar.google.com/scholar?q=Computational+Storage%3A+Where+Are+We+Today%3F4. Cambricon-LLM: Memory-Efficient Large Language Model Inference with Compute-Enabled Flash Memory — Main authors from the Cambricon research team, 2024https://scholar.google.com/scholar?q=Cambricon-LLM%3A+Memory-Efficient+Large+Language+Model+Inference+with+Compute-Enabled+Flash+Memory5. Lincoln: Accelerating Long-Context LLM Inference with In-Flash Computing — Main authors from the Lincoln research team, 2024https://scholar.google.com/scholar?q=Lincoln%3A+Accelerating+Long-Context+LLM+Inference+with+In-Flash+Computing6. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU — Ying Sheng, Yuzhang Wang, Beidi Chen and others, 2024https://scholar.google.com/scholar?q=PowerInfer%3A+Fast+Large+Language+Model+Serving+with+a+Consumer-grade+GPU7. LLM in a Flash: Efficient Large Language Model Inference with Limited Memory — Keiichi Yao, Siddharth Joshi, Priya Goyal and others, 2024https://scholar.google.com/scholar?q=LLM+in+a+Flash%3A+Efficient+Large+Language+Model+Inference+with+Limited+Memory8. Speculative Decoding for Accelerating Large Language Model Inference — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023https://scholar.google.com/scholar?q=Speculative+Decoding+for+Accelerating+Large+Language+Model+Inference9. PagedAttention: Efficient Memory Management for Large Language Model Serving with Paged KV Cache — Woosuk Kwon, Zhihong Shen, Siyuan Zhuang and others, 2023https://scholar.google.com/scholar?q=PagedAttention%3A+Efficient+Memory+Management+for+Large+Language+Model+Serving+with+Paged+KV+Cache10. Cambricon-LLM — Not specified in the provided excerpt, Likely 2024-2025https://scholar.google.com/scholar?q=Cambricon-LLM11. Lincoln — Not specified in the provided excerpt, Likely 2024-2025https://scholar.google.com/scholar?q=Lincoln12. LLaMA 2 — Hugo Touvron et al., 2023https://scholar.google.com/scholar?q=LLaMA+213. Llama 3.1 — Meta AI, 2024https://scholar.google.com/scholar?q=Llama+3.114. PagedAttention / vLLM — Woosuk Kwon et al., 2023https://scholar.google.com/scholar?q=PagedAttention+%2F+vLLM15. FlashAttention — Tri Dao et al., 2022https://scholar.google.com/scholar?q=FlashAttention16. MQA/GQA transformer variants such as GQA in Llama-family models — Various, 2023-2024https://scholar.google.com/scholar?q=MQA%2FGQA+transformer+variants+such+as+GQA+in+Llama-family+models17. Computational storage / near-data processing in SSDs — Various, 2019-2024https://scholar.google.com/scholar?q=Computational+storage+%2F+near-data+processing+in+SSDs18. RazorAttention: Efficient KV Cache Compression through Retrieval Heads — approx. recent LLM systems/ML authors, 2024/2025https://scholar.google.com/scholar?q=RazorAttention%3A+Efficient+KV+Cache+Compression+through+Retrieval+Heads19. Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning — approx. recent LLM efficiency authors, 2024/2025https://scholar.google.com/scholar?q=Not+All+Heads+Matter%3A+A+Head-Level+KV+Cache+Compression+Method+with+Integrated+Retrieval+and+Reasoning20. AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models — approx. recent LLM inference authors, 2024/2025https://scholar.google.com/scholar?q=AhaKV%3A+Adaptive+Holistic+Attention-Driven+KV+Cache+Eviction+for+Efficient+Inference+of+Large+Language+Models21. G-KV: Decoding-Time KV Cache Eviction with Global Attention — approx. recent LLM efficiency authors, 2024/2025https://scholar.google.com/scholar?q=G-KV%3A+Decoding-Time+KV+Cache+Eviction+with+Global+Attention22. LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference — approx. recent LLM inference authors, 2024/2025https://scholar.google.com/scholar?q=LLMs+Know+What+to+Drop%3A+Self-Attention+Guided+KV+Cache+Eviction+for+Efficient+Long-Context+Inference23. Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-Level Caching — approx. recent systems authors, 2024/2025https://scholar.google.com/scholar?q=Harnessing+Your+DRAM+and+SSD+for+Sustainable+and+Accessible+LLM+Inference+with+Mixed-Precision+and+Multi-Level+Caching24. Efficient LLM Inference Using Dynamic Input Pruning and Cache-Aware Masking — approx. recent mobile/edge inference authors, 2024/2025https://scholar.google.com/scholar?q=Efficient+LLM+Inference+Using+Dynamic+Input+Pruning+and+Cache-Aware+Masking25. SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving — approx. recent edge serving authors, 2024/2025https://scholar.google.com/scholar?q=SLED%3A+A+Speculative+LLM+Decoding+Framework+for+Efficient+Edge+Serving26. DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decoding — approx. recent edge/cloud systems authors, 2024/2025https://scholar.google.com/scholar?q=DSSD%3A+Efficient+Edge-Device+LLM+Deployment+and+Collaborative+Inference+via+Distributed+Split+Speculative+Decoding27. AI Post Transformers: SolidAttention: Co-Designing Sparse Attention and SSD I/O — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-18-solidattention-co-designing-sparse-atten-5a8622.mp328. AI Post Transformers: LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-lookaheadkv-fast-and-accurate-kv-9cfc9f.mp329. AI Post Transformers: Lookahead Q-Cache for Consistent KV Eviction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-lookahead-q-cache-for-consistent-kv-evic-d97b09.mp330. AI Post Transformers: QuantSpec: Hierarchical KV Cache for Self-Speculative Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/quantspec-hierarchical-kv-cache-for-self-speculative-decoding/31. AI Post Transformers: CXL-SpecKV: Bridging the LLM Memory Wall with Speculative FPGA Disaggregation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/cxl-speckv-bridging-the-llm-memory-wall-with-speculative-fpga-disaggregation/32. AI Post Transformers: FAST26: Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/fast26-bidaw-enhancing-key-value-caching-for-interactive-llm-serving-via-bidirec/33. AI Post Transformers: FlexGen: High-Throughput LLM Inference on a Single GPU — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/flexgen-high-throughput-llm-inference-on-a-single-gpu/34. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp3Interactive Visualization: DRAM-Free In-Flash Computing for LLM Inference29 min
- Memory Sparse Attention for 100M-Token ScalingThis episode explores a paper proposing Memory Sparse Attention, an end-to-end trainable memory architecture designed to scale language models from ordinary long-context settings to 100 million tokens. The discussion explains why standard dense self-attention becomes infeasible at extreme lengths, distinguishes simple context-window extension from true “lifetime-scale” memory, and situates the approach among alternatives like parameter-based memory, recurrent compression, and external retrieval systems such as RAG. It argues that the paper’s core idea is selective, trainable access to a small set of relevant memory segments rather than treating all past tokens as one continuous stream, while also noting the authors’ ambitious systems claims around practical inference. A listener would find it interesting for its clear framing of what makes ultra-long-context modeling hard, and for its skeptical but concrete examination of whether this architecture meaningfully bridges the gap between long prompts and persistent memory.Sources:1. MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens — Yu Chen, Runkai Chen, Sheng Yi, Xinda Zhao, Xiaohong Li, Jianjin Zhang, Jun Sun, Chuanrui Hu, Yunyun Han, Lidong Bing, Yafeng Deng, Tianqiao Chen, 2026http://arxiv.org/abs/2603.235162. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al., 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention3. Ring Attention with Blockwise Transformers for Near-Infinite Context — Aidan N. Gomez, Sean Dao, and collaborators, 2023https://scholar.google.com/scholar?q=Ring+Attention+with+Blockwise+Transformers+for+Near-Infinite+Context4. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro, 2019https://scholar.google.com/scholar?q=Megatron-LM%3A+Training+Multi-Billion+Parameter+Language+Models+Using+Model+Parallelism5. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Tri Dao, 2023https://scholar.google.com/scholar?q=FlashAttention-2%3A+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning6. RoFormer: Enhanced Transformer with Rotary Position Embedding — Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu, 2021https://scholar.google.com/scholar?q=RoFormer%3A+Enhanced+Transformer+with+Rotary+Position+Embedding7. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — Ofir Press, Noah A. Smith, Mike Lewis, 2021https://scholar.google.com/scholar?q=Train+Short%2C+Test+Long%3A+Attention+with+Linear+Biases+Enables+Input+Length+Extrapolation8. Extending Context Window of Large Language Models via Positional Interpolation — Shouyuan Chen, Sherman Wong, Liangcheng Luo, et al., 2023https://scholar.google.com/scholar?q=Extending+Context+Window+of+Large+Language+Models+via+Positional+Interpolation9. YaRN: Efficient Context Window Extension of Large Language Models — Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enming Luo, 2023https://scholar.google.com/scholar?q=YaRN%3A+Efficient+Context+Window+Extension+of+Large+Language+Models10. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2025https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time11. Infini-attention: Infinite Context for Efficient Transformers — Hao Liu, Wilson Yan, Matei Zaharia, Pieter Abbeel, 2024https://scholar.google.com/scholar?q=Infini-attention%3A+Infinite+Context+for+Efficient+Transformers12. LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens — Yucheng Ding, Li Dong, et al., 2024https://scholar.google.com/scholar?q=LongRoPE%3A+Extending+LLM+Context+Window+Beyond+2+Million+Tokens13. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, Joseph E. Gonzalez, 2024https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems14. RAG: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Patrick Lewis, Ethan Perez, Aleksandara Piktus, et al., 2020https://scholar.google.com/scholar?q=RAG%3A+Retrieval-Augmented+Generation+for+Knowledge-Intensive+NLP+Tasks15. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache — Zhenyu Liu, et al., 2024https://scholar.google.com/scholar?q=KIVI%3A+A+Tuning-Free+Asymmetric+2bit+Quantization+for+KV+Cache16. PagedAttention / vLLM: Efficient Memory Management for Large Language Model Serving — Woosuk Kwon, Zhuohan Li, et al., 2023https://scholar.google.com/scholar?q=PagedAttention+%2F+vLLM%3A+Efficient+Memory+Management+for+Large+Language+Model+Serving17. Memorizing Transformers — Yannic Kilcher? no; actually Jack W. Rae, Sebastian Borgeaud, Trevor Cai, et al., 2022https://scholar.google.com/scholar?q=Memorizing+Transformers18. TransformerFAM / Focused Attention Memory variants for long-context retrieval — Various 2024-2025 authors, 2024-2025https://scholar.google.com/scholar?q=TransformerFAM+%2F+Focused+Attention+Memory+variants+for+long-context+retrieval19. OpenRAG: Optimizing RAG End-to-End via In-Context Retrieval Learning — approx. 2025 authors unclear from snippet, 2025https://scholar.google.com/scholar?q=OpenRAG%3A+Optimizing+RAG+End-to-End+via+In-Context+Retrieval+Learning20. Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation — approx. 2025 authors unclear from snippet, 2025https://scholar.google.com/scholar?q=Beyond+RAG+for+Agent+Memory%3A+Retrieval+by+Decoupling+and+Aggregation21. MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention — approx. 2024/2025 authors unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=MInference+1.0%3A+Accelerating+Pre-filling+for+Long-Context+LLMs+via+Dynamic+Sparse+Attention22. SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention — approx. 2025 authors unclear from snippet, 2025https://scholar.google.com/scholar?q=SampleAttention%3A+Near-Lossless+Acceleration+of+Long+Context+LLM+Inference+with+Adaptive+Structured+Sparse+Attention23. FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference — approx. 2025 authors unclear from snippet, 2025https://scholar.google.com/scholar?q=FlexPrefill%3A+A+Context-Aware+Sparse+Attention+Mechanism+for+Efficient+Long-Sequence+Inference24. Kvlink: Accelerating Large Language Models via Efficient KV Cache Reuse — approx. 2025 authors unclear from snippet, 2025https://scholar.google.com/scholar?q=Kvlink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse25. HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse — approx. 2025 authors unclear from snippet, 2025https://scholar.google.com/scholar?q=HyperRAG%3A+Enhancing+Quality-Efficiency+Tradeoffs+in+Retrieval-Augmented+Generation+with+Reranker+KV-Cache+Reuse26. ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation — approx. 2025 authors unclear from snippet, 2025https://scholar.google.com/scholar?q=ProphetKV%3A+User-Query-Driven+Selective+Recomputation+for+Efficient+KV+Cache+Reuse+in+Retrieval-Augmented+Generation27. Hierarchical Local-Global Transformer With Dynamic Positional Encoding for Document-Level Machine Translation — approx. 2024/2025 authors unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Hierarchical+Local-Global+Transformer+With+Dynamic+Positional+Encoding+for+Document-Level+Machine+Translation28. AI Post Transformers: MEMSEARCHER: Reinforcement Learning for LLM Memory Management — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-memsearcher-reinforcement-learning-for-l-e9ad84.mp329. AI Post Transformers: Kimi Linear: Efficient Expressive Attention Architecture — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/kimi-linear-efficient-expressive-attention-architecture/30. AI Post Transformers: Lookahead Q-Cache for Consistent KV Eviction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-lookahead-q-cache-for-consistent-kv-evic-d97b09.mp331. AI Post Transformers: CacheSlide: Position-Aware KV Cache Reuse for Agent LLMs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-16-cacheslide-position-aware-kv-cache-reuse-cd59c7.mp332. AI Post Transformers: Doc-to-LoRA: Internalizing Context as LoRA — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-29-doc-to-lora-internalizing-context-as-lor-8dd5ec.mp333. AI Post Transformers: From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-22-from-prefix-cache-to-fusion-rag-9c5d39.mp334. AI Post Transformers: TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-turboquant-online-vector-quantiz-1967b7.mp30 min
- TriAttention for Efficient Long-Context KV CompressionThis episode explores TriAttention, a new method for reducing KV-cache memory during long-context inference by modeling how attention behaves under Rotary Positional Embeddings rather than relying on recent attention patterns alone. It explains why common compression methods can fail for long reasoning tasks: under RoPE, queries at different positions are rotated into different coordinate systems, so a small window of recent post-RoPE queries is a poor predictor of which earlier tokens will matter later. The discussion highlights the paper’s dual contribution as both a systems result for making 32K-token-style reasoning more practical and a mechanistic argument that transformer attention has analyzable structure rather than being purely empirical. Listeners interested in efficient LLM serving, long-context reasoning, or the inner geometry of attention will find it compelling because it connects deployment bottlenecks with a concrete theoretical explanation.Sources:1. TriAttention: Efficient Long Reasoning with Trigonometric KV Compression — Weian Mao, Xi Lin, Wei Huang, Yuxin Xie, Tianfu Fu, Bohan Zhuang, Song Han, Yukang Chen, 2026http://arxiv.org/abs/2604.049212. RoFormer: Enhanced Transformer with Rotary Position Embedding — Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu, 2021https://scholar.google.com/scholar?q=RoFormer%3A+Enhanced+Transformer+with+Rotary+Position+Embedding3. A Mathematical Framework for Transformer Circuits — Nelson Elhage, Nicholas Joseph, Ajeya Cotra, Kaidi Cao, Jared Kaplan, et al., 2021https://scholar.google.com/scholar?q=A+Mathematical+Framework+for+Transformer+Circuits4. StreamingLLM: Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yao Fu, Kuanlun Guo, Xuefei Ning, et al., 2023https://scholar.google.com/scholar?q=StreamingLLM%3A+Efficient+Streaming+Language+Models+with+Attention+Sinks5. TriAttention: Efficient Long Reasoning with Trigonometric KV Compression — Weian Mao, Xi Lin, Wei Huang, Yuxin Xie, Tianfu Fu, Bohan Zhuang, Song Han, Yukang Chen, 2026https://scholar.google.com/scholar?q=TriAttention%3A+Efficient+Long+Reasoning+with+Trigonometric+KV+Compression6. What Makes Rotary Positional Encodings Useful? — Federico Barbero, et al., 2025https://scholar.google.com/scholar?q=What+Makes+Rotary+Positional+Encodings+Useful%3F7. Attention Sinks and Massive Activation Values in Transformers — Xiaozhi Xiao, et al., 2025https://scholar.google.com/scholar?q=Attention+Sinks+and+Massive+Activation+Values+in+Transformers8. Heavy Hitter Oracle for Efficient Generative Inference of Large Language Models — Zirui Liu, et al., 2023https://scholar.google.com/scholar?q=Heavy+Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models9. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zirui Liu, et al., 2023https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models10. PyramidKV — Zhang, et al., 2024https://scholar.google.com/scholar?q=PyramidKV11. SnapKV — Li, et al., 2024https://scholar.google.com/scholar?q=SnapKV12. R-KV — Zhang, et al., 2025https://scholar.google.com/scholar?q=R-KV13. Vision Transformer Interpretability via Attention Rollout — Samira Abnar, Willem Zuidema, 2020https://scholar.google.com/scholar?q=Vision+Transformer+Interpretability+via+Attention+Rollout14. An Analysis of Attention Weights as a Proxy for Explanation — Sarthak Jain, Byron C. Wallace, 2019https://scholar.google.com/scholar?q=An+Analysis+of+Attention+Weights+as+a+Proxy+for+Explanation15. RazorAttention: Efficient KV Cache Compression Through Retrieval Heads — approx. Tang et al., 2024/2025https://scholar.google.com/scholar?q=RazorAttention%3A+Efficient+KV+Cache+Compression+Through+Retrieval+Heads16. Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning — approx. 2025 head-aware KV compression paper, 2025https://scholar.google.com/scholar?q=Not+All+Heads+Matter%3A+A+Head-Level+KV+Cache+Compression+Method+with+Integrated+Retrieval+and+Reasoning17. FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference — approx. 2025, 2025https://scholar.google.com/scholar?q=FreeKV%3A+Boosting+KV+Cache+Retrieval+for+Efficient+LLM+Inference18. RAP: KV-Cache Compression via RoPE-Aligned Pruning — approx. 2025, 2025https://scholar.google.com/scholar?q=RAP%3A+KV-Cache+Compression+via+RoPE-Aligned+Pruning19. EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection — approx. 2025, 2025https://scholar.google.com/scholar?q=EliteKV%3A+Scalable+KV+Cache+Compression+via+RoPE+Frequency+Selection+and+Joint+Low-Rank+Projection20. Asymmetric KV Cache Compression using State-Aware Sparsity and Quantization — approx. 2025, 2025https://scholar.google.com/scholar?q=Asymmetric+KV+Cache+Compression+using+State-Aware+Sparsity+and+Quantization21. Efficient Streaming Language Models with Attention Sinks — Xiao et al., 2023https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks22. When Attention Sink Emerges in Language Models: An Empirical View — approx. 2024/2025, 2024/2025https://scholar.google.com/scholar?q=When+Attention+Sink+Emerges+in+Language+Models%3A+An+Empirical+View23. AI Post Transformers: Lookahead Q-Cache for Consistent KV Eviction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-lookahead-q-cache-for-consistent-kv-evic-d97b09.mp324. AI Post Transformers: LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-lookaheadkv-fast-and-accurate-kv-9cfc9f.mp325. AI Post Transformers: TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-turboquant-online-vector-quantiz-1967b7.mp326. AI Post Transformers: Kimi Linear: Efficient Expressive Attention Architecture — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/kimi-linear-efficient-expressive-attention-architecture/27. AI Post Transformers: Real Context Size and Context Rot — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-07-real-context-size-and-context-rot-56cbb4.mp3Interactive Visualization: TriAttention for Efficient Long-Context KV Compression29 min
- Cache Mechanism for Agent RAG SystemsThis episode explores a 2025 paper on cache management for agentic RAG systems, asking whether an annotation-free cache can preserve most of the value of a massive retrieval corpus while using far less storage and reducing latency. It explains how RAG, agent memory, vector databases, embeddings, and approximate nearest neighbor search fit together, arguing that retrieval performance is not just a modeling issue but a core systems constraint for real-world agents. The discussion situates the paper in the broader history of retrieval and agent research, from Word2Vec and BERT to Dense Passage Retrieval, ReAct, and FAISS, showing why externalized knowledge remains useful even as language models grow larger. Listeners would find it interesting because it focuses on a practical but consequential question: how to make retrieval-heavy AI agents cheaper, faster, and more deployable outside large cloud infrastructures.Sources:1. Cache Mechanism for Agent RAG Systems — Shuhang Lin, Zhencan Peng, Lingyao Li, Xiao Lin, Xi Zhu, Yongfeng Zhang, 2025http://arxiv.org/abs/2511.029192. PlanRAG — Lee et al., 2024https://scholar.google.com/scholar?q=PlanRAG3. Generate-then-Ground — Shi et al., 2024https://scholar.google.com/scholar?q=Generate-then-Ground4. RAP — Kagaya et al., 2024https://scholar.google.com/scholar?q=RAP5. RAT — Wang et al., 2024https://scholar.google.com/scholar?q=RAT6. Mei et al. (system engineering / large knowledge repositories) — Mei et al., 2025https://scholar.google.com/scholar?q=Mei+et+al.+%28system+engineering+%2F+large+knowledge+repositories%297. Guo et al. on RAG-powered agent architectures — Guo et al., 2025https://scholar.google.com/scholar?q=Guo+et+al.+on+RAG-powered+agent+architectures8. Long Context vs. RAG for LLMs: An Evaluation and Revisits — approx. recent LLM/RAG evaluation authors, 2024/2025https://scholar.google.com/scholar?q=Long+Context+vs.+RAG+for+LLMs%3A+An+Evaluation+and+Revisits9. Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? — approx. recent long-context LLM systems authors, 2024/2025https://scholar.google.com/scholar?q=Can+Long-Context+Language+Models+Subsume+Retrieval%2C+RAG%2C+SQL%2C+and+More%3F10. Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generation — approx. recent RAG evaluation/prediction authors, 2024/2025https://scholar.google.com/scholar?q=Predicting+Retrieval+Utility+and+Answer+Quality+in+Retrieval-Augmented+Generation11. Relevance Filtering for Embedding-Based Retrieval — approx. recent dense retrieval / IR authors, 2024/2025https://scholar.google.com/scholar?q=Relevance+Filtering+for+Embedding-Based+Retrieval12. Volatility-Driven Decay: Adaptive Memory Retention for RAG Systems Under Unknown Drift — approx. recent continual RAG / memory authors, 2025https://scholar.google.com/scholar?q=Volatility-Driven+Decay%3A+Adaptive+Memory+Retention+for+RAG+Systems+Under+Unknown+Drift13. On the Role of Long-Tail Knowledge in Retrieval Augmented Large Language Models — approx. recent RAG robustness authors, 2024/2025https://scholar.google.com/scholar?q=On+the+Role+of+Long-Tail+Knowledge+in+Retrieval+Augmented+Large+Language+Models14. Graph-Based Retriever Captures the Long Tail of Biomedical Knowledge — approx. recent biomedical retrieval authors, 2024/2025https://scholar.google.com/scholar?q=Graph-Based+Retriever+Captures+the+Long+Tail+of+Biomedical+Knowledge15. FIT-RAG: Black-Box RAG with Factual Information and Token Reduction — approx. recent black-box RAG authors, 2024/2025https://scholar.google.com/scholar?q=FIT-RAG%3A+Black-Box+RAG+with+Factual+Information+and+Token+Reduction16. AI Post Transformers: QVCache for Semantic Caching in ANN Search — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-qvcache-for-semantic-caching-in-ann-sear-415304.mp317. AI Post Transformers: Episode: From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-22-from-prefix-cache-to-fusion-rag-9c5d39.mp318. AI Post Transformers: MEMSEARCHER: Reinforcement Learning for LLM Memory Management — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-memsearcher-reinforcement-learning-for-l-e9ad84.mp319. AI Post Transformers: Doc-to-LoRA: Internalizing Context as LoRA — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-29-doc-to-lora-internalizing-context-as-lor-8dd5ec.mp320. AI Post Transformers: ColBERT and ColBERT v2 — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/colbert-and-colbert-v2/Interactive Visualization: Cache Mechanism for Agent RAG Systems0 min
- When Spectral Gradient Updates Help Deep LearningThis episode explores a theory paper that asks when spectral matrix updates should outperform standard Euclidean gradient methods in deep networks and transformers. It explains how spectral updates replace a gradient matrix with its polar factor—preserving singular-vector directions while flattening singular values—and argues that this geometry can help when incoming activations have low stable rank while gradients have high nuclear-rank-like spread. The discussion connects this criterion to practical excitement around spectral-style optimizers such as Muon, while contrasting them with curvature-based methods like K-FAC and Shampoo. Listeners would find it interesting because the episode turns a seemingly niche optimizer trick into a concrete, testable claim about the hidden geometry of neural network training.Sources:1. When do spectral gradient updates help in deep learning? — Damek Davis, Dmitriy Drusvyatskiy, 2025http://arxiv.org/abs/2512.042992. Shampoo: Preconditioned Stochastic Tensor Optimization — Vineet Gupta, Tomer Koren, Yoram Singer and others, 2018https://scholar.google.com/scholar?q=Shampoo%3A+Preconditioned+Stochastic+Tensor+Optimization3. K-FAC: Kronecker-Factored Approximate Curvature for Neural Network Optimization — James Martens, Roger Grosse, 2015https://scholar.google.com/scholar?q=K-FAC%3A+Kronecker-Factored+Approximate+Curvature+for+Neural+Network+Optimization4. Muon: An optimizer for hidden layers in neural networks — Keller Jordan and collaborators, 2024https://scholar.google.com/scholar?q=Muon%3A+An+optimizer+for+hidden+layers+in+neural+networks5. When do spectral gradient updates help in deep learning? — Damek Davis, Dmitriy Drusvyatskiy, 2026https://scholar.google.com/scholar?q=When+do+spectral+gradient+updates+help+in+deep+learning%3F6. Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation — Anonymous/various authors depending on version; commonly cited in transformer dynamics discussions, 2021https://scholar.google.com/scholar?q=Deep+Transformers+without+Shortcuts%3A+Modifying+Self-attention+for+Faithful+Signal+Propagation7. On the Softmax Bottleneck of Recurrent Language Models — Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, William W. Cohen, Yoshua Bengio, 2018https://scholar.google.com/scholar?q=On+the+Softmax+Bottleneck+of+Recurrent+Language+Models8. Representation Degeneration Problem in Training Natural Language Generation Models — Junxian He, Daniel Spokoyny, Graham Neubig, Taylor Berg-Kirkpatrick, 2020https://scholar.google.com/scholar?q=Representation+Degeneration+Problem+in+Training+Natural+Language+Generation+Models9. Neural Collapse: A Terminal Phase of Deep Learning Training — Vardan Papyan, X. Y. Han, David L. Donoho, 2020https://scholar.google.com/scholar?q=Neural+Collapse%3A+A+Terminal+Phase+of+Deep+Learning+Training10. Understanding Dimensional Collapse in Contrastive Self-supervised Learning — Tianyu Hua, Wenxiao Wang, Zihang Dai and others, 2021https://scholar.google.com/scholar?q=Understanding+Dimensional+Collapse+in+Contrastive+Self-supervised+Learning11. The Intrinsic Dimension of Objective Landscapes — Chunyuan Li, Heerad Farkhoor, Rosanne Liu, Jason Yosinski, 2018https://scholar.google.com/scholar?q=The+Intrinsic+Dimension+of+Objective+Landscapes12. Random Features for Large-Scale Kernel Machines — Ali Rahimi, Benjamin Recht, 2007https://scholar.google.com/scholar?q=Random+Features+for+Large-Scale+Kernel+Machines13. A Random Matrix Perspective on Random Features for Compositional Kernels — Florent Krzakala, Lenka Zdeborová, and collaborators in the random-features theory community, 2019https://scholar.google.com/scholar?q=A+Random+Matrix+Perspective+on+Random+Features+for+Compositional+Kernels14. The Surprising Effectiveness of Random Features for Structured Data — Various authors across theory and applied ML; representative random-feature comparison literature, 2010s-2020shttps://scholar.google.com/scholar?q=The+Surprising+Effectiveness+of+Random+Features+for+Structured+Data15. Spectral Gradient Descent — Yair Carmon, John C. Duchi, Oliver Hinder, Aaron Sidford, 2021https://scholar.google.com/scholar?q=Spectral+Gradient+Descent16. A Kronecker-factored approximate Fisher matrix for convolution layers — Roger Grosse, Jimmy Ba, et al., 2016https://scholar.google.com/scholar?q=A+Kronecker-factored+approximate+Fisher+matrix+for+convolution+layers17. Feature Learning in Infinite-Width Neural Networks — Mario Geiger, Stefano Spigler, Arthur Jacot, Matthieu Wyart, 2020https://scholar.google.com/scholar?q=Feature+Learning+in+Infinite-Width+Neural+Networks18. Neural Collapse: A Review and Synthesis — Vardan Papyan, X.Y. Han, David L. Donoho, 2023https://scholar.google.com/scholar?q=Neural+Collapse%3A+A+Review+and+Synthesis19. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning — Arora et al., 2021https://scholar.google.com/scholar?q=Intrinsic+Dimensionality+Explains+the+Effectiveness+of+Language+Model+Fine-Tuning20. Understanding transformers for time series: Rank structure, flow-of-ranks, and compressibility — approx. recent transformer interpretability / theory authors, recenthttps://scholar.google.com/scholar?q=Understanding+transformers+for+time+series%3A+Rank+structure%2C+flow-of-ranks%2C+and+compressibility21. Tuning stable rank shrinkage: Aiming at the overlooked structural risk in fine-tuning — approx. recent fine-tuning / representation learning authors, recenthttps://scholar.google.com/scholar?q=Tuning+stable+rank+shrinkage%3A+Aiming+at+the+overlooked+structural+risk+in+fine-tuning22. Unraveling the gradient descent dynamics of transformers — approx. recent optimization theory authors, recenthttps://scholar.google.com/scholar?q=Unraveling+the+gradient+descent+dynamics+of+transformers23. AI Post Transformers: Adam: A Method for Stochastic Optimization — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/adam-a-method-for-stochastic-optimization/24. AI Post Transformers: AdamW: Decoupled Weight Decay Regularization for Adaptive Gradient Algorithms — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/adamw-decoupled-weight-decay-regularization-for-adaptive-gradient-algorithms/25. AI Post Transformers: In-Context Learning as Implicit Learning Algorithms — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/in-context-learning-as-implicit-learning-algorithms/Interactive Visualization: When Spectral Gradient Updates Help Deep Learning0 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.