Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- Breaking the Prefix Barrier with Shared KV CacheThis episode explores a 2026 paper proposing that multiple LLM agents should share transformer KV-cache state, not just text, so they can avoid repeatedly paying the prefill cost of rereading the same plans, critiques, and intermediate outputs. It explains the systems background behind prefix caching, vLLM’s PagedAttention, and SGLang, then focuses on why multi-agent workflows break the exact-prefix assumption and make segment-level reuse much harder. The discussion highlights the paper’s core technical tension: the idea is compelling, but reusing cached activations across different prompt positions is fragile because of positional encoding effects such as RoPE misalignment and attention behavior. Listeners would find it interesting because it connects a practical bottleneck in agent systems to deep transformer internals, while also questioning whether the paper truly delivers fine-grained semantic sharing or a narrower form of reusable output caching.Sources:1. Breaking the Prefix Barrier with Shared KV Cachehttps://openreview.net/forum?id=kgzBkyqg6Z2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention3. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2024https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs4. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang, 2025https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion5. KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems — Hancheng Ye, Zhengqi Gao, Mingyuan Ma, Qinsi Wang, Yuzhe Fu, Ming-Yu Chung, Yueqian Lin, Zhijian Liu, Jianyi Zhang, Danyang Zhuo, Yiran Chen, 2025https://scholar.google.com/scholar?q=KVCOMM%3A+Online+Cross-context+KV-cache+Communication+for+Efficient+LLM-based+Multi-agent+Systems6. EPIC: Efficient Position-Independent Caching for Serving Large Language Models — Junhao Hu, Wenrui Huang, Weidong Wang, Haoyi Wang, Tiancheng Hu, Qin Zhang, Hao Feng, Xusheng Chen, Yizhou Shan, Tao Xie, 2025https://scholar.google.com/scholar?q=EPIC%3A+Efficient+Position-Independent+Caching+for+Serving+Large+Language+Models7. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows — Zaifeng Pan, Ajjkumar Patel, Zhengding Hu, Yipeng Shen, Yue Guan, Wan-Lu Li, Lianhui Qin, Yida Wang, Yufei Ding, 2025https://scholar.google.com/scholar?q=KVFlow%3A+Efficient+Prefix+Caching+for+Accelerating+LLM-Based+Multi-Agent+Workflows8. DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving — Yuhan Liu, Yuyang Huang, Jiayi Yao, Shaoting Feng, Zhuohan Gu, Kuntai Du, Hanchen Li, Yihua Cheng, Junchen Jiang, Shan Lu, Madan Musuvathi, Esha Choukse, 2024https://scholar.google.com/scholar?q=DroidSpeak%3A+KV+Cache+Sharing+for+Cross-LLM+Communication+and+Multi-LLM+Serving9. TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing — Zhuohang Bian, Feiyang Wu, Chengrui Zhang, Hangcheng Dong, Yun Liang, Youwei Zhuo, 2026https://scholar.google.com/scholar?q=TokenDance%3A+Scaling+Multi-Agent+LLM+Serving+via+Collective+KV+Cache+Sharing10. HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse — authors unclear from Scholar snippet, 2025https://scholar.google.com/scholar?q=HyperRAG%3A+Enhancing+Quality-Efficiency+Tradeoffs+in+Retrieval-Augmented+Generation+with+Reranker+KV-Cache+Reuse11. ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation — authors unclear from Scholar snippet, 2025https://scholar.google.com/scholar?q=ProphetKV%3A+User-Query-Driven+Selective+Recomputation+for+Efficient+KV+Cache+Reuse+in+Retrieval-Augmented+Generation12. Cache-craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation — authors unclear from Scholar snippet, 2025https://scholar.google.com/scholar?q=Cache-craft%3A+Managing+Chunk-Caches+for+Efficient+Retrieval-Augmented+Generation13. AttentionStore: Cost-Effective Attention Reuse Across Multi-Turn Conversations in Large Language Model Serving — authors unclear from Scholar snippet, 2025https://scholar.google.com/scholar?q=AttentionStore%3A+Cost-Effective+Attention+Reuse+Across+Multi-Turn+Conversations+in+Large+Language+Model+Serving14. BAT: Efficient Generative Recommender Serving with Bipartite Attention — authors unclear from Scholar snippet, 2025https://scholar.google.com/scholar?q=BAT%3A+Efficient+Generative+Recommender+Serving+with+Bipartite+Attention15. AI Post Transformers: Efficient KV Cache Sharing for Multi-LoRA Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-efficient-kv-cache-sharing-for-multi-lor-afda05.mp316. AI Post Transformers: TokenDance for Multi-Agent KV Cache Sharing — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-tokendance-for-multi-agent-kv-cache-shar-aa9b99.mp317. AI Post Transformers: KV Cache TTL for Multi-Turn Agent Scheduling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-09-kv-cache-ttl-for-multi-turn-agent-schedu-996bf1.mp318. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp319. AI Post Transformers: Test-time Scaling for Multi-Agent Collaborative Reasoning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-test-time-scaling-for-multi-agent-collab-082570.mp3Interactive Visualization: Breaking the Prefix Barrier with Shared KV Cache14 min
- Stochastic KV Routing for Cache SharingThis episode explores Apple’s Stochastic KV Routing approach for reducing transformer inference costs by letting some layers reuse key-value caches from earlier layers instead of storing separate caches at every depth. It explains why KV cache memory becomes a major bottleneck for long-context autoregressive decoding, and why depth-wise cache sharing is a different idea from token eviction or temporal compression. The discussion connects the paper to Grouped Query Attention and other prior cache-sharing methods, highlighting Apple’s main argument: train models with stochastic cross-layer routing so they can tolerate many cache-retention layouts and then expose a practical serving-time knob for trading memory use against model quality. A listener would find it interesting because it ties a concrete systems problem in LLM deployment to a training strategy that could make large models more flexible under real hardware constraints.Sources:1. Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing — Anastasiia Filippova, David Grangier, Marco Cuturi, João Monteiro, 2026http://arxiv.org/abs/2604.227822. Fast Transformer Decoding: One Write-Head is All You Need — Noam Shazeer, 2019https://scholar.google.com/scholar?q=Fast+Transformer+Decoding%3A+One+Write-Head+is+All+You+Need3. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, Sumit Sanghai, 2023https://scholar.google.com/scholar?q=GQA%3A+Training+Generalized+Multi-Query+Transformer+Models+from+Multi-Head+Checkpoints4. Layer-Condensed KV Cache for Efficient Inference of Large Language Models — Haoyi Wu, Kewei Tu, 2024https://scholar.google.com/scholar?q=Layer-Condensed+KV+Cache+for+Efficient+Inference+of+Large+Language+Models5. MiniCache: KV Cache Compression in Depth Dimension for Large Language Models — Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, Bohan Zhuang, 2024https://scholar.google.com/scholar?q=MiniCache%3A+KV+Cache+Compression+in+Depth+Dimension+for+Large+Language+Models6. Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing — Anastasiia Filippova, David Grangier, Marco Cuturi, Joao Monteiro, 2026https://scholar.google.com/scholar?q=Stochastic+KV+Routing%3A+Enabling+Adaptive+Depth-Wise+Cache+Sharing7. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention — William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, Jonathan Ragan-Kelley, 2024https://scholar.google.com/scholar?q=Reducing+Transformer+Key-Value+Cache+Size+with+Cross-Layer+Attention8. XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference — Joao Monteiro, Etienne Marcotte, Pierre-Andre Noel, Valentina Zantedeschi, David Vazquez, Nicolas Chapados, Christopher Pal, Perouz Taslakian, 2024https://scholar.google.com/scholar?q=XC-Cache%3A+Cross-Attending+to+Cached+Context+for+Efficient+LLM+Inference9. KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing — not identifiable from snippet, recenthttps://scholar.google.com/scholar?q=KVSharer%3A+Efficient+Inference+via+Layer-Wise+Dissimilar+KV+Cache+Sharing10. Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity — not identifiable from snippet, recenthttps://scholar.google.com/scholar?q=Compressing+KV+Cache+for+Long-Context+LLM+Inference+with+Inter-Layer+Attention+Similarity11. CaR: An Efficient KV Cache Reuse System for Large Language Model Inference — not identifiable from snippet, recenthttps://scholar.google.com/scholar?q=CaR%3A+An+Efficient+KV+Cache+Reuse+System+for+Large+Language+Model+Inference12. Compute or Load KV Cache? Why Not Both? — not identifiable from snippet, recenthttps://scholar.google.com/scholar?q=Compute+or+Load+KV+Cache%3F+Why+Not+Both%3F13. AI Post Transformers: Lookahead Q-Cache for Consistent KV Eviction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-lookahead-q-cache-for-consistent-kv-evic-d97b09.mp314. AI Post Transformers: Prefill-as-a-Service for Cross-Datacenter KV Cache — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-prefill-as-a-service-for-cross-datacente-7560be.mp315. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp316. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp317. AI Post Transformers: Mamba-3 for Efficient Sequence Modeling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-16-mamba-3-for-efficient-sequence-modeling-97a22a.mp318. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3Interactive Visualization: Stochastic KV Routing for Cache Sharing12 min
- Stabilizing Efficient Reasoning with Step-Level Advantage SelectionThis episode explores a 2026 paper on making reasoning models faster and cheaper by shortening their chain-of-thought without sacrificing too much accuracy. It explains the paper’s core claim that short-context RL post-training can itself push models toward concise reasoning, while also unpacking the instability this creates and how the proposed Step-Level Advantage Selection method is meant to stabilize training. The discussion places that idea in the broader arc from chain-of-thought prompting and self-consistency to today’s industry-facing reasoning-budget controls, framing efficient reasoning as a test-time compute management problem rather than a new model architecture. Listeners would find it interesting for its skeptical, engineering-focused look at whether shorter reasoning traces are a real advance or just a fragile optimization hidden behind benchmark gains.Sources:1. Stabilizing Efficient Reasoning with Step-Level Advantage Selection — Han Wang, Xiaodong Yu, Jialian Wu, Jiang Liu, Ximeng Sun, Mohit Bansal, Zicheng Liu, 2026http://arxiv.org/abs/2604.240032. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, 2022https://scholar.google.com/scholar?q=Chain-of-Thought+Prompting+Elicits+Reasoning+in+Large+Language+Models3. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou, 2022https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models4. Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs — Pranjal Aggarwal, Aman Madaan, Yiming Yang, Mausam, 2023https://scholar.google.com/scholar?q=Let%E2%80%99s+Sample+Step+by+Step%3A+Adaptive-Consistency+for+Efficient+Reasoning+and+Coding+with+LLMs5. Training Language Models to Reason Efficiently — Daman Arora, Andrea Zanette, 2025https://scholar.google.com/scholar?q=Training+Language+Models+to+Reason+Efficiently6. DeepScaleR: Effective RL Scaling of Reasoning Models via Iterative Context Lengthening — Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, Ion Stoica, 2025https://scholar.google.com/scholar?q=DeepScaleR%3A+Effective+RL+Scaling+of+Reasoning+Models+via+Iterative+Context+Lengthening7. L1: Controlling How Long a Reasoning Model Thinks with Reinforcement Learning — Pranjal Aggarwal, Sean Welleck, 2025https://scholar.google.com/scholar?q=L1%3A+Controlling+How+Long+a+Reasoning+Model+Thinks+with+Reinforcement+Learning8. ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning — Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, 2025https://scholar.google.com/scholar?q=ThinkPrune%3A+Pruning+Long+Chain-of-Thought+of+LLMs+via+Reinforcement+Learning9. LAPO: Internalizing Reasoning Efficiency via Length-Adaptive Policy Optimization — Xingyu Wu, Yuchen Yan, Shangke Lyu, Linjuan Wu, Yiwen Qiu, Yongliang Shen, Weiming Lu, Jian Shao, Jun Xiao, Yueting Zhuang, 2025https://scholar.google.com/scholar?q=LAPO%3A+Internalizing+Reasoning+Efficiency+via+Length-Adaptive+Policy+Optimization10. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning — Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-Hui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, Junyang Lin, 2025https://scholar.google.com/scholar?q=Beyond+the+80%2F20+Rule%3A+High-Entropy+Minority+Tokens+Drive+Effective+Reinforcement+Learning+for+LLM+Reasoning11. Do NOT Think That Much for 2+3=? On the Overthinking of Long Reasoning Models — Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, Dong Yu, 2025https://scholar.google.com/scholar?q=Do+NOT+Think+That+Much+for+2%2B3%3D%3F+On+the+Overthinking+of+Long+Reasoning+Models12. QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning — Fanqi Wan et al., 2025https://scholar.google.com/scholar?q=QwenLong-L1%3A+Towards+Long-Context+Large+Reasoning+Models+with+Reinforcement+Learning13. LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts — Siyuan Wang et al., 2025https://scholar.google.com/scholar?q=LoongRL%3A+Reinforcement+Learning+for+Advanced+Reasoning+over+Long+Contexts14. LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards — Bowen Ping et al., 2026https://scholar.google.com/scholar?q=LongR%3A+Unleashing+Long-Context+Reasoning+via+Reinforcement+Learning+with+Dense+Utility+Rewards15. SSVPO: Effective Step-Level Credit Assignment for RL Training of Language Models — Yugu Li, Zehong Cao, Jianglin Qiao, Siyi Hu, 2026https://scholar.google.com/scholar?q=SSVPO%3A+Effective+Step-Level+Credit+Assignment+for+RL+Training+of+Language+Models16. GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning — Yao Zhang et al., 2025https://scholar.google.com/scholar?q=GroundedPRM%3A+Tree-Guided+and+Fidelity-Aware+Process+Reward+Modeling+for+Step-Level+Reasoning17. Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning — Miles Turpin, Andy Arditi, Marvin Li, Joe Benton, Julian Michael, 2025https://scholar.google.com/scholar?q=Teaching+Models+to+Verbalize+Reward+Hacking+in+Chain-of-Thought+Reasoning18. Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effort — Xinpeng Wang, Nitish Joshi, Barbara Plank, Rico Angell, He He, 2025https://scholar.google.com/scholar?q=Is+It+Thinking+or+Cheating%3F+Detecting+Implicit+Reward+Hacking+by+Measuring+Reasoning+Effort19. AI Post Transformers: DeepSeek-V4 and Practical Million-Token Context — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-25-deepseek-v4-and-practical-million-token-6f4de1.mp320. AI Post Transformers: AgenticQwen and Small Industrial Tool Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-27-agenticqwen-and-small-industrial-tool-ag-dc676d.mp321. AI Post Transformers: World-R1 Improves 3D Consistency in Text-to-Video — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-28-world-r1-improves-3d-consistency-in-text-f065d9.mp322. AI Post Transformers: Experience-Based Learning Beyond Human Data — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-experience-based-learning-beyond-human-d-b0caa4.mp3Interactive Visualization: Stabilizing Efficient Reasoning with Step-Level Advantage Selection15 min
- World-R1 Improves 3D Consistency in Text-to-VideoThis episode explores World-R1, a post-training method for improving 3D consistency in text-to-video generation without redesigning the underlying model architecture. It explains how the approach combines reinforcement learning, pretrained 3D reconstruction critics, vision-language rewards, and camera-motion-focused prompt data to push generated videos toward more stable geometry under viewpoint changes. The discussion highlights why this matters for scene persistence, occlusion, and camera motion, especially if video models are ever to serve as usable world models rather than just visually plausible clip generators. Listeners would find it interesting because it digs into a concrete attempt to make today’s impressive but fragile video systems behave more like coherent simulated worlds.Sources:1. World-R1: Reinforcing 3D Constraints for Text-to-Video Generation — Weijie Wang, Xiaoxuan He, Youping Gu, Yifan Yang, Zeyu Zhang, Yefei He, Yanbo Ding, Xirui Hu, Donny Y. Chen, Zhiyuan He, Yuqing Yang, Bohan Zhuang, 2026http://arxiv.org/abs/2604.247642. Make-A-Video: Text-to-Video Generation without Text-Video Data — Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, Yaniv Taigman, 2022https://scholar.google.com/scholar?q=Make-A-Video%3A+Text-to-Video+Generation+without+Text-Video+Data3. Imagen Video: High Definition Video Generation with Diffusion Models — Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, Tim Salimans, 2022https://scholar.google.com/scholar?q=Imagen+Video%3A+High+Definition+Video+Generation+with+Diffusion+Models4. Lumiere: A Space-Time Diffusion Model for Video Generation — Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Michael Rubinstein, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel, Inbar Mosseri, 2024https://scholar.google.com/scholar?q=Lumiere%3A+A+Space-Time+Diffusion+Model+for+Video+Generation5. Video generation models as world simulators — Tim Brooks, Bill Peebles, Conor Holmes, Will DePue, Alex Payne, Robin Rombach, Patrick Esser, Jon Barron, Bhargav Chan, and OpenAI collaborators, 2024https://scholar.google.com/scholar?q=Video+generation+models+as+world+simulators6. CameraCtrl: Enabling Camera Control for Text-to-Video Generation — Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, Ceyuan Yang, 2024https://scholar.google.com/scholar?q=CameraCtrl%3A+Enabling+Camera+Control+for+Text-to-Video+Generation7. WorldForge: Unlocking Emergent 3D/4D Generation in Video Diffusion Model via Training-Free Guidance — Chenxi Song, Yanming Yang, Tong Zhao, Ruibo Li, Chi Zhang, 2025https://scholar.google.com/scholar?q=WorldForge%3A+Unlocking+Emergent+3D%2F4D+Generation+in+Video+Diffusion+Model+via+Training-Free+Guidance8. FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Prediction — Yixiang Dai, Fan Jiang, Chiyu Wang, Mu Xu, Yonggang Qi, 2025https://scholar.google.com/scholar?q=FantasyWorld%3A+Geometry-Consistent+World+Modeling+via+Unified+Video+and+3D+Prediction9. 3D and 4D World Modeling: A Survey — Lingdong Kong, Wesley Yang, Jianbiao Mei, Youquan Liu, Ao Liang, Dekai Zhu and many coauthors, 2025https://scholar.google.com/scholar?q=3D+and+4D+World+Modeling%3A+A+Survey10. Wan: Open and Advanced Large-Scale Video Generative Models — WanTeam, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, and many others, 2025https://scholar.google.com/scholar?q=Wan%3A+Open+and+Advanced+Large-Scale+Video+Generative+Models11. Flow-GRPO: Training Flow Matching Models via Online RL — Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, Wanli Ouyang, 2025https://scholar.google.com/scholar?q=Flow-GRPO%3A+Training+Flow+Matching+Models+via+Online+RL12. Depth Anything 3: Recovering the Visual Space from Any Views — Haotong Lin, Sili Chen, Junhao Liew, Donny Y. Chen, Zhenyu Li, Guang Shi, Jiashi Feng, Bingyi Kang, 2025https://scholar.google.com/scholar?q=Depth+Anything+3%3A+Recovering+the+Visual+Space+from+Any+Views13. VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward — Zhaochong An, Orest Kupyn, Theo Uscidda, Andrea Colaco, Karan Ahuja, Serge Belongie, Mar Gonzalez-Franco, Marta Tintore Gazulla, 2026https://scholar.google.com/scholar?q=VGGRPO%3A+Towards+World-Consistent+Video+Generation+with+4D+Latent+Reward14. WorldWarp: Propagating 3D Geometry with Asynchronous Video Diffusion — Hanyang Kong, Xingyi Yang, Xiaoxu Zheng, Xinchao Wang, 2025https://scholar.google.com/scholar?q=WorldWarp%3A+Propagating+3D+Geometry+with+Asynchronous+Video+Diffusion15. Pre-Trained Video Generative Models as World Simulators — Haoran He, Yang Zhang, Liang Lin, Zhongwen Xu, and collaborators, 2025https://scholar.google.com/scholar?q=Pre-Trained+Video+Generative+Models+as+World+Simulators16. Towards Realistic and Consistent Orbital Video Generation via 3D Foundation Priors — authors not visible in the snippet, recent, likely 2025-2026https://scholar.google.com/scholar?q=Towards+Realistic+and+Consistent+Orbital+Video+Generation+via+3D+Foundation+Priors17. Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding — authors not visible in the snippet, recent, likely 2025-2026https://scholar.google.com/scholar?q=Generation+Models+Know+Space%3A+Unleashing+Implicit+3D+Priors+for+Scene+Understanding18. Is a Picture Worth a Thousand Words? Delving into Spatial Reasoning for Vision-Language Models — authors not visible in the snippet, recent, likely 2024-2025https://scholar.google.com/scholar?q=Is+a+Picture+Worth+a+Thousand+Words%3F+Delving+into+Spatial+Reasoning+for+Vision-Language+Models19. Fast Multi-View Consistent 3D Editing with Video Priors — authors not visible in the snippet, recent, likely 2024-2025https://scholar.google.com/scholar?q=Fast+Multi-View+Consistent+3D+Editing+with+Video+Priors20. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp321. AI Post Transformers: DreamerV3 World Models Across 150 Tasks — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-20-dreamerv3-world-models-across-150-tasks-af5edb.mp3Interactive Visualization: World-R1 Improves 3D Consistency in Text-to-Video11 min
- FPGA Neural Network Accelerators for SpaceThis episode explores a survey of FPGA-based neural network accelerators for space applications, focusing on what the literature actually demonstrates rather than treating “space AI” as a single vague category. It explains why FPGAs are appealing for onboard inference, including tight control over hardware, energy efficiency, and the ability to support vision, autonomy, compression, navigation, and selective downlink under severe space constraints like limited bandwidth, latency, power, and thermal limits. The discussion also emphasizes a central argument of the paper: evidence for true space-ready systems is thinner than the hype suggests, with a need to distinguish lab demos on commercial boards from hardware that can handle radiation and mission-critical fault tolerance. Listeners would find it interesting because it connects modern AI hardware design to the harsh realities of spacecraft engineering and shows how a careful survey can reveal where the field is mature, where it is overstating its progress, and what technical gaps still matter most.Sources:1. FPGA-Based Neural Network Accelerators for Space Applications: A Survey — Pedro Antunes, Artur Podobas, 2025http://arxiv.org/abs/2504.161732. An FPGA-Based Hardware Accelerator for CNNs Inference on Board Satellites: Benchmarking with Myriad 2-Based Solution for the CloudScout Case Study — Emilio Rapuano, Gabriele Meoni, Tommaso Pacini, Gianmarco Dinelli, Gianluca Furano, Gianluca Giuffrida, Luca Fanucci, 2021https://scholar.google.com/scholar?q=An+FPGA-Based+Hardware+Accelerator+for+CNNs+Inference+on+Board+Satellites%3A+Benchmarking+with+Myriad+2-Based+Solution+for+the+CloudScout+Case+Study3. Reconfigurable Framework for Resilient Semantic Segmentation for Space Applications — Sebastian Sabogal, Alan George, Gary Crum, 2021https://scholar.google.com/scholar?q=Reconfigurable+Framework+for+Resilient+Semantic+Segmentation+for+Space+Applications4. Systematic Reliability Evaluation of FPGA Implemented CNN Accelerators — Zhen Gao, Shihui Gao, Yi Yao, Qiang Liu, Shulin Zeng, Guangjun Ge, Yu Wang, Anees Ullah, Pedro Reviriego, 2023https://scholar.google.com/scholar?q=Systematic+Reliability+Evaluation+of+FPGA+Implemented+CNN+Accelerators5. Online continual streaming learning for embedded space applications — Van-Tam Nguyen, Alaa Mazouz, 2024https://scholar.google.com/scholar?q=Online+continual+streaming+learning+for+embedded+space+applications6. A Quarter of a Century of Neuromorphic Architectures on FPGAs -- an Overview — Wiktor J. Szczerek, Artur Podobas, 2025https://scholar.google.com/scholar?q=A+Quarter+of+a+Century+of+Neuromorphic+Architectures+on+FPGAs+--+an+Overview7. Hardware platforms enabling edge AI for space applications: A critical review — unknown, approximate recent review authors, 2024 or 2025https://scholar.google.com/scholar?q=Hardware+platforms+enabling+edge+AI+for+space+applications%3A+A+critical+review8. Study of Radiation Effects on FPGA and GPU based Neural Networks Accelerator Designs — unknown, approximate recent systems authors, 2024 or 2025https://scholar.google.com/scholar?q=Study+of+Radiation+Effects+on+FPGA+and+GPU+based+Neural+Networks+Accelerator+Designs9. A Radiation-Hardened Neuromorphic Imager with Self-Healing Spiking Pixels and Unified Spiking Neural Network for Space Robotics — unknown, approximate neuromorphic hardware authors, 2024 or 2025https://scholar.google.com/scholar?q=A+Radiation-Hardened+Neuromorphic+Imager+with+Self-Healing+Spiking+Pixels+and+Unified+Spiking+Neural+Network+for+Space+Robotics10. Onboard Optimization and Learning: A Survey — unknown, approximate recent survey authors, 2024 or 2025https://scholar.google.com/scholar?q=Onboard+Optimization+and+Learning%3A+A+Survey11. Review on hardware devices and software techniques enabling neural network inference onboard satellites — unknown, approximate recent review authors, 2024 or 2025https://scholar.google.com/scholar?q=Review+on+hardware+devices+and+software+techniques+enabling+neural+network+inference+onboard+satellites12. AI Post Transformers: FlatAttention for Tile-Based Accelerator Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-flatattention-for-tile-based-accelerator-56e6ca.mp313. AI Post Transformers: AWQ: On-Device LLM Compression and Acceleration — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/awq-on-device-llm-compression-and-acceleration/14. AI Post Transformers: KVSwap for Disk-Aware Long-Context On-Device Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-16-kvswap-for-disk-aware-long-context-on-de-f3c15e.mp3Interactive Visualization: FPGA Neural Network Accelerators for Space0 min
- Deep Learning in Spiking Neural NetworksThis episode explores how spiking neural networks differ from conventional deep networks by representing information as time-based spikes rather than continuous activations, and why that makes them both appealing and difficult to train. It breaks down the main research directions in the field, including local spike-timing-dependent plasticity, direct gradient-based training of spiking models, and ANN-to-SNN conversion, emphasizing that these approaches solve different problems and should not be treated as interchangeable. The discussion highlights the paper’s core argument that spiking models may help bridge neuroscience, energy-efficient neuromorphic hardware, and competitive machine learning, while also questioning whether claims about progress often blur together engineering wins, biological realism, and benchmark performance. Listeners would find it interesting for its clear explanation of the ANN-SNN gap, the tradeoffs behind biologically inspired AI, and the unresolved question of whether spiking systems can become both practical and scientifically meaningful.Sources:1. Deep Learning in Spiking Neural Networks — Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothee Masquelier, Anthony S. Maida, 2018http://arxiv.org/abs/1804.081502. Networks of Spiking Neurons: The Third Generation of Neural Network Models — Wolfgang Maass, 1997https://scholar.google.com/scholar?q=Networks+of+Spiking+Neurons%3A+The+Third+Generation+of+Neural+Network+Models3. Deep Learning in Spiking Neural Networks — Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothee Masquelier, Anthony Maida, 2019https://scholar.google.com/scholar?q=Deep+Learning+in+Spiking+Neural+Networks4. Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks — Emre O. Neftci, Hesham Mostafa, Friedemann Zenke, 2019https://scholar.google.com/scholar?q=Surrogate+Gradient+Learning+in+Spiking+Neural+Networks%3A+Bringing+the+Power+of+Gradient-Based+Optimization+to+Spiking+Neural+Networks5. Backpropagation and the Brain — Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin J. Akerman, Geoffrey Hinton, 2020https://scholar.google.com/scholar?q=Backpropagation+and+the+Brain6. Fast-Classifying, High-Accuracy Spiking Deep Networks Through Weight and Threshold Balancing — Peter U. Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, Michael Pfeiffer, 2015https://scholar.google.com/scholar?q=Fast-Classifying%2C+High-Accuracy+Spiking+Deep+Networks+Through+Weight+and+Threshold+Balancing7. Training Deep Spiking Neural Networks Using Backpropagation — Jun Haeng Lee, Tobi Delbruck, Michael Pfeiffer, 2016https://scholar.google.com/scholar?q=Training+Deep+Spiking+Neural+Networks+Using+Backpropagation8. Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification — Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, Shih-Chii Liu, 2017https://scholar.google.com/scholar?q=Conversion+of+Continuous-Valued+Deep+Networks+to+Efficient+Event-Driven+Networks+for+Image+Classification9. Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks — Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi, 2018https://scholar.google.com/scholar?q=Spatio-Temporal+Backpropagation+for+Training+High-Performance+Spiking+Neural+Networks10. SLAYER: Spike Layer Error Reassignment in Time — Sumit Bam Shrestha, Garrick Orchard, 2018https://scholar.google.com/scholar?q=SLAYER%3A+Spike+Layer+Error+Reassignment+in+Time11. A spiking neural network with continuous local learning for robust online brain machine interface — approx. unknown from snippet, recenthttps://scholar.google.com/scholar?q=A+spiking+neural+network+with+continuous+local+learning+for+robust+online+brain+machine+interface12. Adaptive deep spiking neural network with global-local learning via balanced excitatory and inhibitory mechanism — approx. unknown from snippet, recenthttps://scholar.google.com/scholar?q=Adaptive+deep+spiking+neural+network+with+global-local+learning+via+balanced+excitatory+and+inhibitory+mechanism13. Delay learning based on temporal coding in spiking neural networks — approx. unknown from snippet, recenthttps://scholar.google.com/scholar?q=Delay+learning+based+on+temporal+coding+in+spiking+neural+networks14. Temporal-coded spiking neural networks with dynamic firing threshold: Learning with event-driven backpropagation — approx. unknown from snippet, 2021https://scholar.google.com/scholar?q=Temporal-coded+spiking+neural+networks+with+dynamic+firing+threshold%3A+Learning+with+event-driven+backpropagation15. Spikingformer: Spike-driven residual learning for transformer-based spiking neural network — approx. unknown from snippet, recenthttps://scholar.google.com/scholar?q=Spikingformer%3A+Spike-driven+residual+learning+for+transformer-based+spiking+neural+network16. Sstformer: Bridging spiking neural network and memory support transformer for frame-event based recognition — approx. unknown from snippet, recenthttps://scholar.google.com/scholar?q=Sstformer%3A+Bridging+spiking+neural+network+and+memory+support+transformer+for+frame-event+based+recognition17. TE-Spikformer: Temporal-enhanced spiking neural network with transformer — approx. unknown from snippet, recenthttps://scholar.google.com/scholar?q=TE-Spikformer%3A+Temporal-enhanced+spiking+neural+network+with+transformer18. NeuronSpark: A Spiking Neural Network Language Model with Selective State Space Dynamics — approx. unknown from snippet, recenthttps://scholar.google.com/scholar?q=NeuronSpark%3A+A+Spiking+Neural+Network+Language+Model+with+Selective+State+Space+Dynamics19. SpikingSSMs: Learning long sequences with sparse and parallel spiking state space models — approx. unknown from snippet, recenthttps://scholar.google.com/scholar?q=SpikingSSMs%3A+Learning+long+sequences+with+sparse+and+parallel+spiking+state+space+models20. Delays in Spiking Neural Networks: A State Space Model Approach — approx. unknown from snippet, recenthttps://scholar.google.com/scholar?q=Delays+in+Spiking+Neural+Networks%3A+A+State+Space+Model+Approach21. AI Post Transformers: Directly Trained Spiking DQNs for Atari — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-20-directly-trained-spiking-dqns-for-atari-947693.mp322. AI Post Transformers: SpikingBrain: Brain-Inspired LLMs for Efficient Long-Context Processing — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/spikingbrain-brain-inspired-llms-for-efficient-long-context-processing/Interactive Visualization: Deep Learning in Spiking Neural Networks16 min
- AgenticQwen and Small Industrial Tool AgentsThis episode explores AgenticQwen, a system for training small open-weight language models to handle industrial-scale tool use through repeated reinforcement learning and synthetic data generation. It explains the paper’s central idea of dual data flywheels: one that turns reasoning failures into harder verifiable tasks, and another that expands simple agent workflows into branching, tool-using trajectories with recovery steps and user interaction. The discussion contrasts imitation from synthetic data with trajectory-level reinforcement learning, arguing that real agent competence depends on rewarding decisions like tool choice, clarification, and error recovery rather than just polished final answers. Listeners would find it interesting for its grounded look at whether small models can become cheap, fast, and genuinely useful agents for high-volume real-world work without relying on massive frontier systems.Sources:1. AgenticQwen: Training Small Agentic Language Models with Dual Data Flywheels for Industrial-Scale Tool Use — Yuanjie Lyu, Chengyu Wang, Haonan Zheng, Yuanhao Yue, Junbing Yan, Ming Wang, Jun Huang, 2026http://arxiv.org/abs/2604.215902. Training Language Models to Follow Instructions with Human Feedback — Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin and others, 2022https://scholar.google.com/scholar?q=Training+Language+Models+to+Follow+Instructions+with+Human+Feedback3. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek-AI, 2025https://scholar.google.com/scholar?q=DeepSeek-R1%3A+Incentivizing+Reasoning+Capability+in+LLMs+via+Reinforcement+Learning4. Agent Lightning: Train ANY AI Agents with Reinforcement Learning — Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang, Siyun Zhao, Dongsheng Li, Luna K. Qiu, Yuqing Yang and others, 2025https://scholar.google.com/scholar?q=Agent+Lightning%3A+Train+ANY+AI+Agents+with+Reinforcement+Learning5. CM2: Reinforcement Learning with Checklist Rewards for Multi-Turn and Multi-Step Agentic Tool Use — Zhen Zhang, Kaiqiang Song, Xun Wang, Yebowen Hu, Weixiang Yan, Chenyang Zhao, Henry Peng Zou and others, 2026https://scholar.google.com/scholar?q=CM2%3A+Reinforcement+Learning+with+Checklist+Rewards+for+Multi-Turn+and+Multi-Step+Agentic+Tool+Use6. Self-Instruct: Aligning Language Models with Self-Generated Instructions — Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi, 2022https://scholar.google.com/scholar?q=Self-Instruct%3A+Aligning+Language+Models+with+Self-Generated+Instructions7. Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, Ahmed Awadallah, 2023https://scholar.google.com/scholar?q=Orca%3A+Progressive+Learning+from+Complex+Explanation+Traces+of+GPT-48. Textbooks Are All You Need — Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio Cesar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi and others, 2023https://scholar.google.com/scholar?q=Textbooks+Are+All+You+Need9. TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — Ronen Eldan, Yuanzhi Li, 2023https://scholar.google.com/scholar?q=TinyStories%3A+How+Small+Can+Language+Models+Be+and+Still+Speak+Coherent+English%3F10. A Survey of Behavior Trees in Robotics and AI — Matteo Iovino, Edvards Scukins, Jonathan Styrud, Petter Ogren, Christian Smith, 2022https://scholar.google.com/scholar?q=A+Survey+of+Behavior+Trees+in+Robotics+and+AI11. Robot Behavior-Tree-Based Task Generation with Large Language Models — Yue Cao, C. S. George Lee, 2023https://scholar.google.com/scholar?q=Robot+Behavior-Tree-Based+Task+Generation+with+Large+Language+Models12. Behavior Trees Enable Structured Programming of Language Model Agents — Richard Kelley, 2024https://scholar.google.com/scholar?q=Behavior+Trees+Enable+Structured+Programming+of+Language+Model+Agents13. Behavior Tree Generation and Adaptation for a Social Robot Control with LLMs — authors from the 2025 Robotics and Autonomous Systems paper, 2025https://scholar.google.com/scholar?q=Behavior+Tree+Generation+and+Adaptation+for+a+Social+Robot+Control+with+LLMs14. Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards — Yuan-Jay Lü, Chengyu Wang, Lei Shen, Jun Huang, Tong Xu, 2026https://scholar.google.com/scholar?q=Mock+Worlds%2C+Real+Skills%3A+Building+Small+Agentic+Language+Models+with+Synthetic+Tasks%2C+Simulated+Environments%2C+and+Rubric-Based+Rewards15. τ^2-Bench: Evaluating Conversational Agents in a Dual-Control Environment — Victor Barres, Honghua Dong, Soham Ray, Xujie Si, Karthik Narasimhan, 2025https://scholar.google.com/scholar?q=%CF%84%5E2-Bench%3A+Evaluating+Conversational+Agents+in+a+Dual-Control+Environment16. Procedural Environment Generation for Tool-Use Agents — Michael Sullivan, Mareike Hartmann, Alexander Koller, 2025https://scholar.google.com/scholar?q=Procedural+Environment+Generation+for+Tool-Use+Agents17. ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Context — approx. ASTRA-bench authors unknown from snippet, recenthttps://scholar.google.com/scholar?q=ASTRA-bench%3A+Evaluating+Tool-Use+Agent+Reasoning+and+Action+Planning+with+Personal+User+Context18. m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks — approx. m&m's benchmark authors unknown from snippet, recenthttps://scholar.google.com/scholar?q=m%26m%27s%3A+A+Benchmark+to+Evaluate+Tool-Use+for+multi-step+multi-modal+Tasks19. EvilGenie: A Reward Hacking Benchmark — approx. EvilGenie authors unknown from snippet, recenthttps://scholar.google.com/scholar?q=EvilGenie%3A+A+Reward+Hacking+Benchmark20. Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking — approx. reward auditing authors unknown from snippet, recenthttps://scholar.google.com/scholar?q=Adversarial+Reward+Auditing+for+Active+Detection+and+Mitigation+of+Reward+Hacking21. AI Post Transformers: Benchmarking Test-Time Scaling for General LLM Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-benchmarking-test-time-scaling-for-gener-8f14f9.mp322. AI Post Transformers: Experience-Based Learning Beyond Human Data — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-experience-based-learning-beyond-human-d-b0caa4.mp323. AI Post Transformers: Learning to Reason with 13 Parameters — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-learning-to-reason-with-13-parameters-54c87f.mp324. AI Post Transformers: Kimi K2.5 and Visual Agent Swarms — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-24-kimi-k25-and-visual-agent-swarms-7d04d7.mp325. AI Post Transformers: Simple Self-Distillation for Better Code Generation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-02-simple-self-distillation-for-better-code-cc88e0.mp3Interactive Visualization: AgenticQwen and Small Industrial Tool Agents0 min
- Compressed Convolutional Attention in Latent SpaceThis episode explores Compressed Convolutional Attention (CCA), a new approach that pushes transformer attention itself into a shared compressed latent space instead of only compressing the KV-cache. It walks through the evolution from standard multi-head attention to MQA, GQA, and MLA, explaining why earlier efficiency methods mostly targeted decode-time memory while leaving much of the attention compute burden intact. The discussion highlights the paper’s main claim that CCA, especially when combined with grouped-query ideas in CCGQA, can reduce parameters, attention FLOPs, and KV-cache size at the same time while outperforming strong baselines in both dense and mixture-of-experts models. Listeners would find it interesting because it gets into the real systems question behind long-context models: whether a cleaner theoretical efficiency idea can actually translate into better economics and practical performance at scale.Sources:1. Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space — Tomas Figliolia, Nicholas Alonso, Rishi Iyer, Quentin Anthony, Beren Millidge, 2025http://arxiv.org/abs/2510.044762. Linformer: Self-Attention with Linear Complexity — Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, Hao Ma, 2020https://scholar.google.com/scholar?q=Linformer%3A+Self-Attention+with+Linear+Complexity3. Perceiver: General Perception with Iterative Attention — Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, Joao Carreira, 2021https://scholar.google.com/scholar?q=Perceiver%3A+General+Perception+with+Iterative+Attention4. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI team, 2024https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model5. Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space — Tomas Figliolia, Nicholas Alonso, Rishi Iyer, Quentin Anthony, Beren Millidge, 2025https://scholar.google.com/scholar?q=Compressed+Convolutional+Attention%3A+Efficient+Attention+in+a+Compressed+Latent+Space6. Fast Transformer Decoding: One Write-Head is All You Need — Noam Shazeer, 2019https://scholar.google.com/scholar?q=Fast+Transformer+Decoding%3A+One+Write-Head+is+All+You+Need7. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, Sumit Sanghai, 2023https://scholar.google.com/scholar?q=GQA%3A+Training+Generalized+Multi-Query+Transformer+Models+from+Multi-Head+Checkpoints8. StarCoder 2 and The Stack v2: The Next Generation — Anton Lozhkov, Raymond Li, Loubna Ben Allal and many others, 2024https://scholar.google.com/scholar?q=StarCoder+2+and+The+Stack+v2%3A+The+Next+Generation9. RoFormer: Enhanced Transformer with Rotary Position Embedding — Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu, 2021https://scholar.google.com/scholar?q=RoFormer%3A+Enhanced+Transformer+with+Rotary+Position+Embedding10. Zyda-2: a 5 Trillion Token High-Quality Dataset — Yury Tokpanov, Paolo Glorioso, Quentin Anthony, Beren Millidge, 2024https://scholar.google.com/scholar?q=Zyda-2%3A+a+5+Trillion+Token+High-Quality+Dataset11. RazorAttention: Efficient KV Cache Compression through Retrieval Heads — approx. recent efficient inference / attention-systems authors, 2024/2025https://scholar.google.com/scholar?q=RazorAttention%3A+Efficient+KV+Cache+Compression+through+Retrieval+Heads12. Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning — approx. recent LLM inference authors, 2024/2025https://scholar.google.com/scholar?q=Not+All+Heads+Matter%3A+A+Head-Level+KV+Cache+Compression+Method+with+Integrated+Retrieval+and+Reasoning13. State-Space Models Can Learn In-Context by Gradient Descent — approx. recent theory / sequence-model authors, 2024/2025https://scholar.google.com/scholar?q=State-Space+Models+Can+Learn+In-Context+by+Gradient+Descent14. Structured State Space Models for In-Context Reinforcement Learning — approx. recent sequence-model authors, 2024/2025https://scholar.google.com/scholar?q=Structured+State+Space+Models+for+In-Context+Reinforcement+Learning15. SHRP: Specialized Head Routing and Pruning for Efficient Encoder Compression — approx. recent model-compression authors, 2024/2025https://scholar.google.com/scholar?q=SHRP%3A+Specialized+Head+Routing+and+Pruning+for+Efficient+Encoder+Compression16. AI Post Transformers: Nemotron 3 Super Hybrid Mamba-Transformer MoE — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-nemotron-3-super-hybrid-mamba-transforme-31ac75.mp317. AI Post Transformers: Batch-Aware Expert Routing for Faster MoE Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-batch-aware-expert-routing-for-faster-mo-683ab6.mp318. AI Post Transformers: Mamba-3 for Efficient Sequence Modeling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-16-mamba-3-for-efficient-sequence-modeling-97a22a.mp319. AI Post Transformers: FlashAttention: IO-Aware Fast and Memory-Efficient Attention — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/flashattention-io-aware-fast-and-memory-efficient-attention/20. AI Post Transformers: Kimi Linear: Efficient Expressive Attention Architecture — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/kimi-linear-efficient-expressive-attention-architecture/21. AI Post Transformers: RoPE — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/rope/22. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp323. AI Post Transformers: KV Cache TTL for Multi-Turn Agent Scheduling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-09-kv-cache-ttl-for-multi-turn-agent-schedu-996bf1.mp3Interactive Visualization: Compressed Convolutional Attention in Latent Space0 min
- Long Short-Term Memory and Vanishing GradientsThis episode explores the 1997 LSTM paper as a targeted fix for why standard recurrent neural networks failed on long-range dependencies: backpropagated error signals either vanish or explode when carried through many time steps. It explains how earlier training methods like BPTT and RTRL worked, why the underlying optimization problem was mathematically ill-conditioned, and how LSTM’s memory cells, Constant Error Carousel, and gating mechanisms were designed to preserve usable gradients over long lags. The discussion argues that the paper’s real contribution was not “solving memory” in a broad sense, but introducing a specific architectural mechanism validated mainly on synthetic delayed-response tasks. Listeners would find it interesting for its clear separation of the vanishing-gradient diagnosis from the LSTM remedy, and for its more nuanced view of a model often remembered only through later hype and tutorials.Sources:1. Long Short-Term Memory and Vanishing Gradientshttps://www.bioinf.jku.at/publications/older/2604.pdf2. Learning Long-Term Dependencies with Gradient Descent is Difficult — Yoshua Bengio, Patrice Simard, Paolo Frasconi, 1994https://scholar.google.com/scholar?q=Learning+Long-Term+Dependencies+with+Gradient+Descent+is+Difficult3. The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions — Sepp Hochreiter, 1998https://scholar.google.com/scholar?q=The+Vanishing+Gradient+Problem+During+Learning+Recurrent+Neural+Nets+and+Problem+Solutions4. On the Difficulty of Training Recurrent Neural Networks — Razvan Pascanu, Tomas Mikolov, Yoshua Bengio, 2013https://scholar.google.com/scholar?q=On+the+Difficulty+of+Training+Recurrent+Neural+Networks5. Understanding the Difficulty of Training Deep Feedforward Neural Networks — Xavier Glorot, Yoshua Bengio, 2010https://scholar.google.com/scholar?q=Understanding+the+Difficulty+of+Training+Deep+Feedforward+Neural+Networks6. Long Short-Term Memory — Sepp Hochreiter, Jürgen Schmidhuber, 1997https://scholar.google.com/scholar?q=Long+Short-Term+Memory7. Learning to Forget: Continual Prediction with LSTM — Felix A. Gers, Jürgen Schmidhuber, Fred Cummins, 2000https://scholar.google.com/scholar?q=Learning+to+Forget%3A+Continual+Prediction+with+LSTM8. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling — Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio, 2014https://scholar.google.com/scholar?q=Empirical+Evaluation+of+Gated+Recurrent+Neural+Networks+on+Sequence+Modeling9. Language Modeling with Gated Convolutional Networks — Yann N. Dauphin, Angela Fan, Michael Auli, David Grangier, 2017https://scholar.google.com/scholar?q=Language+Modeling+with+Gated+Convolutional+Networks10. Backpropagation Through Time: What It Does and How to Do It — Paul J. Werbos, 1990https://scholar.google.com/scholar?q=Backpropagation+Through+Time%3A+What+It+Does+and+How+to+Do+It11. A Learning Algorithm for Continually Running Fully Recurrent Neural Networks — Ronald J. Williams, David Zipser, 1989https://scholar.google.com/scholar?q=A+Learning+Algorithm+for+Continually+Running+Fully+Recurrent+Neural+Networks12. Sequence to Sequence Learning with Neural Networks — Ilya Sutskever, Oriol Vinyals, Quoc V. Le, 2014https://scholar.google.com/scholar?q=Sequence+to+Sequence+Learning+with+Neural+Networks13. Untersuchungen zu dynamischen neuronalen Netzen — Sepp Hochreiter, 1991https://scholar.google.com/scholar?q=Untersuchungen+zu+dynamischen+neuronalen+Netzen14. The Problem of Learning Long-Term Dependencies in Recurrent Networks — Yoshua Bengio, Patrice Simard, Paolo Frasconi, 1994https://scholar.google.com/scholar?q=The+Problem+of+Learning+Long-Term+Dependencies+in+Recurrent+Networks15. Recurrent Networks and Neural Sequence Chunking — Jürgen Schmidhuber, 1992https://scholar.google.com/scholar?q=Recurrent+Networks+and+Neural+Sequence+Chunking16. The Continually Running Fully Recurrent Neural Network and Learning Long-Term Dependencies — Michael C. Mozer, 1992https://scholar.google.com/scholar?q=The+Continually+Running+Fully+Recurrent+Neural+Network+and+Learning+Long-Term+Dependencies17. Finding Structure in Time — Jeffrey L. Elman, 1990https://scholar.google.com/scholar?q=Finding+Structure+in+Time18. Learning Precise Timing with LSTM Recurrent Networks — Felix A. Gers, Jürgen Schmidhuber, Fred Cummins, 2000https://scholar.google.com/scholar?q=Learning+Precise+Timing+with+LSTM+Recurrent+Networks19. Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling — Hasim Sak, Andrew Senior, Françoise Beaufays, 2014https://scholar.google.com/scholar?q=Long+Short-Term+Memory+Recurrent+Neural+Network+Architectures+for+Large+Scale+Acoustic+Modeling20. Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training — unclear from snippet, recenthttps://scholar.google.com/scholar?q=Exploiting+Symmetric+Temporally+Sparse+BPTT+for+Efficient+RNN+Training21. Fast Training of Recurrent Neural Networks with Stationary State Feedbacks — unclear from snippet, recenthttps://scholar.google.com/scholar?q=Fast+Training+of+Recurrent+Neural+Networks+with+Stationary+State+Feedbacks22. Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff — unclear from snippet, recenthttps://scholar.google.com/scholar?q=Simple+Linear+Attention+Language+Models+Balance+the+Recall-Throughput+Tradeoff23. Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory — unclear from snippet, recenthttps://scholar.google.com/scholar?q=Beyond+Attention%3A+Breaking+the+Limits+of+Transformer+Context+Length+with+Recurrent+Memory24. A Systematic Analysis of Hybrid Linear Attention — unclear from snippet, recenthttps://scholar.google.com/scholar?q=A+Systematic+Analysis+of+Hybrid+Linear+Attention25. In-Context Learning as Conditioned Associative Memory Retrieval — unclear from snippet, recenthttps://scholar.google.com/scholar?q=In-Context+Learning+as+Conditioned+Associative+Memory+Retrieval26. Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting — unclear from snippet, recenthttps://scholar.google.com/scholar?q=Dual+Process+Learning%3A+Controlling+Use+of+In-Context+vs.+In-Weights+Strategies+with+Weight+Forgetting27. Memorization in In-Context Learning — unclear from snippet, recenthttps://scholar.google.com/scholar?q=Memorization+in+In-Context+Learning28. Pretraining Without Attention — approx. modern sequence-modeling authors; exact authors unclear from snippet, recenthttps://scholar.google.com/scholar?q=Pretraining+Without+Attention29. AI Post Transformers: Mamba-3 for Efficient Sequence Modeling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-16-mamba-3-for-efficient-sequence-modeling-97a22a.mp330. AI Post Transformers: Kimi Linear: Efficient Expressive Attention Architecture — Hal Turing & Dr. Ada Shannon, Sun,https://podcast.do-not-panic.com/episodes/kimi-linear-efficient-expressive-attention-architecture/Interactive Visualization: Long Short-Term Memory and Vanishing Gradients17 min
- Breiman's Two Cultures of Statistical ModelingThis episode explores Leo Breiman’s “Statistical Modeling: The Two Cultures” as a sharp argument about a core divide in data science: building explicit probabilistic models to explain the world versus training algorithms that simply predict well on new data. It traces how that split maps onto inference versus prediction, connects Breiman’s critique to earlier ideas from Tukey and Box, and shows how later work such as Shmueli’s formalized the distinction. The discussion also grounds the debate in concrete methods, from linear regression and Cox models to CART, bagging, random forests, and neural networks, highlighting why algorithmic approaches gained ground on messy, high-dimensional problems. Listeners would find it interesting because it explains a foundational argument that still shapes modern machine learning, while also probing where interpretable classical models remain essential in areas like medicine, policy, and reliability.Sources:1. Breiman's Two Cultures of Statistical Modelinghttps://www2.math.uu.se/~thulin/mm/breiman.pdf2. The Future of Data Analysis — John W. Tukey, 1962https://scholar.google.com/scholar?q=The+Future+of+Data+Analysis3. Science and Statistics — George E. P. Box, 1976https://scholar.google.com/scholar?q=Science+and+Statistics4. Regression Models and Life-Tables — D. R. Cox, 1972https://scholar.google.com/scholar?q=Regression+Models+and+Life-Tables5. Statistical Modeling: The Two Cultures — Leo Breiman, 2001https://scholar.google.com/scholar?q=Statistical+Modeling%3A+The+Two+Cultures6. Bagging Predictors — Leo Breiman, 1996https://scholar.google.com/scholar?q=Bagging+Predictors7. Random Forests — Leo Breiman, 2001https://scholar.google.com/scholar?q=Random+Forests8. Greedy Function Approximation: A Gradient Boosting Machine — Jerome H. Friedman, 2001https://scholar.google.com/scholar?q=Greedy+Function+Approximation%3A+A+Gradient+Boosting+Machine9. Clinical versus Actuarial Judgment — Robyn M. Dawes, David Faust, Paul E. Meehl, 1989https://scholar.google.com/scholar?q=Clinical+versus+Actuarial+Judgment10. To Explain or To Predict? — Galit Shmueli, 2010https://scholar.google.com/scholar?q=To+Explain+or+To+Predict%3F11. Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning — Tal Yarkoni, Jacob Westfall, 2017https://scholar.google.com/scholar?q=Choosing+Prediction+Over+Explanation+in+Psychology%3A+Lessons+From+Machine+Learning12. Classification and Regression Trees — Leo Breiman, Jerome H. Friedman, Richard A. Olshen, Charles J. Stone, 1984https://scholar.google.com/scholar?q=Classification+and+Regression+Trees13. Induction of Decision Trees — J. R. Quinlan, 1986https://scholar.google.com/scholar?q=Induction+of+Decision+Trees14. A Random Forest Guided Tour — Gérard Biau, Erwan Scornet, 2016https://scholar.google.com/scholar?q=A+Random+Forest+Guided+Tour15. Learning Representations by Back-Propagating Errors — David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams, 1986https://scholar.google.com/scholar?q=Learning+Representations+by+Back-Propagating+Errors16. Multilayer Feedforward Networks Are Universal Approximators — Kurt Hornik, Maxwell Stinchcombe, Halbert White, 1989https://scholar.google.com/scholar?q=Multilayer+Feedforward+Networks+Are+Universal+Approximators17. ImageNet Classification with Deep Convolutional Neural Networks — Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, 2012https://scholar.google.com/scholar?q=ImageNet+Classification+with+Deep+Convolutional+Neural+Networks18. Deep Learning — Yann LeCun, Yoshua Bengio, Geoffrey Hinton, 2015https://scholar.google.com/scholar?q=Deep+Learning19. Arcing Classifiers — Leo Breiman, 1998https://scholar.google.com/scholar?q=Arcing+Classifiers20. Generalized Additive Models — Trevor Hastie and Robert Tibshirani, 1990https://scholar.google.com/scholar?q=Generalized+Additive+Models21. The Elements of Statistical Learning — Trevor Hastie, Robert Tibshirani, and Jerome Friedman, 2001https://scholar.google.com/scholar?q=The+Elements+of+Statistical+Learning22. Sparse Neural Additive Model: Interpretable Deep Learning with Feature Selection via Group Sparsity — Shiyun Xu, Zhiqi Bu, Pratik Chaudhari, Ian J. Barnett, 2022/2023https://scholar.google.com/scholar?q=Sparse+Neural+Additive+Model%3A+Interpretable+Deep+Learning+with+Feature+Selection+via+Group+Sparsity23. Neural Additive Models for Location Scale and Shape: A Framework for Interpretable Neural Regression Beyond the Mean — Anton Frederik Thielmann, Rene-Marcel Kruse, Thomas Kneib, Benjamin Safken, 2024https://scholar.google.com/scholar?q=Neural+Additive+Models+for+Location+Scale+and+Shape%3A+A+Framework+for+Interpretable+Neural+Regression+Beyond+the+Mean24. Conformal Prediction: A Data Perspective — Xiaofan Zhou, Baiting Chen, Yu Gui, Lu Cheng, 2024https://scholar.google.com/scholar?q=Conformal+Prediction%3A+A+Data+Perspective25. Large language model validity via enhanced conformal prediction methods — John J. Cherian, Isaac Gibbs, Emmanuel J. Candes, 2024https://scholar.google.com/scholar?q=Large+language+model+validity+via+enhanced+conformal+prediction+methods26. CPSign: conformal prediction for cheminformatics modeling — Staffan Arvidsson McShane, Ulf Norinder, Jonathan Alvarsson, Ernst Ahlberg, Lars Carlsson, Ola Spjuth, 2024https://scholar.google.com/scholar?q=CPSign%3A+conformal+prediction+for+cheminformatics+modeling27. Open Problems in Mechanistic Interpretability — Lee Sharkey et al., 2025https://scholar.google.com/scholar?q=Open+Problems+in+Mechanistic+Interpretability28. AI Post Transformers: Evaluating LLM Embeddings for Psychometric Personality Prediction — Hal Turing & Dr. Ada Shannon, Tue,https://podcast.do-not-panic.com/episodes/evaluating-llm-embeddings-for-psychometric-personality-prediction/29. AI Post Transformers: Introducing RTEB: Retrieval Embedding Benchmark — Hal Turing & Dr. Ada Shannon, Fri,https://podcast.do-not-panic.com/episodes/introducing-rteb-retrieval-embedding-benchmark/30. AI Post Transformers: Information Bottleneck-based Causal Attention for Medical Image Recognition — Hal Turing & Dr. Ada Shannon, Tue,https://podcast.do-not-panic.com/episodes/information-bottleneck-based-causal-attention-for-medical-image-recognition/Interactive Visualization: Breiman's Two Cultures of Statistical Modeling0 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.