Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- Benchmarking Test-Time Scaling for General LLM AgentsThis episode explores a paper that tests whether general LLM agents remain effective when search, coding, reasoning, and API/tool-use tasks are mixed together under one shared prompt, interface, and tool set rather than optimized benchmark-specific setups. It explains how the benchmark is built by unifying tasks from BrowseComp, WebVoyager, SWE-Bench Verified, Terminal-Bench, MathHay, Tau2-Bench, and MCP-Bench, forcing agents to infer the task type and select tools without domain-specific cues. The discussion highlights the paper’s core argument that conventional benchmarks can overstate capability by pre-structuring the environment, while a general setting better reflects real user requests and exposes weaknesses in planning, tool choice, and adaptation. Listeners would find it interesting for its clear look at test-time scaling in agents—giving the same model more turns or parallel attempts—and for its broader challenge to how agent intelligence should be evaluated.Sources:1. Benchmark Test-Time Scaling of General LLM Agents — Xiaochuan Li, Ryan Ming, Pranav Setlur, Abhijay Paladugu, Andy Tang, Hao Kang, Shuai Shao, Rong Jin, Chenyan Xiong, 2026http://arxiv.org/abs/2602.189982. SWE-Bench — Jimenez et al., 2023https://scholar.google.com/scholar?q=SWE-Bench3. Terminal-Bench — Aleithan et al., 2024https://scholar.google.com/scholar?q=Terminal-Bench4. BrowseComp — presumably cited in paper; exact citation not provided in excerpt, 2024/2025https://scholar.google.com/scholar?q=BrowseComp5. Mind2Web — Deng/He et al. or benchmark authors cited as Wei et al. 2025 / He et al. 2024 in excerpt context, 2024/2025https://scholar.google.com/scholar?q=Mind2Web6. WebVoyager — Zhou et al., 2023https://scholar.google.com/scholar?q=WebVoyager7. Tau2-Bench — not specified in excerpt, likely 2025/2026https://scholar.google.com/scholar?q=Tau2-Bench8. MCP-Bench — not specified in excerpt, likely 2025/2026https://scholar.google.com/scholar?q=MCP-Bench9. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Wang et al., 2022https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models10. Training Verifiers to Solve Math Word Problems — Cobbe et al., 2021https://scholar.google.com/scholar?q=Training+Verifiers+to+Solve+Math+Word+Problems11. Let's Verify Step by Step — Lightman et al., 2023https://scholar.google.com/scholar?q=Let%27s+Verify+Step+by+Step12. Quiet-STaR / test-time reasoning scaling related work — Zelikman et al., 2024https://scholar.google.com/scholar?q=Quiet-STaR+%2F+test-time+reasoning+scaling+related+work13. Snell et al. test-time scaling work — Snell et al., 2024https://scholar.google.com/scholar?q=Snell+et+al.+test-time+scaling+work14. Toolformer — Schick et al., 2023https://scholar.google.com/scholar?q=Toolformer15. Gorilla / APIBench-style tool-use work — Patil et al., 2024https://scholar.google.com/scholar?q=Gorilla+%2F+APIBench-style+tool-use+work16. Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents — approx. unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=Beyond+the+Context+Window%3A+A+Cost-Performance+Analysis+of+Fact-Based+Memory+vs.+Long-Context+LLMs+for+Persistent+Agents17. Memory in the Age of AI Agents — approx. unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=Memory+in+the+Age+of+AI+Agents18. Toward Conversational Agents with Context and Time Sensitive Long-Term Memory — approx. unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=Toward+Conversational+Agents+with+Context+and+Time+Sensitive+Long-Term+Memory19. When LLM Judge Scores Look Good but Best-of-N Decisions Fail — approx. unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=When+LLM+Judge+Scores+Look+Good+but+Best-of-N+Decisions+Fail20. When to Solve, When to Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning — approx. unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=When+to+Solve%2C+When+to+Verify%3A+Compute-Optimal+Problem+Solving+and+Generative+Verification+for+LLM+Reasoning21. Scalable Best-of-N Selection for Large Language Models via Self-Certainty — approx. unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=Scalable+Best-of-N+Selection+for+Large+Language+Models+via+Self-Certainty22. AgentClinic: A Multimodal Agent Benchmark to Evaluate AI in Simulated Clinical Environments — approx. unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=AgentClinic%3A+A+Multimodal+Agent+Benchmark+to+Evaluate+AI+in+Simulated+Clinical+Environments23. DABStep: Data Agent Benchmark for Multi-Step Reasoning — approx. unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=DABStep%3A+Data+Agent+Benchmark+for+Multi-Step+Reasoning24. GTA1: GUI Test-Time Scaling Agent — approx. unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=GTA1%3A+GUI+Test-Time+Scaling+Agent25. AI Post Transformers: SkillsBench for Evaluating Agent Skills — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-skillsbench-for-evaluating-agent-skills-58bb1e.mp326. AI Post Transformers: MEMSEARCHER: Reinforcement Learning for LLM Memory Management — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-memsearcher-reinforcement-learning-for-l-e9ad84.mp327. AI Post Transformers: Memory Sparse Attention for 100M-Token Scaling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-07-memory-sparse-attention-for-100m-token-s-377cff.mp328. AI Post Transformers: IMO-Bench for Robust Mathematical Reasoning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-imo-bench-for-robust-mathematical-reason-143489.mp329. AI Post Transformers: Simple Self-Distillation for Better Code Generation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-02-simple-self-distillation-for-better-code-cc88e0.mp3Interactive Visualization: Benchmarking Test-Time Scaling for General LLM Agents0 min
- TUMIX Multi-Agent Test-Time Scaling with ToolsThis episode explores TUMIX, a test-time scaling framework that turns a single strong language model into a team of specialized agents with different tool-use strategies, including plain-text reasoning, code execution, search, and hybrids. It explains the paper’s core argument that better reasoning may come not from simply sampling one model more times, but from diversifying computational pathways and letting those agents iteratively refine each other under roughly cost-matched settings. The discussion situates TUMIX within prior work on inference-time compute, program-aided reasoning, and tool-using agents, while also probing whether the approach is genuinely novel or mostly a systems-level formalization of practices already emerging in industry. Listeners would find it interesting for its concrete framing of a major open question in AI: how to orchestrate tools and agent diversity to improve reasoning without exploding latency and cost.Sources:1. TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixture — Yongchao Chen, Jiefeng Chen, Rui Meng, Ji Yin, Na Li, Chuchu Fan, Chi Wang, Tomas Pfister, Jinsung Yoon, 2025http://arxiv.org/abs/2510.012792. PAL: Program-aided Language Models — Luyu Gao, Shafiq Joty, Caiming Xiong, Irwin King, Steven C. H. Hoi, 2022https://scholar.google.com/scholar?q=PAL%3A+Program-aided+Language+Models3. ReAct: Synergizing Reasoning and Acting in Language Models — Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao, 2023https://scholar.google.com/scholar?q=ReAct%3A+Synergizing+Reasoning+and+Acting+in+Language+Models4. Mixture-of-Agents Enhances Large Language Model Capabilities — Chi Wang, Xun Wang, Silvio Savarese, Caiming Xiong, Diyi Yang and collaborators, 2024https://scholar.google.com/scholar?q=Mixture-of-Agents+Enhances+Large+Language+Model+Capabilities5. Search-Augmented Factuality in Language Models: Challenges and Opportunities for Retrieval-Grounded Generation — Representative survey literature; e.g., researchers across academia and industry on retrieval-augmented and search-grounded generation, 2023-2025https://scholar.google.com/scholar?q=Search-Augmented+Factuality+in+Language+Models%3A+Challenges+and+Opportunities+for+Retrieval-Grounded+Generation6. Automatic Prompt Engineer — Tristan Zhou, Shuyan Zhou, Tianyi Zhou, Jacob Andreas, Jason Wei and collaborators, 2022https://scholar.google.com/scholar?q=Automatic+Prompt+Engineer7. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines — Omar Khattab, Keshav Santhanam, and collaborators, 2023https://scholar.google.com/scholar?q=DSPy%3A+Compiling+Declarative+Language+Model+Calls+into+Self-Improving+Pipelines8. TextGrad: Automatic 'Differentiation' via Text — Chandar Lab and collaborators, 2024https://scholar.google.com/scholar?q=TextGrad%3A+Automatic+%27Differentiation%27+via+Text9. ADAS: Automated Design of Agentic Systems — Researchers working on LLM-based workflow and agent search, including recent 2024-2025 agentic-systems optimization efforts, 2024https://scholar.google.com/scholar?q=ADAS%3A+Automated+Design+of+Agentic+Systems10. Self-MoA — Li et al., 2025https://scholar.google.com/scholar?q=Self-MoA11. Symbolic-MoE — Unknown from excerpt, 2025https://scholar.google.com/scholar?q=Symbolic-MoE12. DEI — Unknown from excerpt, 2025https://scholar.google.com/scholar?q=DEI13. SciMaster — Unknown from excerpt, 2025https://scholar.google.com/scholar?q=SciMaster14. GSA — Unknown from excerpt, 2025https://scholar.google.com/scholar?q=GSA15. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — Brown et al., 2024https://scholar.google.com/scholar?q=Scaling+LLM+Test-Time+Compute+Optimally+can+be+More+Effective+than+Scaling+Model+Parameters16. Language Models Can Solve Computer Tasks — Madaan et al., 2022https://scholar.google.com/scholar?q=Language+Models+Can+Solve+Computer+Tasks17. Program-of-Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks — Chen et al., 2022https://scholar.google.com/scholar?q=Program-of-Thoughts+Prompting%3A+Disentangling+Computation+from+Reasoning+for+Numerical+Reasoning+Tasks18. Humanity's Last Exam — Phan et al., 2025https://scholar.google.com/scholar?q=Humanity%27s+Last+Exam19. GPQA: A Graduate-Level Google-Proof Q&A Benchmark — Rein et al., 2024https://scholar.google.com/scholar?q=GPQA%3A+A+Graduate-Level+Google-Proof+Q%26A+Benchmark20. OpenAI/Gemini Deep Research comparison paper or report — Comanici et al., 2025https://scholar.google.com/scholar?q=OpenAI%2FGemini+Deep+Research+comparison+paper+or+report21. DeepSeek-R1 or related RL reasoning paper — Guo et al., 2025https://scholar.google.com/scholar?q=DeepSeek-R1+or+related+RL+reasoning+paper22. Recent work showing Code Interpreter underuse in OpenAI models — Chen et al., 2024https://scholar.google.com/scholar?q=Recent+work+showing+Code+Interpreter+underuse+in+OpenAI+models23. Simple Test-Time Scaling — unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Simple+Test-Time+Scaling24. Faster and Better LLMs via Latency-Aware Test-Time Scaling — unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Faster+and+Better+LLMs+via+Latency-Aware+Test-Time+Scaling25. Thought Calibration: Efficient and Confident Test-Time Scaling — unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Thought+Calibration%3A+Efficient+and+Confident+Test-Time+Scaling26. Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling — unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Reasoning+Aware+Self-Consistency%3A+Leveraging+Reasoning+Paths+for+Efficient+LLM+Sampling27. Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer Reasoning — unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Latent+Self-Consistency+for+Reliable+Majority-Set+Selection+in+Short-+and+Long-Answer+Reasoning28. Universal Self-Consistency for Large Language Model Generation — unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Universal+Self-Consistency+for+Large+Language+Model+Generation29. The Hidden Strength of Disagreement: Unraveling the Consensus-Diversity Tradeoff in Adaptive Multi-Agent Systems — unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=The+Hidden+Strength+of+Disagreement%3A+Unraveling+the+Consensus-Diversity+Tradeoff+in+Adaptive+Multi-Agent+Systems30. Stay Focused: Problem Drift in Multi-Agent Debate — unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Stay+Focused%3A+Problem+Drift+in+Multi-Agent+Debate31. Why Do Multi-Agent LLM Systems Fail? — unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Why+Do+Multi-Agent+LLM+Systems+Fail%3F32. LLM-Based Agents for Tool Learning: A Survey — W. Xu et al., 2024/2025https://scholar.google.com/scholar?q=LLM-Based+Agents+for+Tool+Learning%3A+A+Survey33. AI Post Transformers: Multiagent Debate Improves Language Model Reasoning — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/multiagent-debate-improves-language-model-reasoning/34. AI Post Transformers: Experimental Comparison of Agentic and Enhanced RAG — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-experimental-comparison-of-agentic-and-e-37d8bc.mp335. AI Post Transformers: MEMSEARCHER: Reinforcement Learning for LLM Memory Management — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-memsearcher-reinforcement-learning-for-l-e9ad84.mp336. AI Post Transformers: Generalist Reward Modeling with Inference-Time Scaling — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/generalist-reward-modeling-with-inference-time-scaling/Interactive Visualization: TUMIX Multi-Agent Test-Time Scaling with Tools0 min
- Test-time Scaling for Multi-Agent Collaborative ReasoningThis episode explores whether multi-agent systems can benefit from test-time scaling in the same way single models do, focusing on a 2025 paper that combines learned collaborative reasoning with runtime orchestration. It explains the paper’s core setup: a model trained on 500 carefully curated multi-agent reasoning traces (M500) and a separate “CEO” controller that coordinates specialized agents such as planners, critics, and verifiers. The discussion highlights the paper’s central argument that stronger performance may require both better reasoning models and better coordination policies, while also questioning whether the gains justify the added complexity and compute compared with simpler single-agent approaches. Listeners would find it interesting for its clear breakdown of a major emerging AI debate: when collaboration between models is genuinely useful, and when it becomes an expensive “group project” with little payoff.Sources:1. Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoning — Can Jin, Hongwu Peng, Qixin Zhang, Yujin Tang, Dimitris N. Metaxas, Tong Che, 2025http://arxiv.org/abs/2504.097722. AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors — Guo et al., 2023https://scholar.google.com/scholar?q=AgentVerse%3A+Facilitating+Multi-Agent+Collaboration+and+Exploring+Emergent+Behaviors3. DeepSeek-R1 — DeepSeek-AI et al., 2025https://scholar.google.com/scholar?q=DeepSeek-R14. MATH-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations — Wang et al., 2024https://scholar.google.com/scholar?q=MATH-Shepherd%3A+Verify+and+Reinforce+LLMs+Step-by-step+without+Human+Annotations5. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Wang et al., 2023https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models6. Tree of Thoughts: Deliberate Problem Solving with Large Language Models — Yao et al., 2023https://scholar.google.com/scholar?q=Tree+of+Thoughts%3A+Deliberate+Problem+Solving+with+Large+Language+Models7. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation — Wu et al., 2023https://scholar.google.com/scholar?q=AutoGen%3A+Enabling+Next-Gen+LLM+Applications+via+Multi-Agent+Conversation8. CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society — Li et al., 2023https://scholar.google.com/scholar?q=CAMEL%3A+Communicative+Agents+for+%22Mind%22+Exploration+of+Large+Language+Model+Society9. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework — Hong et al., 2024https://scholar.google.com/scholar?q=MetaGPT%3A+Meta+Programming+for+A+Multi-Agent+Collaborative+Framework10. The Agent Company — Xu et al., 2024https://scholar.google.com/scholar?q=The+Agent+Company11. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering — Yang et al., 2024https://scholar.google.com/scholar?q=SWE-agent%3A+Agent-Computer+Interfaces+Enable+Automated+Software+Engineering12. Benchmark Test-Time Scaling of General LLM Agents — unknown from snippet, 2025https://scholar.google.com/scholar?q=Benchmark+Test-Time+Scaling+of+General+LLM+Agents13. Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Parameters for Reasoning — unknown from snippet, 2024/2025https://scholar.google.com/scholar?q=Scaling+LLM+Test-Time+Compute+Optimally+Can+Be+More+Effective+Than+Scaling+Parameters+for+Reasoning14. CONSENSAGENT: Towards Efficient and Effective Consensus in Multi-Agent LLM Interactions Through Sycophancy Mitigation — unknown from snippet, 2025https://scholar.google.com/scholar?q=CONSENSAGENT%3A+Towards+Efficient+and+Effective+Consensus+in+Multi-Agent+LLM+Interactions+Through+Sycophancy+Mitigation15. LLM-Based Multi-agent Systems: Frameworks, Evaluation, Open Challenges, and Research Frontiers — unknown from snippet, 2024/2025https://scholar.google.com/scholar?q=LLM-Based+Multi-agent+Systems%3A+Frameworks%2C+Evaluation%2C+Open+Challenges%2C+and+Research+Frontiers16. Multi-agent Coordination Across Diverse Applications: A Survey — unknown from snippet, 2024/2025https://scholar.google.com/scholar?q=Multi-agent+Coordination+Across+Diverse+Applications%3A+A+Survey17. Decentralized Multi-Agent Goal Assignment for Path Planning Using Large Language Models — unknown from snippet, 2024/2025https://scholar.google.com/scholar?q=Decentralized+Multi-Agent+Goal+Assignment+for+Path+Planning+Using+Large+Language+Models18. AI Post Transformers: Agentic AI and the Next Intelligence Explosion — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-28-agentic-ai-and-the-next-intelligence-exp-d06561.mp319. AI Post Transformers: MetaScale: Test-Time Scaling with Evolving Meta-Thoughts — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/metascale-test-time-scaling-with-evolving-meta-thoughts/20. AI Post Transformers: Simple Self-Distillation for Better Code Generation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-02-simple-self-distillation-for-better-code-cc88e0.mp321. AI Post Transformers: Generalist Reward Modeling with Inference-Time Scaling — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/generalist-reward-modeling-with-inference-time-scaling/22. AI Post Transformers: Nemotron 3 Super Hybrid Mamba-Transformer MoE — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-nemotron-3-super-hybrid-mamba-transforme-31ac75.mp323. AI Post Transformers: SkillsBench for Evaluating Agent Skills — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-skillsbench-for-evaluating-agent-skills-58bb1e.mp3Interactive Visualization: Test-time Scaling for Multi-Agent Collaborative Reasoning0 min
- Program Synthesis with Large Language ModelsThis episode explores a 2021 Google Research paper on whether large language models can synthesize short Python programs directly from natural-language descriptions, moving beyond code autocomplete into true program synthesis. It explains why this is difficult in general-purpose languages, contrasts classical search-based synthesis with transformer-based generation, and highlights the paper’s emphasis on execution-based evaluation, where code must actually run and pass tests rather than merely resemble reference solutions. The discussion covers the MBPP and MathQA-Python benchmarks, the effects of model scale from 244 million to 137 billion parameters, and the finding that larger models improve substantially, with the biggest model solving 59.6% of MBPP in a few-shot setting and fine-tuning on just 374 examples adding roughly 10 points. Listeners would find it interesting for its clear look at an early turning point when code LLMs began to show measurable, testable synthesis ability rather than just fluent code-like text.Sources:1. Program Synthesis with Large Language Models — Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, Charles Sutton, 2021http://arxiv.org/abs/2108.077322. Program Synthesis — Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, 2017https://scholar.google.com/scholar?q=Program+Synthesis3. Neural Program Synthesis: A Survey — Michele Vallecorsa, Luca Quartana, Luca Pasquale and others, 2022https://scholar.google.com/scholar?q=Neural+Program+Synthesis%3A+A+Survey4. Program Synthesis with Large Language Models — Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, Charles Sutton, 2021https://scholar.google.com/scholar?q=Program+Synthesis+with+Large+Language+Models5. A Survey on Neural Code Intelligence: From Program Representation to Program Synthesis — Uri Alon, Miltiadis Allamanis, Marc Brockschmidt and others, 2024https://scholar.google.com/scholar?q=A+Survey+on+Neural+Code+Intelligence%3A+From+Program+Representation+to+Program+Synthesis6. Evaluating Large Language Models Trained on Code — Mark Chen, Jerry Tworek, Heewoo Jun, et al., 2021https://scholar.google.com/scholar?q=Evaluating+Large+Language+Models+Trained+on+Code7. Language Models are Few-Shot Learners — Tom B. Brown, Benjamin Mann, Nick Ryder, et al., 2020https://scholar.google.com/scholar?q=Language+Models+are+Few-Shot+Learners8. CuBERT: BERT Models for Python Source Code Understanding — Rahul Kanade, Petros Maniatis, Gogul Balakrishnan, Kensen Shi, 2020https://scholar.google.com/scholar?q=CuBERT%3A+BERT+Models+for+Python+Source+Code+Understanding9. CodeBERT: A Pre-Trained Model for Programming and Natural Languages — Zhangyin Feng, Daya Guo, Duyu Tang, et al., 2020https://scholar.google.com/scholar?q=CodeBERT%3A+A+Pre-Trained+Model+for+Programming+and+Natural+Languages10. PyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers — Colin Clement, Dawn Drain, Aakanksha S. Bhatia, et al., 2020https://scholar.google.com/scholar?q=PyMT5%3A+Multi-mode+Translation+of+Natural+Language+and+Python+Code+with+Transformers11. DeepCoder: Learning to Write Programs — Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, et al., 2017https://scholar.google.com/scholar?q=DeepCoder%3A+Learning+to+Write+Programs12. RobustFill: Neural Program Learning under Noisy I/O — Rishabh Singh, Abhishek Gulwani, 2017https://scholar.google.com/scholar?q=RobustFill%3A+Neural+Program+Learning+under+Noisy+I%2FO13. DreamCoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning — Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Josh Tenenbaum, Armando Solar-Lezama, 2021https://scholar.google.com/scholar?q=DreamCoder%3A+Bootstrapping+Inductive+Program+Synthesis+with+Wake-Sleep+Library+Learning14. Learning to Infer Graphics Programs from Hand-Drawn Images — Augustus Odena, Charles Sutton, 2020https://scholar.google.com/scholar?q=Learning+to+Infer+Graphics+Programs+from+Hand-Drawn+Images15. MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms — Aida Amini, Saeideh Bakhshi, Sivan Ray Choi, et al., 2019https://scholar.google.com/scholar?q=MathQA%3A+Towards+Interpretable+Math+Word+Problem+Solving+with+Operation-Based+Formalisms16. Allamanis et al. 2018 Survey on Machine Learning for Code — Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, Charles Sutton, 2018https://scholar.google.com/scholar?q=Allamanis+et+al.+2018+Survey+on+Machine+Learning+for+Code17. Chain-of-Code: Reasoning with a Language Model-Augmented Code Emulator — Li et al. (approx.), 2024https://scholar.google.com/scholar?q=Chain-of-Code%3A+Reasoning+with+a+Language+Model-Augmented+Code+Emulator18. OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement — Zhang et al. (approx.), 2024https://scholar.google.com/scholar?q=OpenCodeInterpreter%3A+Integrating+Code+Generation+with+Execution+and+Refinement19. CodePRM: Execution Feedback-Enhanced Process Reward Model for Code Generation — Wang et al. (approx.), 2024https://scholar.google.com/scholar?q=CodePRM%3A+Execution+Feedback-Enhanced+Process+Reward+Model+for+Code+Generation20. CodeMonkeys: Scaling Test-Time Compute for Software Engineering — anonymous/uncertain from snippet, 2024 or 2025https://scholar.google.com/scholar?q=CodeMonkeys%3A+Scaling+Test-Time+Compute+for+Software+Engineering21. AI Post Transformers: CODEGEN: Open Language Model for Code Synthesis — Hal Turing & Dr. Ada Shannon, Fri,https://podcast.do-not-panic.com/episodes/codegen-open-language-model-for-code-synthesis/22. AI Post Transformers: Simple Self-Distillation for Better Code Generation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-02-simple-self-distillation-for-better-code-cc88e0.mp323. AI Post Transformers: CWM: Code Generation with World Models — Hal Turing & Dr. Ada Shannon, Sat,https://podcast.do-not-panic.com/episodes/cwm-code-generation-with-world-models/24. AI Post Transformers: CodeI/O: Reasoning Patterns Through Code Input-Output Prediction — Hal Turing & Dr. Ada Shannon, Tue,https://podcast.do-not-panic.com/episodes/codeio-reasoning-patterns-through-code-input-output-prediction/Interactive Visualization: Program Synthesis with Large Language Models0 min
- DreamerV3 World Models Across 150 TasksThis episode explores DreamerV3, a world-model reinforcement learning system that claims to use one main configuration across more than 150 tasks spanning Atari, ProcGen, DMLab, robot control, visual control, BSuite, and Minecraft. It explains how world models work—learning compact environment dynamics so an agent can train on imagined futures—and why that approach is appealing for sample efficiency but historically difficult because agents can overfit to inaccurate “fantasy” dynamics. The discussion highlights the paper’s central argument that robust world-model design may reduce the need for domain-specific retuning, while also stressing that “fixed hyperparameters” does not eliminate all domain engineering such as wrappers, action discretization, and evaluation choices. Listeners would find it interesting for its clear look at a major RL unification attempt, including why the results matter for scaling, sparse-reward tasks, and expensive real-world settings like robotics.Sources:1. Mastering Diverse Domains through World Models — Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, Timothy Lillicrap, 2023http://arxiv.org/abs/2301.041042. Mastering Atari with Discrete World Models — Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, Jimmy Ba, 2021https://scholar.google.com/scholar?q=Mastering+Atari+with+Discrete+World+Models3. Mastering Visual Continuous Control: Improved Data-Efficient Reinforcement Learning with Dreamer — Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, James Davidson, 2020https://scholar.google.com/scholar?q=Mastering+Visual+Continuous+Control%3A+Improved+Data-Efficient+Reinforcement+Learning+with+Dreamer4. Learning Latent Dynamics for Planning from Pixels — Danijar Hafner, Timothy Lillicrap, Jimmy Ba, Mohammad Norouzi, 2019https://scholar.google.com/scholar?q=Learning+Latent+Dynamics+for+Planning+from+Pixels5. MuZero — Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, et al., 2020https://scholar.google.com/scholar?q=MuZero6. IRIS: Efficient Video Pretraining for Reinforcement Learning — various authors as cited by the paper, 2023https://scholar.google.com/scholar?q=IRIS%3A+Efficient+Video+Pretraining+for+Reinforcement+Learning7. Temporal Difference Models / TD-MPC / TD-MPC2 — various authors including Nicklas Hansen and colleagues, 2022-2024https://scholar.google.com/scholar?q=Temporal+Difference+Models+%2F+TD-MPC+%2F+TD-MPC28. MineRL BASALT / VPT-related Minecraft works — various authors including OpenAI and MineRL participants, 2021-2022https://scholar.google.com/scholar?q=MineRL+BASALT+%2F+VPT-related+Minecraft+works9. DrQ-v2 — Ilya Kostrikov, Denis Yarats, Rob Fergus, 2021https://scholar.google.com/scholar?q=DrQ-v210. R2D2 — Steven Kapturowski, Georg Ostrovski, John Quan, et al., 2019https://scholar.google.com/scholar?q=R2D211. STORM: Efficient Stochastic Transformer-based World Models for Reinforcement Learning — approx. Guo et al., 2023/2024https://scholar.google.com/scholar?q=STORM%3A+Efficient+Stochastic+Transformer-based+World+Models+for+Reinforcement+Learning12. Improving Transformer World Models for Data-Efficient RL — approx. recent 2023/2024 RL world-model authors, 2023/2024https://scholar.google.com/scholar?q=Improving+Transformer+World+Models+for+Data-Efficient+RL13. GIRL: Generative Imagination Reinforcement Learning via Information-Theoretic Hallucination Control — approx. recent MBRL authors, 2024/2025https://scholar.google.com/scholar?q=GIRL%3A+Generative+Imagination+Reinforcement+Learning+via+Information-Theoretic+Hallucination+Control14. Normalization Enhances Generalization in Visual Reinforcement Learning — approx. recent visual RL authors, 2024/2025https://scholar.google.com/scholar?q=Normalization+Enhances+Generalization+in+Visual+Reinforcement+Learning15. Understanding the Mechanisms of Fast Hyperparameter Transfer — approx. recent hyperparameter-transfer authors, 2024/2025https://scholar.google.com/scholar?q=Understanding+the+Mechanisms+of+Fast+Hyperparameter+Transfer16. Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration — approx. recent hyperparameter-transfer authors, 2024/2025https://scholar.google.com/scholar?q=Completed+Hyperparameter+Transfer+across+Modules%2C+Width%2C+Depth%2C+Batch+and+Duration17. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp318. AI Post Transformers: Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contexts — Hal Turing & Dr. Ada Shannon, Tue,https://podcast.do-not-panic.com/episodes/zero-shot-context-generalization-in-reinforcement-learning-from-few-training-con/19. AI Post Transformers: Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning — Hal Turing & Dr. Ada Shannon, Fri,https://podcast.do-not-panic.com/episodes/contrastive-behavioral-similarity-embeddings-for-generalization-in-reinforcement/20. AI Post Transformers: HyperController: Fast, Stable Reinforcement Learning Hyperparameter Optimization — Hal Turing & Dr. Ada Shannon, Fri,https://podcast.do-not-panic.com/episodes/hypercontroller-fast-stable-reinforcement-learning-hyperparameter-optimization/Interactive Visualization: DreamerV3 World Models Across 150 Tasks0 min
- Directly Trained Spiking DQNs for AtariThis episode explores a 2023 paper on Deep Spiking Q-Networks, asking whether a directly trained spiking version of DQN can compete with earlier conversion-based spiking reinforcement learning methods on Atari while retaining the energy-efficiency promise of spiking neural networks. It explains the technical foundations behind spiking networks, including leaky integrate-and-fire neurons, surrogate-gradient training, and why SNNs remain difficult to train and awkward on conventional GPU hardware despite their appeal for neuromorphic chips like TrueNorth and Loihi. The discussion also situates the paper against the legacy of the original DeepMind DQN work, arguing that the paper’s title deliberately invites scrutiny over whether it truly matches the breadth and ambition of the classic Atari benchmark. Listeners would find it interesting for its clear framing of both the hype and the hard practical questions around neuromorphic AI: not just whether spiking RL works, but where, on what hardware, and under what conditions its efficiency claims actually matter.Sources:1. Human-Level Control through Directly-Trained Deep Spiking Q-Networks — Guisong Liu, Wenjie Deng, Xiurui Xie, Li Huang, Huajin Tang, 2021http://arxiv.org/abs/2201.072112. Spiking Neural Networks for Machine Learning: An Overview — Wolfgang Maass and others; overview literature includes major contributors such as Thomas Pfeil, Emre Neftci, and Surya Ganguli across the field, Recent overview genre, especially 2023https://scholar.google.com/scholar?q=Spiking+Neural+Networks+for+Machine+Learning%3A+An+Overview3. Training Spiking Neural Networks Using Lessons From Deep Learning — Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, Wolfgang Maass, 2018https://scholar.google.com/scholar?q=Training+Spiking+Neural+Networks+Using+Lessons+From+Deep+Learning4. Spiking Neural Networks in the Fourth Generation of Artificial Intelligence — Zhaofei Yu, Hanle Zheng, Yujie Wu, and others, 2023https://scholar.google.com/scholar?q=Spiking+Neural+Networks+in+the+Fourth+Generation+of+Artificial+Intelligence5. The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks — Friedemann Zenke, Tim Vogels, 2021https://scholar.google.com/scholar?q=The+Remarkable+Robustness+of+Surrogate+Gradient+Learning+for+Instilling+Complex+Function+in+Spiking+Neural+Networks6. Human-level control through deep reinforcement learning — Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis, 2015https://scholar.google.com/scholar?q=Human-level+control+through+deep+reinforcement+learning7. Asynchronous Methods for Deep Reinforcement Learning — Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy Lillicrap, David Silver, Koray Kavukcuoglu, 2016https://scholar.google.com/scholar?q=Asynchronous+Methods+for+Deep+Reinforcement+Learning8. Deep Reinforcement Learning: An Overview — Yuxi Li, 2017https://scholar.google.com/scholar?q=Deep+Reinforcement+Learning%3A+An+Overview9. Reinforcement Learning: An Introduction — Richard S. Sutton, Andrew G. Barto, 1998; 2nd edition 2018https://scholar.google.com/scholar?q=Reinforcement+Learning%3A+An+Introduction10. Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks — Emre O. Neftci, Hesham Mostafa, Friedemann Zenke, 2019https://scholar.google.com/scholar?q=Surrogate+Gradient+Learning+in+Spiking+Neural+Networks%3A+Bringing+the+Power+of+Gradient-Based+Optimization+to+Spiking+Neural+Networks11. Direct Training for Spiking Neural Networks: Faster, Larger, Better — Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi, 2019https://scholar.google.com/scholar?q=Direct+Training+for+Spiking+Neural+Networks%3A+Faster%2C+Larger%2C+Better12. Going Deeper With Directly-Trained Larger Spiking Neural Networks — Chaoteng Duan, Shikuang Deng, Xingting Wang, Meng Zhang, and others, 2022https://scholar.google.com/scholar?q=Going+Deeper+With+Directly-Trained+Larger+Spiking+Neural+Networks13. Threshold-Dependent Batch Normalization for Training Deep Spiking Neural Networks — Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi, 2021https://scholar.google.com/scholar?q=Threshold-Dependent+Batch+Normalization+for+Training+Deep+Spiking+Neural+Networks14. A million spiking-neuron integrated circuit with a scalable communication network and interface — Paul A. Merolla, John V. Arthur, Rodrigo Alvarez-Icaza, Andrew S. Cassidy, Jun Sawada, Filipp Akopyan, Bryan L. Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, Bernard Brezzo, Ivan Vo, Steven Esser, Rathinakumar Appuswamy, Brian Taba, Arnon Amir, Myron Flickner, William Risk, Rajit Manohar, Dharmendra Modha, 2014https://scholar.google.com/scholar?q=A+million+spiking-neuron+integrated+circuit+with+a+scalable+communication+network+and+interface15. Loihi: A Neuromorphic Manycore Processor with On-Chip Learning — Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al., 2018https://scholar.google.com/scholar?q=Loihi%3A+A+Neuromorphic+Manycore+Processor+with+On-Chip+Learning16. SpiNNaker: A 1-W 18-Core System-on-Chip for Massively-Parallel Neural Network Simulation — Steve B. Furber, Francesco Galluppi, Steve Temple, Luis A. Plana, 2014https://scholar.google.com/scholar?q=SpiNNaker%3A+A+1-W+18-Core+System-on-Chip+for+Massively-Parallel+Neural+Network+Simulation17. Benchmarking Neuromorphic Systems with Nengo — Terry C. Stewart, Dan Rasmussen, Xuan Choo, Aaron Voelker, and others, 2015-2017 era benchmarking workhttps://scholar.google.com/scholar?q=Benchmarking+Neuromorphic+Systems+with+Nengo18. Playing Atari with Deep Reinforcement Learning — Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller, 2013https://scholar.google.com/scholar?q=Playing+Atari+with+Deep+Reinforcement+Learning19. Deep Reinforcement Learning with Double Q-learning — Hado van Hasselt, Arthur Guez, David Silver, 2016https://scholar.google.com/scholar?q=Deep+Reinforcement+Learning+with+Double+Q-learning20. Rainbow: Combining Improvements in Deep Reinforcement Learning — Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver, 2018https://scholar.google.com/scholar?q=Rainbow%3A+Combining+Improvements+in+Deep+Reinforcement+Learning21. Enabling Deep Spiking Neural Networks for Reinforcement Learning — Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, Kaushik Roy, 2020https://scholar.google.com/scholar?q=Enabling+Deep+Spiking+Neural+Networks+for+Reinforcement+Learning22. Going Deeper in Spiking Neural Networks: VGG and Residual Architectures — Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, Kaushik Roy, 2021https://scholar.google.com/scholar?q=Going+Deeper+in+Spiking+Neural+Networks%3A+VGG+and+Residual+Architectures23. Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks — Yuhang Fang, Zhaofei Yu, Tielin Zhang, et al., 2021https://scholar.google.com/scholar?q=Incorporating+Learnable+Membrane+Time+Constant+to+Enhance+Learning+of+Spiking+Neural+Networks24. Deep Residual Learning in Spiking Neural Networks — Yujie Wu, Yuhang Zhao, et al., 2021https://scholar.google.com/scholar?q=Deep+Residual+Learning+in+Spiking+Neural+Networks25. A Unified Optimization Framework of ANN-SNN Conversion: Towards Optimal Mapping from Activation Values to Firing Rates — approx. recent ANN-to-SNN conversion literature, 2023-2024https://scholar.google.com/scholar?q=A+Unified+Optimization+Framework+of+ANN-SNN+Conversion%3A+Towards+Optimal+Mapping+from+Activation+Values+to+Firing+Rates26. Towards High-Performance Spiking Transformers from ANN to SNN Conversion — approx. recent conversion/transformer authors, 2024https://scholar.google.com/scholar?q=Towards+High-Performance+Spiking+Transformers+from+ANN+to+SNN+Conversion27. Towards Training-Free and Accurate ANN-to-SNN Conversion via Activation-Aware Redistribution — approx. recent ANN-to-SNN conversion authors, 2024https://scholar.google.com/scholar?q=Towards+Training-Free+and+Accurate+ANN-to-SNN+Conversion+via+Activation-Aware+Redistribution28. Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks — approx. recent SNN RL authors, 2024-2025https://scholar.google.com/scholar?q=Adaptive+Surrogate+Gradients+for+Sequential+Reinforcement+Learning+in+Spiking+Neural+Networks29. Elucidating the Theoretical Underpinnings of Surrogate Gradient Learning in Spiking Neural Networks — approx. recent theoretical SNN authors, 2023-2024https://scholar.google.com/scholar?q=Elucidating+the+Theoretical+Underpinnings+of+Surrogate+Gradient+Learning+in+Spiking+Neural+Networks30. Spiking Reinforcement Learning Enhanced by Bioinspired Event Source of Multi-Dendrite Spiking Neuron and Dynamic Thresholds — approx. recent spiking RL authors, 2024-2025https://scholar.google.com/scholar?q=Spiking+Reinforcement+Learning+Enhanced+by+Bioinspired+Event+Source+of+Multi-Dendrite+Spiking+Neuron+and+Dynamic+Thresholds31. S2Act: Simple Spiking Actor — approx. recent spiking actor-critic authors, 2024-2025https://scholar.google.com/scholar?q=S2Act%3A+Simple+Spiking+Actor32. AI Post Transformers: Zero-Shot Context Generalization in ReinforcementLearning from Few Training Contexts — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/zero-shot-context-generalization-in-reinforcement-learning-from-few-training-con/33. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp3Interactive Visualization: Directly Trained Spiking DQNs for Atari0 min
- RetrievalAttention for Long-Context LLM InferenceThis episode explores RetrievalAttention, a 2024 paper that tries to make long-context LLM inference much cheaper by retrieving only the most relevant key-value cache entries during decoding instead of scanning the entire history every time. It explains why long-context serving is bottlenecked less by raw FLOPs than by memory traffic and KV-cache growth, citing concrete figures such as roughly 125 GB of KV cache per million tokens for Llama-3-8B and decoding latency that balloons from 32.8 seconds at 128K tokens to 1,765 seconds at 1M. The discussion argues that attention is dynamically sparse in practice, but also emphasizes a key technical caveat: standard vector search is not automatically a good proxy for attention lookup, so retrieval-based sparsity has to be designed carefully. Listeners would find it interesting because it connects transformer modeling, systems bottlenecks, and vector retrieval into a practical strategy for making million-token context windows more usable in real deployments.Sources:1. RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval — Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, Chen Chen, Fan Yang, Yuqing Yang, Lili Qiu, 2024http://arxiv.org/abs/2409.105162. StreamingLLM — Xiao et al., 2024https://scholar.google.com/scholar?q=StreamingLLM3. SnapKV — Li et al. or related 2024 sparse/compact KV-cache work cited in the paper's framing, 2024https://scholar.google.com/scholar?q=SnapKV4. InfLLM — Xiao et al. / related long-context inference work cited by the paper, 2024https://scholar.google.com/scholar?q=InfLLM5. Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs — Malkov and Yashunin, 2018https://scholar.google.com/scholar?q=Efficient+and+robust+approximate+nearest+neighbor+search+using+Hierarchical+Navigable+Small+World+graphs6. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — Katharopoulos et al., 2020https://scholar.google.com/scholar?q=Transformers+are+RNNs%3A+Fast+Autoregressive+Transformers+with+Linear+Attention7. H2O / Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — likely cited among the 2024 heuristic sparse-attention/KV methods, 2023https://scholar.google.com/scholar?q=H2O+%2F+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models8. FlexGen — Sheng et al., 2023https://scholar.google.com/scholar?q=FlexGen9. The Needle in a Haystack test / long-context retrieval benchmark family — various, 2023-2024https://scholar.google.com/scholar?q=The+Needle+in+a+Haystack+test+%2F+long-context+retrieval+benchmark+family10. KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long-Context Capable Approaches — approx. recent benchmark paper, authors unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=KV+Cache+Compression%2C+But+What+Must+We+Give+in+Return%3F+A+Comprehensive+Benchmark+of+Long-Context+Capable+Approaches11. RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression — approx. RocketKV authors, unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=RocketKV%3A+Accelerating+Long-Context+LLM+Inference+via+Two-Stage+KV+Cache+Compression12. Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks — approx. adaptive KV merging authors, unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Model+Tells+You+Where+to+Merge%3A+Adaptive+KV+Cache+Merging+for+LLMs+on+Long-Context+Tasks13. Efficient Low Rank Attention for Long-Context Inference in Large Language Models — approx. LRQK / low-rank attention authors, unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=Efficient+Low+Rank+Attention+for+Long-Context+Inference+in+Large+Language+Models14. KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation — approx. KVPR authors, unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=KVPR%3A+Efficient+LLM+Inference+with+I%2FO-Aware+KV+Cache+Partial+Recomputation15. ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference — approx. ScoutAttention authors, unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=ScoutAttention%3A+Efficient+KV+Cache+Offloading+via+Layer-Ahead+CPU+Pre-computation+for+LLM+Inference16. DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads — approx. DuoAttention authors, unclear from snippet, 2024/2025https://scholar.google.com/scholar?q=DuoAttention%3A+Efficient+Long-Context+LLM+Inference+with+Retrieval+and+Streaming+Heads17. AI Post Transformers: LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-lookaheadkv-fast-and-accurate-kv-9cfc9f.mp318. AI Post Transformers: Lookahead Q-Cache for Consistent KV Eviction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-lookahead-q-cache-for-consistent-kv-evic-d97b09.mp319. AI Post Transformers: KVSwap for Disk-Aware Long-Context On-Device Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-16-kvswap-for-disk-aware-long-context-on-de-f3c15e.mp320. AI Post Transformers: SolidAttention: Co-Designing Sparse Attention and SSD I/O — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-18-solidattention-co-designing-sparse-atten-5a8622.mp321. AI Post Transformers: TriAttention for Efficient Long-Context KV Compression — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-07-triattention-for-efficient-long-context-6c08ee.mp322. AI Post Transformers: Memory Sparse Attention for 100M-Token Scaling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-07-memory-sparse-attention-for-100m-token-s-377cff.mp323. AI Post Transformers: GPU-Accelerated Dynamic Quantized ANNS Graph Search — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-gpu-accelerated-dynamic-quantized-anns-g-f2cd4e.mp324. AI Post Transformers: TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-turboquant-online-vector-quantiz-1967b7.mp325. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp326. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp327. AI Post Transformers: FlexGen: High-Throughput LLM Inference on a Single GPU — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/flexgen-high-throughput-llm-inference-on-a-single-gpu/Interactive Visualization: RetrievalAttention for Long-Context LLM Inference19 min
- Qwen3.5-Omni Thinker-Talker for Omnimodal StreamingThis episode explores the Qwen3.5-Omni technical report as a significant update in omnimodal AI: a system designed to understand and generate text, audio, images, and video within one architecture. It unpacks the model’s Thinker–Talker design, arguing that separating multimodal reasoning from real-time output is especially important for speech, where latency and timing make generation far harder than standard text responses. The discussion also examines why the report leans on Mixture-of-Experts and hybrid attention instead of a plain dense transformer, highlighting the tradeoff between longer context and greater capacity versus routing complexity, infrastructure overhead, and serving difficulty. Listeners would find it interesting for its clear explanation of why claims like 256k context and stable low-latency streaming speech are technically ambitious—and why flashy multimodal demos often hide hard systems problems underneath.Sources:1. Qwen3.5-Omni Technical Report — Qwen Team, 2026http://arxiv.org/abs/2604.158042. A Survey on Text-to-Speech Synthesis — Heiga Zen, Keiichi Tokuda, Alan W. Black, 2009https://scholar.google.com/scholar?q=A+Survey+on+Text-to-Speech+Synthesis3. Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions — Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, et al., 2018https://scholar.google.com/scholar?q=Natural+TTS+Synthesis+by+Conditioning+WaveNet+on+Mel+Spectrogram+Predictions4. FastSpeech: Fast, Robust and Controllable Text to Speech — Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu, 2019https://scholar.google.com/scholar?q=FastSpeech%3A+Fast%2C+Robust+and+Controllable+Text+to+Speech5. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers — Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, et al., 2023https://scholar.google.com/scholar?q=Neural+Codec+Language+Models+are+Zero-Shot+Text+to+Speech+Synthesizers6. Qwen2.5-Omni Technical Report — Xu et al., 2025https://scholar.google.com/scholar?q=Qwen2.5-Omni+Technical+Report7. Qwen3-Omni Technical Report — Xu et al., 2025https://scholar.google.com/scholar?q=Qwen3-Omni+Technical+Report8. Attention Is All You Need — Vaswani et al., 2017https://scholar.google.com/scholar?q=Attention+Is+All+You+Need9. Language Models are Few-Shot Learners — Brown et al., 2020https://scholar.google.com/scholar?q=Language+Models+are+Few-Shot+Learners10. GPT-4 Technical Report / GPT-4-class system references in the report — OpenAI, 2023https://scholar.google.com/scholar?q=GPT-4+Technical+Report+%2F+GPT-4-class+system+references+in+the+report11. Gemini technical reports referenced as Gemini Team (2024) — Gemini Team, 2024https://scholar.google.com/scholar?q=Gemini+technical+reports+referenced+as+Gemini+Team+%282024%2912. Audio language model / omni-audio model references cited as Chu et al. — Chu et al., 2023-2024https://scholar.google.com/scholar?q=Audio+language+model+%2F+omni-audio+model+references+cited+as+Chu+et+al.13. Recent native omnimodal system references cited as OpenAI (2024), Comanici et al. (2025), Xu et al. (2025a;b) — OpenAI; Comanici et al.; Xu et al., 2024-2025https://scholar.google.com/scholar?q=Recent+native+omnimodal+system+references+cited+as+OpenAI+%282024%29%2C+Comanici+et+al.+%282025%29%2C+Xu+et+al.+%282025a%3Bb%2914. TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling — approx. recent speech/SLM authors, 2024/2025https://scholar.google.com/scholar?q=TASTE%3A+Text-Aligned+Speech+Tokenization+and+Embedding+for+Spoken+Language+Modeling15. dMel: Speech Tokenization Made Simple — approx. recent speech tokenizer authors, 2024/2025https://scholar.google.com/scholar?q=dMel%3A+Speech+Tokenization+Made+Simple16. TaDiCodec: Text-Aware Diffusion Speech Tokenizer for Speech Language Modeling — approx. recent codec/SLM authors, 2024/2025https://scholar.google.com/scholar?q=TaDiCodec%3A+Text-Aware+Diffusion+Speech+Tokenizer+for+Speech+Language+Modeling17. DC-Spin: A Speaker-Invariant Speech Tokenizer for Spoken Language Models — approx. recent spoken language model authors, 2024/2025https://scholar.google.com/scholar?q=DC-Spin%3A+A+Speaker-Invariant+Speech+Tokenizer+for+Spoken+Language+Models18. HyperAttention: Long-Context Attention in Near-Linear Time — approx. recent long-context attention authors, 2024/2025https://scholar.google.com/scholar?q=HyperAttention%3A+Long-Context+Attention+in+Near-Linear+Time19. On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention for Long-Context LLM Serving — approx. recent long-context serving authors, 2024/2025https://scholar.google.com/scholar?q=On-the-Fly+Adaptive+Distillation+of+Transformer+to+Dual-State+Linear+Attention+for+Long-Context+LLM+Serving20. MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling — approx. MiniCPM team / recent efficient attention authors, 2024/2025https://scholar.google.com/scholar?q=MiniCPM-SALA%3A+Hybridizing+Sparse+and+Linear+Attention+for+Efficient+Long-Context+Modeling21. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — DeepSeek team, 2024https://scholar.google.com/scholar?q=DeepSeekMoE%3A+Towards+Ultimate+Expert+Specialization+in+Mixture-of-Experts+Language+Models22. Dive into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts — approx. recent MoE reconstruction authors, 2024/2025https://scholar.google.com/scholar?q=Dive+into+MoE%3A+Diversity-Enhanced+Reconstruction+of+Large+Language+Models+from+Dense+into+Mixture-of-Experts23. AI Post Transformers: Qwen3Guard: Streaming Three-Way Safety Classification for LLMs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-16-qwen3guard-streaming-three-way-safety-cl-26b0ef.mp324. AI Post Transformers: Nemotron 3 Super Hybrid Mamba-Transformer MoE — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-nemotron-3-super-hybrid-mamba-transforme-31ac75.mp325. AI Post Transformers: Mamba-3 for Efficient Sequence Modeling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-16-mamba-3-for-efficient-sequence-modeling-97a22a.mp326. AI Post Transformers: Prefill-as-a-Service for Cross-Datacenter KV Cache — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-prefill-as-a-service-for-cross-datacente-7560be.mp327. AI Post Transformers: Kimi Linear: Efficient Expressive Attention Architecture — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/kimi-linear-efficient-expressive-attention-architecture/28. AI Post Transformers: Memory Sparse Attention for 100M-Token Scaling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-07-memory-sparse-attention-for-100m-token-s-377cff.mp3Interactive Visualization: Qwen3.5-Omni Thinker-Talker for Omnimodal Streaming21 min
- ContiguousKV for Faster LLM Prefill KV ReuseThis episode explores a systems paper on speeding up LLM “Re-Prefill,” the step where a model reloads a previously saved shared prefix KV cache from CPU or SSD, computes a request-specific suffix, and produces the first token. It explains why prefix KV reuse is valuable for shared-context workloads like RAG, conversational search, and multi-turn document QA, but argues that offloading creates new bottlenecks: read amplification from mismatched semantic selection and storage block size, plus serialized I/O-and-compute dependencies that hurt first-token latency. The discussion breaks down the paper’s proposed fixes—granularity-aligned contiguous chunk layouts, speculative asynchronous prefetching, and attention-guided cache residency—and examines the headline claim of a 3.85x Re-Prefill speedup over IMPRESS on Qwen2.5 models. Listeners would find it interesting for its practical focus on where real-world LLM serving slows down once transformer math is no longer the only bottleneck, and for its skeptical analysis of whether the reported gains come from sound systems design or evaluation choices.Sources:1. ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Management — Jing Zou, Shangyu Wu, Hancong Duan, Qiao Li, Chun Jason Xue, 2026http://arxiv.org/abs/2601.136312. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang and collaborators, 2023https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models3. Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time — Yuhui Wang and collaborators, 2023https://scholar.google.com/scholar?q=Scissorhands%3A+Exploiting+the+Persistence+of+Importance+Hypothesis+for+LLM+KV+Cache+Compression+at+Test+Time4. SnapKV: LLM Knows What You Are Looking for Before Generation — Yao Fu and collaborators, 2024https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+Are+Looking+for+Before+Generation5. IMPRESS: An Importance-Based KV Cache Offloading System for Long-Context LLM Inference — Authors include researchers working on systems for offloaded long-context inference, 2024https://scholar.google.com/scholar?q=IMPRESS%3A+An+Importance-Based+KV+Cache+Offloading+System+for+Long-Context+LLM+Inference6. IMPRESS — Not specified in the provided excerpt, Not specified in the provided excerpthttps://scholar.google.com/scholar?q=IMPRESS7. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Liu et al., 2024https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving8. MiniCache: KV Cache Compression in Depth Dimension for Large Language Models — Liu et al., 2024https://scholar.google.com/scholar?q=MiniCache%3A+KV+Cache+Compression+in+Depth+Dimension+for+Large+Language+Models9. StreamingLLM — Xiao et al., 2024https://scholar.google.com/scholar?q=StreamingLLM10. vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention — Kwon et al., 2023https://scholar.google.com/scholar?q=vLLM%3A+Easy%2C+Fast%2C+and+Cheap+LLM+Serving+with+PagedAttention11. More tokens, lower precision: Towards the optimal token-precision trade-off in KV cache compression — approx. recent KV-compression authors, 2024/2025https://scholar.google.com/scholar?q=More+tokens%2C+lower+precision%3A+Towards+the+optimal+token-precision+trade-off+in+KV+cache+compression12. KV-Compress: Paged KV-cache compression with variable compression rates per attention head — approx. recent systems/LLM inference authors, 2024/2025https://scholar.google.com/scholar?q=KV-Compress%3A+Paged+KV-cache+compression+with+variable+compression+rates+per+attention+head13. Paged Attention Meets FlexAttention: Unlocking long-context efficiency in deployed inference — approx. recent long-context inference authors, 2024/2025https://scholar.google.com/scholar?q=Paged+Attention+Meets+FlexAttention%3A+Unlocking+long-context+efficiency+in+deployed+inference14. LayerKV: Optimizing large language model serving with layer-wise KV cache management — approx. recent LLM serving authors, 2024/2025https://scholar.google.com/scholar?q=LayerKV%3A+Optimizing+large+language+model+serving+with+layer-wise+KV+cache+management15. AI Post Transformers: From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-22-from-prefix-cache-to-fusion-rag-9c5d39.mp316. AI Post Transformers: SolidAttention: Co-Designing Sparse Attention and SSD I/O — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-18-solidattention-co-designing-sparse-atten-5a8622.mp317. AI Post Transformers: KVSwap for Disk-Aware Long-Context On-Device Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-16-kvswap-for-disk-aware-long-context-on-de-f3c15e.mp318. AI Post Transformers: Prefill-as-a-Service for Cross-Datacenter KV Cache — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-prefill-as-a-service-for-cross-datacente-7560be.mp319. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp320. AI Post Transformers: Lookahead Q-Cache for Consistent KV Eviction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-lookahead-q-cache-for-consistent-kv-evic-d97b09.mp3Interactive Visualization: ContiguousKV for Faster LLM Prefill KV Reuse17 min
- Nemotron 3 Super Hybrid Mamba-Transformer MoEThis episode explores NVIDIA’s Nemotron 3 Super, an open 120B-parameter model with only 12B active parameters per token, and examines whether its hybrid Mamba-transformer Mixture-of-Experts design can approach frontier-model accuracy while delivering much better inference efficiency. The discussion breaks down the paper’s main ingredients—MoE sparsity, LatentMoE for more practical serving, Mamba-style state-space layers for cheaper long-sequence processing, attention for precise retrieval, NVFP4 low-precision training, multi-token prediction, native speculative decoding, and support for contexts up to one million tokens. It argues that the model is interesting because it tries to unify advances that are often presented separately into a full systems story aimed at agentic reasoning workloads like coding, tool use, and long-horizon tasks. Listeners would find it compelling for its clear explanation of why active parameters, memory movement, and deployment realities matter as much as raw benchmark claims, and for its skepticism about which headline speed and capability claims still need stronger proof.Sources:1. Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning — NVIDIA, :, Aakshita Chandiramani, Aaron Blakeman, Abdullahi Olaoye, Abhibha Gupta, Abhilash Somasamudramath, Abhinav Khattar, Adeola Adesoba, Adi Renduchintala, Adil Asif, Aditya Agrawal, Aditya Vavre, Ahmad Kiswani, Aishwarya Padmakumar, Ajay Hotchandani, Akanksha Shukla, Akhiad Bercovich, Aleksander Ficek, Aleksandr Shaposhnikov, Alex Gronskiy, Alex Kondratenko, Alex Neefus, Alex Steiner, Alex Yang, Alexander Bukharin, Alexander Young, Ali Hatamizadeh, Ali Taghibakhshi, Alina Galiautdinova, Alisa Liu, Alok Kumar, Ameya Sunil Mahabaleshwarkar, Amir Klein, Amit Zuker, Amnon Geifman, Anahita Bhiwandiwalla, Ananth Subramaniam, Andrew Tao, Anjaney Shrivastava, Anjulie Agrusa, Ankur Srivastava, Ankur Verma, Ann Guan, Anna Shors, Annamalai Chockalingam, Anubhav Mandarwal, Aparnaa Ramani, Arham Mehta, Arti Jain, Arun Venkatesan, Asha Anoosheh, Ashwath Aithal, Ashwin Poojary, Asif Ahamed, Asit Mishra, Asli Sabanci Demiroz, Asma Kuriparambil Thekkumpate, Atefeh Sohrabizadeh, Avinash Kaur, Ayush Dattagupta, Barath Subramaniam Anandan, Bardiya Sadeghi, Barnaby Simkin, Ben Lanir, Benedikt Schifferer, Benjamin Chislett, Besmira Nushi, Bilal Kartal, Bill Thiede, Bita Darvish Rouhani, Bobby Chen, Boris Ginsburg, Brandon Norick, Branislav Kisacanin, Brian Yu, Bryan Catanzaro, Buvaneswari Mani, Carlo del Mundo, Chankyu Lee, Chanran Kim, Chantal Hwang, Chao Ni, Charles Wang, Charlie Truong, Cheng-Ping Hsieh, Chenhan Yu, Chenjie Luo, Cherie Wang, Chetan Mungekar, Chintan Patel, Chris Alexiuk, Chris Holguin, Chris Wing, Christian Munley, Christopher Parisien, Chuck Desai, Chunyang Sheng, Collin Neale, Cyril Meurillon, Dakshi Kumar, Dan Gil, Dan Su, Dane Corneil, Daniel Afrimi, Daniel Burkhardt Eliuth Triana, Daniel Egert, Daniel Fatade, Daniel Lo, Daniel Rohrer, Daniel Serebrenik, Daniil Sorokin, Daria Gitman, Daria Levy, Darko Stosic, David Edelsohn, David Messina, David Mosallanezhad, David Tamok, Deena Donia, Deepak Narayanan, Devin O'Kelly, Dheeraj Peri, Dhruv Nathawani, Di Wu, Dima Rekesh, Dina Yared, Divyanshu Kakwani, Dmitry Konyagin Brandon Tuttle, Dong Ahn, Dongfu Jiang, Dorrin Poorkay, Douglas O'Flaherty, Duncan Riach, Dusan Stosic, Dustin Van Stee, Edgar Minasyan, Edward Lin, Eileen Peters Long, Elad Segal, Elena Lantz, Elena Lewis, Ellie Evans, Elliott Ning, Eric Chung, Eric Harper, Eric Pham-Hung, Eric W. Tramel, Erick Galinkin, Erik Pounds, Esti Etrog, Evan Briones, Evan Wu, Evelina Bakhturina, Evgeny Tsykunov, Ewa Dobrowolska, Farshad Saberi Movahed, Farzan Memarian, Fay Wang, Fei Jia, Felipe Soares, Felipe Vieira Frujeri, Feng Chen, Fengguang Lin, Ferenc Galko, Fortuna Zhang, Frankie Siino, Frida Hou, Gantavya Bhatt, Gargi Prasad, Geethapriya Venkataramani, Geetika Gupta, George Armstrong, Gerald Shen, Giulio Borghesi, Gordana Neskovic, Gorkem Batmaz, Grace Lam, Grace Wu, Greg Pauloski, Greyson Davis, Grigor Nalbandyan, Guoming Zhang, Guy Farber, Guyue Huang, Haifeng Qian, Haran Kumar Shiv Kumar, Harry Kim, Harsh Sharma, Hayate Iso, Hayley Ross, Herbert Hum, Herman Sahota, Hexin Wang, Himanshu Soni, Hiren Upadhyay, Huy Nguyen, Iain Cunningham, Ido Galil, Ido Shahaf, Igino Padovani, Igor Gitman, Igor Shovkun, Ikroop Dhillon, Ilya Loshchilov, Ingrid Kelly, Itamar Schen, Itay Levy, Ivan Moshkov, Izik Golan, Izzy Putterman, Jain Tu, Jan Baczek, Jan Kautz, Jane Polak Scowcroft, Janica Rosenberg, Jared Casper, Jarrod Pflum, Jason Grant, Jason Sewall, Jatin Mitra, Jeffrey Glick, Jenny Chen, Jesse Oliver, Jiacheng Xu, Jiafan Zhu, Jialin Song, Jian Zhang, Jiaqi Zeng, Jie Lou, Jill Milton, Jim Chow, Jimmy Zhang, Jinhang Choi, Jining Huang, Jocelyn Huang, Joel Caruso, Joey Conway, Joey Guman, Johan Jatko, John Kamalu, Johnny Greco, Jonathan Cohen, Jonathan Raiman, Joseph Jennings, Joyjit Daw, Juan Yu, Julio Tapia, Junkeun Yi, Jupinder Parmar, Jyothi Achar, Kari Briski, Kartik Mattoo, Katherine Cheung, Katherine Luna, Keith Wyss, Kevin Shih, Kezhi Kong, Khanh Nguyen, Khushi Bhardwaj, Kirill Buryak, Kirthi Shankar Sivamani, Konstantinos Krommydas, Kris Murphy, Krishna C. Puvvada, Krzysztof Pawelec, Kumar Anik, Laikh Tewari, Laya Sleiman, Leo Du, Leon Derczynski, Li Ding, Lilach Ilan, Lingjie Wu, Lizzie Wei, Luis Vega, Lun Su, Maarten Van Segbroeck, Maer Rodrigues de Melo, Magaret Zhang, Mahan Fathi, Makesh Narsimhan Sreedhar, Makesh Sreedhar, Makesh Tarun Chandran, Manuel Reyes Gomez, Maor Ashkenazi, Marc Cuevas, Marc Romeijn, Margaret Zhang, Mark Cai, Mark Gabel, Markus Kliegl, Martyna Patelka, Maryam Moosaei, Matthew Varacalli, Matvei Novikov, Mauricio Ferrato, Mehrzad Samadi, Melissa Corpuz, Meng Xin, Mengdi Wang, Mengru Wang, Meredith Price, Micah Schaffer, Michael Andersch, Michael Boone, Michael Evans, Michael Z Wang, Miguel Martinez, Mikail Khona, Mike Chrzanowski, Mike Hollinger, Mingyuan Ma, Minseok Lee, Mohammad Dabbah, Mohammad Shoeybi, Mostofa Patwary, Nabin Mulepati, Nader Khalil, Najeeb Nabwani, Nancy Agarwal, Nanthini Balasubramaniam, Narimane Hennouni, Narsi Kodukula, Natalie Hereth, Nathaniel Pinckney, Nave Assaf, Negar Habibi, Nestor Qin, Neta Zmora, Netanel Haber, Nick Reamaroon, Nickson Quak, Nidhi Bhatia, Nikhil Jukar, Nikki Pope, Nikolai Ludwig, Nima Tajbakhsh, Nir Ailon, Nirmal Juluru, Nirmalya De, Nowel Pitt, Oleg Rybakov, Oleksii Hrinchuk, Oleksii Kuchaiev, Olivier Delalleau, Oluwatobi Olabiyi, Omer Ullman Argov, Omri Almog, Omri Puny, Oren Tropp, Otavio Padovani, Ouye Xie, Parth Chadha, Pasha Shamis, Paul Gibbons, Pavlo Molchanov, Peter Belcak, Peter Jin, Pinky Xu, Piotr Januszewski, Pooya Jannaty, Prachi Shevate, Pradeep Thalasta, Pranav Prashant Thombre, Prasoon Varshney, Prerana Gambhir, Pritam Gundecha, Przemek Tredak, Qing Miao, Qiyu Wan, Quan Tran Minh, Rabeeh Karimi Mahabadi, Rachel Oberman, Rachit Garg, Rahul Kandu, Raina Zhong, Ran El-Yaniv, Ran Zilberstein, Rasoul Shafipour, Renee Yao, Renjie Pi, Richard Mazzarese, Richard Wang, Rick Izzo, Ridhima Singla, Rima Shahbazyan, Rishabh Garg, Ritika Borkar, Ritu Gala, Riyad Islam, Robert Clark, Robert Hesse, Roger Waleffe, Rohit Varma Kalidindi, Rohit Watve, Roi Koren, Ron Fan, Ruchika Kharwar, Ruisi Cai, Ruoxi Zhang, Russell J. Hewett, Ryan Prenger, Ryan Timbrook, Ryota Egashira, Sadegh Mahdavi, Sagar Singh Ashutosh Joshi, Sahil Modi, Samuel Kriman, Sandeep Pombra, Sanjay Kariyappa, Sanjeev Satheesh, Santiago Pombo, Saori Kaji, Satish Pasumarthi, Saurav Mishra, Saurav Muralidharan, Scott Hara, Sean Narenthiran, Sebastian Rogawski, Seonjin Na, Seonmyeong Bak, Sepehr Sameni, Seth Poulos, Shahar Mor, Shantanu Acharya, Shaona Ghosh Adam Lord, Sharath Turuvekere Sreenivas, Shaun Kotek, Shaya Gharghabi, Shelby Thomas, Sheng-Chieh Lin, Shibani Likhite, Shiqing Fan, Shiyang Chen, Shreya Gopal, Shrimai Prabhumoye, Shubham Pachori, Shubham Toshniwal, Shuo Zhang, Shuoyang Ding, Shyam Renjith, Shyamala Prayaga, Siddhartha Jain, Simeng Sun, Sirisha Rella, Sirshak Das, Smita Ithape, Sneha Harishchandra S, Somshubra Majumdar, Soumye Singhal, Sri Harsha Singudasu, Sriharsha Niverty, Stas Sergienko, Stefana Gloginic, Stefania Alborghetti, Stephen Ge, Stephen McCullough, Sugam Dipak Devare, Suguna Varshini Velury, Sukrit Rao, Sumeet Kumar Barua, Sunny Gai, Suseella Panguluri, Sushil Koundinyan, Swathi Patnam, Sweta Priyadarshi, Swetha Bhendigeri, Syeda Nahida Akter, Sylendran Arunagiri, Tailling Yuan, Talor Abramovich, Tan Bui, Tan Yu, Terry Kong, Thanh Do, Thomas Gburek, Thorgane Marques, Tiffany Moore, Tijmen Blankevoort, Tim Moon, Timothy Ma, Tiyasa Mitra, Tomasz Grzegorzek, Tomer Asida, Tomer Bar Natan, Tomer Keren, Tomer Ronen, Traian Rebedea, Trenton Starkey, Tugrul Konuk, Twinkle Vashishth, Tyler Condensa, Udi Karpas, Ushnish De, Vahid Noorozi, Vahid Noroozi, Vanshil Atul Shah, Veena Vaidyanathan, Venkat Srinivasan, Venmugil Elango, Victor Cui, Vijay Korthikanti, Vikas Mehta, Virginia Adams, Virginia Wu, Vitaly Kurin, Vitaly Lavrukhin, Vladimir Anisimov, Wan Seo, Wanli Jiang, Wasi Uddin Ahmad, Wei Du, Wei Ping, Wei-Ming Chen, Wendy Quan, Wenliang Dai, Wenwen Gao, Will Jennings, William Zhang, Xiaowei Ren, Xiaowen Xin, Xin Li, Yang Yu, Yangyi Chen, Yaniv Galron, Yashaswi Karnati, Yejin Choi, Yev Meyer, Yi-Fu Wu, Yian Zhang, Ying Lin, Yonatan Geifman, Yonggan Fu, Yoshi Suhara, Youngeun Kwon, Yuan Zhang, Yuki Huang, Zach Moshe, Zhilin Wang, Zhiyu Cheng, Zhongbo Zhu, Zhuolin Yang, Zihan Liu, Zijia Chen, Zijie Yan, Zuhair Ahmed, 2026http://arxiv.org/abs/2604.123742. LatentMoE — Elango et al., 2026https://scholar.google.com/scholar?q=LatentMoE3. Nemotron 3 Nano — NVIDIA, 2025https://scholar.google.com/scholar?q=Nemotron+3+Nano4. Nemotron 3 — NVIDIA, 2025https://scholar.google.com/scholar?q=Nemotron+35. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — Lepikhin et al., 2020https://scholar.google.com/scholar?q=GShard%3A+Scaling+Giant+Models+with+Conditional+Computation+and+Automatic+Sharding6. DeepSeek-AI (2025c) MoE model/report — DeepSeek-AI, 2025https://scholar.google.com/scholar?q=DeepSeek-AI+%282025c%29+MoE+model%2Freport7. GLM-4.5-Air-Base — GLM-4.5-Team, 2025https://scholar.google.com/scholar?q=GLM-4.5-Air-Base8. Ling-flash-Base-2.0 — Ling-Team, 2025https://scholar.google.com/scholar?q=Ling-flash-Base-2.09. GPT-OSS-120B — OpenAI, 2025https://scholar.google.com/scholar?q=GPT-OSS-120B10. Qwen3.5-122B — Qwen Team, 2025https://scholar.google.com/scholar?q=Qwen3.5-122B11. How Transformers Learn to Plan via Multi-Token Prediction — unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=How+Transformers+Learn+to+Plan+via+Multi-Token+Prediction12. Better & faster large language models via multi-token prediction — unknown from snippet, 2024/2025https://scholar.google.com/scholar?q=Better+%26+faster+large+language+models+via+multi-token+prediction13. MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration — unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=MemoSight%3A+Unifying+Context+Compression+and+Multi+Token+Prediction+for+Reasoning+Acceleration14. Quartet: Native fp4 training can be optimal for large language models — unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=Quartet%3A+Native+fp4+training+can+be+optimal+for+large+language+models15. Adaptive Block-Scaled Data Types — unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=Adaptive+Block-Scaled+Data+Types16. Scaling laws for precision — unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=Scaling+laws+for+precision17. ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts — unknown from snippet, 2025/2026https://scholar.google.com/scholar?q=ReXMoE%3A+Reusing+Experts+with+Minimal+Overhead+in+Mixture-of-Experts18. Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolation — unknown from snippet, 2024/2025https://scholar.google.com/scholar?q=Recall+with+Reasoning%3A+Chain-of-Thought+Distillation+for+Mamba%27s+Long-Context+Memory+and+Extrapolation19. AI Post Transformers: Mamba-3 for Efficient Sequence Modeling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-16-mamba-3-for-efficient-sequence-modeling-97a22a.mp320. AI Post Transformers: Jet-Nemotron and PostNAS for Faster Long Context — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-24-jet-nemotron-and-postnas-for-faster-long-436381.mp321. AI Post Transformers: GLM-5: Transitioning from Vibe Coding to Agentic Engineering — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/glm-5-transitioning-from-vibe-coding-to-agentic-engineering/22. AI Post Transformers: Batch-Aware Expert Routing for Faster MoE Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-batch-aware-expert-routing-for-faster-mo-683ab6.mp323. AI Post Transformers: Advancements in Efficient KV Cache Quantization and Management — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/advancements-in-efficient-kv-cache-quantization-and-management/24. AI Post Transformers: ShadowKV: High-Throughput Long-Context LLM Inference — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/shadowkv-high-throughput-long-context-llm-inference/25. AI Post Transformers: Memory Sparse Attention for 100M-Token Scaling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-07-memory-sparse-attention-for-100m-token-s-377cff.mp326. AI Post Transformers: Kimi Linear: Efficient Expressive Attention Architecture — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/kimi-linear-efficient-expressive-attention-architecture/27. AI Post Transformers: rStar2-Agent: Smarter Math Reasoning Through Agentic RL — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/rstar2-agent-smarter-math-reasoning-through-agentic-rl/28. AI Post Transformers: Learning to Reason with 13 Parameters — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-learning-to-reason-with-13-parameters-54c87f.mp329. AI Post Transformers: CWM: Code Generation with World Models — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/cwm-code-generation-with-world-models/30. AI Post Transformers: LFM2-8B-A1B: Efficient On-Device Mixture-of-Experts — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/lfm2-8b-a1b-efficient-on-device-mixture-of-experts/31. AI Post Transformers: Llama 3: Architecture, Capabilities, and Safety — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/llama-3-architecture-capabilities-and-safety/Interactive Visualization: Nemotron 3 Super Hybrid Mamba-Transformer MoE21 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.