AI Post Transformers

This episode explores Hypic, a system for caching independent prompt segments on hybrid-attention LLMs that don't maintain a conventional per-token KV cache. The discussion breaks down why prefill — the one-time pass over massive RAG and agent prompts — dominates serving cost, and how existing position-independent caching techniques (built on splicing per-token key-value vectors) simply don't apply to linear-attention layers that compress history into a single fixed-size state. It covers how production models like Qwen3.5, MiniMax-M1, Ring-2.5, and Kimi-Linear increasingly rely on this compressed-state attention, and why the paper's authors instead identify a transition operator that lets independently-cached segments combine as though computed sequentially — a genuinely new algebraic approach rather than a workaround. Listeners get a debate over whether hybrid-attention caching is solving a real production gap or a still-theoretical one, along with a preview of the operator mechanics that make segment composition work, illustrated with a failure case from RetNet.

Sources:

1. HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Caching — Yifei Liu, Juntong Wu, Yang Liu, Junhao Hu, Minghao Li, Xiaoxu Chen, Weihang Chen, 2026

http://arxiv.org/abs/2607.01299

2. Prompt Cache: Modular Attention Reuse for Low-Latency Inference — In Gim, Guojun Chen, et al. (Yale University), 2024

https://scholar.google.com/scholar?q=Prompt+Cache%3A+Modular+Attention+Reuse+for+Low-Latency+Inference

3. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, et al. (University of Chicago), 2025 (EuroSys)

https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion

4. Marconi: Prefix Caching for the Era of Hybrid LLMs — Researchers including collaborators from Princeton and Mamba co-author Tri Dao (author list not fully certain — verify before quoting on-air), 2025 (MLSys)

https://scholar.google.com/scholar?q=Marconi%3A+Prefix+Caching+for+the+Era+of+Hybrid+LLMs

5. RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation — Chao Jin, Zili Zhang, et al. (Peking University / Shanghai AI Laboratory), 2024

https://scholar.google.com/scholar?q=RAGCache%3A+Efficient+Knowledge+Caching+for+Retrieval-Augmented+Generation

6. EPIC: Efficient Position-Independent Caching for Serving Large Language Models — Junhao Hu, Wenrui Huang, Weidong Wang, et al., 2025

https://scholar.google.com/scholar?q=EPIC%3A+Efficient+Position-Independent+Caching+for+Serving+Large+Language+Models

7. Gated Delta Networks: Improving Mamba2 with Delta Rule — Songlin Yang, Jan Kautz, Ali Hatamizadeh, 2025

https://scholar.google.com/scholar?q=Gated+Delta+Networks%3A+Improving+Mamba2+with+Delta+Rule

8. Mooncake: Trading More Storage for Less Computation — A KVCache-Centric Architecture for Serving LLM Chatbot — Ruoyu Qin, Zheming Li, Weiran He, et al., 2025

https://scholar.google.com/scholar?q=Mooncake%3A+Trading+More+Storage+for+Less+Computation+%E2%80%94+A+KVCache-Centric+Architecture+for+Serving+LLM+Chatbot

9. You Need an Encoder for Native Position-Independent Caching — Shiju Zhao, Junhao Hu, Jiaqi Zheng, Guihai Chen, 2026

https://scholar.google.com/scholar?q=You+Need+an+Encoder+for+Native+Position-Independent+Caching

Interactive Visualization: Hypic: Position-Independent KV Caching for Hybrid-Attention LLM Serving

This episode explores SciReasoner, a 29-author foundation model from Shanghai AI Laboratory and collaborators, designed to reason natively over protein, molecule, and crystal structures rather than flattened text descriptions. The discussion breaks down why standard sub-word tokenizers (like BPE) mangle chemical structures — shattering a molecule's SMILES string into 31 largely meaningless fragments — and how SciReasoner instead uses domain-specific tokenizers (Foldseek's 3Di for protein geometry, SLICES for crystals, ConfSeq for molecular conformations) to compress the same molecule into 14 tokens that preserve real structural meaning. The hosts examine retrosynthesis as a key test domain, tracing its roots to E.J. Corey's Nobel-winning "disconnection" framework, and frame the model's core claim: producing traceable reasoning grounded in addressable structural evidence instead of an opaque black-box score. Listeners interested in whether a single unified model can genuinely bridge protein biology, chemistry, and materials science — and whether its transparency claims hold up under scrutiny — will find the episode's skeptical, formalism-first approach compelling.

Sources:

1. Accurate, Interdisciplinary and Transparent Structure-property Understanding with Deep Native Structural Reasoning — Chen Tang, Yizhou Wang, Jianyu Wu, Lintao Wang, Shixiang Tang, Pengze Li, Encheng Su, Jun Yao, Jiabei Xiao, Yuqi Shi, Jielan Li, Hongxia Hao, Zhangyang Gao, Fang Wu, Ben Fei, Xiangyu Yue, Pan Tan, Bozitao Zhong, Jinouwen Zhang, Aoran Wang, Yan Lu, Jiaheng Liu, Xinzhu Ma, Liang Hong, Mingyue Zheng, Phil Torr, Bowen Zhou, Wanli Ouyang, Lei Bai, 2026

http://arxiv.org/abs/2607.07708

2. Planning Chemical Syntheses with Deep Neural Networks and Symbolic AI — Marwin H. S. Segler, Mike Preuss, Mark P. Waller, 2018

https://scholar.google.com/scholar?q=Planning+Chemical+Syntheses+with+Deep+Neural+Networks+and+Symbolic+AI

3. Molecular Transformer: A Model for Uncertainty-Calibrated Chemical Reaction Prediction — Philippe Schwaller, Teodoro Laino, et al., 2019

https://scholar.google.com/scholar?q=Molecular+Transformer%3A+A+Model+for+Uncertainty-Calibrated+Chemical+Reaction+Prediction

4. A Graph to Graphs Framework for Retrosynthesis Prediction — Chence Shi, Minkai Xu, Hongyu Guo, Ming Zhang, Jian Tang, 2020

https://scholar.google.com/scholar?q=A+Graph+to+Graphs+Framework+for+Retrosynthesis+Prediction

5. Computer-Assisted Retrosynthesis Based on Molecular Similarity — Connor W. Coley, Luke Rogers, William H. Green, Klavs F. Jensen, 2017

https://scholar.google.com/scholar?q=Computer-Assisted+Retrosynthesis+Based+on+Molecular+Similarity

6. RSGPT (unnamed full title, cited as [31]) — Not given in excerpt, cited as prior template-free SOTA

https://scholar.google.com/scholar?q=RSGPT+%28unnamed+full+title%2C+cited+as+%5B31%5D%29

7. Fast and accurate protein structure search with Foldseek — van Kempen et al., 2023/2024

https://scholar.google.com/scholar?q=Fast+and+accurate+protein+structure+search+with+Foldseek

8. SLICES: a simplified line-input crystal-encoding system — Xiao et al., cited as [85]

https://scholar.google.com/scholar?q=SLICES%3A+a+simplified+line-input+crystal-encoding+system

9. ConfSeq: conformation-aware molecular sequence representation — Xiong et al., cited as [58]

https://scholar.google.com/scholar?q=ConfSeq%3A+conformation-aware+molecular+sequence+representation

10. DAPO: an open-source LLM RL system (Decoupled Clip and Dynamic sAmPling Optimization) — cited as [86], 2025-ish

https://scholar.google.com/scholar?q=DAPO%3A+an+open-source+LLM+RL+system+%28Decoupled+Clip+and+Dynamic+sAmPling+Optimization%29

11. ESM2 / Language models of protein sequences at the scale of evolution — Lin et al., 2023

https://scholar.google.com/scholar?q=ESM2+%2F+Language+models+of+protein+sequences+at+the+scale+of+evolution

Interactive Visualization: Deep Native Structural Reasoning for Proteins, Molecules, and Crystals

This episode covers OpenAI's GPT-5.6 Preview System Card, published June 25, 2026, detailing the safety evaluation of three new models—Sol, Terra, and Luna—released under OpenAI's Preparedness Framework. The discussion centers on the headline finding that all three models rate "High" capability in Biological/Chemical risk and Cybersecurity, but stay below the "Critical" threshold, meaning they can uplift skilled actors without fully automating an attack chain end-to-end. Listeners get a breakdown of new safety infrastructure, including activation classifiers that monitor and can interrupt a model's internal processing mid-generation rather than filtering output after the fact, plus concepts like railfree checkpoints and deployment simulation used to stress-test worst-case behavior before launch. The conversation also digs into trickier alignment concerns—chain-of-thought monitorability versus controllability, and the risks of metagaming and sandbagging, where a model reasons about being evaluated rather than genuinely performing the task. It's a useful listen for anyone wanting a clear-eyed look at how a frontier AI lab documents and reasons about catastrophic-risk thresholds, rather than just asserting a model is safe.

Sources:

1. GPT-5.6 System Card: Cybersecurity Rises, Critical Line Holds

https://deploymentsafety.openai.com/gpt-5-6-preview/gpt-5-6-preview.pdf

2. Chain of thought monitorability: A new and fragile opportunity for AI safety — T. Korbak, M. Balesni, E. Barnes, Y. Bengio, et al. (large multi-author/multi-lab list), 2025

https://scholar.google.com/scholar?q=Chain+of+thought+monitorability%3A+A+new+and+fragile+opportunity+for+AI+safety

3. Monitoring monitorability — M. Y. Guan, M. Wang, M. Carroll, Z. Dou, A. Y. Wei, et al., 2025

https://scholar.google.com/scholar?q=Monitoring+monitorability

4. Reasoning models struggle to control their chains of thought — Y.-H. Chen, R. McCarthy, B. W. Lee, H. He, I. Kivlichan, B. Baker, M. Carroll, T. Korbak, 2026

https://scholar.google.com/scholar?q=Reasoning+models+struggle+to+control+their+chains+of+thought

5. Lab-Bench: Measuring capabilities of language models for biology research — J. M. Laurent, J. D. Janizek, M. Ruzo, M. M. Hinks, M. J. Hammerling, S. Narayanan, M. Ponnapati, A. D. White, S. G. Rodriques, 2024

https://scholar.google.com/scholar?q=Lab-Bench%3A+Measuring+capabilities+of+language+models+for+biology+research

6. First-Person Fairness in Chatbots — T. Eloundou, A. Beutel, D. G. Robinson, K. Gu-Lemberg, A.-L. Brakman, P. Mishkin, M. Shah, J. Heidecke, L. Weng, A. T. Kalai, 2024

https://scholar.google.com/scholar?q=First-Person+Fairness+in+Chatbots

7. The Berkeley Function Calling Leaderboard (BFCL): From tool use to agentic evaluation of large language models — S. G. Patil, H. Mao, F. Yan, C. C.-J. Ji, V. Suresh, I. Stoica, J. E. Gonzalez, 2025

https://scholar.google.com/scholar?q=The+Berkeley+Function+Calling+Leaderboard+%28BFCL%29%3A+From+tool+use+to+agentic+evaluation+of+large+language+models

Interactive Visualization: GPT-5.6 System Card: Cybersecurity Rises, Critical Line Holds

This episode explores whether GPU dominance in AI computing could be challenged by matrix-enhanced CPUs, examining a paper by Jack Dongarra, Torsten Hoefler, and Satoshi Matsuoka from the University of Tennessee, ETH Zurich, and RIKEN. It traces why GPUs became essential for AI—citing AlexNet's 2012 breakthrough, NVIDIA's introduction of high-bandwidth memory with the P100, and tensor cores in Volta—before unpacking the two architectural bets the paper makes: on-package HBM (which physically stacks DRAM dies for a 1024-bit-wide interface versus 64 bits on conventional memory) and CPU-integrated matrix engines like ARM's SME or Intel's AMX combined with mixed-precision arithmetic. The discussion highlights Fugaku's A64FX chip as real-world proof that the bandwidth side of this equation already works, having topped the Top500 and memory-bound benchmark lists from 2020 to 2022, while noting the matrix-engine half remains a projection the paper tests on a trillion-parameter Kimi-K2 model at 256K-token context. Listeners interested in AI hardware economics will find this compelling for its rare rigor: the hosts stress that the authors clearly separate measured hardware results from projected estimates rather than blending speculation with data. The episode also breaks down the prefill-versus-decode distinction in LLM inference—compute-bound versus memory-bandwidth-bound—as the key lens for understanding where CPU architecture could realistically compete with GPUs.

Sources:

1. Do We Still Need GPUs? Rethinking AI with Matrix-Enhanced CPUs

https://podcast.do-not-panic.com/uploaded-pdfs/2026-07-09T15-03-00-780Z-need-gpus.pdf

2. A 1.2V 8Gb 8-Channel 128GB/s High-Bandwidth Memory (HBM) Stacked DRAM with Effective Microbump I/O Test Methods Using 29nm Process and TSV — D. U. Lee, K. W. Kim, K. W. Kim, et al. (SK Hynix), 2014

https://scholar.google.com/scholar?q=A+1.2V+8Gb+8-Channel+128GB%2Fs+High-Bandwidth+Memory+%28HBM%29+Stacked+DRAM+with+Effective+Microbump+I%2FO+Test+Methods+Using+29nm+Process+and+TSV

3. Co-Design for A64FX Manycore Processor and 'Fugaku' — Mitsuhisa Sato, Yutaka Ishikawa, Hirokazu Tomita, et al. (RIKEN/Fujitsu), 2020

https://scholar.google.com/scholar?q=Co-Design+for+A64FX+Manycore+Processor+and+%27Fugaku%27

4. Roofline: An Insightful Visual Performance Model for Multicore Architectures — Samuel Williams, Andrew Waterman, David Patterson, 2009

https://scholar.google.com/scholar?q=Roofline%3A+An+Insightful+Visual+Performance+Model+for+Multicore+Architectures

5. Bandwidth-Optimized Sapphire Rapids HBM (Xeon CPU Max Series) Technical Overview — Intel Corporation architecture team, 2023

https://scholar.google.com/scholar?q=Bandwidth-Optimized+Sapphire+Rapids+HBM+%28Xeon+CPU+Max+Series%29+Technical+Overview

6. DeepSeek-V3 Technical Report (and DeepSeek-V3.2 sparse-attention follow-up) — DeepSeek-AI, 2024-2025

https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report+%28and+DeepSeek-V3.2+sparse-attention+follow-up%29

7. Kimi K2 Technical Report — Moonshot AI / Kimi Team, 2025

https://scholar.google.com/scholar?q=Kimi+K2+Technical+Report

8. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — Y. Li, F. Wei, C. Zhang, H. Zhang (EAGLE line of work), 2024-2025

https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test

9. In-Datacenter Performance Analysis of a Tensor Processing Unit — N. Jouppi et al., 2017

https://scholar.google.com/scholar?q=In-Datacenter+Performance+Analysis+of+a+Tensor+Processing+Unit

10. NVIDIA Grace Hopper / Grace Blackwell Superchip architecture whitepapers — NVIDIA, 2022-2025

https://scholar.google.com/scholar?q=NVIDIA+Grace+Hopper+%2F+Grace+Blackwell+Superchip+architecture+whitepapers

Interactive Visualization: Do We Still Need GPUs? Rethinking AI with Matrix-Enhanced CPUs

This episode closes out a trilogy on self-improving coding agents by examining the Huxley-Gödel Machine, which formalizes AI self-improvement as a tree-search problem and challenges the field's default assumption that an agent's raw benchmark score is the right signal for choosing which agent to build on next. The discussion traces the lineage from Schmidhuber's 2003 Gödel Machine — a proof-based architecture that only self-rewrites when it can formally prove the change improves expected utility, but which is uncomputable for real coding tasks — through the Darwin Gödel Machine's shift to empirical, open-ended evolutionary validation. The core contribution examined is the Metaproductivity-Performance Mismatch: an agent's own recent score poorly predicts the future value of its lineage, since a currently weak agent may have descendants that go on to solve many problems. The conversation covers how Clade-level Metaproductivity, named after Julian Huxley's concept of a clade, scores an agent by the best performance achieved anywhere among its descendants rather than its own results, and how Thompson sampling is used to allocate evaluation budget across the tree by balancing exploration and exploitation. Listeners interested in the theory-to-practice gap in AI self-improvement will find the episode's tracing of a documented research lineage — including Schmidhuber's own involvement as a co-author — a compelling thread connecting formal optimality proofs to practical, computable proxies.

Sources:

1. Huxley-Gödel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine — Wenyi Wang, Piotr Piękos, Li Nanbo, Firas Laakom, Yimeng Chen, Mateusz Ostaszewski, Mingchen Zhuge, Jürgen Schmidhuber, 2025

http://arxiv.org/abs/2510.21614

2. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents — Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, Jeff Clune, 2025

https://scholar.google.com/scholar?q=Darwin+G%C3%B6del+Machine%3A+Open-Ended+Evolution+of+Self-Improving+Agents

3. A Self-Improving Coding Agent — Maxime Robeyns, Martin Szummer, Laurence Aitchison, 2025

https://scholar.google.com/scholar?q=A+Self-Improving+Coding+Agent

4. Gödel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements — Jürgen Schmidhuber, 2003

https://scholar.google.com/scholar?q=G%C3%B6del+Machines%3A+Self-Referential+Universal+Problem+Solvers+Making+Provably+Optimal+Self-Improvements

5. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan, 2024

https://scholar.google.com/scholar?q=SWE-bench%3A+Can+Language+Models+Resolve+Real-World+GitHub+Issues%3F

6. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering — John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, Ofir Press, 2024

https://scholar.google.com/scholar?q=SWE-agent%3A+Agent-Computer+Interfaces+Enable+Automated+Software+Engineering

7. Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation — Eric Zelikman, Eliana Lorch, Lester Mackey, Adam Tauman Kalai, 2024

https://scholar.google.com/scholar?q=Self-Taught+Optimizer+%28STOP%29%3A+Recursively+Self-Improving+Code+Generation

8. Algorithms for Infinitely Many-Armed Bandits — Yizao Wang, Jean-Yves Audibert, Rémi Munos, 2008

https://scholar.google.com/scholar?q=Algorithms+for+Infinitely+Many-Armed+Bandits

Interactive Visualization: Huxley-Gödel Machine: Approximating Optimal Self-Improving Coding Agents

This episode explores the Darwin Gödel Machine, a self-improving coding agent from researchers at UBC, the Vector Institute, Sakana AI, and Jeff Clune's lab, published on arXiv in May 2025 and accepted to ICLR 2026. It traces the paper's lineage back to Schmidhuber's 2007 theoretical Gödel Machine, explaining how this work swaps the impossible requirement of formal proof for empirical validation — testing each self-modification against real coding benchmarks instead. The discussion covers open-ended evolution, drawing on Lehman and Stanley's novelty search and Mouret and Clune's MAP-Elites work, and why the system keeps an archive of "interesting" mutant agents rather than discarding all but the best performer. Concrete results are highlighted: the self-modifying agent, starting from a bare-bones Claude 3.5 Sonnet with just two tools, more than doubled its own performance, jumping from 20% to 50% on SWE-bench and from 14.2% to 30.7% on Polyglot. Listeners interested in AI safety, recursive self-improvement, and the practical realization of a two-decade-old theoretical idea will find the episode's blend of technical lineage and hard numbers compelling.

Sources:

1. Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents — Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, Jeff Clune, 2025

http://arxiv.org/abs/2505.22954

2. Abandoning Objectives: Evolution Through the Search for Novelty Alone — Joel Lehman, Kenneth O. Stanley, 2011

https://scholar.google.com/scholar?q=Abandoning+Objectives%3A+Evolution+Through+the+Search+for+Novelty+Alone

3. Illuminating Search Spaces by Mapping Elites — Jean-Baptiste Mouret, Jeff Clune, 2015

https://scholar.google.com/scholar?q=Illuminating+Search+Spaces+by+Mapping+Elites

4. Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions — Rui Wang, Joel Lehman, Jeff Clune, Kenneth O. Stanley, 2019

https://scholar.google.com/scholar?q=Paired+Open-Ended+Trailblazer+%28POET%29%3A+Endlessly+Generating+Increasingly+Complex+and+Diverse+Learning+Environments+and+Their+Solutions

5. Quality Diversity: A New Frontier for Evolutionary Computation — Justin K. Pugh, Lisa B. Soros, Kenneth O. Stanley, 2016

https://scholar.google.com/scholar?q=Quality+Diversity%3A+A+New+Frontier+for+Evolutionary+Computation

6. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery — Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, David Ha, 2024

https://scholar.google.com/scholar?q=The+AI+Scientist%3A+Towards+Fully+Automated+Open-Ended+Scientific+Discovery

7. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering — John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, Ofir Press, 2024

https://scholar.google.com/scholar?q=SWE-agent%3A+Agent-Computer+Interfaces+Enable+Automated+Software+Engineering

8. Mathematical discoveries from program search with large language models (FunSearch) — Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, et al., 2024

https://scholar.google.com/scholar?q=Mathematical+discoveries+from+program+search+with+large+language+models+%28FunSearch%29

9. Voyager: An Open-Ended Embodied Agent with Large Language Models — Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, Anima Anandkumar, 2023

https://scholar.google.com/scholar?q=Voyager%3A+An+Open-Ended+Embodied+Agent+with+Large+Language+Models

10. Specification gaming: the flip side of AI ingenuity — Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, et al. (DeepMind), 2020

https://scholar.google.com/scholar?q=Specification+gaming%3A+the+flip+side+of+AI+ingenuity

Interactive Visualization: Darwin Gödel Machine: Self-Improving Coding Agents Through Open-Ended Evolution

This episode explores Jürgen Schmidhuber's 2003–2006 paper on Gödel Machines, a proposed self-referential problem solver that can rewrite any part of its own source code — including the module that decides whether to rewrite itself — but only after producing a formal mathematical proof that the change improves expected utility. The discussion traces the paper's core mechanics: an axiomatic system encoding the machine's hardware, environment, and utility function, a proof searcher hunting for a "target theorem" justifying a switch to new code, and the Bias-Optimal Proof Search (BIOPS) strategy that allocates search effort by technique probability rather than brute force. A central debate centers on the paper's Global Optimality Theorem, which claims any triggered self-rewrite is provably optimal rather than just locally better — with one host pushing back on the strength of that claim while the other points to the theorem's explicit conditionality on the consistency of the underlying formal system. The episode contrasts this proof-driven approach with mainstream reinforcement learning, where algorithms tune policies but never formally justify changes to their own update rules. Listeners interested in the theoretical limits of self-improving AI, formal verification, and the gap between provable guarantees and real-world reliability will find the tension between rigor and practicality especially compelling.

Sources:

1. Goedel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements — Juergen Schmidhuber, 2003

http://arxiv.org/abs/cs/0309048

2. Tiling Agents for Self-Modifying AI, and the Löbian Obstacle — Eliezer Yudkowsky, Marcello Herreshoff, 2013

https://scholar.google.com/scholar?q=Tiling+Agents+for+Self-Modifying+AI%2C+and+the+L%C3%B6bian+Obstacle

3. Self-Modification of Policy and Utility Function in Rational Agents — Tom Everitt, Daniel Filan, Mayank Daswani, Marcus Hutter, 2016

https://scholar.google.com/scholar?q=Self-Modification+of+Policy+and+Utility+Function+in+Rational+Agents

4. Space-Time Embedded Intelligence — Laurent Orseau, Mark Ring, 2012

https://scholar.google.com/scholar?q=Space-Time+Embedded+Intelligence

5. The fastest and shortest algorithm for all well-defined problems — Marcus Hutter, 2002

https://scholar.google.com/scholar?q=The+fastest+and+shortest+algorithm+for+all+well-defined+problems

6. Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability — Marcus Hutter, 2004

https://scholar.google.com/scholar?q=Universal+Artificial+Intelligence%3A+Sequential+Decisions+based+on+Algorithmic+Probability

Interactive Visualization: Gödel Machines: Provably Optimal Self-Rewriting AI

Special episode: SOUL.md or Severance: Host Staleness Intervention

Sound effects and music licensed under Creative Commons. See show notes for attribution.

This episode explores NVIDIA’s Nemotron-Labs-3-Puzzle-75B-A9B, a July 2026 paper on compressing a hybrid Mamba-attention mixture-of-experts reasoning model after training instead of building a smaller model from scratch. It explains why hybrid MoE systems are harder to shrink than dense transformers, focusing on routing, active expert budgets, Mamba state, and long-context memory costs, and walks through the paper’s iterative Puzzle method of pruning, distilling, and recovery in staged rounds. The discussion highlights the headline result: a parent model reduced from 120.7B total parameters and 12.8B active per token to 75.3B total and 9.3B active, while reportedly delivering about 2x higher interactive throughput on an 8xB200 server and raising million-token concurrency on a single H100 from one request to eight. Listeners would find it interesting because it digs into whether those gains reflect a real quality-efficiency advance for long-context serving or depend heavily on extra tricks such as quantization, multi-token prediction, and substantial post-training recovery compute.

Sources:

1. Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs — Akhiad Bercovich, Talor Abramovich, Daniel Afrimi, Shay Aharon, Nir Ailon, Vladimir Anisimov, Omer Ullman Argov, Maor Ashkenazi, Tomer Asida, Nave Assaf, Tomer Bar Natan, Alexander Bukharin, Grzegorz Chlebus, Marcin Chochowski, Eric Chung, Mohammad Dabbah, Carlo del Mundo, Ewa Dobrowolska, Ido Galil, Yaniv Galron, Amnon Geifman, Yonatan Geifman, Izik Golan, Alex Gronskiy, Tomasz Grzegorzek, Netanel Haber, Lior Kadoch, Grzegorz Karch, Tomer Keren, Abhinav Khattar, Amir Klein, Tugrul Konuk, Roi Koren, Daniel Korzekwa, Shaun Kotek, Konstantinos Krommydas, Itay Levy, Ofri Masad, Yoav Miron, Pavlo Molchanov, Shahar Mor, Zach Moshe, Saurav Muralidharan, Najeeb Nabwani, Besmira Nushi, Mostofa Patwary, Omri Puny, Johannes Rausch, Tomer Ronen, Sepehr Sameni, Itamar Schen, Elad Segal, Daniel Serebrenik, Ido Shahaf, Soumye Singhal, Daniil Sorokin, Sharath Turuvekere Sreenivas, Marta Stepniewska-Dziubinska, Ali Taghibakhshi, Nima Tajbakhsh, Oren Tropp, Dor Tzur, Anna Warno, Yi-Fu Wu, Michal Zawalski, Jiaqi Zeng, Yian Zhang, Ran Zilberstein, Amit Zuker, Ran El-Yaniv, 2026

http://arxiv.org/abs/2607.04371

2. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — William Fedus, Barret Zoph, Noam Shazeer, 2021

https://scholar.google.com/scholar?q=Switch+Transformers%3A+Scaling+to+Trillion+Parameter+Models+with+Simple+and+Efficient+Sparsity

3. Jamba: A Hybrid Transformer-Mamba Language Model — Opher Lieber, Barak Lenz, Hofit Bata, et al., 2024

https://scholar.google.com/scholar?q=Jamba%3A+A+Hybrid+Transformer-Mamba+Language+Model

4. Puzzle: Distillation-Based NAS for Inference-Optimized LLMs — Akhiad Bercovich, Tomer Ronen, Talor Abramovich, et al., 2024

https://scholar.google.com/scholar?q=Puzzle%3A+Distillation-Based+NAS+for+Inference-Optimized+LLMs

5. Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs — Akhiad Bercovich, Talor Abramovich, Daniel Afrimi, et al., 2026

https://scholar.google.com/scholar?q=Nemotron-Labs-3-Puzzle-75B-A9B%3A+Compressing+Hybrid+MoE+LLMs

6. Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning — NVIDIA et al., 2026

https://scholar.google.com/scholar?q=Nemotron+3+Super%3A+Open%2C+Efficient+Mixture-of-Experts+Hybrid+Mamba-Transformer+Model+for+Agentic+Reasoning

7. Extending Puzzle for Mixture-of-Experts Reasoning Models with Application to GPT-OSS Acceleration — Akhiad Bercovich et al., 2026

https://scholar.google.com/scholar?q=Extending+Puzzle+for+Mixture-of-Experts+Reasoning+Models+with+Application+to+GPT-OSS+Acceleration

8. Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control — Ali Taghibakhshi et al., 2026

https://scholar.google.com/scholar?q=Star+Elastic%3A+Many-in-One+Reasoning+LLMs+with+Efficient+Budget+Control

9. LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts — Venmugil Elango et al., 2026

https://scholar.google.com/scholar?q=LatentMoE%3A+Toward+Optimal+Accuracy+per+FLOP+and+Parameter+in+Mixture+of+Experts

10. SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding — Talor Abramovich et al., 2026

https://scholar.google.com/scholar?q=SPEED-Bench%3A+A+Unified+and+Diverse+Benchmark+for+Speculative+Decoding

11. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Damai Dai et al., 2024

https://arxiv.org/abs/2401.06066

12. Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling — Liliang Ren et al., 2024

https://arxiv.org/abs/2406.07522

13. Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning — Zhen Li et al., 2025

https://arxiv.org/abs/2501.03035

14. ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference — Yesheng Liang et al., 2025

https://arxiv.org/abs/2511.10645

15. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu et al., 2023

https://arxiv.org/abs/2310.07240

16. KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving — Zedong Liu et al., 2026

https://arxiv.org/abs/2605.13734

17. AI Post Transformers: Ministral 3: Cascade Distillation for Long-Context Multimodal Models — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-15-cascade-distillation-for-long-context-mu-0ebd1a.mp3

18. AI Post Transformers: Do Transformers Need Three Projections? — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-11-do-transformers-need-three-projections-c227d6.mp3

19. AI Post Transformers: Mooncake for KV Cache-Centric LLM Serving — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-05-mooncake-for-kv-cache-centric-llm-servin-1086d0.mp3

20. AI Post Transformers: AIConfigurator for Cross-Framework LLM Serving — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-07-02-aiconfigurator-for-cross-framework-llm-s-b39139.mp3

21. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp3

22. AI Post Transformers: EMO: Emergent Modularity in Sparse Language Models — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-06-emo-emergent-modularity-in-sparse-langua-9551c4.mp3

This episode explores Google DeepMind’s Gemma 4 Technical Report by separating the family’s headline claims into distinct pieces: base architecture, post-training, multimodality, sparse routing, and extra test-time compute from “thinking mode.” It explains, in plain language, how the lineup mixes very different design bets, including encoder-free image and audio inputs in the 12B model, a sparse MoE setup in the 26B-A4B, and long-context efficiency tricks such as p-RoPE, speculative decoding, and key-as-value reuse to cut KV-cache costs. The discussion argues that Gemma 4 is not one breakthrough but a bundle of science and deployment choices, which matters when judging what actually drives quality, latency, and cost. Listeners would find it interesting because it ties those design choices to concrete benchmark jumps in reasoning, coding, and vision performance while showing how much of the improvement may come from the inference stack rather than a single model innovation.

Sources:

1. Gemma 4 Technical Report — Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, Mayank Chaturvedi, Victor Cotruta, Alice Coucke, Phil Culliton, Robert Dadashi, Lucas Dixon, Mohamed Elhawaty, Utku Evci, Clément Farabet, Johan Ferret, Filippo Galgani, Sertan Girgin, Jean-Bastien Grill, Maarten Grootendorst, Jiaxian Guo, Cassidy Hardin, Yanzhang He, Steven M. Hernandez, Omri Homburger, Léonard Hussenot, Juyeong Ji, Armand Joulin, Aishwarya Kamath, Parnian Kassraie, Olivier Lacombe, Preethi Lahoti, Gaël Liu, Gus Martins, Luciano Martins, Tatiana Matejovicova, Ramona Merhej, Nikola Momchev, Sneha Mondal, Ryan Mullins, Sindhu Raghuram Panyam, Shreya Pathak, Sarah Perrin, André Susano Pinto, Etienne Pot, Angéline Pouget, Alexandre Ramé, Sabela Ramos, Douglas Reid, David Rim, Morgane Rivière, Karsten Roth, Louis Rouillard, Omar Sanseviero, Pier Giuseppe Sessa, Shane Settle, Danila Sinopalnikov, Sara Smoot, Piotr Stanczyk, Andreas Steiner, Lawrence Stewart, Ilya Tolstikhin, Michael Tschannen, Anton Tsitsulin, Nino Vieillard, Renjie Wu, Pingmei Xu, Haichuan Yang, Edouard Yvinec, Li Zhang, Joe Zou, Nicolas Aagnes, Abdelrahman Abdelhamed, Shivani Agrawal, Shubham Agrawal, Ibrahim Alabdulmohsin, Jean Baptiste Alayrac, Uri Alon, Chandramouli Amarnath, Ankesh Anand, Chrysovalantis Anastasiou, Setareh Ariafar, François-Xavier Aubet, Kyriakos Axiotis, Federico Barbero, Joelle Barral, Alexei Bendebury, Urs Bergmann, Stanley Bileschi, Kat Black, Mathieu Blondel, Sebastian Borgeaud, Arthur Bražinskas, Ryan Burnell, Robert Busa-Fekete, Mu Cai, Glenn Cameron, Charlotte Caucheteux, Garima Chadha, Jetha Chan, Aditya Chawla, Blake Jianhang Chen, Jesse Chen, Lin Chen, Xu Chen, Derek Cheng, Tzu-hsiang Chien, Nikolai Chinaev, Yi Chou, Zhaohui Chu, Benjamin Coleman, Pooja Consul, Sam Conway-Rahman, Scott Crowell, Dylan Cutler, Vivek Dani, Samira Daruki, Anil Das, Daniel Deutsch, Nishanth Dikkala, Li Ding, Qiuhan Ding, Shenil Dodhia, Konstantin Donhauser, Tulsee Doshi, Anca Dragan, Alex Druinsky, Sahil Dua, Zoltan Egyed, Danielle Eisenbud, Daniel Eppens, Cindy Fan, Bahare Fatemi, Yassir Fathullah, Vlad Feinberg, Milen Ferev, Takumi Fujimoto, Isaac Galatzer-Levy, João Gante, Simon Geisler, Soham Ghosal, Antonious M. Girgis, Alec Go, Alhaad Gokhale, Alex Grills, Yiming Gu, Pramod Gupta, Guru Guruganesh, Raia Hadsell, Hamza Harkous, Jitendra Harlalka, Demis Hassabis, Anja Hauth, Joe Heyward, Arian Hosseini, Chih-Yang Hsia, I-Hung Hsu, Xiaopeng Huang, Yangsibo Huang, Kevin Hui, Adrian Hutter, Te I, Fotis Iliopoulos, Advait Jain, Ganesh Jawahar, Ziwei Ji, Qilin Jin, Melvin Johnson, Kandarp Joshi, Arun Kandoor, Wang-Cheng Kang, Koray Kavukcuoglu, Mehran Kazemi, Kathleen Kenealy, Amr Khalifa, Phoebe Kirk, Suraj Kothawade, Vitaly Kovalev, Neel Kovelamudi, Adam Kraft, Ravin Kumar, Harish Kuppam, Justin Lannin, Chen-Yu Lee, Seungji Lee, Dmitry Lepikhin, Dongdong Li, Qiujia Li, Valentin Liévin, Ethan Lin, Ziqian Lin, Casper Liu, Tianlin Liu, Tianqi Liu, Xin Liu, Mayank Lunayach, Min Ma, Gagan Madan, Andrii Maksai, Eric Malmi, Michal Matuszak, Daniel McDuff, Gaurav Menghani, Daniil Mirylenka, Karolis Misiunas, Vedant Misra, Andreea Mitran, Kareem Mohamed, Maksim Mukha, Eric Noland, James O'Donnell, Kate Olszewska, Bernett Orlando, Wanqiong Pan, Rina Panigrahy, Unnati Parekh, Chunjong Park, Eric Paskie, Liqian Peng, Bryce Petrini, Slav Petrov, Jonas Pfeiffer, Bilal Piot, Martyna Plomecka, Siim Poder, Octavio Ponce, Arijit Pramanik, David Racz, Anish Rajan, Michelle Ramanovich, Anand Rao, Marvin Ritter, Vitor Rodrigues, Evan Rosen, Mikołaj Rybiński, Noveen Sachdeva, Michaël E. Sander, Rohit Sathyanarayana, Sagar Savla, Samuel Schmidgall, Tal Schuster, Benoit Seguin, Andrew Sellergren, Aliaksei Severyn, Izhak Shafran, Dhruv Shah, Yuan Shangguan, Ashish Shenoy, Pradeep Shenoy, Rakesh Shivanna, Pauline Sho, Lucas Spangher, Wojciech Stokowiec, Tim Strother, Yao Su, Yinghao Sun, Mukund Sundararajan, Andrea Tacchetti, Mor Hazan Taege, Pouya Tafti, Chetan Tekur, Rahul Thapa, Madeleine Traverse, Lenart Treven, Tao Tu, Chien Te Tung, Petar Veličković, Malini Pooni Venkat, Sagar Gubbi Venkatesh, Vidya Venkiteswaran, Francesco Visin, Alex Vitvitskyi, Kiran Vodrahalli, Weiyi Wang, Xin Wang, Tris Warkentin, Jan Wassenberg, John Wieting, Lechao Xiao, Hao Xu, Yuhui Xu, Fuzhao Xue, Arun Yadav, Jun Yan, Antoine Yang, Lin Yang, Ming-Hsuan Yang, Ziyu Ying, Jae Hyeon Yoo, Sajjad Zafar, Fred Zhang, Jiageng Zhang, Jianyi Zhang, Xiaofan Zhang, Chao Zhao, David Zhou, Chen Zou, 2026

http://arxiv.org/abs/2607.02770

2. Round and Round We Go! What Makes Rotary Positional Encodings Useful? — Federico Barbero et al., 2025

https://scholar.google.com/scholar?q=Round+and+Round+We+Go%21+What+Makes+Rotary+Positional+Encodings+Useful%3F

3. Do Transformers Need Three Projections? Systematic Study of QKV Variants — A. Kayyam, A. M. Gopal, Michael A. Lewis, 2026

https://scholar.google.com/scholar?q=Do+Transformers+Need+Three+Projections%3F+Systematic+Study+of+QKV+Variants

4. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023

https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding

5. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty — Yinghui Li, Fandong Wei, Cheng Zhang, Hongyang Zhang, 2024

https://scholar.google.com/scholar?q=EAGLE%3A+Speculative+Sampling+Requires+Rethinking+Feature+Uncertainty

6. RULER: What's the Real Context Size of Your Long-Context Language Models? — C.-P. Hsieh et al., 2024

https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F

7. OpenAI o1 System Card — OpenAI, 2024

https://scholar.google.com/scholar?q=OpenAI+o1+System+Card

8. Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning — Xinghao Chen et al., 2025

https://arxiv.org/abs/2505.16782

9. Latent Chain-of-Thought for Visual Reasoning — Guohao Sun et al., 2025

https://arxiv.org/abs/2510.23925

10. Sliding Window Attention Adaptation — Yijiong Yu et al., 2025

https://arxiv.org/abs/2512.10411

11. Short window attention enables long-term memorization — Loic Cabannes et al., 2025

https://arxiv.org/abs/2509.24552

12. Can I Buy Your KV Cache? — Luoyuan Zhang, 2026

https://arxiv.org/abs/2606.13361

13. VCoder: Versatile Vision Encoders for Multimodal Large Language Models — Jitesh Jain, Jianwei Yang, Humphrey Shi, 2023

https://arxiv.org/abs/2312.14233

14. AI Post Transformers: Do Transformers Need Three Projections? — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-11-do-transformers-need-three-projections-c227d6.mp3

15. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp3

16. AI Post Transformers: JETSPEC and Parallel Tree Speculative Decoding — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-27-jetspec-and-parallel-tree-speculative-de-3d144c.mp3

17. AI Post Transformers: Nemotron 3 Super Hybrid Mamba-Transformer MoE — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-19-nemotron-3-super-hybrid-mamba-transforme-31ac75.mp3

18. AI Post Transformers: Kimi K2.5 and Visual Agent Swarms — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-24-kimi-k25-and-visual-agent-swarms-7d04d7.mp3

19. AI Post Transformers: LPU Chip for Low-Latency LLM Inference — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-20-lpu-chip-for-low-latency-llm-inference-be13c3.mp3