AI Post Transformers

This episode examines "A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning" (Ross, Gordon, and Bagnell, AISTATS 2011), unpacking why naively training a policy via supervised learning on expert demonstrations produces errors that compound quadratically with task horizon rather than linearly. It traces this compounding-error problem through concrete analogues — a self-driving policy drifting off the expert's trajectory into unseen states, and the exposure bias later rediscovered in sequence-to-sequence language models trained with teacher forcing — showing the same closed-loop failure mode recurring across robotics, structured prediction (like part-of-speech tagging), and NLP. It also revisits the authors' own earlier fix, SMILe (2010), explaining why its stochastic mixture-of-policies approach required an impractically large number of iterations to approach linear regret. Listeners interested in the theoretical foundations connecting imitation learning, sequence generation, and online learning reductions will find this a clear walkthrough of a foundational result that anticipated problems later rediscovered independently in deep learning.

Sources:

1. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning — Stephane Ross, Geoffrey J. Gordon, J. Andrew Bagnell, 2010

http://arxiv.org/abs/1011.0686

2. Efficient Reductions for Imitation Learning — Stéphane Ross, J. Andrew Bagnell, 2010

https://scholar.google.com/scholar?q=Efficient+Reductions+for+Imitation+Learning

3. Search-based Structured Prediction (SEARN) — Hal Daumé III, John Langford, Daniel Marcu, 2009

https://scholar.google.com/scholar?q=Search-based+Structured+Prediction+%28SEARN%29

4. Apprenticeship Learning via Inverse Reinforcement Learning — Pieter Abbeel, Andrew Y. Ng, 2004

https://scholar.google.com/scholar?q=Apprenticeship+Learning+via+Inverse+Reinforcement+Learning

5. Generative Adversarial Imitation Learning — Jonathan Ho, Stefano Ermon, 2016

https://scholar.google.com/scholar?q=Generative+Adversarial+Imitation+Learning

6. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data — John Lafferty, Andrew McCallum, Fernando Pereira, 2001

https://scholar.google.com/scholar?q=Conditional+Random+Fields%3A+Probabilistic+Models+for+Segmenting+and+Labeling+Sequence+Data

7. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks — Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer, 2015

https://scholar.google.com/scholar?q=Scheduled+Sampling+for+Sequence+Prediction+with+Recurrent+Neural+Networks

8. Approximately Optimal Approximate Reinforcement Learning — Sham Kakade, John Langford, 2002

https://scholar.google.com/scholar?q=Approximately+Optimal+Approximate+Reinforcement+Learning

9. Error-Correcting Tournaments / Error Limiting Reductions — Alina Beygelzimer, Varsha Dani, Tom Hayes, John Langford, Bianca Zadrozny (various papers in this line, ~2005), 2005

https://scholar.google.com/scholar?q=Error-Correcting+Tournaments+%2F+Error+Limiting+Reductions

10. Relating Reinforcement Learning Performance to Classification Performance — John Langford, Bianca Zadrozny, 2005

https://scholar.google.com/scholar?q=Relating+Reinforcement+Learning+Performance+to+Classification+Performance

11. Efficient Reductions for Imitation Learning (SMILe, Forward Training) — Stéphane Ross, J. Andrew Bagnell, 2010

https://scholar.google.com/scholar?q=Efficient+Reductions+for+Imitation+Learning+%28SMILe%2C+Forward+Training%29

12. Approximately Optimal Approximate Reinforcement Learning (CPI) — Sham Kakade, John Langford, 2002

https://scholar.google.com/scholar?q=Approximately+Optimal+Approximate+Reinforcement+Learning+%28CPI%29

13. Error Limiting Reductions Between Classification Tasks — Alina Beygelzimer, Varsha Dani, Tom Hayes, John Langford, Bianca Zadrozny, 2005

https://scholar.google.com/scholar?q=Error+Limiting+Reductions+Between+Classification+Tasks

14. Max-Margin Markov Networks — Ben Taskar, Carlos Guestrin, Daphne Koller, 2003

https://scholar.google.com/scholar?q=Max-Margin+Markov+Networks

15. On the Generalization Ability of Online Strongly Convex Programming Algorithms — Sham Kakade, Ambuj Tewari, 2009

https://scholar.google.com/scholar?q=On+the+Generalization+Ability+of+Online+Strongly+Convex+Programming+Algorithms

Interactive Visualization: Reduction of Imitation Learning to No-Regret Online Learning

This episode examines Piper, a training system from Oak Ridge National Laboratory designed to fix catastrophic GPU underutilization in large-scale Mixture-of-Experts training, where the leading framework X-MoE hits only about 5% utilization on a 545-billion-parameter model. The discussion traces MoE's evolution from GShard and Switch Transformer's coarse-grained experts to DeepSeek-MoE's fine-grained approach with hundreds of small experts, and explains why expert parallelism's all-to-all communication becomes a severe bottleneck on Frontier's Dragonfly network topology, where bandwidth varies sharply with GPU distance. Piper's core innovation is repurposing pipeline parallelism, normally used only to split layers across dense models, to also confine expensive expert-parallel communication within small, physically local GPU groups arranged in a pipeline-by-expert-parallel grid. The conversation details how an analytical resource model prunes infeasible configurations for memory and communication cost before a micro-benchmarking pass measures real hardware throughput to select the optimal setup, claiming a two-to-three-and-a-half-times utilization improvement. Listeners interested in the practical gap between theoretical FLOPs and real supercomputer throughput will find this a concrete look at what it takes to make trillion-parameter training economically viable on shared HPC infrastructure rather than purpose-built AI clusters.

Sources:

1. Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism — Sajal Dash, Feiyi Wang, 2026

http://arxiv.org/abs/2605.05049

2. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen, 2020

https://scholar.google.com/scholar?q=GShard%3A+Scaling+Giant+Models+with+Conditional+Computation+and+Automatic+Sharding

3. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — William Fedus, Barret Zoph, Noam Shazeer, 2021

https://scholar.google.com/scholar?q=Switch+Transformers%3A+Scaling+to+Trillion+Parameter+Models+with+Simple+and+Efficient+Sparsity

4. Tutel: Adaptive Mixture-of-Experts at Scale — Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, Joe Chau, Peng Cheng, Fan Yang, Mao Yang, Yongqiang Xiong, 2022 (updated 2023)

https://scholar.google.com/scholar?q=Tutel%3A+Adaptive+Mixture-of-Experts+at+Scale

5. FasterMoE: Modeling and Optimizing Training of Large-Scale Dynamic Pre-Trained Models — Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, Jie Tang, 2022

https://scholar.google.com/scholar?q=FasterMoE%3A+Modeling+and+Optimizing+Training+of+Large-Scale+Dynamic+Pre-Trained+Models

6. FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement — Xiaonan Nie, Xupeng Miao, Zilong Wang, Zichao Yang, Jilong Xue, Lingxiao Ma, Gang Cao, Bin Cui, 2023 (SIGMOD/VLDB)

https://scholar.google.com/scholar?q=FlexMoE%3A+Scaling+Large-scale+Sparse+Pre-trained+Model+Training+via+Dynamic+Device+Placement

7. SmartMoE: Efficiently Training Sparsely-Activated Models through Combining Offline and Online Parallelization — Mingshu Zhai, Jiaao He, Zixuan Ma, Zan Zong, Runqing Zhang, Jidong Zhai, 2023 (USENIX ATC)

https://scholar.google.com/scholar?q=SmartMoE%3A+Efficiently+Training+Sparsely-Activated+Models+through+Combining+Offline+and+Online+Parallelization

8. Lina: Enabling Sparse-Aware Distributed Training via Efficient Communication Scheduling — Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, Hong Xu, 2023 (SIGCOMM/ASPLOS)

https://scholar.google.com/scholar?q=Lina%3A+Enabling+Sparse-Aware+Distributed+Training+via+Efficient+Communication+Scheduling

9. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning — Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He, 2021 (SC)

https://scholar.google.com/scholar?q=ZeRO-Infinity%3A+Breaking+the+GPU+Memory+Wall+for+Extreme+Scale+Deep+Learning

Interactive Visualization: Piper: Fixing GPU Utilization in MoE Training on Frontier

This episode explores MoM (Mixture-of-Memories), a linear sequence modeling architecture from researchers at Shanghai AI Laboratory and collaborating universities that tackles a core weakness in efficient Transformer alternatives: their tendency to forget information from earlier in a sequence. The discussion traces the trade-off at the heart of the field — Transformers preserve every token via a growing key-value cache at quadratic cost, while linear models like Mamba and RWKV compress everything into a single fixed-size memory state, trading recall precision for constant-time efficiency. MoM's proposed fix draws on two distinct sources: a neuroscience-inspired analogy to how the hippocampus uses separate oscillatory channels to keep simultaneous memories from blending together, and the Mixture-of-Experts routing mechanism, applied here to memory states rather than feed-forward layers. The result is an architecture with multiple independent memory slots plus a shared accumulating memory, with a lightweight router directing each token to the appropriate slot. Listeners interested in efficient sequence modeling, long-context recall, or the cross-pollination between neuroscience and deep learning architecture design will find the mechanics of this capacity-versus-interference problem — and its proposed solution — a compelling deep dive.

Sources:

1. MoM: Linear Sequence Modeling with Mixture-of-Memories — Jusen Du, Weigao Sun, Disen Lan, Jiaxi Hu, Yu Cheng, 2025

http://arxiv.org/abs/2502.13685v4

2. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret, 2020

https://scholar.google.com/scholar?q=Transformers+are+RNNs%3A+Fast+Autoregressive+Transformers+with+Linear+Attention

3. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023 (updated 2024)

https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces

4. Gated Linear Attention Transformers with Hardware-Efficient Training — Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, Yoon Kim, 2023

https://scholar.google.com/scholar?q=Gated+Linear+Attention+Transformers+with+Hardware-Efficient+Training

5. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean, 2017

https://scholar.google.com/scholar?q=Outrageously+Large+Neural+Networks%3A+The+Sparsely-Gated+Mixture-of-Experts+Layer

6. Gated Delta Networks: Improving Mamba2 with Delta Rule — Songlin Yang, Jan Kautz, Ali Hatamizadeh, 2024

https://scholar.google.com/scholar?q=Gated+Delta+Networks%3A+Improving+Mamba2+with+Delta+Rule

7. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2024

https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time

8. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — William Fedus, Barret Zoph, Noam Shazeer, 2022

https://scholar.google.com/scholar?q=Switch+Transformers%3A+Scaling+to+Trillion+Parameter+Models+with+Simple+and+Efficient+Sparsity

9. Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff (BASED) — Simran Arora, Sabri Eyuboglu, Michael Zhang, et al., 2024

https://scholar.google.com/scholar?q=Simple+Linear+Attention+Language+Models+Balance+the+Recall-Throughput+Tradeoff+%28BASED%29

10. Gated Slot Attention for Efficient Linear-Time Sequence Modeling — Yu Zhang, Songlin Yang, Ruijie Zhu, et al., 2024

https://scholar.google.com/scholar?q=Gated+Slot+Attention+for+Efficient+Linear-Time+Sequence+Modeling

Interactive Visualization: MoM: Mixture-of-Memories Fixes Linear Attention Recall

This episode examines a Lightmatter-authored paper claiming photonic interconnects can cut inference prefill latency by up to 8.5x for long-context, Mixture-of-Experts workloads. The hosts unpack why prefill has become a dominant cost center as agentic coding pushes median prompt lengths toward 96K tokens, and explain the technical distinction between compute-bound prefill and memory-bandwidth-bound decode. They dig into the physics behind the claim: copper's one-meter reach limit at 224 Gbps per lane forces multi-rack scale-out, while 3D-integrated photonics decouples I/O from a chip's shoreline, enabling far higher bandwidth density. Throughout, the co-hosts push back on taking the headline multiplier at face value, stressing that the bandwidth specs come from Lightmatter's own published sheet and the performance gains from a simulator the company itself built and controls. It's a useful listen for anyone wanting a grounded, skeptical walkthrough of interconnect physics versus vendor-reported benchmarks in AI infrastructure claims.

Sources:

1. Scaling Inference Prefill with High-Radix Photonic Interconnects — Arulselvan Madhavan, Peter Carson, Taylor Groves, Thomas Graham, 2026

http://arxiv.org/abs/2609.01821

2. Accelerating Frontier MoE Training with 3D Integrated Optics — M. Bernadskiy, P. Carson, T. Graham, T. Groves, H. J. Lee, E. Yeh, 2025

https://scholar.google.com/scholar?q=Accelerating+Frontier+MoE+Training+with+3D+Integrated+Optics

3. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized LLM Serving — Y. Zhong, S. Liu, et al., 2024

https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-Optimized+LLM+Serving

4. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference — A. Agrawal, N. Kedia, A. Panwar, et al., 2024 (OSDI)

https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference

5. Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast LLM Inference — Q. Li, B. Zhang, L. Ye, et al., 2024

https://scholar.google.com/scholar?q=Flash+Communication%3A+Reducing+Tensor+Parallelization+Bottleneck+for+Fast+LLM+Inference

6. DeepSeek-V3 Technical Report — DeepSeek-AI, 2024

https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report

Interactive Visualization: Photonic Interconnects and the Race to Cut Prefill Latency

This episode revisits "The Case for Learned Index Structures" by Tim Kraska and coauthors from MIT and Google, which proposes replacing classic data structures like B-Trees, hash maps, and Bloom filters with small trained neural networks. The discussion covers how a B-Tree traversal is mathematically equivalent to estimating a cumulative distribution function, and how a tiny two-layer model can predict a key's position directly, turning a branchy O(log N) search into near-constant-time arithmetic suited to SIMD and GPU hardware. It also digs into how the same idea extends to hash functions tuned to real key distributions and to Bloom filters reframed as classifiers, complete with a backup filter to catch the model's false negatives. The hosts push back on each other over the paper's headline claims of 70% faster lookups and order-of-magnitude memory savings, debating whether results measured on static, read-only, in-memory workloads generalize or overstate the case. Listeners interested in database internals, the intersection of machine learning and systems design, or the tradeoffs between hand-engineered and learned structures will find the back-and-forth a useful gut check on a widely cited but contested idea.

Sources:

1. The Case for Learned Index Structures — Tim Kraska, Alex Beutel, Ed H. Chi, Jeffrey Dean, Neoklis Polyzotis, 2017

http://arxiv.org/abs/1712.01208

2. SOSD: A Benchmark for Learned Indexes — Andreas Kipf, Ryan Marcus, Alexander van Renen, Mihail Stoian, Alfons Kemper, Tim Kraska, Thomas Neumann, 2019

https://scholar.google.com/scholar?q=SOSD%3A+A+Benchmark+for+Learned+Indexes

3. ALEX: An Updatable Adaptive Learned Index — Jialin Ding, Umar Farooq Minhas, Jia Yu, Chi Wang, Jaeyoung Do, Yinan Li, Hantian Zhang, Badrish Chandramouli, Johannes Gehrke, Donald Kossmann, David Lomet, Tim Kraska, 2020

https://scholar.google.com/scholar?q=ALEX%3A+An+Updatable+Adaptive+Learned+Index

4. The PGM-index: A Fully-Dynamic Compressed Learned Index with Provable Worst-Case Bounds — Paolo Ferragina, Giorgio Vinciguerra, 2020

https://scholar.google.com/scholar?q=The+PGM-index%3A+A+Fully-Dynamic+Compressed+Learned+Index+with+Provable+Worst-Case+Bounds

5. FITing-Tree: A Data-aware Index Structure — Alex Galakatos, Michael Markovitch, Carsten Binnig, Rodrigo Fonseca, Tim Kraska, 2019

https://scholar.google.com/scholar?q=FITing-Tree%3A+A+Data-aware+Index+Structure

6. Organization and Maintenance of Large Ordered Indexes — Rudolf Bayer, Edward M. McCreight, 1972

https://scholar.google.com/scholar?q=Organization+and+Maintenance+of+Large+Ordered+Indexes

7. The Ubiquitous B-Tree — Douglas Comer, 1979

https://scholar.google.com/scholar?q=The+Ubiquitous+B-Tree

8. Making B+-Trees Cache Conscious in Main Memory — Jun Rao, Kenneth A. Ross, 2000

https://scholar.google.com/scholar?q=Making+B%2B-Trees+Cache+Conscious+in+Main+Memory

9. Space/Time Trade-offs in Hash Coding with Allowable Errors — Burton H. Bloom, 1970

https://scholar.google.com/scholar?q=Space%2FTime+Trade-offs+in+Hash+Coding+with+Allowable+Errors

10. Network Applications of Bloom Filters: A Survey — Andrei Broder, Michael Mitzenmacher, 2004

https://scholar.google.com/scholar?q=Network+Applications+of+Bloom+Filters%3A+A+Survey

11. A Model for Learned Bloom Filters and Optimizing by Sandwiching — Michael Mitzenmacher, 2018

https://scholar.google.com/scholar?q=A+Model+for+Learned+Bloom+Filters+and+Optimizing+by+Sandwiching

12. Cuckoo Filter: Practically Better Than Bloom — Bin Fan, Dave G. Andersen, Michael Kaminsky, Michael D. Mitzenmacher, 2014

https://scholar.google.com/scholar?q=Cuckoo+Filter%3A+Practically+Better+Than+Bloom

13. A model for learned bloom filters and related structures — M. Mitzenmacher, 2018

https://scholar.google.com/scholar?q=A+model+for+learned+bloom+filters+and+related+structures

14. A-Tree: A Bounded Approximate Index Structure — A. Galakatos, M. Markovitch, C. Binnig, R. Fonseca, T. Kraska, 2018

https://scholar.google.com/scholar?q=A-Tree%3A+A+Bounded+Approximate+Index+Structure

15. FAST: Fast Architecture Sensitive Tree Search on Modern CPUs and GPUs — C. Kim, J. Chhugani, N. Satish, et al., 2010

https://scholar.google.com/scholar?q=FAST%3A+Fast+Architecture+Sensitive+Tree+Search+on+Modern+CPUs+and+GPUs

16. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — N. Shazeer, A. Mirhoseini, K. Maziarz, et al., 2017

https://scholar.google.com/scholar?q=Outrageously+Large+Neural+Networks%3A+The+Sparsely-Gated+Mixture-of-Experts+Layer

Interactive Visualization: The Case for Learned Index Structures Revisited

This episode examines "AI Coaching for Accelerating Human Skill Development with Reinforcement Learning," a University of Pennsylvania and Johns Hopkins paper that challenges the assumption that AI assistance always benefits learners, arguing that copilots optimized for immediate task success can quietly prevent people from ever mastering a skill independently. The discussion traces the tension between over-assistance, which produces clean performance but no learning, and under-assistance, which produces uninstructive failure, connecting this to Manu Kapur's productive-failure research and decades of shared-control work from Dragan, Srinivasa, Reddy, and Levine. The core innovation discussed is framing coaching as a non-cooperative dynamic game rather than a cooperative one: the learner optimizes for immediate performance while the coach is trained on Value of Independence, a counterfactual measure of how well the human would perform if the AI were removed entirely. The hosts debate whether this framing is truly adversarial or just a shared goal on different timescales, concluding the reward structures can genuinely conflict moment-to-moment. The episode also situates the work against closest prior art, including the Cyber Racing Coach's fixed assistance-decay schedule, highlighting why a skill-aware, game-theoretic approach marks a meaningful departure from prior fading-assistance methods.

Sources:

1. AI Coaching for Accelerating Human Skill Development with Reinforcement Learning — Wei Wang, Enlin Gu, Antonio Loquercio, Haimin Hu, Rahul Mangharam, 2026

http://arxiv.org/abs/2606.25337v1

2. A Policy-Blending Formalism for Shared Control — Anca D. Dragan, Siddhartha S. Srinivasa, 2013

https://scholar.google.com/scholar?q=A+Policy-Blending+Formalism+for+Shared+Control

3. Shared Autonomy via Hindsight Optimization — Shervin Javdani, Siddhartha S. Srinivasa, J. Andrew Bagnell, 2015

https://scholar.google.com/scholar?q=Shared+Autonomy+via+Hindsight+Optimization

4. Shared Autonomy via Deep Reinforcement Learning — Siddharth Reddy, Anca D. Dragan, Sergey Levine, 2018

https://scholar.google.com/scholar?q=Shared+Autonomy+via+Deep+Reinforcement+Learning

5. Highway Driving with a Semi-Autonomous Alliance of Human and Machine (Shared Control for Highway Driving) — Jake Brawer / Vaibhav Gupta / related shared-control-for-driving lineage (e.g., Broad, Arkin, Ratliff, Howard, Argall), 2017-2019

https://scholar.google.com/scholar?q=Highway+Driving+with+a+Semi-Autonomous+Alliance+of+Human+and+Machine+%28Shared+Control+for+Highway+Driving%29

6. Dynamic Noncooperative Game Theory — Tamer Basar, Geert Jan Olsder, 1982 (2nd ed. 1999)

https://scholar.google.com/scholar?q=Dynamic+Noncooperative+Game+Theory

7. Planning for Autonomous Cars that Leverage Effects on Human Actions — Dorsa Sadigh, S. Shankar Sastry, Sanjit A. Seshia, Anca D. Dragan, 2016

https://scholar.google.com/scholar?q=Planning+for+Autonomous+Cars+that+Leverage+Effects+on+Human+Actions

8. Efficient Iterative Linear-Quadratic Approximations for Nonlinear Multi-Player General-Sum Differential Games (ILQGames) — David Fridovich-Keil, Ellis Ratner, Lasse Peters, Anca D. Dragan, Claire J. Tomlin, 2020

https://scholar.google.com/scholar?q=Efficient+Iterative+Linear-Quadratic+Approximations+for+Nonlinear+Multi-Player+General-Sum+Differential+Games+%28ILQGames%29

9. Who Plays First? Optimizing the Order of Play in Stackelberg Games with Many Robots (and related Stackelberg human-AV interaction work) — Haimin Hu, Zixu Zhang, Kensuke Nakamura, Andrea Bajcsy, Jaime F. Fisac (and related Fisac/Dragan lineage), 2023

https://scholar.google.com/scholar?q=Who+Plays+First%3F+Optimizing+the+Order+of+Play+in+Stackelberg+Games+with+Many+Robots+%28and+related+Stackelberg+human-AV+interaction+work%29

10. Legibility and Predictability of Robot Motion — Anca D. Dragan, Kenton C.T. Lee, Siddhartha S. Srinivasa, 2013

https://scholar.google.com/scholar?q=Legibility+and+Predictability+of+Robot+Motion

11. The Physical Presence of a Robot Tutor Increases Cognitive Learning Gains — Daniel Leyzberg, Samuel Spaulding, Mariya Toneva, Brian Scassellati, 2012

https://scholar.google.com/scholar?q=The+Physical+Presence+of+a+Robot+Tutor+Increases+Cognitive+Learning+Gains

12. Trajectory Deformations from Physical Human-Robot Interaction — Dylan P. Losey, Marcia K. O'Malley, 2017

https://scholar.google.com/scholar?q=Trajectory+Deformations+from+Physical+Human-Robot+Interaction

13. Productive Failure — Manu Kapur, 2008

https://scholar.google.com/scholar?q=Productive+Failure

14. Challenge Point: A Framework for Conceptualizing the Effects of Various Practice Conditions in Motor Learning — Mark A. Guadagnoli, Timothy D. Lee, 2004

https://scholar.google.com/scholar?q=Challenge+Point%3A+A+Framework+for+Conceptualizing+the+Effects+of+Various+Practice+Conditions+in+Motor+Learning

15. The Role of Tutoring in Problem Solving — David Wood, Jerome S. Bruner, Gail Ross, 1976

https://scholar.google.com/scholar?q=The+Role+of+Tutoring+in+Problem+Solving

16. The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems — Kurt VanLehn, 2011

https://scholar.google.com/scholar?q=The+Relative+Effectiveness+of+Human+Tutoring%2C+Intelligent+Tutoring+Systems%2C+and+Other+Tutoring+Systems

17. Cyber Racing Coach: A Haptic Shared Control Framework for Teaching Advanced Driving Skills — C. Shen et al., 2025

https://scholar.google.com/scholar?q=Cyber+Racing+Coach%3A+A+Haptic+Shared+Control+Framework+for+Teaching+Advanced+Driving+Skills

18. Shared Autonomy for Proximal Teaching (Z-COACH) — M. Srivastava et al., 2025

https://scholar.google.com/scholar?q=Shared+Autonomy+for+Proximal+Teaching+%28Z-COACH%29

19. AssistanceZero: Scalably Solving Assistance Games — C. Laidlaw et al., 2025

https://scholar.google.com/scholar?q=AssistanceZero%3A+Scalably+Solving+Assistance+Games

20. Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development — J. Kulveit et al., 2025

https://scholar.google.com/scholar?q=Gradual+Disempowerment%3A+Systemic+Existential+Risks+from+Incremental+AI+Development

21. Champion-Level Drone Racing Using Deep Reinforcement Learning — E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, D. Scaramuzza, 2023

https://scholar.google.com/scholar?q=Champion-Level+Drone+Racing+Using+Deep+Reinforcement+Learning

22. Human-Robot Mutual Adaptation in Collaborative Tasks: Models and Experiments — S. Nikolaidis, D. Hsu, S. Srinivasa, 2017

https://scholar.google.com/scholar?q=Human-Robot+Mutual+Adaptation+in+Collaborative+Tasks%3A+Models+and+Experiments

23. Does Using Artificial Intelligence Assistance Accelerate Skill Decay and Hinder Skill Development Without Performers' Awareness? — B. N. Macnamara et al., 2024

https://scholar.google.com/scholar?q=Does+Using+Artificial+Intelligence+Assistance+Accelerate+Skill+Decay+and+Hinder+Skill+Development+Without+Performers%27+Awareness%3F

Interactive Visualization: AI Coaching That Preserves Human Skill Development

This episode surveys "Recursive Self-Improvement in AI," a paper by Mingguang Chen and colleagues that classifies 1,250 papers on how AI systems attempt to improve themselves. The discussion establishes precise definitions distinguishing agents, harnesses, and evaluators, then draws a critical line between bounded self-refinement (improvement against a fixed external evaluator) and open-ended recursive self-improvement (where the system also modifies its own criteria for success). It traces the intellectual lineage from I.J. Good's 1965 "intelligence explosion" concept through Schmidhuber's provably-optimal but practically unusable Gödel machines, showing how the field traded mathematical proof for empirically checkable but weaker signals like benchmarks and tests. The episode also introduces a verification hierarchy ranking formal verifiers above execution feedback, learned judges, and self-assessment, citing key findings that scoring reasoning steps beats scoring final answers, and that language models largely cannot self-correct without external feedback. Listeners interested in AI safety, agent architectures, or the theoretical limits of self-improving systems will find this a rigorous framework for cutting through loose talk about "self-improving AI."

Sources:

1. Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops — Mingguang Chen, Licheng Wang, Bo Qu, 2026

http://arxiv.org/abs/2607.07663

2. Speculations Concerning the First Ultraintelligent Machine — I. J. Good, 1965

https://scholar.google.com/scholar?q=Speculations+Concerning+the+First+Ultraintelligent+Machine

3. Gödel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements — Jürgen Schmidhuber, 2003 (extended 2006)

https://scholar.google.com/scholar?q=G%C3%B6del+Machines%3A+Self-Referential+Universal+Problem+Solvers+Making+Provably+Optimal+Self-Improvements

4. Self-Rewarding Language Models — Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, Jason Weston (Meta AI/NYU), 2024

https://scholar.google.com/scholar?q=Self-Rewarding+Language+Models

5. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents — Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, Jeff Clune (Sakana AI / UBC), 2025

https://scholar.google.com/scholar?q=Darwin+G%C3%B6del+Machine%3A+Open-Ended+Evolution+of+Self-Improving+Agents

6. AI models collapse when trained on recursively generated data — Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, Yarin Gal, 2024

https://scholar.google.com/scholar?q=AI+models+collapse+when+trained+on+recursively+generated+data

7. Large Language Models Cannot Self-Correct Reasoning Yet — Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, Denny Zhou, 2024 (ICLR)

https://scholar.google.com/scholar?q=Large+Language+Models+Cannot+Self-Correct+Reasoning+Yet

8. Let's Verify Step by Step — Hunter Lightman et al., 2023

https://scholar.google.com/scholar?q=Let%27s+Verify+Step+by+Step

9. Scaling Laws for Reward Model Overoptimization — Leo Gao, John Schulman, Jacob Hilton, 2022/2023

https://scholar.google.com/scholar?q=Scaling+Laws+for+Reward+Model+Overoptimization

10. Recursive Self-Improvement (Anthropic Institute blog post) — Anthropic, 2026

https://scholar.google.com/scholar?q=Recursive+Self-Improvement+%28Anthropic+Institute+blog+post%29

Interactive Visualization: The Great Debate: Bounded Refinement vs Open-Ended Self-Improvement

This episode examines "Post-Training Language Models for Gold-Medal Performance in Coding Competitions" from NVIDIA, whose Ultra-CC system scored 535.4 out of 600 at the 2026 International Olympiad in Informatics — beating not just the gold cutoff of 361.12 but the top human competitor's 498.27, live and under real contest conditions. The discussion breaks down why IOI-style problems are a harder test than typical coding benchmarks, since they demand inventing novel algorithms rather than recognizing familiar patterns, with partial credit across subtasks revealing the difference between no idea, the right idea with wrong complexity, and a fully correct solution. It covers the four-stage training pipeline behind the result: curating 22,000 competitive programming problems, distilling teacher-model reasoning traces for supervised fine-tuning, applying reinforcement learning from verifiable code-execution rewards via GRPO, and using a test-time strategy called GenCorrect to refine candidate solutions before submission. The episode also contrasts the two models built, a smaller mixture-of-experts Nano-CC with RL and a much larger Ultra-CC trained only with SFT, tracing the underlying architecture and training ideas back to foundational work like Shazeer's mixture-of-experts paper, Switch Transformer, InstructGPT, and DeepSeek's R1. Listeners interested in how far AI reasoning has come against elite human problem-solvers will find the specific mechanics behind this milestone result compelling.

Sources:

1. Post-Training Language Models for Gold-Medal Performance in Coding Competitions — Aleksander Ficek, Sean Narenthiran, Mehrzad Samadi, Somshubra Majumdar, Boris Ginsburg, 2026

http://arxiv.org/abs/2609.02849

2. Competitive programming with large reasoning models (o1-ioi, o3) — OpenAI et al., 2025

https://scholar.google.com/scholar?q=Competitive+programming+with+large+reasoning+models+%28o1-ioi%2C+o3%29

3. Scaling test-time compute to achieve IOI gold medal with open-weight models (GenCluster) — Mehrzad Samadi, Aleksander Ficek, Sean Narenthiran, et al., 2026

https://scholar.google.com/scholar?q=Scaling+test-time+compute+to+achieve+IOI+gold+medal+with+open-weight+models+%28GenCluster%29

4. Nemotron-cascade 2: Post-training LLMs with cascade RL and multi-domain on-policy distillation — Zhuolin Yang et al., 2026

https://scholar.google.com/scholar?q=Nemotron-cascade+2%3A+Post-training+LLMs+with+cascade+RL+and+multi-domain+on-policy+distillation

5. Large Language Models Cannot Self-Correct Reasoning Yet — Jie Huang, Xinyun Chen, Swaroop Mishra, et al., 2024

https://scholar.google.com/scholar?q=Large+Language+Models+Cannot+Self-Correct+Reasoning+Yet

6. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (DeepSeek-V3.2-Speciale) — DeepSeek-AI, 2025

https://scholar.google.com/scholar?q=DeepSeek-V3.2%3A+Pushing+the+Frontier+of+Open+Large+Language+Models+%28DeepSeek-V3.2-Speciale%29

Interactive Visualization: NVIDIA Ultra-CC Beats Top Human at IOI 2026

This episode examines MegaTrain, a method for full-precision training of 100-billion-parameter-plus language models on a single H200 GPU paired with 1.5 terabytes of host RAM, from a Notre Dame and Lehigh University team. The discussion centers on why memory, not compute, is the real bottleneck for most researchers, citing a survey showing only two of 167 surveyed U.S. universities average more than one H100 per student, while post-training work like instruction tuning and alignment increasingly demands full parameter and optimizer states without full pretraining-scale hardware. The hosts walk through the GPU memory hierarchy — from on-chip SRAM through HBM, host DDR5, and NVMe — and the 12-bytes-per-parameter cost of Adam optimizer state that makes a 70B model require 840 gigabytes of persistent storage. They contrast MegaTrain's approach with prior offloading systems like ZeRO-Offload and ZeRO-Infinity, highlighting the key architectural inversion: host memory becomes the authoritative store for all parameters and optimizer state, while GPU HBM is reduced to a transient scratchpad streaming one layer at a time across PCIe. Listeners interested in democratizing large-model training on constrained hardware will find the systems-level tradeoffs and pointed debate over whether this is genuinely novel or a repackaging of known offloading techniques particularly engaging.

Sources:

1. MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU — Zhengqing Yuan, Hanchi Sun, Lichao Sun, Yanfang Ye, 2026

http://arxiv.org/abs/2604.05091

2. vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design — Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, Stephen W. Keckler, 2016

https://scholar.google.com/scholar?q=vDNN%3A+Virtualized+Deep+Neural+Networks+for+Scalable%2C+Memory-Efficient+Neural+Network+Design

3. ZeRO-Offload: Democratizing Billion-Scale Model Training — Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He, 2021

https://scholar.google.com/scholar?q=ZeRO-Offload%3A+Democratizing+Billion-Scale+Model+Training

4. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning — Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He, 2021

https://scholar.google.com/scholar?q=ZeRO-Infinity%3A+Breaking+the+GPU+Memory+Wall+for+Extreme+Scale+Deep+Learning

5. JAX: composable transformations of Python+NumPy programs — James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, Skye Wanderman-Milne, 2018

https://scholar.google.com/scholar?q=JAX%3A+composable+transformations+of+Python%2BNumPy+programs

6. Chainer: A Next-Generation Open Source Framework for Deep Learning — Seiya Tokui, Kenta Oono, Shohei Hido, Justin Clayton, 2015

https://scholar.google.com/scholar?q=Chainer%3A+A+Next-Generation+Open+Source+Framework+for+Deep+Learning

7. Automatic Differentiation in PyTorch — Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, et al., 2017

https://scholar.google.com/scholar?q=Automatic+Differentiation+in+PyTorch

8. Training Deep Nets with Sublinear Memory Cost — Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin, 2016

https://scholar.google.com/scholar?q=Training+Deep+Nets+with+Sublinear+Memory+Cost

9. Reducing Activation Recomputation in Large Transformer Models — Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, Bryan Catanzaro, 2022

https://scholar.google.com/scholar?q=Reducing+Activation+Recomputation+in+Large+Transformer+Models

10. Efficient Rematerialization for Deep Networks — Ravi Kumar, Manish Purohit, Zoya Svitkina, Erik Vee, Joshua Wang, 2019

https://scholar.google.com/scholar?q=Efficient+Rematerialization+for+Deep+Networks

11. Ratel: Optimizing Holistic Data Movement to Fine-Tune 100B Model on a Consumer GPU — Changyue Liao, Mo Sun, Zihan Yang, Jun Xie, Kaiqi Chen, Binhang Yuan, Fei Wu, Zeke Wang, 2025

https://scholar.google.com/scholar?q=Ratel%3A+Optimizing+Holistic+Data+Movement+to+Fine-Tune+100B+Model+on+a+Consumer+GPU

12. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, et al., 2023

https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU

13. PatrickStar: Parallel Training of Pre-trained Models via a Chunk-based Memory Management — Jiarui Fang, Zilin Zhu, Shenggui Li, Hui Su, Yang Yu, Jie Zhou, Yang You, 2022

https://scholar.google.com/scholar?q=PatrickStar%3A+Parallel+Training+of+Pre-trained+Models+via+a+Chunk-based+Memory+Management

14. GaLore / other optimizer-state-aware memory-efficient training methods — Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, Yuandong Tian, 2024

https://scholar.google.com/scholar?q=GaLore+%2F+other+optimizer-state-aware+memory-efficient+training+methods

Interactive Visualization: MegaTrain: Training 120B-Parameter Models on a Single GPU

This episode examines a comparative study of three KV cache management strategies for LLM inference — vLLM's PagedAttention memory management, H2O's static sparsification, and InfiniGen's dynamic CPU-offload selection — tested side by side on identical hardware for the first time. The standout finding: both H2O and InfiniGen hit out-of-memory errors around 10,000 tokens, less than 10% of the 128K context window modern models claim to support, revealing that many eviction-based approaches can't even survive prefill on long documents. The discussion traces why KV caches exist at all (avoiding quadratic recomputation cost), how Grouped Query Attention reduces steady-state cache size but does nothing for the transient attention-score matrix that must be materialized during prefill to decide what to evict, and why that structural gap explains the paradigms' divergent failure modes. Testing spans Llama-3.1-8B and 70B, GPT-OSS-20B, and multiple benchmark datasets across four H100 GPUs. Listeners interested in the practical limits of long-context LLM serving — and why architectural tricks like GQA don't fully solve the memory problem — will find the paper's empirical exposure of these failure points compelling.

Sources:

1. Comparative Characterization of KV Cache Management Strategies for LLM Inference — Oteo Mamo, Olga Kogiou, Hyunjin Yi, Weikuan Yu, 2026

http://arxiv.org/abs/2604.05012

2. Efficient Memory Management for Large Language Model Serving with PagedAttention — W. Kwon et al., 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Z. Zhang et al., 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models

4. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — W. Lee, J. Lee, J. Seo, J. Sim, 2024

https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management

5. Characterizing the Behavior and Impact of KV Caching on Transformer Inferences under Concurrency — J. Ye, J. Cernuda, A. Maurya, X.-H. Sun, A. Kougkas, B. Nicolae, 2025

https://scholar.google.com/scholar?q=Characterizing+the+Behavior+and+Impact+of+KV+Caching+on+Transformer+Inferences+under+Concurrency

6. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — G. Xiao et al., 2023

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29

7. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — J. Tang et al., 2024

https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference

8. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization — C. Hooper et al., 2024

https://scholar.google.com/scholar?q=KVQuant%3A+Towards+10+Million+Context+Length+LLM+Inference+with+KV+Cache+Quantization

Interactive Visualization: KV Cache Management Faces Its First Head-to-Head Test