AI Post Transformers

This episode explores Tensor Cache, a memory architecture for transformer inference that addresses the tradeoff between unbounded KV cache growth and the "amnesia" problem of sliding-window attention. Rather than deleting evicted tokens, the approach folds them into a fixed-size associative memory matrix using an outer-product mechanism rooted in Schmidhuber's 1992 fast-weight memory concept, combined with a linear-attention identity from Schlag et al. that lets a single matrix multiply approximate attention over everything compressed into it. The discussion details the two-tier design—an exact local ring-buffer cache (L1) paired with a compressed overflow matrix (L2)—and clarifies how it differs from related approaches like mLSTM, Infini-attention, RetNet, Mamba, and importance-based eviction schemes such as H2O and SnapKV. Listeners get a clear picture of the learned, per-head gating and decay mechanisms that control how much compressed memory blends into each layer's output, along with a look at the practical challenge of training this eviction-conditioned system efficiently across batches without simulating token-by-token eviction at every gradient step.

Sources:

1. Tensor Cache: Eviction-conditioned Associative Memory for Transformers — Kabir Swain, Sijie Han, Daniel Karl I. Weidele, Mauro Martino, Antonio Torralba, 2026

http://arxiv.org/abs/2605.22884

2. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhang, Sheng, Zhou, Chen, Zheng, Cai, Song, Tian, Ré, Barrett, Wang, Chen, 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models

3. SnapKV: LLM Knows What You Are Looking For Before Generation — Li, Huang, Yang, Venkitesh, Locatelli, Ye, Cai, Lewis, Chen, 2024

https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+Are+Looking+For+Before+Generation

4. CAOTE: KV Caching through Attention Output Error Based Token Eviction — Goel, Park, Gagrani, Jones, Morse, Langston, Lee, Lott, 2025

https://scholar.google.com/scholar?q=CAOTE%3A+KV+Caching+through+Attention+Output+Error+Based+Token+Eviction

5. Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference — Dong, Yang, Zhang, Wang, Chi, Chen, 2024

https://scholar.google.com/scholar?q=Get+More+with+LESS%3A+Synthesizing+Recurrence+with+KV+Cache+Compression+for+Efficient+LLM+Inference

6. Parallelizing Linear Transformers with the Delta Rule over Sequence Length — Yang, Wang, Zhang, Shen, Kim, 2024

https://scholar.google.com/scholar?q=Parallelizing+Linear+Transformers+with+the+Delta+Rule+over+Sequence+Length

7. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Liu, Li, Cheng, Ray, Huang, Zhang, Du, Yao, Lu, Ananthanarayanan, Maire, Hoffmann, Holtzman, Jiang, 2023

https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving

Interactive Visualization: Tensor Cache: Compressing Evicted Tokens into Fixed-Size Memory

This episode explores why identical reinforcement learning recipes produce wildly different reasoning ability in similarly-sized language models, examining "Cognitive Behaviors that Enable Self-Improving Reasoners" from Stanford and SynthLabs. Training Qwen-2.5-3B and Llama-3.2-3B on the number-puzzle game Countdown with identical PPO settings, Qwen jumps to roughly 60% accuracy while Llama plateaus around 30% — despite matched architecture size, algorithm, and hyperparameters. The discussion traces this gap to four cognitive behaviors already present in Qwen's pretrained weights before any RL begins: verification, backtracking, subgoal setting, and backward chaining (the last borrowed straight from 1970s-80s expert-system logic). Using GPT-4o-mini as an automated classifier across thousands of reasoning traces, the hosts unpack how these behaviors' presence — or absence — in a base model predicts whether reinforcement learning takes off or stalls, reframing the "just scale it up" narrative around what a model already knows how to do before training starts. It sets up the next question the arc will tackle: whether these behaviors can be deliberately installed in a model that lacks them.

Sources:

1. Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs — Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, Noah D. Goodman, 2025

http://arxiv.org/abs/2503.01307

2. STaR: Bootstrapping Reasoning With Reasoning — Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman, 2022

https://scholar.google.com/scholar?q=STaR%3A+Bootstrapping+Reasoning+With+Reasoning

3. Let's Verify Step by Step — Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe, 2023

https://scholar.google.com/scholar?q=Let%27s+Verify+Step+by+Step

4. Tree of Thoughts: Deliberate Problem Solving with Large Language Models — Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, Karthik Narasimhan, 2023

https://scholar.google.com/scholar?q=Tree+of+Thoughts%3A+Deliberate+Problem+Solving+with+Large+Language+Models

5. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek-AI (Daya Guo et al.), 2025

https://scholar.google.com/scholar?q=DeepSeek-R1%3A+Incentivizing+Reasoning+Capability+in+LLMs+via+Reinforcement+Learning

6. LAMBADA: Backward Chaining for Automated Reasoning in Natural Language — Seyed Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, Deepak Ramachandran, 2023

https://scholar.google.com/scholar?q=LAMBADA%3A+Backward+Chaining+for+Automated+Reasoning+in+Natural+Language

7. Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning — Antonia Creswell, Murray Shanahan, Irina Higgins, 2022

https://scholar.google.com/scholar?q=Selection-Inference%3A+Exploiting+Large+Language+Models+for+Interpretable+Logical+Reasoning

8. A Machine-Oriented Logic Based on the Resolution Principle — J. A. Robinson, 1965

https://scholar.google.com/scholar?q=A+Machine-Oriented+Logic+Based+on+the+Resolution+Principle

9. Demystifying Long Chain-of-Thought Reasoning in LLMs — Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, Xiang Yue, 2025

https://scholar.google.com/scholar?q=Demystifying+Long+Chain-of-Thought+Reasoning+in+LLMs

10. Stream of Search (SoS): Learning to Search in Language — Kanishk Gandhi, Denise H.J. Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, Noah Goodman, 2024

https://scholar.google.com/scholar?q=Stream+of+Search+%28SoS%29%3A+Learning+to+Search+in+Language

11. LLMs Can Easily Learn to Reason from Demonstrations: Structure, not Content, is What Matters! — Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Shishir G. Patil, Matei Zaharia, Joseph E. Gonzalez, Ion Stoica, 2025

https://scholar.google.com/scholar?q=LLMs+Can+Easily+Learn+to+Reason+from+Demonstrations%3A+Structure%2C+not+Content%2C+is+What+Matters%21

12. There May Not Be Aha Moment in R1-Zero-like Training — A Pilot Study — Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, Min Lin, 2025

https://scholar.google.com/scholar?q=There+May+Not+Be+Aha+Moment+in+R1-Zero-like+Training+%25E2%2580%2594+A+Pilot+Study

13. Human Problem Solving: The State of the Theory in 1970 — Herbert A. Simon, Allen Newell, 1971

https://scholar.google.com/scholar?q=Human+Problem+Solving%3A+The+State+of+the+Theory+in+1970

Interactive Visualization: Cognitive Behaviors Behind Self-Improving Language Model Reasoners

This episode examines SoundnessBench, a new benchmark testing whether frontier LLMs can judge the underlying soundness of a research proposal before any experiments are run, rather than just executing and scoring completed work like prior agent benchmarks (MLE-Bench, PaperBench, InnovatorBench). Built from 1,099 ICLR proposals labeled with reviewers' soundness sub-scores rather than acceptance outcomes, the benchmark found that twelve frontier models produced a 74% false-positive rate — repeatedly rating flawed proposals as sound. The hosts debate whether this stems from a sycophancy-style bias inherited from RLHF training, pointing to a striking result where switching to "aggressive" fault-hunting prompts flips the same models' verdicts on the same proposals, suggesting the failure is about framing sensitivity rather than missing domain knowledge. The discussion lands on why this matters for autonomous AI research agents: an unreliable judge sitting at the "first gate" risks industrializing well-executed experiments built on dead-on-arrival ideas.

Sources:

1. SoundnessBench: Exposing AI Reviewers' Blind Spots

https://arxiv.org/pdf/2605.30329

2. Discovering Language Model Behaviors with Model-Written Evaluations — Ethan Perez, Sam Ringer, Kamile Lukosiute, et al. (Anthropic), 2022

https://scholar.google.com/scholar?q=Discovering+Language+Model+Behaviors+with+Model-Written+Evaluations

3. Towards Understanding Sycophancy in Language Models — Mrinank Sharma, Meg Tong, Tomasz Korbak, et al. (Anthropic, with academic collaborators), 2023

https://scholar.google.com/scholar?q=Towards+Understanding+Sycophancy+in+Language+Models

4. Simple Synthetic Data Reduces Sycophancy in Large Language Models — Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, Quoc V. Le (Google DeepMind / Google Brain), 2023

https://scholar.google.com/scholar?q=Simple+Synthetic+Data+Reduces+Sycophancy+in+Large+Language+Models

5. Prompt Sensitivity Evaluations of Large Language Models — Kate Elkins, Jon Chun (and related follow-on prompt-robustness studies, e.g. Geng et al.), 2025

https://scholar.google.com/scholar?q=Prompt+Sensitivity+Evaluations+of+Large+Language+Models

6. Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers — Chenglei Si, Diyi Yang, Tatsunori Hashimoto, 2025

https://scholar.google.com/scholar?q=Can+LLMs+Generate+Novel+Research+Ideas%3F+A+Large-Scale+Human+Study+with+100%2B+NLP+Researchers

7. The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas — Chenglei Si, Tatsunori Hashimoto, Diyi Yang, 2025

https://scholar.google.com/scholar?q=The+Ideation-Execution+Gap%3A+Execution+Outcomes+of+LLM-Generated+versus+Human+Research+Ideas

8. The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search — Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, David Ha, 2025

https://scholar.google.com/scholar?q=The+AI+Scientist-v2%3A+Workshop-Level+Automated+Scientific+Discovery+via+Agentic+Tree+Search

9. PaperBench: Evaluating AI's Ability to Replicate AI Research — Giulio Starace, Oliver Jaffe, Dane Sherburn, et al. (OpenAI), 2025

https://scholar.google.com/scholar?q=PaperBench%3A+Evaluating+AI%27s+Ability+to+Replicate+AI+Research

Interactive Visualization: SoundnessBench: Exposing AI Reviewers' Blind Spots

This episode examines HyperOffload, a graph-driven scheduling system from Shanghai Jiao Tong University and Huawei that shifts LLM memory offload and prefetch decisions from a reactive runtime into compile-time graph scheduling for terabyte-scale "SuperNode" hardware. The hosts scrutinize the paper's headline 26% peak memory reduction, arguing it's largely definitional since it comes from offloading the entire KV cache in one configuration, while pointing to the defragmentation results (57 stalls eliminated) and bandwidth-robustness curves as the figures that actually demonstrate the scheduler's value. They flag a notable gap: the paper's motivating anecdote about a 2.7x slowdown from reactive prefetching is never directly retested against HyperOffload, leaving its central justification unconfirmed. The discussion also surfaces missing citations to ZeRO-Infinity and a lack of engagement with PagedAttention as a competing paradigm, plus the fact that all results are confined to Ascend NPUs and MindSpore with no evidence of portability to CUDA or PyTorch. Listeners interested in memory management for large-scale LLM serving and training will find a sharp critique of how benchmark framing can overstate a system's true contribution.

Sources:

1. HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architectures — Fangxin Liu, Qinghua Zhang, Hanjing Shen, Zhibo Liang, Li Jiang, Haibing Guan, Chong Bao, Xuefeng Jin, 2026

http://arxiv.org/abs/2602.00748

2. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning — Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He (Microsoft), 2021

https://scholar.google.com/scholar?q=ZeRO-Infinity%3A+Breaking+the+GPU+Memory+Wall+for+Extreme+Scale+Deep+Learning

3. Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization — Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, Ion Stoica, 2020 (MLSys)

https://scholar.google.com/scholar?q=Checkmate%3A+Breaking+the+Memory+Wall+with+Optimal+Tensor+Rematerialization

4. AutoTM: Automatic Tensor Movement in Heterogeneous Memory Systems using Integer Linear Programming — Michael Hildebrand, Jawad Khan, Sanjeev Trika, Jason Lowe-Power, Venkatesh Akella, 2020 (ASPLOS)

https://scholar.google.com/scholar?q=AutoTM%3A+Automatic+Tensor+Movement+in+Heterogeneous+Memory+Systems+using+Integer+Linear+Programming

5. G10: Enabling An Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrations — Haoyang Zhang, Yirui Zhou, Yuqi Xue, Yiqi Liu, Jian Huang, 2023 (MICRO)

https://scholar.google.com/scholar?q=G10%3A+Enabling+An+Efficient+Unified+GPU+Memory+and+Storage+Architecture+with+Smart+Tensor+Migrations

6. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, Ion Stoica, 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

7. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2024

https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving

8. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan et al. (DeepSeek-AI), 2025

https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention

Interactive Visualization: HyperOffload's Scheduling Claims Under Scrutiny

This episode explores STEEL, a sparsity-aware fused attention design for running long-sequence prefill inference efficiently on AMD's XDNA neural processing unit. The discussion contrasts spatial-dataflow NPU architectures, where compute tiles are explicitly scheduled with no dynamic cache management, against GPU SIMT execution, and explains how the causal attention mask creates load imbalance that a fixed pipeline can't easily absorb the way a GPU scheduler can. Building on FlashAttention-2's tiling and online-softmax approach, the paper restructures the computation into a three-stage pipeline across dedicated compute cores to address that imbalance directly. The hosts walk through why this matters for on-device AI agents that need low latency, privacy, and battery efficiency without offloading to cloud GPUs. Reported results include over 9.5x latency reduction versus prior state-of-the-art NPU implementations, over 9x energy savings against a CPU baseline, and more than 22x speedup over a naive layer-by-layer approach.

Sources:

1. STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU — Victor J. B. Jung, Gagandeep Singh, Joseph Melber, Kristof Denolf, Francesco Conti, Luca Benini, 2026

http://arxiv.org/abs/2607.09385v1

2. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022

https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness

3. Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks — Yu-Hsin Chen, Joel Emer, Vivienne Sze, 2016

https://scholar.google.com/scholar?q=Eyeriss%3A+An+Energy-Efficient+Reconfigurable+Accelerator+for+Deep+Convolutional+Neural+Networks

4. In-Datacenter Performance Analysis of a Tensor Processing Unit — Norman P. Jouppi et al. (Google), 2017

https://scholar.google.com/scholar?q=In-Datacenter+Performance+Analysis+of+a+Tensor+Processing+Unit

5. Plasticine: A Reconfigurable Architecture for Parallel Patterns — Raghu Prabhakar, Yaqi Zhang, David Koeplinger, Matt Feldman, Tian Zhao, Stefan Hadjis, Ardavan Pedram, Christos Kozyrakis, Kunle Olukotun, 2017

https://scholar.google.com/scholar?q=Plasticine%3A+A+Reconfigurable+Architecture+for+Parallel+Patterns

6. Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

7. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Ainslie et al., 2023

https://scholar.google.com/scholar?q=GQA%3A+Training+Generalized+Multi-Query+Transformer+Models+from+Multi-Head+Checkpoints

8. Efficiently Scaling Transformer Inference — Pope et al., 2022

https://scholar.google.com/scholar?q=Efficiently+Scaling+Transformer+Inference

9. FlashDecoding++: Faster Large Language Model Inference on GPUs — Hong et al., 2024

https://scholar.google.com/scholar?q=FlashDecoding%2B%2B%3A+Faster+Large+Language+Model+Inference+on+GPUs

10. NITRO: LLM Inference on Intel Laptop NPUs — Fei and Abdelfattah, 2024

https://scholar.google.com/scholar?q=NITRO%3A+LLM+Inference+on+Intel+Laptop+NPUs

Interactive Visualization: AMD XDNA NPU Sparsity-Aware Attention for Long-Context Prefill

This episode explores how inference-time scaling breaks down once large language models shift from short chat responses to long chain-of-thought reasoning, drawing on Micron and Argonne National Laboratory's research spanning models from 8 billion to 671 billion parameters. It explains the divide between compute-bound prefill and bandwidth-bound decode phases, and how reasoning traces exceeding ten thousand tokens push systems into a "capacity-bound" regime where the KV cache — not raw FLOPs — becomes the limiting resource. The discussion contrasts three parallelism strategies (data, tensor, and pipeline) and shows why data parallelism, the industry default, hits a capacity wall under reasoning workloads even though it remains optimal for short prompts. It also covers how architectural choices like Grouped-Query Attention versus DeepSeek-R1's Mixture-of-Experts design and Multi-Head Latent Attention change how much cache pressure a model generates per token. Listeners interested in the practical engineering tradeoffs behind serving reasoning models at scale will find concrete guidance on when each parallelism strategy actually wins, backed by measurements on an 8x H200 NVLink node.

Sources:

1. Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principles — Moiz Arif, Avinash Maurya, Sudharshan Vazhkudai, Bogdan Nicolae, 2026

http://arxiv.org/abs/2605.19775

2. PyTorch Distributed: Experiences on Accelerating Data Parallel Training — Shen Li, Yanli Zhao, Rohan Varma, et al. (Meta AI / PyTorch team), 2020

https://scholar.google.com/scholar?q=PyTorch+Distributed%3A+Experiences+on+Accelerating+Data+Parallel+Training

3. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He (Microsoft), 2020

https://scholar.google.com/scholar?q=ZeRO%3A+Memory+Optimizations+Toward+Training+Trillion+Parameter+Models

4. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. (UC Berkeley), 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%29

5. Orca: A Distributed Serving System for Transformer-Based Generative Models — Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun (Seoul National University / FriendliAI), 2022

https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models

6. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024

https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving

7. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini, 2024

https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting

8. Llumnix: Dynamic Scheduling for Large Language Model Serving — Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, Wei Lin, 2024

https://scholar.google.com/scholar?q=Llumnix%3A+Dynamic+Scheduling+for+Large+Language+Model+Serving

9. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention — Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar, 2024

https://scholar.google.com/scholar?q=vAttention%3A+Dynamic+Memory+Management+for+Serving+LLMs+without+PagedAttention

10. Efficient Memory Management for Large Language Model Serving with PagedAttention (already cited [22]) — cross-check against KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache — Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, Xia Hu, 2024

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28already+cited+%5B22%5D%29+%E2%80%94+cross-check+against+KIVI%3A+A+Tuning-Free+Asymmetric+2bit+Quantization+for+KV+Cache

Interactive Visualization: Understanding Inference Scaling: Prefill, Decode, and Reasoning Bottlenecks

This episode explores how speculative decoding — the standard trick for speeding up autoregressive LLM inference by having a cheap draft model propose tokens for batch verification — breaks down when applied to Mixture-of-Experts models. The discussion traces why decoding is memory-bandwidth bound rather than compute bound, then shows how MoE routing decouples a draft token's acceptance probability from its actual verification cost: tokens that route to disjoint experts (termed "expert scattering") force costly extra weight fetches even when a confidence-only selector rates them highly. The paper introduces EcoSpec, a cost-aware draft selector that accounts for expert-loading overhead rather than optimizing acceptance length (alpha) alone, and the hosts examine tradeoffs in Table 1 where EcoSpec sacrifices a small amount of acceptance probability on models like Qwen3 and GPT-OSS in exchange for reduced memory traffic. Listeners interested in LLM inference serving, hardware-aware systems design, or the practical limits of applying dense-model optimizations to sparse architectures will find the episode's reframing of a three-year-old assumption particularly compelling.

Sources:

1. Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts — Jincheng Xie, Runheng Liu, Heyan Huang, Yawen Ling, Hanbin Dai, Yu Zheng, Wen Hu, 2026

http://arxiv.org/abs/2607.12696

2. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023

https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding

3. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024

https://scholar.google.com/scholar?q=EAGLE-2%3A+Faster+Inference+of+Language+Models+with+Dynamic+Draft+Trees

4. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Tri Dao, et al., 2024

https://scholar.google.com/scholar?q=Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads

5. Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models — Keisuke Kamahori, Yile Gu, Kan Zhu, Baris Kasikci, 2024

https://scholar.google.com/scholar?q=Fiddler%3A+CPU-GPU+Orchestration+for+Fast+Inference+of+Mixture-of-Experts+Models

6. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — Li, Y., Wei, F., Zhang, C., Zhang, H., 2026

https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test

7. MoE-Spec: Expert Budgeting for Efficient Speculative Decoding — McDanel, B., Li, S., Surineni, S., Khaitan, H., 2026

https://scholar.google.com/scholar?q=MoE-Spec%3A+Expert+Budgeting+for+Efficient+Speculative+Decoding

8. SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference — Chen, L., Wen, Z., Wu, T., Zhang, X., Wu, C., 2025

https://scholar.google.com/scholar?q=SP-MoE%3A+Speculative+Decoding+and+Prefetching+for+Accelerating+MoE-based+Model+Inference

9. MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts — Wang, W., Liu, J., Hou, X., Xia, X., Tang, P., Zhang, M., Li, C., Guo, M., 2025

https://scholar.google.com/scholar?q=MoE-SpeQ%3A+Speculative+Quantized+Decoding+with+Proactive+Expert+Prefetching+and+Offloading+for+Mixture-of-Experts

10. Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding (GTO) — Hu, S., Li, J., Lu, Z., Zhou, P., 2026

https://scholar.google.com/scholar?q=Bridging+Draft+Policy+Misalignment%3A+Group+Tree+Optimization+for+Speculative+Decoding+%28GTO%29

11. MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache — Xue, L., Fu, Y., Lu, Z., Mai, L., Marina, M., 2025

https://scholar.google.com/scholar?q=MoE-Infinity%3A+Efficient+MoE+Inference+on+Personal+Machines+with+Sparsity-Aware+Expert+Cache

12. A Survey on Inference Optimization Techniques for Mixture of Experts Models — Liu, J., Tang, P., Wang, W., Ren, Y., Hou, X., Heng, P.A., Guo, M., Li, C., 2026

https://scholar.google.com/scholar?q=A+Survey+on+Inference+Optimization+Techniques+for+Mixture+of+Experts+Models

Interactive Visualization: Cost-Aware Speculative Decoding for Mixture-of-Experts Models

This episode explores Parcae, a paper on scaling laws for stable looped language models, where instead of stacking distinct transformer layers, a single block is applied repeatedly through the residual stream — echoing Universal Transformers and ALBERT's weight-tying but tackling the training instability that has historically plagued the approach. The discussion centers on reframing looped inference as a linear time-invariant dynamical system, showing that the spectral norm of the transition matrix A determines whether the residual stream stays bounded or explodes exponentially — turning a mysterious loss-spike failure mode into a measurable, checkable quantity. It also covers how the authors diagnose this concretely by examining the eigenvalues of A (contrasting how different prelude-embedding injection methods affect stability), and how they extend Chinchilla-style isoFLOP curve-fitting with a third axis — recurrence depth — to find the FLOP-optimal number of loops at a given compute budget. Listeners interested in efficient inference, edge deployment, or the mechanics of why prior recurrent-depth models like RDM needed fragile tuning will find the control-theory framing a clarifying, math-grounded alternative to typical trial-and-error architecture papers.

Sources:

1. Parcae: Scaling Laws For Stable Looped Language Models — Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, Daniel Y. Fu, 2026

http://arxiv.org/abs/2604.12946

2. Universal Transformers — Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Lukasz Kaiser, 2018

https://scholar.google.com/scholar?q=Universal+Transformers

3. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations — Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut, 2019

https://scholar.google.com/scholar?q=ALBERT%3A+A+Lite+BERT+for+Self-supervised+Learning+of+Language+Representations

4. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping et al., 2025

https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach

5. Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA — Sangmin Bae et al., 2024

https://scholar.google.com/scholar?q=Relaxed+Recursive+Transformers%3A+Effective+Parameter+Sharing+with+Layer-wise+LoRA

6. A Proposal on Machine Learning via Dynamical Systems — Weinan E, 2017

https://scholar.google.com/scholar?q=A+Proposal+on+Machine+Learning+via+Dynamical+Systems

7. Neural Ordinary Differential Equations — Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud, 2018

https://scholar.google.com/scholar?q=Neural+Ordinary+Differential+Equations

8. Stable Architectures for Deep Neural Networks — Eldad Haber, Lars Ruthotto, 2017

https://scholar.google.com/scholar?q=Stable+Architectures+for+Deep+Neural+Networks

9. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023

https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces

10. Transformers are SSMs: Generalized Models and Efficient Algorithms through Structured State Space Duality — Tri Dao, Albert Gu, 2024

https://scholar.google.com/scholar?q=Transformers+are+SSMs%3A+Generalized+Models+and+Efficient+Algorithms+through+Structured+State+Space+Duality

11. Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation — Sangmin Bae, Yujin Kim, Reza Bayat, et al., 2025

https://scholar.google.com/scholar?q=Mixture-of-Recursions%3A+Learning+Dynamic+Recursive+Depths+for+Adaptive+Token-Level+Computation

12. Scaling Latent Reasoning via Looped Language Models — Rui-Jie Zhu, Zixuan Wang, Kai Hua, et al., 2025

https://scholar.google.com/scholar?q=Scaling+Latent+Reasoning+via+Looped+Language+Models

13. Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence — Sean McLeish, Ang Li, John Kirchenbauer, et al., 2025

https://scholar.google.com/scholar?q=Teaching+Pretrained+Language+Models+to+Think+Deeper+with+Retrofitted+Recurrence

14. Reasoning with Latent Thoughts: On the Power of Looped Transformers — Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, Sashank J. Reddi, 2025

https://scholar.google.com/scholar?q=Reasoning+with+Latent+Thoughts%3A+On+the+Power+of+Looped+Transformers

Interactive Visualization: Parcae: Stabilizing Looped Language Models with Control Theory

This episode examines ELDR (Expert-Locality-Aware Decode Routing), a routing scheme for serving mixture-of-experts models under prefill-decode disaggregation, from researchers at KAIST, Microsoft Research, and the Shanghai Xingyunzhili Artificial Intelligence Institute. The discussion lays out why decode is memory-bandwidth bound rather than compute bound, and how MoE sparsity — which lowers per-token cost for a single request — becomes a liability at batch scale, since latency now depends on the union of experts a batch touches rather than just token count or load. The key insight is that expert activation patterns are structured rather than random: prompts from similar domains or languages route to overlapping experts, and the model's prefill-time gating decisions already preview which experts a request's decode phase will need. ELDR exploits this by using prefill activations as an early signature to route decode requests toward workers with "warm" overlapping experts, targeting time-per-output-token latency specifically, distinct from ordinary load balancing which only tracks request count or capacity. The conversation grounds this in prior work — DistServe's phase-disaggregation argument, and MoE foundations from Switch Transformers and GShard — before probing how well the paper's offline expert-locality clustering, calibrated on a fixed domain mix, generalizes to shifting real-world traffic.

Sources:

1. Expert-Locality-Aware Decode Routing for MoE Serving

https://arxiv.org/pdf/2607.00466

2. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — William Fedus, Barret Zoph, Noam Shazeer, 2021

https://scholar.google.com/scholar?q=Switch+Transformers%3A+Scaling+to+Trillion+Parameter+Models+with+Simple+and+Efficient+Sparsity

3. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen, 2020

https://scholar.google.com/scholar?q=GShard%3A+Scaling+Giant+Models+with+Conditional+Computation+and+Automatic+Sharding

4. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024

https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving

5. Mooncake: A KV Cache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu (Moonshot AI / Kimi), 2024

https://scholar.google.com/scholar?q=Mooncake%3A+A+KV+Cache-centric+Disaggregated+Architecture+for+LLM+Serving

6. Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns — Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park, et al., 2026

https://scholar.google.com/scholar?q=Scaling+Multi-Node+Mixture-of-Experts+Inference+Using+Expert+Activation+Patterns

7. Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens (METRO) — Yanpeng Yu, Haiyue Ma, Krish Agarwal, et al., 2025

https://scholar.google.com/scholar?q=Efficient+MoE+Serving+in+the+Memory-Bound+Regime%3A+Balance+Activated+Experts%2C+Not+Tokens+%28METRO%29

8. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, et al., 2025

https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving

9. DeepSeek-V3 Technical Report / EPLB: Expert Parallelism Load Balancer — DeepSeek-AI, 2024/2025

https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report+%2F+EPLB%3A+Expert+Parallelism+Load+Balancer

Interactive Visualization: Expert-Locality-Aware Decode Routing for MoE Serving

This episode explores SkillOpt-Lite, a framework for improving AI agents by editing their "skill" documents — the text-based instructions a frozen LLM reads to approach a task — rather than retraining the underlying model. The hosts unpack how the authors reframe skill editing as zeroth-order optimization, mapping techniques from prior systems like SkillOpt, SkillCat, and SkillAdapter onto classical concepts such as one-point gradient estimators, central differences, and coordinate descent. A key argument is that agent execution traces offer a far richer optimization signal than a single loss value, since failures can be traced to specific planning steps or errors. The discussion also covers how PAC-learning generalization bounds are used to strip away architectural complexity inherited from earlier systems, testing which components actually earn their keep versus which are dead weight. Listeners interested in agent design will find the payoff notable: the leaner pipeline reportedly outperforms full SkillOpt, with one result showing a smaller model using this framework beating a flagship model running the older approach.

Sources:

1. SkillOpt-Lite: Better and Faster Agent Self-evolution via One Line of Vibe — Yifei Shen, Bo Li, Xinjie Zhang, 2026

http://arxiv.org/abs/2607.03451

2. Large Language Models as Optimizers — Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen (Google DeepMind), 2023

https://scholar.google.com/scholar?q=Large+Language+Models+as+Optimizers

3. Voyager: An Open-Ended Embodied Agent with Large Language Models — Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, Anima Anandkumar (NVIDIA, Caltech, UT Austin, Stanford), 2023

https://scholar.google.com/scholar?q=Voyager%3A+An+Open-Ended+Embodied+Agent+with+Large+Language+Models

4. Reflexion: Language Agents with Verbal Reinforcement Learning — Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao, 2023

https://scholar.google.com/scholar?q=Reflexion%3A+Language+Agents+with+Verbal+Reinforcement+Learning

5. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines — Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, Christopher Potts, 2023

https://scholar.google.com/scholar?q=DSPy%3A+Compiling+Declarative+Language+Model+Calls+into+Self-Improving+Pipelines

6. SkillOpt: Executive strategy for self-evolving agent skills — Yang, Gong, Huang, Yang, Zhou, Huang, Li, Gao, Dai, Liu, et al., 2026 (arXiv:2605.23904)

https://scholar.google.com/scholar?q=SkillOpt%3A+Executive+strategy+for+self-evolving+agent+skills

7. A primer on zeroth-order optimization in signal processing and machine learning — Liu, Chen, Kailkhura, Zhang, Hero III, Varshney, 2020

https://scholar.google.com/scholar?q=A+primer+on+zeroth-order+optimization+in+signal+processing+and+machine+learning

8. Learnability and stability in the Vapnik-Chervonenkis sense — Shalev-Shwartz, Shamir, Srebro, Sridharan, 2010

https://scholar.google.com/scholar?q=Learnability+and+stability+in+the+Vapnik-Chervonenkis+sense

9. Meta-harness: End-to-end optimization of model harnesses — Lee, Nair, Zhang, Lee, Khattab, Finn, 2026 (arXiv:2603.28052)

https://scholar.google.com/scholar?q=Meta-harness%3A+End-to-end+optimization+of+model+harnesses

10. Harness updating is not harness benefit: Disentangling evolution capabilities in self-evolving LLM agents — Harness Self-Evolution (anonymous/collective), 2026

https://scholar.google.com/scholar?q=Harness+updating+is+not+harness+benefit%3A+Disentangling+evolution+capabilities+in+self-evolving+LLM+agents

11. SpreadsheetBench: Towards challenging real world spreadsheet manipulation — Ma, Zhang, Zhang, Yu, Zhang, Zhang, Luo, Wang, Tang, 2024

https://scholar.google.com/scholar?q=SpreadsheetBench%3A+Towards+challenging+real+world+spreadsheet+manipulation

Interactive Visualization: SkillOpt-Lite: Rethinking Agent Skill Optimization with Zeroth-Order Simplicity