AI Post Transformers

This episode examines FlatCake, a proposed KV cache storage architecture for LLM serving that drops the usual GPU, host DRAM, and NVMe SSD hierarchy. Instead of migrating blocks between tiers, it decides at write time whether each block lives in DRAM or on SSD, and it never moves the block afterward. The discussion explains why the authors think tiering breaks down. Reuse is bimodal: in several conversational and API traces, most blocks are used only once. In a Claude Code agentic trace, sessions are so large that a few concurrent ones overflow DRAM. Staging buffers also run out under load, and blocks written in eviction order end up scattered across the drive. The hosts walk through the proposed fix, which lays each session out contiguously on the SSD so a restore becomes a few large reads, and they note that GPUDirect Storage fixes the bounce buffer but not the scatter. They also point out the cost of the approach: placement relies on hand-tuned thresholds with no published values, and a wrong placement choice sticks. Listeners interested in serving infrastructure and storage layout get a critical look at how much the paper's experiments actually test its claims.

Sources:

1. FlatCake: A Flat KVCache Storage Architecture for LLM Serving — Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu, Wei Yang, Chenxi Wang, Tao Xie, Youwei Zhuo, 2026

http://arxiv.org/abs/2610.38355

2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu (Moonshot AI / Tsinghua), 2024 (FAST 2025)

https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving

3. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention (AttentionStore) — Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, Pengfei Zuo, 2024 (USENIX ATC 2024)

https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention+%28AttentionStore%29

4. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yihua Cheng, Yuhan Liu, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du, Junchen Jiang et al., 2025

https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference

5. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang, 2023 (ICML 2023)

https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU

6. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning — Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He, 2021 (SC 2021)

https://scholar.google.com/scholar?q=ZeRO-Infinity%3A+Breaking+the+GPU+Memory+Wall+for+Extreme+Scale+Deep+Learning

7. GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecture — Zaid Qureshi, Vikram Sharma Mailthody, Isaac Gelado, Seung Won Min, Amna Masood, Jeongmin Park, Jinjun Xiong, C. J. Newburn, Dmitri Vainbrand, I-Hsin Chung, Michael Garland, William Dally, Wen-mei Hwu, 2023 (ASPLOS 2023)

https://scholar.google.com/scholar?q=GPU-Initiated+On-Demand+High-Throughput+Storage+Access+in+the+BaM+System+Architecture

8. GIDS: Accelerating Sampling and Aggregation Operations in GNN Frameworks with GPU Initiated Direct Storage Accesses — Jeongmin Brian Park, Vikram Sharma Mailthody, Zaid Qureshi, Wen-mei Hwu, 2024 (PVLDB 2024)

https://scholar.google.com/scholar?q=GIDS%3A+Accelerating+Sampling+and+Aggregation+Operations+in+GNN+Frameworks+with+GPU+Initiated+Direct+Storage+Accesses

9. KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider — Jiahao Wang, Jinbo Han, Xingda Wei, Sijie Shen, Dingyan Zhang, Chenguang Fang, Rong Chen, Wenyuan Yu, Haibo Chen, 2025 (USENIX ATC 2025)

https://scholar.google.com/scholar?q=KVCache+Cache+in+the+Wild%3A+Characterizing+and+Optimizing+KVCache+Cache+at+a+Large+Cloud+Provider

10. Preble: Efficient Distributed Prompt Scheduling for LLM Serving — Vikranth Srivastava, Zijian He, Shuowei Jin (Yingxuan Jin?), Yilong Zhao, Yibo Zhu, Ion Stoica, Hao Zhang and others (UCSD / Berkeley), 2024 (ICLR 2025)

https://scholar.google.com/scholar?q=Preble%3A+Efficient+Distributed+Prompt+Scheduling+for+LLM+Serving

11. FIFO Queues are All You Need for Cache Eviction — Juncheng Yang, Yazhuo Zhang, Ziyue Qiu, Yao Yue, Rashmi Vinayak, 2023 (SOSP 2023)

https://scholar.google.com/scholar?q=FIFO+Queues+are+All+You+Need+for+Cache+Eviction

12. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu, Hanchen Li, Yihua Cheng, et al., 2024

https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving

13. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, et al., 2025

https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion

14. AttentionStore / CachedAttention: Cost-Efficient LLM Serving for Multi-turn Conversations — Bin Gao, Zhuomin He, Puru Sharma, et al., 2024

https://scholar.google.com/scholar?q=AttentionStore+%2F+CachedAttention%3A+Cost-Efficient+LLM+Serving+for+Multi-turn+Conversations

15. Managing Scalable Direct Storage Accesses for GPUs with GoFS — Shaobo Li, Yirui Eric Zhou, Yuqi Xue, Yuan Xu, Jian Huang, 2025

https://scholar.google.com/scholar?q=Managing+Scalable+Direct+Storage+Accesses+for+GPUs+with+GoFS

16. Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional Computation–Storage Awareness — Shipeng Hu, Guangyan Zhang, et al., 2026

https://scholar.google.com/scholar?q=Bidaw%3A+Enhancing+Key-Value+Caching+for+Interactive+LLM+Serving+via+Bidirectional+Computation%E2%80%93Storage+Awareness

17. Tutti: GPU-initiated io_uring queues for KV storage — Cited as [28] in the paper, 2025–2026

https://scholar.google.com/scholar?q=Tutti%3A+GPU-initiated+io_uring+queues+for+KV+storage

Interactive Visualization: FlatCake: Flat KVCache Storage Beats Tiered Memory Hierarchies

This episode examines why speculative decoding, long thought to give little benefit for sparse mixture-of-experts models at batch size one, actually helps most in that regime for many newer models. It covers ten MoE models from 30 billion to a trillion parameters, and explains how a drafter proposes tokens that the target model verifies in a single pass. It then contrasts coarse-grained experts (Mixtral-style) with fine-grained ones (Qwen3, Kimi-K2.5), where each token touches only a tiny slice of the layer. The discussion challenges the bandwidth-bound cost model from prior work, which predicts a speedup curve that rises, peaks at mid-sized batches, then falls. The authors point to two missing factors: adjacent tokens routing to overlapping experts, and real GPU execution dynamics. The episode also explains target efficiency, a metric that isolates verification cost from draft acceptance, and outlines the vLLM benchmarking setup on H100 and B200 GPUs. Listeners who care about low-latency serving for agents, test-time scaling, or RL rollouts will find it useful for deciding when speculative decoding is worth enabling.

Sources:

1. Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models — Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, Acyr Locatelli, 2026

http://arxiv.org/abs/2610.38981

2. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2022 (arXiv); ICML 2023

https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding

3. Accelerating Large Language Model Decoding with Speculative Sampling — Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John Jumper, 2023

https://scholar.google.com/scholar?q=Accelerating+Large+Language+Model+Decoding+with+Speculative+Sampling

4. MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE — Zongle Huang, Lei Zhu, Zongyuan Zhan, Ting Hu, Weikai Mao, Xianzhi Yu, Yongpan Liu, Tianyu Zhang (verify author list), 2025

https://scholar.google.com/scholar?q=MoESD%3A+Unveil+Speculative+Decoding%27s+Potential+for+Accelerating+Sparse+MoE

5. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2025

https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test

6. Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference — Jinghan Yao, Quentin Anthony, Aamir Shafi, Hari Subramoni, Dhabaleswar K. Panda, 2024

https://scholar.google.com/scholar?q=Exploiting+Inter-Layer+Expert+Affinity+for+Accelerating+Mixture-of-Experts+Model+Inference

7. Fast Inference of Mixture-of-Experts Language Models with Offloading — Artyom Eliseev, Denis Mazur, 2023

https://scholar.google.com/scholar?q=Fast+Inference+of+Mixture-of-Experts+Language+Models+with+Offloading

8. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Damai Dai, Chengqi Deng, Chenggang Zhao, and others (DeepSeek-AI), 2024

https://scholar.google.com/scholar?q=DeepSeekMoE%3A+Towards+Ultimate+Expert+Specialization+in+Mixture-of-Experts+Language+Models

9. Mixtral of Experts — Albert Q. Jiang and others (Mistral AI), 2024

https://scholar.google.com/scholar?q=Mixtral+of+Experts

10. On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals are Known — W. Edwards Deming, Frederick F. Stephan, 1940

https://scholar.google.com/scholar?q=On+a+Least+Squares+Adjustment+of+a+Sampled+Frequency+Table+When+the+Expected+Marginal+Totals+are+Known

11. I-Divergence Geometry of Probability Distributions and Minimization Problems — Imre Csiszár, 1975

https://scholar.google.com/scholar?q=I-Divergence+Geometry+of+Probability+Distributions+and+Minimization+Problems

12. Information Theory and Statistical Mechanics — E. T. Jaynes, 1957

https://scholar.google.com/scholar?q=Information+Theory+and+Statistical+Mechanics

13. Sinkhorn Distances: Lightspeed Computation of Optimal Transport — Marco Cuturi, 2013

https://scholar.google.com/scholar?q=Sinkhorn+Distances%3A+Lightspeed+Computation+of+Optimal+Transport

14. Roofline: An Insightful Visual Performance Model for Multicore Architectures — Samuel Williams, Andrew Waterman, David Patterson, 2009

https://scholar.google.com/scholar?q=Roofline%3A+An+Insightful+Visual+Performance+Model+for+Multicore+Architectures

15. Efficiently Scaling Transformer Inference — Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, Jeff Dean, 2022

https://scholar.google.com/scholar?q=Efficiently+Scaling+Transformer+Inference

16. MegaBlocks: Efficient Sparse Training with Mixture-of-Experts — Trevor Gale, Deepak Narayanan, Cliff Young, Matei Zaharia, 2023

https://scholar.google.com/scholar?q=MegaBlocks%3A+Efficient+Sparse+Training+with+Mixture-of-Experts

17. Dissecting the NVIDIA Hopper Architecture through Microbenchmarking and Multiple Level Analysis — Weile Luo, Ruibo Fan, Zeyu Li, Dayou Du, Qiang Wang, Xiaowen Chu, 2025

https://scholar.google.com/scholar?q=Dissecting+the+NVIDIA+Hopper+Architecture+through+Microbenchmarking+and+Multiple+Level+Analysis

18. Utility-driven Speculative Decoding for Mixture-of-Experts — Saxena, Tsai, Taneja, Jaleel, Qureshi, 2025

https://scholar.google.com/scholar?q=Utility-driven+Speculative+Decoding+for+Mixture-of-Experts

19. MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding — Sadhukhan et al., 2025

https://scholar.google.com/scholar?q=MagicDec%3A+Breaking+the+Latency-Throughput+Tradeoff+for+Long+Context+Generation+with+Speculative+Decoding

20. ExpertFlow: Efficient MoE Inference via Predictive Expert Caching and Token Scheduling — He et al., 2026

https://scholar.google.com/scholar?q=ExpertFlow%3A+Efficient+MoE+Inference+via+Predictive+Expert+Caching+and+Token+Scheduling

21. Probing Semantic Routing in Large Mixture-of-Expert Models — Olson et al., 2025

https://scholar.google.com/scholar?q=Probing+Semantic+Routing+in+Large+Mixture-of-Expert+Models

22. On the Spatial Structure of Mixture-of-Experts in Transformers — Bershatsky, Oseledets, 2025

https://scholar.google.com/scholar?q=On+the+Spatial+Structure+of+Mixture-of-Experts+in+Transformers

23. Scaling Laws for Fine-Grained Mixture of Experts — Ludziejewski et al., 2024

https://scholar.google.com/scholar?q=Scaling+Laws+for+Fine-Grained+Mixture+of+Experts

24. On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals are Known / I-divergence Geometry of Probability Distributions and Minimization Problems — Deming & Stephan (1940); Csiszár (1975), 1940 / 1975

https://scholar.google.com/scholar?q=On+a+Least+Squares+Adjustment+of+a+Sampled+Frequency+Table+When+the+Expected+Marginal+Totals+are+Known+%2F+I-divergence+Geometry+of+Probability+Distributions+and+Minimization+Problems

Interactive Visualization: Speculative Decoding Surprisingly Excels on Sparse MoE Models

This episode examines a paper claiming that a pretrained LLM can be cut down to a sliding window of the last 64 tokens plus four "attention sink" starting tokens, with no training, and still keep about 99% of its benchmark score. That result would match months of work on retrofitting models into linear attention. The discussion builds up the background: the KV cache and how eviction differs from fixed-size linear-attention state, why StreamingLLM's sinks keep windowing from collapsing, and how retrofit costs range from roughly 20 million to 100 billion training tokens. It then questions what "99% recovery" means, noting that log-likelihood multiple-choice benchmarks like MMLU and HellaSwag mostly need only nearby context, unlike generative tasks such as GSM8K and HumanEval. It also asks whether the comparison is fair when memory budgets aren't held equal. Listeners get a skeptical look at whether a simple free baseline is outperforming retrofit methods, or whether the benchmarks are just too local to tell.

Sources:

1. Sliding-window beats linear attention — Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron, Emy Gervais, 2026

http://arxiv.org/abs/2608.28444

2. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023 (ICLR 2024)

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks

3. Longformer: The Long-Document Transformer — Iz Beltagy, Matthew E. Peters, Arman Cohan, 2020

https://scholar.google.com/scholar?q=Longformer%3A+The+Long-Document+Transformer

4. Mistral 7B — Albert Q. Jiang et al. (Mistral AI), 2023

https://scholar.google.com/scholar?q=Mistral+7B

5. Gemma 2: Improving Open Language Models at a Practical Size (and Gemma 3 Technical Report) — Gemma Team, Google DeepMind, 2024 / 2025

https://scholar.google.com/scholar?q=Gemma+2%3A+Improving+Open+Language+Models+at+a+Practical+Size+%28and+Gemma+3+Technical+Report%29

6. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret, 2020

https://scholar.google.com/scholar?q=Transformers+are+RNNs%3A+Fast+Autoregressive+Transformers+with+Linear+Attention

7. Finetuning Pretrained Transformers into RNNs — Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, Noah A. Smith, 2021

https://scholar.google.com/scholar?q=Finetuning+Pretrained+Transformers+into+RNNs

8. LoLCATs: On Low-Rank Linearizing of Large Language Models — Michael Zhang, Simran Arora, Rahul Chalamala, Alan Wu, Benjamin Spector, Aaryan Singhal, Krithik Ramesh, Christopher Ré, 2024 (ICLR 2025)

https://scholar.google.com/scholar?q=LoLCATs%3A+On+Low-Rank+Linearizing+of+Large+Language+Models

9. Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (MOHAWK) — Aviv Bick, Kevin Y. Li, Eric P. Xing, J. Zico Kolter, Albert Gu, 2024

https://scholar.google.com/scholar?q=Transformers+to+SSMs%3A+Distilling+Quadratic+Knowledge+to+Subquadratic+Models+%28MOHAWK%29

10. The Mamba in the Llama: Distilling and Accelerating Hybrid Models — Junxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush, Tri Dao, 2024

https://scholar.google.com/scholar?q=The+Mamba+in+the+Llama%3A+Distilling+and+Accelerating+Hybrid+Models

11. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2024

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29

12. Why do LLMs attend to the first token? — Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, Razvan Pascanu, 2025

https://scholar.google.com/scholar?q=Why+do+LLMs+attend+to+the+first+token%3F

13. Short window attention enables long-term memorization — Loïc Cabannes, Maximilian Beck, Maria Lomeli, Gergely Szilvasy, Matthijs Douze, Jade Copet, Pierre-Emmanuel Mazaré, Gabriel Synnaeve, Hervé Jégou, 2026

https://scholar.google.com/scholar?q=Short+window+attention+enables+long-term+memorization

14. SWAA: Sliding Window Attention Adaptation for Efficient and Quality Preserving Long Context Processing — Yijiong Yu, Jiale Liu, Qingyun Wu, Huazheng Wang, Ji Pei, 2025

https://scholar.google.com/scholar?q=SWAA%3A+Sliding+Window+Attention+Adaptation+for+Efficient+and+Quality+Preserving+Long+Context+Processing

15. STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs — Weikang Meng, Liangyu Huo, Yadan Luo, Jiawen Guan, Jingyi Zhang, Yingjian Li, Zheng Zhang, 2026

https://scholar.google.com/scholar?q=STILL%3A+Selecting+Tokens+for+Intra-Layer+Hybrid+Attention+to+Linearize+LLMs

16. Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (MOHAWK) and Llamba: Scaling Distilled Recurrent Models — Aviv Bick, Kevin Y. Li, Eric P. Xing, J. Zico Kolter, Albert Gu; Aviv Bick, Tobias Katsch, Nimit Sohoni, Arjun Desai, Albert Gu, 2024/2025

https://scholar.google.com/scholar?q=Transformers+to+SSMs%3A+Distilling+Quadratic+Knowledge+to+Subquadratic+Models+%28MOHAWK%29+and+Llamba%3A+Scaling+Distilled+Recurrent+Models

17. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models — Soham De, Samuel L. Smith, Anushan Fernando, et al., 2024

https://scholar.google.com/scholar?q=Griffin%3A+Mixing+Gated+Linear+Recurrences+with+Local+Attention+for+Efficient+Language+Models

18. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models; SnapKV: LLM Knows What You Are Looking For Before Generation — Zhenyu Zhang et al.; Yuhong Li et al., 2023/2024

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models%3B+SnapKV%3A+LLM+Knows+What+You+Are+Looking+For+Before+Generation

19. RULER: What's the Real Context Size of Your Long-Context Language Models?; BABILong — Cheng-Ping Hsieh et al.; Yuri Kuratov et al., 2024

https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F%3B+BABILong

20. Gated Delta Networks: Improving Mamba2 with Delta Rule; Parallelizing Linear Transformers with the Delta Rule — Songlin Yang, Jan Kautz, Ali Hatamizadeh; Songlin Yang et al., 2025/2024

https://scholar.google.com/scholar?q=Gated+Delta+Networks%3A+Improving+Mamba2+with+Delta+Rule%3B+Parallelizing+Linear+Transformers+with+the+Delta+Rule

Interactive Visualization: Sliding-Window Attention Beats Linear Attention Without Any Training

This episode examines "Harness 2," a paper on recursive agent harnessing. Its premise is that the wrapper around a frozen model can be improved automatically on a stream of unlabeled tasks, with no ground-truth verifier. The "harness" here is eight editable files: prompts, guardrails, memory, skills, tools, scripts, subagents, and middleware. The discussion places the work against prior approaches. These include SWE-agent's interface findings, Meta-Harness's freeze-after-search approach, per-instance scaling, ReasoningBank, Reflexion, and the Darwin Gödel Machine. It then explains the two coupled loops. Task-level refinement edits the execution harness for the current task. Harnessing-level recursion folds the lessons into a separate improvement harness, made of a library of adopted harnesses plus an EXPERIENCE.md file, while each new task still starts from a common base. The episode begins a walkthrough of the Propose, Probe, Compose refinement step, using a JobBench case that rose from 43.1 to 77.8. Listeners interested in self-improving agents will find it useful for seeing why self-correction without external feedback is hard and how this design tries to work around that.

Sources:

1. Harness^2: Recursive Agent Harnessing for an Open World — Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu, Bhavana Dalvi Mishra, Yifei Ming, Han Yu, Rujun Han, Chen-Yu Lee, Tomas Pfister, 2026

http://arxiv.org/abs/2610.14224

2. Reflexion: Language Agents with Verbal Reinforcement Learning — Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao, 2023

https://scholar.google.com/scholar?q=Reflexion%3A+Language+Agents+with+Verbal+Reinforcement+Learning

3. Automated Design of Agentic Systems (ADAS) — Shengran Hu, Cong Lu, Jeff Clune, 2024

https://scholar.google.com/scholar?q=Automated+Design+of+Agentic+Systems+%28ADAS%29

4. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents — Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, Jeff Clune, 2025

https://scholar.google.com/scholar?q=Darwin+G%C3%B6del+Machine%3A+Open-Ended+Evolution+of+Self-Improving+Agents

5. ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory — Siru Ouyang et al. (Google Cloud AI Research and collaborators), 2025

https://scholar.google.com/scholar?q=ReasoningBank%3A+Scaling+Agent+Self-Evolving+with+Reasoning+Memory

6. Self-Refine: Iterative Refinement with Self-Feedback — Aman Madaan, Niket Tandon, Peter Clark, Yiming Yang, et al., 2023

https://scholar.google.com/scholar?q=Self-Refine%3A+Iterative+Refinement+with+Self-Feedback

7. Large Language Models Cannot Self-Correct Reasoning Yet — Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, Denny Zhou, 2023

https://scholar.google.com/scholar?q=Large+Language+Models+Cannot+Self-Correct+Reasoning+Yet

8. Large Language Models as Optimizers (OPRO) — Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen, 2023

https://scholar.google.com/scholar?q=Large+Language+Models+as+Optimizers+%28OPRO%29

9. Meta-Harness (Lee et al.) — Lee et al. (as cited in the Harness2 paper; full author list not verified), 2026

https://scholar.google.com/scholar?q=Meta-Harness+%28Lee+et+al.%29

10. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering — John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, Ofir Press, 2024

https://scholar.google.com/scholar?q=SWE-agent%3A+Agent-Computer+Interfaces+Enable+Automated+Software+Engineering

11. Meta-Harness: End-to-end optimization of model harnesses — Y. Lee, R. S. Nair, Q. Zhang, K. Lee, O. Khattab, C. Finn, 2026

https://scholar.google.com/scholar?q=Meta-Harness%3A+End-to-end+optimization+of+model+harnesses

12. Rethinking the evaluation of harness evolution for agents — Y. Wang, H. Zhu, Z. Hu, et al., 2026

https://scholar.google.com/scholar?q=Rethinking+the+evaluation+of+harness+evolution+for+agents

13. Recursive harness self-improvement; and TTHE: Test-time harness evolution — H. Lee, J. Xu, J. Seely, D. Lee, M. Zaharia, Y. Tang (2026a); J. Nie et al. (2026b), 2026

https://scholar.google.com/scholar?q=Recursive+harness+self-improvement%3B+and+TTHE%3A+Test-time+harness+evolution

14. Hyperagents; Darwin Gödel Machine; Self-taught Optimizer (STOP) — J. Zhang et al. (Hyperagents 2026, DGM 2026); E. Zelikman et al. (STOP 2024), 2024-2026

https://scholar.google.com/scholar?q=Hyperagents%3B+Darwin+G%C3%B6del+Machine%3B+Self-taught+Optimizer+%28STOP%29

15. Harness continual learning (HCL) and Hierarchical self-improvement (HSI) — B. Kang et al. (2026); T. Zhou (2026), 2026

https://scholar.google.com/scholar?q=Harness+continual+learning+%28HCL%29+and+Hierarchical+self-improvement+%28HSI%29

16. GEPA: Reflective prompt evolution can outperform reinforcement learning — L. A. Agrawal et al., 2026

https://scholar.google.com/scholar?q=GEPA%3A+Reflective+prompt+evolution+can+outperform+reinforcement+learning

17. Recursive Self-Aggregation Unlocks Deep Thinking in LLMs — S. Venkatraman et al., 2025

https://scholar.google.com/scholar?q=Recursive+Self-Aggregation+Unlocks+Deep+Thinking+in+LLMs

18. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena; LLM Evaluators Recognize and Favor Their Own Generations — L. Zheng et al. (2023); A. Panickssery et al. (2024), 2023-2024

https://scholar.google.com/scholar?q=Judging+LLM-as-a-Judge+with+MT-Bench+and+Chatbot+Arena%3B+LLM+Evaluators+Recognize+and+Favor+Their+Own+Generations

19. Scaling Laws for Reward Model Overoptimization — L. Gao, J. Schulman, J. Hilton, 2023

https://scholar.google.com/scholar?q=Scaling+Laws+for+Reward+Model+Overoptimization

20. Reflexion: Language agents with verbal reinforcement learning; Self-Refine — N. Shinn et al. (2023); A. Madaan et al. (2023), 2023

https://scholar.google.com/scholar?q=Reflexion%3A+Language+agents+with+verbal+reinforcement+learning%3B+Self-Refine

21. Automated Design of Agentic Systems (ADAS); AFlow; DSPy — S. Hu, C. Lu, J. Clune (2025); J. Zhang et al. (2025); O. Khattab et al. (2024), 2024-2025

https://scholar.google.com/scholar?q=Automated+Design+of+Agentic+Systems+%28ADAS%29%3B+AFlow%3B+DSPy

Interactive Visualization: Harness2: Recursive Self-Improving Agent Harnesses Without Verifiers

This episode examines "Vector Policy Optimization," a proposed drop-in replacement for the GRPO advantage that trains models to produce diverse candidates rather than a single best answer, so that test-time search such as best-of-N and evolutionary code search has more to work with. It lays out why standard RL post-training collapses onto one high-reward mode. The evidence cited is that RL models beat base models at small k but fall behind at large k, and that KL-regularized RL is argued to collapse structurally. The method treats rewards as vectors, such as per-test-case or per-hop scores. It samples random scalarization weights from a uniform Dirichlet during training, so candidates can land at different points on the Pareto frontier. The discussion compares this with pass@k optimization, entropy bonuses and embedding-based diversity rewards, and with a multi-answer baseline that gives the model room to diversify but no incentive to do so. It also defines the evaluation metrics, pass@k, best@k and reward-space diversity. For listeners, the interest is the argument that giving up some single-shot accuracy can pay off once a search procedure sits on top of the model.

Sources:

1. Vector Policy Optimization: Training for Diversity Improves Test-Time Search — Ryan Bahlous-Boldi, Isha Puri, Idan Shenfeld, Akarsh Kumar, Mehul Damani, Sebastian Risi, Omar Khattab, Zhang-Wei Hong, Pulkit Agrawal, 2026

http://arxiv.org/abs/2605.22817

2. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. (DeepSeek-AI), 2024

https://scholar.google.com/scholar?q=DeepSeekMath%3A+Pushing+the+Limits+of+Mathematical+Reasoning+in+Open+Language+Models

3. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? — Yang Yue, Zhiqi Chen, Rui Lu, et al., 2025

https://scholar.google.com/scholar?q=Does+Reinforcement+Learning+Really+Incentivize+Reasoning+Capacity+in+LLMs+Beyond+the+Base+Model%3F

4. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — Bradley Brown, Jordan Juravsky, Ryan Ehrlich, et al., 2024

https://scholar.google.com/scholar?q=Large+Language+Monkeys%3A+Scaling+Inference+Compute+with+Repeated+Sampling

5. Understanding the Effects of RLHF on LLM Generalisation and Diversity — Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, et al., 2024

https://scholar.google.com/scholar?q=Understanding+the+Effects+of+RLHF+on+LLM+Generalisation+and+Diversity

6. A Survey of Multi-Objective Sequential Decision-Making — Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, Richard Dazeley, 2013

https://scholar.google.com/scholar?q=A+Survey+of+Multi-Objective+Sequential+Decision-Making

7. A Practical Guide to Multi-Objective Reinforcement Learning and Planning — Conor F. Hayes, Roxana Rădulescu, Eugenio Bargiacchi, et al., 2022

https://scholar.google.com/scholar?q=A+Practical+Guide+to+Multi-Objective+Reinforcement+Learning+and+Planning

8. A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation — Runzhe Yang, Xingyuan Sun, Karthik Narasimhan, 2019

https://scholar.google.com/scholar?q=A+Generalized+Algorithm+for+Multi-Objective+Reinforcement+Learning+and+Policy+Adaptation

9. Rewarded Soups: Towards Pareto-Optimal Alignment by Interpolating Weights Fine-Tuned on Diverse Rewards — Alexandre Ramé, Guillaume Couairon, Corentin Dancette, et al., 2023

https://scholar.google.com/scholar?q=Rewarded+Soups%3A+Towards+Pareto-Optimal+Alignment+by+Interpolating+Weights+Fine-Tuned+on+Diverse+Rewards

10. Pass@k Policy Optimization: Solving Harder Reinforcement Learning Problems — Christian Walder, Deep Karkhanis, 2025

https://scholar.google.com/scholar?q=Pass%40k+Policy+Optimization%3A+Solving+Harder+Reinforcement+Learning+Problems

11. Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models — Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, et al. (Google DeepMind), 2024

https://scholar.google.com/scholar?q=Inference-Aware+Fine-Tuning+for+Best-of-N+Sampling+in+Large+Language+Models

12. AlphaEvolve: A coding agent for scientific and algorithmic discovery — Alexander Novikov, Ngân Vũ, Marvin Eisenberger, et al. (Google DeepMind), 2025

https://scholar.google.com/scholar?q=AlphaEvolve%3A+A+coding+agent+for+scientific+and+algorithmic+discovery

13. Mathematical discoveries from program search with large language models (FunSearch) — Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, et al., 2024

https://scholar.google.com/scholar?q=Mathematical+discoveries+from+program+search+with+large+language+models+%28FunSearch%29

14. ShinkaEvolve: Towards Open-Ended and Sample-Efficient Program Evolution — Robert Tjarko Lange, Yuki Imajuku, Edoardo Cetin (Sakana AI), 2025

https://scholar.google.com/scholar?q=ShinkaEvolve%3A+Towards+Open-Ended+and+Sample-Efficient+Program+Evolution

15. Illuminating search spaces by mapping elites (MAP-Elites) / Quality Diversity: A New Frontier for Evolutionary Computation — Jean-Baptiste Mouret and Jeff Clune (2015); Justin K. Pugh, Lisa B. Soros, Kenneth O. Stanley (2016), 2015-2016

https://scholar.google.com/scholar?q=Illuminating+search+spaces+by+mapping+elites+%28MAP-Elites%29+%2F+Quality+Diversity%3A+A+New+Frontier+for+Evolutionary+Computation

16. Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models — Isha Puri, Mehul Damani, Idan Shenfeld, Marzyeh Ghassemi, Jacob Andreas, Yoon Kim, 2026

https://scholar.google.com/scholar?q=Reaching+Beyond+the+Mode%3A+RL+for+Distributional+Reasoning+in+Language+Models

17. KL-Regularized Reinforcement Learning is Designed to Mode Collapse — Anthony GX-Chen, Jatin Prakash, Jeff Guo, Rob Fergus, Rajesh Ranganath, 2025

https://scholar.google.com/scholar?q=KL-Regularized+Reinforcement+Learning+is+Designed+to+Mode+Collapse

18. The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisation — Farid Bagirov, Mikhail Arkhipov, Ksenia Sycheva, Evgeniy Glukhov, Egor Bogomolov, 2025

https://scholar.google.com/scholar?q=The+Best+of+N+Worlds%3A+Aligning+Reinforcement+Learning+with+Best-of-N+Sampling+via+max%40k+Optimisation

19. Maximum Likelihood Reinforcement Learning (MaxRL) — Fahim Tajwar, Guanning Zeng, Yueer Zhou, Yuda Song, et al., 2026

https://scholar.google.com/scholar?q=Maximum+Likelihood+Reinforcement+Learning+%28MaxRL%29

20. GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization — Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, et al. (NVIDIA), 2026

https://scholar.google.com/scholar?q=GDPO%3A+Group+reward-Decoupled+Normalization+Policy+Optimization+for+Multi-reward+RL+Optimization

21. A Survey of Multi-Objective Sequential Decision-Making / A Practical Guide to Multi-Objective RL and Planning — Roijers, Vamplew, Whiteson, Dazeley (2013); Hayes et al. (2022), 2013/2022

https://scholar.google.com/scholar?q=A+Survey+of+Multi-Objective+Sequential+Decision-Making+%2F+A+Practical+Guide+to+Multi-Objective+RL+and+Planning

22. Lexicase selection (Spector 2012; La Cava et al. 2019; Ni et al. 2024, DALex) — Lee Spector; William La Cava, Thomas Helmuth, et al.; Andrew Ni, Li Ding, Lee Spector, 2012/2019/2024

https://scholar.google.com/scholar?q=Lexicase+selection+%28Spector+2012%3B+La+Cava+et+al.+2019%3B+Ni+et+al.+2024%2C+DALex%29

23. Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models / BOND / BoNBoN / InfAlign — Chow et al.; Sessa et al.; Gui, Gârbacea, Veitch; Balashankar et al., 2024-2025

https://scholar.google.com/scholar?q=Inference-Aware+Fine-Tuning+for+Best-of-N+Sampling+in+Large+Language+Models+%2F+BOND+%2F+BoNBoN+%2F+InfAlign

24. AlphaEvolve / FunSearch / ShinkaEvolve / OpenEvolve — Novikov et al.; Romera-Paredes et al.; Lange, Imajuku, Cetin; Sharma, 2023-2025

https://scholar.google.com/scholar?q=AlphaEvolve+%2F+FunSearch+%2F+ShinkaEvolve+%2F+OpenEvolve

25. Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment — Rui Yang, Xiaoman Pan, Feng Luo, et al., 2024

https://scholar.google.com/scholar?q=Rewards-in-Context%3A+Multi-objective+Alignment+of+Foundation+Models+with+Dynamic+Preference+Adjustment

26. Rewarded Soups / Directional Preference Alignment — Rame et al. (2023); Wang et al. (2024), 2023/2024

https://scholar.google.com/scholar?q=Rewarded+Soups+%2F+Directional+Preference+Alignment

27. DARLING (diversity-aware RL) and entropy-preserving GRPO variants (DAPO clip-higher) — Li et al.; Yu et al., 2025

https://scholar.google.com/scholar?q=DARLING+%28diversity-aware+RL%29+and+entropy-preserving+GRPO+variants+%28DAPO+clip-higher%29

Interactive Visualization: Vector Policy Optimization: Training for Diversity to Improve Test-Time Search

This episode examines the OpenAI paper on large-batch training, which asks why ImageNet training stalls at batches of tens of thousands of images while Dota 2 agents train on millions of observations. It explains the tradeoff between serial optimizer steps and total examples processed. This tradeoff forms a Pareto frontier whose knee is the critical batch size, where compute efficiency is 50 percent. The central claim is that the gradient noise scale, a statistic computable from a single ordinary training run, predicts that critical batch size to within an order of magnitude. The discussion walks through the second-order loss derivation behind it, where progress per step is halved when batch size equals the noise scale. It also covers the theory's predictions: the noise scale grows during training, is larger for harder tasks and for reinforcement learning, and depends only weakly on model size. The authors admit the derivation rests on many unfounded assumptions. A listener would find it useful for understanding how to choose batch sizes, including why batch sizes that increase over training make sense, without brute-force sweeps.

Sources:

1. An Empirical Model of Large-Batch Training — Sam McCandlish, Jared Kaplan, Dario Amodei, OpenAI Dota Team, 2018

http://arxiv.org/abs/1812.06162

2. An Empirical Model of Large-Batch Training — Sam McCandlish, Jared Kaplan, Dario Amodei, OpenAI Dota Team, 2018

https://scholar.google.com/scholar?q=An+Empirical+Model+of+Large-Batch+Training

3. Measuring the Effects of Data Parallelism on Neural Network Training — Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, George E. Dahl, 2018

https://scholar.google.com/scholar?q=Measuring+the+Effects+of+Data+Parallelism+on+Neural+Network+Training

4. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour — Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, Kaiming He, 2017

https://scholar.google.com/scholar?q=Accurate%2C+Large+Minibatch+SGD%3A+Training+ImageNet+in+1+Hour

5. Large Batch Training of Convolutional Networks (LARS) — Yang You, Igor Gitman, Boris Ginsburg, 2017

https://scholar.google.com/scholar?q=Large+Batch+Training+of+Convolutional+Networks+%28LARS%29

6. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima — Nitish Shirish Keskar, Dheevatcha Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang, 2016

https://scholar.google.com/scholar?q=On+Large-Batch+Training+for+Deep+Learning%3A+Generalization+Gap+and+Sharp+Minima

7. Don't Decay the Learning Rate, Increase the Batch Size — Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le, 2017

https://scholar.google.com/scholar?q=Don%27t+Decay+the+Learning+Rate%2C+Increase+the+Batch+Size

8. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, 2020

https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models

9. Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model — Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E. Dahl, Christopher J. Shallue, Roger Grosse, 2019

https://scholar.google.com/scholar?q=Which+Algorithmic+Choices+Matter+at+Which+Batch+Sizes%3F+Insights+From+a+Noisy+Quadratic+Model

10. Three Factors Influencing Minima in SGD — Stanisław Jastrzębski, Zachary Kenton, Devansh Arpit, et al., 2017

https://scholar.google.com/scholar?q=Three+Factors+Influencing+Minima+in+SGD

11. Dota 2 with Large Scale Deep Reinforcement Learning — OpenAI, Christopher Berner, Greg Brockman, et al., 2019

https://scholar.google.com/scholar?q=Dota+2+with+Large+Scale+Deep+Reinforcement+Learning

12. Online Batch Selection / Adaptive sampling via gradient noise (Balles, Romero, Hennig) — Lukas Balles, Javier Romero, Philipp Hennig, 2016

https://scholar.google.com/scholar?q=Online+Batch+Selection+%2F+Adaptive+sampling+via+gradient+noise+%28Balles%2C+Romero%2C+Hennig%29

Interactive Visualization: Gradient Noise Scale Predicts Critical Batch Size for Large-Batch Training

This episode examines FastPersist, a DeepSpeed paper on making deep learning model checkpointing fast enough to save every training iteration. It lays out why checkpoint time has grown to 50–96% of iteration time as data parallelism scaled, and why a checkpoint is so large: mixed-precision Adam state works out to about 14 bytes per parameter, or 173 GB for a 13B model. The discussion traces the slowness to torch.save issuing small, unaligned writes. That leaves NVMe drives well below peak, with the baseline reaching under 20% of the 24.8 GB/s a RAID-0 array can deliver. It also covers the structural opportunity to overlap checkpointing with the next forward and backward passes, and to spread writes across data-parallel replicas that hold identical state. The episode compares the approach with CheckFreq, Check-N-Run and Gemini, and it questions whether checkpointing every iteration is the right target. It also notes what the paper leaves out, such as SSD write-cache behavior and flash wear.

Sources:

1. FastPersist: Accelerating Model Checkpointing in Deep Learning — Guanhua Wang, Olatunji Ruwase, Bing Xie, Yuxiong He, 2024

http://arxiv.org/abs/2406.13768

2. CheckFreq: Frequent, Fine-Grained DNN Checkpointing — Jayashree Mohan, Amar Phanishayee, Vijay Chidambaram, 2021

https://scholar.google.com/scholar?q=CheckFreq%3A+Frequent%2C+Fine-Grained+DNN+Checkpointing

3. Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints — Zhuang Wang, Zhen Jia, Shuai Zheng, Zhen Zhang, Xinwei Fu, T. S. Eugene Ng, Yida Wang, 2023

https://scholar.google.com/scholar?q=Gemini%3A+Fast+Failure+Recovery+in+Distributed+Training+with+In-Memory+Checkpoints

4. Check-N-Run: A Checkpointing System for Training Deep Learning Recommendation Models — Assaf Eisenman, Kiran Kumar Matam, Steven Ingram, Dheevatsa Mudigere, Raghuraman Krishnamoorthi, Krishnakumar Nair, Misha Smelyanskiy, Murali Annavaram, 2022

https://scholar.google.com/scholar?q=Check-N-Run%3A+A+Checkpointing+System+for+Training+Deep+Learning+Recommendation+Models

5. ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development — Borui Wan, Mingji Han, Yiyao Sheng, Yanghua Peng, Haibin Lin, Mofan Zhang, Zhichao Lai, Menghan Yu, Junda Zhang, Zuquan Song, Xin Liu, Chuan Wu, 2024

https://scholar.google.com/scholar?q=ByteCheckpoint%3A+A+Unified+Checkpointing+System+for+Large+Foundation+Model+Development

6. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning — Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He, 2021

https://scholar.google.com/scholar?q=ZeRO-Infinity%3A+Breaking+the+GPU+Memory+Wall+for+Extreme+Scale+Deep+Learning

7. Characterizing Deep Learning Training Workloads on Alibaba-PAI / Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training — Mark Zhao, Niket Agarwal, Aarti Basant, Buğra Gedik, Satadru Pan, Mustafa Ozdal, Rakesh Komuravelli, Jerry Pan, Tianshu Bao, Haowei Lu, Sundaram Narayanan, Jack Langman, Kevin Wilfong, Harsha Rastogi, Carole-Jean Wu, Christos Kozyrakis, Parik Pol, 2022

https://scholar.google.com/scholar?q=Characterizing+Deep+Learning+Training+Workloads+on+Alibaba-PAI+%2F+Understanding+Data+Storage+and+Ingestion+for+Large-Scale+Deep+Recommendation+Model+Training

8. What Modern NVMe Storage Can Do, And How To Exploit It: High-Performance I/O for High-Performance Storage Engines — Gabriel Haas, Viktor Leis, 2023

https://scholar.google.com/scholar?q=What+Modern+NVMe+Storage+Can+Do%2C+And+How+To+Exploit+It%3A+High-Performance+I%2FO+for+High-Performance+Storage+Engines

9. PyTorch Distributed Checkpoint (DCP) and async_save — PyTorch team (Meta), 2023-2024

https://scholar.google.com/scholar?q=PyTorch+Distributed+Checkpoint+%28DCP%29+and+async_save

10. Reducing Activation Recomputation / OPT-175B logbook and Llama 3 herd paper (failure analyses) — Susan Zhang et al.; Llama Team, Meta AI, 2022; 2024

https://scholar.google.com/scholar?q=Reducing+Activation+Recomputation+%2F+OPT-175B+logbook+and+Llama+3+herd+paper+%28failure+analyses%29

11. The Young/Daly optimal checkpoint interval and multi-level checkpointing (e.g., SCR, FTI) — J. Young; J. Daly; Moody et al.; Bautista-Gomez et al., 1974; 2006; 2010-2011

https://scholar.google.com/scholar?q=The+Young%2FDaly+optimal+checkpoint+interval+and+multi-level+checkpointing+%28e.g.%2C+SCR%2C+FTI%29

12. GPUDirect Storage / cuFile and SPIN — NVIDIA; Bergman, Brokhman, Cohen, Silberstein, 2017-2022

https://scholar.google.com/scholar?q=GPUDirect+Storage+%2F+cuFile+and+SPIN

13. ReCycle / Oobleck / Bamboo style resilient training without checkpoint restart — Various, 2023-2024

https://scholar.google.com/scholar?q=ReCycle+%2F+Oobleck+%2F+Bamboo+style+resilient+training+without+checkpoint+restart

Interactive Visualization: FastPersist: Making Per-Iteration Deep Learning Checkpointing Nearly Free

This episode examines DepthBench, a study of why stacking more transformer layers often adds little useful computation, and which residual and normalization designs fix that when parameter count and training recipe are held constant. It reviews the Pre-LN "Curse of Depth" argument, in which growing residual-stream magnitude makes late layers act like identity. It then surveys ten architectures in three families: normalization tweaks such as LayerNorm Scaling and DeepNorm, multi-stream residuals such as Hyper-Connections and mHC, and cross-layer access such as Attention Residuals. The central claim is that the benefit of going deeper and narrower depends on the architecture: it can hurt under standard Pre-LN but keeps helping under two other designs. That would mean the common shallow-wide shapes are partly an artifact of the residual design. The discussion sets this against Kaplan-style scaling results, layer-pruning studies, and the rival explanation that deep models are simply mis-scaled in training. Listeners get a clear framework for "effective computational depth" and for judging whether the evidence supports the claim.

Sources:

1. DepthBench: Measuring How Residual Connections Enable More Computational Depth — Keyu Wang, Yangyi Huang, Jiale Kang, David González-Martínez, Weiyang Liu, Shiwei Liu, 2026

http://arxiv.org/abs/2609.32534

2. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, et al., 2020

https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models

3. Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers — Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler, 2021

https://scholar.google.com/scholar?q=Scale+Efficiently%3A+Insights+from+Pre-training+and+Fine-tuning+Transformers

4. MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases — Zechun Liu, Changsheng Zhao, Forrest Iandola, et al. (Meta), 2024

https://scholar.google.com/scholar?q=MobileLLM%3A+Optimizing+Sub-billion+Parameter+Language+Models+for+On-Device+Use+Cases

5. The Curse of Depth in Large Language Models — Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, Shiwei Liu, 2025

https://scholar.google.com/scholar?q=The+Curse+of+Depth+in+Large+Language+Models

6. Hyper-Connections (and follow-up: mHC, Manifold-Constrained Hyper-Connections) — Defa Zhu, Hongzhi Huang, Zihao Huang, et al. (ByteDance Seed) for HC; DeepSeek-AI researchers (Xie et al.) for mHC, 2024-2025

https://scholar.google.com/scholar?q=Hyper-Connections+%28and+follow-up%3A+mHC%2C+Manifold-Constrained+Hyper-Connections%29

7. Do Language Models Use Their Depth Efficiently? — Róbert Csordás, Christopher D. Manning, Christopher Potts, 2025

https://scholar.google.com/scholar?q=Do+Language+Models+Use+Their+Depth+Efficiently%3F

8. The Unreasonable Ineffectiveness of the Deeper Layers — Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, Daniel A. Roberts, 2024

https://scholar.google.com/scholar?q=The+Unreasonable+Ineffectiveness+of+the+Deeper+Layers

9. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect — Xin Men et al., 2024

https://scholar.google.com/scholar?q=ShortGPT%3A+Layers+in+Large+Language+Models+are+More+Redundant+Than+You+Expect

10. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (muP) — Greg Yang et al., 2022

https://scholar.google.com/scholar?q=Tensor+Programs+V%3A+Tuning+Large+Neural+Networks+via+Zero-Shot+Hyperparameter+Transfer+%28muP%29

11. Don't be lazy: CompleteP enables compute-efficient deep transformers — Nolan Dey et al., 2025

https://scholar.google.com/scholar?q=Don%27t+be+lazy%3A+CompleteP+enables+compute-efficient+deep+transformers

12. Scaling Laws with Model Shape / Gemstones: A Model Suite for Multi-Faceted Scaling Laws — Sean McLeish et al., 2025

https://scholar.google.com/scholar?q=Scaling+Laws+with+Model+Shape+%2F+Gemstones%3A+A+Model+Suite+for+Multi-Faceted+Scaling+Laws

13. The Impact of Depth on Compositional Generalization in Transformer Language Models — Jackson Petty et al., 2024

https://scholar.google.com/scholar?q=The+Impact+of+Depth+on+Compositional+Generalization+in+Transformer+Language+Models

14. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping et al., 2025

https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach

15. Transformers, Parallel Computation, and Logarithmic Depth — Clayton Sanford, Daniel Hsu, Matus Telgarsky, 2024

https://scholar.google.com/scholar?q=Transformers%2C+Parallel+Computation%2C+and+Logarithmic+Depth

16. The Expressive Power of Transformers with Chain of Thought / The Parallelism Tradeoff — William Merrill, Ashish Sabharwal, 2023-2024

https://scholar.google.com/scholar?q=The+Expressive+Power+of+Transformers+with+Chain+of+Thought+%2F+The+Parallelism+Tradeoff

17. Resolving Discrepancies in Compute-Optimal Scaling of Language Models — Tomer Porian et al., 2024

https://scholar.google.com/scholar?q=Resolving+Discrepancies+in+Compute-Optimal+Scaling+of+Language+Models

18. DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging — Matteo Pagliardini et al., 2024

https://scholar.google.com/scholar?q=DenseFormer%3A+Enhancing+Information+Flow+in+Transformers+via+Depth+Weighted+Averaging

19. Peri-LN: Revisiting Layer Normalization in the Transformer Architecture — Jeonghoon Kim et al., 2025

https://scholar.google.com/scholar?q=Peri-LN%3A+Revisiting+Layer+Normalization+in+the+Transformer+Architecture

Interactive Visualization: DepthBench: How Residual Connections Unlock Computational Depth in Transformers

This episode examines a paper that uses an LLM-driven evolutionary search (OpenEvolve with Claude Mythos 5) to find quantum arithmetic circuits needing fewer Toffoli gates and fewer qubits than the best published designs across eight tasks. These tasks include decoders for decoded quantum interferometry, modular inversion, and chemistry and lattice-gauge workloads. It explains why quantum arithmetic is hard to write: coherent control flow forces every branch to run, reversibility requires uncomputation, and a stated failure budget can be traded for savings. It also covers how candidates are checked classically by tracking basis states and phases, so hallucinated circuits are rejected by an exact evaluator. The discussion places the work against FunSearch, AlphaEvolve and rewrite-based optimizers, and notes the trade-off that correctness is statistical rather than proved. It also flags the Anthropic affiliation and the simplification of equally weighting gate count and qubit count into one cost metric.

Sources:

1. Discovering state-of-the-art quantum circuits with large language models — Andi Gu, Alek H. Kemeny, Susanne F. Yelin, Mikhail D. Lukin, 2026

http://arxiv.org/abs/2610.22367

2. How to factor 2048 bit RSA integers in 8 hours using 20 million noisy qubits — Craig Gidney, Martin Ekera, 2019 (Quantum 2021)

https://scholar.google.com/scholar?q=How+to+factor+2048+bit+RSA+integers+in+8+hours+using+20+million+noisy+qubits

3. Encoding Electronic Spectra in Quantum Circuits with Linear T Complexity — Ryan Babbush, Craig Gidney, Dominic Berry, Nathan Wiebe, Jarrod McClean, Alexandru Paler, Austin Fowler, Hartmut Neven, 2018

https://scholar.google.com/scholar?q=Encoding+Electronic+Spectra+in+Quantum+Circuits+with+Linear+T+Complexity

4. AlphaEvolve: A coding agent for scientific and algorithmic discovery (and the preceding FunSearch: Mathematical discoveries from program search with large language models) — Alexander Novikov, Ngan Vu, Marvin Eisenberger, et al. (Google DeepMind); FunSearch: Bernardino Romera-Paredes, Mohammadamin Barekatain, et al., 2025 (AlphaEvolve); 2023 (FunSearch, Nature 2024)

https://scholar.google.com/scholar?q=AlphaEvolve%3A+A+coding+agent+for+scientific+and+algorithmic+discovery+%28and+the+preceding+FunSearch%3A+Mathematical+discoveries+from+program+search+with+large+language+models%29

5. Discovering faster matrix multiplication algorithms with reinforcement learning (AlphaTensor) and Quantum Circuit Optimization with AlphaTensor (AlphaTensor-Quantum) — Alhussein Fawzi et al.; Francisco J. R. Ruiz et al. (DeepMind), 2022 (AlphaTensor); 2024 (AlphaTensor-Quantum, Nature Machine Intelligence 2025)

https://scholar.google.com/scholar?q=Discovering+faster+matrix+multiplication+algorithms+with+reinforcement+learning+%28AlphaTensor%29+and+Quantum+Circuit+Optimization+with+AlphaTensor+%28AlphaTensor-Quantum%29

6. Logical Reversibility of Computation — Charles H. Bennett, 1973

https://scholar.google.com/scholar?q=Logical+Reversibility+of+Computation

7. Quantum Resource Estimates for Computing Elliptic Curve Discrete Logarithms — Martin Roetteler, Michael Naehrig, Krysta M. Svore, Kristin Lauter, 2017

https://scholar.google.com/scholar?q=Quantum+Resource+Estimates+for+Computing+Elliptic+Curve+Discrete+Logarithms

8. Fast constant-time gcd computation and modular inversion — Daniel J. Bernstein, Bo-Yin Yang, 2019

https://scholar.google.com/scholar?q=Fast+constant-time+gcd+computation+and+modular+inversion

9. Reversible Residual Network: Backpropagation Without Storing Activations (RevNet) — Aidan N. Gomez, Mengye Ren, Raquel Urtasun, Roger B. Grosse, 2017

https://scholar.google.com/scholar?q=Reversible+Residual+Network%3A+Backpropagation+Without+Storing+Activations+%28RevNet%29

10. Mathematical discoveries from program search with large language models (FunSearch) — Romera-Paredes et al., 2023/2024

https://scholar.google.com/scholar?q=Mathematical+discoveries+from+program+search+with+large+language+models+%28FunSearch%29

11. AlphaEvolve: A coding agent for scientific and algorithmic discovery — Novikov et al. (Google DeepMind), 2025

https://scholar.google.com/scholar?q=AlphaEvolve%3A+A+coding+agent+for+scientific+and+algorithmic+discovery

12. Quartz / PyZX / TODD / T-par style circuit optimizers — Xu et al. (Quartz); Kissinger & van de Wetering (PyZX); Heyfron & Campbell (TODD); Amy, Maslov & Mosca (T-par), 2019-2022

https://scholar.google.com/scholar?q=Quartz+%2F+PyZX+%2F+TODD+%2F+T-par+style+circuit+optimizers

13. Faster quantum arithmetic / AlphaTensor-Quantum: Quantum circuit optimization with deep reinforcement learning — Ruiz et al. (DeepMind), 2024/2025

https://scholar.google.com/scholar?q=Faster+quantum+arithmetic+%2F+AlphaTensor-Quantum%3A+Quantum+circuit+optimization+with+deep+reinforcement+learning

14. Decoded Quantum Interferometry — Jordan et al., 2024

https://scholar.google.com/scholar?q=Decoded+Quantum+Interferometry

15. Compiling decoded quantum interferometry / reversible Reed-Solomon decoders for DQI (Khattar et al.) — Khattar et al., 2025

https://scholar.google.com/scholar?q=Compiling+decoded+quantum+interferometry+%2F+reversible+Reed-Solomon+decoders+for+DQI+%28Khattar+et+al.%29

16. Fast constant-time gcd computation and modular inversion (safegcd / divsteps) — Bernstein & Yang, 2019

https://scholar.google.com/scholar?q=Fast+constant-time+gcd+computation+and+modular+inversion+%28safegcd+%2F+divsteps%29

17. How to factor 2048 bit RSA integers with less than a million noisy qubits — Gidney, 2025

https://scholar.google.com/scholar?q=How+to+factor+2048+bit+RSA+integers+with+less+than+a+million+noisy+qubits

18. Halving the cost of quantum addition / Measurement-based uncomputation of quantum circuits — Gidney (2018); Berry et al. (2019), 2018/2019

https://scholar.google.com/scholar?q=Halving+the+cost+of+quantum+addition+%2F+Measurement-based+uncomputation+of+quantum+circuits

19. Coset representation of modular integers / approximate residue arithmetic — Zalka; Gidney, 2006/2019

https://scholar.google.com/scholar?q=Coset+representation+of+modular+integers+%2F+approximate+residue+arithmetic

20. Efficient implementations of reversible computation / Bennett pebble game space-time trade-offs — Bennett; Knill; Chan & Gidney, 1989/1995/2013

https://scholar.google.com/scholar?q=Efficient+implementations+of+reversible+computation+%2F+Bennett+pebble+game+space-time+trade-offs

21. The Reversible Residual Network: Backpropagation Without Storing Activations — Gomez et al., 2017

https://scholar.google.com/scholar?q=The+Reversible+Residual+Network%3A+Backpropagation+Without+Storing+Activations

Interactive Visualization: LLM Evolutionary Search Finds Cheaper Quantum Arithmetic Circuits

This episode examines "Latent Compaction with Test-Time Training," a preprint that asks whether a frozen model can shrink a document's KV cache by 16 to 32 times by running gradient descent on a small set of new key-value entries. The compaction happens before any questions are known. The discussion explains why the cache bottlenecks long-context memory and batching, and why text summaries lose details like file paths or IDs. It then contrasts the approach with eviction methods such as H2O, SnapKV, PyramidKV and KVZip, which can only select existing entries and never create blended ones. It also places the work among prefix-tuning, Cartridges, Attention Matching and other test-time training approaches, and clarifies that the "32x" figure counts cache slots, not bytes, with no measured speedup or memory savings. Listeners get a clear primer on query-aware versus query-agnostic compression, along with a skeptical look at what the headline RULER result does and doesn't show.

Sources:

1. Latent Compaction with Test-Time Training — Yingfa Chen, Xingyu Shen, Chaojun Xiao, Xu Han, 2026

http://arxiv.org/abs/2610.20582v1

2. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts — Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, Moritz Hardt, 2020

https://scholar.google.com/scholar?q=Test-Time+Training+with+Self-Supervision+for+Generalization+under+Distribution+Shifts

3. Learning to (Learn at Test Time): RNNs with Expressive Hidden States — Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, 2024

https://scholar.google.com/scholar?q=Learning+to+%28Learn+at+Test+Time%29%3A+RNNs+with+Expressive+Hidden+States

4. The Surprising Effectiveness of Test-Time Training for Few-Shot Learning — Ekin Akyürek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, Jacob Andreas, 2024

https://scholar.google.com/scholar?q=The+Surprising+Effectiveness+of+Test-Time+Training+for+Few-Shot+Learning

5. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2024

https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time

6. H2O: Heavy-Hitter Oracle: Efficient Generative Inference of Large Language Models with Heavy Hitters — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Zhangyang Wang, Beidi Chen, et al., 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Heavy+Hitters

7. SnapKV: LLM Knows What You are Looking for Before Generation — Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, Deming Chen, 2024

https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation

8. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction — Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee, Sangdoo Yun, Hyun Oh Song, 2025

https://scholar.google.com/scholar?q=KVzip%3A+Query-Agnostic+KV+Cache+Compression+with+Context+Reconstruction

9. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023

https://scholar.google.com/scholar?q=Learning+to+Compress+Prompts+with+Gist+Tokens

10. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, Christopher Ré, 2025

https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study

11. Fast KV Compaction via Attention Matching — Zweiger et al., 2026

https://scholar.google.com/scholar?q=Fast+KV+Compaction+via+Attention+Matching

12. The Power of Scale for Parameter-Efficient Prompt Tuning — Brian Lester, Rami Al-Rfou, Noah Constant, 2021

https://scholar.google.com/scholar?q=The+Power+of+Scale+for+Parameter-Efficient+Prompt+Tuning

13. Learning by Distilling Context — Charlie Snell, Dan Klein, Ruiqi Zhong, 2022

https://scholar.google.com/scholar?q=Learning+by+Distilling+Context

14. KVzap: Fast, adaptive, and faithful KV cache pruning — Jégou, Jeblick, 2026

https://scholar.google.com/scholar?q=KVzap%3A+Fast%2C+adaptive%2C+and+faithful+KV+cache+pruning

15. End-to-End Test-Time Training for Long Context — Tandon, Dalal, Li, Koceja, Rød, Buchanan, Wang, Leskovec, Koyejo, Hashimoto, Guestrin, McCaleb, Choi, Sun, 2025

https://scholar.google.com/scholar?q=End-to-End+Test-Time+Training+for+Long+Context

16. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Li, Liang, 2021

https://scholar.google.com/scholar?q=Prefix-Tuning%3A+Optimizing+Continuous+Prompts+for+Generation

17. Adapting Language Models to Compress Contexts (AutoCompressors) — Chevalier, Wettig, Ajith, Chen, 2023

https://scholar.google.com/scholar?q=Adapting+Language+Models+to+Compress+Contexts+%28AutoCompressors%29

18. In-context Autoencoder for Context Compression in a Large Language Model — Ge, Hu, Wang, Wang, Chen, Wei, 2024

https://scholar.google.com/scholar?q=In-context+Autoencoder+for+Context+Compression+in+a+Large+Language+Model

19. Repeat After Me: Transformers are Better than State Space Models at Copying — Jelassi, Brandfonbrener, Kakade, Malach, 2024

https://scholar.google.com/scholar?q=Repeat+After+Me%3A+Transformers+are+Better+than+State+Space+Models+at+Copying

20. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Xiao, Tian, Chen, Han, Lewis, 2024

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29

21. RULER: What's the Real Context Size of Your Long-Context Language Models? — Hsieh, Kriman, Acharya, Rekesh, Jia, Zhang, Ginsburg and others, 2024

https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F

22. Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation — Feng, Lv, Cao, Xie, Zhou, 2025

https://scholar.google.com/scholar?q=Ada-KV%3A+Optimizing+KV+Cache+Eviction+by+Adaptive+Budget+Allocation

23. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache — Liu, Yuan, Jin, Zhong, Xu, Braverman, Chen, Hu, 2024

https://scholar.google.com/scholar?q=KIVI%3A+A+Tuning-Free+Asymmetric+2bit+Quantization+for+KV+Cache

Interactive Visualization: Latent Compaction with Test-Time Training: Learning Compact KV Caches