AI Post Transformers

This episode examines "Memory Caching: RNNs with Growing Memory," a February 2026 paper proposing that recurrent models save checkpoints of their own memory as a sequence unfolds. The aim is to recover the recall that fixed-size states lose, at a cost between linear and quadratic. It sets up the problem with needle-in-a-haystack tests, then traces how linear attention, Titans and Deep Linear Attention treat memory as a small network trained during the forward pass. The paper's variants are introduced: summing cached memories, gating them per token, routing to a sparse few, and merging their parameters in "Memory Soup." The discussion ties Memory Soup to model soups and linear mode connectivity. Averaging weights only helps if the cached memories share a loss basin, so the episode treats that as the claim to test. Listeners get a clear explanation of why recurrent models forget and a skeptical way to judge whether this fix works.

Sources:

1. Memory Caching: Giving Recurrent Models Growing Memory

https://arxiv.org/pdf/2602.24281v1

2. Memory Caching: RNNs with Growing Memory — Ali Behrouz, Zeman Li, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, Vahab Mirrokni, 2026

https://scholar.google.com/scholar?q=Memory+Caching%3A+RNNs+with+Growing+Memory

3. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time — Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, et al., 2022

https://scholar.google.com/scholar?q=Model+soups%3A+averaging+weights+of+multiple+fine-tuned+models+improves+accuracy+without+increasing+inference+time

4. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2025

https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time

5. Averaging Weights Leads to Wider Optima and Better Generalization (Stochastic Weight Averaging) — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018

https://scholar.google.com/scholar?q=Averaging+Weights+Leads+to+Wider+Optima+and+Better+Generalization+%28Stochastic+Weight+Averaging%29

6. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret, 2020

https://scholar.google.com/scholar?q=Transformers+are+RNNs%3A+Fast+Autoregressive+Transformers+with+Linear+Attention

7. Linear Mode Connectivity and the Lottery Ticket Hypothesis — Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, Michael Carbin, 2020

https://scholar.google.com/scholar?q=Linear+Mode+Connectivity+and+the+Lottery+Ticket+Hypothesis

8. Git Re-Basin: Merging Models modulo Permutation Symmetries — Samuel K. Ainsworth, Jonathan Hayase, Siddhartha Srinivasa, 2023

https://scholar.google.com/scholar?q=Git+Re-Basin%3A+Merging+Models+modulo+Permutation+Symmetries

9. Averaging Weights Leads to Wider Optima and Better Generalization (SWA) — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018

https://scholar.google.com/scholar?q=Averaging+Weights+Leads+to+Wider+Optima+and+Better+Generalization+%28SWA%29

10. Editing Models with Task Arithmetic; TIES-Merging; Model Breadcrumbs — Ilharco et al. 2023; Yadav et al. 2023; Davari & Belilovsky 2023, 2023

https://scholar.google.com/scholar?q=Editing+Models+with+Task+Arithmetic%3B+TIES-Merging%3B+Model+Breadcrumbs

11. Log-Linear Attention — Han Guo, Songlin Yang, Tarushii Goel, Eric P. Xing, Tri Dao, Yoon Kim, 2025

https://scholar.google.com/scholar?q=Log-Linear+Attention

12. Test-Time Training with Self-Supervision / Learning to (Learn at Test Time): RNNs with Expressive Hidden States — Yu Sun, Xinhao Li, Karan Dalal, et al., 2024

https://scholar.google.com/scholar?q=Test-Time+Training+with+Self-Supervision+%2F+Learning+to+%28Learn+at+Test+Time%29%3A+RNNs+with+Expressive+Hidden+States

13. Gated Delta Networks: Improving Mamba2 with Delta Rule — Songlin Yang, Jan Kautz, Ali Hatamizadeh, 2025

https://scholar.google.com/scholar?q=Gated+Delta+Networks%3A+Improving+Mamba2+with+Delta+Rule

14. Zoology: Measuring and Improving Recall in Efficient Language Models (MQAR); Based — Simran Arora et al., 2024

https://scholar.google.com/scholar?q=Zoology%3A+Measuring+and+Improving+Recall+in+Efficient+Language+Models+%28MQAR%29%3B+Based

15. Transformer-XL; Infini-attention; Compressive Transformer — Dai et al. 2019; Munkhdalai et al. 2024; Rae et al. 2020, 2019-2024

https://scholar.google.com/scholar?q=Transformer-XL%3B+Infini-attention%3B+Compressive+Transformer

16. Mixture of Block Attention (MoBA); Native Sparse Attention — Lu et al. 2025; Yuan et al. 2025, 2025

https://scholar.google.com/scholar?q=Mixture+of+Block+Attention+%28MoBA%29%3B+Native+Sparse+Attention

Interactive Visualization: Memory Caching: Giving Recurrent Models Growing Memory

This episode examines "Automatic Curriculum Learning through Value Disagreement," a NeurIPS 2020 paper proposing that a reinforcement learning agent can choose its own training goals. It walks through the background: goal-conditioned RL with sparse rewards, Hindsight Experience Replay as an implicit curriculum, and earlier explicit approaches like Goal GAN and learning-progress teachers, which found that plain uniform sampling is a surprisingly strong baseline. The core idea is to train an ensemble of Q-functions and sample goals in proportion to the standard deviation of their predictions. The reasoning is that the ensemble agrees on easy goals and on hopeless ones, but splits on goals at the frontier of competence. The discussion explains the method as a simple loop: draw candidate goals, score each by ensemble disagreement, and sample one. It needs no adversary or success classifier, and the policy learner never sees the ensemble. Throughout, the hosts treat the disagreement-marks-the-frontier hypothesis as a claim to test rather than accept, which makes this a useful look at what a cheap uncertainty signal can and can't tell a learning agent.

Sources:

1. Automatic Curriculum Learning Through Value Disagreement in Reinforcement Learning

https://proceedings.neurips.cc/paper/2020/file/566f0ea4f6c2e947f36795c8f58ba901-Paper.pdf

2. Curriculum Learning — Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston, 2009

https://scholar.google.com/scholar?q=Curriculum+Learning

3. Automatic Goal Generation for Reinforcement Learning Agents (Goal GAN) — Carlos Florensa, David Held, Xinyang Geng, Pieter Abbeel, 2018

https://scholar.google.com/scholar?q=Automatic+Goal+Generation+for+Reinforcement+Learning+Agents+%28Goal+GAN%29

4. Teacher-Student Curriculum Learning — Tambet Matiisen, Avital Oliver, Taco Cohen, John Schulman, 2017 (IEEE TNNLS 2019)

https://scholar.google.com/scholar?q=Teacher-Student+Curriculum+Learning

5. Automatic Curriculum Learning For Deep RL: A Short Survey — Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, Pierre-Yves Oudeyer, 2020

https://scholar.google.com/scholar?q=Automatic+Curriculum+Learning+For+Deep+RL%3A+A+Short+Survey

6. Universal Value Function Approximators — Tom Schaul, Daniel Horgan, Karol Gregor, David Silver, 2015

https://scholar.google.com/scholar?q=Universal+Value+Function+Approximators

7. Hindsight Experience Replay — Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba, 2017 (NeurIPS)

https://scholar.google.com/scholar?q=Hindsight+Experience+Replay

8. Reinforcement Learning with Deep Energy-Based Policies / Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research — Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, Wojciech Zaremba, 2018

https://scholar.google.com/scholar?q=Reinforcement+Learning+with+Deep+Energy-Based+Policies+%2F+Multi-Goal+Reinforcement+Learning%3A+Challenging+Robotics+Environments+and+Request+for+Research

9. Goal-Conditioned Reinforcement Learning: Problems and Solutions — Minghuan Liu, Menghui Zhu, Weinan Zhang, 2022 (IJCAI survey)

https://scholar.google.com/scholar?q=Goal-Conditioned+Reinforcement+Learning%3A+Problems+and+Solutions

10. Teacher algorithms for curriculum learning of deep RL in continuously parameterized environments — Rémy Portelas, Cédric Colas, Katja Hofmann, Pierre-Yves Oudeyer, 2019

https://scholar.google.com/scholar?q=Teacher+algorithms+for+curriculum+learning+of+deep+RL+in+continuously+parameterized+environments

11. Automatic Goal Generation for Reinforcement Learning Agents — Carlos Florensa, David Held, Xinyang Geng, Pieter Abbeel, 2018

https://scholar.google.com/scholar?q=Automatic+Goal+Generation+for+Reinforcement+Learning+Agents

12. Self-Supervised Exploration via Disagreement — Deepak Pathak, Dhiraj Gandhi, Abhinav Gupta, 2019

https://scholar.google.com/scholar?q=Self-Supervised+Exploration+via+Disagreement

13. Deep Exploration via Bootstrapped DQN — Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin Van Roy, 2016

https://scholar.google.com/scholar?q=Deep+Exploration+via+Bootstrapped+DQN

14. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles — Balaji Lakshminarayanan, Alexander Pritzel, Charles Blundell, 2017

https://scholar.google.com/scholar?q=Simple+and+Scalable+Predictive+Uncertainty+Estimation+using+Deep+Ensembles

15. Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play — Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, Rob Fergus, 2018

https://scholar.google.com/scholar?q=Intrinsic+Motivation+and+Automatic+Curricula+via+Asymmetric+Self-Play

16. Prioritized Level Replay — Minqi Jiang, Edward Grefenstette, Tim Rocktäschel, 2021

https://scholar.google.com/scholar?q=Prioritized+Level+Replay

17. Emergent Tool Use From Multi-Agent Autocurricula — Bowen Baker et al., 2019

https://scholar.google.com/scholar?q=Emergent+Tool+Use+From+Multi-Agent+Autocurricula

18. Paired Open-Ended Trailblazer (POET) and Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design (PAIRED) — Rui Wang, Joel Lehman, Jeff Clune, Kenneth Stanley; Michael Dennis et al., 2019/2020

https://scholar.google.com/scholar?q=Paired+Open-Ended+Trailblazer+%28POET%29+and+Emergent+Complexity+and+Zero-shot+Transfer+via+Unsupervised+Environment+Design+%28PAIRED%29

19. DAPO: An Open-Source LLM Reinforcement Learning System at Scale — Qiying Yu et al., 2025

https://scholar.google.com/scholar?q=DAPO%3A+An+Open-Source+LLM+Reinforcement+Learning+System+at+Scale

Interactive Visualization: Automatic Curriculum Learning Through Value Disagreement in Reinforcement Learning

This episode examines "Emergent Tool Use From Multi-Agent Autocurricula," in which hiders and seekers in a physics simulator develop strategies like building box forts, using ramps, and locking objects. All of this comes from a reward that only says whether a hider is visible. It sets the work against hand-written rewards and curiosity-driven baselines such as count-based exploration and Random Network Distillation. It also explains the setup: team-based self-play with PPO, movable boxes and ramps, and a head start for the hiders. The discussion asks whether the claimed six phases are measured or only watched. It looks at the statistics the authors track, such as box movement and locking, and notes that the paper mentions Elo and TrueSkill but doesn't run them. It also covers transfer tasks as a way to test what the agents learned. Listeners get a skeptical look at how self-play progress is measured, plus a clear account of why co-adapting agents make that hard.

Sources:

1. Emergent Tool Use From Multi-Agent Autocurricula in Hide-and-Seek

https://arxiv.org/pdf/1909.07528

2. Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research — Joel Z. Leibo, Edward Hughes, Marc Lanctot, Thore Graepel, 2019

https://scholar.google.com/scholar?q=Autocurricula+and+the+Emergence+of+Innovation+from+Social+Interaction%3A+A+Manifesto+for+Multi-Agent+Intelligence+Research

3. Emergent Complexity via Multi-Agent Competition — Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, Igor Mordatch, 2018

https://scholar.google.com/scholar?q=Emergent+Complexity+via+Multi-Agent+Competition

4. Grandmaster level in StarCraft II using multi-agent reinforcement learning — Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, et al. (DeepMind), 2019

https://scholar.google.com/scholar?q=Grandmaster+level+in+StarCraft+II+using+multi-agent+reinforcement+learning

5. Dota 2 with Large Scale Deep Reinforcement Learning — OpenAI: Christopher Berner, Greg Brockman, Brooke Chan, et al., 2019

https://scholar.google.com/scholar?q=Dota+2+with+Large+Scale+Deep+Reinforcement+Learning

6. Emergent Coordination Through Competition — Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, Thore Graepel, 2019

https://scholar.google.com/scholar?q=Emergent+Coordination+Through+Competition

7. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (MADDPG) — Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch, 2017

https://scholar.google.com/scholar?q=Multi-Agent+Actor-Critic+for+Mixed+Cooperative-Competitive+Environments+%28MADDPG%29

8. Emergence of Grounded Compositional Language in Multi-Agent Populations — Igor Mordatch, Pieter Abbeel, 2018

https://scholar.google.com/scholar?q=Emergence+of+Grounded+Compositional+Language+in+Multi-Agent+Populations

9. Evolving 3D Morphology and Behavior by Competition — Karl Sims, 1994

https://scholar.google.com/scholar?q=Evolving+3D+Morphology+and+Behavior+by+Competition

10. Exploration by Random Network Distillation — Yuri Burda, Harrison Edwards, Amos Storkey, Oleg Klimov, 2018 (ICLR 2019)

https://scholar.google.com/scholar?q=Exploration+by+Random+Network+Distillation

11. Curiosity-driven Exploration by Self-supervised Prediction — Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell, 2017

https://scholar.google.com/scholar?q=Curiosity-driven+Exploration+by+Self-supervised+Prediction

12. Large-Scale Study of Curiosity-Driven Learning — Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, Alexei A. Efros, 2018

https://scholar.google.com/scholar?q=Large-Scale+Study+of+Curiosity-Driven+Learning

13. Unifying Count-Based Exploration and Intrinsic Motivation — Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Rémi Munos, 2016

https://scholar.google.com/scholar?q=Unifying+Count-Based+Exploration+and+Intrinsic+Motivation

14. How transferable are features in deep neural networks? — Jason Yosinski, Jeff Clune, Yoshua Bengio, Hod Lipson, 2014

https://scholar.google.com/scholar?q=How+transferable+are+features+in+deep+neural+networks%3F

15. Quantifying Generalization in Reinforcement Learning — Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, John Schulman, 2019

https://scholar.google.com/scholar?q=Quantifying+Generalization+in+Reinforcement+Learning

16. On the Measure of Intelligence — François Chollet, 2019

https://scholar.google.com/scholar?q=On+the+Measure+of+Intelligence

17. A Survey of Zero-shot Generalisation in Deep Reinforcement Learning — Robert Kirk, Amy Zhang, Edward Grefenstette, Tim Rocktäschel, 2021 (JAIR 2023)

https://scholar.google.com/scholar?q=A+Survey+of+Zero-shot+Generalisation+in+Deep+Reinforcement+Learning

18. Open-Ended Learning Leads to Generally Capable Agents (XLand) — Open-Ended Learning Team (DeepMind: Stooke, Mahajan, Barros, et al.), 2021

https://scholar.google.com/scholar?q=Open-Ended+Learning+Leads+to+Generally+Capable+Agents+%28XLand%29

19. Grandmaster level in StarCraft II using multi-agent reinforcement learning (AlphaStar league) — Vinyals et al., 2019

https://scholar.google.com/scholar?q=Grandmaster+level+in+StarCraft+II+using+multi-agent+reinforcement+learning+%28AlphaStar+league%29

20. Real World Games Look Like Spinning Tops — Czarnecki, Gidel, Tracey, Tuyls, Omidshafiei, Balduzzi, Jaderberg, 2020

https://scholar.google.com/scholar?q=Real+World+Games+Look+Like+Spinning+Tops

21. Open-ended Learning in Symmetric Zero-sum Games — Balduzzi, Garnelo, Bachrach, Czarnecki, Perolat, Jaderberg, Graepel, 2019

https://scholar.google.com/scholar?q=Open-ended+Learning+in+Symmetric+Zero-sum+Games

22. Paired Open-Ended Trailblazer (POET) and Enhanced POET — Wang, Lehman, Clune, Stanley (2019); Wang, Lehman, Rawal, Zhi, Li, Clune, Stanley (2020), 2019/2020

https://scholar.google.com/scholar?q=Paired+Open-Ended+Trailblazer+%28POET%29+and+Enhanced+POET

23. Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design (PAIRED) — Dennis, Jaques, Vinitsky, Bayen, Russell, Critch, Levine, 2020

https://scholar.google.com/scholar?q=Emergent+Complexity+and+Zero-shot+Transfer+via+Unsupervised+Environment+Design+%28PAIRED%29

24. Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play — Sukhbaatar, Lin, Kostrikov, Synnaeve, Szlam, Fergus, 2018

https://scholar.google.com/scholar?q=Intrinsic+Motivation+and+Automatic+Curricula+via+Asymmetric+Self-Play

25. Go-Explore: a New Approach for Hard-Exploration Problems / First return, then explore — Ecoffet, Huizinga, Lehman, Stanley, Clune, 2019/2021

https://scholar.google.com/scholar?q=Go-Explore%3A+a+New+Approach+for+Hard-Exploration+Problems+%2F+First+return%2C+then+explore

26. Never Give Up: Learning Directed Exploration Strategies (NGU) and Agent57 — Badia et al., 2020

https://scholar.google.com/scholar?q=Never+Give+Up%3A+Learning+Directed+Exploration+Strategies+%28NGU%29+and+Agent57

27. Diversity is All You Need: Learning Skills without a Reward Function (DIAYN) — Eysenbach, Gupta, Ibarz, Levine, 2019

https://scholar.google.com/scholar?q=Diversity+is+All+You+Need%3A+Learning+Skills+without+a+Reward+Function+%28DIAYN%29

28. The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (MAPPO) — Yu, Velu, Vinitsky, Gao, Wang, Bayen, Wu, 2021/2022

https://scholar.google.com/scholar?q=The+Surprising+Effectiveness+of+PPO+in+Cooperative+Multi-Agent+Games+%28MAPPO%29

29. Specification gaming: the flip side of AI ingenuity / The Surprising Creativity of Digital Evolution — Krakovna et al. (2020); Lehman et al. (2018), 2018/2020

https://scholar.google.com/scholar?q=Specification+gaming%3A+the+flip+side+of+AI+ingenuity+%2F+The+Surprising+Creativity+of+Digital+Evolution

30. Embodied Intelligence via Learning and Evolution — Gupta, Savarese, Ganguli, Fei-Fei, 2021

https://scholar.google.com/scholar?q=Embodied+Intelligence+via+Learning+and+Evolution

31. SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn RL; Absolute Zero: Reinforced Self-play Reasoning with Zero Data — Liu et al. (SPIRAL); Zhao et al. (Absolute Zero), 2025

https://scholar.google.com/scholar?q=SPIRAL%3A+Self-Play+on+Zero-Sum+Games+Incentivizes+Reasoning+via+Multi-Agent+Multi-Turn+RL%3B+Absolute+Zero%3A+Reinforced+Self-play+Reasoning+with+Zero+Data

Interactive Visualization: Emergent Tool Use From Multi-Agent Autocurricula in Hide-and-Seek

This episode explores MARCH, a recurrent memory architecture that extends Gated DeltaNet with periodic "state anchors" — frozen snapshots of the recurrent state paired with content-derived routing keys, allowing tokens to retrieve earlier versions of memory that would otherwise be overwritten. The discussion frames the core limitation of linear-attention and state-space models as a fixed-state bottleneck: unlike Transformers, which retain every key-value pair, recurrent models can only read the current, most-revised version of their compressed state, permanently losing access to earlier associations once they're gated or delta-updated away. MARCH addresses this through temporal expansion rather than capacity expansion, checkpointing state every C tokens and using a learned anchor token to generate a compact routing key describing each snapshot's content. The reported results are striking: a 94.8% improvement over vanilla Gated DeltaNet on Complex Needle-in-a-Haystack retrieval, plus gains of 47.7% on single-needle retrieval and 49.6% on RULER. Listeners interested in long-context efficiency, recurrent versus attention-based architectures, or the tradeoffs between memory compression and retrievability will find a clear, mechanistic explanation of why recurrent models forget and how content-routed checkpointing offers a fix without altering the underlying recurrence.

Sources:

1. MARCH: Scaling Recurrent Memory with Content-Routed State Anchors — Ming Zhang, Kaisen Yang, Shu Yu, Ermo Hua, Ning Ding, Xia Hu, Bowen Zhou, Chaochao Lu, Youbang Sun, 2026

http://arxiv.org/abs/2608.12435v1

2. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2024

https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time

3. Memory Caching: RNNs with Growing Memory — Ali Behrouz, Zeman Li, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, Vahab Mirrokni, 2026

https://scholar.google.com/scholar?q=Memory+Caching%3A+RNNs+with+Growing+Memory

4. Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity — Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy, Matthijs Douze, Maria Lomeli, Ilze Amanda Auzina, Justin Carpentier, Gabriel Synnaeve, Hervé Jégou, 2026

https://scholar.google.com/scholar?q=Sparse+Delta+Memory%3A+Scaling+the+State+of+Linear+RNNs+through+Sparsity

5. MoM: Linear Sequence Modeling with Mixture-of-Memories — Jusen Du, Weigao Sun, Disen Lan, Jiaxi Hu, Yu Cheng, 2025

https://scholar.google.com/scholar?q=MoM%3A+Linear+Sequence+Modeling+with+Mixture-of-Memories

6. Scaling Linear Attention with Sparse State Expansion — Yuqi Pan, Yongqi An, Zheng Li, Yuhong Chou, Ruijie Zhu, Xiaohui Wang, Mingxuan Wang, Jinqiao Wang, Guoqi Li, 2025

https://scholar.google.com/scholar?q=Scaling+Linear+Attention+with+Sparse+State+Expansion

7. Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff — Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, Christopher Ré, 2024

https://scholar.google.com/scholar?q=Simple+Linear+Attention+Language+Models+Balance+the+Recall-Throughput+Tradeoff

8. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024

https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference

This episode examines Raven, a sequence-modeling architecture from EPFL, Carnegie Mellon, MBZUAI, and Cartesia AI that claims near-perfect recall at sixteen times its training context length. The discussion traces the underlying tradeoff across three existing approaches — Transformers' unbounded but expensive KV cache, state-space models like Mamba that compress history into a fixed-size state at the cost of interference between tokens, and sliding-window attention's exact but position-based eviction — and frames linear attention as mathematically equivalent to the SSM case. Raven's contribution is "Routing Slot Memories": a fixed set of memory slots where a learned, content-based router (using sigmoid scoring, top-K selection, and Gumbel noise during training) decides which slots get written and decayed, recasting SSMs and sliding-window attention as special cases of one general mechanism. Listeners interested in long-context modeling will find the reframing valuable for understanding why linear models forget distant facts and how architecture choices about "where to write memory" — not just how much to compress — determine recall performance.

Sources:

1. Raven: High-Recall Sequence Modeling with Sparse Memory Routing — Arshia Afzal, Aviv Bick, Eric P. Xing, Volkan Cevher, Albert Gu, 2026

http://arxiv.org/abs/2607.25357

2. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023

https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces

3. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret, 2020

https://scholar.google.com/scholar?q=Transformers+are+RNNs%3A+Fast+Autoregressive+Transformers+with+Linear+Attention

4. Longformer: The Long-Document Transformer — Iz Beltagy, Matthew E. Peters, Arman Cohan, 2020

https://scholar.google.com/scholar?q=Longformer%3A+The+Long-Document+Transformer

5. Neural Turing Machines — Alex Graves, Greg Wayne, Ivo Danihelka, 2014

https://scholar.google.com/scholar?q=Neural+Turing+Machines

6. Simple linear attention language models balance the recall-throughput tradeoff (Based) — Simran Arora, Sabri Eyuboglu, Michael Zhang, et al., 2024

https://scholar.google.com/scholar?q=Simple+linear+attention+language+models+balance+the+recall-throughput+tradeoff+%28Based%29

7. The Illusion of State in State-Space Models — William Merrill, Jackson Petty, Ashish Sabharwal, 2024

https://scholar.google.com/scholar?q=The+Illusion+of+State+in+State-Space+Models

8. Stuffed Mamba: Oversized States Lead to the Inability to Forget — Yingfa Chen, Xinrong Zhang, Shengding Hu, et al., 2025

https://scholar.google.com/scholar?q=Stuffed+Mamba%3A+Oversized+States+Lead+to+the+Inability+to+Forget

9. Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism — Aviv Bick, Eric Xing, Albert Gu, 2025

https://scholar.google.com/scholar?q=Understanding+the+Skill+Gap+in+Recurrent+Language+Models%3A+The+Role+of+the+Gather-and-Aggregate+Mechanism

10. MoM: Linear Sequence Modeling with Mixture-of-Memories — Jusen Du, Weigao Sun, Disen Lan, Jiaxi Hu, Yu Cheng, 2025

https://scholar.google.com/scholar?q=MoM%3A+Linear+Sequence+Modeling+with+Mixture-of-Memories

11. Scaling Linear Attention with Sparse State Expansion (SSE) — Yuqi Pan, Yongqi An, Zheng Li, et al., 2025

https://scholar.google.com/scholar?q=Scaling+Linear+Attention+with+Sparse+State+Expansion+%28SSE%29

This episode examines a paper on translating the KV cache, the per-layer key and value tensors a transformer builds while reading a document, from one model family into another. The aim is for a second model to answer as if it had read the text itself, so a multi-model system reads a document once instead of once per model. The discussion explains why caches are private to each model and why prefill cost matters for time to first token in cache-augmented generation. It also surveys related work, including latent-space alignment, Cache-to-Cache projection, and Cartridges as a non-translation point of comparison. The central idea is a tension between propagation error when translating early and correction-deficit error when translating late. This is illustrated by a U-shaped loss curve on a small same-family GPT-2 setup, and the hosts caution that it motivates the design more than it proves the result generalizes. The episode then walks through the five-step inference pipeline and begins the Mixture-of-Translators idea, which replaces a single translator with several. Listeners get a clear, skeptical read on how to interpret the headline recovery numbers against a native-prefill baseline.

Sources:

1. KV Cache Translation across Heterogeneous Large Language Models — Jin-woo Lee, Minkyung Song, Junghyun Oh, Seunghoon Han, Gwangseon Jang, Soyoung Park, Sungsu Lim, 2026

http://arxiv.org/abs/2607.28979v2

2. Cache-to-Cache: Direct Semantic Communication Between Large Language Models — Tianyu Fu et al., 2025

https://scholar.google.com/scholar?q=Cache-to-Cache%3A+Direct+Semantic+Communication+Between+Large+Language+Models

3. DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving — Yuhan Liu et al., 2024

https://scholar.google.com/scholar?q=DroidSpeak%3A+KV+Cache+Sharing+for+Cross-LLM+Communication+and+Multi-LLM+Serving

4. Communicating Activations Between Language Model Agents — Vignav Ramesh, Kenneth Li, 2025

https://scholar.google.com/scholar?q=Communicating+Activations+Between+Language+Model+Agents

5. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, et al., 2024 (EuroSys 2025)

https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion

6. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon et al., 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

7. Distilling the Knowledge in a Neural Network — Geoffrey Hinton, Oriol Vinyals, Jeff Dean, 2015

https://scholar.google.com/scholar?q=Distilling+the+Knowledge+in+a+Neural+Network

8. FitNets: Hints for Thin Deep Nets — Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, et al., 2015

https://scholar.google.com/scholar?q=FitNets%3A+Hints+for+Thin+Deep+Nets

9. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes — Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, et al., 2024

https://scholar.google.com/scholar?q=On-Policy+Distillation+of+Language+Models%3A+Learning+from+Self-Generated+Mistakes

10. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, et al., 2025

https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study

11. Cache-to-Cache (C2C): Direct semantic communication between LLMs via KV-cache fusion — Fu et al., 2026

https://scholar.google.com/scholar?q=Cache-to-Cache+%28C2C%29%3A+Direct+semantic+communication+between+LLMs+via+KV-cache+fusion

12. Latent space communication via K-V cache alignment — Dery et al., 2026

https://scholar.google.com/scholar?q=Latent+space+communication+via+K-V+cache+alignment

13. KVComm: Selective KV sharing between LLMs — Shi et al., 2026

https://scholar.google.com/scholar?q=KVComm%3A+Selective+KV+sharing+between+LLMs

14. Enabling agents to communicate entirely in latent space (Interlat) — Du et al., 2026

https://scholar.google.com/scholar?q=Enabling+agents+to+communicate+entirely+in+latent+space+%28Interlat%29

15. Model stitching: Looking for functional similarity between representations — Bansal, Nakkiran, Barak, 2021

https://scholar.google.com/scholar?q=Model+stitching%3A+Looking+for+functional+similarity+between+representations

16. A reduction of imitation learning and structured prediction to no-regret online learning (DAgger) — Ross, Gordon, Bagnell, 2011

https://scholar.google.com/scholar?q=A+reduction+of+imitation+learning+and+structured+prediction+to+no-regret+online+learning+%28DAgger%29

17. Efficient memory management for LLM serving with PagedAttention — Kwon et al., 2023

https://scholar.google.com/scholar?q=Efficient+memory+management+for+LLM+serving+with+PagedAttention

Interactive Visualization: KV Cache Translation Across Heterogeneous Large Language Models

This episode explores RaReCache, a 2026 preprint on reusing a small model's KV cache inside a much larger model from the same family. It starts with the handoff cost in cascades and routers: when a cheap model escalates to a big one, the big model has to redo the full prefill. The discussion covers prior work on selective recomputation (token-level in CacheBlend, layer-level in DroidSpeak) and a closed-form ridge-regression map that translates KV entries between model sizes. The motivating result uses Qwen3-0.6B as the source and Qwen3-14B as the target on GSM8K. Plain cache mapping reaches 71.6% accuracy against 95.1% for a native prefill. The paper's argument is that the errors cluster in a few positions, so the large model only needs to recompute a fraction of tokens. The hosts also note how this repair differs from lossless speculative decoding. They flag that the calibration data comes from the same task's training split, which limits what the results show. The episode ends on why attention-based selection of positions fails: the 14B model puts most of its attention on a sink token and template text, and very little on the actual question.

Sources:

1. RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation — Sreetama Sarkar, Saptarshi Mitra, Sitao Huang, Souvik Kundu, Peter A. Beerel, 2026

http://arxiv.org/abs/2610.11358v1

2. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang, 2024 (arXiv); EuroSys 2025

https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion

3. DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving — Yuhan Liu, Yuyang Huang, Jiayi Yao, Shan Lu, Junchen Jiang and collaborators, 2024 (arXiv); later versions through 2026

https://scholar.google.com/scholar?q=DroidSpeak%3A+KV+Cache+Sharing+for+Cross-LLM+Communication+and+Multi-LLM+Serving

4. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, Ion Stoica, 2023 (SOSP)

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%29

5. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance — Lingjiao Chen, Matei Zaharia, James Zou, 2023

https://scholar.google.com/scholar?q=FrugalGPT%3A+How+to+Use+Large+Language+Models+While+Reducing+Cost+and+Improving+Performance

6. H2O: Heavy-Hitter Oracle: Efficient Generative Inference of Large Language Models with Heavy Hitters — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen, 2023 (NeurIPS)

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Heavy+Hitters

7. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, 2022 (NeurIPS)

https://scholar.google.com/scholar?q=LLM.int8%28%29%3A+8-bit+Matrix+Multiplication+for+Transformers+at+Scale

8. Palu: Compressing KV-Cache with Low-Rank Projection — Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, Kai-Chiang Wu, 2024 (arXiv)

https://scholar.google.com/scholar?q=Palu%3A+Compressing+KV-Cache+with+Low-Rank+Projection

9. Cross-model KV cache transfer in LLM families: A closed-form linear mapping for prefill reuse — Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, et al., 2026

https://scholar.google.com/scholar?q=Cross-model+KV+cache+transfer+in+LLM+families%3A+A+closed-form+linear+mapping+for+prefill+reuse

10. Cache-to-Cache: Direct Semantic Communication Between Large Language Models — Tianyu Fu, Zihan Min, Hanling Zhang, et al., 2026

https://scholar.google.com/scholar?q=Cache-to-Cache%3A+Direct+Semantic+Communication+Between+Large+Language+Models

11. Latent Space Communication via K-V Cache Alignment — Lucio M. Dery, Zohar Yahav, Henry Prior, et al., 2026

https://scholar.google.com/scholar?q=Latent+Space+Communication+via+K-V+Cache+Alignment

12. Reduced-rank regression for the multivariate linear model — Alan Julian Izenman, 1975

https://scholar.google.com/scholar?q=Reduced-rank+regression+for+the+multivariate+linear+model

13. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2024

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29

14. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of LLMs; SnapKV: LLM Knows What You Are Looking for Before Generation — Zhenyu Zhang et al.; Yuhong Li et al., 2023 / 2024

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+LLMs%3B+SnapKV%3A+LLM+Knows+What+You+Are+Looking+for+Before+Generation

15. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023

https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding

16. Palu: Compressing KV-Cache with Low-Rank Projection; Eigen Attention — Chi-Chih Chang et al.; Utkarsh Saxena et al., 2024

https://scholar.google.com/scholar?q=Palu%3A+Compressing+KV-Cache+with+Low-Rank+Projection%3B+Eigen+Attention

17. FrugalGPT / RouteLLM — Lingjiao Chen, Matei Zaharia, James Zou; Isaac Ong et al., 2023 / 2024

https://scholar.google.com/scholar?q=FrugalGPT+%2F+RouteLLM

Interactive Visualization: RaReCache: Rank-Disagreement Recomputation for Cross-Model KV Cache Reuse

This episode examines Self-Evolving Curriculum, a 2025 paper proposing that a small multi-armed bandit can choose which problem categories an LLM trains on during reinforcement learning from verifiable rewards. It explains why GRPO gets no learning signal from problems the model always solves or never solves, since every advantage in the group is zero. It contrasts the bandit approach with fixed easy-to-hard schedules, random shuffling, and DAPO-style dynamic sampling, which pays for rollouts it then discards. The discussion covers the non-stationary bandit setup: each difficulty level or problem type is an arm, and the signal is reused from advantages GRPO already computes. It also looks at the Countdown, Zebra, ARC-1D and MATH experiments, including the point that the "out-of-distribution" claims are cleaner for the puzzles than for the math benchmarks. Listeners get a clear walkthrough of the mechanism and a skeptical look at whether the evidence supports the paper's claims.

Sources:

1. Self-Evolving Curriculum for LLM Reasoning — Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, Ehsan Kamalloo, 2025

http://arxiv.org/abs/2505.14970

2. Curriculum Learning — Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston, 2009

https://scholar.google.com/scholar?q=Curriculum+Learning

3. Automated Curriculum Learning for Neural Networks — Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, Koray Kavukcuoglu, 2017

https://scholar.google.com/scholar?q=Automated+Curriculum+Learning+for+Neural+Networks

4. Teacher-Student Curriculum Learning — Tambet Matiisen, Avital Oliver, Taco Cohen, John Schulman, 2017 (arXiv); published in IEEE Transactions on Neural Networks and Learning Systems, 2019

https://scholar.google.com/scholar?q=Teacher-Student+Curriculum+Learning

5. Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey — Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, Peter Stone, 2020

https://scholar.google.com/scholar?q=Curriculum+Learning+for+Reinforcement+Learning+Domains%3A+A+Framework+and+Survey

6. Efficient Online Data Mixing For Language Model Pre-Training — Alon Albalak, Liangming Pan, Colin Raffel, William Yang Wang, 2023

https://scholar.google.com/scholar?q=Efficient+Online+Data+Mixing+For+Language+Model+Pre-Training

7. DAPO: An Open-Source LLM Reinforcement Learning System at Scale — Qiying Yu et al., 2025

https://scholar.google.com/scholar?q=DAPO%3A+An+Open-Source+LLM+Reinforcement+Learning+System+at+Scale

8. DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training — Zhenting Wang, Guofeng Cui, Kun Wan, Wentian Zhao, 2025

https://scholar.google.com/scholar?q=DUMP%3A+Automated+Distribution-Level+Curriculum+Learning+for+RL-based+LLM+Post-training

9. Efficient Reinforcement Finetuning via Adaptive Curriculum Learning (AdaRFT) — Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, Jieyu Zhao, 2025

https://scholar.google.com/scholar?q=Efficient+Reinforcement+Finetuning+via+Adaptive+Curriculum+Learning+%28AdaRFT%29

10. Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning — Sanghwan Bae et al., 2025

https://scholar.google.com/scholar?q=Online+Difficulty+Filtering+for+Reasoning+Oriented+Reinforcement+Learning

11. Prioritized Experience Replay — Tom Schaul, John Quan, Ioannis Antonoglou, David Silver, 2015

https://scholar.google.com/scholar?q=Prioritized+Experience+Replay

12. Not All Samples Are Created Equal: Deep Learning with Importance Sampling — Angelos Katharopoulos, François Fleuret, 2018

https://scholar.google.com/scholar?q=Not+All+Samples+Are+Created+Equal%3A+Deep+Learning+with+Importance+Sampling

13. Evolving Curricula with Regret-Based Environment Design (ACCEL) — Jack Parker-Holder, Minqi Jiang, Michael Dennis, et al., 2022

https://scholar.google.com/scholar?q=Evolving+Curricula+with+Regret-Based+Environment+Design+%28ACCEL%29

14. Prioritized Level Replay — Minqi Jiang, Edward Grefenstette, Tim Rocktäschel, 2021

https://scholar.google.com/scholar?q=Prioritized+Level+Replay

15. Dr. GRPO / Understanding R1-Zero-Like Training: A Critical Perspective — Zichen Liu et al., 2025

https://scholar.google.com/scholar?q=Dr.+GRPO+%2F+Understanding+R1-Zero-Like+Training%3A+A+Critical+Perspective

16. Variance-based / Prompt-difficulty selection for RLVR (e.g., Online Difficulty Filtering, GRESO, Prompt Curriculum Learning) — Various, 2025, 2025

https://scholar.google.com/scholar?q=Variance-based+%2F+Prompt-difficulty+selection+for+RLVR+%28e.g.%2C+Online+Difficulty+Filtering%2C+GRESO%2C+Prompt+Curriculum+Learning%29

Interactive Visualization: Self-Evolving Curriculum: Bandit-Selected Problems for LLM Reasoning RL

This episode examines GraniKV, a KV-cache paging scheme for multi-agent workloads where many requests share a very long prompt prefix. It argues that engines like SGLang treat the shared prefix and each request's private suffix the same way, using 16-token pages, even though the two want opposite things from memory. The discussion first covers the background: PagedAttention, RadixAttention prefix reuse, and the point that reuse saves prefill compute but not the repeated memory reads during decode. It then separates two prior kernel ideas, cascade attention, which reads the prefix once to cut memory traffic, and Hydragen, which batches queries into dense matrix multiplies but needs a contiguous prefix. GraniKV's claim is about storage and integration rather than new kernels. It keeps each prefix as one contiguous slab in a reserved "hot" pool, keeps suffixes at single-token granularity in a "cold" pool, and chooses an attention backend at each decode step. The episode tests where the reported 2.16x throughput gain over stock SGLang at a 16K-token prefix comes from, asking whether the layout or the kernel does the work, with a self-built cascade baseline as the comparison. It is useful for listeners who want to understand why prefix-heavy agent serving is memory-bound and how storage layout interacts with attention kernels.

Sources:

1. GraniKV: Asymmetric Granularity KV-Cache Paging for Multi-Agent Systems with Long Shared Prefix — Jinhyun Jeon, Sungjoo Yoo, 2026

http://arxiv.org/abs/2608.15584v1

2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

3. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2024

https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs

4. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention — Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar, 2024

https://scholar.google.com/scholar?q=vAttention%3A+Dynamic+Memory+Management+for+Serving+LLMs+without+PagedAttention

5. Preble: Efficient Distributed Prompt Scheduling for LLM Serving — Vikranth Srivastava, Shu Liu, Yifan Qiao, Yang Zhou, Xingyu Xiang and colleagues (UC San Diego / UC Berkeley), 2024

https://scholar.google.com/scholar?q=Preble%3A+Efficient+Distributed+Prompt+Scheduling+for+LLM+Serving

6. Hydragen: High-Throughput LLM Inference with Shared Prefixes — Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christopher Ré, Azalia Mirhoseini, 2024

https://scholar.google.com/scholar?q=Hydragen%3A+High-Throughput+LLM+Inference+with+Shared+Prefixes

7. Cascade Inference: Memory-Bandwidth Efficient Shared Prefix Batch Decoding / FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving — Zihao Ye, Ruihang Lai, Bo-Ru Lu, Chien-Yu Lin, Size Zheng, Lequn Chen, Tianqi Chen, Luis Ceze (and co-authors), 2024-2025

https://scholar.google.com/scholar?q=Cascade+Inference%3A+Memory-Bandwidth+Efficient+Shared+Prefix+Batch+Decoding+%2F+FlashInfer%3A+Efficient+and+Customizable+Attention+Engine+for+LLM+Inference+Serving

8. ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition — Lu Ye, Ze Tao, Yong Huang, Yang Li, 2024

https://scholar.google.com/scholar?q=ChunkAttention%3A+Efficient+Self-Attention+with+Prefix-Aware+KV+Cache+and+Two-Phase+Partition

9. Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs — Ben Athiwaratkun, Sujan Kumar Gonugondla, Sanjay Krishna Gouda, Haifeng Qian and colleagues (AWS), 2024

https://scholar.google.com/scholar?q=Bifurcated+Attention%3A+Accelerating+Massively+Parallel+Decoding+with+Shared+Prefixes+in+LLMs

10. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024

https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve

11. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024

https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving

12. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving — Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, Luis Ceze, 2025

https://scholar.google.com/scholar?q=FlashInfer%3A+Efficient+and+Customizable+Attention+Engine+for+LLM+Inference+Serving

13. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning / Flash-Decoding for long-context inference — Tri Dao (FlashAttention-2); Tri Dao, Daniel Haziza, Francisco Massa, Grigory Sizov (Flash-Decoding), 2023

https://scholar.google.com/scholar?q=FlashAttention-2%3A+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning+%2F+Flash-Decoding+for+long-context+inference

14. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving (including the cascade / multi-level cascade API) — Ye, Chen, Lai, Shao, Zhao, Zheng, Jin, et al., 2025

https://scholar.google.com/scholar?q=FlashInfer%3A+Efficient+and+Customizable+Attention+Engine+for+LLM+Inference+Serving+%28including+the+cascade+%2F+multi-level+cascade+API%29

15. SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention) — Zheng, Yin, Xie, Sun, Huang, Yu, Cao, Kozyrakis, Stoica, Gonzalez, Barrett, Sheng, 2024

https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs+%28RadixAttention%29

16. Efficient Memory Management for LLM Serving with PagedAttention — Kwon, Li, Zhuang, Sheng, Yu, Gonzalez, Zhang, Stoica, 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+LLM+Serving+with+PagedAttention

17. HotPrefix: Hotness-aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems — Li, Gu, Huan, Wang, Yao, Tian, Chen, 2025

https://scholar.google.com/scholar?q=HotPrefix%3A+Hotness-aware+KV+Cache+Scheduling+for+Efficient+Prefix+Sharing+in+LLM+Inference+Systems

18. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Qin, Li, He, Zhang, Wu, Zheng, Xu, 2025

https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving

19. FlashAttention-2 / FlashDecoding (split-KV) — Dao; Dao, Haziza, Massa, Sizov, 2023-2024

https://scholar.google.com/scholar?q=FlashAttention-2+%2F+FlashDecoding+%28split-KV%29

20. Multi-Level / Prefix-sharing attention in other engines (e.g. DeFT, Prefix-sharing tree attention) — Yao et al. (DeFT) and related tree-attention work, 2024

https://scholar.google.com/scholar?q=Multi-Level+%2F+Prefix-sharing+attention+in+other+engines+%28e.g.+DeFT%2C+Prefix-sharing+tree+attention%29

Interactive Visualization: GraniKV: Asymmetric KV-Cache Paging for Shared-Prefix Agents

This episode examines LLMLingua, a method for compressing prompts five to twenty times by using a small local language model to delete low-information tokens, so that a black-box model such as GPT-3.5 can still answer well. It explains why prompts get so long (chain-of-thought demonstrations, retrieved documents, conversation history). It also explains why quantization, pruning, and learned soft-prompt compression don't work on closed APIs. The discussion ties token surprisal and perplexity to Shannon's redundancy argument and the view of language modeling as compression. It then shows why the Selective-Context baseline fails: it scores every token independently, so a phrase repeated across demonstrations can be deleted entirely. The episode walks through the paper's components, including aligning the small model to the target and a budget controller that barely trims the instruction and question but cuts the demonstrations heavily. Along the way it flags caveats: predictable tokens aren't always unimportant, scoring is left-to-right only, and the alignment step adds under a point on GSM8K.

Sources:

1. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023

http://arxiv.org/abs/2310.05736

2. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023 (EMNLP 2023)

https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models

3. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression — Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023 (ACL 2024)

https://scholar.google.com/scholar?q=LongLLMLingua%3A+Accelerating+and+Enhancing+LLMs+in+Long+Context+Scenarios+via+Prompt+Compression

4. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression — Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, et al., 2024 (ACL Findings 2024)

https://scholar.google.com/scholar?q=LLMLingua-2%3A+Data+Distillation+for+Efficient+and+Faithful+Task-Agnostic+Prompt+Compression

5. Compressing Context to Enhance Inference Efficiency of Large Language Models (Selective Context) — Yucheng Li, Bo Dong, Frank Guerin, Chenghua Lin, 2023 (EMNLP 2023)

https://scholar.google.com/scholar?q=Compressing+Context+to+Enhance+Inference+Efficiency+of+Large+Language+Models+%28Selective+Context%29

6. Prediction and Entropy of Printed English — Claude E. Shannon, 1951

https://scholar.google.com/scholar?q=Prediction+and+Entropy+of+Printed+English

7. Language Models are Unsupervised Multitask Learners (GPT-2) and perplexity-based data filtering in CCNet — Radford et al. (GPT-2, 2019); Wenzek et al. (CCNet, 2020), 2019/2020

https://scholar.google.com/scholar?q=Language+Models+are+Unsupervised+Multitask+Learners+%28GPT-2%29+and+perplexity-based+data+filtering+in+CCNet

8. Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering (Selective Context) — Yucheng Li, 2023

https://scholar.google.com/scholar?q=Unlocking+Context+Constraints+of+LLMs%3A+Enhancing+Context+Efficiency+of+LLMs+with+Self-Information-Based+Content+Filtering+%28Selective+Context%29

9. Language Modeling Is Compression — Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, et al., 2023

https://scholar.google.com/scholar?q=Language+Modeling+Is+Compression

10. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023

https://scholar.google.com/scholar?q=Learning+to+Compress+Prompts+with+Gist+Tokens

11. Adapting Language Models to Compress Contexts (AutoCompressors) — Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen, 2023

https://scholar.google.com/scholar?q=Adapting+Language+Models+to+Compress+Contexts+%28AutoCompressors%29

12. In-context Autoencoder for Context Compression in a Large Language Model (ICAE) — Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, Furu Wei, 2023

https://scholar.google.com/scholar?q=In-context+Autoencoder+for+Context+Compression+in+a+Large+Language+Model+%28ICAE%29

13. Semantic Compression with Large Language Models — Henry Gilbert, Michael Sandborn, Douglas C. Schmidt, Jesse Spencer-Smith, Jules White, 2023

https://scholar.google.com/scholar?q=Semantic+Compression+with+Large+Language+Models

14. Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance — Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, Tushar Khot, 2023

https://scholar.google.com/scholar?q=Chain-of-Thought+Hub%3A+A+Continuous+Effort+to+Measure+Large+Language+Models%27+Reasoning+Performance

15. Training Verifiers to Solve Math Word Problems (GSM8K) — Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al., 2021

https://scholar.google.com/scholar?q=Training+Verifiers+to+Solve+Math+Word+Problems+%28GSM8K%29

16. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, et al., 2023

https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts

17. Efficient Streaming Language Models with Attention Sinks / H2O / SnapKV (KV-cache eviction line) — Xiao et al.; Zhang et al.; Li et al., 2023-2024

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%2F+H2O+%2F+SnapKV+%28KV-cache+eviction+line%29

Interactive Visualization: LLMLingua: Compressing Prompts with Small-Model Token Pruning