AI Post Transformers

This episode examines "In-Context Autoencoder for Context Compression in a Large Language Model" (ICLR 2024), which uses a LoRA-adapted copy of an LLM to squeeze a context into 128 learned memory slots that an unmodified, frozen decoder reads in place of the original tokens, for roughly 4x compression. It covers the two-stage training: self-supervised pretraining on the Pile with autoencoding and language-modeling objectives, then instruction tuning on a GPT-4-generated prompt-and-answer dataset. The discussion contrasts this design with Gist tokens and AutoCompressors, and with architectural fixes for long context such as sparse attention and state-space models. It also draws on the "Lost in the Middle" findings as motivation. The hosts question whether the paper's evidence supports its "long context" framing, since compressing a single passage is a different claim from handling truly long inputs. They also flag that the synthetic training data and the GPT-4 judge deserve scrutiny. Listeners get a clear explanation of how soft-prompt-style memory slots work and a critical lens on what the results do and don't show.

Sources:

1. In-context Autoencoder for Context Compression in a Large Language Model — Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, Furu Wei, 2023

http://arxiv.org/abs/2307.06945

2. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023

https://scholar.google.com/scholar?q=Learning+to+Compress+Prompts+with+Gist+Tokens

3. Adapting Language Models to Compress Contexts — Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen, 2023

https://scholar.google.com/scholar?q=Adapting+Language+Models+to+Compress+Contexts

4. Compressive Transformers for Long-Range Sequence Modelling — Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Timothy P. Lillicrap, 2019

https://scholar.google.com/scholar?q=Compressive+Transformers+for+Long-Range+Sequence+Modelling

5. Perceiver: General Perception with Iterative Attention — Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, Joao Carreira, 2021

https://scholar.google.com/scholar?q=Perceiver%3A+General+Perception+with+Iterative+Attention

6. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023

https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts

7. Adapting Language Models to Compress Contexts (AutoCompressors) — Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen, 2023

https://scholar.google.com/scholar?q=Adapting+Language+Models+to+Compress+Contexts+%28AutoCompressors%29

8. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023

https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models

9. Language Modeling Is Compression — Grégoire Delétang et al., 2023

https://scholar.google.com/scholar?q=Language+Modeling+Is+Compression

10. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29

11. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of LLMs — Zhenyu Zhang et al., 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+LLMs

12. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache — Zirui Liu et al., 2024

https://scholar.google.com/scholar?q=KIVI%3A+A+Tuning-Free+Asymmetric+2bit+Quantization+for+KV+Cache

13. Efficient Memory Management for LLM Serving with PagedAttention (vLLM) — Woosuk Kwon et al., 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+LLM+Serving+with+PagedAttention+%28vLLM%29

14. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Tri Dao, 2023

https://scholar.google.com/scholar?q=FlashAttention-2%3A+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning

15. Learning to Compress Contexts / xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token — Xin Cheng et al., 2024

https://scholar.google.com/scholar?q=Learning+to+Compress+Contexts+%2F+xRAG%3A+Extreme+Context+Compression+for+Retrieval-augmented+Generation+with+One+Token

16. In-Context Former / 500xCompressor: Generalized Prompt Compression — Zongqian Li, Yixuan Su, Nigel Collier, 2024

https://scholar.google.com/scholar?q=In-Context+Former+%2F+500xCompressor%3A+Generalized+Prompt+Compression

17. Perceiver / Flamingo (Perceiver Resampler) and BLIP-2 (Q-Former) — Andrew Jaegle et al.; Jean-Baptiste Alayrac et al.; Junnan Li et al., 2021-2023

https://scholar.google.com/scholar?q=Perceiver+%2F+Flamingo+%28Perceiver+Resampler%29+and+BLIP-2+%28Q-Former%29

18. Prompt Cache / Cartridges-style KV compression (Learning to compress context by training a compact KV cache) — In Gim et al.; Sabri Eyuboglu et al., 2024-2025

https://scholar.google.com/scholar?q=Prompt+Cache+%2F+Cartridges-style+KV+compression+%28Learning+to+compress+context+by+training+a+compact+KV+cache%29

Interactive Visualization: In-Context Autoencoder: Compressing Long Contexts into Memory Slots

This episode examines "Efficient Streaming Language Models with Attention Sinks" (ICLR 2024), which asks how a model trained on a fixed attention window can run over an endless input stream at constant memory and constant per-token cost, with no fine-tuning. It explains why plain window attention collapses once the first tokens are evicted from the KV cache (perplexity in the thousands), and why recomputing every window works but is slow. The key argument is that softmax forces attention to land somewhere, so models learn to park surplus attention on the initial tokens, which act as "sinks". On Llama-2-13B, keeping just the first four tokens plus a rolling window brings perplexity from about 5,158 down to 5.40, close to the 5.43 of full recomputation. Replacing those four tokens with newlines still works, which shows that absolute position matters and content doesn't. The discussion also stresses that the method delivers stable generation, not long-term memory, and it ends on how positions are assigned by cache slot rather than by original text position.

Sources:

1. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023

http://arxiv.org/abs/2309.17453

2. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023 (ICLR 2024)

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks

3. Massive Activations in Large Language Models — Mingjie Sun, Xinlei Chen, J. Zico Kolter, Zhuang Liu, 2024

https://scholar.google.com/scholar?q=Massive+Activations+in+Large+Language+Models

4. Vision Transformers Need Registers — Timothée Darcet, Maxime Oquab, Julien Mairal, Piotr Bojanowski, 2023

https://scholar.google.com/scholar?q=Vision+Transformers+Need+Registers

5. Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing — Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort, 2023

https://scholar.google.com/scholar?q=Quantizable+Transformers%3A+Removing+Outliers+by+Helping+Attention+Heads+Do+Nothing

6. H2O: Heavy-Hitter Oracle: Efficient Generative Inference of Large Language Models with Heavy-Hitter Oracle for KV Cache — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Zhangyang Wang, Beidi Chen, et al., 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Heavy-Hitter+Oracle+for+KV+Cache

7. Longformer: The Long-Document Transformer — Iz Beltagy, Matthew E. Peters, Arman Cohan, 2020

https://scholar.google.com/scholar?q=Longformer%3A+The+Long-Document+Transformer

8. Efficiently Scaling Transformer Inference — Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, et al., 2022

https://scholar.google.com/scholar?q=Efficiently+Scaling+Transformer+Inference

9. Efficient Streaming Language Models with Attention Sinks (follow-up analyses): When Attention Sink Emerges in Language Models: An Empirical View — Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, Min Lin, 2024

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28follow-up+analyses%29%3A+When+Attention+Sink+Emerges+in+Language+Models%3A+An+Empirical+View

10. Attention is Off by One (Softmax1) — Evan Miller, 2023

https://scholar.google.com/scholar?q=Attention+is+Off+by+One+%28Softmax1%29

11. LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models — Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, Sinong Wang, 2023

https://scholar.google.com/scholar?q=LM-Infinite%3A+Simple+On-the-Fly+Length+Generalization+for+Large+Language+Models

12. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen, 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models

13. SnapKV: LLM Knows What You are Looking for Before Generation — Yuhong Li, Yingbing Huang, Bowen Yang, et al., 2024

https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation

14. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023

https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts

15. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

16. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023

https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces

17. Efficient Transformers: A Survey / Longformer — Iz Beltagy, Matthew E. Peters, Arman Cohan, 2020

https://scholar.google.com/scholar?q=Efficient+Transformers%3A+A+Survey+%2F+Longformer

Interactive Visualization: Attention Sinks Enable Streaming Language Models Over Endless Input

This episode examines "Speculative Prefill," an ICML 2025 method that cuts time to first token by having a small model (Llama-3.1-8B) estimate which prompt tokens matter, so a large model (Llama-405B) prefills only those. It explains why prefill dominates in real traffic, where prompts are often about ten times longer than outputs. It also explains why dropping tokens differs from sparse-attention approaches like MInference, since it also removes MLP and tensor-parallel communication costs. The discussion places the work among related approaches (SnapKV, LazyLLM, GemFilter, SwiftKV, LLMLingua) as a training-free method that doesn't rely on the big model to pick tokens. It also spells out the two bets behind it: that token importance transfers between models, and that the big model tolerates a gappy prompt at a given keep rate. The episode cautions that the headline 7.66x TTFT and 7x QPS figures come from different experiments and best-case conditions. It also notes that the importance score is an attention heuristic rather than a formal information-theoretic measure.

Sources:

1. Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation — Jingyu Liu, Beidi Chen, Ce Zhang, 2025

http://arxiv.org/abs/2502.02789

2. SnapKV: LLM Knows What You are Looking for Before Generation — Yuhong Li, Yingbing Huang, Bowen Yang, et al., 2024

https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation

3. LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference — Qichen Fu et al., 2024

https://scholar.google.com/scholar?q=LazyLLM%3A+Dynamic+Token+Pruning+for+Efficient+Long+Context+LLM+Inference

4. Not All Heads Matter / Quest, and RetrievalAttention-style dynamic sparse attention — Jiaming Tang et al. (Quest); Di Liu et al. (RetrievalAttention), 2024

https://scholar.google.com/scholar?q=Not+All+Heads+Matter+%2F+Quest%2C+and+RetrievalAttention-style+dynamic+sparse+attention

5. GemFilter: Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction — Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, Shafiq Joty, 2024

https://scholar.google.com/scholar?q=GemFilter%3A+Discovering+the+Gems+in+Early+Layers%3A+Accelerating+Long-Context+LLMs+with+1000x+Input+Token+Reduction

6. SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation — Aurick Qiao et al. (Snowflake), 2024

https://scholar.google.com/scholar?q=SwiftKV%3A+Fast+Prefill-Optimized+Inference+with+Knowledge-Preserving+Model+Transformation

7. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023

https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding

8. LLMLingua-2 / LongLLMLingua: Question-aware prompt compression — Huiqiang Jiang, Qianhui Wu, et al., 2023-2024

https://scholar.google.com/scholar?q=LLMLingua-2+%2F+LongLLMLingua%3A+Question-aware+prompt+compression

9. Sarathi-Serve / Chunked prefill and Mooncake / prefill-decode disaggregation — Agrawal et al. (Sarathi-Serve); Qin et al. (Mooncake), 2024

https://scholar.google.com/scholar?q=Sarathi-Serve+%2F+Chunked+prefill+and+Mooncake+%2F+prefill-decode+disaggregation

10. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu et al., 2023

https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts

11. Marconi / prefix-caching for hybrid LLMs and LMCache/CacheBlend — Pan et al. (Marconi); Yao et al. (CacheBlend), 2024-2025

https://scholar.google.com/scholar?q=Marconi+%2F+prefix-caching+for+hybrid+LLMs+and+LMCache%2FCacheBlend

Interactive Visualization: Speculative Prefill: Training-Free Token Importance for Faster TTFT

This episode examines "Jenga," a paper from the team behind vLLM's PagedAttention, which asks what that memory manager costs on modern models that break its two core assumptions: that every layer stores identically sized per-token state, and that every layer needs the full prefix. It covers sliding-window layers, Mamba-style recurrent states, and cross-attention vision models. In a Llama 3.2 Vision example, text layers end up reserving slots for image tokens they never read. The discussion also covers why prefix caching and a single global LRU eviction policy no longer work when each layer type needs a different part of the prefix. It then walks through the proposed fix, a slab-style two-level allocator that sizes large pages at the least common multiple of the layer types' entry sizes (a worked toy of 256 and 384 bytes gives 768-byte pages) and lets each layer type split them into its own small pages. Listeners get a clear explanation of how this recovers memory and throughput while working with existing attention kernels.

Sources:

1. Jenga: Effective Memory Management for Serving LLM with Heterogeneity — Chen Zhang, Kuntai Du, Shu Liu, Woosuk Kwon, Xiangxi Mo, Yufeng Wang, Xiaoxuan Liu, Kaichao You, Zhuohan Li, Mingsheng Long, Jidong Zhai, Joseph Gonzalez, Ion Stoica, 2025

http://arxiv.org/abs/2503.18292

2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, Ion Stoica, 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

3. Jenga: Effective Memory Management for Serving LLM with Heterogeneity — Chen Zhang, Kuntai Du, Shu Liu, Woosuk Kwon, Xiangxi Mo, Yufeng Wang, Xiaoxuan Liu, Kaichao You, Zhuohan Li, Mingsheng Long, Jidong Zhai, Joseph Gonzalez, Ion Stoica, 2025

https://scholar.google.com/scholar?q=Jenga%3A+Effective+Memory+Management+for+Serving+LLM+with+Heterogeneity

4. Jamba: A Hybrid Transformer-Mamba Language Model — Opher Lieber et al. (AI21 Labs), 2024

https://scholar.google.com/scholar?q=Jamba%3A+A+Hybrid+Transformer-Mamba+Language+Model

5. Hymba: A Hybrid-head Architecture for Small Language Models — Xin Dong et al. (NVIDIA), 2024

https://scholar.google.com/scholar?q=Hymba%3A+A+Hybrid-head+Architecture+for+Small+Language+Models

6. SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention) — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, et al., 2023/2024

https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs+%28RadixAttention%29

7. Marconi: Prefix Caching for the Era of Hybrid LLMs — Rui Pan, Zhuang Wang, Zhen Jia, Can Karakus, Luca Zancato, Tri Dao, Yida Wang, Ravi Netravali, 2024

https://scholar.google.com/scholar?q=Marconi%3A+Prefix+Caching+for+the+Era+of+Hybrid+LLMs

8. Gemma 2: Improving Open Language Models at a Practical Size — Gemma Team, Google DeepMind, 2024

https://scholar.google.com/scholar?q=Gemma+2%3A+Improving+Open+Language+Models+at+a+Practical+Size

9. The Slab Allocator: An Object-Caching Kernel Memory Allocator — Jeff Bonwick, 1994

https://scholar.google.com/scholar?q=The+Slab+Allocator%3A+An+Object-Caching+Kernel+Memory+Allocator

10. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention — Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar, 2024

https://scholar.google.com/scholar?q=vAttention%3A+Dynamic+Memory+Management+for+Serving+LLMs+without+PagedAttention

11. SmartSpec: Optimizing Speculative Decoding in a Serving System — Xiaoxuan Liu et al., 2024

https://scholar.google.com/scholar?q=SmartSpec%3A+Optimizing+Speculative+Decoding+in+a+Serving+System

12. MuxServe: Flexible Multiplexing for Efficient Multiple LLM Serving — Jiangfei Duan et al., 2024

https://scholar.google.com/scholar?q=MuxServe%3A+Flexible+Multiplexing+for+Efficient+Multiple+LLM+Serving

13. Mooncake / CachedAttention (KV cache offloading to CPU and disk) — Qin et al.; Gao et al., 2024

https://scholar.google.com/scholar?q=Mooncake+%2F+CachedAttention+%28KV+cache+offloading+to+CPU+and+disk%29

Interactive Visualization: Jenga: Two-Level Memory Allocation for Heterogeneous LLM Serving

This episode examines AttnComp, a method that uses a language model's own middle-layer attention, lightly fine-tuned, to compress retrieved documents in retrieval-augmented generation. It covers why large retrieved contexts hurt: noisy passages can push accuracy below no retrieval, key facts get lost in the middle, and every extra token adds cost. It contrasts slow abstractive compressors with extractive ones that use fixed budgets and score each sentence alone, which misses bridge documents in multi-hop questions. The discussion explains how attention acts as a shared budget, so document scores compete jointly, and how a nucleus-sampling-style rule keeps as many documents as the attention requires, including none. The attention share on the instruction and first token serves as a signal for how relevant the context is. The episode also flags caveats, including attention sinks and the warning that attention is not always an explanation.

Sources:

1. AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation — Lvzhou Luo, Yixuan Cao, Ping Luo, 2025

http://arxiv.org/abs/2509.17486

2. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation — Fangyuan Xu, Weijia Shi, Eunsol Choi, 2023

https://scholar.google.com/scholar?q=RECOMP%3A+Improving+Retrieval-Augmented+LMs+with+Compression+and+Selective+Augmentation

3. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models (and LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression) — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu (Microsoft), 2023

https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models+%28and+LongLLMLingua%3A+Accelerating+and+Enhancing+LLMs+in+Long+Context+Scenarios+via+Prompt+Compression%29

4. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023 (TACL 2024)

https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts

5. Making Retrieval-Augmented Language Models Robust to Irrelevant Context — Ori Yoran, Tomer Wolfson, Ori Ram, Jonathan Berant, 2024 (ICLR)

https://scholar.google.com/scholar?q=Making+Retrieval-Augmented+Language+Models+Robust+to+Irrelevant+Context

6. Distilling Knowledge from Reader to Retriever for Question Answering — Gautier Izacard, Edouard Grave, 2020 (ICLR 2021)

https://scholar.google.com/scholar?q=Distilling+Knowledge+from+Reader+to+Retriever+for+Question+Answering

7. Retrieval Head Mechanistically Explains Long-Context Factuality — Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, Yao Fu, 2024

https://scholar.google.com/scholar?q=Retrieval+Head+Mechanistically+Explains+Long-Context+Factuality

8. Attention is not Explanation — Sarthak Jain, Byron C. Wallace, 2019

https://scholar.google.com/scholar?q=Attention+is+not+Explanation

9. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, et al., 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models

10. The Curious Case of Neural Text Degeneration — Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi, 2019 (ICLR 2020)

https://scholar.google.com/scholar?q=The+Curious+Case+of+Neural+Text+Degeneration

11. Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity — Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, Jong C. Park, 2024 (NAACL)

https://scholar.google.com/scholar?q=Adaptive-RAG%3A+Learning+to+Adapt+Retrieval-Augmented+Large+Language+Models+through+Question+Complexity

12. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection — Akari Asai, Zeqiu Wu, Ramakanth Pasunuru, Hannaneh Hajishirzi, et al., 2023 (ICLR 2024)

https://scholar.google.com/scholar?q=Self-RAG%3A+Learning+to+Retrieve%2C+Generate%2C+and+Critique+through+Self-Reflection

13. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks

14. EXIT: Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generation — Taeho Hwang, Sukmin Cho, Soyeong Jeong, Hoyun Song, SeungYoon Han, Jong C. Park, 2024

https://scholar.google.com/scholar?q=EXIT%3A+Context-Aware+Extractive+Compression+for+Enhancing+Retrieval-Augmented+Generation

15. Provence: efficient and robust context pruning for retrieval-augmented generation — Nadezhda Chirkova, Thibault Formal, Vassilina Nikoulina, Stéphane Clinchant, 2025

https://scholar.google.com/scholar?q=Provence%3A+efficient+and+robust+context+pruning+for+retrieval-augmented+generation

16. SnapKV: LLM Knows What You Are Looking For Before Generation — Yuhong Li et al., 2024

https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+Are+Looking+For+Before+Generation

17. The Llama 3 Herd of Models / Mistral 7B / Qwen2 — Grattafiori et al.; Jiang et al.; Yang et al., 2023-2024

https://scholar.google.com/scholar?q=The+Llama+3+Herd+of+Models+%2F+Mistral+7B+%2F+Qwen2

Interactive Visualization: AttnComp: Attention-Guided Adaptive Context Compression for RAG

This episode examines "Memory Layers at Scale" from Meta FAIR, which asks whether a language model can gain huge numbers of fact-storing parameters without adding any arithmetic per token. It explains how a memory layer works: a token's query retrieves the top-k rows from a large learned key-value table, and product keys keep the search cheap. It also covers where the idea came from, including the 2019 product-key work, which this paper scales up to billions of parameters with custom GPU kernels, and the related million-expert approach. The discussion examines what "matched FLOPs" leaves out, namely stored parameters and memory-bandwidth and communication costs. It also looks at how closed-book question answering on NaturalQuestions and TriviaQA is used to test the factual-recall claims against dense and mixture-of-experts baselines. Listeners interested in how models store facts, and in the gap between FLOP counts and real hardware cost, will find it useful.

Sources:

1. Memory Layers at Scale — Vincent-Pierre Berges, Barlas Oğuz, Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, Gargi Ghosh, 2024

http://arxiv.org/abs/2412.09764

2. Mixture of A Million Experts (PEER) — Xu Owen He, 2024

https://scholar.google.com/scholar?q=Mixture+of+A+Million+Experts+%28PEER%29

3. Large Memory Layers with Product Keys — Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, Hervé Jégou, 2019

https://scholar.google.com/scholar?q=Large+Memory+Layers+with+Product+Keys

4. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Damai Dai et al., 2024

https://scholar.google.com/scholar?q=DeepSeekMoE%3A+Towards+Ultimate+Expert+Specialization+in+Mixture-of-Experts+Language+Models

5. Scaling Laws for Fine-Grained Mixture of Experts — Jan Ludziejewski et al., 2024

https://scholar.google.com/scholar?q=Scaling+Laws+for+Fine-Grained+Mixture+of+Experts

6. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws — Zeyuan Allen-Zhu, Yuanzhi Li, 2024

https://scholar.google.com/scholar?q=Physics+of+Language+Models%3A+Part+3.3%2C+Knowledge+Capacity+Scaling+Laws

7. Generalization through Memorization: Nearest Neighbor Language Models (kNN-LM) — Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, Mike Lewis, 2020

https://scholar.google.com/scholar?q=Generalization+through+Memorization%3A+Nearest+Neighbor+Language+Models+%28kNN-LM%29

8. Improving language models by retrieving from trillions of tokens (RETRO) — Sebastian Borgeaud et al., 2022

https://scholar.google.com/scholar?q=Improving+language+models+by+retrieving+from+trillions+of+tokens+%28RETRO%29

9. Product Key Memory with Mixture of Experts / Mixture of Memory-style hybrids; and Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2024

https://scholar.google.com/scholar?q=Product+Key+Memory+with+Mixture+of+Experts+%2F+Mixture+of+Memory-style+hybrids%3B+and+Titans%3A+Learning+to+Memorize+at+Test+Time

10. Locating and Editing Factual Associations in GPT (ROME) and Transformer Feed-Forward Layers Are Key-Value Memories — Kevin Meng et al.; Mor Geva et al., 2022 / 2021

https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+GPT+%28ROME%29+and+Transformer+Feed-Forward+Layers+Are+Key-Value+Memories

11. Memory Mosaics / Mixture of Lookup Experts / Engram-style conditional-memory work (2025) — Various, 2025

https://scholar.google.com/scholar?q=Memory+Mosaics+%2F+Mixture+of+Lookup+Experts+%2F+Engram-style+conditional-memory+work+%282025%29

Interactive Visualization: Memory Layers at Scale: Trainable Key-Value Lookup for Facts

This episode examines LongAttnComp, a method that uses a small trained scorer to pick about 16,000 relevant tokens out of roughly 115,000. The goal is for other companies' models to find code bugs about as well as with the full input. It explains why long prompts are costly (prefill dominates time-to-first-token) and how extractive compression differs from summarization. It also ties the approach to Speculative Prefill, AttnComp, and SnapKV, which score chunks by query-aligned attention, and frames compression as the retrieval half of a retrieve-then-read pipeline. The method freezes the first 13 layers of Llama-3.1-8B-Instruct, adds one trainable cross-attention layer, and trains it on synthetic needle-in-a-haystack data. It keeps the top-scoring chunks under a token budget and sends them to the target model in their original order. Throughout, the discussion stays skeptical of the claim that compression "exceeds" full context. The baseline is middle truncation, which cuts the region where models already read poorly. The paper also never measures latency itself, and the training data is synthetic, so transfer to real inputs is open.

Sources:

1. LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning — Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li, Chen Wu, 2026

http://arxiv.org/abs/2606.01336

2. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023

https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models

3. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression — Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023 (ACL 2024)

https://scholar.google.com/scholar?q=LongLLMLingua%3A+Accelerating+and+Enhancing+LLMs+in+Long+Context+Scenarios+via+Prompt+Compression

4. Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation — Jingyu Liu, Beidi Chen, Ce Zhang, 2025

https://scholar.google.com/scholar?q=Speculative+Prefill%3A+Turbocharging+TTFT+with+Lightweight+and+Training-Free+Token+Importance+Estimation

5. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation — Fangyuan Xu, Weijia Shi, Eunsol Choi, 2023

https://scholar.google.com/scholar?q=RECOMP%3A+Improving+Retrieval-Augmented+LMs+with+Compression+and+Selective+Augmentation

6. MuSiQue: Multihop Questions via Single-hop Question Composition — Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal, 2022

https://scholar.google.com/scholar?q=MuSiQue%3A+Multihop+Questions+via+Single-hop+Question+Composition

7. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps (2WikiMultiHopQA) — Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, Akiko Aizawa, 2020

https://scholar.google.com/scholar?q=Constructing+A+Multi-hop+QA+Dataset+for+Comprehensive+Evaluation+of+Reasoning+Steps+%282WikiMultiHopQA%29

8. From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data — Zheyang Xiong, Vasilis Papageorgiou, Kangwook Lee, Dimitris Papailiopoulos, 2024

https://scholar.google.com/scholar?q=From+Artificial+Needles+to+Real+Haystacks%3A+Improving+Retrieval+Capabilities+in+LLMs+by+Finetuning+on+Synthetic+Data

9. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks — Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li, 2024

https://scholar.google.com/scholar?q=LongBench+v2%3A+Towards+Deeper+Understanding+and+Reasoning+on+Realistic+Long-context+Multitasks

10. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression — Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, et al., 2024

https://scholar.google.com/scholar?q=LLMLingua-2%3A+Data+Distillation+for+Efficient+and+Faithful+Task-Agnostic+Prompt+Compression

11. SnapKV: LLM Knows What You are Looking for Before Generation — Yuhong Li, Yingqi Huang, Bowen Yang, et al., 2024

https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation

12. Make Your LLM Fully Utilize the Context (IN2 training) — Shengnan An, Zexiong Ma, Zeqi Lin, et al., 2024

https://scholar.google.com/scholar?q=Make+Your+LLM+Fully+Utilize+the+Context+%28IN2+training%29

13. Provence: Efficient and Robust Context Pruning for Retrieval-Augmented Generation — Nadezhda Chirkova, Thibault Formal, Vassilina Nikoulina, Stéphane Clinchant, 2025

https://scholar.google.com/scholar?q=Provence%3A+Efficient+and+Robust+Context+Pruning+for+Retrieval-Augmented+Generation

14. Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Models — Shubhangi Upasani, Ravi Shanker Raju, Bo Li, Mengmeng Ji, et al., 2026

https://scholar.google.com/scholar?q=Cross-Family+Speculative+Prefill%3A+Training-Free+Long-Context+Compression+with+Small+Draft+Models

15. Compressing Context to Enhance Inference Efficiency of Large Language Models (Selective Context) — Yucheng Li, Bo Dong, Chenghua Lin, Frank Guerin, 2023

https://scholar.google.com/scholar?q=Compressing+Context+to+Enhance+Inference+Efficiency+of+Large+Language+Models+%28Selective+Context%29

16. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2024

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29

17. Retrieval Head Mechanistically Explains Long-Context Factuality — Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, Yao Fu, 2024

https://scholar.google.com/scholar?q=Retrieval+Head+Mechanistically+Explains+Long-Context+Factuality

Interactive Visualization: LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning

This episode examines "Large Memory Layers with Product Keys," which replaces a transformer's feed-forward network with a sparse memory of over a million trainable slots. The layer's central trick is splitting each query in half and scoring it against two small codebooks of sub-keys. This avoids scoring every key and cuts search cost by roughly a thousandfold. The hosts work through a toy example and the argument for why taking the top-k of each half still recovers the exact top-k. They also cover the motivation for training on the 28-billion-word CC-News corpus rather than One Billion Word, which overfits. Throughout, they stay skeptical of the headline claim that a 12-layer model with memory beats a 24-layer one at nearly twice the speed. They note that the paper doesn't test directly against approximate-nearest-neighbor memory or mixture-of-experts baselines, which makes it a useful listen for anyone wondering what "free capacity" actually costs.

Sources:

1. Product Keys Give Transformers Million-Slot Memory Layers

https://proceedings.neurips.cc/paper_files/paper/2019/file/9d8df73a3cfbf3c5b47bc9b50f214aff-Paper.pdf

2. Large Memory Layers with Product Keys — Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, Hervé Jégou, 2019

https://scholar.google.com/scholar?q=Large+Memory+Layers+with+Product+Keys

3. Product Quantization for Nearest Neighbor Search — Hervé Jégou, Matthijs Douze, Cordelia Schmid, 2011

https://scholar.google.com/scholar?q=Product+Quantization+for+Nearest+Neighbor+Search

4. Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes — Jack W. Rae, Jonathan J. Hunt, Tim Harley, Ivo Danihelka, Andrew Senior, Greg Wayne, Alex Graves, Timothy Lillicrap, 2016

https://scholar.google.com/scholar?q=Scaling+Memory-Augmented+Neural+Networks+with+Sparse+Reads+and+Writes

5. Memory Layers at Scale — Vincent-Pierre Berges, Barlas Oğuz, Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, Gargi Ghosh, 2024

https://scholar.google.com/scholar?q=Memory+Layers+at+Scale

6. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean, 2017

https://scholar.google.com/scholar?q=Outrageously+Large+Neural+Networks%3A+The+Sparsely-Gated+Mixture-of-Experts+Layer

7. Generalization through Memorization: Nearest Neighbor Language Models — Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, Mike Lewis, 2020

https://scholar.google.com/scholar?q=Generalization+through+Memorization%3A+Nearest+Neighbor+Language+Models

8. Transformer Feed-Forward Layers Are Key-Value Memories — Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy, 2021

https://scholar.google.com/scholar?q=Transformer+Feed-Forward+Layers+Are+Key-Value+Memories

9. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — William Fedus, Barret Zoph, Noam Shazeer, 2021

https://scholar.google.com/scholar?q=Switch+Transformers%3A+Scaling+to+Trillion+Parameter+Models+with+Simple+and+Efficient+Sparsity

10. Improving language models by retrieving from trillions of tokens (RETRO) — Sebastian Borgeaud et al., 2022

https://scholar.google.com/scholar?q=Improving+language+models+by+retrieving+from+trillions+of+tokens+%28RETRO%29

11. Mixture of A Million Experts (PEER) — Xu Owen He, 2024

https://scholar.google.com/scholar?q=Mixture+of+A+Million+Experts+%28PEER%29

12. Scaling Laws for Fine-Grained Mixture of Experts — Jan Ludziejewski et al., 2024

https://scholar.google.com/scholar?q=Scaling+Laws+for+Fine-Grained+Mixture+of+Experts

13. Large Product Key Memory for Pretrained Language Models — Gyuwan Kim, Tae-Hwan Jung, 2020

https://scholar.google.com/scholar?q=Large+Product+Key+Memory+for+Pretrained+Language+Models

14. Adaptive Input Representations for Neural Language Modeling — Alexei Baevski, Michael Auli, 2019

https://scholar.google.com/scholar?q=Adaptive+Input+Representations+for+Neural+Language+Modeling

Interactive Visualization: Product Keys Give Transformers Million-Slot Memory Layers

This episode explores "Context Language Models," a paper that lets a model rewrite its own context window with ordinary Bash or Python edits, instead of relying on harness-defined truncation or a fixed menu of context actions. It covers the three tiers of context control, the serving catch (an edit in the middle or at the start of the context can cost several times more prefill compute than an append, because it breaks the shared prefix), and the ContextBench diagnostic. On the BrowseComp-Plus deep-research benchmark, the approach scores 59.4 percent, about six points above a summary-compaction baseline, with roughly a fifth less compute. The discussion also covers the models' emergent behavior, including an orchestrator that updates a subagent scoreboard in place 163 times and a model that writes and calls its own compaction function. Listeners get a critical look at whether these gains hold up, given that the cost metric is analytic FLOPs rather than measured latency.

Sources:

1. Context Language Models — Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li, Minheng Wang, Hamish Ivison, Radha Poovendran, Nathan Lambert, Teng Xiao, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Pang Wei Koh, 2026

http://arxiv.org/abs/2609.37725

2. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023

https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems

3. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023 (TACL 2024)

https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts

4. Recursive Language Models — Alex L. Zhang, Tim Kraska, Omar Khattab, 2025

https://scholar.google.com/scholar?q=Recursive+Language+Models

5. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models (ACE) — Qizheng Zhang et al. (Stanford, SambaNova and collaborators), 2025

https://scholar.google.com/scholar?q=Agentic+Context+Engineering%3A+Evolving+Contexts+for+Self-Improving+Language+Models+%28ACE%29

6. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

7. SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention) — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2023 (NeurIPS 2024)

https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs+%28RadixAttention%29

8. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang, 2024 (EuroSys 2025)

https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion

9. Prompt Cache: Modular Attention Reuse for Low-Latency Inference — In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, Lin Zhong, 2023 (MLSys 2024)

https://scholar.google.com/scholar?q=Prompt+Cache%3A+Modular+Attention+Reuse+for+Low-Latency+Inference

10. EPIC: Efficient Position-Independent Caching for Serving Large Language Models — Junhao Hu, Wenrui Huang, Weidong Wang, et al., 2025

https://scholar.google.com/scholar?q=EPIC%3A+Efficient+Position-Independent+Caching+for+Serving+Large+Language+Models

11. Let the Code LLM Edit Itself When You Edit the Code (PIE) — Zhenyu He, Jun Zhang, Shengjie Luo, Jingjing Xu, Zhi Zhang, Di He, 2025

https://scholar.google.com/scholar?q=Let+the+Code+LLM+Edit+Itself+When+You+Edit+the+Code+%28PIE%29

12. Memento: Teaching LLMs to Manage Their Own Context — Vasilis Kontonis, Yuchen Zeng, Shivam Garg, et al., 2026

https://scholar.google.com/scholar?q=Memento%3A+Teaching+LLMs+to+Manage+Their+Own+Context

13. Scaling Long-Horizon LLM Agent via Context-Folding — Weiwei Sun, Miao Lu, Zhan Ling, et al., 2025

https://scholar.google.com/scholar?q=Scaling+Long-Horizon+LLM+Agent+via+Context-Folding

14. MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents — Zijian Zhou, Ao Qu, Zhaoxuan Wu, et al., 2026

https://scholar.google.com/scholar?q=MEM1%3A+Learning+to+Synergize+Memory+and+Reasoning+for+Efficient+Long-Horizon+Agents

15. Meta-Harness: End-to-End Optimization of Model Harnesses — Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn, 2026

https://scholar.google.com/scholar?q=Meta-Harness%3A+End-to-End+Optimization+of+Model+Harnesses

16. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning — Lakshya A. Agrawal, et al., 2026

https://scholar.google.com/scholar?q=GEPA%3A+Reflective+Prompt+Evolution+Can+Outperform+Reinforcement+Learning

17. Efficient Memory Management for Large Language Model Serving with PagedAttention; and SGLang / RadixAttention — Kwon et al. (vLLM); Lianmin Zheng et al. (SGLang), 2023/2024

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention%3B+and+SGLang+%2F+RadixAttention

18. Self-Generated Prompt Injections in Compaction Summaries — OpenAI, 2026

https://scholar.google.com/scholar?q=Self-Generated+Prompt+Injections+in+Compaction+Summaries

19. Training Language Models to Reason Efficiently; and Think Dense, Not Long (DDCA) — Daman Arora, Andrea Zanette; Keqin Peng et al., 2026

https://scholar.google.com/scholar?q=Training+Language+Models+to+Reason+Efficiently%3B+and+Think+Dense%2C+Not+Long+%28DDCA%29

20. Titans / Mamba / Gated DeltaNet and learned-memory architectures — Behrouz et al.; Gu & Dao; Yang et al., 2024–2025

https://scholar.google.com/scholar?q=Titans+%2F+Mamba+%2F+Gated+DeltaNet+and+learned-memory+architectures

Interactive Visualization: Context Language Models: Letting the Model Rewrite Its Own Context

This episode examines Chakra, an MLCommons effort to define one open, graph-based execution-trace format. The format is meant to carry a distributed AI workload from production observation through benchmark replay to hardware simulation, without exposing a proprietary model. It explains how a trace is a per-device directed acyclic graph of compute, memory, and communication nodes linked by control, data, and synchronization dependencies. Process groups encode tensor, pipeline, data, and expert parallelism, and the traces carry no weights or data. The discussion separates post-execution traces, which are recorded from real runs and carry that machine's timings, from pre-execution traces, which are portable but need timing from elsewhere. It also covers how PyTorch's execution graph observer and Kineto profiler are merged to capture traces, and how replay, simulation (ASTRA-sim), and emulation consume them. The hosts treat the ecosystem story as strong but press on whether the evidence supports the paper's boldest claims, which makes this useful for anyone interested in benchmarking and co-designing AI infrastructure.

Sources:

1. MLCommons Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Traces — Srinivas Sridharan, Theodor-Adrian Badea, Andy Balogh, Bradford M. Beckmann, Brian Coutinho, Louis Feng, Sheng Fu, Sanshan Gao, Mehryar Garakani, Taekyung Heo, David Kanter, Josh Ladd, Ziwei Li, Winston Liu, Changhai Man, Dan Mihailescu, Spandan More, Joongun Park, Ashwin Ramachandran, Vinay Ramakrishnaiah, Saeed Rashidi, Vijay Janapa Reddi, Puneet Sharma, Phio Tian, William Won, Hanjiang Wu, Huan Xu, Jinsun Yoo, Tushar Krishna, 2026

http://arxiv.org/abs/2605.11333

2. ASTRA-sim: Enabling SW/HW Co-Design Exploration for Distributed DL Training Platforms — Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, Tushar Krishna, 2020

https://scholar.google.com/scholar?q=ASTRA-sim%3A+Enabling+SW%2FHW+Co-Design+Exploration+for+Distributed+DL+Training+Platforms

3. ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale — William Won, Taekyung Heo, Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, Tushar Krishna, 2023

https://scholar.google.com/scholar?q=ASTRA-sim2.0%3A+Modeling+Hierarchical+Networks+and+Disaggregated+Systems+for+Large-model+Training+at+Scale

4. Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Traces — Srinivas Sridharan, Taekyung Heo, Louis Feng, Zhaodong Wang, Melanie Bhatt, Wei Wen, Tushar Krishna, and collaborators, 2023

https://scholar.google.com/scholar?q=Chakra%3A+Advancing+Performance+Benchmarking+and+Co-design+using+Standardized+Execution+Traces

5. PyTorch Profiler / Kineto and the PyTorch Execution Trace Observer (project documentation) — Meta PyTorch team, 2021-2023

https://scholar.google.com/scholar?q=PyTorch+Profiler+%2F+Kineto+and+the+PyTorch+Execution+Trace+Observer+%28project+documentation%29

6. MLPerf Training Benchmark — Peter Mattson, Christine Cheng, Gregory Diamos, Cody Coleman, Paulius Micikevicius, David Patterson, and many co-authors, 2020

https://scholar.google.com/scholar?q=MLPerf+Training+Benchmark

7. MLPerf Inference Benchmark — Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, and many co-authors, 2020

https://scholar.google.com/scholar?q=MLPerf+Inference+Benchmark

8. MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI — Arya Tschand, Arun Tejusve Raghunath Rajan, Sachin Idgunji, Anirban Ghosh, Jeremy Holleman, Csaba Kiraly, Pawan Ambalkar, Ritika Borkar, Ramesh Chukka, Trevor Cockrell, Oliver Curtis, Grigori Fursin, Miro Hodak, Hiwot Kassa, Anton Lokhmotov, Dejan Miskovic, Yuechao Pan, Manu Prasad Manmathan, Liz Raymond, Tom St. John, Arjun Suresh, Rowan Taubitz, Sean Zhan, Scott Wasson, David Kanter, Vijay Janapa Reddi, 2024

https://scholar.google.com/scholar?q=MLPerf+Power%3A+Benchmarking+the+Energy+Efficiency+of+Machine+Learning+Systems+from+Microwatts+to+Megawatts+for+Sustainable+AI

9. SPEC CPU benchmark suite documentation and rationale (Standard Performance Evaluation Corporation) — SPEC, with historical analysis by John L. Henning and others, 1989-2017

https://scholar.google.com/scholar?q=SPEC+CPU+benchmark+suite+documentation+and+rationale+%28Standard+Performance+Evaluation+Corporation%29

10. In-Datacenter Performance Analysis of a Tensor Processing Unit — Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, and many co-authors, 2017

https://scholar.google.com/scholar?q=In-Datacenter+Performance+Analysis+of+a+Tensor+Processing+Unit

11. A Domain-Specific Architecture for Deep Neural Networks / A New Golden Age for Computer Architecture — John L. Hennessy and David A. Patterson, 2019

https://scholar.google.com/scholar?q=A+Domain-Specific+Architecture+for+Deep+Neural+Networks+%2F+A+New+Golden+Age+for+Computer+Architecture

12. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022

https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness

13. Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models — Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, and many co-authors at Meta, 2022

https://scholar.google.com/scholar?q=Software-Hardware+Co-design+for+Fast+and+Scalable+Training+of+Deep+Learning+Recommendation+Models

14. Mystique: Enabling Accurate and Scalable Generation of Production AI Benchmarks — Mengyuan (Mingyu) Liang, Wenyin Fu, Louis Feng, Zhongyi Lin, Pavani Panakanti, Shengbao Zheng, Srinivas Sridharan, Christina Delimitrou, 2023

https://scholar.google.com/scholar?q=Mystique%3A+Enabling+Accurate+and+Scalable+Generation+of+Production+AI+Benchmarks

15. Scaling Deep Learning Training with MPI/Collective Communication: NCCL and related collective library documentation (NVIDIA NCCL, AMD RCCL) — NVIDIA and AMD library teams, 2016-2025

https://scholar.google.com/scholar?q=Scaling+Deep+Learning+Training+with+MPI%2FCollective+Communication%3A+NCCL+and+related+collective+library+documentation+%28NVIDIA+NCCL%2C+AMD+RCCL%29

16. Characterizing and Understanding Distributed Training Workloads / Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads — Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, Fan Yang, 2019

https://scholar.google.com/scholar?q=Characterizing+and+Understanding+Distributed+Training+Workloads+%2F+Analysis+of+Large-Scale+Multi-Tenant+GPU+Clusters+for+DNN+Training+Workloads

17. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters — Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, Yu Ding, 2022

https://scholar.google.com/scholar?q=MLaaS+in+the+Wild%3A+Workload+Analysis+and+Scheduling+in+Large-Scale+Heterogeneous+GPU+Clusters

18. Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Training — Liang et al., 2025

https://scholar.google.com/scholar?q=Lumos%3A+Efficient+Performance+Modeling+and+Estimation+for+Large-scale+LLM+Training

19. Holistic Trace Analysis (HTA) — Meta / FacebookResearch, 2023

https://scholar.google.com/scholar?q=Holistic+Trace+Analysis+%28HTA%29

20. SimAI: Unifying Architecture Design and Performance Tuning for Large-Scale LLM Training with Scalability and Precision — Wang et al. (Alibaba), 2025

https://scholar.google.com/scholar?q=SimAI%3A+Unifying+Architecture+Design+and+Performance+Tuning+for+Large-Scale+LLM+Training+with+Scalability+and+Precision

21. Echo / Proteus / vTrain / Calculon style performance models (vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal LLM Training) — Bang et al., 2024

https://scholar.google.com/scholar?q=Echo+%2F+Proteus+%2F+vTrain+%2F+Calculon+style+performance+models+%28vTrain%3A+A+Simulation+Framework+for+Evaluating+Cost-effective+and+Compute-optimal+LLM+Training%29

22. Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization — Unger et al., 2022

https://scholar.google.com/scholar?q=Unity%3A+Accelerating+DNN+Training+Through+Joint+Optimization+of+Algebraic+Transformations+and+Parallelization

23. GOAL: A Generalized Goal-oriented Application Language for Network/HPC Trace Simulation (LogGOPSim lineage) — Hoefler et al. / Shen et al., 2010 / 2025

https://scholar.google.com/scholar?q=GOAL%3A+A+Generalized+Goal-oriented+Application+Language+for+Network%2FHPC+Trace+Simulation+%28LogGOPSim+lineage%29

24. Llumnix / Splitwise / DistServe / Mooncake — Patel et al.; Zhong et al.; Qin et al., 2024

https://scholar.google.com/scholar?q=Llumnix+%2F+Splitwise+%2F+DistServe+%2F+Mooncake

25. Characterization of Large Language Model Development in the Datacenter (Acme) — Hu et al., 2024

https://scholar.google.com/scholar?q=Characterization+of+Large+Language+Model+Development+in+the+Datacenter+%28Acme%29

26. MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs — Jiang et al. (ByteDance), 2024

https://scholar.google.com/scholar?q=MegaScale%3A+Scaling+Large+Language+Model+Training+to+More+Than+10%2C000+GPUs

27. STAGE: Symbolic Tensor Graph Engine for LLM Workload Synthesis — Man et al., 2026

https://scholar.google.com/scholar?q=STAGE%3A+Symbolic+Tensor+Graph+Engine+for+LLM+Workload+Synthesis

28. MLPerf Training / Inference benchmark papers — Mattson et al.; Reddi et al., 2020

https://scholar.google.com/scholar?q=MLPerf+Training+%2F+Inference+benchmark+papers

Interactive Visualization: MLCommons Chakra: Standardized Execution Traces for AI Benchmarking and Co-design