AI Post Transformers

This episode explores Context Language Models, a proposal that lets a model write directly to its own context as an editable file instead of relying on human-designed harnesses. It first surveys the existing approaches: fixed-policy compaction, action-based tools such as Context Folding, and recursive language models. It then shows why they fall short on tasks like in-place Sudoku edits, using a diagnostic benchmark that varies context pressure. The discussion explains why edits to the middle of a context break prefix caching and force costly recomputation, and why the paper measures cost in prefix-reuse FLOPs. It also lays out the paper's five claims: zero-shot gains on deep research and coding, swarms of agents with file-based contexts, evolved natural-language skills, online reinforcement learning, and a serving change called Suffix Cache Reuse. The hosts stress the difference between percentage points and relative percent, and they note that the strongest evidence is zero-shot with a large model while the training evidence is thinner.

Sources:

1. Context Language Models — Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li, Minheng Wang, Hamish Ivison, Radha Poovendran, Nathan Lambert, Teng Xiao, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Pang Wei Koh, 2026

http://arxiv.org/abs/2609.37725

2. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO) — Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. (DeepSeek-AI), 2024

https://scholar.google.com/scholar?q=DeepSeekMath%3A+Pushing+the+Limits+of+Mathematical+Reasoning+in+Open+Language+Models+%28introduces+GRPO%29

3. MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents — Zijian Zhou et al., 2025

https://scholar.google.com/scholar?q=MEM1%3A+Learning+to+Synergize+Memory+and+Reasoning+for+Efficient+Long-Horizon+Agents

4. Scaling Long-Horizon LLM Agent via Context-Folding — Weiwei Sun et al. (ByteDance Seed / CMU), 2025

https://scholar.google.com/scholar?q=Scaling+Long-Horizon+LLM+Agent+via+Context-Folding

5. BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent — Zijian Chen, Xueguang Ma, Shengyao Zhuang, et al. (Waterloo and collaborators), 2025

https://scholar.google.com/scholar?q=BrowseComp-Plus%3A+A+More+Fair+and+Transparent+Evaluation+Benchmark+of+Deep-Research+Agent

6. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023

https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems

7. How we built our multi-agent research system (Anthropic engineering report) / Effective context engineering for AI agents — Anthropic engineering team, 2025

https://scholar.google.com/scholar?q=How+we+built+our+multi-agent+research+system+%28Anthropic+engineering+report%29+%2F+Effective+context+engineering+for+AI+agents

8. Don't Build Multi-Agents (Cognition AI blog post) — Walden Yan (Cognition AI), 2025

https://scholar.google.com/scholar?q=Don%27t+Build+Multi-Agents+%28Cognition+AI+blog+post%29

9. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models (ACE) — Qizheng Zhang et al. (Stanford, SambaNova and collaborators), 2025

https://scholar.google.com/scholar?q=Agentic+Context+Engineering%3A+Evolving+Contexts+for+Self-Improving+Language+Models+%28ACE%29

10. ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization — Wu et al., 2025

https://scholar.google.com/scholar?q=ReSum%3A+Unlocking+Long-Horizon+Search+Intelligence+via+Context+Summarization

11. Training Language Models to Reason Efficiently — Arora and Zanette, 2025 (NeurIPS)

https://scholar.google.com/scholar?q=Training+Language+Models+to+Reason+Efficiently

12. Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning (DDCA) — Peng et al., 2026

https://scholar.google.com/scholar?q=Think+Dense%2C+Not+Long%3A+Dynamic+Decoupled+Conditional+Advantage+for+Efficient+Reasoning+%28DDCA%29

13. CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents — Li, Hou, Jing, Tang, Dong, 2026

https://scholar.google.com/scholar?q=CompactionRL%3A+Reinforcement+Learning+with+Context+Compaction+for+Long-Horizon+Agents

14. Memento: Teaching LLMs to Manage Their Own Context — Kontonis et al., 2026

https://scholar.google.com/scholar?q=Memento%3A+Teaching+LLMs+to+Manage+Their+Own+Context

15. CacheBlend: Fast LLM Serving for RAG with Cached Knowledge Fusion; EPIC: Efficient Position-Independent Caching — Yao et al.; Hu et al., 2025

https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+LLM+Serving+for+RAG+with+Cached+Knowledge+Fusion%3B+EPIC%3A+Efficient+Position-Independent+Caching

16. Recursive Language Models — Zhang, Kraska, Khattab, 2025

https://scholar.google.com/scholar?q=Recursive+Language+Models

17. Sculptor: Empowering LLMs with Cognitive Agency via Active Context Management — Li et al., 2026

https://scholar.google.com/scholar?q=Sculptor%3A+Empowering+LLMs+with+Cognitive+Agency+via+Active+Context+Management

18. Self-Generated Prompt Injections in Compaction Summaries — OpenAI, 2026

https://scholar.google.com/scholar?q=Self-Generated+Prompt+Injections+in+Compaction+Summaries

19. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning — Agrawal et al., 2026

https://scholar.google.com/scholar?q=GEPA%3A+Reflective+Prompt+Evolution+Can+Outperform+Reinforcement+Learning

20. Meta-Harness: End-to-End Optimization of Model Harnesses — Lee, Nair, Zhang, Lee, Khattab, Finn, 2026

https://scholar.google.com/scholar?q=Meta-Harness%3A+End-to-End+Optimization+of+Model+Harnesses

Interactive Visualization: Context Language Models: Letting Models Edit Their Own Context

This episode examines "500xCompressor," a soft prompt compression method that squeezes hundreds of tokens of context into as little as a single learned token. A frozen Llama-3-8B decoder then answers questions from that token's key-value (KV) vectors alone. It covers why long contexts are costly (KV cache memory and attention cost grow with length). It also contrasts hard compression, which deletes text (Selective Context, LLMLingua), with soft approaches (gist tokens, AutoCompressor, ICAE, xRAG). The discussion walks through how a LoRA-adapted encoder and autoencoder-style regeneration pretraining produce those KV values, and what the headline 6x–480x ratios and 62–73% retained capability actually measure. The hosts also scrutinize the evaluation: arXiv abstracts from early 2024 are used to avoid pretraining leakage, and F1 versus Exact Match are explained. Listeners get a critical look at whether KV beats embeddings and whether the benchmark can be trusted.

Sources:

1. 500xCompressor: Generalized Prompt Compression for Large Language Models — Zongqian Li, Yixuan Su, Nigel Collier, 2024

http://arxiv.org/abs/2408.03094v1

2. Compressing Context to Enhance Inference Efficiency of Large Language Models (Selective Context) — Li, Dong, Guerin, Lin, 2023

https://scholar.google.com/scholar?q=Compressing+Context+to+Enhance+Inference+Efficiency+of+Large+Language+Models+%28Selective+Context%29

3. Learning to Compress Prompts with Gist Tokens — Mu, Li, Goodman, 2023

https://scholar.google.com/scholar?q=Learning+to+Compress+Prompts+with+Gist+Tokens

4. Adapting Language Models to Compress Contexts (AutoCompressor) — Chevalier, Wettig, Ajith, Chen, 2023

https://scholar.google.com/scholar?q=Adapting+Language+Models+to+Compress+Contexts+%28AutoCompressor%29

5. xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token — Cheng et al., 2024

https://scholar.google.com/scholar?q=xRAG%3A+Extreme+Context+Compression+for+Retrieval-augmented+Generation+with+One+Token

6. In-context Autoencoder for Context Compression in a Large Language Model (ICAE) — Ge, Jing, Wang, Wang, Chen, Wei, 2024

https://scholar.google.com/scholar?q=In-context+Autoencoder+for+Context+Compression+in+a+Large+Language+Model+%28ICAE%29

7. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Li, Liang, 2021

https://scholar.google.com/scholar?q=Prefix-Tuning%3A+Optimizing+Continuous+Prompts+for+Generation

8. Cartridges: Lightweight and general-purpose long context representations via self-study — Eyuboglu et al., 2025

https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+general-purpose+long+context+representations+via+self-study

9. Function Vectors in Large Language Models — Todd et al., 2024

https://scholar.google.com/scholar?q=Function+Vectors+in+Large+Language+Models

10. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Xiao et al., 2024

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29

11. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhang et al., 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models

Interactive Visualization: 500xCompressor: Compressing Prompts into a Single Learned Token

This episode examines "Investigating the Effectiveness of HyperTuning via Gisting," which asks whether a few-shot examples can be compressed into just sixteen tokens of state so a frozen Llama-2 adapts to a task without backpropagation, instead of re-reading the examples. It sets up the cost trade-offs of in-context learning (examples re-read on every request), parameter-efficient fine-tuning (training needed per task), and hypernetworks (amortized one-time training that generates adapters in a single forward pass). The central insight is that Gist tokens' key and value states are effectively a soft prefix, so a Gisting model can act as a prefix generator using only an attention-mask change, with no dedicated parameter-generation heads or expensive hyperpretraining stage like the earlier HyperT5. The episode also describes the HyperLlama design, a single 4-bit quantized Llama-2-7B that plays hypernetwork or frozen model depending on whether its LoRA adapter is on. Listeners get a clear explanation of how the approach is built, why it is cheap, and why its author reports only mixed performance against simply reading the examples.

Sources:

1. Investigating the Effectiveness of HyperTuning via Gisting — Jason Phang, 2024

http://arxiv.org/abs/2402.16817

2. HyperNetworks — David Ha, Andrew M. Dai, Quoc V. Le, 2017 (ICLR 2017)

https://scholar.google.com/scholar?q=HyperNetworks

3. HyperTuning: Toward Adapting Large Language Models without Back-propagation — Jason Phang, Yi Mao, Pengcheng He, Weizhu Chen, 2023 (ICML 2023)

https://scholar.google.com/scholar?q=HyperTuning%3A+Toward+Adapting+Large+Language+Models+without+Back-propagation

4. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023 (NeurIPS 2023)

https://scholar.google.com/scholar?q=Learning+to+Compress+Prompts+with+Gist+Tokens

5. HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation — Hamish Ivison, Akshita Bhagia, Yizhong Wang, Hannaneh Hajishirzi, Matthew E. Peters, 2023 (ACL 2023)

https://scholar.google.com/scholar?q=HINT%3A+Hypernetwork+Instruction+Tuning+for+Efficient+Zero-+%26+Few-Shot+Generalisation

6. HyperTuning: Toward Adapting Large Language Models without Backpropagation — Jason Phang, Yi Mao, Pengcheng He, Weizhu Chen, 2023

https://scholar.google.com/scholar?q=HyperTuning%3A+Toward+Adapting+Large+Language+Models+without+Backpropagation

7. Adapting Language Models to Compress Contexts (AutoCompressors) — Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen, 2023

https://scholar.google.com/scholar?q=Adapting+Language+Models+to+Compress+Contexts+%28AutoCompressors%29

8. In-context Autoencoder for Context Compression in a Large Language Model (ICAE) — Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, Furu Wei, 2023

https://scholar.google.com/scholar?q=In-context+Autoencoder+for+Context+Compression+in+a+Large+Language+Model+%28ICAE%29

9. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Xiang Lisa Li, Percy Liang, 2021

https://scholar.google.com/scholar?q=Prefix-Tuning%3A+Optimizing+Continuous+Prompts+for+Generation

10. Transformers Learn In-Context by Gradient Descent — Johannes von Oswald et al., 2023

https://scholar.google.com/scholar?q=Transformers+Learn+In-Context+by+Gradient+Descent

11. Symbol tuning improves in-context learning in language models — Jerry Wei et al., 2023

https://scholar.google.com/scholar?q=Symbol+tuning+improves+in-context+learning+in+language+models

12. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu et al., 2025

https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study

13. Text-to-LoRA: Instant Transformer Adaption — Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, Robert Lange, 2025

https://scholar.google.com/scholar?q=Text-to-LoRA%3A+Instant+Transformer+Adaption

14. Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights — Zhiyuan Liang et al., 2025

https://scholar.google.com/scholar?q=Drag-and-Drop+LLMs%3A+Zero-Shot+Prompt-to-Weights

15. Hypernetworks (HyperNetworks) — David Ha, Andrew Dai, Quoc Le, 2017

https://scholar.google.com/scholar?q=Hypernetworks+%28HyperNetworks%29

16. 500xCompressor: Generalized Prompt Compression for Large Language Models — Zongqian Li, Yixuan Su, Nigel Collier, 2024

https://scholar.google.com/scholar?q=500xCompressor%3A+Generalized+Prompt+Compression+for+Large+Language+Models

Interactive Visualization: HyperTuning via Gisting: Cheap Adapter Generation, Mixed Results

This episode examines "HyperTuning," an ICML 2023 paper that uses a hypernetwork (HyperT5) to adapt a frozen language model to a new task in a single forward pass, with no back-propagation through the downstream model. It explains how the hypermodel reads up to sixteen labeled examples and emits prefix or LoRA parameters that the downstream model reuses for every query without seeing the examples again. The discussion covers why per-task adaptation is costly, how parameter-efficient methods still require a full backward pass, and how the work relates to earlier hypernetwork, meta-learning, and instruction-to-parameter approaches. It also sets up a skeptical reading of the paper's three claims: that HyperT5 roughly matches full fine-tuning on held-out tasks, that it beats shared-parameter PEFT baselines, and that its output works as a PEFT initialization. The skeptical questions are where the training cost actually moves to, and whether the headline margins hold beyond single-run noise.

Sources:

1. HyperTuning: Adapting Language Models Without Back-propagation Using Hypernetworks

https://proceedings.mlr.press/v202/phang23a/phang23a.pdf

2. Multitask Prompted Training Enables Zero-Shot Task Generalization (T0) — Victor Sanh, Albert Webson, Colin Raffel, et al., 2022 (ICLR; arXiv 2021)

https://scholar.google.com/scholar?q=Multitask+Prompted+Training+Enables+Zero-Shot+Task+Generalization+%28T0%29

3. Finetuned Language Models Are Zero-Shot Learners (FLAN) — Jason Wei, Maarten Bosma, Vincent Zhao, et al., 2022 (ICLR; arXiv 2021)

https://scholar.google.com/scholar?q=Finetuned+Language+Models+Are+Zero-Shot+Learners+%28FLAN%29

4. Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, et al., 2022

https://scholar.google.com/scholar?q=Super-NaturalInstructions%3A+Generalization+via+Declarative+Instructions+on+1600%2B+NLP+Tasks

5. MetaICL: Learning to Learn In Context — Sewon Min, Mike Lewis, Luke Zettlemoyer, Hannaneh Hajishirzi, 2022

https://scholar.google.com/scholar?q=MetaICL%3A+Learning+to+Learn+In+Context

6. HyperNetworks — David Ha, Andrew Dai, Quoc V. Le, 2016

https://scholar.google.com/scholar?q=HyperNetworks

7. HINT: Hypernetwork Instruction Tuning for Efficient Zero-Shot Generalisation — Hamish Ivison, Akshita Bhagia, Yizhong Wang, Hannaneh Hajishirzi, Matthew Peters, 2022

https://scholar.google.com/scholar?q=HINT%3A+Hypernetwork+Instruction+Tuning+for+Efficient+Zero-Shot+Generalisation

8. Boosting Natural Language Generation from Instructions with Meta-Learning — Budhaditya Deb, Guoqing Zheng, Ahmed Hassan Awadallah, 2022

https://scholar.google.com/scholar?q=Boosting+Natural+Language+Generation+from+Instructions+with+Meta-Learning

9. Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (HyperFormer) — Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, James Henderson, 2021

https://scholar.google.com/scholar?q=Parameter-efficient+Multi-task+Fine-tuning+for+Transformers+via+Shared+Hypernetworks+%28HyperFormer%29

10. SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer — Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, Daniel Cer, 2021

https://scholar.google.com/scholar?q=SPoT%3A+Better+Frozen+Model+Adaptation+through+Soft+Prompt+Transfer

11. Reducing Retraining by Recycling Parameter-Efficient Prompts — Brian Lester, Joshua Yurtsever, Siamak Shakeri, Noah Constant, 2022

https://scholar.google.com/scholar?q=Reducing+Retraining+by+Recycling+Parameter-Efficient+Prompts

12. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (MAML) — Chelsea Finn, Pieter Abbeel, Sergey Levine, 2017

https://scholar.google.com/scholar?q=Model-Agnostic+Meta-Learning+for+Fast+Adaptation+of+Deep+Networks+%28MAML%29

13. Hypernetworks and meta-learned weight generators for LLMs (e.g. Text-to-LoRA, Drag-and-Drop LLMs) — Sakana AI (Text-to-LoRA); Liang et al. (Drag-and-Drop LLMs), 2025

https://scholar.google.com/scholar?q=Hypernetworks+and+meta-learned+weight+generators+for+LLMs+%28e.g.+Text-to-LoRA%2C+Drag-and-Drop+LLMs%29

14. Transformers generalize differently from information stored in context vs in weights — Stephanie Chan, Ishita Dasgupta, Junkyung Kim, Dharshan Kumaran, Andrew Lampinen, Felix Hill, 2022

https://scholar.google.com/scholar?q=Transformers+generalize+differently+from+information+stored+in+context+vs+in+weights

Interactive Visualization: HyperTuning: Adapting Language Models Without Back-propagation Using Hypernetworks

This episode examines "Learning to Compress Prompts with Gist Tokens," which trains a language model to squeeze an instruction into a few cached gist-token activations. The trick is a modified attention mask: tokens after the gist can't see the original prompt, so its information has to pass through the gist positions. The discussion covers why the KV cache and per-task methods like prefix tuning, prompt tuning, and context distillation don't fully solve the cost of reused prompts. It frames gisting as a hypernetwork that predicts the compressed prefix in one forward pass, trained with ordinary instruction tuning and no extra objective. It also questions the headline 26x compression figure. That number is a ratio of averages from prompts of about 26 tokens, which may not support the long system-prompt use case the paper opens with. Listeners get a clear explanation of the mechanism and a skeptical look at what the evidence actually shows.

Sources:

1. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023

http://arxiv.org/abs/2304.08467

2. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023 (NeurIPS 2023; arXiv 2304.08467)

https://scholar.google.com/scholar?q=Learning+to+Compress+Prompts+with+Gist+Tokens

3. The Power of Scale for Parameter-Efficient Prompt Tuning — Brian Lester, Rami Al-Rfou, Noah Constant, 2021

https://scholar.google.com/scholar?q=The+Power+of+Scale+for+Parameter-Efficient+Prompt+Tuning

4. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Xiang Lisa Li, Percy Liang, 2021

https://scholar.google.com/scholar?q=Prefix-Tuning%3A+Optimizing+Continuous+Prompts+for+Generation

5. Adapting Language Models to Compress Contexts (AutoCompressors) — Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen, 2023

https://scholar.google.com/scholar?q=Adapting+Language+Models+to+Compress+Contexts+%28AutoCompressors%29

6. In-context Autoencoder for Context Compression in a Large Language Model (ICAE) — Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, Furu Wei, 2023

https://scholar.google.com/scholar?q=In-context+Autoencoder+for+Context+Compression+in+a+Large+Language+Model+%28ICAE%29

7. Language Modeling with Learned Summary or Memory Tokens: Compressed Context Memory for Online Language Model Interaction — Jang-Hyun Kim, Junyoung Yeom, Sangdoo Yun, Hyun Oh Song, 2023

https://scholar.google.com/scholar?q=Language+Modeling+with+Learned+Summary+or+Memory+Tokens%3A+Compressed+Context+Memory+for+Online+Language+Model+Interaction

8. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29

9. Longformer: The Long-Document Transformer — Iz Beltagy, Matthew E. Peters, Arman Cohan, 2020

https://scholar.google.com/scholar?q=Longformer%3A+The+Long-Document+Transformer

10. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023

https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models

11. Learning to Compress Contexts / Compressive Transformers for Long-Range Sequence Modelling — Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, Timothy P. Lillicrap, 2020

https://scholar.google.com/scholar?q=Learning+to+Compress+Contexts+%2F+Compressive+Transformers+for+Long-Range+Sequence+Modelling

12. Learning by Distilling Context — Charlie Snell, Dan Klein, Ruiqi Zhong, 2022

https://scholar.google.com/scholar?q=Learning+by+Distilling+Context

13. Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models — David Wingate, Mohammad Shoeybi, Taylor Sorensen, 2022

https://scholar.google.com/scholar?q=Prompt+Compression+and+Contrastive+Conditioning+for+Controllability+and+Toxicity+Reduction+in+Language+Models

14. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, et al., 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%29

15. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Joshua Ainslie, James Lee-Thorp, Michiel de Jong, et al., 2023

https://scholar.google.com/scholar?q=GQA%3A+Training+Generalized+Multi-Query+Transformer+Models+from+Multi-Head+Checkpoints

16. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al., 2023

https://scholar.google.com/scholar?q=Judging+LLM-as-a-Judge+with+MT-Bench+and+Chatbot+Arena

Interactive Visualization: Gist Tokens: Compressing Prompts Up to 26x

This episode examines "Mixture of A Million Experts," a 2024 paper proposing a learned router that selects among more than a million single-neuron experts at sublinear cost. It explains why conventional mixture-of-experts routing stalls around 128 experts: scoring every expert against every token scales linearly. It also covers why fixed hash routers fall short. The discussion traces the paper's building blocks: fine-grained MoE scaling laws that favor ever-smaller experts, Product Key Memory's Cartesian-product trick for cheap retrieval, and the view of feedforward layers as key-value memories. Along the way, the hosts flag caveats, including the paper's redefinition of granularity, the extrapolation of a scaling law far beyond where it was fitted, and lifelong learning being motivation only, with no experiment. They also raise whether FLOP-based isoFLOP wins will hold up on real hardware, where scattered lookups into a million-entry table are a poor fit for large matrix multiplies.

Sources:

1. Mixture of A Million Experts — Xu Owen He, 2024

http://arxiv.org/abs/2407.04153

2. Scaling Laws for Fine-Grained Mixture of Experts — Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, Marek Cygan, Sebastian Jaszczur, 2024

https://scholar.google.com/scholar?q=Scaling+Laws+for+Fine-Grained+Mixture+of+Experts

3. Unified Scaling Laws for Routed Language Models — Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew Johnson, Katie Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Jack Rae, Erich Elsen, Koray Kavukcuoglu, Karen Simonyan, 2022

https://scholar.google.com/scholar?q=Unified+Scaling+Laws+for+Routed+Language+Models

4. Mixture of A Million Experts (PEER) — Xu Owen He, 2024

https://scholar.google.com/scholar?q=Mixture+of+A+Million+Experts+%28PEER%29

5. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Damai Dai, Chengqi Deng, Chenggang Zhao, R.X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, Wenfeng Liang, 2024

https://scholar.google.com/scholar?q=DeepSeekMoE%3A+Towards+Ultimate+Expert+Specialization+in+Mixture-of-Experts+Language+Models

6. Large Memory Layers with Product Keys — Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, Hervé Jégou, 2019

https://scholar.google.com/scholar?q=Large+Memory+Layers+with+Product+Keys

7. Transformer Feed-Forward Layers Are Key-Value Memories — Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy, 2021

https://scholar.google.com/scholar?q=Transformer+Feed-Forward+Layers+Are+Key-Value+Memories

8. Mixture-of-Experts with Expert Choice Routing — Yanqi Zhou et al., 2022

https://scholar.google.com/scholar?q=Mixture-of-Experts+with+Expert+Choice+Routing

9. Memory augmented language models through mixture of word experts — Cicero Nogueira dos Santos et al., 2023

https://scholar.google.com/scholar?q=Memory+augmented+language+models+through+mixture+of+word+experts

10. Fast Feedforward Networks — Peter Belcak, Roger Wattenhofer, 2023

https://scholar.google.com/scholar?q=Fast+Feedforward+Networks

11. Approximating Two-Layer Feedforward Networks for Efficient Transformers — Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber, 2023

https://scholar.google.com/scholar?q=Approximating+Two-Layer+Feedforward+Networks+for+Efficient+Transformers

12. Lifelong Language Pretraining with Distribution-Specialized Experts (Lifelong-MoE) — Wuyang Chen et al., 2023

https://scholar.google.com/scholar?q=Lifelong+Language+Pretraining+with+Distribution-Specialized+Experts+%28Lifelong-MoE%29

13. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — Noam Shazeer et al., 2017

https://scholar.google.com/scholar?q=Outrageously+Large+Neural+Networks%3A+The+Sparsely-Gated+Mixture-of-Experts+Layer

Interactive Visualization: Mixture of a Million Experts: Scaling Routing with Product Keys

This episode examines a 2024 paper arguing that mixture-of-experts models outperform dense Transformers at every compute budget, with a gap that widens at scale. That claim contradicts earlier Meta and DeepMind studies, which saw the MoE advantage shrink, and the authors blame two hidden assumptions: a fixed training duration and a fixed expert size. The discussion builds the background from dense feed-forward layers, routers, and the difference between active and total parameters, through the Kaplan and Chinchilla scaling laws. It then introduces "granularity", the ratio of dense feed-forward width to expert width, as a third scaling axis. Granularity is a design choice that was inherited by convention and never shown to be optimal. The episode also stays skeptical of the evidence. The law is fit on a ragged grid of small models (3M to 85M active parameters), yet its headline claims extrapolate to a trillion active parameters, so listeners get a clear view of both the idea and its limits.

Sources:

1. Scaling Laws for Fine-Grained Mixture of Experts — Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, Marek Cygan, Sebastian Jaszczur, 2024

http://arxiv.org/abs/2402.07871

2. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Damai Dai, Chengqi Deng, Chenggang Zhao, et al. (DeepSeek-AI), 2024

https://scholar.google.com/scholar?q=DeepSeekMoE%3A+Towards+Ultimate+Expert+Specialization+in+Mixture-of-Experts+Language+Models

3. Scaling Laws for Fine-Grained Mixture of Experts — Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Sebastian Jaszczur, et al., 2024

https://scholar.google.com/scholar?q=Scaling+Laws+for+Fine-Grained+Mixture+of+Experts

4. Unified Scaling Laws for Routed Language Models — Aidan Clark, Diego de las Casas, Aurelia Guy, et al. (DeepMind), 2022

https://scholar.google.com/scholar?q=Unified+Scaling+Laws+for+Routed+Language+Models

5. Mixture of A Million Experts — Xu Owen He (Google DeepMind), 2024

https://scholar.google.com/scholar?q=Mixture+of+A+Million+Experts

6. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, et al. (OpenAI), 2020

https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models

7. Training Compute-Optimal Large Language Models — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. (DeepMind), 2022

https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models

8. Scaling Laws for Sparsely-Connected Foundation Models / Scaling Laws for MoE variants (e.g., Artetxe et al., 'Efficient Large Scale Language Modeling with Mixtures of Experts') — Mikel Artetxe, Shruti Bhosale, Naman Goyal, et al. (Meta AI), 2022

https://scholar.google.com/scholar?q=Scaling+Laws+for+Sparsely-Connected+Foundation+Models+%2F+Scaling+Laws+for+MoE+variants+%28e.g.%2C+Artetxe+et+al.%2C+%27Efficient+Large+Scale+Language+Modeling+with+Mixtures+of+Experts%27%29

9. Scaling Laws for Fine-Grained Mixture of Experts (ICML 2024 version) and follow-ups on joint scaling of E, G and D — Krajewski, Ludziejewski, Adamczewski, et al., 2024

https://scholar.google.com/scholar?q=Scaling+Laws+for+Fine-Grained+Mixture+of+Experts+%28ICML+2024+version%29+and+follow-ups+on+joint+scaling+of+E%2C+G+and+D

10. Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient — Ludziejewski, Pióro, Krajewski, et al., 2025

https://scholar.google.com/scholar?q=Joint+MoE+Scaling+Laws%3A+Mixture+of+Experts+Can+Be+Memory+Efficient

11. Efficient Large Scale Language Modeling with Mixtures of Experts — Mikel Artetxe, Shruti Bhosale, et al., 2022

https://scholar.google.com/scholar?q=Efficient+Large+Scale+Language+Modeling+with+Mixtures+of+Experts

12. Training Compute-Optimal Large Language Models (Chinchilla) — Jordan Hoffmann, Sebastian Borgeaud, et al., 2022

https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models+%28Chinchilla%29

13. Mixture-of-Experts with Expert Choice Routing — Yanqi Zhou, Tao Lei, et al., 2022

https://scholar.google.com/scholar?q=Mixture-of-Experts+with+Expert+Choice+Routing

14. Mixture of A Million Experts (PEER) — Xu Owen He, 2024

https://scholar.google.com/scholar?q=Mixture+of+A+Million+Experts+%28PEER%29

15. Switch Transformers; GShard; Mixtral of Experts — Fedus, Zoph, Shazeer; Lepikhin et al.; Jiang et al., 2021-2024

https://scholar.google.com/scholar?q=Switch+Transformers%3B+GShard%3B+Mixtral+of+Experts

16. Transformer Feed-Forward Layers Are Key-Value Memories — Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy, 2021

https://scholar.google.com/scholar?q=Transformer+Feed-Forward+Layers+Are+Key-Value+Memories

17. Scaling Laws for Sparsely-Connected Foundation Models — Elias Frantar, Carlos Riquelme, et al., 2023

https://scholar.google.com/scholar?q=Scaling+Laws+for+Sparsely-Connected+Foundation+Models

18. Scaling Laws for Upcycling Mixture-of-Experts Language Models; Scaling Laws for Sparsely-Activated models at inference — Liew et al.; related inference-cost work, 2025

https://scholar.google.com/scholar?q=Scaling+Laws+for+Upcycling+Mixture-of-Experts+Language+Models%3B+Scaling+Laws+for+Sparsely-Activated+models+at+inference

Interactive Visualization: Scaling Laws for Fine-Grained Mixture of Experts

This episode examines "How Linear Attention Remembers," a preprint that asks how a single fact is written into, retained in, and read back out of the fixed-size recurrent state that replaces the KV cache. It contrasts Gated Linear Attention with Gated DeltaNet (whose delta rule overwrites old bindings instead of piling on top of them) and with hybrids that keep a few full-attention layers. The authors use associative-recall tests with causal interventions on pretrained models. They swap a head's state between near-identical "Paris" and "London" runs and measure the shift in the logit margin. Screening picks out one "hub" head per model, and the episode lays out the paper's four claims: content-specific writes, an identifiable read path, shared-state coupling that degrades recall as load grows, and attention layers taking over direct recall in hybrids. Listeners get the background on why a sum-based memory can't be pointed at the way a KV cache can, and a look at what that means for long-context serving on phones, edge devices, and robots.

Sources:

1. How Linear Attention Remembers — Kichang Lee, JaeYeon Park, Songkuk Kim, JeongGil Ko, 2026

http://arxiv.org/abs/2609.33093

2. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, Francois Fleuret, 2020

https://scholar.google.com/scholar?q=Transformers+are+RNNs%3A+Fast+Autoregressive+Transformers+with+Linear+Attention

3. Linear Transformers Are Secretly Fast Weight Programmers — Imanol Schlag, Kazuki Irie, Juergen Schmidhuber, 2021

https://scholar.google.com/scholar?q=Linear+Transformers+Are+Secretly+Fast+Weight+Programmers

4. Gated Delta Networks: Improving Mamba2 with Delta Rule — Songlin Yang, Jan Kautz, Ali Hatamizadeh, 2025

https://scholar.google.com/scholar?q=Gated+Delta+Networks%3A+Improving+Mamba2+with+Delta+Rule

5. Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff (Based) — Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, Christopher Re, 2024

https://scholar.google.com/scholar?q=Simple+Linear+Attention+Language+Models+Balance+the+Recall-Throughput+Tradeoff+%28Based%29

6. Repeat After Me: Transformers are Better than State Space Models at Copying — Samy Jelassi, David Brandfonbrener, Sham M. Kakade, Eran Malach, 2024

https://scholar.google.com/scholar?q=Repeat+After+Me%3A+Transformers+are+Better+than+State+Space+Models+at+Copying

7. Locating and Editing Factual Associations in GPT — Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov, 2022

https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+GPT

8. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small — Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, Jacob Steinhardt, 2022

https://scholar.google.com/scholar?q=Interpretability+in+the+Wild%3A+a+Circuit+for+Indirect+Object+Identification+in+GPT-2+small

9. In-context Learning and Induction Heads — Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, and others (Anthropic), 2022

https://scholar.google.com/scholar?q=In-context+Learning+and+Induction+Heads

10. The Hidden Attention of Mamba Models — Ameen Ali, Itamar Zimerman, Lior Wolf, 2024

https://scholar.google.com/scholar?q=The+Hidden+Attention+of+Mamba+Models

11. Zoology: Measuring and Improving Recall in Efficient Language Models — Simran Arora, Sabri Eyuboglu, Aman Timalsina, et al., 2023

https://scholar.google.com/scholar?q=Zoology%3A+Measuring+and+Improving+Recall+in+Efficient+Language+Models

12. Hopfield Networks is All You Need — Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, et al., 2020

https://scholar.google.com/scholar?q=Hopfield+Networks+is+All+You+Need

13. Learning to (Learn at Test Time): RNNs with Expressive Hidden States — Yu Sun, Xinhao Li, Karan Dalal, et al., 2024

https://scholar.google.com/scholar?q=Learning+to+%28Learn+at+Test+Time%29%3A+RNNs+with+Expressive+Hidden+States

14. Mamba: Linear-Time Sequence Modeling with Selective State Spaces / Transformers are SSMs (Mamba-2) — Albert Gu, Tri Dao, 2023/2024

https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces+%2F+Transformers+are+SSMs+%28Mamba-2%29

15. Locating and Editing Factual Associations in Mamba — Arnab Sen Sharma, David Atkinson, David Bau, 2024

https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+Mamba

16. Mamba-Based Language Models: An Empirical Study (Nemotron-H / An Empirical Study of Mamba-based Language Models) — Roger Waleffe, Wonmin Byeon, et al., 2024

https://scholar.google.com/scholar?q=Mamba-Based+Language+Models%3A+An+Empirical+Study+%28Nemotron-H+%2F+An+Empirical+Study+of+Mamba-based+Language+Models%29

17. Investigating the Indirect Object Identification circuit in GPT-2 small / Interpretability in the Wild — Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, Jacob Steinhardt, 2022

https://scholar.google.com/scholar?q=Investigating+the+Indirect+Object+Identification+circuit+in+GPT-2+small+%2F+Interpretability+in+the+Wild

18. Zamba / Jamba / Samba hybrid papers (e.g., Jamba: A Hybrid Transformer-Mamba Language Model) — Opher Lieber et al. (Jamba); Liliang Ren et al. (Samba), 2024

https://scholar.google.com/scholar?q=Zamba+%2F+Jamba+%2F+Samba+hybrid+papers+%28e.g.%2C+Jamba%3A+A+Hybrid+Transformer-Mamba+Language+Model%29

19. Understanding the Differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks (Test-time regression view) — Jerome Sieber, Carmen Amo Alonso, et al. / Ke Alexander Wang, Jiaxin Shi, Emily B. Fox (Test-time regression), 2024/2025

https://scholar.google.com/scholar?q=Understanding+the+Differences+in+Foundation+Models%3A+Attention%2C+State+Space+Models%2C+and+Recurrent+Neural+Networks+%28Test-time+regression+view%29

20. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods — Fred Zhang, Neel Nanda, 2024

https://scholar.google.com/scholar?q=Towards+Best+Practices+of+Activation+Patching+in+Language+Models%3A+Metrics+and+Methods

Interactive Visualization: How Linear Attention Remembers: Writing and Recalling Facts

This episode explores a Lean 4 formalization of Hopfield networks and Boltzmann machines, covering about sixteen thousand new lines on top of mathlib, which the authors say is the first for either model in any proof assistant. It walks through Hopfield energy descent and asynchronous convergence, Hebbian weights and the cross-talk term for non-orthogonal patterns, and Gibbs sampling as a random-scan Markov chain with detailed balance. It also covers the zero-temperature limit that recovers the Hopfield rule. The main finding is that formalizing convergence required every neuron to be updated infinitely often, a fairness condition the textbook theorem left implicit. The episode also points out that the Boltzmann machine formalized is fully visible, with no hidden units, so the ergodicity result covers less than the word suggests. It places the work against modern context, including the link between continuous Hopfield updates and softmax attention and dense associative memory. Listeners interested in what machine-checked proofs expose in classic results will find it useful.

Sources:

1. Formalized Hopfield Networks and Boltzmann Machines — Matteo Cipollina, Michail Karatarakis, Freek Wiedijk, 2025

http://arxiv.org/abs/2512.07766

2. A Learning Algorithm for Boltzmann Machines — David H. Ackley, Geoffrey E. Hinton, Terrence J. Sejnowski, 1985

https://scholar.google.com/scholar?q=A+Learning+Algorithm+for+Boltzmann+Machines

3. Training Products of Experts by Minimizing Contrastive Divergence — Geoffrey E. Hinton, 2002

https://scholar.google.com/scholar?q=Training+Products+of+Experts+by+Minimizing+Contrastive+Divergence

4. A Fast Learning Algorithm for Deep Belief Nets — Geoffrey E. Hinton, Simon Osindero, Yee-Whye Teh, 2006

https://scholar.google.com/scholar?q=A+Fast+Learning+Algorithm+for+Deep+Belief+Nets

5. Deep Boltzmann Machines — Ruslan Salakhutdinov, Geoffrey E. Hinton, 2009

https://scholar.google.com/scholar?q=Deep+Boltzmann+Machines

6. The PageRank Citation Ranking: Bringing Order to the Web — Lawrence Page, Sergey Brin, Rajeev Motwani, Terry Winograd, 1998

https://scholar.google.com/scholar?q=The+PageRank+Citation+Ranking%3A+Bringing+Order+to+the+Web

7. Markov Chains and Mixing Times — David A. Levin, Yuval Peres, Elizabeth L. Wilmer, 2009 (2nd ed. 2017, with Propp)

https://scholar.google.com/scholar?q=Markov+Chains+and+Mixing+Times

8. Non-negative Matrices and Markov Chains — Eugene Seneta, 1981 (2nd ed.; originally 1973)

https://scholar.google.com/scholar?q=Non-negative+Matrices+and+Markov+Chains

9. Hopfield Networks is All You Need — Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, et al. (Sepp Hochreiter's group), 2020

https://scholar.google.com/scholar?q=Hopfield+Networks+is+All+You+Need

10. Dense Associative Memory for Pattern Recognition — Dmitry Krotov, John Hopfield, 2016

https://scholar.google.com/scholar?q=Dense+Associative+Memory+for+Pattern+Recognition

11. Neural networks and physical systems with emergent collective computational abilities — John J. Hopfield, 1982

https://scholar.google.com/scholar?q=Neural+networks+and+physical+systems+with+emergent+collective+computational+abilities

12. On Contrastive Divergence Learning — Miguel Á. Carreira-Perpiñán, Geoffrey E. Hinton, 2005

https://scholar.google.com/scholar?q=On+Contrastive+Divergence+Learning

13. Stochastic Ising machines and p-bits / probabilistic computing — Kerem Y. Camsari, Brian M. Sutton, Supriyo Datta (and related work), 2017-2019

https://scholar.google.com/scholar?q=Stochastic+Ising+machines+and+p-bits+%2F+probabilistic+computing

14. Formalizing Perron–Frobenius / Perron-Frobenius Theorem for Spectral Radius Analysis — René Thiemann, Jose Divasón, et al., 2016-2019

https://scholar.google.com/scholar?q=Formalizing+Perron%E2%80%93Frobenius+%2F+Perron-Frobenius+Theorem+for+Spectral+Radius+Analysis

15. Certigrad: Practical Verification of Stochastic Computation Graphs — Daniel Selsam, Percy Liang, David Dill, 2017

https://scholar.google.com/scholar?q=Certigrad%3A+Practical+Verification+of+Stochastic+Computation+Graphs

16. Rank collapse causes over-smoothing and over-squashing / Attention is not all you need: pure attention loses rank doubly exponentially with depth — Yihe Dong, Jean-Baptiste Cordonnier, Andreas Loukas, 2021

https://scholar.google.com/scholar?q=Rank+collapse+causes+over-smoothing+and+over-squashing+%2F+Attention+is+not+all+you+need%3A+pure+attention+loses+rank+doubly+exponentially+with+depth

17. Equation of State Calculations by Fast Computing Machines / Optimization by Simulated Annealing — Metropolis et al.; Kirkpatrick, Gelatt, Vecchi, 1953 / 1983

https://scholar.google.com/scholar?q=Equation+of+State+Calculations+by+Fast+Computing+Machines+%2F+Optimization+by+Simulated+Annealing

Interactive Visualization: Formalizing Hopfield Networks and Boltzmann Machines in Lean 4

This episode examines "Verifying Peephole Rewriting In SSA Compiler IRs," which uses the Lean proof assistant to give MLIR-style compilers a push-button workflow for proving small rewrites correct, without the cost of a fully verified compiler. It explains peephole rewrites and why SSA def-use chains let a matcher catch patterns that straight-line matching misses. It then compares the approach with Alive and Alive2, which rely on SMT solvers, and with CompCert-style end-to-end verification, which takes years of expert effort and is tied to one IR. The discussion covers MLIR dialects and regions, and why a ring-aware prover helps for domains like fully homomorphic encryption, where an algebraically trivial rewrite becomes a heroic proof at the LLVM level. It also covers refinement semantics with poison and undefined behavior. The episode adds a caveat from Csmith's results: proving local rewrites correct does not cover bugs in parsing, lowering, or pass ordering. Listeners get a clear picture of where formal compiler verification can reach into new domain-specific IRs and where it stops.

Sources:

1. Verifying Peephole Rewriting In SSA Compiler IRs — Siddharth Bhat, Alex Keizer, Chris Hughes, Andrés Goens, Tobias Grosser, 2024

http://arxiv.org/abs/2407.03685

2. Peephole Optimization — William M. McKeeman, 1965

https://scholar.google.com/scholar?q=Peephole+Optimization

3. Provably Correct Peephole Optimizations with Alive — Nuno P. Lopes, David Menendez, Santosh Nagarakatte, John Regehr, 2015

https://scholar.google.com/scholar?q=Provably+Correct+Peephole+Optimizations+with+Alive

4. Alive2: Bounded Translation Validation for LLVM — Nuno P. Lopes, Juneyoung Lee, Chung-Kil Hur, Zhengyang Liu, John Regehr, 2021

https://scholar.google.com/scholar?q=Alive2%3A+Bounded+Translation+Validation+for+LLVM

5. Verified Peephole Optimizations for CompCert — Eric Mullen, Daryl Zuniga, Zachary Tatlock, Dan Grossman, 2016

https://scholar.google.com/scholar?q=Verified+Peephole+Optimizations+for+CompCert

6. MLIR: Scaling Compiler Infrastructure for Domain Specific Computation — Chris Lattner, Mehdi Amini, Uday Bondhugula, Albert Cohen, Andy Davis, Jacques Pienaar, River Riddle, Tatiana Shpeisman, Nicolas Vasilache, Oleksandr Zinenko, 2021

https://scholar.google.com/scholar?q=MLIR%3A+Scaling+Compiler+Infrastructure+for+Domain+Specific+Computation

7. Static Single Assignment Form and the Control Dependence Graph — Ron Cytron, Jeanne Ferrante, Barry K. Rosen, Mark N. Wegman, F. Kenneth Zadeck, 1991

https://scholar.google.com/scholar?q=Static+Single+Assignment+Form+and+the+Control+Dependence+Graph

8. Compiling Lean programs with MLIR / LEAN to MLIR (lz and lean-mlir) — Siddharth Bhat, Tobias Grosser and collaborators, 2022

https://scholar.google.com/scholar?q=Compiling+Lean+programs+with+MLIR+%2F+LEAN+to+MLIR+%28lz+and+lean-mlir%29

9. Verifying Peephole Rewriting In SSA Compiler IRs — Siddharth Bhat, Alex Keizer, Chris Hughes, Andrés Goens, Tobias Grosser, 2024

https://scholar.google.com/scholar?q=Verifying+Peephole+Rewriting+In+SSA+Compiler+IRs

10. Formal Verification of a Realistic Compiler — Xavier Leroy, 2009

https://scholar.google.com/scholar?q=Formal+Verification+of+a+Realistic+Compiler

11. Finding and Understanding Bugs in C Compilers — Xuejun Yang, Yang Chen, Eric Eide, John Regehr, 2011

https://scholar.google.com/scholar?q=Finding+and+Understanding+Bugs+in+C+Compilers

12. CakeML: A Verified Implementation of ML — Ramana Kumar, Magnus O. Myreen, Michael Norrish, Scott Owens, 2014

https://scholar.google.com/scholar?q=CakeML%3A+A+Verified+Implementation+of+ML

13. Formalizing the LLVM Intermediate Representation for Verified Program Transformations (Vellvm) — Jianzhou Zhao, Santosh Nagarakatte, Milo M. K. Martin, Steve Zdancewic, 2012

https://scholar.google.com/scholar?q=Formalizing+the+LLVM+Intermediate+Representation+for+Verified+Program+Transformations+%28Vellvm%29

14. Taming Undefined Behavior in LLVM — Juneyoung Lee, Yoonseung Kim, Youngju Song, Chung-Kil Hur, Sanjoy Das, David Majnemer, John Regehr, Nuno P. Lopes, 2017

https://scholar.google.com/scholar?q=Taming+Undefined+Behavior+in+LLVM

15. Bringing the WebAssembly Standard up to Speed with SpecTec / Mechanising and Verifying the WebAssembly Specification — Conrad Watt (and collaborators including Andreas Rossberg), 2018

https://scholar.google.com/scholar?q=Bringing+the+WebAssembly+Standard+up+to+Speed+with+SpecTec+%2F+Mechanising+and+Verifying+the+WebAssembly+Specification

16. Finding and Understanding Bugs in C Compilers (Csmith) — Xuejun Yang, Yang Chen, Eric Eide, John Regehr, 2011

https://scholar.google.com/scholar?q=Finding+and+Understanding+Bugs+in+C+Compilers+%28Csmith%29

17. Formal Verification of a Realistic Compiler (CompCert) — Xavier Leroy, 2009

https://scholar.google.com/scholar?q=Formal+Verification+of+a+Realistic+Compiler+%28CompCert%29

18. Vellvm: Formalizing the LLVM Intermediate Representation for Verified Program Transformations — Jianzhou Zhao, Santosh Nagarakatte, Milo M. K. Martin, Steve Zdancewic, 2012

https://scholar.google.com/scholar?q=Vellvm%3A+Formalizing+the+LLVM+Intermediate+Representation+for+Verified+Program+Transformations

19. Interaction Trees: Representing Recursive and Impure Programs in Coq — Li-yao Xia, Yannick Zakowski, Paul He, Chung-Kil Hur, Gregory Malecha, Benjamin C. Pierce, Steve Zdancewic, 2020

https://scholar.google.com/scholar?q=Interaction+Trees%3A+Representing+Recursive+and+Impure+Programs+in+Coq

20. Souper: A Synthesizing Superoptimizer — Raimondas Sasnauskas, Yang Chen, Peter Collingbourne, Jeroen Ketema, Gratian Lup, Jubi Taneja, John Regehr, 2017

https://scholar.google.com/scholar?q=Souper%3A+A+Synthesizing+Superoptimizer

21. TASO: Optimizing Deep Learning Computation with Automatic Generation of Graph Substitutions — Zhihao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, Alex Aiken, 2019

https://scholar.google.com/scholar?q=TASO%3A+Optimizing+Deep+Learning+Computation+with+Automatic+Generation+of+Graph+Substitutions

22. PET: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections — Haojie Wang, Jidong Zhai, Mingyu Gao, Zixuan Ma, et al., 2021

https://scholar.google.com/scholar?q=PET%3A+Optimizing+Tensor+Programs+with+Partially+Equivalent+Transformations+and+Automated+Corrections

23. egg: Fast and Extensible Equality Saturation — Max Willsey, Chandrakana Nandi, Yisu Remy Wang, Oliver Flatt, Zachary Tatlock, Pavel Panchekha, 2021

https://scholar.google.com/scholar?q=egg%3A+Fast+and+Extensible+Equality+Saturation

24. Verified Compilation of Neural Network / Tensor Programs (Alive-tv for MLIR) — Seongwon Bang, Seunghyeon Nam, Inwhan Chun, Ho Young Jhoo, Juneyoung Lee, 2022

https://scholar.google.com/scholar?q=Verified+Compilation+of+Neural+Network+%2F+Tensor+Programs+%28Alive-tv+for+MLIR%29

25. Learning to Superoptimize Programs / LLM-guided compiler optimization (e.g., Meta LLM Compiler) — Rudy Bunel et al.; Chris Cummins et al., 2016 / 2024

https://scholar.google.com/scholar?q=Learning+to+Superoptimize+Programs+%2F+LLM-guided+compiler+optimization+%28e.g.%2C+Meta+LLM+Compiler%29

Interactive Visualization: Verifying Peephole Rewrites in SSA Compiler IRs with Lean