This episode explores IMPRESS, a systems paper from Zhejiang University and Huawei Cloud researchers presented at USENIX FAST 2025, which tackles a specific bottleneck in large language model inference: the time delay before a model produces its first response token when cached context has spilled onto disk. The discussion traces how modern LLM applications—retrieval-augmented generation, multi-turn chatbots, and plugin frameworks—prepend large chunks of context that inflate prefill costs superlinearly, with one cited example showing a 2,600-token plugin prompt stretching time-to-first-token by nine times on OPT-30B. It covers how prior work like vLLM's PagedAttention and AttentionStore addressed prefix KV caching but hit a wall once caches outgrew GPU and CPU memory and moved to slower disk storage, where I/O latency can consume up to 98 percent of total delay. The conversation traces IMPRESS's key insight: repurposing an importance-scoring technique from H2O—originally used to decide what to evict during decoding—to instead decide what's worth loading from disk before prefill even starts, claiming up to 2.8x lower latency with comparable accuracy. Listeners interested in the practical engineering tradeoffs behind making large-context LLM applications faster and cheaper to run will find the discussion's grounding in measured attention patterns, rather than benchmark tweaking, particularly compelling.

Sources:

1. IMPRESS: A Multi-Tier KV Storage System for Faster LLM Prefill

https://www.usenix.org/system/files/fast25-chen-weijian-impress.pdf

2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica (UC Berkeley), 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen, 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models

4. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention (AttentionStore) — Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, and colleagues (industry/academic collaboration), 2024

https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention+%28AttentionStore%29

5. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu (Moonshot AI, Tsinghua University), 2024

https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving

6. AttentionStore: Cost-Effective Attention Reuse across Multi-Turn Conversations in Large Language Model Serving — Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, Pengfei Zuo, 2024

https://scholar.google.com/scholar?q=AttentionStore%3A+Cost-Effective+Attention+Reuse+across+Multi-Turn+Conversations+in+Large+Language+Model+Serving

7. Retrieval Head Mechanistically Explains Long-Context Factuality — Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, Yao Fu, 2024

https://scholar.google.com/scholar?q=Retrieval+Head+Mechanistically+Explains+Long-Context+Factuality

8. RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation — Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, Xin Jin, 2024

https://scholar.google.com/scholar?q=RAGCache%3A+Efficient+Knowledge+Caching+for+Retrieval-Augmented+Generation

Interactive Visualization: IMPRESS: A Multi-Tier KV Storage System for Faster LLM Prefill

Fast State Restoration in LLM Serving with HCache tackles a hidden cost of running LLM chat services: when GPU memory pressure forces eviction of a conversation's KV cache, restoring that state currently means either recomputing it from scratch (20-26x slower than no restoration) or streaming the full cache back from storage over PCIe (6.5-13x slower). Drawing on traces from ShareGPT4 and L-Eval, the discussion lays out why eviction is the common case rather than an edge case — a single A100-40GB holds only enough KV cache for a handful of live conversations at once. The episode walks through the researchers' proposed middle path: caching the hidden state (one layer upstream of the key/value projection) instead of the KV cache itself, then reconstructing K and V on demand via a cheap matrix multiplication. It's a systems paper grounded in first-principles reasoning about transformer architecture before any benchmarks are run, making the case for why this approach should be faster on theoretical grounds alone. Listeners interested in the practical engineering trade-offs behind serving long, multi-turn LLM conversations at scale will find the framing of "recompute vs. offload vs. something smaller in between" a clear lens on a problem most users never realize is happening.

Sources:

1. Fast State Restoration in LLM Serving with HCache — Shiwei Gao, Youmin Chen, Jiwu Shu, 2024

http://arxiv.org/abs/2410.05004

2. Training Deep Nets with Sublinear Memory Cost — Tianqi Chen, Bin Xu, Chiyuan Zhang, Carlos Guestrin, 2016

https://scholar.google.com/scholar?q=Training+Deep+Nets+with+Sublinear+Memory+Cost

3. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al., 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

4. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, et al. (Moonshot AI and Tsinghua University), 2024/2025

https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving

5. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu, Junchen Jiang, et al. (University of Chicago), 2024

https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving

6. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, et al., 2025

https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion

7. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Ying Sheng, et al., 2024

https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs

8. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention — Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, Pengfei Zuo, 2024 (USENIX ATC)

https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention

9. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, Sumit Sanghai, 2023 (EMNLP)

https://scholar.google.com/scholar?q=GQA%3A+Training+Generalized+Multi-Query+Transformer+Models+from+Multi-Head+Checkpoints

10. Prompt Cache: Modular Attention Reuse for Low-Latency Inference — In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, Lin Zhong, 2024 (MLSys)

https://scholar.google.com/scholar?q=Prompt+Cache%3A+Modular+Attention+Reuse+for+Low-Latency+Inference

11. Efficiently Programming Large Language Models using SGLang — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2023/2024

https://scholar.google.com/scholar?q=Efficiently+Programming+Large+Language+Models+using+SGLang

12. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, Ce Zhang, 2023 (ICML)

https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU

Interactive Visualization: Fast State Restoration for Evicted LLM KV Caches

This episode examines "Compute Or Load KV Cache? Why not Both?" (Jin, Liu, et al., University of Michigan), which introduces Cake, a scheduling system for LLM inference. The discussion traces the KV cache problem from its roots in the attention mechanism through the rise of prefix caching in production systems like OpenAI, Anthropic, and DeepSeek, then explains why loading a cached prefix isn't automatically cheap — most cache hits land on slow disk tiers rather than fast GPU memory. The core insight covered is that compute cost per chunk rises across a sequence while I/O cost stays flat, which lets Cake run a "meet in the middle" two-pointer scheduler that computes early chunks on GPU while simultaneously loading later chunks from storage. Listeners interested in LLM serving infrastructure will find concrete numbers throughout — like the 30-second prefill tax on a 72,000-token input — that ground the tradeoff between recomputation and I/O in real system constraints rather than abstract theory.

Sources:

1. Compute Or Load KV Cache? Why Not Both? — Shuowei Jin, Xueshen Liu, Qingzhao Zhang, Z. Morley Mao, 2024

http://arxiv.org/abs/2410.03065

2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Qin, R. et al. (Moonshot AI / Kimi), 2024

https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving

3. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI, 2024

https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model

4. CacheBlend: Fast Large Language Model Serving with Cached Knowledge Fusion — Yao, J., Li, H., Liu, Y., et al., 2024

https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+with+Cached+Knowledge+Fusion

5. Preble: Efficient Distributed Prompt Scheduling for LLM Serving — Srivatsa, V. et al., 2024

https://scholar.google.com/scholar?q=Preble%3A+Efficient+Distributed+Prompt+Scheduling+for+LLM+Serving

Interactive Visualization: Compute or Load? Cake's Smart KV Cache Scheduler

This episode explores "Post-Training Science for Supervised Fine-Tuning," a study from Baseten researchers that treats SFT hyperparameter choices — learning rate, batch size, LoRA rank, epochs, and optimizer — as empirical questions rather than inherited folklore. Using controlled, one-variable-at-a-time sweeps across Qwen3 and Llama models ranging from 0.6 billion to 235 billion parameters (including dense and mixture-of-experts architectures), the hosts unpack findings like a surprisingly stable optimal LoRA learning rate that holds flat across two orders of magnitude in model scale, and a batch size that behaves more like a compute-cost tradeoff than a quality lever. They dig into how LoRA stacks up against full fine-tuning, with LoRA recovering a median 98% of full fine-tuning's gains using a fraction of the trainable parameters, and discuss where increasing LoRA rank stops paying off. Along the way, they flag a methodological wrinkle worth scrutinizing: the same evaluator used to construct the training data is also used to judge the fine-tuned model's output quality. Listeners interested in practical, evidence-based guidance for production fine-tuning — rather than another one-off trick — will find concrete, scale-tested defaults here.

Sources:

1. Post-Training Science: Scaling Laws for SFT and LoRA

https://www.datocms-assets.com/104802/1781805778-baseten-research-sft.pdf

2. LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen (Microsoft Research), 2021 (ICLR 2022)

https://scholar.google.com/scholar?q=LoRA%3A+Low-Rank+Adaptation+of+Large+Language+Models

3. QLoRA: Efficient Finetuning of Quantized LLMs — Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer (University of Washington), 2023 (NeurIPS 2023)

https://scholar.google.com/scholar?q=QLoRA%3A+Efficient+Finetuning+of+Quantized+LLMs

4. LIMA: Less Is More for Alignment — Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy (Meta AI / external collaborators), 2023 (NeurIPS 2023)

https://scholar.google.com/scholar?q=LIMA%3A+Less+Is+More+for+Alignment

5. Muon: An optimizer for hidden layers in neural networks / Muon is Scalable for LLM Training — Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, Jeremy Bernstein (original Muon, 2024); Moonshot AI / Kimi team, led by Jingyuan Liu and Jianlin Su, et al. (scaling follow-up, 2025), 2024-2025

https://scholar.google.com/scholar?q=Muon%3A+An+optimizer+for+hidden+layers+in+neural+networks+%2F+Muon+is+Scalable+for+LLM+Training

6. Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy Without Increasing Inference Time — Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, et al., 2022

https://scholar.google.com/scholar?q=Model+Soups%3A+Averaging+Weights+of+Multiple+Fine-Tuned+Models+Improves+Accuracy+Without+Increasing+Inference+Time

7. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning — Armen Aghajanyan, Luke Zettlemoyer, Sonal Gupta, 2020

https://scholar.google.com/scholar?q=Intrinsic+Dimensionality+Explains+the+Effectiveness+of+Language+Model+Fine-Tuning

8. S-LoRA: Serving Thousands of Concurrent LoRA Adapters — Ying Sheng, Shiyi Cao, Dacheng Li, et al., 2023

https://scholar.google.com/scholar?q=S-LoRA%3A+Serving+Thousands+of+Concurrent+LoRA+Adapters

Interactive Visualization: Post-Training Science: Scaling Laws for SFT and LoRA

This episode explores CONTXT, a training-free method for correcting distribution shift by adding a single precomputed "context vector" directly into a model's internal activations—no fine-tuning, no paired prompts, and no gradient updates required. The discussion traces the paper's neuroscience grounding in dual-process theory, where the hippocampus rapidly encodes context and hands it to the prefrontal cortex to amplify relevant features and suppress irrelevant ones, and examines how faithfully that analogy maps onto a simple additive vector operation. It also clarifies the distinction between domain generalization (no access to target data at all) and test-time adaptation (unlabeled target data available at inference), situating CONTXT within existing activation-steering approaches like those requiring token-level paired prompts. Listeners get a walkthrough of the core math—h plus alpha times an index vector, extendable to multiple stacked contexts for simultaneous edits like adjusting tone while removing sarcasm—before the hosts turn to concrete demonstrations, including a striking out-of-distribution image classification example. The conversation is notable for its skepticism: one host pushes back hard on whether a two-region brain theory can really license a one-line vector subtraction, making this as much a critique of steering-paper rigor as an explainer of the method itself.

Sources:

1. Context is All You Need — Jean Erik Delanois, Shruti Joshi, Ryan Golden, Teresa Nick, Maxim Bazhenov, 2026

http://arxiv.org/abs/2604.04364

2. In Search of Lost Domain Generalization — Ishaan Gulrajani, David Lopez-Paz, 2020

https://scholar.google.com/scholar?q=In+Search+of+Lost+Domain+Generalization

3. Deep CORAL: Correlation Alignment for Deep Domain Adaptation — Baochen Sun, Kate Saenko, 2016

https://scholar.google.com/scholar?q=Deep+CORAL%3A+Correlation+Alignment+for+Deep+Domain+Adaptation

4. Domain-Adversarial Training of Neural Networks — Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, Victor Lempitsky, 2016

https://scholar.google.com/scholar?q=Domain-Adversarial+Training+of+Neural+Networks

5. Deeper, Broader and Artier Domain Generalization (the PACS dataset) — Da Li, Yongxin Yang, Yi-Zhe Song, Timothy M. Hospedales, 2017

https://scholar.google.com/scholar?q=Deeper%2C+Broader+and+Artier+Domain+Generalization+%28the+PACS+dataset%29

6. Tent: Fully Test-Time Adaptation by Entropy Minimization — Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, Trevor Darrell, 2021

https://scholar.google.com/scholar?q=Tent%3A+Fully+Test-Time+Adaptation+by+Entropy+Minimization

7. Improving Robustness Against Common Corruptions by Covariate Shift Adaptation — Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, Matthias Bethge, 2020

https://scholar.google.com/scholar?q=Improving+Robustness+Against+Common+Corruptions+by+Covariate+Shift+Adaptation

8. Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation — Jian Liang, Dapeng Hu, Jiashi Feng, 2020

https://scholar.google.com/scholar?q=Do+We+Really+Need+to+Access+the+Source+Data%3F+Source+Hypothesis+Transfer+for+Unsupervised+Domain+Adaptation

9. MEMO: Test Time Robustness via Adaptation and Augmentation — Marvin Zhang, Sergey Levine, Chelsea Finn, 2022

https://scholar.google.com/scholar?q=MEMO%3A+Test+Time+Robustness+via+Adaptation+and+Augmentation

10. Steering Llama 2 via Contrastive Activation Addition — Panickssery, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., Turner, A. M., 2023

https://scholar.google.com/scholar?q=Steering+Llama+2+via+Contrastive+Activation+Addition

11. Representation Engineering: A Top-Down Approach to AI Transparency — Zou, A., Gao, L., Greenblatt, R., et al., 2023

https://scholar.google.com/scholar?q=Representation+Engineering%3A+A+Top-Down+Approach+to+AI+Transparency

12. Extracting Latent Steering Vectors from Pretrained Language Models — Subramani, N., Suresh, N., Peters, M. E., 2022

https://scholar.google.com/scholar?q=Extracting+Latent+Steering+Vectors+from+Pretrained+Language+Models

13. Distributionally Robust Neural Networks for Group Shifts (GroupDRO) — Sagawa, S., Koh, P. W., Hashimoto, T. B., Liang, P., 2020

https://scholar.google.com/scholar?q=Distributionally+Robust+Neural+Networks+for+Group+Shifts+%28GroupDRO%29

Interactive Visualization: Context is All You Need: Fixing OOD Drift Without Retraining

This episode explores EventTensor, a compiler abstraction from Carnegie Mellon and collaborators (presented at MLSys 2026) that treats synchronization events as first-class tensors for compiling GPU megakernels. The discussion covers how encoding true data dependencies—rather than waiting for entire kernels to finish—enables fine-grained scheduling, illustrated through a split-K summation example and a symbolic batch-size template that avoids recompilation when shapes change. A key focus is how the system handles Mixture-of-Experts routing, where dependencies aren't known until runtime, via data-dependent event counters and task triggering computed from router outputs. The hosts also unpack the tradeoffs between static and dynamic scheduling, showing that static wins on predictable dense workloads while dynamic pays off only under genuine irregularity like MoE. Benchmark results show up to 1.40x speedups over cuBLAS+NCCL, 1.23x over Triton/FlashInfer on MoE layers, and end-to-end gains of 1.48x over vLLM, making this a concrete look at how compile-time and runtime scheduling can be unified without sacrificing performance.

Sources:

1. Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel — Hongyi Jin, Bohan Hou, Guanjie Wang, Ruihang Lai, Jinqi Chen, Zihao Ye, Yaxing Cai, Yixin Dong, Xinhao Cheng, Zhihao Zhang, Yilong Zhao, Yingyi Huang, Lijie Yang, Jinchen Jiang, Gabriele Oliaro, Jianan Ji, Xupeng Miao, Vinod Grover, Todd C. Mowry, Zhihao Jia, Tianqi Chen, 2026

http://arxiv.org/abs/2604.13327

2. Legion: Expressing Locality and Independence with Logical Regions — Michael Bauer, Sean Treichler, Elliott Slaughter, Alex Aiken, 2012

https://scholar.google.com/scholar?q=Legion%3A+Expressing+Locality+and+Independence+with+Logical+Regions

3. StarPU: A Unified Platform for Task Scheduling on Heterogeneous Multicore Architectures — Cédric Augonnet, Samuel Thibault, Raymond Namyst, Pierre-André Wacrenier, 2011

https://scholar.google.com/scholar?q=StarPU%3A+A+Unified+Platform+for+Task+Scheduling+on+Heterogeneous+Multicore+Architectures

4. Dynamic Control Flow in Large-Scale Machine Learning — Yuan Yu, Martín Abadi, Paul Barham, Eugene Brevdo, Mike Burrows, Andy Davis, Jeff Dean, Sanjay Ghemawat, Tim Harley, Peter Hawkins, Mark Hong, Rajat Monga, Derek Murray, Xiaoqiang Zheng, and others (Google Brain), 2018

https://scholar.google.com/scholar?q=Dynamic+Control+Flow+in+Large-Scale+Machine+Learning

5. Ray: A Distributed Framework for Emerging AI Applications — Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, Ion Stoica, 2018

https://scholar.google.com/scholar?q=Ray%3A+A+Distributed+Framework+for+Emerging+AI+Applications

6. Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programs — Cheng, X., Zhang, Z., Zhou, Y., Ji, J., Jiang, J., Zhao, Z., et al. (overlapping author list with this paper), 2025

https://scholar.google.com/scholar?q=Mirage+Persistent+Kernel%3A+A+Compiler+and+Runtime+for+Mega-Kernelizing+Tensor+Programs

7. Look ma, no bubbles! Designing a low-latency megakernel for Llama-1B — Spector, B., Juravsky, J., Sul, S., Dugan, O., Lim, D., Fu, D., Arora, S., R, C., 2025

https://scholar.google.com/scholar?q=Look+ma%2C+no+bubbles%21+Designing+a+low-latency+megakernel+for+Llama-1B

8. A Framework for Fine-Grained Synchronization of Dependent GPU Kernels (CuSync) — Jangda, A., Maleki, S., Dehnavi, M. M., Musuvathi, M., Saarikivi, O., 2024

https://scholar.google.com/scholar?q=A+Framework+for+Fine-Grained+Synchronization+of+Dependent+GPU+Kernels+%28CuSync%29

9. Graphene: An IR for Optimized Tensor Computations on GPUs — Hagedorn, B., Fan, B., Chen, H., Cecka, C., Garland, M., Grover, V., 2023

https://scholar.google.com/scholar?q=Graphene%3A+An+IR+for+Optimized+Tensor+Computations+on+GPUs

10. FlashMoE: Fast Distributed MoE in a Single Kernel — Aimuyo, O. J., Oh, B., Singh, R., 2025

https://scholar.google.com/scholar?q=FlashMoE%3A+Fast+Distributed+MoE+in+a+Single+Kernel

Interactive Visualization: Prompt Boundary-Aware Scheduling with Event Tensors for Dynamic Kernels

This episode examines the "Reversal Curse," a 2023 finding from researchers at Vanderbilt, UK AI Safety Institute, Apollo Research, NYU, Sussex, and Oxford showing that autoregressive language models trained on "A is B" statements fail to infer "B is A," even though the two are logically equivalent. Using the example of Valentina Tereshkova, the discussion shows how a model finetuned on "Tereshkova was the first woman in space" answers forward questions perfectly but performs at chance when the question is reversed, and extends this to real-world GPT-4 results where "who is Tom Cruise's mother" scores 79% accuracy versus just 33% for the reverse query about Mary Lee Pfeiffer. The hosts unpack why this happens mechanically — next-token prediction bakes facts into weights as one-directional associations rather than symmetric relations like a knowledge-graph edge — and contrast this with in-context learning, where the same reversal works flawlessly, pinpointing the failure specifically to generalization from gradient-based training rather than a reasoning limitation. A debate over whether this is just a data-coverage gap versus a deeper meta-learning failure leads into the researchers' attempted fix: training on facts stated in both directions to see if models can pick up the general pattern. It's a compelling listen for anyone curious about the hidden asymmetries in how LLMs actually store knowledge versus how humans intuitively reason about it.

Sources:

1. The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" — Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, Owain Evans, 2023

http://arxiv.org/abs/2309.12288

2. Locating and Editing Factual Associations in GPT (ROME) — Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov, 2022

https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+GPT+%28ROME%29

3. Language Models as Knowledge Bases? — Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, Sebastian Riedel, 2019

https://scholar.google.com/scholar?q=Language+Models+as+Knowledge+Bases%3F

4. Reverse Training to Nurse the Reversal Curse — Olga Golovneva, Zeyuan Allen-Zhu, Jason Weston, Sainbayar Sukhbaatar (Meta AI), 2024

https://scholar.google.com/scholar?q=Reverse+Training+to+Nurse+the+Reversal+Curse

5. Physics of Language Models: Part 3.2, Knowledge Manipulation — Zeyuan Allen-Zhu, Yuanzhi Li, 2024

https://scholar.google.com/scholar?q=Physics+of+Language+Models%3A+Part+3.2%2C+Knowledge+Manipulation

6. Studying Large Language Model Generalization with Influence Functions — Roger Grosse, Juhan Bae, Cem Anil, et al., 2023

https://scholar.google.com/scholar?q=Studying+Large+Language+Model+Generalization+with+Influence+Functions

7. Transformer Feed-Forward Layers Are Key-Value Memories — Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy, 2021

https://scholar.google.com/scholar?q=Transformer+Feed-Forward+Layers+Are+Key-Value+Memories

8. Large Language Models Struggle to Learn Long-Tail Knowledge — Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, Colin Raffel, 2023

https://scholar.google.com/scholar?q=Large+Language+Models+Struggle+to+Learn+Long-Tail+Knowledge

9. Taken Out of Context: On Measuring Situational Awareness in LLMs — Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, et al., 2023

https://scholar.google.com/scholar?q=Taken+Out+of+Context%3A+On+Measuring+Situational+Awareness+in+LLMs

Interactive Visualization: The Reversal Curse: When A Is B But Not B Is A

This episode explores Patchscopes, a unifying framework from Ghandeharioun et al. (Google Research and Tel Aviv University, ICML 2024) for inspecting hidden representations of language models. Rather than decoding internal states through narrow tools like probing classifiers, logit lens, tuned lens, or activation patching, Patchscopes patches a hidden representation from a source prompt directly into a separate target prompt designed to elicit a plain-language explanation of what it holds. The hosts detail how this single mechanism — defined by prompt, layer, position, and an optional transform — subsumes existing interpretability methods as special-case configurations, including logit lens, tuned lens, causal tracing, and attention knockout. They emphasize that Patchscopes remains strictly read-only inspection, distinct from knowledge editing that alters model weights, and discuss how it overcomes the closed-vocabulary and early-layer failure modes that limit earlier techniques. The conversation makes a compelling case that many interpretability tools researchers already use are really the same underlying operation with different settings, offering listeners a clearer theoretical map of the field.

Sources:

1. Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models — Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, Mor Geva, 2024

http://arxiv.org/abs/2401.06102

2. Locating and Editing Factual Associations in GPT — Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov, 2022

https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+GPT

3. Investigating Gender Bias in Language Models Using Causal Mediation Analysis — Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, Stuart Shieber, 2020

https://scholar.google.com/scholar?q=Investigating+Gender+Bias+in+Language+Models+Using+Causal+Mediation+Analysis

4. Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small — Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, Jacob Steinhardt, 2023

https://scholar.google.com/scholar?q=Interpretability+in+the+Wild%3A+A+Circuit+for+Indirect+Object+Identification+in+GPT-2+Small

5. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods — Fred Zhang, Neel Nanda, 2024

https://scholar.google.com/scholar?q=Towards+Best+Practices+of+Activation+Patching+in+Language+Models%3A+Metrics+and+Methods

6. interpreting GPT: the logit lens — nostalgebraist, 2020

https://scholar.google.com/scholar?q=interpreting+GPT%3A+the+logit+lens

7. Eliciting Latent Predictions from Transformers with the Tuned Lens — Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, Jacob Steinhardt, 2023

https://scholar.google.com/scholar?q=Eliciting+Latent+Predictions+from+Transformers+with+the+Tuned+Lens

8. Analyzing Transformers in Embedding Space — Guy Dar, Mor Geva, Ankit Gupta, Jonathan Berant, 2023

https://scholar.google.com/scholar?q=Analyzing+Transformers+in+Embedding+Space

9. Jump to Conclusions: Short-Cutting Transformers With Linear Transformations — Alexander Yom Din, Taelin Karidi, Leshem Choshen, Mor Geva, 2023

https://scholar.google.com/scholar?q=Jump+to+Conclusions%3A+Short-Cutting+Transformers+With+Linear+Transformations

10. Locating and Editing Factual Associations in GPT (ROME) — Meng, Bau, Andonian, Belinkov, 2022

https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+GPT+%28ROME%29

11. Linearity of Relation Decoding in Transformer Language Models (LRE) — Hernandez, Sharma, Haklay, Meng, Wattenberg, Andreas, Belinkov, Bau, 2023

https://scholar.google.com/scholar?q=Linearity+of+Relation+Decoding+in+Transformer+Language+Models+%28LRE%29

12. Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models — Hase, Bansal, Kim, Ghandeharioun, 2023

https://scholar.google.com/scholar?q=Does+Localization+Inform+Editing%3F+Surprising+Differences+in+Causality-Based+Localization+vs.+Knowledge+Editing+in+Language+Models

13. The Expressive Power of Transformers with Chain of Thought — Merrill, Sabharwal, 2024

https://scholar.google.com/scholar?q=The+Expressive+Power+of+Transformers+with+Chain+of+Thought

14. Understanding and Patching Compositional Reasoning in LLMs — Li, Jiang, Xie, Song, Lian, Wei, 2024

https://scholar.google.com/scholar?q=Understanding+and+Patching+Compositional+Reasoning+in+LLMs

Interactive Visualization: How Patchscopes Reveals What Language Models Really Think

This episode traces the "Knowing-Using Gap" — the puzzling phenomenon where fine-tuning a language model on a new fact produces instant, perfect recall of that fact in isolation, yet the model fails when asked to actually reason with it in multi-step tasks. Drawing on a July 2026 arXiv paper from HKUST researchers, the discussion covers how the authors use a novel "self-patching" technique — building on ROME's causal tracing and PatchScope — to trace exactly where a memorized fact sits inside a model's layers and why it's inaccessible to reasoning circuits. The central finding is the "knowledge-circuit misalignment hypothesis": the fact isn't missing from the model at all, it's simply stored in the wrong layers — filed in storage/recall circuits rather than the mid-layer circuits that handle chaining and intersection reasoning. The conversation situates this within the broader landscape of knowledge injection methods (RAG, model editing like ROME/MEMIT, and fine-tuning) and prior benchmarks like MQuAKE and RippleEdits that documented the same failure without explaining it. Listeners interested in interpretability, LLM training dynamics, or why fine-tuned knowledge often doesn't "stick" for reasoning will find the mechanistic account — and the paper's numbers on how much of that lost reasoning is actually recoverable — a compelling departure from purely behavioral benchmarking.

Sources:

1. Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning — Lu Dai, Ziyang Rao, Yili Wang, Hanqing Wang, Hao Liu, Hui Xiong, 2026

http://arxiv.org/abs/2607.08393

2. Locating and Editing Factual Associations in GPT — Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov, 2022

https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+GPT

3. MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions — Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, Danqi Chen, 2023

https://scholar.google.com/scholar?q=MQuAKE%3A+Assessing+Knowledge+Editing+in+Language+Models+via+Multi-Hop+Questions

4. Do Large Language Models Latently Perform Multi-Hop Reasoning? — Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, Sebastian Riedel, 2024

https://scholar.google.com/scholar?q=Do+Large+Language+Models+Latently+Perform+Multi-Hop+Reasoning%3F

5. Progress Measures for Grokking via Mechanistic Interpretability — Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, Jacob Steinhardt, 2023

https://scholar.google.com/scholar?q=Progress+Measures+for+Grokking+via+Mechanistic+Interpretability

6. Physics of Language Models: Part 3.2, Knowledge Manipulation — Zeyuan Allen-Zhu, Yuanzhi Li, 2023

https://scholar.google.com/scholar?q=Physics+of+Language+Models%3A+Part+3.2%2C+Knowledge+Manipulation

7. Hopping too late: Exploring the limitations of large language models on multi-hop queries — Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, Amir Globerson, 2024

https://scholar.google.com/scholar?q=Hopping+too+late%3A+Exploring+the+limitations+of+large+language+models+on+multi-hop+queries

8. Cake: Circuit-aware editing enables generalizable knowledge learners — Yunzhi Yao, Jizhan Fang, Jia-Chen Gu, Ningyu Zhang, Shumin Deng, Huajun Chen, Nanyun Peng, 2025

https://scholar.google.com/scholar?q=Cake%3A+Circuit-aware+editing+enables+generalizable+knowledge+learners

9. Model editing at scale leads to gradual and catastrophic forgetting — Akshat Gupta, Anurag Rao, Gopala Anumanchipalli, 2024

https://scholar.google.com/scholar?q=Model+editing+at+scale+leads+to+gradual+and+catastrophic+forgetting

Interactive Visualization: The First Fact That Wouldn't Reason

This episode examines a solo-authored arXiv paper by Charles O'Neill (Baseten), "Can a Language Model Learn Facts Continually in Its Weights?", which asks whether facts written into a model's weights via LoRA adapters remain usable after dozens or even a hundred subsequent training updates, rather than simply measuring whether accuracy holds up. The discussion traces the theoretical lineage behind the question, from McCloskey and Cohen's 1989 catastrophic forgetting findings through the reversal curse and Gekhman et al.'s 2024 work showing fine-tuned facts fail at paraphrase and multi-hop reasoning even when the same fact works fine when placed directly in a prompt. To isolate what a written fact actually retains, O'Neill invents fictional entities and facts, writes them into Qwen3-4B via per-fact LoRA adapters, and tests recall, paraphrase, application, composition, and counterfactual reasoning against two benchmarks: an untouched base model and a prompt-injected ceiling. A lenient-versus-strict grading scheme introduces the "entailment gap," a metric for how often a model merely restates a trained premise instead of producing the actual answer. Listeners interested in knowledge editing, continual learning, or the mechanics of what it really means for a model to "know" something will find the paper's more rigorous framing of memory durability a useful corrective to accuracy-only benchmarks in this space.

Sources:

1. Can a Language Model Learn Facts Continually in Its Weights? — Charles O'Neill, 2026

http://arxiv.org/abs/2607.11020

2. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem — Michael McCloskey, Neal J. Cohen, 1989

https://scholar.google.com/scholar?q=Catastrophic+Interference+in+Connectionist+Networks%3A+The+Sequential+Learning+Problem

3. Overcoming Catastrophic Forgetting in Neural Networks — James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, et al. (DeepMind), 2017

https://scholar.google.com/scholar?q=Overcoming+Catastrophic+Forgetting+in+Neural+Networks

4. An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks — Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, Yoshua Bengio, 2013

https://scholar.google.com/scholar?q=An+Empirical+Investigation+of+Catastrophic+Forgetting+in+Gradient-Based+Neural+Networks

5. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning — Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, Yue Zhang, 2023

https://scholar.google.com/scholar?q=An+Empirical+Study+of+Catastrophic+Forgetting+in+Large+Language+Models+During+Continual+Fine-tuning

6. Locating and Editing Factual Associations in GPT — Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov, 2022

https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+GPT

7. Mass-Editing Memory in a Transformer — Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, David Bau, 2023

https://scholar.google.com/scholar?q=Mass-Editing+Memory+in+a+Transformer

8. Fast Model Editing at Scale — Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, Christopher D. Manning, 2022

https://scholar.google.com/scholar?q=Fast+Model+Editing+at+Scale

9. MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions — Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, Danqi Chen, 2023

https://scholar.google.com/scholar?q=MQuAKE%3A+Assessing+Knowledge+Editing+in+Language+Models+via+Multi-Hop+Questions

10. RL's razor: Why online reinforcement learning forgets less — Idan Shenfeld, Jyothish Pari, Pulkit Agrawal, 2025

https://scholar.google.com/scholar?q=RL%27s+razor%3A+Why+online+reinforcement+learning+forgets+less

11. Does localization inform editing? Surprising differences in causality-based localization vs. knowledge editing — Peter Hase, Mohit Bansal, Been Kim, Asma Ghandeharioun, 2023

https://scholar.google.com/scholar?q=Does+localization+inform+editing%3F+Surprising+differences+in+causality-based+localization+vs.+knowledge+editing

12. Towards mechanistically understanding why memorized knowledge fails to generalize in large language model finetuning — Lu Dai, Ziyang Rao, Yili Wang, Hanqing Wang, Hao Liu, Hui Xiong, 2026

https://scholar.google.com/scholar?q=Towards+mechanistically+understanding+why+memorized+knowledge+fails+to+generalize+in+large+language+model+finetuning

13. Model editing at scale leads to gradual and catastrophic forgetting — Akshat Gupta, Anurag Rao, Gopala Anumanchipalli, 2024

https://scholar.google.com/scholar?q=Model+editing+at+scale+leads+to+gradual+and+catastrophic+forgetting

14. AI models collapse when trained on recursively generated data — Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, Yarin Gal, 2024

https://scholar.google.com/scholar?q=AI+models+collapse+when+trained+on+recursively+generated+data

15. LoRA vs full fine-tuning: An illusion of equivalence — Reece Shuttleworth, Jacob Andreas, Antonio Torralba, Pratyusha Sharma, 2025

https://scholar.google.com/scholar?q=LoRA+vs+full+fine-tuning%3A+An+illusion+of+equivalence

16. Availability versus accessibility of information in memory for words — Endel Tulving, Zena Pearlstone, 1966

https://scholar.google.com/scholar?q=Availability+versus+accessibility+of+information+in+memory+for+words

Interactive Visualization: Language Model Continual Learning: Do Written Facts Survive Repeated Weight Updates?