AI Post Transformers

This episode examines "Compute Or Load KV Cache? Why not Both?" (Jin, Liu, et al., University of Michigan), which introduces Cake, a scheduling system for LLM inference. The discussion traces the KV cache problem from its roots in the attention mechanism through the rise of prefix caching in production systems like OpenAI, Anthropic, and DeepSeek, then explains why loading a cached prefix isn't automatically cheap — most cache hits land on slow disk tiers rather than fast GPU memory. The core insight covered is that compute cost per chunk rises across a sequence while I/O cost stays flat, which lets Cake run a "meet in the middle" two-pointer scheduler that computes early chunks on GPU while simultaneously loading later chunks from storage. Listeners interested in LLM serving infrastructure will find concrete numbers throughout — like the 30-second prefill tax on a 72,000-token input — that ground the tradeoff between recomputation and I/O in real system constraints rather than abstract theory.

Sources:

1. Compute Or Load KV Cache? Why Not Both? — Shuowei Jin, Xueshen Liu, Qingzhao Zhang, Z. Morley Mao, 2024

http://arxiv.org/abs/2410.03065

2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Qin, R. et al. (Moonshot AI / Kimi), 2024

https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving

3. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI, 2024

https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model

4. CacheBlend: Fast Large Language Model Serving with Cached Knowledge Fusion — Yao, J., Li, H., Liu, Y., et al., 2024

https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+with+Cached+Knowledge+Fusion

5. Preble: Efficient Distributed Prompt Scheduling for LLM Serving — Srivatsa, V. et al., 2024

https://scholar.google.com/scholar?q=Preble%3A+Efficient+Distributed+Prompt+Scheduling+for+LLM+Serving

Interactive Visualization: Compute or Load? Cake's Smart KV Cache Scheduler

This episode examines the "last mile" problem in multi-GPU scale-up fabrics: when a remote memory request arrives at a destination GPU carrying a Network Physical Address, that address means nothing locally until it's converted back to a System Physical Address through Reverse Address Translation. The discussion traces why this destination-side translation problem is genuinely new — decades of TLB optimization research has assumed the initiating processor controls the access pattern, while NVLink and UALink fabrics flip that model, forcing the receiving GPU to translate incoming requests with no warning and no control. The hosts connect this seemingly niche hardware detail to a concrete workload: Mixture-of-Experts models rely on All-to-All dispatch and gather collectives, implemented in libraries like NCCL and RCCL, that cross this translation step twice per layer across dozens of layers. To quantify the impact, the paper extends the ASTRA-sim2.0 simulator with an Omnet++ network backend to model packet-level UALink Clos topologies, generating realistic All-to-All traffic via Microsoft's MSCCLang. Listeners interested in the unglamorous plumbing beneath large-scale AI infrastructure will find a compelling case that address translation, not just compute or bandwidth, may be a hidden bottleneck for the collective communication patterns underpinning today's largest inference deployments.

Sources:

1. Amel Fatima's Reverse Address Translation in GPU Fabrics

https://arxiv.org/pdf/2604.02473

2. Trans-FW: Short Circuiting Page Table Walk in Multi-GPU Systems via Remote Forwarding — Bingyao Li, Jieming Yin, Anup Holey, Youtao Zhang, Jun Yang, Xulong Tang, 2023

https://scholar.google.com/scholar?q=Trans-FW%3A+Short+Circuiting+Page+Table+Walk+in+Multi-GPU+Systems+via+Remote+Forwarding

3. ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale — William Won, Taekyung Heo, Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, Tushar Krishna, 2023

https://scholar.google.com/scholar?q=ASTRA-sim2.0%3A+Modeling+Hierarchical+Networks+and+Disaggregated+Systems+for+Large-model+Training+at+Scale

4. MSCCLang: Microsoft Collective Communication Language — Meghan Cowan, Saeed Maleki, Madanlal Musuvathi, Olli Saarikivi, Yifan Xiong, 2023

https://scholar.google.com/scholar?q=MSCCLang%3A+Microsoft+Collective+Communication+Language

5. Tutel: Adaptive Mixture-of-Experts at Scale — Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al., 2023

https://scholar.google.com/scholar?q=Tutel%3A+Adaptive+Mixture-of-Experts+at+Scale

6. Optimizing distributed ML communication with fused computation-collective operations — Kishore Punniyamurthy, Khaled Hamidouche, Bradford M. Beckmann, 2024

https://scholar.google.com/scholar?q=Optimizing+distributed+ML+communication+with+fused+computation-collective+operations

7. TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference — Raja Gond, Nipun Kwatra, Ramachandran Ramjee, 2025

https://scholar.google.com/scholar?q=TokenWeave%3A+Efficient+Compute-Communication+Overlap+for+Distributed+LLM+Inference

Interactive Visualization: Small Collectives, Big TLB Cost: Reverse Address Translation in Scale-Up GPU Pods

This episode explores "Post-Training Science for Supervised Fine-Tuning," a study from Baseten researchers that treats SFT hyperparameter choices — learning rate, batch size, LoRA rank, epochs, and optimizer — as empirical questions rather than inherited folklore. Using controlled, one-variable-at-a-time sweeps across Qwen3 and Llama models ranging from 0.6 billion to 235 billion parameters (including dense and mixture-of-experts architectures), the hosts unpack findings like a surprisingly stable optimal LoRA learning rate that holds flat across two orders of magnitude in model scale, and a batch size that behaves more like a compute-cost tradeoff than a quality lever. They dig into how LoRA stacks up against full fine-tuning, with LoRA recovering a median 98% of full fine-tuning's gains using a fraction of the trainable parameters, and discuss where increasing LoRA rank stops paying off. Along the way, they flag a methodological wrinkle worth scrutinizing: the same evaluator used to construct the training data is also used to judge the fine-tuned model's output quality. Listeners interested in practical, evidence-based guidance for production fine-tuning — rather than another one-off trick — will find concrete, scale-tested defaults here.

Sources:

1. Post-Training Science: Scaling Laws for SFT and LoRA

https://www.datocms-assets.com/104802/1781805778-baseten-research-sft.pdf

2. LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen (Microsoft Research), 2021 (ICLR 2022)

https://scholar.google.com/scholar?q=LoRA%3A+Low-Rank+Adaptation+of+Large+Language+Models

3. QLoRA: Efficient Finetuning of Quantized LLMs — Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer (University of Washington), 2023 (NeurIPS 2023)

https://scholar.google.com/scholar?q=QLoRA%3A+Efficient+Finetuning+of+Quantized+LLMs

4. LIMA: Less Is More for Alignment — Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy (Meta AI / external collaborators), 2023 (NeurIPS 2023)

https://scholar.google.com/scholar?q=LIMA%3A+Less+Is+More+for+Alignment

5. Muon: An optimizer for hidden layers in neural networks / Muon is Scalable for LLM Training — Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, Jeremy Bernstein (original Muon, 2024); Moonshot AI / Kimi team, led by Jingyuan Liu and Jianlin Su, et al. (scaling follow-up, 2025), 2024-2025

https://scholar.google.com/scholar?q=Muon%3A+An+optimizer+for+hidden+layers+in+neural+networks+%2F+Muon+is+Scalable+for+LLM+Training

6. Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy Without Increasing Inference Time — Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, et al., 2022

https://scholar.google.com/scholar?q=Model+Soups%3A+Averaging+Weights+of+Multiple+Fine-Tuned+Models+Improves+Accuracy+Without+Increasing+Inference+Time

7. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning — Armen Aghajanyan, Luke Zettlemoyer, Sonal Gupta, 2020

https://scholar.google.com/scholar?q=Intrinsic+Dimensionality+Explains+the+Effectiveness+of+Language+Model+Fine-Tuning

8. S-LoRA: Serving Thousands of Concurrent LoRA Adapters — Ying Sheng, Shiyi Cao, Dacheng Li, et al., 2023

https://scholar.google.com/scholar?q=S-LoRA%3A+Serving+Thousands+of+Concurrent+LoRA+Adapters

Interactive Visualization: Post-Training Science: Scaling Laws for SFT and LoRA

This episode explores CONTXT, a training-free method for correcting distribution shift by adding a single precomputed "context vector" directly into a model's internal activations—no fine-tuning, no paired prompts, and no gradient updates required. The discussion traces the paper's neuroscience grounding in dual-process theory, where the hippocampus rapidly encodes context and hands it to the prefrontal cortex to amplify relevant features and suppress irrelevant ones, and examines how faithfully that analogy maps onto a simple additive vector operation. It also clarifies the distinction between domain generalization (no access to target data at all) and test-time adaptation (unlabeled target data available at inference), situating CONTXT within existing activation-steering approaches like those requiring token-level paired prompts. Listeners get a walkthrough of the core math—h plus alpha times an index vector, extendable to multiple stacked contexts for simultaneous edits like adjusting tone while removing sarcasm—before the hosts turn to concrete demonstrations, including a striking out-of-distribution image classification example. The conversation is notable for its skepticism: one host pushes back hard on whether a two-region brain theory can really license a one-line vector subtraction, making this as much a critique of steering-paper rigor as an explainer of the method itself.

Sources:

1. Context is All You Need — Jean Erik Delanois, Shruti Joshi, Ryan Golden, Teresa Nick, Maxim Bazhenov, 2026

http://arxiv.org/abs/2604.04364

2. In Search of Lost Domain Generalization — Ishaan Gulrajani, David Lopez-Paz, 2020

https://scholar.google.com/scholar?q=In+Search+of+Lost+Domain+Generalization

3. Deep CORAL: Correlation Alignment for Deep Domain Adaptation — Baochen Sun, Kate Saenko, 2016

https://scholar.google.com/scholar?q=Deep+CORAL%3A+Correlation+Alignment+for+Deep+Domain+Adaptation

4. Domain-Adversarial Training of Neural Networks — Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, Victor Lempitsky, 2016

https://scholar.google.com/scholar?q=Domain-Adversarial+Training+of+Neural+Networks

5. Deeper, Broader and Artier Domain Generalization (the PACS dataset) — Da Li, Yongxin Yang, Yi-Zhe Song, Timothy M. Hospedales, 2017

https://scholar.google.com/scholar?q=Deeper%2C+Broader+and+Artier+Domain+Generalization+%28the+PACS+dataset%29

6. Tent: Fully Test-Time Adaptation by Entropy Minimization — Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, Trevor Darrell, 2021

https://scholar.google.com/scholar?q=Tent%3A+Fully+Test-Time+Adaptation+by+Entropy+Minimization

7. Improving Robustness Against Common Corruptions by Covariate Shift Adaptation — Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, Matthias Bethge, 2020

https://scholar.google.com/scholar?q=Improving+Robustness+Against+Common+Corruptions+by+Covariate+Shift+Adaptation

8. Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation — Jian Liang, Dapeng Hu, Jiashi Feng, 2020

https://scholar.google.com/scholar?q=Do+We+Really+Need+to+Access+the+Source+Data%3F+Source+Hypothesis+Transfer+for+Unsupervised+Domain+Adaptation

9. MEMO: Test Time Robustness via Adaptation and Augmentation — Marvin Zhang, Sergey Levine, Chelsea Finn, 2022

https://scholar.google.com/scholar?q=MEMO%3A+Test+Time+Robustness+via+Adaptation+and+Augmentation

10. Steering Llama 2 via Contrastive Activation Addition — Panickssery, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., Turner, A. M., 2023

https://scholar.google.com/scholar?q=Steering+Llama+2+via+Contrastive+Activation+Addition

11. Representation Engineering: A Top-Down Approach to AI Transparency — Zou, A., Gao, L., Greenblatt, R., et al., 2023

https://scholar.google.com/scholar?q=Representation+Engineering%3A+A+Top-Down+Approach+to+AI+Transparency

12. Extracting Latent Steering Vectors from Pretrained Language Models — Subramani, N., Suresh, N., Peters, M. E., 2022

https://scholar.google.com/scholar?q=Extracting+Latent+Steering+Vectors+from+Pretrained+Language+Models

13. Distributionally Robust Neural Networks for Group Shifts (GroupDRO) — Sagawa, S., Koh, P. W., Hashimoto, T. B., Liang, P., 2020

https://scholar.google.com/scholar?q=Distributionally+Robust+Neural+Networks+for+Group+Shifts+%28GroupDRO%29

Interactive Visualization: Context is All You Need: Fixing OOD Drift Without Retraining

This episode examines the "Reversal Curse," a 2023 finding from researchers at Vanderbilt, UK AI Safety Institute, Apollo Research, NYU, Sussex, and Oxford showing that autoregressive language models trained on "A is B" statements fail to infer "B is A," even though the two are logically equivalent. Using the example of Valentina Tereshkova, the discussion shows how a model finetuned on "Tereshkova was the first woman in space" answers forward questions perfectly but performs at chance when the question is reversed, and extends this to real-world GPT-4 results where "who is Tom Cruise's mother" scores 79% accuracy versus just 33% for the reverse query about Mary Lee Pfeiffer. The hosts unpack why this happens mechanically — next-token prediction bakes facts into weights as one-directional associations rather than symmetric relations like a knowledge-graph edge — and contrast this with in-context learning, where the same reversal works flawlessly, pinpointing the failure specifically to generalization from gradient-based training rather than a reasoning limitation. A debate over whether this is just a data-coverage gap versus a deeper meta-learning failure leads into the researchers' attempted fix: training on facts stated in both directions to see if models can pick up the general pattern. It's a compelling listen for anyone curious about the hidden asymmetries in how LLMs actually store knowledge versus how humans intuitively reason about it.

Sources:

1. The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" — Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, Owain Evans, 2023

http://arxiv.org/abs/2309.12288

2. Locating and Editing Factual Associations in GPT (ROME) — Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov, 2022

https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+GPT+%28ROME%29

3. Language Models as Knowledge Bases? — Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, Sebastian Riedel, 2019

https://scholar.google.com/scholar?q=Language+Models+as+Knowledge+Bases%3F

4. Reverse Training to Nurse the Reversal Curse — Olga Golovneva, Zeyuan Allen-Zhu, Jason Weston, Sainbayar Sukhbaatar (Meta AI), 2024

https://scholar.google.com/scholar?q=Reverse+Training+to+Nurse+the+Reversal+Curse

5. Physics of Language Models: Part 3.2, Knowledge Manipulation — Zeyuan Allen-Zhu, Yuanzhi Li, 2024

https://scholar.google.com/scholar?q=Physics+of+Language+Models%3A+Part+3.2%2C+Knowledge+Manipulation

6. Studying Large Language Model Generalization with Influence Functions — Roger Grosse, Juhan Bae, Cem Anil, et al., 2023

https://scholar.google.com/scholar?q=Studying+Large+Language+Model+Generalization+with+Influence+Functions

7. Transformer Feed-Forward Layers Are Key-Value Memories — Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy, 2021

https://scholar.google.com/scholar?q=Transformer+Feed-Forward+Layers+Are+Key-Value+Memories

8. Large Language Models Struggle to Learn Long-Tail Knowledge — Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, Colin Raffel, 2023

https://scholar.google.com/scholar?q=Large+Language+Models+Struggle+to+Learn+Long-Tail+Knowledge

9. Taken Out of Context: On Measuring Situational Awareness in LLMs — Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, et al., 2023

https://scholar.google.com/scholar?q=Taken+Out+of+Context%3A+On+Measuring+Situational+Awareness+in+LLMs

Interactive Visualization: The Reversal Curse: When A Is B But Not B Is A

This episode explores EventTensor, a compiler abstraction from Carnegie Mellon and collaborators (presented at MLSys 2026) that treats synchronization events as first-class tensors for compiling GPU megakernels. The discussion covers how encoding true data dependencies—rather than waiting for entire kernels to finish—enables fine-grained scheduling, illustrated through a split-K summation example and a symbolic batch-size template that avoids recompilation when shapes change. A key focus is how the system handles Mixture-of-Experts routing, where dependencies aren't known until runtime, via data-dependent event counters and task triggering computed from router outputs. The hosts also unpack the tradeoffs between static and dynamic scheduling, showing that static wins on predictable dense workloads while dynamic pays off only under genuine irregularity like MoE. Benchmark results show up to 1.40x speedups over cuBLAS+NCCL, 1.23x over Triton/FlashInfer on MoE layers, and end-to-end gains of 1.48x over vLLM, making this a concrete look at how compile-time and runtime scheduling can be unified without sacrificing performance.

Sources:

1. Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel — Hongyi Jin, Bohan Hou, Guanjie Wang, Ruihang Lai, Jinqi Chen, Zihao Ye, Yaxing Cai, Yixin Dong, Xinhao Cheng, Zhihao Zhang, Yilong Zhao, Yingyi Huang, Lijie Yang, Jinchen Jiang, Gabriele Oliaro, Jianan Ji, Xupeng Miao, Vinod Grover, Todd C. Mowry, Zhihao Jia, Tianqi Chen, 2026

http://arxiv.org/abs/2604.13327

2. Legion: Expressing Locality and Independence with Logical Regions — Michael Bauer, Sean Treichler, Elliott Slaughter, Alex Aiken, 2012

https://scholar.google.com/scholar?q=Legion%3A+Expressing+Locality+and+Independence+with+Logical+Regions

3. StarPU: A Unified Platform for Task Scheduling on Heterogeneous Multicore Architectures — Cédric Augonnet, Samuel Thibault, Raymond Namyst, Pierre-André Wacrenier, 2011

https://scholar.google.com/scholar?q=StarPU%3A+A+Unified+Platform+for+Task+Scheduling+on+Heterogeneous+Multicore+Architectures

4. Dynamic Control Flow in Large-Scale Machine Learning — Yuan Yu, Martín Abadi, Paul Barham, Eugene Brevdo, Mike Burrows, Andy Davis, Jeff Dean, Sanjay Ghemawat, Tim Harley, Peter Hawkins, Mark Hong, Rajat Monga, Derek Murray, Xiaoqiang Zheng, and others (Google Brain), 2018

https://scholar.google.com/scholar?q=Dynamic+Control+Flow+in+Large-Scale+Machine+Learning

5. Ray: A Distributed Framework for Emerging AI Applications — Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, Ion Stoica, 2018

https://scholar.google.com/scholar?q=Ray%3A+A+Distributed+Framework+for+Emerging+AI+Applications

6. Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programs — Cheng, X., Zhang, Z., Zhou, Y., Ji, J., Jiang, J., Zhao, Z., et al. (overlapping author list with this paper), 2025

https://scholar.google.com/scholar?q=Mirage+Persistent+Kernel%3A+A+Compiler+and+Runtime+for+Mega-Kernelizing+Tensor+Programs

7. Look ma, no bubbles! Designing a low-latency megakernel for Llama-1B — Spector, B., Juravsky, J., Sul, S., Dugan, O., Lim, D., Fu, D., Arora, S., R, C., 2025

https://scholar.google.com/scholar?q=Look+ma%2C+no+bubbles%21+Designing+a+low-latency+megakernel+for+Llama-1B

8. A Framework for Fine-Grained Synchronization of Dependent GPU Kernels (CuSync) — Jangda, A., Maleki, S., Dehnavi, M. M., Musuvathi, M., Saarikivi, O., 2024

https://scholar.google.com/scholar?q=A+Framework+for+Fine-Grained+Synchronization+of+Dependent+GPU+Kernels+%28CuSync%29

9. Graphene: An IR for Optimized Tensor Computations on GPUs — Hagedorn, B., Fan, B., Chen, H., Cecka, C., Garland, M., Grover, V., 2023

https://scholar.google.com/scholar?q=Graphene%3A+An+IR+for+Optimized+Tensor+Computations+on+GPUs

10. FlashMoE: Fast Distributed MoE in a Single Kernel — Aimuyo, O. J., Oh, B., Singh, R., 2025

https://scholar.google.com/scholar?q=FlashMoE%3A+Fast+Distributed+MoE+in+a+Single+Kernel

Interactive Visualization: Prompt Boundary-Aware Scheduling with Event Tensors for Dynamic Kernels

This episode explores Patchscopes, a unifying framework from Ghandeharioun et al. (Google Research and Tel Aviv University, ICML 2024) for inspecting hidden representations of language models. Rather than decoding internal states through narrow tools like probing classifiers, logit lens, tuned lens, or activation patching, Patchscopes patches a hidden representation from a source prompt directly into a separate target prompt designed to elicit a plain-language explanation of what it holds. The hosts detail how this single mechanism — defined by prompt, layer, position, and an optional transform — subsumes existing interpretability methods as special-case configurations, including logit lens, tuned lens, causal tracing, and attention knockout. They emphasize that Patchscopes remains strictly read-only inspection, distinct from knowledge editing that alters model weights, and discuss how it overcomes the closed-vocabulary and early-layer failure modes that limit earlier techniques. The conversation makes a compelling case that many interpretability tools researchers already use are really the same underlying operation with different settings, offering listeners a clearer theoretical map of the field.

Sources:

1. Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models — Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, Mor Geva, 2024

http://arxiv.org/abs/2401.06102

2. Locating and Editing Factual Associations in GPT — Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov, 2022

https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+GPT

3. Investigating Gender Bias in Language Models Using Causal Mediation Analysis — Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, Stuart Shieber, 2020

https://scholar.google.com/scholar?q=Investigating+Gender+Bias+in+Language+Models+Using+Causal+Mediation+Analysis

4. Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small — Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, Jacob Steinhardt, 2023

https://scholar.google.com/scholar?q=Interpretability+in+the+Wild%3A+A+Circuit+for+Indirect+Object+Identification+in+GPT-2+Small

5. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods — Fred Zhang, Neel Nanda, 2024

https://scholar.google.com/scholar?q=Towards+Best+Practices+of+Activation+Patching+in+Language+Models%3A+Metrics+and+Methods

6. interpreting GPT: the logit lens — nostalgebraist, 2020

https://scholar.google.com/scholar?q=interpreting+GPT%3A+the+logit+lens

7. Eliciting Latent Predictions from Transformers with the Tuned Lens — Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, Jacob Steinhardt, 2023

https://scholar.google.com/scholar?q=Eliciting+Latent+Predictions+from+Transformers+with+the+Tuned+Lens

8. Analyzing Transformers in Embedding Space — Guy Dar, Mor Geva, Ankit Gupta, Jonathan Berant, 2023

https://scholar.google.com/scholar?q=Analyzing+Transformers+in+Embedding+Space

9. Jump to Conclusions: Short-Cutting Transformers With Linear Transformations — Alexander Yom Din, Taelin Karidi, Leshem Choshen, Mor Geva, 2023

https://scholar.google.com/scholar?q=Jump+to+Conclusions%3A+Short-Cutting+Transformers+With+Linear+Transformations

10. Locating and Editing Factual Associations in GPT (ROME) — Meng, Bau, Andonian, Belinkov, 2022

https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+GPT+%28ROME%29

11. Linearity of Relation Decoding in Transformer Language Models (LRE) — Hernandez, Sharma, Haklay, Meng, Wattenberg, Andreas, Belinkov, Bau, 2023

https://scholar.google.com/scholar?q=Linearity+of+Relation+Decoding+in+Transformer+Language+Models+%28LRE%29

12. Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models — Hase, Bansal, Kim, Ghandeharioun, 2023

https://scholar.google.com/scholar?q=Does+Localization+Inform+Editing%3F+Surprising+Differences+in+Causality-Based+Localization+vs.+Knowledge+Editing+in+Language+Models

13. The Expressive Power of Transformers with Chain of Thought — Merrill, Sabharwal, 2024

https://scholar.google.com/scholar?q=The+Expressive+Power+of+Transformers+with+Chain+of+Thought

14. Understanding and Patching Compositional Reasoning in LLMs — Li, Jiang, Xie, Song, Lian, Wei, 2024

https://scholar.google.com/scholar?q=Understanding+and+Patching+Compositional+Reasoning+in+LLMs

Interactive Visualization: How Patchscopes Reveals What Language Models Really Think

This episode traces the "Knowing-Using Gap" — the puzzling phenomenon where fine-tuning a language model on a new fact produces instant, perfect recall of that fact in isolation, yet the model fails when asked to actually reason with it in multi-step tasks. Drawing on a July 2026 arXiv paper from HKUST researchers, the discussion covers how the authors use a novel "self-patching" technique — building on ROME's causal tracing and PatchScope — to trace exactly where a memorized fact sits inside a model's layers and why it's inaccessible to reasoning circuits. The central finding is the "knowledge-circuit misalignment hypothesis": the fact isn't missing from the model at all, it's simply stored in the wrong layers — filed in storage/recall circuits rather than the mid-layer circuits that handle chaining and intersection reasoning. The conversation situates this within the broader landscape of knowledge injection methods (RAG, model editing like ROME/MEMIT, and fine-tuning) and prior benchmarks like MQuAKE and RippleEdits that documented the same failure without explaining it. Listeners interested in interpretability, LLM training dynamics, or why fine-tuned knowledge often doesn't "stick" for reasoning will find the mechanistic account — and the paper's numbers on how much of that lost reasoning is actually recoverable — a compelling departure from purely behavioral benchmarking.

Sources:

1. Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning — Lu Dai, Ziyang Rao, Yili Wang, Hanqing Wang, Hao Liu, Hui Xiong, 2026

http://arxiv.org/abs/2607.08393

2. Locating and Editing Factual Associations in GPT — Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov, 2022

https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+GPT

3. MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions — Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, Danqi Chen, 2023

https://scholar.google.com/scholar?q=MQuAKE%3A+Assessing+Knowledge+Editing+in+Language+Models+via+Multi-Hop+Questions

4. Do Large Language Models Latently Perform Multi-Hop Reasoning? — Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, Sebastian Riedel, 2024

https://scholar.google.com/scholar?q=Do+Large+Language+Models+Latently+Perform+Multi-Hop+Reasoning%3F

5. Progress Measures for Grokking via Mechanistic Interpretability — Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, Jacob Steinhardt, 2023

https://scholar.google.com/scholar?q=Progress+Measures+for+Grokking+via+Mechanistic+Interpretability

6. Physics of Language Models: Part 3.2, Knowledge Manipulation — Zeyuan Allen-Zhu, Yuanzhi Li, 2023

https://scholar.google.com/scholar?q=Physics+of+Language+Models%3A+Part+3.2%2C+Knowledge+Manipulation

7. Hopping too late: Exploring the limitations of large language models on multi-hop queries — Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, Amir Globerson, 2024

https://scholar.google.com/scholar?q=Hopping+too+late%3A+Exploring+the+limitations+of+large+language+models+on+multi-hop+queries

8. Cake: Circuit-aware editing enables generalizable knowledge learners — Yunzhi Yao, Jizhan Fang, Jia-Chen Gu, Ningyu Zhang, Shumin Deng, Huajun Chen, Nanyun Peng, 2025

https://scholar.google.com/scholar?q=Cake%3A+Circuit-aware+editing+enables+generalizable+knowledge+learners

9. Model editing at scale leads to gradual and catastrophic forgetting — Akshat Gupta, Anurag Rao, Gopala Anumanchipalli, 2024

https://scholar.google.com/scholar?q=Model+editing+at+scale+leads+to+gradual+and+catastrophic+forgetting

Interactive Visualization: The First Fact That Wouldn't Reason

This episode examines a solo-authored arXiv paper by Charles O'Neill (Baseten), "Can a Language Model Learn Facts Continually in Its Weights?", which asks whether facts written into a model's weights via LoRA adapters remain usable after dozens or even a hundred subsequent training updates, rather than simply measuring whether accuracy holds up. The discussion traces the theoretical lineage behind the question, from McCloskey and Cohen's 1989 catastrophic forgetting findings through the reversal curse and Gekhman et al.'s 2024 work showing fine-tuned facts fail at paraphrase and multi-hop reasoning even when the same fact works fine when placed directly in a prompt. To isolate what a written fact actually retains, O'Neill invents fictional entities and facts, writes them into Qwen3-4B via per-fact LoRA adapters, and tests recall, paraphrase, application, composition, and counterfactual reasoning against two benchmarks: an untouched base model and a prompt-injected ceiling. A lenient-versus-strict grading scheme introduces the "entailment gap," a metric for how often a model merely restates a trained premise instead of producing the actual answer. Listeners interested in knowledge editing, continual learning, or the mechanics of what it really means for a model to "know" something will find the paper's more rigorous framing of memory durability a useful corrective to accuracy-only benchmarks in this space.

Sources:

1. Can a Language Model Learn Facts Continually in Its Weights? — Charles O'Neill, 2026

http://arxiv.org/abs/2607.11020

2. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem — Michael McCloskey, Neal J. Cohen, 1989

https://scholar.google.com/scholar?q=Catastrophic+Interference+in+Connectionist+Networks%3A+The+Sequential+Learning+Problem

3. Overcoming Catastrophic Forgetting in Neural Networks — James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, et al. (DeepMind), 2017

https://scholar.google.com/scholar?q=Overcoming+Catastrophic+Forgetting+in+Neural+Networks

4. An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks — Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, Yoshua Bengio, 2013

https://scholar.google.com/scholar?q=An+Empirical+Investigation+of+Catastrophic+Forgetting+in+Gradient-Based+Neural+Networks

5. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning — Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, Yue Zhang, 2023

https://scholar.google.com/scholar?q=An+Empirical+Study+of+Catastrophic+Forgetting+in+Large+Language+Models+During+Continual+Fine-tuning

6. Locating and Editing Factual Associations in GPT — Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov, 2022

https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+GPT

7. Mass-Editing Memory in a Transformer — Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, David Bau, 2023

https://scholar.google.com/scholar?q=Mass-Editing+Memory+in+a+Transformer

8. Fast Model Editing at Scale — Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, Christopher D. Manning, 2022

https://scholar.google.com/scholar?q=Fast+Model+Editing+at+Scale

9. MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions — Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, Danqi Chen, 2023

https://scholar.google.com/scholar?q=MQuAKE%3A+Assessing+Knowledge+Editing+in+Language+Models+via+Multi-Hop+Questions

10. RL's razor: Why online reinforcement learning forgets less — Idan Shenfeld, Jyothish Pari, Pulkit Agrawal, 2025

https://scholar.google.com/scholar?q=RL%27s+razor%3A+Why+online+reinforcement+learning+forgets+less

11. Does localization inform editing? Surprising differences in causality-based localization vs. knowledge editing — Peter Hase, Mohit Bansal, Been Kim, Asma Ghandeharioun, 2023

https://scholar.google.com/scholar?q=Does+localization+inform+editing%3F+Surprising+differences+in+causality-based+localization+vs.+knowledge+editing

12. Towards mechanistically understanding why memorized knowledge fails to generalize in large language model finetuning — Lu Dai, Ziyang Rao, Yili Wang, Hanqing Wang, Hao Liu, Hui Xiong, 2026

https://scholar.google.com/scholar?q=Towards+mechanistically+understanding+why+memorized+knowledge+fails+to+generalize+in+large+language+model+finetuning

13. Model editing at scale leads to gradual and catastrophic forgetting — Akshat Gupta, Anurag Rao, Gopala Anumanchipalli, 2024

https://scholar.google.com/scholar?q=Model+editing+at+scale+leads+to+gradual+and+catastrophic+forgetting

14. AI models collapse when trained on recursively generated data — Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, Yarin Gal, 2024

https://scholar.google.com/scholar?q=AI+models+collapse+when+trained+on+recursively+generated+data

15. LoRA vs full fine-tuning: An illusion of equivalence — Reece Shuttleworth, Jacob Andreas, Antonio Torralba, Pratyusha Sharma, 2025

https://scholar.google.com/scholar?q=LoRA+vs+full+fine-tuning%3A+An+illusion+of+equivalence

16. Availability versus accessibility of information in memory for words — Endel Tulving, Zena Pearlstone, 1966

https://scholar.google.com/scholar?q=Availability+versus+accessibility+of+information+in+memory+for+words

Interactive Visualization: Language Model Continual Learning: Do Written Facts Survive Repeated Weight Updates?

This episode explores a technique called "Still," which compresses transformer key-value caches into a fixed-size representation in a single forward pass. The discussion walks through why KV caches balloon with long-context agents, the two-axis taxonomy of compression methods (selection versus synthesis, per-context versus amortized), and how prior work only amortized selection while synthesis remained slow. Using a Perceiver-based module descended from DeepMind's Flamingo resampler, Still cross-attends into each layer's cache and distills it down, requiring a careful workaround for RoPE position rotations so blended content from different token positions doesn't destabilize. The conversation highlights why this fills a genuine gap — manufacturing new compressed representations rather than just picking survivors — and why that matters for memory-constrained, long-horizon agent workloads. Listeners interested in LLM systems efficiency and the mechanics behind emerging cache-compression techniques will find the design-space walkthrough particularly clarifying.

Sources:

1. KV Cache Compaction Beyond Selection: Introducing Still

https://arxiv.org/pdf/2606.07878v1

2. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen, 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models

3. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks

4. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023

https://scholar.google.com/scholar?q=Learning+to+Compress+Prompts+with+Gist+Tokens

5. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu et al., 2025

https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study

6. Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks — Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, Yee Whye Teh, 2019

https://scholar.google.com/scholar?q=Set+Transformer%3A+A+Framework+for+Attention-based+Permutation-Invariant+Neural+Networks

7. Perceiver: General Perception with Iterative Attention — Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, Joao Carreira, 2021

https://scholar.google.com/scholar?q=Perceiver%3A+General+Perception+with+Iterative+Attention

8. Flamingo: a Visual Language Model for Few-Shot Learning — Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, et al., 2022

https://scholar.google.com/scholar?q=Flamingo%3A+a+Visual+Language+Model+for+Few-Shot+Learning

9. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models — Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi, 2023

https://scholar.google.com/scholar?q=BLIP-2%3A+Bootstrapping+Language-Image+Pre-training+with+Frozen+Image+Encoders+and+Large+Language+Models

10. Learned structure in cartridges: Keys as shareable routers in self-studied representations — Maurizio Diaz, 2025

https://scholar.google.com/scholar?q=Learned+structure+in+cartridges%3A+Keys+as+shareable+routers+in+self-studied+representations

11. Fast KV compaction via attention matching — Adam Zweiger, Xinghong Fu, Han Guo, Yoon Kim, 2026

https://scholar.google.com/scholar?q=Fast+KV+compaction+via+attention+matching

12. KV-Distill: Nearly lossless learnable context compression for LLMs — Vivek Chari, Guanghui Qin, Benjamin Van Durme, 2025

https://scholar.google.com/scholar?q=KV-Distill%3A+Nearly+lossless+learnable+context+compression+for+LLMs

13. Learning to evict from key-value cache (KVP) — Luca Moschella, Laura Manduchi, Ozan Sener, 2026

https://scholar.google.com/scholar?q=Learning+to+evict+from+key-value+cache+%28KVP%29

14. DeepSeek-V4: Toward highly efficient million-token context intelligence — DeepSeek-AI, 2026

https://scholar.google.com/scholar?q=DeepSeek-V4%3A+Toward+highly+efficient+million-token+context+intelligence

Interactive Visualization: KV Cache Compaction Beyond Selection: Introducing Still