Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- The Illusion of State in State-Space ModelsThis episode examines "The Illusion of State in State-Space Models," which asks whether S4 and Mamba, despite their recurrent form, can track state any better than transformers. It frames state tracking as the monoid word problem, from a single parity switch up to the non-commutative group S5 (five people swapping coins). Barrington's theorem makes S5 complete for NC1, while constant-depth transformers and SSMs both land in TC0. The argument rests on the unproven conjecture that TC0 ≠ NC1, and it assumes log-precision numbers and worst-case inputs. The discussion explains why linear, nonlinearity-free recurrences can be computed with a parallel scan, and why that same property caps their expressive power. It then walks through the proof that S4's fixed matrix powers and Mamba's input-dependent diagonal matrices both stay within TC0. Listeners get a clear account of why "has a recurrent form" does not imply "has recurrent power."Sources:1. The Illusion of State in State-Space Models — William Merrill, Jackson Petty, Ashish Sabharwal, 2024http://arxiv.org/abs/2404.088192. Transformers Learn Shortcuts to Automata — Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, Cyril Zhang, 2023https://scholar.google.com/scholar?q=Transformers+Learn+Shortcuts+to+Automata3. Bounded-width polynomial-size branching programs recognize exactly those languages in NC1 — David A. Barrington, 1986 (conference 1985)https://scholar.google.com/scholar?q=Bounded-width+polynomial-size+branching+programs+recognize+exactly+those+languages+in+NC14. The Parallelism Tradeoff: Limitations of Log-Precision Transformers — William Merrill, Ashish Sabharwal, 2023https://scholar.google.com/scholar?q=The+Parallelism+Tradeoff%3A+Limitations+of+Log-Precision+Transformers5. The Illusion of State in State-Space Models — William Merrill, Jackson Petty, Ashish Sabharwal, 2024https://scholar.google.com/scholar?q=The+Illusion+of+State+in+State-Space+Models6. The Expressive Power of Transformers with Chain of Thought — William Merrill, Ashish Sabharwal, 2024https://scholar.google.com/scholar?q=The+Expressive+Power+of+Transformers+with+Chain+of+Thought7. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces8. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (Mamba-2) — Tri Dao, Albert Gu, 2024https://scholar.google.com/scholar?q=Transformers+are+SSMs%3A+Generalized+Models+and+Efficient+Algorithms+Through+Structured+State+Space+Duality+%28Mamba-2%299. Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues — Riccardo Grazzi, Julien Siems, Arber Zela, Jörg K.H. Franke, Frank Hutter, Massimiliano Pontil, 2024https://scholar.google.com/scholar?q=Unlocking+State-Tracking+in+Linear+RNNs+Through+Negative+Eigenvalues10. Combining Recurrent, Convolutional, and Continuous-time Models with Linear State Space Layers (LSSL) — Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, Christopher Ré, 2021https://scholar.google.com/scholar?q=Combining+Recurrent%2C+Convolutional%2C+and+Continuous-time+Models+with+Linear+State+Space+Layers+%28LSSL%2911. Liquid Structural State-Space Models — Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, Daniela Rus, 2023https://scholar.google.com/scholar?q=Liquid+Structural+State-Space+Models12. Chess as a Testbed for Language Model State Tracking — Shubham Toshniwal, Sam Wiseman, Karen Livescu, Kevin Gimpel, 2021https://scholar.google.com/scholar?q=Chess+as+a+Testbed+for+Language+Model+State+Tracking13. Entity Tracking in Language Models — Najoung Kim, Sebastian Schuster, 2023https://scholar.google.com/scholar?q=Entity+Tracking+in+Language+Models14. Parallelizing Linear Transformers with the Delta Rule over Sequence Length (DeltaNet) — Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, Yoon Kim, 2024https://scholar.google.com/scholar?q=Parallelizing+Linear+Transformers+with+the+Delta+Rule+over+Sequence+Length+%28DeltaNet%2915. The Expressive Capacity of State Space Models: A Formal Language Perspective — Yash Sarrof, Yana Veitsman, Michael Hahn, 2024https://scholar.google.com/scholar?q=The+Expressive+Capacity+of+State+Space+Models%3A+A+Formal+Language+Perspective16. The Illusion of Thought / Transformers, parallel computation, and logarithmic depth — Clayton Sanford, Daniel Hsu, Matus Telgarsky, 2024https://scholar.google.com/scholar?q=The+Illusion+of+Thought+%2F+Transformers%2C+parallel+computation%2C+and+logarithmic+depth17. Barrington's theorem: Bounded-width polynomial-size branching programs recognize exactly those languages in NC1 — David A. Barrington, 1989https://scholar.google.com/scholar?q=Barrington%27s+theorem%3A+Bounded-width+polynomial-size+branching+programs+recognize+exactly+those+languages+in+NC1Interactive Visualization: The Illusion of State in State-Space Models0 min
- Cartridges++: Compressed KV Memory Without Off-Context DerailmentThis episode examines "Cartridges++," an Apple paper on learned, compressed KV-cache memories of a document (Cartridges) and a failure they cause on questions unrelated to that document. It explains how KV-cache compression works and compares three families: eviction methods like H2O and SnapKV, text summaries, and learned methods such as Cartridges, prefix-tuning and Attention Matching. The central argument is that a Cartridge's training objective only rewards matching the teacher on document-related queries and says nothing about unrelated ones. So the memory that best reproduces the document can derail the model elsewhere, for example by quoting a stored policy document when asked what Saturn is made of. The discussion separates context fidelity from capability preservation and flags interference, meaning fabricated quotes from the document. It argues the fair baseline is the full uncompressed document pasted as text, which already distracts models somewhat. Listeners get a clear account of why scoring a compressed memory on document questions alone hides a real cost, and why the problem looks like catastrophic forgetting applied to cache entries.Sources:1. Cartridges++: KV Cache Compression without Off-Context Derailment — Sonia Laguna, Joao Monteiro, Marco Cuturi, Pierre Ablin, Eleonora Gualdoni, 2026http://arxiv.org/abs/2609.356212. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, Christopher Ré, 2025https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study3. Overcoming catastrophic forgetting in neural networks — James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, Raia Hadsell, 2017https://scholar.google.com/scholar?q=Overcoming+catastrophic+forgetting+in+neural+networks4. LoRA Learns Less and Forgets Less — Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, and colleagues (Databricks / Columbia), 2024https://scholar.google.com/scholar?q=LoRA+Learns+Less+and+Forgets+Less5. Large Language Models Can Be Easily Distracted by Irrelevant Context — Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, Denny Zhou, 2023https://scholar.google.com/scholar?q=Large+Language+Models+Can+Be+Easily+Distracted+by+Irrelevant+Context6. Fast KV Compaction via Attention Matching — Adam Zweiger et al., 2026https://scholar.google.com/scholar?q=Fast+KV+Compaction+via+Attention+Matching7. Cartridges at Scale — Hardalov et al., 2026https://scholar.google.com/scholar?q=Cartridges+at+Scale8. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Xiang Lisa Li, Percy Liang, 2021https://scholar.google.com/scholar?q=Prefix-Tuning%3A+Optimizing+Continuous+Prompts+for+Generation9. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao et al., 2024https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks10. Conformal Risk Control — Anastasios Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, Tal Schuster, 2024https://scholar.google.com/scholar?q=Conformal+Risk+Control11. Lost in the Middle / Large Language Models Can Be Easily Distracted by Irrelevant Context — Freda Shi et al. (distraction); Nelson Liu et al. (lost in the middle), 2023https://scholar.google.com/scholar?q=Lost+in+the+Middle+%2F+Large+Language+Models+Can+Be+Easily+Distracted+by+Irrelevant+Context12. Training plug-and-play knowledge modules with deep context distillation — Lucas Caccia, Alan Ansell, Edoardo Ponti, Ivan Vulić, Alessandro Sordoni, 2025https://scholar.google.com/scholar?q=Training+plug-and-play+knowledge+modules+with+deep+context+distillation13. Doc-to-LoRA: Learning to instantly internalize contexts — Rujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, Robert Tjarko Lange, 2026https://scholar.google.com/scholar?q=Doc-to-LoRA%3A+Learning+to+instantly+internalize+contexts14. The pitfalls of KV cache compression — Alex Chen, Renato Geh, Aditya Grover, Guy Van den Broeck, Daniel Israel, 2026https://scholar.google.com/scholar?q=The+pitfalls+of+KV+cache+compression15. KV-Distill: Nearly lossless learnable context compression for LLMs — Vivek Chari, Guanghui Qin, Benjamin Van Durme, 2025https://scholar.google.com/scholar?q=KV-Distill%3A+Nearly+lossless+learnable+context+compression+for+LLMsInteractive Visualization: Cartridges++: Compressed KV Memory Without Off-Context Derailment16 min
- Agentic Coding in the Wild: GitHub Copilot's Cache Behavior at ScaleThis episode examines a production-scale characterization of GitHub Copilot's coding agent, built on sampled client-side telemetry from one week in June 2026 (13.5 million sessions and 760 million LLM calls, though some totals are inconsistent). It argues that agent traffic is session-shaped rather than stateless chat. About 87% of LLM calls are agent-initiated, the median call carries roughly 68,000 prompt tokens to produce about 247 output tokens, and the human appears only at turn boundaries. The discussion explains how prefix caching should give very high reuse inside a turn, and why the cache is lost after a pause, a model switch, or context compaction. It also separates "container idle" from "KV-cache idle", which suggests turn boundaries are the natural point to reclaim resources. The episode places the work alongside prior agentic KV-retention research and serverless idle-time studies. Listeners get a clear framework for why request-level scheduling and plain LRU eviction fall short for agent workloads, along with caveats about inferring server behavior from client data alone.Sources:1. Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale — Banruo Liu, Haoran Qiu, Íñigo Goiri, Rodrigo Fonseca, Ricardo Bianchini, Esha Choukse, 2026http://arxiv.org/abs/2608.00101v12. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan, 2023 (ICLR 2024)https://scholar.google.com/scholar?q=SWE-bench%3A+Can+Language+Models+Resolve+Real-World+GitHub+Issues%3F3. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, et al. (Microsoft), 2023 (ISCA 2024)https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting4. BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems — Yuxin Wang, Yuhan Chen, Zeyu Li, et al., 2024https://scholar.google.com/scholar?q=BurstGPT%3A+A+Real-world+Workload+Dataset+to+Optimize+LLM+Serving+Systems5. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheng Li, Weiran He, et al. (Moonshot AI), 2024 (FAST 2025 best paper)https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving6. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems7. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023 (TACL 2024)https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts8. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu (Microsoft), 2023 (EMNLP)https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models9. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al., 2023 (SOSP)https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention10. Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider — Mohammad Shahrad, Rodrigo Fonseca, Íñigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, Ricardo Bianchini, 2020 (USENIX ATC)https://scholar.google.com/scholar?q=Serverless+in+the+Wild%3A+Characterizing+and+Optimizing+the+Serverless+Workload+at+a+Large+Cloud+Provider11. Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live — Hanchen Li, Qiuyang Mang, Runyuan He, Qizheng Zhang, Huanzhi Mao, et al., 2025https://scholar.google.com/scholar?q=Continuum%3A+Efficient+and+Robust+Multi-Turn+LLM+Agent+Scheduling+with+KV+Cache+Time-to-Live12. FaaSCache: Keeping Serverless Computing Alive with Greedy-Dual Caching — Alexander Fuerst, Prateek Sharma, 2021 (ASPLOS)https://scholar.google.com/scholar?q=FaaSCache%3A+Keeping+Serverless+Computing+Alive+with+Greedy-Dual+Caching13. Efficient Agentic LLM Inference with KV Cache Retention (the 'CacheTTL'/Continuum line of work, ref [19]) — Hanchen Li, Azalia Mirhoseini, Ion Stoica, 2025https://scholar.google.com/scholar?q=Efficient+Agentic+LLM+Inference+with+KV+Cache+Retention+%28the+%27CacheTTL%27%2FContinuum+line+of+work%2C+ref+%5B19%5D%2914. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention — Bin Gao, Zhuomin He, Puru Sharma, et al., 2024https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention15. InferCept: Efficient Intercept Support for Augmented Large Language Model Inference — Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, Yiying Zhang, 2024https://scholar.google.com/scholar?q=InferCept%3A+Efficient+Intercept+Support+for+Augmented+Large+Language+Model+Inference16. Pensieve: Retention of Past Key-Values for Stateful LLM Serving (Fast and Expressive Multi-Turn Inference with Stateful Serving) — Lingfan Yu, Jinyang Li, et al., 2023/2025https://scholar.google.com/scholar?q=Pensieve%3A+Retention+of+Past+Key-Values+for+Stateful+LLM+Serving+%28Fast+and+Expressive+Multi-Turn+Inference+with+Stateful+Serving%2917. Preble: Efficient Distributed Prompt Scheduling for LLM Serving — Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, Yiying Zhang, 2024https://scholar.google.com/scholar?q=Preble%3A+Efficient+Distributed+Prompt+Scheduling+for+LLM+Serving18. H2O, SnapKV and StreamingLLM (KV eviction/compression family) — Zhang et al.; Li et al.; Xiao et al., 2023–2024https://scholar.google.com/scholar?q=H2O%2C+SnapKV+and+StreamingLLM+%28KV+eviction%2Fcompression+family%2919. Cartridges: Lightweight and general-purpose long context representations via self-study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, et al., 2025https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+general-purpose+long+context+representations+via+self-study20. TraceLab: Characterizing Coding Agent Workloads for LLM Serving, and Agentic AI Workload Characteristics — Kan Zhu et al. (arXiv:2606.30560); Yichao Yuan et al. (arXiv:2605.26297), 2026https://scholar.google.com/scholar?q=TraceLab%3A+Characterizing+Coding+Agent+Workloads+for+LLM+Serving%2C+and+Agentic+AI+Workload+Characteristics21. Marconi: Prefix Caching for the Era of Hybrid LLMs — Rui Pan, Zhuang Wang, et al., 2024/2025https://scholar.google.com/scholar?q=Marconi%3A+Prefix+Caching+for+the+Era+of+Hybrid+LLMsInteractive Visualization: Agentic Coding in the Wild: GitHub Copilot's Cache Behavior at Scale17 min
- SanSi: Looped Typed Decision Models for System 1.5 ThinkingThis episode examines SanSi, a small "typed decision" model that returns a probability for each caller-declared option (such as spam or not spam) in a single forward pass, with no generated text to parse. It explains how the model gets more computation by looping the same 24-layer stack up to eight times, with a readout after every loop, so one model can serve any budget from one loop to eight. The discussion covers how this "System 1.5" approach differs from one-pass classifiers and from generated chain-of-thought. It also covers the earlier work it builds on: Universal Transformers, adaptive computation time, and pre-trained looped models such as Ouro. On training, the model uses a frozen backbone with LoRA adapters (about 61 million trainable parameters) and a loss averaged across all loops. Supervising only the final loop dropped loop-1 accuracy to 35.6%, against 58.4% with per-loop training. The episode also raises caveats: the Kahneman-style labels are framing rather than evidence, the RL case study uses the judge's own probability as reward rather than a truly verifiable one, and it is unclear whether looping beats spending the same compute another way.Sources:1. SanSi: A Looped Typed Decision Model for System 1.5 Thinking — Shuyu Gan, Young-Jun Lee, Dongyeop Kang, 2026http://arxiv.org/abs/2610.077302. SanSi: A Looped Typed Decision Model for System 1.5 Thinking — Shuyu Gan, Young-Jun Lee, Dongyeop Kang, 2026https://scholar.google.com/scholar?q=SanSi%3A+A+Looped+Typed+Decision+Model+for+System+1.5+Thinking3. Strictly Proper Scoring Rules, Prediction, and Estimation — Tilmann Gneiting, Adrian E. Raftery, 2007https://scholar.google.com/scholar?q=Strictly+Proper+Scoring+Rules%2C+Prediction%2C+and+Estimation4. On Calibration of Modern Neural Networks — Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger, 2017https://scholar.google.com/scholar?q=On+Calibration+of+Modern+Neural+Networks5. Generative Verifiers: Reward Modeling as Next-Token Prediction — Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, Rishabh Agarwal, 2024https://scholar.google.com/scholar?q=Generative+Verifiers%3A+Reward+Modeling+as+Next-Token+Prediction6. Universal Transformers — Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Łukasz Kaiser, 2019https://scholar.google.com/scholar?q=Universal+Transformers7. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Tom Goldstein, 2025https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach8. Scaling Latent Reasoning via Looped Language Models (Ouro) — Rui-Jie Zhu et al., 2025https://scholar.google.com/scholar?q=Scaling+Latent+Reasoning+via+Looped+Language+Models+%28Ouro%299. Adaptive Computation Time for Recurrent Neural Networks — Alex Graves, 2016https://scholar.google.com/scholar?q=Adaptive+Computation+Time+for+Recurrent+Neural+Networks10. Multi-Scale Dense Networks for Resource Efficient Image Classification — Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, Kilian Q. Weinberger, 2018https://scholar.google.com/scholar?q=Multi-Scale+Dense+Networks+for+Resource+Efficient+Image+Classification11. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (Huginn) — Jonas Geiping et al., 2025https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach+%28Huginn%2912. PonderNet: Learning to Ponder — Andrea Banino, Jan Balaguer, Charles Blundell, 2021https://scholar.google.com/scholar?q=PonderNet%3A+Learning+to+Ponder13. Shallow-Deep Networks: Understanding and Mitigating Network Overthinking — Yigitcan Kaya, Sanghyun Hong, Tudor Dumitras, 2019https://scholar.google.com/scholar?q=Shallow-Deep+Networks%3A+Understanding+and+Mitigating+Network+Overthinking14. End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking — Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, Tom Goldstein, 2022https://scholar.google.com/scholar?q=End-to-end+Algorithm+Synthesis+with+Recurrent+Networks%3A+Logical+Extrapolation+Without+Overthinking15. Proper Scoring Rules / Strictly Proper Scoring Rules, Prediction, and Estimation — Tilmann Gneiting, Adrian Raftery, 2007https://scholar.google.com/scholar?q=Proper+Scoring+Rules+%2F+Strictly+Proper+Scoring+Rules%2C+Prediction%2C+and+Estimation16. Large Language Models Can Self-Improve / TTRL: Test-Time Reinforcement Learning — Yuxin Zuo et al. (TTRL); Jiaxin Huang et al., 2025https://scholar.google.com/scholar?q=Large+Language+Models+Can+Self-Improve+%2F+TTRL%3A+Test-Time+Reinforcement+Learning17. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Lianmin Zheng et al., 2023https://scholar.google.com/scholar?q=Judging+LLM-as-a-Judge+with+MT-Bench+and+Chatbot+Arena18. Chain-of-Thought Reasoning Without Prompting / Faithfulness work on CoT (Turpin et al., Language Models Don't Always Say What They Think) — Miles Turpin et al., 2023https://scholar.google.com/scholar?q=Chain-of-Thought+Reasoning+Without+Prompting+%2F+Faithfulness+work+on+CoT+%28Turpin+et+al.%2C+Language+Models+Don%27t+Always+Say+What+They+Think%29Interactive Visualization: SanSi: Looped Typed Decision Models for System 1.5 Thinking17 min
- Soup-of-Experts: Pretraining Specialists via Parameter AveragingThis episode examines "Soup-of-Experts," an Apple paper on pretraining one bank of parameters that can be averaged into a small specialist model for any data mixture, with no retraining. It sets up the background: clustered importance sampling over 4096 RedPajama-v2 clusters, model soups from fine-tuning, per-domain expert baselines, and the hypernetwork framing. It also notes closely related prior work, Branch-Train-Merge and c-BTM, that the paper doesn't compare against. The discussion then moves to the mechanism: shared parameters plus 128 experts, combined by a small MLP that maps a domain histogram to mixing coefficients. Listeners get a clear look at the amortization argument, which is paying one large pretraining cost so each later specialist is nearly free. They also get a skeptical read on whether the headline comparison holds up.Sources:1. Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging — Pierre Ablin, Angelos Katharopoulos, Skyler Seto, David Grangier, 2025http://arxiv.org/abs/2502.018042. Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models — Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Noah A. Smith, Luke Zettlemoyer, 2022https://scholar.google.com/scholar?q=Branch-Train-Merge%3A+Embarrassingly+Parallel+Training+of+Expert+Language+Models3. Scaling Expert Language Models with Unsupervised Domain Discovery (c-BTM) — Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, Luke Zettlemoyer, 2023https://scholar.google.com/scholar?q=Scaling+Expert+Language+Models+with+Unsupervised+Domain+Discovery+%28c-BTM%294. Hypernetworks — David Ha, Andrew Dai, Quoc V. Le, 2017https://scholar.google.com/scholar?q=Hypernetworks5. HyperTuning / Hypernetwork-based task adaptation (e.g., HyperFormer, Hyper-X) — Various (Mahabadi et al.; Üstün et al.; Phang et al.), 2021-2023https://scholar.google.com/scholar?q=HyperTuning+%2F+Hypernetwork-based+task+adaptation+%28e.g.%2C+HyperFormer%2C+Hyper-X%296. Mixture of Experts Models / Soft MoE (From Sparse to Soft Mixtures of Experts) — Joan Puigcerver, Carlos Riquelme, Basil Mustafa, Neil Houlsby, 2024https://scholar.google.com/scholar?q=Mixture+of+Experts+Models+%2F+Soft+MoE+%28From+Sparse+to+Soft+Mixtures+of+Experts%297. Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training — Zexuan Zhong, Mengzhou Xia, Danqi Chen, Mike Lewis, 2024https://scholar.google.com/scholar?q=Lory%3A+Fully+Differentiable+Mixture-of-Experts+for+Autoregressive+Language+Model+Pre-training8. Sheared LLaMA / structured pruning to a target-task model — Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, Danqi Chen, 2023https://scholar.google.com/scholar?q=Sheared+LLaMA+%2F+structured+pruning+to+a+target-task+model9. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pre-training — Sang Michael Xie et al., 2023https://scholar.google.com/scholar?q=DoReMi%3A+Optimizing+Data+Mixtures+Speeds+Up+Language+Model+Pre-training10. Editing Models with Task Arithmetic / TIES-Merging / DARE — Ilharco et al.; Yadav et al.; Yu et al., 2022-2024https://scholar.google.com/scholar?q=Editing+Models+with+Task+Arithmetic+%2F+TIES-Merging+%2F+DARE11. Language Models Scale Reliably With Over-Training and on Downstream Tasks — Samir Yitzhak Gadre et al., 2024https://scholar.google.com/scholar?q=Language+Models+Scale+Reliably+With+Over-Training+and+on+Downstream+Tasks12. Model Merging in LLMs, MLLMs, and Beyond / Merging pretrained models survey — Various, 2024-2025, 2024https://scholar.google.com/scholar?q=Model+Merging+in+LLMs%2C+MLLMs%2C+and+Beyond+%2F+Merging+pretrained+models+surveyInteractive Visualization: Soup-of-Experts: Pretraining Specialists via Parameter Averaging16 min
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory EfficientThis episode explores the paper "Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient," which asks whether a mixture-of-experts model can beat a dense model on loss when both train with the same compute and are capped at the same total parameter count. It covers how Switch-style top-1 routing works with experts the same size as the dense feed-forward layer. It also covers the difference between active and total parameters, using a 1.1 billion total, 426 million active, four-expert example. The discussion explains how a smaller active size lets an MoE see more tokens for the same FLOPs. That ties into deliberate overtraining under Chinchilla-style tradeoffs, and into a memory definition that includes the KV cache. The hosts also place the work alongside related studies on sparsity, inference-aware scaling and hardware efficiency. They flag the gap between a loss-based claim and real serving hardware as the question to keep in mind.Sources:1. Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficienthttps://arxiv.org/pdf/2502.051722. Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models — Samira Abnar, Harshay Shah, Dan Busbridge, et al., 2025https://scholar.google.com/scholar?q=Parameters+vs+FLOPs%3A+Scaling+Laws+for+Optimal+Sparsity+for+Mixture-of-Experts+Language+Models3. Toward Inference-Optimal Mixture-of-Expert Large Language Models — Longfei Yun, Yonghao Zhuang, Yao Fu, Eric P. Xing, Hao Zhang, 2024https://scholar.google.com/scholar?q=Toward+Inference-Optimal+Mixture-of-Expert+Large+Language+Models4. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws — Nikhil Sardana, Jacob Portes, Sasha Doubov, Jonathan Frankle, 2024https://scholar.google.com/scholar?q=Beyond+Chinchilla-Optimal%3A+Accounting+for+Inference+in+Language+Model+Scaling+Laws5. Unified Scaling Laws for Routed Language Models — Aidan Clark et al., 2022https://scholar.google.com/scholar?q=Unified+Scaling+Laws+for+Routed+Language+Models6. Scaling Laws for Fine-Grained Mixture of Experts — Jan Ludziejewski et al., 2024https://scholar.google.com/scholar?q=Scaling+Laws+for+Fine-Grained+Mixture+of+Experts7. Reconciling Kaplan and Chinchilla Scaling Laws / Resolving Discrepancies in Compute-Optimal Scaling of Language Models — Tomer Porian et al.; Tim Pearce & Jinyeop Song, 2024-2025https://scholar.google.com/scholar?q=Reconciling+Kaplan+and+Chinchilla+Scaling+Laws+%2F+Resolving+Discrepancies+in+Compute-Optimal+Scaling+of+Language+Models8. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations — Alexander Hägele et al., 2024https://scholar.google.com/scholar?q=Scaling+Laws+and+Compute-Optimal+Training+Beyond+Fixed+Training+Durations9. OLMoE: Open Mixture-of-Experts Language Models — Niklas Muennighoff et al., 2024https://scholar.google.com/scholar?q=OLMoE%3A+Open+Mixture-of-Experts+Language+Models10. Fast Inference of Mixture-of-Experts Language Models with Offloading / expert-offloading systems work (e.g., Mixtral offloading, MoE-Infinity) — Eliseev & Mazur; Xue et al., 2023-2024https://scholar.google.com/scholar?q=Fast+Inference+of+Mixture-of-Experts+Language+Models+with+Offloading+%2F+expert-offloading+systems+work+%28e.g.%2C+Mixtral+offloading%2C+MoE-Infinity%2911. GQA: Training Generalized Multi-Query Transformer Models / DeepSeek-V2 (MLA) — Ainslie et al.; DeepSeek-AI, 2023-2024https://scholar.google.com/scholar?q=GQA%3A+Training+Generalized+Multi-Query+Transformer+Models+%2F+DeepSeek-V2+%28MLA%29Interactive Visualization: Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient16 min
- Hydragen: Shared-Prefix Attention for High-Throughput LLM InferenceThis episode examines Hydragen, a technique for speeding up LLM inference when many requests share a long prompt prefix, and asks how much of its claimed 32x end-to-end throughput gain comes from the idea and how much from the baseline. It explains why decode attention is memory-bound: each sequence runs a matrix-vector product against its own KV cache, which leaves tensor cores idle and makes batching unable to help. The central argument is that vLLM's PagedAttention deduplicates the storage of shared prefixes but not the reads, so every sequence still streams the full prefix from memory. Hydragen targets those reads by splitting attention into prefix and suffix parts and combining them using log-sum-exp weights. The authors present this as exact attention built on the FlashAttention family, with no approximation. The episode ends as the hosts ask why splitting the softmax is still exact.Sources:1. Hydragen: High-Throughput LLM Inference with Shared Prefixes — Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christopher Ré, Azalia Mirhoseini, 2024http://arxiv.org/abs/2402.050992. Hydragen: High-Throughput LLM Inference with Shared Prefixes — Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christopher Ré, Azalia Mirhoseini, 2024https://scholar.google.com/scholar?q=Hydragen%3A+High-Throughput+LLM+Inference+with+Shared+Prefixes3. Cascade Inference: Memory Bandwidth Efficient Shared Prefix Batch Decoding (FlashInfer blog) and FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving — Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, Luis Ceze, 2024 (blog) / 2025 (MLSys paper)https://scholar.google.com/scholar?q=Cascade+Inference%3A+Memory+Bandwidth+Efficient+Shared+Prefix+Batch+Decoding+%28FlashInfer+blog%29+and+FlashInfer%3A+Efficient+and+Customizable+Attention+Engine+for+LLM+Inference+Serving4. Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs — Ben Athiwaratkun et al. (Amazon / AWS AI Labs), 2024https://scholar.google.com/scholar?q=Bifurcated+Attention%3A+Accelerating+Massively+Parallel+Decoding+with+Shared+Prefixes+in+LLMs5. ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition — Lu Ye, Ze Tao, Yong Huang, Yang Li, 2024https://scholar.google.com/scholar?q=ChunkAttention%3A+Efficient+Self-Attention+with+Prefix-Aware+KV+Cache+and+Two-Phase+Partition6. Online normalizer calculation for softmax; Self-attention Does Not Need O(n^2) Memory; FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Maxim Milakov & Natalia Gimelshein (2018); Markus Rabe & Charles Staats (2021); Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré (2022), 2018 / 2021 / 2022https://scholar.google.com/scholar?q=Online+normalizer+calculation+for+softmax%3B+Self-attention+Does+Not+Need+O%28n%5E2%29+Memory%3B+FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness7. SGLang: Efficient Execution of Structured Language Model Programs (introduces RadixAttention) — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2023 (arXiv) / 2024 (NeurIPS)https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs+%28introduces+RadixAttention%298. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%299. DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inference — Jinwei Yao, Kaiqi Chen, Kexun Zhang, Jiaxuan You, Binhang Yuan, Zeke Wang, Tao Lin, 2024https://scholar.google.com/scholar?q=DeFT%3A+Decoding+with+Flash+Tree-attention+for+Efficient+Tree-structured+LLM+Inference10. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, Azalia Mirhoseini, 2024https://scholar.google.com/scholar?q=Large+Language+Monkeys%3A+Scaling+Inference+Compute+with+Repeated+Sampling11. SpecInfer: Accelerating Generative LLM Serving with Tree-based Speculative Inference and Verification; Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Xupeng Miao et al. (2023); Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, Tri Dao (2024), 2023 / 2024https://scholar.google.com/scholar?q=SpecInfer%3A+Accelerating+Generative+LLM+Serving+with+Tree-based+Speculative+Inference+and+Verification%3B+Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads12. Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang, Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention13. SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention) — Zheng, Yin, Xie, Huang, Sun, Yu, Cao, Kozyrakis, Stoica, Gonzalez, Barrett, Sheng, 2023/2024https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs+%28RadixAttention%2914. FlashAttention / FlashAttention-2 — Dao, Fu, Ermon, Rudra, Re; Dao, 2022/2023https://scholar.google.com/scholar?q=FlashAttention+%2F+FlashAttention-215. Online normalizer calculation for softmax; Self-attention Does Not Need O(n^2) Memory — Milakov and Gimelshein; Rabe and Staats, 2018; 2021https://scholar.google.com/scholar?q=Online+normalizer+calculation+for+softmax%3B+Self-attention+Does+Not+Need+O%28n%5E2%29+Memory16. Flash-Decoding for long-context inference — Dao, Haziza, Massa, Sizov (PyTorch/Together blog), 2023https://scholar.google.com/scholar?q=Flash-Decoding+for+long-context+inference17. Cascade Inference: Memory Bandwidth Efficient Shared Prefix Batch Decoding (FlashInfer) — Ye, Chen, Lai, et al., 2024https://scholar.google.com/scholar?q=Cascade+Inference%3A+Memory+Bandwidth+Efficient+Shared+Prefix+Batch+Decoding+%28FlashInfer%2918. Efficiently Scaling Transformer Inference — Pope, Douglas, Chowdhery, Devlin, Bradbury, Levskaya, Heek, Xiao, Agrawal, Dean, 2022https://scholar.google.com/scholar?q=Efficiently+Scaling+Transformer+Inference19. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Yao et al., 2024/2025https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion20. AlphaCode / Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — Li et al.; Brown, Juravsky, Ehrlich, et al., 2022; 2024https://scholar.google.com/scholar?q=AlphaCode+%2F+Large+Language+Monkeys%3A+Scaling+Inference+Compute+with+Repeated+SamplingInteractive Visualization: Hydragen: Shared-Prefix Attention for High-Throughput LLM Inference16 min
- DigitalCoach: Gaps in Human and AI Software CoachingThis episode examines DIGITALCOACH, a dataset of 72 human coaching sessions (22,752 dialogue turns across Excel, FL Studio, Blender, Figma and Onshape) built to test whether multimodal models can coach novices through real software rather than just automate it. It separates what coaches say (dialogue acts), how they teach (methods drawn from cognitive apprenticeship, such as explanation, diagnosis and elicitation), and how they tie guidance to the learner's screen. Human coaches orient and explain as well as direct, and about 27% of their turns carry annotations drawn on the learner's screen. Current models mostly give bare instructions like "click Extrude". The discussion also questions how far the machine-labeled distributions can be trusted, since the labels were validated on only 200 utterances. It then sets up the offline next-utterance comparison and the live study of whether novices actually learn.Sources:1. DigitalCoach: Gaps in Human and AI Software Coachinghttps://arxiv.org/pdf/2606.319802. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments — Tianbao Xie, Danyang Zhang, Jixuan Chen, et al., 2024https://scholar.google.com/scholar?q=OSWorld%3A+Benchmarking+Multimodal+Agents+for+Open-Ended+Tasks+in+Real+Computer+Environments3. Mind2Web: Towards a Generalist Agent for the Web — Xiang Deng, Yu Gu, Boyuan Zheng, et al., 2023https://scholar.google.com/scholar?q=Mind2Web%3A+Towards+a+Generalist+Agent+for+the+Web4. Principles of Mixed-Initiative User Interfaces — Eric Horvitz, 1999https://scholar.google.com/scholar?q=Principles+of+Mixed-Initiative+User+Interfaces5. Executing Instructions in Situated Collaborative Interactions — Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, Yoav Artzi, 2019https://scholar.google.com/scholar?q=Executing+Instructions+in+Situated+Collaborative+Interactions6. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering — Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, Devi Parikh, 2017https://scholar.google.com/scholar?q=Making+the+V+in+VQA+Matter%3A+Elevating+the+Role+of+Image+Understanding+in+Visual+Question+Answering7. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents — Kanzhi Cheng, Qiushi Sun, Yougang Chu, et al., 2024https://scholar.google.com/scholar?q=SeeClick%3A+Harnessing+GUI+Grounding+for+Advanced+Visual+GUI+Agents8. Visual Dialog — Abhishek Das, Satwik Kottur, Khushi Gupta, et al., 2017https://scholar.google.com/scholar?q=Visual+Dialog9. Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering — Aishwarya Agrawal, Dhruv Batra, Devi Parikh, Aniruddha Kembhavi, 2018https://scholar.google.com/scholar?q=Don%27t+Just+Assume%3B+Look+and+Answer%3A+Overcoming+Priors+for+Visual+Question+Answering10. Learning from Human Tutoring — Michelene T. H. Chi, Stephanie A. Siler, Heisawn Jeong, Takashi Yamauchi, Robert G. Hausmann, 2001https://scholar.google.com/scholar?q=Learning+from+Human+Tutoring11. The ICAP Framework: Linking Cognitive Engagement to Active Learning Outcomes — Michelene T. H. Chi, Ruth Wylie, 2014https://scholar.google.com/scholar?q=The+ICAP+Framework%3A+Linking+Cognitive+Engagement+to+Active+Learning+Outcomes12. The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems — Kurt VanLehn, 2011https://scholar.google.com/scholar?q=The+Relative+Effectiveness+of+Human+Tutoring%2C+Intelligent+Tutoring+Systems%2C+and+Other+Tutoring+Systems13. Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech — Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, et al., 2000https://scholar.google.com/scholar?q=Dialogue+Act+Modeling+for+Automatic+Tagging+and+Recognition+of+Conversational+Speech14. Generative AI Can Harm Learning — Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı, Rei Mariman, 2024https://scholar.google.com/scholar?q=Generative+AI+Can+Harm+Learning15. Evaluating Human-Language Model Interaction — Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, et al., 2022https://scholar.google.com/scholar?q=Evaluating+Human-Language+Model+Interaction16. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference — Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, et al., 2024https://scholar.google.com/scholar?q=Chatbot+Arena%3A+An+Open+Platform+for+Evaluating+LLMs+by+Human+Preference17. Studying the Effect of AI Code Generators on Supporting Novice Learners in Introductory Programming — Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, Tovi Grossman, 2023https://scholar.google.com/scholar?q=Studying+the+Effect+of+AI+Code+Generators+on+Supporting+Novice+Learners+in+Introductory+Programming18. Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems: The Relative Effectiveness of Tutoring (VanLehn's review) — Kurt VanLehn, 2011https://scholar.google.com/scholar?q=Human+Tutoring%2C+Intelligent+Tutoring+Systems%2C+and+Other+Tutoring+Systems%3A+The+Relative+Effectiveness+of+Tutoring+%28VanLehn%27s+review%2919. MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors — Jakub Macina et al., 2025https://scholar.google.com/scholar?q=MathTutorBench%3A+A+Benchmark+for+Measuring+Open-ended+Pedagogical+Capabilities+of+LLM+Tutors20. From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning — David Dinucu-Jianu, Jakub Macina, Nico Daheim, Ido Hakimi, Iryna Gurevych, Mrinmaya Sachan, 2025https://scholar.google.com/scholar?q=From+Problem-Solving+to+Teaching+Problem-Solving%3A+Aligning+LLMs+with+Pedagogy+using+Reinforcement+Learning21. Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach (LearnLM) — Irina Jurenka et al. (Google), 2024https://scholar.google.com/scholar?q=Towards+Responsible+Development+of+Generative+AI+for+Education%3A+An+Evaluation-Driven+Approach+%28LearnLM%2922. Vid2Coach: Transforming How-To Videos into Task Assistants — Mina Huh, Zihui Xue, Ujjaini Das, Kumar Ashutosh, Kristen Grauman, Amy Pavel, 2025https://scholar.google.com/scholar?q=Vid2Coach%3A+Transforming+How-To+Videos+into+Task+Assistants23. Can Multi-modal LLMs Provide Live Step-by-Step Task Guidance? — Apratim Bhattacharyya et al., 2025https://scholar.google.com/scholar?q=Can+Multi-modal+LLMs+Provide+Live+Step-by-Step+Task+Guidance%3F24. ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use — Kaixin Li et al., 2025https://scholar.google.com/scholar?q=ScreenSpot-Pro%3A+GUI+Grounding+for+Professional+High-Resolution+Computer+Use25. Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task — Nataliya Kosmyna et al., 2025https://scholar.google.com/scholar?q=Your+Brain+on+ChatGPT%3A+Accumulation+of+Cognitive+Debt+when+Using+an+AI+Assistant+for+Essay+Writing+Task26. Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation — Xinyi Lu, Xu Wang, 2024https://scholar.google.com/scholar?q=Generative+Students%3A+Using+LLM-Simulated+Student+Profiles+to+Support+Question+Item+EvaluationInteractive Visualization: DigitalCoach: Gaps in Human and AI Software Coaching17 min
- Model Soups: Averaging Fine-Tuned Weights Beats Ensembles' CostThis episode examines "Model soups," a paper showing that averaging the weights of several fine-tuned models can raise accuracy without adding any inference cost. It explains why this should be risky: the loss landscape has many basins, and averaging across a barrier breaks the network. Models fine-tuned from the same pre-trained checkpoint tend to share a basin, which is what makes the averaging work. The discussion covers the uniform soup and the greedy soup. The greedy soup adds checkpoints in order of validation accuracy and keeps one only if accuracy doesn't drop. It also compares soups against ensembles, which need k forward passes, and against stochastic weight averaging, which averages along a single training run. Listeners get a clear account of the ImageNet distribution-shift benchmarks, the oracle baseline, and an open question about whether the gains on shifted data come from soups simply pulling weights back toward the initialization.Sources:1. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time — Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt, 2022http://arxiv.org/abs/2203.054822. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time — Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt, 2022 (ICML)https://scholar.google.com/scholar?q=Model+soups%3A+averaging+weights+of+multiple+fine-tuned+models+improves+accuracy+without+increasing+inference+time3. Robust fine-tuning of zero-shot models (WiSE-FT) — Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, Ludwig Schmidt, 2021 (CVPR 2022)https://scholar.google.com/scholar?q=Robust+fine-tuning+of+zero-shot+models+%28WiSE-FT%294. Averaging Weights Leads to Wider Optima and Better Generalization (Stochastic Weight Averaging) — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018 (UAI)https://scholar.google.com/scholar?q=Averaging+Weights+Leads+to+Wider+Optima+and+Better+Generalization+%28Stochastic+Weight+Averaging%295. Merging Models with Fisher-Weighted Averaging — Michael Matena, Colin Raffel, 2022 (NeurIPS)https://scholar.google.com/scholar?q=Merging+Models+with+Fisher-Weighted+Averaging6. Do Better ImageNet Models Transfer Better? — Simon Kornblith, Jonathon Shlens, Quoc V. Le, 2019 (CVPR)https://scholar.google.com/scholar?q=Do+Better+ImageNet+Models+Transfer+Better%3F7. Learning Transferable Visual Models From Natural Language Supervision (CLIP) — Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever, 2021 (ICML)https://scholar.google.com/scholar?q=Learning+Transferable+Visual+Models+From+Natural+Language+Supervision+%28CLIP%298. Scaling Vision Transformers — Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, Lucas Beyer, 2021 (CVPR 2022)https://scholar.google.com/scholar?q=Scaling+Vision+Transformers9. What is being transferred in transfer learning? — Behnam Neyshabur, Hanie Sedghi, Chiyuan Zhang, 2020 (NeurIPS)https://scholar.google.com/scholar?q=What+is+being+transferred+in+transfer+learning%3F10. Linear Mode Connectivity and the Lottery Ticket Hypothesis — Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin, 2020 (ICML)https://scholar.google.com/scholar?q=Linear+Mode+Connectivity+and+the+Lottery+Ticket+Hypothesis11. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs — Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson, 2018 (NeurIPS)https://scholar.google.com/scholar?q=Loss+Surfaces%2C+Mode+Connectivity%2C+and+Fast+Ensembling+of+DNNs12. The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks — Rahim Entezari, Hanie Sedghi, Olga Saukh, Behnam Neyshabur, 2021 (ICLR 2022)https://scholar.google.com/scholar?q=The+Role+of+Permutation+Invariance+in+Linear+Mode+Connectivity+of+Neural+Networks13. Sharpness-Aware Minimization for Efficiently Improving Generalization — Pierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam Neyshabur, 2020 (ICLR 2021)https://scholar.google.com/scholar?q=Sharpness-Aware+Minimization+for+Efficiently+Improving+Generalization14. Averaging Weights Leads to Wider Optima and Better Generalization (SWA) — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018https://scholar.google.com/scholar?q=Averaging+Weights+Leads+to+Wider+Optima+and+Better+Generalization+%28SWA%2915. Git Re-Basin: Merging Models modulo Permutation Symmetries — Samuel K. Ainsworth, Jonathan Hayase, Siddhartha Srinivasa, 2023https://scholar.google.com/scholar?q=Git+Re-Basin%3A+Merging+Models+modulo+Permutation+Symmetries16. Editing Models with Task Arithmetic — Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, et al., 2023https://scholar.google.com/scholar?q=Editing+Models+with+Task+Arithmetic17. TIES-Merging: Resolving Interference When Merging Models — Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, Mohit Bansal, 2023https://scholar.google.com/scholar?q=TIES-Merging%3A+Resolving+Interference+When+Merging+Models18. Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization (Model Ratatouille / DiWA / Rewarded Soups line) — Alexandre Ramé, Matthieu Kirchmeyer, et al., 2022https://scholar.google.com/scholar?q=Ensemble+of+Averages%3A+Improving+Model+Selection+and+Boosting+Performance+in+Domain+Generalization+%28Model+Ratatouille+%2F+DiWA+%2F+Rewarded+Soups+line%2919. Deep Ensembles: A Loss Landscape Perspective — Stanislav Fort, Huiyi Hu, Balaji Lakshminarayanan, 2019https://scholar.google.com/scholar?q=Deep+Ensembles%3A+A+Loss+Landscape+Perspective20. LoRA: Low-Rank Adaptation of Large Language Models — Edward Hu et al., 2021https://scholar.google.com/scholar?q=LoRA%3A+Low-Rank+Adaptation+of+Large+Language+ModelsInteractive Visualization: Model Soups: Averaging Fine-Tuned Weights Beats Ensembles' Cost15 min
- Meta and AMD Co-Design MI350X for Recommendation Model TrainingThis episode examines a Meta and AMD paper claiming that AMD's MI350X can train Meta's production recommendation models at roughly the throughput of NVIDIA's B200. It explains why recommendation models are a harder training target than LLMs: terabyte-scale sharded embedding tables, irregular gathers, heavy all-to-all communication, odd-shaped matrix multiplies, and atomic-update-heavy backward passes. It compares the two chips. Peak compute and memory bandwidth are close, the MI350X has about 1.6 times the HBM capacity, and the B200 has faster scale-up interconnect. The discussion defines "co-design" as adapting the model recipe to a purchased chip, for example by dropping activation checkpointing to use the extra memory, rather than building the platform as in Meta's earlier ZionEX work. It also raises fairness concerns about the headline parity: the recipe was changed for the AMD hardware, several co-authors are AMD engineers, and the MI350X ran at 1000 watts against the B200's 750-watt cap. The episode previews the paper's analytical GEMM tile-selection model and its agentic CUDA-to-AMD kernel porting.Sources:1. Architecting for Hardware Diversity: Meta's End-to-End Co-Design for High-Performance AI Training on AMD GPUs — Haoyu Zhang, Arshad Rahman, Adnan Aziz, Ajith Prabhudev, Amin Farmahini, Ben Rhoads, Benson Ma, Brian Shi, Cen Zhao, CK Luk, Dingming Wu, Emad El-Haraty, Eric Chen, Frank Li, Gary Huang, Gefei Zuo, Haoran Zhang, Hiwot Kassa, Hongyi Zeng, Jack Taylor, Jeremy Hadidjojo, John Donati, Jonathan Sheffield, Julio Machado Silva, Karthik Ganesan, Kent Russell, KR Kishore, Lei Tian, Liu Ke, Muhammad Osama, Peng Chen, Rakesh Hemnani, Ryan Swann, Saleel Kudchadker, Shankar Yanamandram, Shobhit Kanaujia, Shucai Xiao, Srinath Bayareddy, Srinivas Vaidyanathan, Sudharssun Subramanian, Supadchaya Puangpontip, Taylor Robie, Tianhao Huang, Tyler Hart, Viswesh Sankaran, Weijun Jiang, Wenkai Du, Xiaodong Wang, Xiaohu Guo, Yao Rong, Yinbin Ma, Yue Dong, Yuzhen Huang, Yvonne Liu, Zhan Shu, Dan Rabinovitsj, Ajit Mathews, Chunqiang Tang, 2026http://arxiv.org/abs/2610.222302. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness3. Deep Learning Recommendation Model for Personalization and Recommendation Systems — Maxim Naumov, Dheevatsa Mudigere, et al. (Facebook AI), 2019https://scholar.google.com/scholar?q=Deep+Learning+Recommendation+Model+for+Personalization+and+Recommendation+Systems4. Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models — Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, et al., 2022https://scholar.google.com/scholar?q=Software-Hardware+Co-design+for+Fast+and+Scalable+Training+of+Deep+Learning+Recommendation+Models5. DeepSeek-V3 Technical Report (and the follow-up 'Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures') — DeepSeek-AI, 2024-2025https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report+%28and+the+follow-up+%27Insights+into+DeepSeek-V3%3A+Scaling+Challenges+and+Reflections+on+Hardware+for+AI+Architectures%27%296. Ansor: Generating High-Performance Tensor Programs for Deep Learning — Lianmin Zheng, Chengfan Jia, Minmin Sun, et al., 2020https://scholar.google.com/scholar?q=Ansor%3A+Generating+High-Performance+Tensor+Programs+for+Deep+Learning7. A Learned Performance Model for Tensor Processing Units — Samuel J. Kaufman, Phitchaya Mangpo Phothilimthana, Yanqi Zhou, et al., 2021https://scholar.google.com/scholar?q=A+Learned+Performance+Model+for+Tensor+Processing+Units8. Roofline: An Insightful Visual Performance Model for Multicore Architectures — Samuel Williams, Andrew Waterman, David Patterson, 2009https://scholar.google.com/scholar?q=Roofline%3A+An+Insightful+Visual+Performance+Model+for+Multicore+Architectures9. Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPU — Muhammad Osama, Duane Merrill, Cris Cecka, Michael Garland, John D. Owens, 2023https://scholar.google.com/scholar?q=Stream-K%3A+Work-centric+Parallel+Decomposition+for+Dense+Matrix-Matrix+Multiplication+on+the+GPU10. The Architectural Implications of Facebook's DNN-based Personalized Recommendation — Udit Gupta, Carole-Jean Wu, Xiaodong Wang, et al., 2020https://scholar.google.com/scholar?q=The+Architectural+Implications+of+Facebook%27s+DNN-based+Personalized+Recommendation11. Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations — Jiaqi Zhai, Lucy Liao, Xing Liu, et al., 2024https://scholar.google.com/scholar?q=Actions+Speak+Louder+than+Words%3A+Trillion-Parameter+Sequential+Transducers+for+Generative+Recommendations12. KernelBench: Can LLMs Write Efficient GPU Kernels? — Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, Azalia Mirhoseini, 2025https://scholar.google.com/scholar?q=KernelBench%3A+Can+LLMs+Write+Efficient+GPU+Kernels%3F13. The AI CUDA Engineer: Agentic CUDA Kernel Discovery, Optimization and Composition — Robert Tjarko Lange, Aaditya Prasad, Qi Sun, Maxence Faldor, Yujin Tang, David Ha, 2025https://scholar.google.com/scholar?q=The+AI+CUDA+Engineer%3A+Agentic+CUDA+Kernel+Discovery%2C+Optimization+and+Composition14. Kevin: Multi-Turn RL for Generating CUDA Kernels — Carlo Baronio, Pietro Marsella, Ben Pan, Simon Guo, Silas Alberti, 2025https://scholar.google.com/scholar?q=Kevin%3A+Multi-Turn+RL+for+Generating+CUDA+Kernels15. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations — Philippe Tillet, H. T. Kung, David Cox, 2019https://scholar.google.com/scholar?q=Triton%3A+An+Intermediate+Language+and+Compiler+for+Tiled+Neural+Network+Computations16. tritonBLAS: Triton-based Analytical Approach for GEMM Kernel Parameter Selection — Ryan Swann, Muhammad Osama, Xiaohu Guo et al., 2025https://scholar.google.com/scholar?q=tritonBLAS%3A+Triton-based+Analytical+Approach+for+GEMM+Kernel+Parameter+Selection17. KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta — Gang Liao et al., 2026https://scholar.google.com/scholar?q=KernelEvolve%3A+Scaling+Agentic+Kernel+Coding+for+Heterogeneous+AI+Accelerators+at+Meta18. Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations (HSTU) — Jiaqi Zhai et al., 2024https://scholar.google.com/scholar?q=Actions+Speak+Louder+than+Words%3A+Trillion-Parameter+Sequential+Transducers+for+Generative+Recommendations+%28HSTU%2919. Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design — Quentin Anthony et al. (Zyphra/AMD), 2025https://scholar.google.com/scholar?q=Training+Foundation+Models+on+a+Full-Stack+AMD+Platform%3A+Compute%2C+Networking%2C+and+System+Design20. FlashAttention-3: Fast and Accurate Attention with Asynchronicity and Low-precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024https://scholar.google.com/scholar?q=FlashAttention-3%3A+Fast+and+Accurate+Attention+with+Asynchronicity+and+Low-precision21. Silent Data Corruptions at Scale — Harish Dixit et al. (Meta), 2021https://scholar.google.com/scholar?q=Silent+Data+Corruptions+at+Scale22. Meta's Second Generation AI Chip: Model-Chip Co-Design and Productionization Experiences (MTIA) — Joel Coburn et al., 2025https://scholar.google.com/scholar?q=Meta%27s+Second+Generation+AI+Chip%3A+Model-Chip+Co-Design+and+Productionization+Experiences+%28MTIA%29Interactive Visualization: Meta and AMD Co-Design MI350X for Recommendation Model Training17 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.