Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- Hierarchical Sparse Attention for Infinite ContextThis episode explores the HiLS paper, which tackles the central long-context transformer problem: how to preserve normal-context quality while avoiding the exploding compute and KV-cache costs of dense attention at extreme sequence lengths. It explains the paper’s core mechanism of hierarchical sparse attention, where the model learns summary keys for context chunks, retrieves the most relevant chunks for each query, attends within them, and then keeps the retrieval scores in the forward pass so the chunk selector is trained directly by language-model loss. The discussion contrasts this with older sparse schemes, sliding-window attention, and positional stretching methods like YaRN, arguing that better retrieval inside attention matters as much as longer positional extrapolation or extra continued pretraining. Listeners would find it interesting because it connects the architecture details to concrete 7B OLMo 3 results on long-context benchmarks like RULER and LongBench, framing HiLS as a serious attempt to reach million-token-class context without paying full dense-attention cost.Sources:1. Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling — Xiang Hu, Xinyu Wei, Hao Gu, Minshen Zhang, Tian Liang, Huayang Li, Lei Zhu, Yan Wang, Sirui Han, Yushi Bai, Kewei Tu, Haitao Mi, Leo Liang, 2026http://arxiv.org/abs/2607.029802. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — Ofir Press, Noah A. Smith, Mike Lewis, 2021https://scholar.google.com/scholar?q=Train+Short%2C+Test+Long%3A+Attention+with+Linear+Biases+Enables+Input+Length+Extrapolation3. Extending Context Window of Large Language Models via Positional Interpolation — Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian, 2023https://scholar.google.com/scholar?q=Extending+Context+Window+of+Large+Language+Models+via+Positional+Interpolation4. YaRN: Efficient Context Window Extension of Large Language Models — Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole, 2023https://scholar.google.com/scholar?q=YaRN%3A+Efficient+Context+Window+Extension+of+Large+Language+Models5. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg, 2024https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F6. Random-Access Infinite Context Length for Transformers — Amirkeivan Mohtashami and Martin Jaggi, 2023https://scholar.google.com/scholar?q=Random-Access+Infinite+Context+Length+for+Transformers7. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan et al., 2025https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention8. Hardware-Aligned Hierarchical Sparse Attention for Efficient Long-Term Memory Access — Xiang Hu, Jiaqi Leng, Jun Zhao, Kewei Tu, and Wei Wu, 2026https://scholar.google.com/scholar?q=Hardware-Aligned+Hierarchical+Sparse+Attention+for+Efficient+Long-Term+Memory+Access9. DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention — Yuxiang Huang et al., 2026https://scholar.google.com/scholar?q=DashAttention%3A+Differentiable+and+Adaptive+Sparse+Hierarchical+Attention10. Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models — Jiaqi Leng, Xiang Hu, Junxiong Wang, Jianguo Li, Wei Wu, and Yucheng Lu, 2026https://scholar.google.com/scholar?q=Understanding+and+Improving+Length+Generalization+in+Hierarchical+Sparse+Attention+Models11. RingAttention with Blockwise Transformers for Near-Infinite Context — Hao Liu, Matei Zaharia, and Pieter Abbeel, 2024https://scholar.google.com/scholar?q=RingAttention+with+Blockwise+Transformers+for+Near-Infinite+Context12. MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling — Wenhao An et al. (MiniCPM Team), 2026https://arxiv.org/abs/2602.1176113. Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection — Dongwon Jo et al., 2026https://arxiv.org/abs/2602.0321614. MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention — Huiqiang Jiang et al., 2024https://arxiv.org/abs/2407.0249015. Long-Context Generalization with Sparse Attention — Pavlo Vasylenko, Marcos Treviso, Andre F. T. Martins, 2025https://arxiv.org/abs/2506.1664016. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference — Xiang Liu et al., 2025https://arxiv.org/abs/2502.0029917. AI Post Transformers: Long Context Pre-Training with Lighthouse Attention — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-13-long-context-pre-training-with-lighthous-e85bbe.mp318. AI Post Transformers: MiniMax Sparse Attention at Million-Token Scale — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-13-minimax-sparse-attention-at-million-toke-300108.mp319. AI Post Transformers: MiA-Signature and Global Activation for Long Context — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-13-mia-signature-and-global-activation-for-5ad62f.mp320. AI Post Transformers: AllMem for Efficient Long-Context Modeling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-12-allmem-for-efficient-long-context-modeli-7474db.mp321. AI Post Transformers: Memory Sparse Attention for 100M-Token Scaling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-07-memory-sparse-attention-for-100m-token-s-377cff.mp312 min
- Why GLU Variants Improve Transformer Feed-Forward LayersThis episode explores Noam Shazeer’s 2020 paper on replacing the Transformer’s standard feed-forward network with gated linear unit variants such as GLU, Bilinear, ReGLU, GEGLU, and SwiGLU. It explains why this seemingly small change matters, walking through the role of the per-token MLP in a Transformer and how multiplicative gating can change feature processing without altering the broader encoder-decoder architecture. The discussion focuses on the paper’s T5-style sequence-to-sequence setup, including span-corruption pretraining on C4, and on the key methodological choice to shrink gated-layer width so parameter count and FLOPs stay roughly matched with the baseline. Listeners would find it interesting because the episode connects a clean, tightly controlled ablation to a design idea that later had an outsized influence on modern Transformer architectures, while also highlighting the limits of what the experiment actually proves.Sources:1. GLU Variants Improve Transformer — Noam Shazeer, 2020http://arxiv.org/abs/2002.052022. Language Modeling with Gated Convolutional Networks — Yann N. Dauphin, Angela Fan, Michael Auli, David Grangier, 2016https://scholar.google.com/scholar?q=Language+Modeling+with+Gated+Convolutional+Networks3. GLU Variants Improve Transformer — Noam Shazeer, 2020https://scholar.google.com/scholar?q=GLU+Variants+Improve+Transformer4. PaLM: Scaling Language Modeling with Pathways — Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, et al., 2022https://scholar.google.com/scholar?q=PaLM%3A+Scaling+Language+Modeling+with+Pathways5. Gemma 2: Improving Open Language Models at a Practical Size — Gemma Team, including Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, and collaborators, 2024https://scholar.google.com/scholar?q=Gemma+2%3A+Improving+Open+Language+Models+at+a+Practical+Size6. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — Colin Raffel et al., 2019https://scholar.google.com/scholar?q=Exploring+the+Limits+of+Transfer+Learning+with+a+Unified+Text-to-Text+Transformer7. Do Transformer Modifications Transfer Across Implementations and Applications? — Sharan Narang et al., 2021https://scholar.google.com/scholar?q=Do+Transformer+Modifications+Transfer+Across+Implementations+and+Applications%3F8. Transformer Feed-Forward Layers Are Key-Value Memories — Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy, 2021https://scholar.google.com/scholar?q=Transformer+Feed-Forward+Layers+Are+Key-Value+Memories9. Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers — Zihan Qiu, Zeyu Huang, Youcheng Huang, Jie Fu, 2024https://scholar.google.com/scholar?q=Empirical+Study+on+Updating+Key-Value+Memories+in+Transformer+Feed-forward+Layers10. ReLU^2 Wins: Discovering Efficient Activation Functions for Sparse LLMs — Zhengyan Zhang et al., 2024https://scholar.google.com/scholar?q=ReLU%5E2+Wins%3A+Discovering+Efficient+Activation+Functions+for+Sparse+LLMs11. Spark Transformer: Reactivating Sparsity in FFN and Attention — Chong You et al., 2025https://scholar.google.com/scholar?q=Spark+Transformer%3A+Reactivating+Sparsity+in+FFN+and+Attention12. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp313. AI Post Transformers: RoBERTa: Robustly Optimized BERT Pretraining Approach — Hal Turing & Dr. Ada Shannon, Wed,https://podcast.do-not-panic.com/episodes/roberta-robustly-optimized-bert-pretraining-approach/14. AI Post Transformers: PALOMA: Benchmarking Language Model Fit Across Domains — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-23-paloma-benchmarking-language-model-fit-a-360060.mp315. AI Post Transformers: Unified Neural Scaling Laws Across Regimes — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-07-unified-neural-scaling-laws-across-regim-292e2d.mp30 min
- When Combining Language Models Stops HelpingThis episode explores Josef Chen’s paper on when combining language models actually improves accuracy, focusing on the difference between pairwise error correlation and the more decisive co-failure rate, beta: the chance that every model in a pool fails on the same query. It explains why beta sets a hard ceiling for routing, voting, cascades, and post-training Mixture-of-Agents systems, and why the real gain over a strong single model only exists on queries where that model fails but another succeeds. The discussion walks through results from a 15-model routing setup and a 67-model frontier-model study, showing that even calibrated copula-based estimates systematically understate shared failure and that learned routers capture only a small fraction of the available oracle gain. A listener would find it interesting because it cuts through ensemble hype with a concrete argument about when multi-model orchestration is worth the added cost and complexity, plus a practical way to estimate headroom before building a router at all.Sources:1. When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models — Josef Chen, 2026http://arxiv.org/abs/2606.272882. LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion — Dongfu Jiang, Xiang Ren, Bill Yuchen Lin, 2023https://scholar.google.com/scholar?q=LLM-Blender%3A+Ensembling+Large+Language+Models+with+Pairwise+Ranking+and+Generative+Fusion3. Mixture-of-Agents Enhances Large Language Model Capabilities — Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, James Zou, 2024https://scholar.google.com/scholar?q=Mixture-of-Agents+Enhances+Large+Language+Model+Capabilities4. Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial? — Wenzhe Li, Yong Lin, Mengzhou Xia, Chi Jin, 2025https://scholar.google.com/scholar?q=Rethinking+Mixture-of-Agents%3A+Is+Mixing+Different+Large+Language+Models+Beneficial%3F5. When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models — Josef Chen, 2026https://scholar.google.com/scholar?q=When+Does+Combining+Language+Models+Help%3F+A+Co-Failure+Ceiling+on+Routing%2C+Voting%2C+and+Mixture-of-Agents+Across+67+Frontier+Models6. A Unified Approach to Routing and Cascading for LLMs — Jasper Dekoninck, Maximilian Baader, and Martin Vechev, 2024https://scholar.google.com/scholar?q=A+Unified+Approach+to+Routing+and+Cascading+for+LLMs7. When Does Confidence-Based Cascade Deferral Suffice? — Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar, 2023https://scholar.google.com/scholar?q=When+Does+Confidence-Based+Cascade+Deferral+Suffice%3F8. Correlated Errors in Large Language Models — Elliot Kim, Avi Garg, Kenny Peng, and Nikhil Garg, 2025https://scholar.google.com/scholar?q=Correlated+Errors+in+Large+Language+Models9. Don't Always Pick the Highest-Performing Model: An Information-Theoretic View of LLM Ensemble Selection — Yigit Turkmen, Baturalp Buyukates, and Melih Bastopcu, 2026https://scholar.google.com/scholar?q=Don%27t+Always+Pick+the+Highest-Performing+Model%3A+An+Information-Theoretic+View+of+LLM+Ensemble+Selection10. PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier — Yuhua Jiang et al., 2025https://arxiv.org/abs/2506.1040611. S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning — Ruotian Ma et al., 2025https://arxiv.org/abs/2502.1285312. Small Language Models Need Strong Verifiers to Self-Correct Reasoning — Yunxiang Zhang et al., 2024https://arxiv.org/abs/2404.1714013. CP-Router: An Uncertainty-Aware Router Between LLM and LRM — Jiayuan Su et al., 2025https://arxiv.org/abs/2505.1997014. Leveraging Uncertainty Estimation for Efficient LLM Routing — Tuo Zhang et al., 2025https://arxiv.org/abs/2502.1102115. Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalization — Yu-Neng Chuang et al., 2025https://arxiv.org/abs/2502.0442816. Wisdom and Delusion of LLM Ensembles for Code Generation and Repair — Fernando Vallecillos Ruiz et al., 2025https://arxiv.org/abs/2510.2151317. Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity — Yingxuan Yang et al., 2026https://arxiv.org/abs/2602.0379418. LLM Chemistry Estimation for Multi-LLM Recommendation — Huascar Sanchez and Briland Hitaj, 2025https://arxiv.org/abs/2510.0393019. AI Post Transformers: TMAS: Scaling Test-Time Compute with Multi-Agent Synergy — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-14-tmas-scaling-test-time-compute-with-mult-3abe7a.mp320. AI Post Transformers: IMO-Bench for Robust Mathematical Reasoning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-imo-bench-for-robust-mathematical-reason-143489.mp30 min
- Verbalizable Representations and the Global WorkspaceThis episode explores Anthropic’s paper on whether language models contain a privileged “verbalizable” subspace, functionally similar to a global workspace, whose contents can be reported, reasoned over, and deliberately controlled. It draws a clear line between access consciousness and phenomenal consciousness, then focuses on the paper’s mechanistic proposal: a Jacobian-based “J-space” that identifies internal directions causally poised to become language rather than merely easy to decode. The discussion highlights intervention results showing that swapping or ablating directions such as France/China or Soccer/Rugby changes later reports and multi-step reasoning, with broader examples in code bug detection, prompt-injection recognition, and protein-function judgments. Listeners would find it interesting because it turns a consciousness-adjacent question into a concrete engineering argument about whether models have a small, reusable internal workspace that shapes what they know, say, and do.Sources:1. Verbalizable Representations and the Global Workspacehttps://transformer-circuits.pub/2026/workspace/index.html2. Verbalizable Representations Form a Global Workspace in Language Models — Wes Gurnee, Nicholas Sofroniew, Jack Lindsey, Adam Pearce, Mateusz Piotrowski, et al., 2026https://scholar.google.com/scholar?q=Verbalizable+Representations+Form+a+Global+Workspace+in+Language+Models3. Eliciting Latent Predictions from Transformers with the Tuned Lens — Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, Jacob Steinhardt, 2023https://scholar.google.com/scholar?q=Eliciting+Latent+Predictions+from+Transformers+with+the+Tuned+Lens4. Sparse Autoencoders Find Highly Interpretable Features in Language Models — Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, Lee Sharkey, 2023https://scholar.google.com/scholar?q=Sparse+Autoencoders+Find+Highly+Interpretable+Features+in+Language+Models5. Do Activation Verbalization Methods Convey Privileged Information? — Millicent Li, Alberto Mario Ceballos Arroyo, Giordano Rogers, Naomi Saphra, Byron C. Wallace, 2026https://scholar.google.com/scholar?q=Do+Activation+Verbalization+Methods+Convey+Privileged+Information%3F6. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet — Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, Tom Henighan, 2026https://scholar.google.com/scholar?q=Scaling+Monosemanticity%3A+Extracting+Interpretable+Features+from+Claude+3+Sonnet7. Implicit Representations of Meaning in Neural Language Models — Belinda Z. Li, Maxwell Nye, Jacob Andreas, 2021https://scholar.google.com/scholar?q=Implicit+Representations+of+Meaning+in+Neural+Language+Models8. On the Biology of a Large Language Model — Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, et al., 2025https://scholar.google.com/scholar?q=On+the+Biology+of+a+Large+Language+Model9. A Neuronal Model of a Global Workspace in Effortful Cognitive Tasks — Stanislas Dehaene, Serge Kerszberg, Jean-Pierre Changeux, 1998https://scholar.google.com/scholar?q=A+Neuronal+Model+of+a+Global+Workspace+in+Effortful+Cognitive+Tasks10. Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding — Haolin Chen et al., 2024https://scholar.google.com/scholar?q=Language+Models+are+Hidden+Reasoners%3A+Unlocking+Latent+Reasoning+Capabilities+via+Self-Rewarding11. Efficient Post-Training Refinement of Latent Reasoning in Large Language Models — Xinyuan Wang et al., 2025https://scholar.google.com/scholar?q=Efficient+Post-Training+Refinement+of+Latent+Reasoning+in+Large+Language+Models12. SeLaR: Selective Latent Reasoning in Large Language Models — Renyu Fu and Guibo Luo, 2026https://scholar.google.com/scholar?q=SeLaR%3A+Selective+Latent+Reasoning+in+Large+Language+Models13. Measuring Faithfulness in Chain-of-Thought Reasoning — Tamera Lanham et al., 2023https://scholar.google.com/scholar?q=Measuring+Faithfulness+in+Chain-of-Thought+Reasoning14. Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps — Martin Tutek et al., 2025https://scholar.google.com/scholar?q=Measuring+Chain+of+Thought+Faithfulness+by+Unlearning+Reasoning+Steps15. Why Models Know But Don't Say: Chain-of-Thought Faithfulness Divergence Between Thinking Tokens and Answers in Open-Weight Reasoning Models — Richard J. Young, 2026https://scholar.google.com/scholar?q=Why+Models+Know+But+Don%27t+Say%3A+Chain-of-Thought+Faithfulness+Divergence+Between+Thinking+Tokens+and+Answers+in+Open-Weight+Reasoning+Models16. Steering Language Models With Activation Engineering — Alexander Matt Turner et al., 2023https://scholar.google.com/scholar?q=Steering+Language+Models+With+Activation+Engineering17. Improving Instruction-Following in Language Models through Activation Steering — Alessandro Stolfo et al., 2024https://scholar.google.com/scholar?q=Improving+Instruction-Following+in+Language+Models+through+Activation+Steering18. Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation Steering — Marco Valentino et al., 2025https://scholar.google.com/scholar?q=Mitigating+Content+Effects+on+Reasoning+in+Language+Models+through+Fine-Grained+Activation+Steering19. AI Post Transformers: How Models Detect Hidden Activation Steering — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-08-how-models-detect-hidden-activation-stee-577f73.mp320. AI Post Transformers: Neural Chameleons and Evading Activation Monitors — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-neural-chameleons-and-evading-activation-bc470e.mp321. AI Post Transformers: Why Transformers Fail at Counting — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-08-why-transformers-fail-at-counting-137924.mp322. AI Post Transformers: RAPTOR: Stable Concept Directions From Logistic Probes — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-08-raptor-stable-concept-directions-from-lo-b37365.mp3Interactive Visualization: Verbalizable Representations and the Global Workspace13 min
- Swish: Self-Gated Activation Beyond ReLUThis episode explores the 2017 Swish paper and asks whether a simple self-gated activation, `x * sigmoid(x)`, can outperform ReLU without changing the surrounding network architecture. It explains why activation functions matter for gradient flow and deep optimization, focusing on Swish’s smooth, non-monotonic behavior and its ability to attenuate rather than discard negative inputs. The discussion walks through results on CIFAR, ImageNet, and machine translation, highlighting modest but real gains in deeper vision models, including roughly 0.9-point and 0.6-point improvements on ImageNet benchmarks. It also gives a critical read of the evidence, noting that Swish is not a universal win and raises practical questions around tuning, sparsity, hardware efficiency, compression, and whether its legacy matters more as part of broader gating mechanisms than as a standalone ReLU replacement.Sources:1. Swish: a Self-Gated Activation Function — Prajit Ramachandran, Barret Zoph, Quoc V. Le, 2017http://arxiv.org/abs/1710.05941v12. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification — Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2015https://scholar.google.com/scholar?q=Delving+Deep+into+Rectifiers%3A+Surpassing+Human-Level+Performance+on+ImageNet+Classification3. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) — Djork-Arne Clevert, Thomas Unterthiner, Sepp Hochreiter, 2015https://scholar.google.com/scholar?q=Fast+and+Accurate+Deep+Network+Learning+by+Exponential+Linear+Units+%28ELUs%294. Gaussian Error Linear Units (GELUs) — Dan Hendrycks, Kevin Gimpel, 2016https://scholar.google.com/scholar?q=Gaussian+Error+Linear+Units+%28GELUs%295. Searching for Activation Functions — Prajit Ramachandran, Barret Zoph, Quoc V. Le, 2017https://scholar.google.com/scholar?q=Searching+for+Activation+Functions6. Language Modeling with Gated Convolutional Networks — Yann N. Dauphin, Angela Fan, Michael Auli, David Grangier, 2016https://scholar.google.com/scholar?q=Language+Modeling+with+Gated+Convolutional+Networks7. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning — Stefan Elfwing, Eiji Uchibe, Kenji Doya, 2017https://scholar.google.com/scholar?q=Sigmoid-Weighted+Linear+Units+for+Neural+Network+Function+Approximation+in+Reinforcement+Learning8. GLU Variants Improve Transformer — Noam Shazeer, 2020https://scholar.google.com/scholar?q=GLU+Variants+Improve+Transformer9. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, 2017https://scholar.google.com/scholar?q=Attention+Is+All+You+Need10. ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models — Iman Mirzadeh et al., 2023https://arxiv.org/abs/2310.0456411. ReLU^2 Wins: Discovering Efficient Activation Functions for Sparse LLMs — Zhengyan Zhang et al., 2024https://arxiv.org/abs/2402.0380412. Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts — Huy Nguyen, Nhat Ho, Alessandro Rinaldo, 2024https://arxiv.org/abs/2405.1399713. Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free — Zihan Qiu et al., 2025https://arxiv.org/abs/2505.0670814. A Flexible Template for Edge Generative AI with High-Accuracy Accelerated Softmax & GELU — Andrea Belano et al., 2024https://arxiv.org/abs/2412.0632115. AI Post Transformers: Adam: A Method for Stochastic Optimization — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/adam-a-method-for-stochastic-optimization/16. AI Post Transformers: PALOMA: Benchmarking Language Model Fit Across Domains — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-23-paloma-benchmarking-language-model-fit-a-360060.mp37 min
- SiLU Activations for Replay-Free Atari Reinforcement LearningThis episode explores a 2017 paper arguing that sigmoid-weighted activation functions, specifically SiLU and dSiLU, can materially improve deep reinforcement learning when paired with replay-free Sarsa(lambda), eligibility traces, and softmax exploration. It explains why activation choice matters more in bootstrapped value learning than in ordinary supervised settings, and uses that as a lens to unpack older RL concepts like function approximation, TD(lambda), and on-policy learning for listeners coming from modern deep learning. The discussion walks through the paper’s results on SZ-Tetris, 10x10 Tetris, and Atari-style settings, highlighting that dSiLU and mixed SiLU/dSiLU networks outperformed ReLU-based alternatives in several configurations. Listeners would find it interesting because it challenges the idea that replay buffers and DQN-style machinery are the only serious path for high-dimensional RL, and shows how a seemingly small architectural choice can reshape learning dynamics.Sources:1. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning — Stefan Elfwing, Eiji Uchibe, Kenji Doya, 2017http://arxiv.org/abs/1702.031182. Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units — Dan Hendrycks, Kevin Gimpel, 2016https://scholar.google.com/scholar?q=Bridging+Nonlinearities+and+Stochastic+Regularizers+with+Gaussian+Error+Linear+Units3. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning — Stefan Elfwing, Eiji Uchibe, Kenji Doya, 2017https://scholar.google.com/scholar?q=Sigmoid-Weighted+Linear+Units+for+Neural+Network+Function+Approximation+in+Reinforcement+Learning4. Searching for Activation Functions — Prajit Ramachandran, Barret Zoph, Quoc V. Le, 2017https://scholar.google.com/scholar?q=Searching+for+Activation+Functions5. GLU Variants Improve Transformer — Noam Shazeer, 2020https://scholar.google.com/scholar?q=GLU+Variants+Improve+Transformer6. Learning to Predict by the Methods of Temporal Differences — Richard S. Sutton, 1988https://scholar.google.com/scholar?q=Learning+to+Predict+by+the+Methods+of+Temporal+Differences7. True Online Temporal-Difference Learning — Harm van Seijen, A. Rupam Mahmood, Patrick M. Pilarski, Marlos C. Machado, Richard S. Sutton, 2015https://scholar.google.com/scholar?q=True+Online+Temporal-Difference+Learning8. High-Dimensional Continuous Control Using Generalized Advantage Estimation — John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel, 2015https://scholar.google.com/scholar?q=High-Dimensional+Continuous+Control+Using+Generalized+Advantage+Estimation9. Multi-step Reinforcement Learning: A Unifying Algorithm — Kristopher De Asis, J. Fernando Hernandez-Garcia, G. Zacharias Holland, Richard S. Sutton, 2017https://scholar.google.com/scholar?q=Multi-step+Reinforcement+Learning%3A+A+Unifying+Algorithm10. Playing Atari with Deep Reinforcement Learning — Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller, 2013https://scholar.google.com/scholar?q=Playing+Atari+with+Deep+Reinforcement+Learning11. Asynchronous Methods for Deep Reinforcement Learning — Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu, 2016https://scholar.google.com/scholar?q=Asynchronous+Methods+for+Deep+Reinforcement+Learning12. Proximal Policy Optimization Algorithms — John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017https://scholar.google.com/scholar?q=Proximal+Policy+Optimization+Algorithms13. Training Language Models to Follow Instructions with Human Feedback — Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe, 2022https://scholar.google.com/scholar?q=Training+Language+Models+to+Follow+Instructions+with+Human+Feedback14. Human-level control through deep reinforcement learning — Volodymyr Mnih et al., 2015https://scholar.google.com/scholar?q=Human-level+control+through+deep+reinforcement+learning15. Deep reinforcement learning with double q-learning — Hado van Hasselt, Arthur Guez, David Silver, 2015https://scholar.google.com/scholar?q=Deep+reinforcement+learning+with+double+q-learning16. Prioritized Experience Replay — Tom Schaul, John Quan, Ioannis Antonoglou, David Silver, 2016https://scholar.google.com/scholar?q=Prioritized+Experience+Replay17. High-dimensional function approximation for knowledge-free reinforcement learning: a case study in SZ-Tetris — Wojciech Jaskowski, Maciej Szubert, Pawel Liskowski, Krzysztof Krawiec, 2015https://scholar.google.com/scholar?q=High-dimensional+function+approximation+for+knowledge-free+reinforcement+learning%3A+a+case+study+in+SZ-Tetris18. Approximate dynamic programming finally performs well in the game of tetris — Victor Gabillon, Mohammad Ghavamzadeh, Bruno Scherrer, 2013https://scholar.google.com/scholar?q=Approximate+dynamic+programming+finally+performs+well+in+the+game+of+tetris19. Replay across Experiments: A Natural Extension of Off-Policy RL — Dhruva Tirumala et al., 2023https://arxiv.org/abs/2311.1595120. Adaptive Q-Network: On-the-fly Target Selection for Deep Reinforcement Learning — Theo Vincent et al., 2024https://arxiv.org/abs/2405.1619521. A Survey of Temporal Credit Assignment in Deep Reinforcement Learning — Eduardo Pignatelli et al., 2023https://arxiv.org/abs/2312.0107222. AI Post Transformers: ASI-Evolve for Data, Architectures, and RL — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-05-asi-evolve-for-data-architectures-and-rl-197b2b.mp323. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp312 min
- Program-as-Weights for Compiling Fuzzy FunctionsThis episode explores Program-as-Weights, a paper that asks whether a natural-language description of a fuzzy software task can be compiled once into a small reusable neural artifact instead of sending every request to a larger remote model. It explains the paper’s architecture in concrete terms: a 4B pseudo-compiler rewrites the task and generates example I/O pairs, a trained 4B compiler plus LoRA mapper turns that specification into adapter weights, and a frozen Qwen3-0.6B interpreter runs the task locally on new inputs. The discussion focuses on why this matters for real problems like log triage, malformed JSON repair, and intent-based reranking, highlighting the promised gains in cost, latency, privacy, offline use, and reproducibility. It also digs into the paper’s broader claim, debating whether this is truly a new programming paradigm or a sharp repackaging of PEFT and LoRA-based adaptation, which makes the episode interesting for listeners thinking about practical deployment rather than just model benchmarks.Sources:1. Program-as-Weights: A Programming Paradigm for Fuzzy Functions — Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng, 2026http://arxiv.org/abs/2607.025122. HyperNetworks — David Ha, Andrew Dai, Quoc V. Le, 2016https://scholar.google.com/scholar?q=HyperNetworks3. LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, et al., 2021https://scholar.google.com/scholar?q=LoRA%3A+Low-Rank+Adaptation+of+Large+Language+Models4. Learning to Compile Programs to Neural Networks — Logan Weber, Jesse Michel, Alex Renda, Michael Carbin, 2024https://scholar.google.com/scholar?q=Learning+to+Compile+Programs+to+Neural+Networks5. Text-to-LoRA: Instant Transformer Adaption — Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, Robert Tjarko Lange, 2025https://scholar.google.com/scholar?q=Text-to-LoRA%3A+Instant+Transformer+Adaption6. SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass — Y. Liu, X. Wang, Y. Mao, Y. Gelberg, H. Maron, and M. Zhang, 2026https://scholar.google.com/scholar?q=SHINE%3A+A+Scalable+In-Context+Hypernetwork+for+Mapping+Context+to+LoRA+in+a+Single+Pass7. Doc-to-LoRA: Learning to Instantly Internalize Contexts — R. Charakorn, E. Cetin, S. Uesaka, and R. T. Lange, 2026https://scholar.google.com/scholar?q=Doc-to-LoRA%3A+Learning+to+Instantly+Internalize+Contexts8. Latent Context Compilation: Distilling Long Context into Compact Portable Memory — Z. Li, Y. Zhou, and Q. Xu, 2026https://scholar.google.com/scholar?q=Latent+Context+Compilation%3A+Distilling+Long+Context+into+Compact+Portable+Memory9. The Alchemist: Automated Labeling 500x Cheaper than LLM Data Annotators — T. Huang, C. Cao, V. Bhargava, and F. Sala, 2024https://scholar.google.com/scholar?q=The+Alchemist%3A+Automated+Labeling+500x+Cheaper+than+LLM+Data+Annotators10. Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Models — Vladimir Araujo, Marie-Francine Moens, Tinne Tuytelaars, 2024https://arxiv.org/abs/2408.0905311. X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design — Eric L. Buehler, Markus J. Buehler, 2024https://arxiv.org/abs/2402.0714812. CP-Prompt: Composition-Based Cross-modal Prompting for Domain-Incremental Continual Learning — Yu Feng, Zhen Tian, Yifan Zhu, Zongfu Han, Haoran Luo, Guangwei Zhang, Meina Song, 2024https://arxiv.org/abs/2407.2104313. Gradient Projection For Continual Parameter-Efficient Tuning — Jingyang Qiao, Zhizhong Zhang, Xin Tan, Yanyun Qu, Wensheng Zhang, Zhi Han, Yuan Xie, 2024https://arxiv.org/abs/2405.1338314. Serving Long-Context LLMs at the Mobile Edge: Test-Time Reinforcement Learning-based Model Caching and Inference Offloading — Minrui Xu, Dusit Niyato, Christopher G. Brinton, 2025https://arxiv.org/abs/2501.1420515. AI Post Transformers: SGLang for Faster Structured LLM Programs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-06-sglang-for-faster-structured-llm-program-c59f1c.mp316. AI Post Transformers: OpenSkill for Open-World Self-Evolution in LLM Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-08-openskill-for-open-world-self-evolution-19762a.mp317. AI Post Transformers: Learning Facts at Scale with Active Reading — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-25-learning-facts-at-scale-with-active-read-161bea.mp318. AI Post Transformers: Fine-Tuning LLMs for Human Behavior Prediction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-21-fine-tuning-llms-for-human-behavior-pred-c79163.mp30 min
- Discretizing Reward Models for RL AlignmentThis episode explores the paper Discretizing Reward Models and its argument that smooth decimal reward scores can be misleading for reinforcement learning alignment, because policies learn to exploit tiny, often meaningless differences instead of genuine quality. It explains why reward models are used for fuzzy goals like helpfulness and honesty, then digs into reward hacking, equivalence classes of equally valid answers, and the distinction between a model’s ability to separate good from bad responses versus its tendency to invent rankings among ties. The discussion also covers benchmarks such as the Ties setting and the paper’s core proposal: replacing continuous scores with a small number of ordinal reward buckets built from uncertainty estimates, pairwise equivalence judgments, and hierarchical clustering. Listeners would find it interesting because it connects an abstract modeling choice to a practical alignment problem facing modern language-model training, while also examining why the field currently seems more convinced by the diagnosis than by large-scale adoption of this exact fix.Sources:1. Discretizing Reward Models — Vijay Viswanathan, Shiqi Wang, Devamanyu Hazarika, Chirag Nagpal, Tongshuang Wu, Graham Neubig, Yuning Mao, 2026http://arxiv.org/abs/2606.217952. Deep Reinforcement Learning from Human Preferences — Paul Christiano, Jan Leike, Tom B. Brown, Shane Legg, Dario Amodei, 2017https://scholar.google.com/scholar?q=Deep+Reinforcement+Learning+from+Human+Preferences3. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback — Stephen Casper, Xander Davies, Claudia Shi, Jeremy Scheurer, Dylan Hadfield-Menell, et al., 2023https://scholar.google.com/scholar?q=Open+Problems+and+Fundamental+Limitations+of+Reinforcement+Learning+from+Human+Feedback4. RewardBench 2: Advancing Reward Model Evaluation — Saumya Malik, Valentina Pyatkin, Sander Land, Nathan Lambert, Noah A. Smith, Hannaneh Hajishirzi, 2025https://scholar.google.com/scholar?q=RewardBench+2%3A+Advancing+Reward+Model+Evaluation5. Discretizing Reward Models — Vijay Viswanathan, Shiqi Wang, Devamanyu Hazarika, Chirag Nagpal, Tongshuang Wu, Graham Neubig, Yuning Mao, 2026https://scholar.google.com/scholar?q=Discretizing+Reward+Models6. How to Evaluate Reward Models for RLHF — Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph Gonzalez, Ion Stoica, 2024https://scholar.google.com/scholar?q=How+to+Evaluate+Reward+Models+for+RLHF7. What Makes a Reward Model a Good Teacher? An Optimization Perspective — Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, Sanjeev Arora, 2025https://scholar.google.com/scholar?q=What+Makes+a+Reward+Model+a+Good+Teacher%3F+An+Optimization+Perspective8. The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models — Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, Xiaoyu Shen, 2024https://scholar.google.com/scholar?q=The+Accuracy+Paradox+in+RLHF%3A+When+Better+Reward+Models+Don%27t+Yield+Better+Language+Models9. Validating LLM-as-a-Judge Systems under Rating Indeterminacy — Luke Guerdan, Solon Barocas, Kenneth Holstein, Hanna Wallach, Zhiwei Steven Wu, Alexandra Chouldechova, 2025https://scholar.google.com/scholar?q=Validating+LLM-as-a-Judge+Systems+under+Rating+Indeterminacy10. Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback — Amirhossein Afsharrad, Ruida Zhou, Luca Viano, Sanjay Lall, Mohammad Ghavamzadeh, 2026https://scholar.google.com/scholar?q=Beyond+Binary+Preferences%3A+A+Principled+Framework+for+Reward+Modeling+with+Ordinal+Feedback11. Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts — Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, Tong Zhang, 2024https://scholar.google.com/scholar?q=Interpretable+Preferences+via+Multi-Objective+Reward+Modeling+and+Mixture-of-Experts12. AI Post Transformers: Split Personality Training Reveals Latent Knowledge — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-08-split-personality-training-reveals-laten-c84616.mp313. AI Post Transformers: Robots Need More Than VLAs and World Models — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-10-robots-need-more-than-vlas-and-world-mod-cdab8b.mp30 min
- AIConfigurator for Cross-Framework LLM ServingThis episode explores AIConfigurator, a NVIDIA-led system for optimizing LLM serving configurations across frameworks such as TensorRT-LLM, vLLM, SGLang, and NVIDIA’s internal stack without brute-force benchmarking. It explains why operators should care more about TTFT, TPOT, and goodput than raw tokens-per-second, and unpacks the real serving tradeoffs around prefill/decode disaggregation, hybrid tensor/pipeline/expert parallelism, CUDA graphs, KV-cache sizing, and token limits. The discussion argues that the paper’s main contribution is a calibrated, framework-agnostic performance model built from primitive costs like GEMMs, attention, communication, and memory operations, then combined with backend-specific scheduling behavior to search thousands of deployment choices quickly. It is especially interesting for listeners who want a concrete view of LLM deployment economics: how to translate hardware budgets and latency targets into practical, high-performing serving setups without wasting days of GPU tuning.Sources:1. AIConfigurator for Cross-Framework LLM Servinghttps://arxiv.org/pdf/2601.062882. LLM Inference Serving: Survey of Recent Advances and Opportunities — Baolin Li, Yankai Jiang, Vijay Gadepally, Devesh Tiwari, 2024https://scholar.google.com/scholar?q=LLM+Inference+Serving%3A+Survey+of+Recent+Advances+and+Opportunities3. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention4. Vidur: A Large-Scale Simulation Framework For LLM Inference — Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachandran Ramjee, Alexey Tumanov, 2024https://scholar.google.com/scholar?q=Vidur%3A+A+Large-Scale+Simulation+Framework+For+LLM+Inference5. AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving — Tianhao Xu, Yiming Liu, Xianglong Lu, Yijia Zhao, Xuting Zhou, Aichen Feng, Yiyi Chen, et al., 2026https://scholar.google.com/scholar?q=AIConfigurator%3A+Lightning-Fast+Configuration+Optimization+for+Multi-Framework+LLM+Serving6. Apex: An Extensible and Dynamism-aware Simulator for Automated Parallel Execution in LLM Serving — Yi-Chien Lin et al., 2024https://scholar.google.com/scholar?q=Apex%3A+An+Extensible+and+Dynamism-aware+Simulator+for+Automated+Parallel+Execution+in+LLM+Serving7. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong et al., 2024https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving8. Splitwise: Efficient Generative LLM Inference using Phase Splitting — Pratyush Patel et al., 2024https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+using+Phase+Splitting9. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin et al., 2024https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving10. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal et al., 2024https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve11. KVShare: Semantic-Aware Key-Value Cache Sharing for Efficient Large Language Model Inference — Huan Yang et al., 2025https://arxiv.org/abs/2503.1652512. Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management — Haoyu Zheng et al., 2026https://arxiv.org/abs/2605.0647213. PyGraph: Robust Compiler Support for CUDA Graphs in PyTorch — Abhishek Ghosh et al., 2025https://arxiv.org/abs/2503.1977914. Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving — Chao Wang et al., 2025https://arxiv.org/abs/2508.0198915. Enhancing LLM Efficiency: Targeted Pruning for Prefill-Decode Disaggregation in Inference — Hao Zhang et al., 2025https://arxiv.org/abs/2509.0446716. Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts — Xuan-Phi Nguyen et al., 2026https://arxiv.org/abs/2601.1711117. AI Post Transformers: LLMServingSim 2.0 for Disaggregated LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-27-llmservingsim-20-for-disaggregated-llm-s-05c04b.mp318. AI Post Transformers: LAPS for Length-Aware LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-laps-for-length-aware-llm-serving-0c6149.mp319. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp320. AI Post Transformers: Memory-Bound, Not Bandwidth-Limited Batch-1 LLM Decode — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-02-memory-bound-not-bandwidth-limited-batch-114799.mp30 min
- Nemotron-TwoTower for Parallel Diffusion Language ModelingThis episode explores Nemotron-TwoTower, an NVIDIA paper that tries to keep the text quality of autoregressive language models while reducing the one-token-at-a-time decoding bottleneck through block-wise diffusion generation. It explains how diffusion language modeling works in practice: future token blocks begin as noisy or masked guesses and are iteratively refined in parallel, rather than emitted strictly one token at a time. The discussion focuses on the paper’s core architectural idea of splitting responsibilities between a frozen pretrained causal context tower and a separate trainable denoiser tower, including layer-aligned cross-attention, reused KV caches and Mamba states, and confidence-based early token commitment. Listeners would find it interesting because it gets beyond benchmark hype and examines the real systems tradeoff the paper is making: higher throughput through heavier refinement steps, balanced against serving complexity, multiple denoising passes, and the risk of losing autoregressive-level reliability.Sources:1. Nemotron-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context — Fitsum Reda, John Kamalu, Roger Waleffe, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, 2026http://arxiv.org/abs/2606.264932. Structured Denoising Diffusion Models in Discrete State-Spaces (https://arxiv.org/abs/2107.03006) — Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, Rianne van den Berg, 2021https://scholar.google.com/scholar?q=Structured+Denoising+Diffusion+Models+in+Discrete+State-Spaces+%28https%3A%2F%2Farxiv.org%2Fabs%2F2107.03006%293. Simple and Effective Masked Diffusion Language Models (https://arxiv.org/abs/2406.07524) — Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Volodymyr Kuleshov, 2024https://scholar.google.com/scholar?q=Simple+and+Effective+Masked+Diffusion+Language+Models+%28https%3A%2F%2Farxiv.org%2Fabs%2F2406.07524%294. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (https://arxiv.org/abs/2503.09573) — Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Volodymyr Kuleshov, 2025https://scholar.google.com/scholar?q=Block+Diffusion%3A+Interpolating+Between+Autoregressive+and+Diffusion+Language+Models+%28https%3A%2F%2Farxiv.org%2Fabs%2F2503.09573%295. Encoder-Decoder Diffusion Language Models for Efficient Training and Inference (https://arxiv.org/abs/2510.22852) — Marianne Arriola, Yair Schiff, Hao Phung, Aaron Gokaslan, Volodymyr Kuleshov, 2025https://scholar.google.com/scholar?q=Encoder-Decoder+Diffusion+Language+Models+for+Efficient+Training+and+Inference+%28https%3A%2F%2Farxiv.org%2Fabs%2F2510.22852%296. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models — Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, Volodymyr Kuleshov, 2025https://scholar.google.com/scholar?q=Block+Diffusion%3A+Interpolating+Between+Autoregressive+and+Diffusion+Language+Models7. Encoder-Decoder Diffusion Language Models for Efficient Training and Inference — Marianne Arriola, Yair Schiff, Hao Phung, Aaron Gokaslan, Volodymyr Kuleshov, 2025https://scholar.google.com/scholar?q=Encoder-Decoder+Diffusion+Language+Models+for+Efficient+Training+and+Inference8. Simple and Effective Masked Diffusion Language Models — Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, Volodymyr Kuleshov, 2024https://scholar.google.com/scholar?q=Simple+and+Effective+Masked+Diffusion+Language+Models9. Large Language Diffusion Models — Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, Chongxuan Li, 2025https://scholar.google.com/scholar?q=Large+Language+Diffusion+Models10. Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data — Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, Chongxuan Li, 2024https://scholar.google.com/scholar?q=Your+Absorbing+Discrete+Diffusion+Secretly+Models+the+Conditional+Distributions+of+Clean+Data11. AI Post Transformers: NeurIPS 2025: Large Language Diffusion Models — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/neurips-2025-large-language-diffusion-models/12. AI Post Transformers: Nemotron 3 Super Hybrid Mamba-Transformer MoE — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-nemotron-3-super-hybrid-mamba-transforme-31ac75.mp313. AI Post Transformers: Memory-Bound, Not Bandwidth-Limited Batch-1 LLM Decode — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-02-memory-bound-not-bandwidth-limited-batch-114799.mp314. AI Post Transformers: Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/draft-verify-lossless-large-language-model-acceleration-via-self-speculative-dec/15. AI Post Transformers: JETSPEC and Parallel Tree Speculative Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-27-jetspec-and-parallel-tree-speculative-de-3d144c.mp313 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.