Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- The Sparsity Wall: What Reiner Pope Told Dwarkesh About MoE and Sparse AttentionEpisode title: The Sparsity Wall: What Reiner Pope Told Dwarkesh About MoE and Sparse AttentionA two-paper deep dive framed around the Dwarkesh Patel x Reiner Popeblackboard lecture on training and serving frontier LLMs. The hostswork through "Unified Scaling Laws for Routed Language Models" (Clarket al., DeepMind 2022, arXiv 2202.01169) for the mixture-of-expertsside and the DeepSeek sparse-attention paper (arXiv 2512.02556) forthe attention side, treating Pope's blackboard framing on thepodcast as the pedagogical lens. The episode separates what thepapers establish from what Pope's practitioner intuition adds ontop, with particular attention to how MoE on the FFN side and sparseattention on the QK side attack independent cost pools and cancompound rather than compete.Sources:arXiv 2202.01169 — "Unified Scaling Laws for Routed Language Models"https://arxiv.org/pdf/2202.01169arXiv 2512.02556 — DeepSeek sparse-attention paperhttps://arxiv.org/pdf/2512.02556Dwarkesh Podcast — "Reiner Pope: The math behind how LLMs are trained and served" (April 29 2026)https://www.dwarkesh.com/p/reiner-popeTranscript: https://gist.github.com/dwarkeshsp/79100f0fdeed69d76241903bb0604dbeOlder MoE context: GShard (arXiv 2006.16668), Switch Transformer (arXiv 2101.03961)Chinchilla scaling laws (arXiv 2203.15556) — referenced in the Pope episodeInteractive Visualization: The Sparsity Wall: What Reiner Pope Told Dwarkesh About MoE and Sparse Attention12 min
- Trace Rewriting Against Unauthorized LLM DistillationThis episode explores a 2026 paper on defending language models against unauthorized distillation by rewriting chain-of-thought traces before they are returned through an API. It explains the core idea of making reasoning outputs remain useful and correct for human users while becoming less effective as training data for a smaller model trying to copy the teacher, and it connects that strategy to data poisoning and watermarking. The discussion focuses on two defense families, especially LLM-based trace rewriting, and highlights reported results showing strong student degradation on reasoning-heavy tasks like MATH while often preserving or even improving teacher performance. It also digs into the paper’s main ambiguity: whether the defense truly poisons the student’s learning signal, or whether a stronger rewrite model is simply producing cleaner, differently structured reasoning that smaller distilled models fail to absorb well.Sources:1. Protecting Language Models Against Unauthorized Distillation through Trace Rewriting — Xinhang Ma, William Yeoh, Ning Zhang, Yevgeniy Vorobeychik, 2026http://arxiv.org/abs/2602.151432. Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation? — Leyi Pan, Aiwei Liu, Shiyu Huang, Yijian Lu, Xuming Hu, Lijie Wen, Irwin King, Philip S. Yu, 2025http://arxiv.org/abs/2502.115983. A Survey of Deep Neural Network Watermarking Techniques — Yusuke Li, Huili Wang, Mauro Barni, 2021https://scholar.google.com/scholar?q=A+Survey+of+Deep+Neural+Network+Watermarking+Techniques4. Distillation-Resistant Watermarking for Model Protection in NLP — Xuandong Zhao, Lei Li, Yu-Xiang Wang, 2022https://scholar.google.com/scholar?q=Distillation-Resistant+Watermarking+for+Model+Protection+in+NLP5. Protecting Language Generation Models via Invisible Watermarking — Xuandong Zhao, Yu-Xiang Wang, Lei Li, 2023https://scholar.google.com/scholar?q=Protecting+Language+Generation+Models+via+Invisible+Watermarking6. Scalable Watermarking for Identifying Large Language Model Outputs — Sumanth Dathathri, Abigail See, S. Ghaisas and colleagues, 2024https://scholar.google.com/scholar?q=Scalable+Watermarking+for+Identifying+Large+Language+Model+Outputs7. Poisoning Attacks against Support Vector Machines — Battista Biggio, Blaine Nelson, Pavel Laskov, 2012https://scholar.google.com/scholar?q=Poisoning+Attacks+against+Support+Vector+Machines8. Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses — Micah Goldblum and colleagues, 2020https://scholar.google.com/scholar?q=Dataset+Security+for+Machine+Learning%3A+Data+Poisoning%2C+Backdoor+Attacks%2C+and+Defenses9. Online Data Poisoning Attacks — Xuezhou Zhang, Xiaojin Zhu, Laurent Lessard, 2020https://scholar.google.com/scholar?q=Online+Data+Poisoning+Attacks10. Poisoning Language Models During Instruction Tuning — Alexander Wan, Eric Wallace, Sheng Shen, Dan Klein, 2023https://scholar.google.com/scholar?q=Poisoning+Language+Models+During+Instruction+Tuning11. Adversarial Training Methods for Semi-Supervised Text Classification — Takeru Miyato, Andrew M. Dai, Ian Goodfellow, 2017https://scholar.google.com/scholar?q=Adversarial+Training+Methods+for+Semi-Supervised+Text+Classification12. HotFlip: White-Box Adversarial Examples for Text Classification — Javid Ebrahimi, Anyi Rao, Daniel Lowd, Dejing Dou, 2018https://scholar.google.com/scholar?q=HotFlip%3A+White-Box+Adversarial+Examples+for+Text+Classification13. Universal Adversarial Triggers for Attacking and Analyzing NLP — Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, Sameer Singh, 2019https://scholar.google.com/scholar?q=Universal+Adversarial+Triggers+for+Attacking+and+Analyzing+NLP14. Bridge the Gap Between CV and NLP! A Gradient-based Textual Adversarial Attack Framework — Lifan Yuan, Yichi Zhang, Yangyi Chen, Wei Wei, 2023https://scholar.google.com/scholar?q=Bridge+the+Gap+Between+CV+and+NLP%21+A+Gradient-based+Textual+Adversarial+Attack+Framework15. Antidistillation Sampling — Yash Savani, Asher Trockman, Zhili Feng, Yixuan Even Xu, Avi Schwarzschild, Alexander Robey, Marc Finzi, J. Zico Kolter, 2025https://scholar.google.com/scholar?q=Antidistillation+Sampling16. DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillation — Pingzhi Li, Zhen Tan, Mohan Zhang, Huaizhi Qu, Huan Liu, Tianlong Chen, 2025https://scholar.google.com/scholar?q=DOGe%3A+Defensive+Output+Generation+for+LLM+Protection+Against+Knowledge+Distillation17. Information-Preserving Reformulation of Reasoning Traces for Antidistillation — Jiayu Ding, Lei Cui, Li Dong, Nanning Zheng, Furu Wei, 2025https://scholar.google.com/scholar?q=Information-Preserving+Reformulation+of+Reasoning+Traces+for+Antidistillation18. Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation? — Leyi Pan, Aiwei Liu, Shiyu Huang, Yijian Lu, Xuming Hu, Lijie Wen, Irwin King, Philip S. Yu, 2025https://scholar.google.com/scholar?q=Can+LLM+Watermarks+Robustly+Prevent+Unauthorized+Knowledge+Distillation%3F19. Unified Attacks to Large Language Model Watermarks: Spoofing and Scrubbing in Unauthorized Knowledge Distillation — Xin Yi, Yue Li, Shunfan Zheng, Linlin Wang, Xiaoling Wang, Liang He, 2025https://scholar.google.com/scholar?q=Unified+Attacks+to+Large+Language+Model+Watermarks%3A+Spoofing+and+Scrubbing+in+Unauthorized+Knowledge+Distillation20. CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks — Xuanli He, Qiongkai Xu, Yi Zeng, Lingjuan Lyu, Fangzhao Wu, Jiwei Li, Ruoxi Jia, 2022https://scholar.google.com/scholar?q=CATER%3A+Intellectual+Property+Protection+on+Text+Generation+APIs+via+Conditional+Watermarks21. DistillGuard: Evaluating Defenses Against LLM Knowledge Distillation — Bo Jiang, 2026https://scholar.google.com/scholar?q=DistillGuard%3A+Evaluating+Defenses+Against+LLM+Knowledge+Distillation22. Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents — Hyungjoo Chae, Yongho Song, Kai Tzu-iunn Ong, Taeyoon Kwon, Minjin Kim, Youngjae Yu, Dongha Lee, Dongyeop Kang, Jinyoung Yeo, 2023https://scholar.google.com/scholar?q=Dialogue+Chain-of-Thought+Distillation+for+Commonsense-aware+Conversational+Agents23. Towards Faithful Multi-step Reasoning through Fine-Grained Causal-aware Attribution Reasoning Distillation — Zheng Chu, Jingchang Chen, Zhongjie Wang, Guo Tang, Qianglong Chen, Ming Liu, Bing Qin, 2025https://scholar.google.com/scholar?q=Towards+Faithful+Multi-step+Reasoning+through+Fine-Grained+Causal-aware+Attribution+Reasoning+Distillation24. Large Language Model Watermark Stealing With Mixed Integer Programming — Zhaoxi Zhang, Xiaomei Zhang, Yanjun Zhang, Leo Yu Zhang, Chao Chen, Shengshan Hu, Asif Gill, Shirui Pan, 2024https://scholar.google.com/scholar?q=Large+Language+Model+Watermark+Stealing+With+Mixed+Integer+Programming25. Distilling the Thought, Watermarking the Answer: A Principle Semantic Guided Watermark for Large Reasoning Models — Shuliang Liu, Xingyu Li, Hongyi Liu, Dong Fang, Yibo Yan, Bingchen Duan, Qi Zheng, Lingfeng Su, Xuming Hu, 2026https://scholar.google.com/scholar?q=Distilling+the+Thought%2C+Watermarking+the+Answer%3A+A+Principle+Semantic+Guided+Watermark+for+Large+Reasoning+Models26. AI Post Transformers: Self-Improving Pretraining With Post-Trained Models — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-02-self-improving-pretraining-with-post-tra-e37460.mp327. AI Post Transformers: Distilling Multi-Agent Reasoning into a Single LLM — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-distilling-multi-agent-reasoning-into-a-143263.mp328. AI Post Transformers: AgenticQwen and Small Industrial Tool Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-27-agenticqwen-and-small-industrial-tool-ag-dc676d.mp3Interactive Visualization: Trace Rewriting Against Unauthorized LLM Distillation0 min
- Agentic AI as a Path to AGIThis episode explores a position paper arguing that agentic AI systems, built from task decomposition, routing, specialized components, and explicit graph-like workflows, may offer a more credible path to AGI than simply scaling a single monolithic model. It examines how the paper frames AGI through both broad competence across environments and efficient skill acquisition, then asks whether real-world tasks are structured enough for modular systems to outperform one-model-fits-all approaches. The discussion connects that claim to prior work on universal intelligence, compositional generalization, graph-based inductive biases, hierarchical planning, and modular prompting, while stressing that the core debate is about whether intelligence needs external structure rather than just more parameters. A listener would find it interesting for its sharp, theory-driven challenge to the dominant scaling narrative and its concrete attempt to formalize when multi-agent systems should have an advantage.Sources:1. Agentic AI as a Path to AGIhttps://arxiv.org/pdf/2605.129662. HTN Planning: Complexity and Expressivity — Kutluhan Erol, James Hendler, Dana S. Nau, 1994https://scholar.google.com/scholar?q=HTN+Planning%3A+Complexity+and+Expressivity3. Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition — Thomas G. Dietterich, 2000https://scholar.google.com/scholar?q=Hierarchical+Reinforcement+Learning+with+the+MAXQ+Value+Function+Decomposition4. Decomposed Prompting: A Modular Approach for Solving Complex Tasks — Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, Ashish Sabharwal, 2022https://scholar.google.com/scholar?q=Decomposed+Prompting%3A+A+Modular+Approach+for+Solving+Complex+Tasks5. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face — Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, Yueting Zhuang, 2023https://scholar.google.com/scholar?q=HuggingGPT%3A+Solving+AI+Tasks+with+ChatGPT+and+its+Friends+in+Hugging+Face6. Graph of Thoughts: Solving Elaborate Problems with Large Language Models — Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, Torsten Hoefler, 2023https://scholar.google.com/scholar?q=Graph+of+Thoughts%3A+Solving+Elaborate+Problems+with+Large+Language+Models7. TDAG: A Multi-Agent Framework based on Dynamic Task Decomposition and Agent Generation — Yaoxiang Wang, Zhiyong Wu, Junfeng Yao, Jinsong Su, 2024https://scholar.google.com/scholar?q=TDAG%3A+A+Multi-Agent+Framework+based+on+Dynamic+Task+Decomposition+and+Agent+Generation8. DAWN: Distributed LLM Multi-Agent Workflow Synthesis — Guancheng Wan, Mo Zhou, Ziyi Wang, Xiaoran Shang, Eric Hanchen Jiang, Guibin Zhang, Jinhe Bi, Yunpu Ma, Zaixi Zhang, Ke Liang, Wenke Huang, 2026https://scholar.google.com/scholar?q=DAWN%3A+Distributed+LLM+Multi-Agent+Workflow+Synthesis9. From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agents — Ling Yue, Kushal Raj Bhandari, Ching-Yun Ko, Dhaval Patel, Shuxin Lin, Nianjun Zhou, Jianxi Gao, Pin-Yu Chen, Shaowu Pan, 2026https://scholar.google.com/scholar?q=From+Static+Templates+to+Dynamic+Runtime+Graphs%3A+A+Survey+of+Workflow+Optimization+for+LLM+Agents10. A Generalist Agent — Scott Reed et al., 2022https://scholar.google.com/scholar?q=A+Generalist+Agent11. The Measure of Intelligence — François Chollet, 2019https://scholar.google.com/scholar?q=The+Measure+of+Intelligence12. On the Measure of Intelligence — Shane Legg and Marcus Hutter, 2007https://scholar.google.com/scholar?q=On+the+Measure+of+Intelligence13. Relational Inductive Biases, Deep Learning, and Graph Networks — Peter W. Battaglia et al., 2018https://scholar.google.com/scholar?q=Relational+Inductive+Biases%2C+Deep+Learning%2C+and+Graph+Networks14. No Free Lunch Theorems for Optimization — David H. Wolpert and William G. Macready, 1997https://scholar.google.com/scholar?q=No+Free+Lunch+Theorems+for+Optimization15. Scaling can lead to compositional generalization — Florian Redhardt, Yassir Akram, Simon Schug, 2025https://scholar.google.com/scholar?q=Scaling+can+lead+to+compositional+generalization16. Single-agent or Multi-agent Systems? Why Not Both? — Mingyan Gao et al., 2025https://scholar.google.com/scholar?q=Single-agent+or+Multi-agent+Systems%3F+Why+Not+Both%3F17. When Single-Agent with Skills Replace Multi-Agent Systems and When They Fail — Xiaoxiao Li, 2026https://scholar.google.com/scholar?q=When+Single-Agent+with+Skills+Replace+Multi-Agent+Systems+and+When+They+Fail18. Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance? — Qisheng Hu, Quanyu Long, Wenya Wang, 2024/2025https://scholar.google.com/scholar?q=Decomposition+Dilemmas%3A+Does+Claim+Decomposition+Boost+or+Burden+Fact-Checking+Performance%3F19. Unifying Language Agent Algorithms with Graph-based Orchestration Engine for Reproducible Agent Research — Qianqian Zhang et al., 2025https://scholar.google.com/scholar?q=Unifying+Language+Agent+Algorithms+with+Graph-based+Orchestration+Engine+for+Reproducible+Agent+Research20. AI Post Transformers: ASI-Evolve for Data, Architectures, and RL — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-05-asi-evolve-for-data-architectures-and-rl-197b2b.mp321. AI Post Transformers: Kimi K2.5 and Visual Agent Swarms — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-24-kimi-k25-and-visual-agent-swarms-7d04d7.mp322. AI Post Transformers: AI Co-Mathematician for Mathematical Research — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-14-ai-co-mathematician-for-mathematical-res-4aa2d4.mp323. AI Post Transformers: TMAS: Scaling Test-Time Compute with Multi-Agent Synergy — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-14-tmas-scaling-test-time-compute-with-mult-3abe7a.mp324. AI Post Transformers: Agentic Discovery for Test-Time Scaling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-12-agentic-discovery-for-test-time-scaling-f9a81f.mp325. AI Post Transformers: AgenticQwen and Small Industrial Tool Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-27-agenticqwen-and-small-industrial-tool-ag-dc676d.mp326. AI Post Transformers: MEMSEARCHER: Reinforcement Learning for LLM Memory Management — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-memsearcher-reinforcement-learning-for-l-e9ad84.mp327. AI Post Transformers: Neural Computers as Learned Latent Runtimes — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-11-neural-computers-as-learned-latent-runti-9fa282.mp3Interactive Visualization: Agentic AI as a Path to AGI0 min
- When Many-Shot CoT Becomes Test-Time LearningThis episode explores a May 13, 2026 arXiv paper arguing that many-shot chain-of-thought prompting can act less like simple retrieval and more like a form of test-time learning, where structured context helps a model reason during inference without changing its weights. It examines the paper’s main claims that adding many reasoning demonstrations does not reliably help across all settings, that semantically similar examples can fail when their reasoning procedures are not actually usable, and that the order of demonstrations becomes more important as prompts grow longer. The discussion also focuses on the paper’s Curvilinear Demonstration Selection approach, which treats prompt construction more like designing a lesson plan than doing nearest-neighbor search, with especially notable gains on geometry tasks. Listeners would find it interesting because it challenges a common assumption behind huge context windows: more examples are not automatically better, and effective prompting may depend on curriculum design, model capabilities, and the compatibility of reasoning traces.Sources:1. When Many-Shot CoT Becomes Test-Time Learninghttps://arxiv.org/pdf/2605.135112. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, 2022https://arxiv.org/abs/2201.119033. Large Language Models are Zero-Shot Reasoners — Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, Yusuke Iwasawa, 2022https://arxiv.org/abs/2205.119164. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou, 2022https://arxiv.org/abs/2203.111715. Automatic Chain of Thought Prompting in Large Language Models — Zhuosheng Zhang, Aston Zhang, Mu Li, Alex Smola, 2022https://arxiv.org/abs/2210.034936. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts — Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, Moritz Hardt, 2020https://arxiv.org/abs/1909.132317. What Learning Algorithm is In-Context Learning? Investigations with Linear Models — Ekin Akyurek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, Denny Zhou, 2022https://arxiv.org/abs/2211.156618. Transformers Learn In-Context by Gradient Descent — Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, Max Vladymyrov, 2022https://arxiv.org/abs/2212.076779. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar, 2024https://arxiv.org/abs/2408.0331410. What Makes Good In-Context Examples for GPT-3? — Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, Weizhu Chen, 2021https://arxiv.org/abs/2101.0680411. Learning To Retrieve Prompts for In-Context Learning — Ohad Rubin, Jonathan Herzig, Jonathan Berant, 2021https://arxiv.org/abs/2112.0863312. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity — Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, Pontus Stenetorp, 2021https://arxiv.org/abs/2104.0878613. Many-Shot CoT-ICL: Making In-Context Learning Truly Learn — Tsz Ting Chung, Lemao Liu, Mo Yu, Dit-Yan Yeung, 2026https://arxiv.org/abs/2605.1351114. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? — Sewon Min, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer, 2022https://scholar.google.com/scholar?q=Rethinking+the+Role+of+Demonstrations%3A+What+Makes+In-Context+Learning+Work%3F15. Test-Time Compute: Scaling Language Models with More Thinking — OpenAI, 2024https://scholar.google.com/scholar?q=Test-Time+Compute%3A+Scaling+Language+Models+with+More+Thinking16. ALR2: A Retrieve-then-Reason Framework for Long-context Question Answering — Huayang Li, Pat Verga, Priyanka Sen, Bowen Yang, Vijay Viswanathan, Patrick Lewis, Taro Watanabe, Yixuan Su, 2024https://scholar.google.com/scholar?q=ALR2%3A+A+Retrieve-then-Reason+Framework+for+Long-context+Question+Answering17. Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models — Yifu Qiu, Varun Embar, Yizhe Zhang, Navdeep Jaitly, Shay B. Cohen, Benjamin Han, 2025https://scholar.google.com/scholar?q=Eliciting+In-context+Retrieval+and+Reasoning+for+Long-context+Large+Language+Models18. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions — Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal, 2023https://scholar.google.com/scholar?q=Interleaving+Retrieval+with+Chain-of-Thought+Reasoning+for+Knowledge-Intensive+Multi-Step+Questions19. Active Prompting with Chain-of-Thought for Large Language Models — Shizhe Diao, Pengcheng Wang, Yong Lin, Rui Pan, Xiang Liu, Tong Zhang, 2023https://scholar.google.com/scholar?q=Active+Prompting+with+Chain-of-Thought+for+Large+Language+Models20. Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models — Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, Weizhu Chen, 2023https://scholar.google.com/scholar?q=Synthetic+Prompting%3A+Generating+Chain-of-Thought+Demonstrations+for+Large+Language+Models21. Contrastive Chain-of-Thought Prompting — Yew Ken Chia, Guizhen Chen, Luu Anh Tuan, Soujanya Poria, Lidong Bing, 2023https://scholar.google.com/scholar?q=Contrastive+Chain-of-Thought+Prompting22. Let's (not) just put things in Context: Test-Time Training for Long-Context LLMs — Rachit Bansal, Aston Zhang, Rishabh Tiwari, Lovish Madaan, Sai Surya Duvvuri, Devvrit Khatri, David Brandfonbrener, David Alvarez-Melis, Prajjwal Bhargava, Mihir Sanjay Kale, Samy Jelassi, 2025https://scholar.google.com/scholar?q=Let%27s+%28not%29+just+put+things+in+Context%3A+Test-Time+Training+for+Long-Context+LLMs23. AI Post Transformers: In-Place Test-Time Training for Transformers — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-09-in-place-test-time-training-for-transfor-d0b976.mp324. AI Post Transformers: Training LLMs for Divide-and-Conquer Reasoning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-04-training-llms-for-divide-and-conquer-rea-ea6e22.mp325. AI Post Transformers: Reasoning Theater and Unfaithful Chain-of-Thought — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-reasoning-theater-and-unfaithful-chain-o-a4507e.mp326. AI Post Transformers: Test-time Scaling for Multi-Agent Collaborative Reasoning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-test-time-scaling-for-multi-agent-collab-082570.mp327. AI Post Transformers: δ-mem and Online Memory for LLMs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-13-d-mem-and-online-memory-for-llms-6622fa.mp3Interactive Visualization: When Many-Shot CoT Becomes Test-Time Learning0 min
- Ministral 3: Cascade Distillation for Long-Context Multimodal ModelsThis episode explores Ministral 3, a family of 3B, 8B, and 14B long-context multimodal models built from a 24B parent through structured pruning and cascade distillation rather than separate full-scale training runs. It explains how the method works step by step, from teacher-student distillation and capacity-gap concerns to the staged pruning pipeline that extends each child model to 256k context windows while preserving useful capabilities. The discussion places the paper in context with earlier distillation and pruning work such as Hinton’s original distillation paper, DistilBERT, teacher-assistant distillation, and NVIDIA’s Minitron, arguing that the contribution is a practical model-family construction recipe rather than a brand-new paradigm. Listeners would find it interesting because it gets at a central 2026 question in AI deployment: whether smaller, cheaper models can stay competitive on long-context and multimodal tasks by amortizing one expensive parent run across several deployable descendants.Sources:1. Ministral 3 — Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, Alexandre Sablayrolles, Amélie Héliou, Amos You, Andy Ehrenberg, Andy Lo, Anton Eliseev, Antonia Calvi, Avinash Sooriyarachchi, Baptiste Bout, Baptiste Rozière, Baudouin De Monicault, Clémence Lanfranchi, Corentin Barreau, Cyprien Courtot, Daniele Grattarola, Darius Dabert, Diego de las Casas, Elliot Chane-Sane, Faruk Ahmed, Gabrielle Berrada, Gaëtan Ecrepont, Gauthier Guinet, Georgii Novikov, Guillaume Kunsch, Guillaume Lample, Guillaume Martin, Gunshi Gupta, Jan Ludziejewski, Jason Rute, Joachim Studnia, Jonas Amar, Joséphine Delas, Josselin Somerville Roberts, Karmesh Yadav, Khyathi Chandu, Kush Jain, Laurence Aitchison, Laurent Fainsin, Léonard Blier, Lingxiao Zhao, Louis Martin, Lucile Saulnier, Luyu Gao, Maarten Buyl, Margaret Jennings, Marie Pellat, Mark Prins, Mathieu Poirée, Mathilde Guillaumin, Matthieu Dinot, Matthieu Futeral, Maxime Darrin, Maximilian Augustin, Mia Chiquier, Michel Schimpf, Nathan Grinsztajn, Neha Gupta, Nikhil Raghuraman, Olivier Bousquet, Olivier Duchenne, Patricia Wang, Patrick von Platen, Paul Jacob, Paul Wambergue, Paula Kurylowicz, Pavankumar Reddy Muddireddy, Philomène Chagniot, Pierre Stock, Pravesh Agrawal, Quentin Torroba, Romain Sauvestre, Roman Soletskyi, Rupert Menneer, Sagar Vaze, Samuel Barry, Sanchit Gandhi, Siddhant Waghjale, Siddharth Gandhi, Soham Ghosh, Srijan Mishra, Sumukh Aithal, Szymon Antoniak, Teven Le Scao, Théo Cachet, Theo Simon Sorg, Thibaut Lavril, Thiziri Nait Saada, Thomas Chabal, Thomas Foubert, Thomas Robert, Thomas Wang, Tim Lawson, Tom Bewley, Tom Bewley, Tom Edwards, Umar Jamil, Umberto Tomasini, Valeriia Nemychnikova, Van Phung, Vincent Maladière, Virgile Richard, Wassim Bouaziz, Wen-Ding Li, William Marshall, Xinghui Li, Xinyu Yang, Yassine El Ouahidi, Yihan Wang, Yunhao Tang, Zaccharie Ramzi, 2026http://arxiv.org/abs/2601.085842. Distilling the Knowledge in a Neural Network — Geoffrey Hinton, Oriol Vinyals, Jeff Dean, 2015https://scholar.google.com/scholar?q=Distilling+the+Knowledge+in+a+Neural+Network3. Improved Knowledge Distillation via Teacher Assistant — Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, Hassan Ghasemzadeh, 2019https://scholar.google.com/scholar?q=Improved+Knowledge+Distillation+via+Teacher+Assistant4. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter — Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf, 2019https://scholar.google.com/scholar?q=DistilBERT%2C+a+distilled+version+of+BERT%3A+smaller%2C+faster%2C+cheaper+and+lighter5. Compact Language Models via Pruning and Knowledge Distillation — Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, Pavlo Molchanov, 2024https://scholar.google.com/scholar?q=Compact+Language+Models+via+Pruning+and+Knowledge+Distillation6. LLM Pruning and Distillation in Practice: The Minitron Approach — S. T. Sreenivas, S. Muralidharan, R. Joshi, M. Chochowski, A. S. Mahabaleshwarkar, G. Shen, J. Zeng, Z. Chen, Y. Suhara, S. Diao, C. Yu, W. Chen, H. Ross, O. Olabiyi, A. Aithal, O. Kuchaiev, D. Korzekwa, P. Molchanov, M. Patwary, M. Shoeybi, J. Kautz, and B. Catanzaro, 2024https://scholar.google.com/scholar?q=LLM+Pruning+and+Distillation+in+Practice%3A+The+Minitron+Approach7. Distillation Scaling Laws — D. Busbridge, A. Shidani, F. Weers, J. Ramapuram, E. Littwin, and R. Webb, 2025https://scholar.google.com/scholar?q=Distillation+Scaling+Laws8. Distilled Pretraining: A Modern Lens of Data, In-Context Learning and Test-Time Scaling — S. Goyal, D. Lopez-Paz, and K. Ahuja, 2025https://scholar.google.com/scholar?q=Distilled+Pretraining%3A+A+Modern+Lens+of+Data%2C+In-Context+Learning+and+Test-Time+Scaling9. Pixtral 12B — P. Agrawal, S. Antoniak, E. B. Hanna, B. Bout, D. Chaplot, J. Chudnovsky, D. Costa, B. De Monicault, S. Garg, T. Gervet, et al., 2024https://scholar.google.com/scholar?q=Pixtral+12B10. Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs — Lu Yin et al., 2024https://scholar.google.com/scholar?q=Junk+DNA+Hypothesis%3A+Pruning+Small+Pre-Trained+Weights+Irreversibly+and+Monotonically+Impairs+%22Difficult%22+Downstream+Tasks+in+LLMs11. SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot — Elias Frantar and Dan Alistarh, 2023https://scholar.google.com/scholar?q=SparseGPT%3A+Massive+Language+Models+Can+Be+Accurately+Pruned+in+One-Shot12. Fast and Effective Weight Update for Pruned Large Language Models — Vladimir Boza, 2024https://scholar.google.com/scholar?q=Fast+and+Effective+Weight+Update+for+Pruned+Large+Language+Models13. Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs — Ruihan Jin et al., 2026https://scholar.google.com/scholar?q=Exploring+Knowledge+Purification+in+Multi-Teacher+Knowledge+Distillation+for+LLMs14. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models — Siyan Zhao et al., 2026https://scholar.google.com/scholar?q=Self-Distilled+Reasoner%3A+On-Policy+Self-Distillation+for+Large+Language+Models15. Data Engineering for Scaling Language Models to 128K Context — Yao Fu et al., 2024https://scholar.google.com/scholar?q=Data+Engineering+for+Scaling+Language+Models+to+128K+Context16. How to Train Long-Context Language Models (Effectively) — Tianyu Gao et al., 2025https://scholar.google.com/scholar?q=How+to+Train+Long-Context+Language+Models+%28Effectively%2917. Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models — Jun Zhang et al., 2025https://scholar.google.com/scholar?q=Train+Small%2C+Infer+Large%3A+Memory-Efficient+LoRA+Training+for+Large+Language+Models18. AI Post Transformers: DeepSeek-V4 and Practical Million-Token Context — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-25-deepseek-v4-and-practical-million-token-6f4de1.mp319. AI Post Transformers: Muon Is Scalable for LLM Training — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-25-muon-is-scalable-for-llm-training-587ed8.mp320. AI Post Transformers: Learning to Reason with 13 Parameters — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-learning-to-reason-with-13-parameters-54c87f.mp321. AI Post Transformers: AgenticQwen and Small Industrial Tool Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-27-agenticqwen-and-small-industrial-tool-ag-dc676d.mp3Interactive Visualization: Ministral 3: Cascade Distillation for Long-Context Multimodal Models0 min
- Causal-JEPA for Object-Level World ModelsThis episode explores Causal-JEPA, a world-modeling approach that masks whole object trajectories rather than image patches to force a model to reason about interactions between entities. It explains how the method combines object-centric representations with JEPA-style latent prediction, asking the model to reconstruct hidden objects from scene context and then predict future dynamics, instead of relying on pixel reconstruction or simple autoregressive rollouts. The discussion highlights the paper’s core argument that this training setup makes counterfactual and causal reasoning more necessary by blocking shortcut strategies like temporal interpolation and self-contained single-object motion prediction. Listeners would find it interesting for its sharp comparison between patch-based scaling and object-centric structure, and for its claim that better world models may come from making interaction reasoning unavoidable rather than merely possible.Sources:1. Causal-JEPA: Learning World Models through Object-Level Latent Interventions — Heejeong Nam, Quentin Le Lidec, Lucas Maes, Yann LeCun, Randall Balestriero, 2026http://arxiv.org/abs/2602.113892. MONet: Unsupervised Scene Decomposition and Representation — Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, Alexander Lerchner, 2019https://scholar.google.com/scholar?q=MONet%3A+Unsupervised+Scene+Decomposition+and+Representation3. Multi-Object Representation Learning with Iterative Variational Inference — Klaus Greff, Raphael Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, Alexander Lerchner, 2019https://scholar.google.com/scholar?q=Multi-Object+Representation+Learning+with+Iterative+Variational+Inference4. Object-Centric Learning with Slot Attention — Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, Thomas Kipf, 2020https://scholar.google.com/scholar?q=Object-Centric+Learning+with+Slot+Attention5. Bridging the Gap to Real-World Object-Centric Learning — Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, Francesco Locatello, 2023https://scholar.google.com/scholar?q=Bridging+the+Gap+to+Real-World+Object-Centric+Learning6. A Path Towards Autonomous Machine Intelligence — Yann LeCun, 2022https://scholar.google.com/scholar?q=A+Path+Towards+Autonomous+Machine+Intelligence7. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture — Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, Nicolas Ballas, 2023https://scholar.google.com/scholar?q=Self-Supervised+Learning+from+Images+with+a+Joint-Embedding+Predictive+Architecture8. Revisiting Feature Prediction for Learning Visual Representations from Video — Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, Nicolas Ballas, 2024https://scholar.google.com/scholar?q=Revisiting+Feature+Prediction+for+Learning+Visual+Representations+from+Video9. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning — Mido Assran, Adrien Bardes, David Fan and many others including Yann LeCun, Michael Rabbat, Nicolas Ballas, 2025https://scholar.google.com/scholar?q=V-JEPA+2%3A+Self-Supervised+Video+Models+Enable+Understanding%2C+Prediction+and+Planning10. CLEVRER: CoLlision Events for Video REpresentation and Reasoning — Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, 2020https://scholar.google.com/scholar?q=CLEVRER%3A+CoLlision+Events+for+Video+REpresentation+and+Reasoning11. Counterfactual VQA: A Cause-Effect Look at Language Bias — Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, Ji-Rong Wen, 2021https://scholar.google.com/scholar?q=Counterfactual+VQA%3A+A+Cause-Effect+Look+at+Language+Bias12. What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models — Letian Zhang, Xiaotong Zhai, Zhongkai Zhao, Yongshuo Zong, Xin Wen, Bingchen Zhao, 2024https://scholar.google.com/scholar?q=What+If+the+TV+Was+Off%3F+Examining+Counterfactual+Reasoning+Abilities+of+Multi-modal+Language+Models13. ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos — Te-Lin Wu, Zi-Yi Dou, Qingyuan Hu, Yu Hou, Nischal Reddy Chandra, Marjorie Freedman, Ralph M. Weischedel, Nanyun Peng, 2023https://scholar.google.com/scholar?q=ACQUIRED%3A+A+Dataset+for+Answering+Counterfactual+Questions+In+Real-Life+Videos14. Towards Causal Representation Learning — Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, Yoshua Bengio, 2021https://scholar.google.com/scholar?q=Towards+Causal+Representation+Learning15. Interventional Causal Representation Learning — Kartik Ahuja, Divyat Mahajan, Yixin Wang, Yoshua Bengio, 2023https://scholar.google.com/scholar?q=Interventional+Causal+Representation+Learning16. Desiderata for Representation Learning: A Causal Perspective — Yixin Wang, Michael I. Jordan, 2024https://scholar.google.com/scholar?q=Desiderata+for+Representation+Learning%3A+A+Causal+Perspective17. Provably Learning Object-Centric Representations — Stefan Bauer, Bernhard Schölkopf and collaborators, 2023https://scholar.google.com/scholar?q=Provably+Learning+Object-Centric+Representations18. SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models — Yuhang Wu, Yueting Zhuang, Francesco Locatello, et al., 2022https://scholar.google.com/scholar?q=SlotFormer%3A+Unsupervised+Visual+Dynamics+Simulation+with+Object-Centric+Models19. Object-Centric Video Prediction via Decoupling of Object Dynamics and Interactions — Angel Villar-Corrales, Ismail Wahdan, Sven Behnke, 2023https://scholar.google.com/scholar?q=Object-Centric+Video+Prediction+via+Decoupling+of+Object+Dynamics+and+Interactions20. DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning — Gaoyue Zhou, Hengkai Pan, Yann LeCun, Lerrel Pinto, 2024https://scholar.google.com/scholar?q=DINO-WM%3A+World+Models+on+Pre-trained+Visual+Features+enable+Zero-shot+Planning21. Conditional Object-Centric Learning from Video — Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, Klaus Greff, 2022https://scholar.google.com/scholar?q=Conditional+Object-Centric+Learning+from+Video22. Attention over Learned Object Embeddings Enables Complex Visual Reasoning — David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, Matt Botvinick, 2021https://scholar.google.com/scholar?q=Attention+over+Learned+Object+Embeddings+Enables+Complex+Visual+Reasoning23. Dyn-O: Building Structured World Models with Object-Centric Representations — Zizhao Wang, Kaixin Wang, Li Zhao, Peter Stone, Jiang Bian, 2025https://scholar.google.com/scholar?q=Dyn-O%3A+Building+Structured+World+Models+with+Object-Centric+Representations24. Learning Interactive World Model for Object-Centric Reinforcement Learning — Fan Feng, Phillip Lippe, Sara Magliacane, 2025https://scholar.google.com/scholar?q=Learning+Interactive+World+Model+for+Object-Centric+Reinforcement+Learning25. Object-Centric World Model for Language-Guided Manipulation — Youngjoon Jeong, Junha Chun, Soonwoo Cha, Taesup Kim, 2025https://scholar.google.com/scholar?q=Object-Centric+World+Model+for+Language-Guided+Manipulation26. Planning in 8 Tokens: A Compact Discrete Tokenizer for Latent World Model — Dongwon Kim, Gawon Seo, Jinsung Lee, Minsu Cho, Suha Kwak, 2026https://scholar.google.com/scholar?q=Planning+in+8+Tokens%3A+A+Compact+Discrete+Tokenizer+for+Latent+World+Model27. Learning nonparametric latent causal graphs with unknown interventions — Yibo Jiang, Bryon Aragam, 2023https://scholar.google.com/scholar?q=Learning+nonparametric+latent+causal+graphs+with+unknown+interventions28. Learning Linear Causal Representations from Interventions under General Nonlinear Mixing — Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, Pradeep Ravikumar, 2023https://scholar.google.com/scholar?q=Learning+Linear+Causal+Representations+from+Interventions+under+General+Nonlinear+Mixing29. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp330. AI Post Transformers: Learning Latent Action World Models from Video — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-09-learning-latent-action-world-models-from-1570a4.mp331. AI Post Transformers: DreamerV3 World Models Across 150 Tasks — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-20-dreamerv3-world-models-across-150-tasks-af5edb.mp3Interactive Visualization: Causal-JEPA for Object-Level World Models0 min
- Scaling Laws for Multilingual Code PretrainingThis episode explores whether code models should allocate training data evenly across programming languages or tune the mix based on how each language scales. It walks through the paper’s core experiments on monolingual scaling, bilingual transfer, translation-style code pairing, and multilingual token allocation, with close attention to languages like Python, JavaScript, Rust, and TypeScript. The discussion highlights the paper’s main argument that programming languages differ in scaling behavior and cross-lingual usefulness, which could make non-uniform token budgets more effective than treating all code equally. Listeners would find it interesting for its concrete take on how multilingual software ecosystems challenge simple scaling-law assumptions and for its careful scrutiny of whether the evidence really supports those stronger optimization claims.Sources:1. Scaling Laws for Multilingual Code Pretraininghttps://arxiv.org/pdf/2512.134722. Unsupervised Translation of Programming Languages — Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, Guillaume Lample, 2020https://scholar.google.com/scholar?q=Unsupervised+Translation+of+Programming+Languages3. XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence — Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, Chandan K. Reddy, 2022https://scholar.google.com/scholar?q=XLCoST%3A+A+Benchmark+Dataset+for+Cross-lingual+Code+Intelligence4. MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation — Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, Abhinav Jangda, 2022https://scholar.google.com/scholar?q=MultiPL-E%3A+A+Scalable+and+Extensible+Approach+to+Benchmarking+Neural+Code+Generation5. CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X — Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, Jie Tang, 2023https://scholar.google.com/scholar?q=CodeGeeX%3A+A+Pre-Trained+Model+for+Code+Generation+with+Multilingual+Benchmarking+on+HumanEval-X6. Scaling Laws for Code: A More Data-Hungry Regime — Xianzhen Luo, Wenzhen Zheng, Qingfu Zhu, Rongyi Zhang, Houyi Li, Siming Huang, YuanTao Fan, Wanxiang Che, 2025https://scholar.google.com/scholar?q=Scaling+Laws+for+Code%3A+A+More+Data-Hungry+Regime7. Cross-lingual Transfer in Programming Languages: An Extensive Empirical Study — Razan Baltaji, Saurabh Pujar, Louis Mandel, Martin Hirzel, Luca Buratti, Lav Varshney, 2025https://scholar.google.com/scholar?q=Cross-lingual+Transfer+in+Programming+Languages%3A+An+Extensive+Empirical+Study8. CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution — Ruiyang Xu, Jialun Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, Le Sun, 2024https://scholar.google.com/scholar?q=CRUXEval-X%3A+A+Benchmark+for+Multilingual+Code+Reasoning%2C+Understanding+and+Execution9. RepoTransBench: A Real-World Benchmark for Repository-Level Code Translation — Yanli Wang, Yanlin Wang, Suiquan Wang, Daya Guo, Jiachi Chen, John Grundy, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, Zibin Zheng, 2024https://scholar.google.com/scholar?q=RepoTransBench%3A+A+Real-World+Benchmark+for+Repository-Level+Code+Translation10. Towards Multi-Language Repository-Level Code Generation: From-Scratch to Guided Tasks — authors unclear from snippet, unknownhttps://scholar.google.com/scholar?q=Towards+Multi-Language+Repository-Level+Code+Generation%3A+From-Scratch+to+Guided+Tasks11. Do Not Treat Code as Natural Language: Implications for Repository-Level Code Generation and Beyond — authors unclear from snippet, unknownhttps://scholar.google.com/scholar?q=Do+Not+Treat+Code+as+Natural+Language%3A+Implications+for+Repository-Level+Code+Generation+and+Beyond12. Code-Switching In-Context Learning for Cross-Lingual Transfer of Large Language Models — authors unclear from snippet, unknownhttps://scholar.google.com/scholar?q=Code-Switching+In-Context+Learning+for+Cross-Lingual+Transfer+of+Large+Language+Models13. Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfer — authors unclear from snippet, unknownhttps://scholar.google.com/scholar?q=Bridging+the+Language+Gap%3A+Enhancing+Multilingual+Prompt-Based+Code+Generation+in+LLMs+via+Zero-Shot+Cross-Lingual+Transfer14. Advancing Code Translation With Context-Aware Pre-Training in Data-Scarce Environments — authors unclear from snippet, unknownhttps://scholar.google.com/scholar?q=Advancing+Code+Translation+With+Context-Aware+Pre-Training+in+Data-Scarce+Environments15. Semi-Supervised Code Translation Overcoming the Scarcity of Parallel Code Data — authors unclear from snippet, unknownhttps://scholar.google.com/scholar?q=Semi-Supervised+Code+Translation+Overcoming+the+Scarcity+of+Parallel+Code+Data16. Scaling Laws for Predicting Downstream Performance in LLMs — authors unclear from snippet, unknownhttps://scholar.google.com/scholar?q=Scaling+Laws+for+Predicting+Downstream+Performance+in+LLMs17. AI Post Transformers: CODEGEN: Open Language Model for Code Synthesis — Hal Turing & Dr. Ada Shannon, Fri,https://podcast.do-not-panic.com/episodes/codegen-open-language-model-for-code-synthesis/18. AI Post Transformers: Llama 3: Architecture, Capabilities, and Safety — Hal Turing & Dr. Ada Shannon, Sun,https://podcast.do-not-panic.com/episodes/llama-3-architecture-capabilities-and-safety/19. AI Post Transformers: Program Synthesis with Large Language Models — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-20-program-synthesis-with-large-language-mo-b962ec.mp320. AI Post Transformers: MTEB & MMTEB: The Massive Text Embedding Benchmark — Hal Turing & Dr. Ada Shannon, Fri,https://podcast.do-not-panic.com/episodes/mteb-mmteb-the-massive-text-embedding-benchmark/Interactive Visualization: Scaling Laws for Multilingual Code Pretraining0 min
- JANUS for Scalable MoE InferenceThis episode explores JANUS, a systems approach to serving mixture-of-experts transformers efficiently by separating attention layers from expert layers instead of deploying the whole model as a single monolithic unit. It explains why MoE models can still be expensive and latency-prone in practice: even if only a few experts activate per token, the system must still manage large expert memory footprints, skewed expert demand, and strict token-level latency targets such as time per output token. The discussion focuses on JANUS’s core ideas, including separate GPU pools for attention and expert computation, an adaptive two-phase communication scheme that reduces cross-node messaging overhead, and SLO-aware scaling that adjusts attention and expert capacity independently. Listeners would find it interesting because it turns MoE inference from a simple “sparse compute saves money” story into a deeper argument about distributed systems design, load balancing, and the real bottlenecks that determine whether advanced models feel fast in production.Sources:1. Janus: Disaggregating Attention and Experts for Scalable MoE Inference — Zhexiang Zhang, Ye Wang, Yumiao Zhao, Jiayu Xiao, Qianjing Yang, Xiangyu Wang, Jingzhe Jiang, Qizhen Weng, Ruichuan Chen, Shaohuai Shi, Adel N. Toosi, Yin Chen, Minchen Yu, 2025http://arxiv.org/abs/2512.135252. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — William Fedus, Barret Zoph, Noam Shazeer, 2021https://scholar.google.com/scholar?q=Switch+Transformers%3A+Scaling+to+Trillion+Parameter+Models+with+Simple+and+Efficient+Sparsity3. FastMoE: A Fast Mixture-of-Expert Training System — Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, Jie Tang, 2021https://scholar.google.com/scholar?q=FastMoE%3A+A+Fast+Mixture-of-Expert+Training+System4. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving5. MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism — Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang and others, 2025https://scholar.google.com/scholar?q=MegaScale-Infer%3A+Serving+Mixture-of-Experts+at+Scale+with+Disaggregated+Expert+Parallelism6. eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference — Suraiya Tairin, Shohaib Mahmud, Haiying Shen, Anand Iyer, 2025https://scholar.google.com/scholar?q=eMoE%3A+Task-aware+Memory+Efficient+Mixture-of-Experts-Based+%28MoE%29+Model+Inference7. SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference — Luchang Li, Dongfang Li, Bozhao Gong, Yu Zhang, 2026https://scholar.google.com/scholar?q=SLO-Aware+Compute+Resource+Allocation+for+Prefill-Decode+Disaggregated+LLM+Inference8. MoEless: Efficient MoE LLM Serving via Serverless Computing — Hanfei Yu, Bei Ouyang, Shwai He, Ang Li, Hao Wang, 2026https://scholar.google.com/scholar?q=MoEless%3A+Efficient+MoE+LLM+Serving+via+Serverless+Computing9. xDeepServe: Model-as-a-Service on Huawei CloudMatrix384 — Ao Xiao et al., 2025https://scholar.google.com/scholar?q=xDeepServe%3A+Model-as-a-Service+on+Huawei+CloudMatrix38410. Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling — Yan Li, Zhenyu Zhang, Zhengang Wang, Pengfei Chen, Pengfei Zheng, 2025https://scholar.google.com/scholar?q=Semantic+Parallelism%3A+Redefining+Efficient+MoE+Inference+via+Model-Data+Co-Scheduling11. GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference — Yu Han, Lehan Pan, Jie Peng, Ziyang Tao, Hanqi Zhu, Wuyang Zhang, Yanyong Zhang, 2025https://scholar.google.com/scholar?q=GRACE-MoE%3A+Grouping+and+Replication+with+Locality-Aware+Routing+for+Efficient+Distributed+MoE+Inference12. BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems — Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, Xiaowen Chu, 2024https://scholar.google.com/scholar?q=BurstGPT%3A+A+Real-world+Workload+Dataset+to+Optimize+LLM+Serving+Systems13. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI et al., 2024https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model14. Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference — Ranggi Hwang et al., 2023/2024https://arxiv.org/abs/2308.1206615. HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference — Peng Tang et al., 2024https://arxiv.org/abs/2411.0143316. DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference — Yujie Zhang, Shivam Aggarwal, Tulika Mitra, 2025https://arxiv.org/abs/2501.1037517. AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding — Zikun Li et al., 2025https://arxiv.org/abs/2501.1216218. SLOs-Serve: Optimized Serving of Multi-SLO LLMs — Siyuan Chen et al., 2025https://arxiv.org/abs/2504.0878419. Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement — Tian Wu et al., 2025https://arxiv.org/abs/2508.1285120. AI Post Transformers: Batch-Aware Expert Routing for Faster MoE Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-batch-aware-expert-routing-for-faster-mo-683ab6.mp321. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp322. AI Post Transformers: Computation-Bandwidth-Memory Trade-offs for AI Infrastructure — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-09-computation-bandwidth-memory-trade-offs-a83f2b.mp323. AI Post Transformers: Why LLM Serving Needs Mathematical Optimization — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-why-llm-serving-needs-mathematical-optim-647fc6.mp324. AI Post Transformers: LAPS for Length-Aware LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-laps-for-length-aware-llm-serving-0c6149.mp325. AI Post Transformers: DeepSeek-V4 and Practical Million-Token Context — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-25-deepseek-v4-and-practical-million-token-6f4de1.mp326. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp3Interactive Visualization: JANUS for Scalable MoE Inference13 min
- Lossless Sparse Deltas for RL NetworksThis episode explores a systems paper on making reinforcement-learning post-training for large language models practical over ordinary Ethernet and even WAN links, rather than requiring expensive RDMA clusters. It explains why trainer-actor RL creates a synchronization bottleneck, how full policy refreshes can dominate runtime on 1 to 10 gigabit networks, and why that turns bandwidth into a hidden limiter of who can run serious RL workloads. The discussion centers on the paper’s proposed solution: lossless sparse delta checkpoints that send only changed parameters, along with carefully encoded indices, streamed in parallel with rollout generation so actors can reconstruct the exact updated model without quantization or approximation. Listeners would find it interesting because it connects low-level systems design to the economics and accessibility of modern LLM training, asking whether better synchronization methods could open RL post-training to labs and startups outside elite infrastructure environments.Sources:1. RL over Commodity Networks: Overcoming the Bandwidth Barrier with Lossless Sparse Deltas — Chaoyi Ruan, Geng Luo, Xinyi Wan, Long Zhao, Qinghe Wang, Jiaan Zhu, Duling Xu, Guanbin Xu, Dehui Wei, Xiang Liu, Cheng Li, Haifeng Sun, Congcong Miao, Jialin Li, 2026http://arxiv.org/abs/2602.114562. Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL — Erfan Miahi, Eugene Belilovsky, 2026https://scholar.google.com/scholar?q=Understanding+and+Exploiting+Weight+Update+Sparsity+for+Communication-Efficient+Distributed+RL3. StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation — Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, Hongyu Zhou, Yimin Jiang, Yibo Zhu, Daxin Jiang, 2025https://scholar.google.com/scholar?q=StreamRL%3A+Scalable%2C+Heterogeneous%2C+and+Elastic+RL+for+LLMs+with+Disaggregated+Stream+Generation4. HybridFlow: A Flexible and Efficient RLHF Framework — Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, Chuan Wu, 2024https://scholar.google.com/scholar?q=HybridFlow%3A+A+Flexible+and+Efficient+RLHF+Framework5. OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework — Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, Weikai Fang, Xianyu, Yu Cao, Haotian Xu, Yiming Liu, 2024https://scholar.google.com/scholar?q=OpenRLHF%3A+An+Easy-to-use%2C+Scalable+and+High-performance+RLHF+Framework6. How Can We Train Deep Learning Models Across Clouds and Continents? An Experimental Study — Alexander Erben, Ruben Mayer, Hans-Arno Jacobsen, 2023https://scholar.google.com/scholar?q=How+Can+We+Train+Deep+Learning+Models+Across+Clouds+and+Continents%3F+An+Experimental+Study7. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention8. Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models — authors not confirmed from snippet, recent, unconfirmed from snippethttps://scholar.google.com/scholar?q=Asynchronous+RLHF%3A+Faster+and+More+Efficient+Off-Policy+RL+for+Language+Models9. Faster, More Efficient RLHF through Off-Policy Asynchronous Learning — authors not confirmed from snippet, recent, unconfirmed from snippethttps://scholar.google.com/scholar?q=Faster%2C+More+Efficient+RLHF+through+Off-Policy+Asynchronous+Learning10. Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs — authors not confirmed from snippet, recent, unconfirmed from snippethttps://scholar.google.com/scholar?q=Stable+Asynchrony%3A+Variance-Controlled+Off-Policy+RL+for+LLMs11. Efficient Online RFT with Plug-and-Play LLM Judges: Unlocking State-of-the-Art Performance — authors not confirmed from snippet, recent, unconfirmed from snippethttps://scholar.google.com/scholar?q=Efficient+Online+RFT+with+Plug-and-Play+LLM+Judges%3A+Unlocking+State-of-the-Art+Performance12. Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding — authors not confirmed from snippet, recent, unconfirmed from snippethttps://scholar.google.com/scholar?q=Accelerating+RL+Post-Training+Rollouts+via+System-Integrated+Speculative+Decoding13. Beat the Long Tail: Distribution-Aware Speculative Decoding for RL Training — authors not confirmed from snippet, recent, unconfirmed from snippethttps://scholar.google.com/scholar?q=Beat+the+Long+Tail%3A+Distribution-Aware+Speculative+Decoding+for+RL+Training14. AI Post Transformers: HALoS: Hierarchical Asynchronous LLM Training over Slow Networks — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/halos-hierarchical-asynchronous-llm-training-over-slow-networks/15. AI Post Transformers: TensorFlow for Distributed Machine Learning Systems — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-06-tensorflow-for-distributed-machine-learn-b7fa52.mp316. AI Post Transformers: AgenticQwen and Small Industrial Tool Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-27-agenticqwen-and-small-industrial-tool-ag-dc676d.mp3Interactive Visualization: Lossless Sparse Deltas for RL Networks0 min
- FlashFuser and Hopper-Era FFN Kernel FusionThis episode explores how the FlashFuser paper uses Hopper GPU inter-core communication to push kernel fusion beyond the usual single-SM memory limits, especially for transformer feed-forward networks and gated FFNs. It explains why this matters now: H100-class GPUs have gained compute far faster than memory bandwidth, making activation spills to HBM an increasingly painful bottleneck for workloads that can consume 40 to 60 percent of inference time. The discussion walks through Hopper’s distributed shared memory model and FlashFuser’s core idea of coordinating reduce, shuffle, and multiply patterns across SM clusters so large intermediate activations can stay on chip longer. Listeners would find it interesting because it connects compiler techniques, GPU architecture, and real transformer inference bottlenecks into a concrete argument about when newer hardware may finally make more aggressive fusion worthwhile.Sources:1. FlashFuser: Expanding the Scale of Kernel Fusion for Compute-Intensive Operators via Inter-Core Connection — Ziyu Huang, Yangjie Zhou, Zihan Liu, Xinhao Luo, Yijia Diao, Minyi Guo, Jidong Zhai, Yu Feng, Chen Zhang, Anbang Wu, Jingwen Leng, 2025http://arxiv.org/abs/2512.129492. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning — Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, Arvind Krishnamurthy, 2018https://scholar.google.com/scholar?q=TVM%3A+An+Automated+End-to-End+Optimizing+Compiler+for+Deep+Learning3. FusionStitching: Boosting Memory Intensive Computations for Deep Learning Workloads — Zhen Zheng, Pengzhan Zhao, Guoping Long, Feiwen Zhu, Kai Zhu, Wenyi Zhao, Lansong Diao, Jun Yang, Wei Lin, 2020https://scholar.google.com/scholar?q=FusionStitching%3A+Boosting+Memory+Intensive+Computations+for+Deep+Learning+Workloads4. Operator Fusion in XLA: Analysis and Evaluation — Daniel Snider, Ruofan Liang, 2023https://scholar.google.com/scholar?q=Operator+Fusion+in+XLA%3A+Analysis+and+Evaluation5. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Re, 2022https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness6. Benchmarking and Dissecting the Nvidia Hopper GPU Architecture — Weile Luo, Ruibo Fan, Zeyu Li, Dayou Du, Qiang Wang, Xiaowen Chu, 2024https://scholar.google.com/scholar?q=Benchmarking+and+Dissecting+the+Nvidia+Hopper+GPU+Architecture7. A Case Study in CUDA Kernel Fusion: Implementing FlashAttention-2 on NVIDIA Hopper Architecture using the CUTLASS Library — Ganesh Bikshandi, Jay Shah, 2023https://scholar.google.com/scholar?q=A+Case+Study+in+CUDA+Kernel+Fusion%3A+Implementing+FlashAttention-2+on+NVIDIA+Hopper+Architecture+using+the+CUTLASS+Library8. Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor with T10 — Yiqi Liu, Yuqi Xue, Yu Cheng, Lingxiao Ma, Ziming Miao, Jilong Xue, Jian Huang, 2024https://scholar.google.com/scholar?q=Scaling+Deep+Learning+Computation+over+the+Inter-Core+Connected+Intelligence+Processor+with+T109. FlashFuser: Expanding the Scale of Kernel Fusion for Compute-Intensive Operators via Inter-Core Connection — Ziyu Huang, Yangjie Zhou, Zihan Liu, Xinhao Luo, Yijia Diao, Minyi Guo, Jidong Zhai, Yu Feng, Chen Zhang, Anbang Wu, Jingwen Leng, 2025https://scholar.google.com/scholar?q=FlashFuser%3A+Expanding+the+Scale+of+Kernel+Fusion+for+Compute-Intensive+Operators+via+Inter-Core+Connection10. Chimera: An Analytical Optimizing Framework for Effective Compute-intensive Operators Fusion — Size Zheng, Siyuan Chen, Peidi Song, Renze Chen, Xiuhong Li, Shengen Yan, Dahua Lin, Jingwen Leng, Yun Liang, 2023https://scholar.google.com/scholar?q=Chimera%3A+An+Analytical+Optimizing+Framework+for+Effective+Compute-intensive+Operators+Fusion11. BOLT: Bridging the Gap between Auto-tuners and Hardware-native Performance — Jiarong Xing, Leyuan Wang, Shang Zhang, Jack Chen, Ang Chen, Yibo Zhu, 2022https://scholar.google.com/scholar?q=BOLT%3A+Bridging+the+Gap+between+Auto-tuners+and+Hardware-native+Performance12. MCFuser: High-Performance and Rapid Fusion of Memory-Bound Compute-Intensive Operators — Zheng Zhang, Donglin Yang, Xiaobo Zhou, Dazhao Cheng, 2024https://scholar.google.com/scholar?q=MCFuser%3A+High-Performance+and+Rapid+Fusion+of+Memory-Bound+Compute-Intensive+Operators13. Deep Kernel Fusion for Transformers — Zixi Zhang, Zhiwen Mo, Yiren Zhao, Robert Mullins, 2026https://scholar.google.com/scholar?q=Deep+Kernel+Fusion+for+Transformers14. Benchmarking thread block cluster — approximate; unclear from snippet, 2023-2026https://scholar.google.com/scholar?q=Benchmarking+thread+block+cluster15. ClusterSim: modeling thread block clusters in Hopper GPUs — approximate; unclear from snippet, 2023-2026https://scholar.google.com/scholar?q=ClusterSim%3A+modeling+thread+block+clusters+in+Hopper+GPUs16. Analysing and Reducing Costs of Deep Learning Compiler Auto-tuning — approximate; unclear from snippet, 2023-2026https://scholar.google.com/scholar?q=Analysing+and+Reducing+Costs+of+Deep+Learning+Compiler+Auto-tuning17. AI Post Transformers: Why LLM Serving Needs Mathematical Optimization — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-why-llm-serving-needs-mathematical-optim-647fc6.mp318. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp319. AI Post Transformers: LAPS for Length-Aware LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-laps-for-length-aware-llm-serving-0c6149.mp320. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp321. AI Post Transformers: Caffeine: A Unified FPGA for CNNs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-06-caffeine-a-unified-fpga-for-cnns-e8acbe.mp322. AI Post Transformers: Gated Linear Attention for Efficient Long Sequences — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-18-gated-linear-attention-for-efficient-lon-c858ab.mp3Interactive Visualization: FlashFuser and Hopper-Era FFN Kernel Fusion15 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.