Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- Raptor: First 3D-DRAM Accelerator for Generative Inference SiliconThis episode examines an ISCA 2026 paper on early silicon for a logic-on-3D-DRAM inference card, which targets about 100 TB/s of memory bandwidth for generative decoding. It explains why decode is memory-bound, using a Llama-70B KV-cache example that grows to roughly 20 GB per user at 128K context. It then compares three memory options: SRAM (fast but tiny), HBM (large but bandwidth-starved), and 3D-DRAM as a middle point. The discussion covers the main design problems: single-cycle microbump interfaces that rule out conventional burst-based data bus inversion, which motivates a pinless "stream flipping" scheme claimed to save about 18% energy. It also covers channel mapping, where a 128-byte flit across three banks per channel causes overfetch, along with bank sparing, RowHammer, and thermal limits. The episode keeps bench-silicon results (power, bandwidth, refresh) separate from simulated throughput claims, so listeners can judge how much of the story is actually measured.Sources:1. Raptor: First 3D-DRAM Accelerator for Generative Inference Siliconhttps://ramyadhadidi.github.io/files/dMatrix-Raptor-ISCA.pdf2. Bus-Invert Coding for Low-Power I/O — Mircea R. Stan, Wayne P. Burleson, 1995https://scholar.google.com/scholar?q=Bus-Invert+Coding+for+Low-Power+I%2FO3. Toggle-Aware Compression for GPUs — Gennady Pekhimenko, Evgeny Bolotin, Nandita Vijaykumar, Onur Mutlu, Todd C. Mowry, Stephen W. Keckler, 2016 (HPCA; earlier CAL version 2015)https://scholar.google.com/scholar?q=Toggle-Aware+Compression+for+GPUs4. Fine-Grained DRAM: Energy-Efficient DRAM for Extreme Bandwidth Systems — Mike O'Connor, Niladrish Chatterjee, Donghyuk Lee, John Wilson, Aditya Agrawal, Stephen W. Keckler, William J. Dally, 2017 (MICRO)https://scholar.google.com/scholar?q=Fine-Grained+DRAM%3A+Energy-Efficient+DRAM+for+Extreme+Bandwidth+Systems5. Input-Dependent Power Usage in GPUs — Theo Gregersen, Pratyush Patel, Esha Choukse, 2024https://scholar.google.com/scholar?q=Input-Dependent+Power+Usage+in+GPUs6. Flipping Bits in Memory Without Accessing Them: An Experimental Study of DRAM Disturbance Errors — Yoongu Kim, Ross Daly, Jeremie Kim, Chris Fallin, Ji Hye Lee, Donghyuk Lee, Chris Wilkerson, Konrad Lai, Onur Mutlu, 2014https://scholar.google.com/scholar?q=Flipping+Bits+in+Memory+Without+Accessing+Them%3A+An+Experimental+Study+of+DRAM+Disturbance+Errors7. RAIDR: Retention-Aware Intelligent DRAM Refresh — Jamie Liu, Ben Jaiyen, Richard Veras, Onur Mutlu, 2012https://scholar.google.com/scholar?q=RAIDR%3A+Retention-Aware+Intelligent+DRAM+Refresh8. Scaling Deep Learning on GPU and Knights Landing Clusters / Hybrid Memory Cube and HBM3 energy-per-bit disclosures — Various (JEDEC HBM3/HBM4 standard documents; Micron HMC specification), 2022-2025https://scholar.google.com/scholar?q=Scaling+Deep+Learning+on+GPU+and+Knights+Landing+Clusters+%2F+Hybrid+Memory+Cube+and+HBM3+energy-per-bit+disclosures9. MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism — Ruidong Zhu et al., 2025https://scholar.google.com/scholar?q=MegaScale-Infer%3A+Serving+Mixture-of-Experts+at+Scale+with+Disaggregated+Expert+Parallelism10. Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Serving — Yue Pan et al., 2025https://scholar.google.com/scholar?q=Stratum%3A+System-Hardware+Co-Design+with+Tiered+Monolithic+3D-Stackable+DRAM+for+Efficient+MoE+Serving11. H2-LLM: Hardware-Dataflow Co-Exploration for Heterogeneous Hybrid-Bonding-based Low-Batch LLM Inference — Cong Li et al., 2025https://scholar.google.com/scholar?q=H2-LLM%3A+Hardware-Dataflow+Co-Exploration+for+Heterogeneous+Hybrid-Bonding-based+Low-Batch+LLM+Inference12. AI and Memory Wall — Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, Kurt Keutzer, 2024https://scholar.google.com/scholar?q=AI+and+Memory+WallInteractive Visualization: Raptor: First 3D-DRAM Accelerator for Generative Inference Silicon17 min
- AI R&D Automation and the Intelligence Explosion QuestionThis episode examines a policy paper asking what happens if AI automates AI research itself, setting off an "intelligence explosion" that compresses years of progress into months. It reviews the evidence that automation is already underway, including the AI share of approved code at one lab rising from low single digits to over 80 percent, autonomous R&D work growing from 1 to 26 percent, and task-length horizons doubling about every three months. It also covers the caveats on that evidence, since coding benchmarks don't necessarily translate to research ability. The hosts explain the paper's feedback-loop model, in which a ratio of labor returns to idea-finding difficulty decides whether progress fades, holds steady or accelerates. They also cover the four frictions that could stop the loop (diminishing returns, compute limits, hard-to-automate research taste, and slow training runs) and the stakes of loss of control and eroded checks on power. Listeners get a clear walkthrough of what the argument claims and what it doesn't: it argues for preparation, not that an explosion is certain.Sources:1. What if automating AI R&D triggers an intelligence explosion? — Alan Chan, Christoph Winter, Andrew Barto, Jakub Pachocki, Geoffrey Hinton, Eric Horvitz, Yoshua Bengio, Dawn Song, Jack Clark, Hilary Greaves, Anton Korinek, Samuel Hammond, Thore Graepel, Ben Bariach, Philip H. S. Torr, Sheila A. McIlraith, Jeff Clune, Sam Manning, Girish Sastry, Tom Davidson, Daniel Eth, Sören Mindermann, 2026http://arxiv.org/abs/2609.360542. Speculations Concerning the First Ultraintelligent Machine — I.J. Good, 1965https://scholar.google.com/scholar?q=Speculations+Concerning+the+First+Ultraintelligent+Machine3. Intelligence Explosion Microeconomics — Eliezer Yudkowsky, 2013https://scholar.google.com/scholar?q=Intelligence+Explosion+Microeconomics4. Are Ideas Getting Harder to Find? — Nicholas Bloom, Charles I. Jones, John Van Reenen, Michael Webb, 2020https://scholar.google.com/scholar?q=Are+Ideas+Getting+Harder+to+Find%3F5. Algorithmic progress in language models — Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carson Guo, David Atkinson, Neil Thompson, Jaime Sevilla (Epoch AI), 2024https://scholar.google.com/scholar?q=Algorithmic+progress+in+language+models6. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts — Hjalmar Wijk, Tao Lin, Joel Becker, et al. (METR), 2024https://scholar.google.com/scholar?q=RE-Bench%3A+Evaluating+frontier+AI+R%26D+capabilities+of+language+model+agents+against+human+experts7. Measuring AI Ability to Complete Long Tasks — Thomas Kwa, Ben West, Joel Becker, et al. (METR), 2025https://scholar.google.com/scholar?q=Measuring+AI+Ability+to+Complete+Long+Tasks8. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery — Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, David Ha, 2024https://scholar.google.com/scholar?q=The+AI+Scientist%3A+Towards+Fully+Automated+Open-Ended+Scientific+Discovery9. AI 2027 — Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, Romeo Dean, 2025https://scholar.google.com/scholar?q=AI+202710. The Alignment Problem from a Deep Learning Perspective — Richard Ngo, Lawrence Chan, Sören Mindermann, 2022https://scholar.google.com/scholar?q=The+Alignment+Problem+from+a+Deep+Learning+Perspective11. AI Control: Improving Safety Despite Intentional Subversion — Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, Fabien Roger, 2023https://scholar.google.com/scholar?q=AI+Control%3A+Improving+Safety+Despite+Intentional+Subversion12. Alignment faking in large language models — Ryan Greenblatt, Carson Denison, Benjamin Wright, et al. (Anthropic and Redwood Research), 2024https://scholar.google.com/scholar?q=Alignment+faking+in+large+language+models13. Managing extreme AI risks amid rapid progress — Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, et al., 2024https://scholar.google.com/scholar?q=Managing+extreme+AI+risks+amid+rapid+progress14. Will compute bottlenecks prevent an intelligence explosion? — Parker Whitfill, Cheryl Wu, 2025https://scholar.google.com/scholar?q=Will+compute+bottlenecks+prevent+an+intelligence+explosion%3F15. The software intelligence explosion debate needs experiments — Anson Ho, Parker Whitfill, 2025https://scholar.google.com/scholar?q=The+software+intelligence+explosion+debate+needs+experiments16. Will AI R&D Automation Cause a Software Intelligence Explosion? — Daniel Eth, Tom Davidson, 2025https://scholar.google.com/scholar?q=Will+AI+R%26D+Automation+Cause+a+Software+Intelligence+Explosion%3F17. Measuring AI Ability to Complete Long Tasks / Time Horizon 1.1 — METR, 2025/2026https://scholar.google.com/scholar?q=Measuring+AI+Ability+to+Complete+Long+Tasks+%2F+Time+Horizon+1.118. When does automating AI research produce explosive growth? Feedback loops in innovation networks — Tom Davidson, Basil Halperin, Thomas Houlden, Anton Korinek, 2026https://scholar.google.com/scholar?q=When+does+automating+AI+research+produce+explosive+growth%3F+Feedback+loops+in+innovation+networks19. The economics of recursive self-improvement — Tom Cunningham et al., 2026https://scholar.google.com/scholar?q=The+economics+of+recursive+self-improvement20. The bounded parallelizability of R&D: Theory and application to AI — Philip Trammell, 2026https://scholar.google.com/scholar?q=The+bounded+parallelizability+of+R%26D%3A+Theory+and+application+to+AI21. On the origin of algorithmic progress in AI — Hans Gundlach et al., 2025https://scholar.google.com/scholar?q=On+the+origin+of+algorithmic+progress+in+AI22. The dynamics of intelligence explosions — Toby Ord, 2026https://scholar.google.com/scholar?q=The+dynamics+of+intelligence+explosions23. PostTrainBench: Can LLM agents automate LLM post-training? — Rank et al., 2026https://scholar.google.com/scholar?q=PostTrainBench%3A+Can+LLM+agents+automate+LLM+post-training%3F24. Towards end-to-end automation of AI research (AI Scientist) — Chris Lu, Cong Lu, Robert Lange, Jeff Clune et al., 2026https://scholar.google.com/scholar?q=Towards+end-to-end+automation+of+AI+research+%28AI+Scientist%2925. Frontier AI auditing: Toward rigorous third-party assessment of safety and security practices at leading AI companies — Miles Brundage et al., 2026https://scholar.google.com/scholar?q=Frontier+AI+auditing%3A+Toward+rigorous+third-party+assessment+of+safety+and+security+practices+at+leading+AI+companiesInteractive Visualization: AI R&D Automation and the Intelligence Explosion Question18 min
- ATLAS: Full-Stack Simulator for 3D-DRAM LLM AcceleratorsThis episode examines ATLAS, a simulator for 3D-DRAM-based LLM accelerators that the authors say was checked against real silicon. It explains why decoding is memory-bound on GPUs, and how hybrid bonding shrinks the data path from millimeters to micrometers. That cuts access energy by 77 to 83 percent compared with HBM. It covers how the design differs from processing-in-memory, and the trade-offs of small capacity and a heat limit of 85 degrees Celsius. The discussion shows why existing tools like Ramulator and earlier accelerator simulators fall short. It also goes through DRAM details such as logical rows, row-buffer locality, channel interleaving, and the tension block-paged KV cache adds. The episode ends on a skeptical question: how far conclusions can be trusted beyond the specific chip points that were validated.Sources:1. A Full-Stack Performance Evaluation Infrastructure for 3D-DRAM-based LLM Accelerators — Cong Li, Chenhao Xue, Yi Ren, Xiping Dong, Yu Cheng, Yinbo Hu, Fujun Bai, Yixin Guo, Xiping Jiang, Qiang Wu, Zhi Yang, Zhe Cheng, Yuan Xie, Guangyu Sun, 2026http://arxiv.org/abs/2604.08044v12. Tetris: Scalable and Efficient Neural Network Acceleration with 3D Memory — Mingyu Gao, Jing Pu, Xuan Yang, Mark Horowitz, Christos Kozyrakis, 2017https://scholar.google.com/scholar?q=Tetris%3A+Scalable+and+Efficient+Neural+Network+Acceleration+with+3D+Memory3. Neurocube: A Programmable Digital Neuromorphic Architecture with High-Density 3D Memory — Duckhwan Kim, Jaeha Kung, Sek Chai, Sudhakar Yalamanchili, Saibal Mukhopadhyay, 2016https://scholar.google.com/scholar?q=Neurocube%3A+A+Programmable+Digital+Neuromorphic+Architecture+with+High-Density+3D+Memory4. AttAcc! Unleashing the Power of PIM for Batched Transformer-based Generative Model Inference — Jaehyun Park, Jaewan Choi, Kwanhee Kyung, Michael Jaemin Kim, Yongsuk Kwon, Nam Sung Kim, Jung Ho Ahn, 2024https://scholar.google.com/scholar?q=AttAcc%21+Unleashing+the+Power+of+PIM+for+Batched+Transformer-based+Generative+Model+Inference5. A 1ynm 1.25V 8Gb, 16Gb/s/pin GDDR6-based Accelerator-in-Memory supporting 1TFLOPS MAC Operation and Various Activation Functions for Deep-Learning Applications — Seongju Lee, Sukhan Lee, et al. (SK hynix), 2022https://scholar.google.com/scholar?q=A+1ynm+1.25V+8Gb%2C+16Gb%2Fs%2Fpin+GDDR6-based+Accelerator-in-Memory+supporting+1TFLOPS+MAC+Operation+and+Various+Activation+Functions+for+Deep-Learning+Applications6. The gem5 Simulator — Nathan Binkert, Bradford Beckmann, Gabriel Black, Steven Reinhardt, et al., 2011https://scholar.google.com/scholar?q=The+gem5+Simulator7. Ramulator: A Fast and Extensible DRAM Simulator — Yoongu Kim, Weikun Yang, Onur Mutlu, 2016https://scholar.google.com/scholar?q=Ramulator%3A+A+Fast+and+Extensible+DRAM+Simulator8. Accel-Sim: An Extensible Simulation Framework for Validated GPU Modeling — Mahmoud Khairy, Zhesheng Shen, Tor M. Aamodt, Timothy G. Rogers, 2020https://scholar.google.com/scholar?q=Accel-Sim%3A+An+Extensible+Simulation+Framework+for+Validated+GPU+Modeling9. LLMCompass: Enabling Efficient Hardware Design for Large Language Model Inference — Hengrui Zhang, August Ning, Rohan Baskar Prabhakar, David Wentzlaff, 2024https://scholar.google.com/scholar?q=LLMCompass%3A+Enabling+Efficient+Hardware+Design+for+Large+Language+Model+Inference10. Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines — Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, Saman Amarasinghe, 2013https://scholar.google.com/scholar?q=Halide%3A+A+Language+and+Compiler+for+Optimizing+Parallelism%2C+Locality%2C+and+Recomputation+in+Image+Processing+Pipelines11. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning — Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, Arvind Krishnamurthy, 2018https://scholar.google.com/scholar?q=TVM%3A+An+Automated+End-to-End+Optimizing+Compiler+for+Deep+Learning12. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations — Philippe Tillet, H. T. Kung, David Cox, 2019https://scholar.google.com/scholar?q=Triton%3A+An+Intermediate+Language+and+Compiler+for+Tiled+Neural+Network+Computations13. Spatial: A Language and Compiler for Application Accelerators — David Koeplinger, Matthew Feldman, Raghu Prabhakar, Yaqi Zhang, Stefan Hadjis, Ruben Fiszel, Tian Zhao, Luigi Nardi, Ardavan Pedram, Christos Kozyrakis, Kunle Olukotun, 2018https://scholar.google.com/scholar?q=Spatial%3A+A+Language+and+Compiler+for+Application+Accelerators14. Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Serving — Yue Pan et al., 2025https://scholar.google.com/scholar?q=Stratum%3A+System-Hardware+Co-Design+with+Tiered+Monolithic+3D-Stackable+DRAM+for+Efficient+MoE+Serving15. H2-LLM: Hardware-Dataflow Co-Exploration for Heterogeneous Hybrid-Bonding-based Low-Batch LLM Inference — Cong Li, Yihan Yin, et al., 2025https://scholar.google.com/scholar?q=H2-LLM%3A+Hardware-Dataflow+Co-Exploration+for+Heterogeneous+Hybrid-Bonding-based+Low-Batch+LLM+Inference16. Ramulator 2.0: A Modern, Modular, and Extensible DRAM Simulator — Haocong Luo et al., 2023https://scholar.google.com/scholar?q=Ramulator+2.0%3A+A+Modern%2C+Modular%2C+and+Extensible+DRAM+Simulator17. Vidur: A Large-Scale Simulation Framework for LLM Inference — Amey Agrawal et al., 2024https://scholar.google.com/scholar?q=Vidur%3A+A+Large-Scale+Simulation+Framework+for+LLM+Inference18. Timeloop: A Systematic Approach to DNN Accelerator Evaluation — Angshuman Parashar et al., 2019https://scholar.google.com/scholar?q=Timeloop%3A+A+Systematic+Approach+to+DNN+Accelerator+Evaluation19. HotSpot 7.0 / Thermal Modeling for 3D ICs — Skadron et al.; Zhang et al., 2004-2020https://scholar.google.com/scholar?q=HotSpot+7.0+%2F+Thermal+Modeling+for+3D+ICs20. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon et al., 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention21. Samsung HBM-PIM / SK hynix AiM / AttAcc / NeuPIMs — Kwon et al.; Park et al.; Heo et al., 2021-2024https://scholar.google.com/scholar?q=Samsung+HBM-PIM+%2F+SK+hynix+AiM+%2F+AttAcc+%2F+NeuPIMs22. Splitwise / DistServe / Mooncake (PD disaggregation) — Patel et al.; Zhong et al.; Qin et al., 2024https://scholar.google.com/scholar?q=Splitwise+%2F+DistServe+%2F+Mooncake+%28PD+disaggregation%2923. DeepSeek-V3 / MLA technical reports — DeepSeek-AI, 2024-2025https://scholar.google.com/scholar?q=DeepSeek-V3+%2F+MLA+technical+reportsInteractive Visualization: ATLAS: Full-Stack Simulator for 3D-DRAM LLM Accelerators15 min
- RedFuser: Automatic Fusion of Cascaded Reductions for AI AcceleratorsThis episode examines RedFuser, an Alibaba Cloud framework that automatically fuses cascaded reductions, such as softmax feeding a matrix multiply, into a single GPU kernel. It starts with why chained reductions resist standard compiler fusion: each stage needs the finished result of the one before, which forces repeated memory reads and prevents overlap. The discussion then works through the online-softmax rescaling trick with a small numeric example. Multiplying the running denominator by exp(old max − new max) gives the same answer as recomputing from scratch, and the same correction extends to the output accumulator and to merging partial results across blocks, as in FlashAttention and FlashDecoding. It also notes that the paper doesn't appear to cite the earlier online-softmax and chunked-attention work. The central question is whether a compiler can derive this kind of algebraic rewrite for patterns no one has hand-derived, rather than only tuning schedules for a fixed algorithm. That question matters to anyone who wants new attention variants, MoE routing, or FP8 quantization pipelines to get fast kernels without a new manual derivation each time.Sources:1. RedFuser: An Automatic Operator Fusion Framework for Cascaded Reductions on AI Accelerators — Xinsheng Tang, Yangcheng Li, Nan Wang, Zhiyi Shu, Xingyu Ling, Junna Xing, Peng Zhou, Qiang Liu, 2026http://arxiv.org/abs/2603.10026v12. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness3. Welder: Scheduling Deep Learning Memory Access via Tile-graph — Yining Shi, Zhi Yang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Ziming Miao, Yuxiao Guo, Fan Yang, Lidong Zhou, 2023https://scholar.google.com/scholar?q=Welder%3A+Scheduling+Deep+Learning+Memory+Access+via+Tile-graph4. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations — Philippe Tillet, H. T. Kung, David Cox, 2019https://scholar.google.com/scholar?q=Triton%3A+An+Intermediate+Language+and+Compiler+for+Tiled+Neural+Network+Computations5. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning — Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, Arvind Krishnamurthy, 2018https://scholar.google.com/scholar?q=TVM%3A+An+Automated+End-to-End+Optimizing+Compiler+for+Deep+Learning6. Ansor: Generating High-Performance Tensor Programs for Deep Learning — Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, Joseph E. Gonzalez, Ion Stoica, 2020https://scholar.google.com/scholar?q=Ansor%3A+Generating+High-Performance+Tensor+Programs+for+Deep+Learning7. Learning to Optimize Tensor Programs (AutoTVM) — Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang, Thierry Moreau, Luis Ceze, Carlos Guestrin, Arvind Krishnamurthy, 2018https://scholar.google.com/scholar?q=Learning+to+Optimize+Tensor+Programs+%28AutoTVM%298. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024https://scholar.google.com/scholar?q=FlashAttention-3%3A+Fast+and+Accurate+Attention+with+Asynchrony+and+Low-precision9. Triton / TorchInductor: PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation — Jason Ansel and the PyTorch team, 2024https://scholar.google.com/scholar?q=Triton+%2F+TorchInductor%3A+PyTorch+2%3A+Faster+Machine+Learning+Through+Dynamic+Python+Bytecode+Transformation+and+Graph+Compilation10. Online normalizer calculation for softmax — Maxim Milakov, Natalia Gimelshein, 2018https://scholar.google.com/scholar?q=Online+normalizer+calculation+for+softmax11. Self-attention Does Not Need O(n^2) Memory — Markus N. Rabe, Charles Staats, 2021https://scholar.google.com/scholar?q=Self-attention+Does+Not+Need+O%28n%5E2%29+Memory12. Flash-Decoding for long-context inference (and FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning) — Tri Dao; Daniel Haziza, Francisco Massa, Grigory Sizov and collaborators, 2023https://scholar.google.com/scholar?q=Flash-Decoding+for+long-context+inference+%28and+FlashAttention-2%3A+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning%2913. Ring Attention with Blockwise Transformers for Near-Infinite Context — Hao Liu, Matei Zaharia, Pieter Abbeel, 2023https://scholar.google.com/scholar?q=Ring+Attention+with+Blockwise+Transformers+for+Near-Infinite+Context14. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving — Zihao Ye et al., 2025https://scholar.google.com/scholar?q=FlashInfer%3A+Efficient+and+Customizable+Attention+Engine+for+LLM+Inference+Serving15. FlexAttention: The Flexibility of PyTorch with the Performance of FlashAttention — Juechu Dong et al. (PyTorch team), 2024https://scholar.google.com/scholar?q=FlexAttention%3A+The+Flexibility+of+PyTorch+with+the+Performance+of+FlashAttention16. TileLang: A Composable Tiled Programming Model for AI Systems — Lei Wang et al., 2025https://scholar.google.com/scholar?q=TileLang%3A+A+Composable+Tiled+Programming+Model+for+AI+Systems17. Mirage: A Multi-Level Superoptimizer for Tensor Programs — Mengdi Wu, Xinhao Cheng, et al., 2025https://scholar.google.com/scholar?q=Mirage%3A+A+Multi-Level+Superoptimizer+for+Tensor+Programs18. Neptune: Advanced ML Operator Fusion for Locality and Parallelism on GPUs — Yifan Zhao et al., 2025https://scholar.google.com/scholar?q=Neptune%3A+Advanced+ML+Operator+Fusion+for+Locality+and+Parallelism+on+GPUs19. Souffle: Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions — Bojian Zheng et al., 2024https://scholar.google.com/scholar?q=Souffle%3A+Optimizing+Deep+Learning+Inference+via+Global+Analysis+and+Tensor+Expressions20. Lean Attention / Stream-K style decomposition for decode attention — Rya Sanovar et al., 2024https://scholar.google.com/scholar?q=Lean+Attention+%2F+Stream-K+style+decomposition+for+decode+attention21. DeepGEMM / DeepSeek FP8 fine-grained scaling — DeepSeek-AI, 2025https://scholar.google.com/scholar?q=DeepGEMM+%2F+DeepSeek+FP8+fine-grained+scalingInteractive Visualization: RedFuser: Automatic Fusion of Cascaded Reductions for AI Accelerators16 min
- 98× Faster LLM Routing with Flash Attention for vLLM Semantic RouterThis episode examines a March 2026 paper on the vLLM Semantic Router. The paper claims to cut routing time for an 8K-token prompt from about 4.9 seconds to 50 milliseconds without a dedicated GPU. It explains how the router runs jailbreak, PII and domain classifiers inside Envoy before the first token. It also explains why those classifiers must fit in a few hundred megabytes of GPU memory left over by vLLM, since quadratic attention score and mask matrices blow past that budget. The discussion covers the three-stage fix. The first stage is FlashAttention through a custom AMD Composable Kernel operator for ONNX Runtime on MI300X. The second is compressing prompts to about 512 tokens with classical sentence scoring instead of a neural compressor. The third is near-streaming request handling with zero-copy JSON parsing. The hosts treat the 98× figure as a pipeline-versus-pipeline comparison and try to separate how much comes from the attention kernel and how much from simply moving off the CPU. Listeners interested in inference infrastructure get a clear look at the hidden latency costs that sit in front of an LLM.Sources:1. 98$\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router — Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo, Haichen Zhang, Huamin Chen, 2026http://arxiv.org/abs/2603.126462. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness3. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Tri Dao, 2023https://scholar.google.com/scholar?q=FlashAttention-2%3A+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning4. Online normalizer calculation for softmax — Maxim Milakov, Natalia Gimelshein, 2018https://scholar.google.com/scholar?q=Online+normalizer+calculation+for+softmax5. Smarter Hardware, Not Harder Math: Warner et al., Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (ModernBERT) — Benjamin Warner, Antoine Chaffin, Benjamin Clavié, et al., 2024https://scholar.google.com/scholar?q=Smarter+Hardware%2C+Not+Harder+Math%3A+Warner+et+al.%2C+Smarter%2C+Better%2C+Faster%2C+Longer%3A+A+Modern+Bidirectional+Encoder+for+Fast%2C+Memory+Efficient%2C+and+Long+Context+Finetuning+and+Inference+%28ModernBERT%296. Parsing Gigabytes of JSON per Second — Geoff Langdale, Daniel Lemire, 2019https://scholar.google.com/scholar?q=Parsing+Gigabytes+of+JSON+per+Second7. The Tail at Scale — Jeffrey Dean, Luiz André Barroso, 2013https://scholar.google.com/scholar?q=The+Tail+at+Scale8. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention9. Gandiva: Introspective Cluster Scheduling for Deep Learning — Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, et al., 2018https://scholar.google.com/scholar?q=Gandiva%3A+Introspective+Cluster+Scheduling+for+Deep+Learning10. Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications — Peifeng Yu, Mosharaf Chowdhury, 2020https://scholar.google.com/scholar?q=Salus%3A+Fine-Grained+GPU+Sharing+Primitives+for+Deep+Learning+Applications11. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference — Benjamin Warner, Antoine Chaffin, Benjamin Clavié, et al., 2024https://scholar.google.com/scholar?q=Smarter%2C+Better%2C+Faster%2C+Longer%3A+A+Modern+Bidirectional+Encoder+for+Fast%2C+Memory+Efficient%2C+and+Long+Context+Finetuning+and+Inference12. Dynamic Sparse Flash Attention / FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024https://scholar.google.com/scholar?q=Dynamic+Sparse+Flash+Attention+%2F+FlashAttention-3%3A+Fast+and+Accurate+Attention+with+Asynchrony+and+Low-precision13. Longformer: The Long-Document Transformer — Iz Beltagy, Matthew E. Peters, Arman Cohan, 2020https://scholar.google.com/scholar?q=Longformer%3A+The+Long-Document+Transformer14. Self-Attention Does Not Need O(n^2) Memory — Markus N. Rabe, Charles Staats, 2021https://scholar.google.com/scholar?q=Self-Attention+Does+Not+Need+O%28n%5E2%29+Memory15. RouteLLM: Learning to Route LLMs with Preference Data — Isaac Ong et al., 2024https://scholar.google.com/scholar?q=RouteLLM%3A+Learning+to+Route+LLMs+with+Preference+Data16. Jailbreaking Black Box Large Language Models in Twenty Queries (PAIR) and JailbreakBench — Patrick Chao et al., 2023-2024https://scholar.google.com/scholar?q=Jailbreaking+Black+Box+Large+Language+Models+in+Twenty+Queries+%28PAIR%29+and+JailbreakBench17. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression — Huiqiang Jiang et al., 2024https://scholar.google.com/scholar?q=LongLLMLingua%3A+Accelerating+and+Enhancing+LLMs+in+Long+Context+Scenarios+via+Prompt+Compression18. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations — Hakan Inan et al., 2023https://scholar.google.com/scholar?q=Llama+Guard%3A+LLM-based+Input-Output+Safeguard+for+Human-AI+Conversations19. NVIDIA MPS / MIG and AMD Instinct compute partitioning (SPX/CPX/NPS) documentation — NVIDIA; AMD, 2023-2025https://scholar.google.com/scholar?q=NVIDIA+MPS+%2F+MIG+and+AMD+Instinct+compute+partitioning+%28SPX%2FCPX%2FNPS%29+documentation20. Orca / Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference with Chunked Prefills — Amey Agrawal et al., 2024https://scholar.google.com/scholar?q=Orca+%2F+Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Chunked+Prefills21. simdjson: Parsing Gigabytes of JSON per Second — Geoff Langdale, Daniel Lemire, 2019https://scholar.google.com/scholar?q=simdjson%3A+Parsing+Gigabytes+of+JSON+per+SecondInteractive Visualization: 98× Faster LLM Routing with Flash Attention for vLLM Semantic Router18 min
- Model-Aware Tokenizer Transfer: Fitting New Vocabularies to Frozen LLMsThis episode explores Model-Aware Tokenizer Transfer, a method for swapping a language model's tokenizer without breaking it. The motivating example is Gemma 3 on Ukrainian, where a Ukrainian-focused vocabulary raises compression from about 2.98 to 4.44 characters per token, a near 50% inference speedup. The cost is more than 80,000 new tokens with no trained embeddings. The discussion explains why the transformer body is tied to its original vocabulary. It then contrasts earlier approaches with the new one. WECHSEL, FOCUS, Transtokenizers and TokAlign build new vectors by averaging similar old ones. Zero-Shot Tokenizer Transfer and Token Distillation use the model's own behavior. The paper's approach, Attention Influence Modeling, treats the original model as a frozen teacher and trains only the new-token embeddings so the student's attention outputs match the teacher's. The two tokenizers segment text differently, so the comparison is made over the smallest character spans that neither tokenizer splits. The episode also flags an unverified assumption that the feed-forward layers will handle correctly placed inputs, and it sets up a test of whether attention is really what drives the gains.Sources:1. Model-Aware Tokenizer Transfer — Mykola Haltiuk, Aleksander Smywinski-Pohl, 2025http://arxiv.org/abs/2510.219542. As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages — Wietse de Vries, Malvina Nissim, 2020 (arXiv; Findings of ACL 2021)https://scholar.google.com/scholar?q=As+Good+as+New.+How+to+Successfully+Recycle+English+GPT-2+to+Make+Models+for+Other+Languages3. WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models — Benjamin Minixhofer, Fabian Paischer, Navid Rekabsaz, 2022 (NAACL)https://scholar.google.com/scholar?q=WECHSEL%3A+Effective+initialization+of+subword+embeddings+for+cross-lingual+transfer+of+monolingual+language+models4. FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models — Konstantin Dobler, Gerard de Melo, 2023 (EMNLP)https://scholar.google.com/scholar?q=FOCUS%3A+Effective+Embedding+Initialization+for+Monolingual+Specialization+of+Multilingual+Models5. Zero-Shot Tokenizer Transfer — Benjamin Minixhofer, Edoardo M. Ponti, Ivan Vulić, 2024 (NeurIPS)https://scholar.google.com/scholar?q=Zero-Shot+Tokenizer+Transfer6. Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer — Sergey Zagoruyko, Nikos Komodakis, 2017 (ICLR)https://scholar.google.com/scholar?q=Paying+More+Attention+to+Attention%3A+Improving+the+Performance+of+Convolutional+Neural+Networks+via+Attention+Transfer7. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers — Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou, 2020 (NeurIPS)https://scholar.google.com/scholar?q=MiniLM%3A+Deep+Self-Attention+Distillation+for+Task-Agnostic+Compression+of+Pre-Trained+Transformers8. LoLCATs: On Low-Rank Linearizing of Large Language Models — Michael Zhang, Simran Arora, Rahul Chalamala, Alan Wu, Benjamin Spector, Aaryan Singhal, Krithik Ramesh, Christopher Ré, 2024https://scholar.google.com/scholar?q=LoLCATs%3A+On+Low-Rank+Linearizing+of+Large+Language+Models9. Token Distillation: Attention-aware input embeddings for new tokens — Konstantin Dobler, Desmond Elliott, Gerard de Melo, 2026https://scholar.google.com/scholar?q=Token+Distillation%3A+Attention-aware+input+embeddings+for+new+tokens10. Trans-tokenization and cross-lingual vocabulary transfers: Language adaptation of LLMs for low-resource NLP — François Remy et al., 2024https://scholar.google.com/scholar?q=Trans-tokenization+and+cross-lingual+vocabulary+transfers%3A+Language+adaptation+of+LLMs+for+low-resource+NLP11. TokAlign: Efficient vocabulary adaptation via token alignment — Chong Li, Jiajun Zhang, Chengqing Zong, 2025https://scholar.google.com/scholar?q=TokAlign%3A+Efficient+vocabulary+adaptation+via+token+alignment12. Transformer Feed-Forward Layers Are Key-Value Memories — Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy, 2021https://scholar.google.com/scholar?q=Transformer+Feed-Forward+Layers+Are+Key-Value+Memories13. From Tokens to Words: On the Inner Lexicon of LLMs — Guy Kaplan, Matanel Oren, Yuval Reif, Roy Schwartz, 2024https://scholar.google.com/scholar?q=From+Tokens+to+Words%3A+On+the+Inner+Lexicon+of+LLMs14. Byte Latent Transformer: Patches Scale Better Than Tokens — Artidoro Pagnoni et al., 2025https://scholar.google.com/scholar?q=Byte+Latent+Transformer%3A+Patches+Scale+Better+Than+Tokens15. Tokenizer Choice for LLM Training: Negligible or Crucial? — Mehdi Ali et al., 2024https://scholar.google.com/scholar?q=Tokenizer+Choice+for+LLM+Training%3A+Negligible+or+Crucial%3F16. Large Vocabulary Size Improves Large Language Models — Sho Takase et al., 2025https://scholar.google.com/scholar?q=Large+Vocabulary+Size+Improves+Large+Language+ModelsInteractive Visualization: Model-Aware Tokenizer Transfer: Fitting New Vocabularies to Frozen LLMs18 min
- Abstract Chain-of-Thought: Reasoning with Discrete Latent TokensThis episode examines "Thinking Without Words," a paper proposing Abstract Chain-of-Thought, which replaces long written-out reasoning with a short run of up to 128 tokens drawn from a new 64-token codebook that has no meaning in English. It sets the approach against its predecessors: pause tokens, which tend to fail when added after pre-training, stepwise internalization, and continuous latent methods such as Coconut and CODI. Against these, it presents the claim that the method works through post-training alone, with ordinary discrete decoding. The motivation is that verbal reasoning is costly (a Qwen3-8B model trained with RL averages roughly 1,671 tokens per MATH-500 problem) and may not faithfully reflect what drives the answer. The discussion also covers the warm-up stage, where an attention mask hides the verbal reasoning from the answer so the new tokens have to carry the content. It treats the paper's "information bottleneck" framing carefully, noting that it motivates the structure without anything being measured or optimized in information-theoretic terms.Sources:1. Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought — Keshav Ramji, Tahira Naseem, Ramón Fernandez Astudillo, 2026http://arxiv.org/abs/2604.227092. Training Large Language Models to Reason in a Continuous Latent Space (Coconut) — Shibo Hao et al., 2024/2025https://scholar.google.com/scholar?q=Training+Large+Language+Models+to+Reason+in+a+Continuous+Latent+Space+%28Coconut%293. CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation — Zhenyi Shen et al., 2025https://scholar.google.com/scholar?q=CODI%3A+Compressing+Chain-of-Thought+into+Continuous+Space+via+Self-Distillation4. Think before you speak: Training Language Models with Pause Tokens — Sachin Goyal et al., 2024https://scholar.google.com/scholar?q=Think+before+you+speak%3A+Training+Language+Models+with+Pause+Tokens5. L1: Controlling How Long a Reasoning Model Thinks with Reinforcement Learning — Pranjal Aggarwal, Sean Welleck, 2025https://scholar.google.com/scholar?q=L1%3A+Controlling+How+Long+a+Reasoning+Model+Thinks+with+Reinforcement+Learning6. Let's Think Dot by Dot: Hidden Computation in Transformer Language Models — Jacob Pfau, William Merrill, Samuel R. Bowman, 2024https://scholar.google.com/scholar?q=Let%27s+Think+Dot+by+Dot%3A+Hidden+Computation+in+Transformer+Language+Models7. Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety — Tomek Korbak et al., 2025https://scholar.google.com/scholar?q=Chain+of+Thought+Monitorability%3A+A+New+and+Fragile+Opportunity+for+AI+Safety8. Soft Tokens, Hard Truths — Natasha Butt et al., 2025https://scholar.google.com/scholar?q=Soft+Tokens%2C+Hard+Truths9. Learning to Compress Prompts with Gist Tokens — Jesse Mu et al., 2023https://scholar.google.com/scholar?q=Learning+to+Compress+Prompts+with+Gist+Tokens10. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — Miles Turpin et al., 2023https://scholar.google.com/scholar?q=Language+Models+Don%27t+Always+Say+What+They+Think%3A+Unfaithful+Explanations+in+Chain-of-Thought+PromptingInteractive Visualization: Abstract Chain-of-Thought: Reasoning with Discrete Latent Tokens17 min
- Workload–Router–Pool: Unifying LLM Routing, Models, and GPU FleetsThis episode explores a vision paper arguing that choosing a model, routing a request, and sizing a GPU fleet are a single coupled problem. The paper organizes LLM inference into workload, router, and pool dimensions, and it presents a 3-by-3 matrix of their interactions. The discussion defines the latency and serving terms (TTFT, TPOT, prefill versus decode, KV-cache limits, warm versus cold requests). It then examines how workload shape drives the design. The examples are short chat requests that waste long-context KV slots, and agent sessions whose caches get evicted during tool pauses. It also covers how routers can infer workload type from cheap signals, how feedback-driven adaptation differs for chat and agents, and a three-tier vision-language cascade with a projected 78 percent cost reduction. Throughout, the episode stresses that the paper reports no new experiments. Listeners get a way to separate measured results from simulated or projected ones.Sources:1. The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project — Huamin Chen, Xunzhuo Liu, Bowei He, Fuyuan Lyu, Yankai Chen, Xue Liu, Yuhan Liu, Junchen Jiang, 2026http://arxiv.org/abs/2603.213542. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance — Lingjiao Chen, Matei Zaharia, James Zou, 2023https://scholar.google.com/scholar?q=FrugalGPT%3A+How+to+Use+Large+Language+Models+While+Reducing+Cost+and+Improving+Performance3. RouteLLM: Learning to Route LLMs with Preference Data — Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, Ion Stoica, 2024https://scholar.google.com/scholar?q=RouteLLM%3A+Learning+to+Route+LLMs+with+Preference+Data4. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing — Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks V. S. Lakshmanan, Ahmed Hassan Awadallah, 2024 (ICLR)https://scholar.google.com/scholar?q=Hybrid+LLM%3A+Cost-Efficient+and+Quality-Aware+Query+Routing5. RouterBench: A Benchmark for Multi-LLM Routing System — Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, Shriyash Kaustubh Upadhyay, 2024https://scholar.google.com/scholar?q=RouterBench%3A+A+Benchmark+for+Multi-LLM+Routing+System6. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini, 2024 (ISCA)https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting7. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024 (OSDI)https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving8. Vidur: A Large-Scale Simulation Framework for LLM Inference — Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachandran Ramjee, Alexey Tumanov, 2024 (MLSys)https://scholar.google.com/scholar?q=Vidur%3A+A+Large-Scale+Simulation+Framework+for+LLM+Inference9. Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity — Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, Ion Stoica, 2024https://scholar.google.com/scholar?q=M%C3%A9lange%3A+Cost+Efficient+Large+Language+Model+Serving+by+Exploiting+GPU+Heterogeneity10. BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems — Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, Xiaowen Chu, 2024https://scholar.google.com/scholar?q=BurstGPT%3A+A+Real-world+Workload+Dataset+to+Optimize+LLM+Serving+Systems11. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2024 (FAST 2025)https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving12. LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset — Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, Hao Zhang, 2023https://scholar.google.com/scholar?q=LMSYS-Chat-1M%3A+A+Large-Scale+Real-World+LLM+Conversation+Dataset13. Orca: A Distributed Serving System for Transformer-Based Generative Models and Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Gyeong-In Yu et al. (Orca, OSDI 2022); Woosuk Kwon et al. (vLLM, SOSP 2023), 2022 / 2023https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models+and+Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%2914. GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching — Sajal Regmi, Chetan Phakami Pun, 2024https://scholar.google.com/scholar?q=GPT+Semantic+Cache%3A+Reducing+LLM+Costs+and+Latency+via+Semantic+Embedding+Caching15. MeanCache: User-Centric Semantic Caching for LLM Web Services — Waris Gill, Mohamed Elidrisi, Pallavi Kalapatapu, Ammar Ahmed, Ali Anwar, Muhammad Ali Gulzar, 2024https://scholar.google.com/scholar?q=MeanCache%3A+User-Centric+Semantic+Caching+for+LLM+Web+Services16. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (completion cache component) — Lingjiao Chen, Matei Zaharia, James Zou, 2023https://scholar.google.com/scholar?q=FrugalGPT%3A+How+to+Use+Large+Language+Models+While+Reducing+Cost+and+Improving+Performance+%28completion+cache+component%2917. Efficient Memory Management for Large Language Model Serving with PagedAttention (prefix/KV reuse contrast) — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023 (SOSP)https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28prefix%2FKV+reuse+contrast%2918. AutoMix: Automatically Mixing Language Models — Pranjal Aggarwal et al., 2023https://scholar.google.com/scholar?q=AutoMix%3A+Automatically+Mixing+Language+Models19. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving — Yinmin Zhong et al., 2024https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+LLM+Serving20. ServeGen: Workload Characterization of Large Language Model Serving in Production — Yuxin Wang et al., 2025https://scholar.google.com/scholar?q=ServeGen%3A+Workload+Characterization+of+Large+Language+Model+Serving+in+Production21. GPTCache / MeanCache: user-centric semantic caching for LLM web services — Waris Gill et al. (MeanCache); Bang (GPTCache), 2023–2025https://scholar.google.com/scholar?q=GPTCache+%2F+MeanCache%3A+user-centric+semantic+caching+for+LLM+web+services22. Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning — Haozhen Zhang et al., 2025https://scholar.google.com/scholar?q=Router-R1%3A+Teaching+LLMs+Multi-Round+Routing+and+Aggregation+via+Reinforcement+Learning23. Preble: Efficient Distributed Prompt Scheduling for LLM Serving — Vikranth Srivatsa et al., 2024https://scholar.google.com/scholar?q=Preble%3A+Efficient+Distributed+Prompt+Scheduling+for+LLM+ServingInteractive Visualization: Workload–Router–Pool: Unifying LLM Routing, Models, and GPU Fleets0 min
- Coconut: Reasoning in Continuous Latent Space Instead of WordsThis episode examines the Coconut paper, which trains a language model to feed its last hidden state back in as the next input, so it reasons in continuous "thoughts" instead of written words. It first covers why chain-of-thought forces every step through a single discrete token, then sets Coconut against nearby approaches. Those are pause tokens, filler tokens, iCoT's gradual removal of reasoning tokens, and Tree of Thoughts, which runs its search outside the model. The discussion explains the curriculum that swaps written steps for latent ones, and why a vector rather than a token lets gradients flow through the whole chain. It also notes the cost of n+1 sequential forward passes in training. The central claim is that latent thoughts can hold several candidates at once, like breadth-first search, tested on the authors' own ProsQA graph-reasoning dataset alongside ProntoQA and GSM8k. The hosts stay skeptical, pointing out that ProsQA was built to favor search and that the search interpretation is read off a trained model rather than trained for.Sources:1. Training Large Language Models to Reason in a Continuous Latent Space — Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, Yuandong Tian, 2024http://arxiv.org/abs/2412.067692. Training Large Language Models to Reason in a Continuous Latent Space — Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, Yuandong Tian, 2024 (arXiv 2412.06769; revised through 2026)https://scholar.google.com/scholar?q=Training+Large+Language+Models+to+Reason+in+a+Continuous+Latent+Space3. Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought — Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, Yuandong Tian, 2025https://scholar.google.com/scholar?q=Reasoning+by+Superposition%3A+A+Theoretical+Perspective+on+Chain+of+Continuous+Thought4. Tree of Thoughts: Deliberate Problem Solving with Large Language Models — Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, Karthik Narasimhan, 2023https://scholar.google.com/scholar?q=Tree+of+Thoughts%3A+Deliberate+Problem+Solving+with+Large+Language+Models5. Reasoning with Language Model is Planning with World Model (RAP) — Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, Zhiting Hu, 2023https://scholar.google.com/scholar?q=Reasoning+with+Language+Model+is+Planning+with+World+Model+%28RAP%296. From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step — Yuntian Deng, Yejin Choi, Stuart Shieber, 2024https://scholar.google.com/scholar?q=From+Explicit+CoT+to+Implicit+CoT%3A+Learning+to+Internalize+CoT+Step+by+Step7. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Tom Goldstein, 2025https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach8. Emergence of Superposition: Unveiling the Training Dynamics of Chain of Continuous Thought — Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, Yuandong Tian, 2025https://scholar.google.com/scholar?q=Emergence+of+Superposition%3A+Unveiling+the+Training+Dynamics+of+Chain+of+Continuous+Thought9. Think before you speak: Training language models with pause tokens — Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, Vaishnavh Nagarajan, 2023https://scholar.google.com/scholar?q=Think+before+you+speak%3A+Training+language+models+with+pause+tokens10. Let's Think Dot by Dot: Hidden Computation in Transformer Language Models — Jacob Pfau, William Merrill, Samuel R. Bowman, 2024https://scholar.google.com/scholar?q=Let%27s+Think+Dot+by+Dot%3A+Hidden+Computation+in+Transformer+Language+Models11. Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping — Lucas Lehnert, Sainbayar Sukhbaatar, Paul McVay, Michael Rabbat, Yuandong Tian, 2024https://scholar.google.com/scholar?q=Beyond+A%2A%3A+Better+Planning+with+Transformers+via+Search+Dynamics+Bootstrapping12. Stream of Search (SoS): Learning to Search in Language — Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, Noah D. Goodman, 2024https://scholar.google.com/scholar?q=Stream+of+Search+%28SoS%29%3A+Learning+to+Search+in+Language13. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — Miles Turpin, Julian Michael, Ethan Perez, Samuel Bowman, 2024https://scholar.google.com/scholar?q=Language+Models+Don%27t+Always+Say+What+They+Think%3A+Unfaithful+Explanations+in+Chain-of-Thought+Prompting14. Large Concept Models: Language Modeling in a Sentence Representation Space — Loïc Barrault et al., 2024https://scholar.google.com/scholar?q=Large+Concept+Models%3A+Language+Modeling+in+a+Sentence+Representation+Space15. Energy-Based Transformers are Scalable Learners and Thinkers — Alexi Gladstone et al., 2025https://scholar.google.com/scholar?q=Energy-Based+Transformers+are+Scalable+Learners+and+Thinkers16. Recurrent Memory Transformer — Aydar Bulatov, Yury Kuratov, Mikhail Burtsev, 2022https://scholar.google.com/scholar?q=Recurrent+Memory+TransformerInteractive Visualization: Coconut: Reasoning in Continuous Latent Space Instead of Words16 min
- vLLM Semantic Router: Signal-Driven Routing for Mixture-of-Modality ModelsThis episode examines the vLLM Semantic Router paper, which treats request routing as a layer of named "signals" (keywords, language, domain, complexity, modality, PII, jailbreak, fact-checking need) combined into Boolean decision policies that pick a model and run plugins such as semantic caching or hallucination checks. It compares this with earlier routers like RouteLLM, RouterDC and AutoMix, which the system wraps as selection algorithms rather than competing with. It also separates the paper's "Mixture-of-Modality" and "Mixture-of-Models" from sparse Mixture-of-Experts: here the experts are whole, separately trained models, and the gates are auditable symbolic rules rather than gradient-trained ones. The discussion covers the signal machinery: cheap heuristic signals versus learned BERT-style classifiers, and a decision engine that computes only the signals a policy references. The episode also questions the gap between the paper's strong claims, such as one system serving privacy-regulated, cost-optimized and multi-cloud deployments, and its evidence. The latency savings and the information-theory framing are design estimates or schematics, not measured results. Listeners get a clear view of what this routing architecture offers and what the paper does not yet show about routing quality.Sources:1. vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models — Xunzhuo Liu, Huamin Chen, Samzong Lu, Yossi Ovadia, Guohong Wen, Hao Wu, Zhengda Tan, Jintao Zhang, Senan Zedan, Yehudit Kerido, Liav Weiss, Haichen Zhang, Bishen Yu, Asaad Balum, Noa Limoy, Abdallah Samara, Baofa Fan, Brent Salisbury, Ryan Cook, Zhijie Wang, Qiping Pan, Rehan Khan, Avishek Goswami, Houston H. Zhang, Shuyi Wang, Ziang Tang, Fang Han, Zohaib Hassan, Jianqiao Zheng, Avinash Changrani, Xue, Liu, Bowei He, 2026http://arxiv.org/abs/2603.044442. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance — Lingjiao Chen, Matei Zaharia, James Zou, 2023https://scholar.google.com/scholar?q=FrugalGPT%3A+How+to+Use+Large+Language+Models+While+Reducing+Cost+and+Improving+Performance3. RouteLLM: Learning to Route LLMs with Preference Data — Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, Ion Stoica, 2024https://scholar.google.com/scholar?q=RouteLLM%3A+Learning+to+Route+LLMs+with+Preference+Data4. RouterBench: A Benchmark for Multi-LLM Routing System — Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, Shriyash Kaustubh Upadhyay, 2024https://scholar.google.com/scholar?q=RouterBench%3A+A+Benchmark+for+Multi-LLM+Routing+System5. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing — Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V.S. Lakshmanan, Ahmed Hassan Awadallah, 2024https://scholar.google.com/scholar?q=Hybrid+LLM%3A+Cost-Efficient+and+Quality-Aware+Query+Routing6. A Mathematical Theory of Communication — Claude E. Shannon, 1948https://scholar.google.com/scholar?q=A+Mathematical+Theory+of+Communication7. A Symbolic Analysis of Relay and Switching Circuits — Claude E. Shannon, 1938https://scholar.google.com/scholar?q=A+Symbolic+Analysis+of+Relay+and+Switching+Circuits8. The Information Bottleneck Method — Naftali Tishby, Fernando C. Pereira, William Bialek, 1999https://scholar.google.com/scholar?q=The+Information+Bottleneck+Method9. Elements of Information Theory — Thomas M. Cover, Joy A. Thomas, 2006 (2nd edition; 1st edition 1991)https://scholar.google.com/scholar?q=Elements+of+Information+Theory10. Neurosymbolic Programming — Swarat Chaudhuri, Kevin Ellis, Oleksandr Polozov, Rishabh Singh, Armando Solar-Lezama, Yisong Yue, 2021https://scholar.google.com/scholar?q=Neurosymbolic+Programming11. DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines — Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, Christopher Potts, 2023 (ICLR 2024)https://scholar.google.com/scholar?q=DSPy%3A+Compiling+Declarative+Language+Model+Calls+into+State-of-the-Art+Pipelines12. NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails — Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, Jonathan Cohen, 2023https://scholar.google.com/scholar?q=NeMo+Guardrails%3A+A+Toolkit+for+Controllable+and+Safe+LLM+Applications+with+Programmable+Rails13. LMQL: Prompting Is Programming: A Query Language for Large Language Models — Luca Beurer-Kellner, Marc Fischer, Martin Vechev, 2023https://scholar.google.com/scholar?q=LMQL%3A+Prompting+Is+Programming%3A+A+Query+Language+for+Large+Language+Models14. Scallop: A Language for Neurosymbolic Programming — Ziyang Li, Jiani Huang, Mayur Naik, 2023 (PLDI)https://scholar.google.com/scholar?q=Scallop%3A+A+Language+for+Neurosymbolic+Programming15. RouterArena / RouterEval and related open benchmarks for LLM routers — Various (RouterBench authors; RouterArena team), 2024–2026https://scholar.google.com/scholar?q=RouterArena+%2F+RouterEval+and+related+open+benchmarks+for+LLM+routers16. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines — Omar Khattab et al., 2023https://scholar.google.com/scholar?q=DSPy%3A+Compiling+Declarative+Language+Model+Calls+into+Self-Improving+Pipelines17. Llama Guard / Llama Guard 3, and Presidio — Inan et al. (Meta); Microsoft Presidio team, 2023–2024https://scholar.google.com/scholar?q=Llama+Guard+%2F+Llama+Guard+3%2C+and+Presidio18. Universal Model Routing for Efficient LLM Inference / Model-SAT and other routers that generalize to new models — Various, 2025https://scholar.google.com/scholar?q=Universal+Model+Routing+for+Efficient+LLM+Inference+%2F+Model-SAT+and+other+routers+that+generalize+to+new+models19. Conformal / calibrated routing and selective prediction with abstention for LLM cascades — Various (e.g., conformal prediction for cascades; selective classification literature), 2023–2025https://scholar.google.com/scholar?q=Conformal+%2F+calibrated+routing+and+selective+prediction+with+abstention+for+LLM+cascades20. Information Bottleneck principle; and Tishby, Pereira, Bialek — Naftali Tishby, Fernando Pereira, William Bialek, 1999https://scholar.google.com/scholar?q=Information+Bottleneck+principle%3B+and+Tishby%2C+Pereira%2C+Bialek21. Prompt injection and multi-turn jailbreak benchmarks (e.g., Crescendo, many-shot jailbreaking, HarmBench) — Russinovich et al.; Anil et al.; Mazeika et al., 2024https://scholar.google.com/scholar?q=Prompt+injection+and+multi-turn+jailbreak+benchmarks+%28e.g.%2C+Crescendo%2C+many-shot+jailbreaking%2C+HarmBench%2922. Not All Layers / Mixture-of-Agents and LLM ensembling (Mixture-of-Agents; LLM-Blender) — Wang et al.; Jiang et al., 2024https://scholar.google.com/scholar?q=Not+All+Layers+%2F+Mixture-of-Agents+and+LLM+ensembling+%28Mixture-of-Agents%3B+LLM-Blender%29Interactive Visualization: vLLM Semantic Router: Signal-Driven Routing for Mixture-of-Modality Models19 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.