AI Post Transformers

This episode examines Kimi K3, Moonshot AI's open-weight frontier model boasting 2.8 trillion total parameters with only 104 billion active per token, a million-token context window, and native multimodal training from the ground up. The discussion traces the architectural lineage behind the model's claimed 2.5x scaling-efficiency gain over its predecessor Kimi K2, connecting its Mixture-of-Experts design back to Shazeer's 2017 sparsely-gated MoE work and contrasting its hybrid attention approach with the limits of standard residual connections from the 2015 ResNet paper. It also unpacks the systems-engineering side of running a model this large, particularly how Expert Parallelism turns token routing into a datacenter networking problem once hundreds of experts are sharded across GPUs. Listeners get a clear breakdown of why native multimodal training tends to be more stable than bolting a pretrained vision encoder onto a text-only model after the fact. The episode sets up a deeper dive into which of K3's four credited innovations — Kimi Delta Attention, Attention Residuals, Stable LatentMoE, and refined training recipes — is actually doing the heavy lifting.

Sources:

1. Kimi K3 Goes Trillion-Scale With Sparse MoE Efficiency

https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf

2. Deep Residual Learning for Image Recognition — Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2015 (CVPR 2016)

https://scholar.google.com/scholar?q=Deep+Residual+Learning+for+Image+Recognition

3. DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging — Matteo Pagliardini, Amirkeivan Mohtashami, Francois Fleuret, Martin Jaggi, 2024

https://scholar.google.com/scholar?q=DenseFormer%3A+Enhancing+Information+Flow+in+Transformers+via+Depth+Weighted+Averaging

4. Value Residual Learning for Alleviating Attention Concentration in Transformers — Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, Zhenzhong Lan (and collaborators), 2024

https://scholar.google.com/scholar?q=Value+Residual+Learning+for+Alleviating+Attention+Concentration+in+Transformers

5. Hyper-Connections — Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, Xun Zhou (ByteDance Seed), 2024

https://scholar.google.com/scholar?q=Hyper-Connections

6. Attention Residuals — Kimi Team, 2026

https://scholar.google.com/scholar?q=Attention+Residuals

7. Kimi Linear: An Expressive, Efficient Attention Architecture — Kimi Team et al., 2025 (arXiv:2510.26692)

https://scholar.google.com/scholar?q=Kimi+Linear%3A+An+Expressive%2C+Efficient+Attention+Architecture

8. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI, 2024

https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model

9. ReplaySSM: Cache SSM Inputs, Not State — Dao AI Lab, 2026 (blog)

https://scholar.google.com/scholar?q=ReplaySSM%3A+Cache+SSM+Inputs%2C+Not+State

Interactive Visualization: Kimi K3 Goes Trillion-Scale With Sparse MoE Efficiency

This episode examines HOPE (Hilbert Operator for Progressive Encoding), a structured pruning framework from Google DeepMind and UC Berkeley researchers that treats network compression as a diagnostic tool for understanding what deep networks actually learn, rather than just a deployment optimization. The discussion traces the approach's roots to the Information Bottleneck principle while carefully distinguishing HOPE's falsifiable measurement machinery from that unproven theory, and covers why magnitude-based pruning fails due to scale symmetry in batch-normalized networks, and how data-dependent pruning can quietly degrade long-tail class performance. The core innovation discussed is representing neurons as objects in a Hilbert space—comparing what function each neuron computes rather than the size of its weights—using only batch norm statistics already stored in a checkpoint, with no forward passes on real data and no hyperparameter tuning required. Listeners interested in interpretability, pruning theory, or the ongoing debate over why deep learning generalizes will find the hosts' back-and-forth on contested claims particularly engaging, as they push back on overstating the Information Bottleneck's explanatory power while crediting HOPE's mathematically rigorous, data-free approach to isolating a network's predictive core.

Sources:

1. Hilbert Operator Reveals What Networks Actually Learn

https://arxiv.org/pdf/2607.21366

2. Optimal Brain Damage — Yann LeCun, John S. Denker, Sara A. Solla, 1989

https://scholar.google.com/scholar?q=Optimal+Brain+Damage

3. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — Jonathan Frankle, Michael Carbin, 2019

https://scholar.google.com/scholar?q=The+Lottery+Ticket+Hypothesis%3A+Finding+Sparse%2C+Trainable+Neural+Networks

4. Distilling the Knowledge in a Neural Network — Geoffrey Hinton, Oriol Vinyals, Jeff Dean, 2015

https://scholar.google.com/scholar?q=Distilling+the+Knowledge+in+a+Neural+Network

5. SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot — Elias Frantar, Dan Alistarh, 2023

https://scholar.google.com/scholar?q=SparseGPT%3A+Massive+Language+Models+Can+Be+Accurately+Pruned+in+One-Shot

6. Priors for Infinite Networks (in Bayesian Learning for Neural Networks) — Radford M. Neal, 1996

https://scholar.google.com/scholar?q=Priors+for+Infinite+Networks+%28in+Bayesian+Learning+for+Neural+Networks%29

7. Neural Tangent Kernel: Convergence and Generalization in Neural Networks — Arthur Jacot, Franck Gabriel, Clément Hongler, 2018

https://scholar.google.com/scholar?q=Neural+Tangent+Kernel%3A+Convergence+and+Generalization+in+Neural+Networks

8. Fourier Neural Operator for Parametric Partial Differential Equations — Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar, 2021

https://scholar.google.com/scholar?q=Fourier+Neural+Operator+for+Parametric+Partial+Differential+Equations

9. Measuring Statistical Dependence with Hilbert-Schmidt Norms — Arthur Gretton, Olivier Bousquet, Alex Smola, Bernhard Schölkopf, 2005

https://scholar.google.com/scholar?q=Measuring+Statistical+Dependence+with+Hilbert-Schmidt+Norms

10. What Do Compressed Deep Neural Networks Forget? — Sara Hooker, Aaron Courville, Gregory Clark, Yann Yannakakis, Kevin Murphy, 2019

https://scholar.google.com/scholar?q=What+Do+Compressed+Deep+Neural+Networks+Forget%3F

11. Towards Monosemanticity: Decomposing Language Models with Dictionary Learning — Trenton Bricken, Adly Templeton, Joshua Batson, et al. (Anthropic), 2023

https://scholar.google.com/scholar?q=Towards+Monosemanticity%3A+Decomposing+Language+Models+with+Dictionary+Learning

12. LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu, Yelong Shen, Phillip Wallis, et al., 2022

https://scholar.google.com/scholar?q=LoRA%3A+Low-Rank+Adaptation+of+Large+Language+Models

13. Language Modeling Is Compression — Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, et al., 2023

https://scholar.google.com/scholar?q=Language+Modeling+Is+Compression

14. LLM-Pruner: On the Structural Pruning of Large Language Models — Xinyin Ma, Gongfan Fang, Xinchao Wang, 2023

https://scholar.google.com/scholar?q=LLM-Pruner%3A+On+the+Structural+Pruning+of+Large+Language+Models

Interactive Visualization: Hilbert Operator Reveals What Networks Actually Learn

This episode explores NaturalReasoning, a Meta and NYU dataset of 2.8 million reasoning questions built to break the bottleneck facing today's verifiable-reward training methods, which only work in domains like math and code where answers can be checked automatically. The discussion traces the technique's lineage from 2016 machine-translation backtranslation through Meta's 2023 instruction-backtranslation work, explaining how the team flags reasoning-rich passages in pretraining corpora and has a strong model work backward to invent the question a given passage would answer — spanning physics, economics, and social science, not just checkable benchmarks. It covers how the resulting question set is used both for straightforward distillation from a teacher model and for a self-rewarding setup where one model generates, verifies, and judges its own answers without external reward models or human labels. Listeners interested in how reasoning-focused LLM training might scale past math and code, and in the mechanics of synthetic data generation via backtranslation, will find the paper's structural argument — rethinking where training data comes from rather than just scaling it — a useful frame for where the field goes next.

Sources:

1. NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions — Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Ilia Kulikov, Kyunghyun Cho, Dong Wang, Yuandong Tian, Jason E Weston, Xian Li, 2025

http://arxiv.org/abs/2502.13124

2. Self-Alignment with Instruction Backtranslation — Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Jason Weston, Mike Lewis (Meta AI), 2023

https://scholar.google.com/scholar?q=Self-Alignment+with+Instruction+Backtranslation

3. MAmmoTH2: Scaling Instructions from the Web — Xiang Yue, Tuney Zheng, Ge Zhang, Wenhu Chen, 2024

https://scholar.google.com/scholar?q=MAmmoTH2%3A+Scaling+Instructions+from+the+Web

4. Improving Neural Machine Translation Models with Monolingual Data — Rico Sennrich, Barry Haddow, Alexandra Birch, 2016

https://scholar.google.com/scholar?q=Improving+Neural+Machine+Translation+Models+with+Monolingual+Data

5. LongForm: Optimizing Instruction Tuning for Long Text Generation with Corpus Extraction — Abdullatif Köksal, Timo Schick, Anna Korhonen, Hinrich Schütze, 2023

https://scholar.google.com/scholar?q=LongForm%3A+Optimizing+Instruction+Tuning+for+Long+Text+Generation+with+Corpus+Extraction

6. Self-Rewarding Language Models — Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, Jason Weston, 2024

https://scholar.google.com/scholar?q=Self-Rewarding+Language+Models

7. Constitutional AI: Harmlessness from AI Feedback — Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al. (Anthropic), 2022

https://scholar.google.com/scholar?q=Constitutional+AI%3A+Harmlessness+from+AI+Feedback

8. RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback — Harrison Lee, Samrat Phatale, Hassan Mansoor, et al. (Google), 2023

https://scholar.google.com/scholar?q=RLAIF%3A+Scaling+Reinforcement+Learning+from+Human+Feedback+with+AI+Feedback

9. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. (UC Berkeley / LMSYS), 2023

https://scholar.google.com/scholar?q=Judging+LLM-as-a-Judge+with+MT-Bench+and+Chatbot+Arena

10. MAmmoTH2: Scaling Instructions from the Web (WebInstruct) — Xiang Yue, Tuney Zheng, Ge Zhang, Wenhu Chen, 2024

https://scholar.google.com/scholar?q=MAmmoTH2%3A+Scaling+Instructions+from+the+Web+%28WebInstruct%29

11. STaR: Bootstrapping Reasoning with Reasoning — Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah Goodman, 2022

https://scholar.google.com/scholar?q=STaR%3A+Bootstrapping+Reasoning+with+Reasoning

12. Let's Verify Step by Step — Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe, 2023

https://scholar.google.com/scholar?q=Let%27s+Verify+Step+by+Step

13. OpenThoughts: Data Recipes for Reasoning Models — Etash Guha, Ryan Marten, Sedrick Keh, et al., 2025

https://scholar.google.com/scholar?q=OpenThoughts%3A+Data+Recipes+for+Reasoning+Models

Interactive Visualization: NaturalReasoning: Backtranslating Reasoning Questions at Scale

This episode explores ASAP, a Huawei-developed serving system for mixture-of-experts models that physically separates the attention and expert computation stages onto different hardware with non-blocking communication between them. The discussion traces the diagnosis behind the design: production traces reveal a "straggler effect" where global synchronization barriers between attention's data-parallel groups and the shared expert pool force every group to wait on the slowest one, and this imbalance is mathematically unavoidable since attention cost scales with the sum of squares of sequence lengths rather than total tokens. The hosts draw a parallel to the classic MapReduce straggler problem, framing this as a structural consequence of combining Expert Parallelism for MoE layers with Data Parallelism for attention rather than something a better scheduler could fix. Listeners get a grounded walkthrough of prefill versus decode, time-to-first-token, and why hybrid parallelism setups like DeepSeek-V3's TP=8/DP=4/EP=32 configuration create this bottleneck in the first place, setting up the paper's disaggregated architecture as a direct response to a precisely characterized failure mode rather than a speculative fix.

Sources:

1. ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill — Weiwei Chen, Shuang Chen, Lele Li, Qiang Hu, Han Li, Xin Ye, Ming Yan, Zhibin Yu, 2026

http://arxiv.org/abs/2606.22541

2. Revealing the challenges of attention-ffn disaggregation for modern MoE models and hardware systems — Guowei Liu, Hongming Li, Yaning Guo, Yongxi Lyu, Mo Zhou, Yi Liu, Zhaogeng Li, Yanpeng Wang, 2026

https://scholar.google.com/scholar?q=Revealing+the+challenges+of+attention-ffn+disaggregation+for+modern+MoE+models+and+hardware+systems

3. Expert-as-a-Service: Towards efficient, scalable, and robust large-scale MoE serving — Ziming Liu, Boyu Tian, Guoteng Wang, Zhen Jiang, et al., 2025

https://scholar.google.com/scholar?q=Expert-as-a-Service%3A+Towards+efficient%2C+scalable%2C+and+robust+large-scale+MoE+serving

4. MegaScale-Infer — Cited as [55] in ASAP, 2025/2026

https://scholar.google.com/scholar?q=MegaScale-Infer

5. Step-3 — Cited as [44] in ASAP, 2025/2026

https://scholar.google.com/scholar?q=Step-3

6. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbots — Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2025

https://scholar.google.com/scholar?q=Mooncake%3A+Trading+more+storage+for+less+computation+%E2%80%94+a+KVCache-centric+architecture+for+serving+LLM+chatbots

7. Sarathi: Efficient LLM inference by piggybacking decodes with chunked prefills — Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, Ramachandran Ramjee, 2023

https://scholar.google.com/scholar?q=Sarathi%3A+Efficient+LLM+inference+by+piggybacking+decodes+with+chunked+prefills

8. TokenWeave: Efficient compute-communication overlap for distributed LLM inference — Raja Gond, Nipun Kwatra, Ramachandran Ramjee, 2025

https://scholar.google.com/scholar?q=TokenWeave%3A+Efficient+compute-communication+overlap+for+distributed+LLM+inference

Interactive Visualization: ASAP: Disaggregating Attention and Experts for Faster MoE Prefill

This episode explores MegaScale-Infer, a ByteDance Seed/Peking University system for serving large Mixture-of-Experts models more efficiently during inference. The discussion breaks down why decode-phase attention is memory-bandwidth-bound rather than compute-bound, and how MoE's top-k expert routing—while cutting theoretical FLOPs—actually shrinks the effective batch size each expert sees, tanking GPU utilization (illustrated with a concrete Mixtral 8x22B example dropping to 25% utilization). The core proposed fix is disaggregation: physically separating attention and expert computation onto independently scaled GPU pools so attention replicas can pool enough requests to keep experts saturated, building on prior work like DistServe's prefill/decode split. Listeners interested in the gap between algorithmic efficiency claims and real-world GPU serving costs will find the roofline-model analysis a sharp corrective to "sparsity equals free lunch" thinking.

Sources:

1. MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism — Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, Xin Liu, 2025

http://arxiv.org/abs/2504.02263v1

2. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024

https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-Optimized+Large+Language+Model+Serving

3. Mooncake: Kimi's KVCache-centric Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2024

https://scholar.google.com/scholar?q=Mooncake%3A+Kimi%27s+KVCache-centric+Architecture+for+LLM+Serving

4. DeepSeek-V3 Technical Report — DeepSeek-AI (Aixin Liu et al.), 2024

https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report

5. MoE-Lightning: High-Throughput MoE Inference on Memory-Constrained GPUs — Shiyi Cao, Shu Liu, Tyler Griggs, Peter Schafhalter, Xiaoxuan Liu, Ying Sheng, Joseph E. Gonzalez, Matei Zaharia, Ion Stoica, 2024

https://scholar.google.com/scholar?q=MoE-Lightning%3A+High-Throughput+MoE+Inference+on+Memory-Constrained+GPUs

6. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen, 2020

https://scholar.google.com/scholar?q=GShard%3A+Scaling+Giant+Models+with+Conditional+Computation+and+Automatic+Sharding

Interactive Visualization: MegaScale-Infer: Disaggregating Experts for Faster MoE Serving

This episode examines HORIZON, a system from NVIDIA researchers (Cunxi Yu and colleagues) that treats RTL hardware design as repository-level code evolution, wrapping generation in a git-native loop where an LLM agent edits Verilog, an automated evaluator compiles and simulates each candidate, and only passing changes get committed. The conversation traces the idea's lineage from AlphaEvolve's evolutionary code loops through Yu's own SATLUTION and ABCEvo work, framing HORIZON as the next rung: evolving the hardware artifact itself rather than the tools used to build it. It contrasts HORIZON with two existing approaches to LLM-based RTL generation — domain-tuned one-shot generators like RTLCoder and ChipNeMo, and iterative repair systems like AutoChip and RTLFixer — and explains why hardware's unforgiving, concurrent, tape-out-or-bust nature makes "mostly correct" useless in a way it isn't for software. It also introduces CVDP, a 783-problem benchmark built to stress-test agentic and non-agentic Verilog generation now that older benchmarks are saturating. Listeners interested in whether AI coding-agent techniques can transfer to safety-critical, irreversible engineering domains will find the structural argument here more compelling than the raw benchmark numbers.

Sources:

1. Agentic Hardware Design as Repository-Level Code Evolution — Cunxi Yu, Chenhui Deng, Nathaniel Pinckney, Brucek Khailany, 2026

http://arxiv.org/abs/2606.28279v1

2. AlphaEvolve: A coding agent for scientific and algorithmic discovery — Novikov et al. (Google DeepMind), 2025

https://scholar.google.com/scholar?q=AlphaEvolve%3A+A+coding+agent+for+scientific+and+algorithmic+discovery

3. SATLUTION: scaling repository-level self-evolution to SAT solvers — Yu et al., 2025

https://scholar.google.com/scholar?q=SATLUTION%3A+scaling+repository-level+self-evolution+to+SAT+solvers

4. ABCEvo: self-evolving the ABC logic-synthesis system — Yu et al., 2026

https://scholar.google.com/scholar?q=ABCEvo%3A+self-evolving+the+ABC+logic-synthesis+system

5. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering — Yang et al., 2024

https://scholar.google.com/scholar?q=SWE-agent%3A+Agent-Computer+Interfaces+Enable+Automated+Software+Engineering

6. VeriGen: A Large Language Model for Verilog Code Generation — Thakur et al., 2024

https://scholar.google.com/scholar?q=VeriGen%3A+A+Large+Language+Model+for+Verilog+Code+Generation

7. RTLCoder: Fully Open-Source and Efficient LLM-Assisted RTL Code Generation — Liu et al., 2025

https://scholar.google.com/scholar?q=RTLCoder%3A+Fully+Open-Source+and+Efficient+LLM-Assisted+RTL+Code+Generation

8. ChipNeMo: Domain-Adapted LLMs for Chip Design — Liu et al., 2023

https://scholar.google.com/scholar?q=ChipNeMo%3A+Domain-Adapted+LLMs+for+Chip+Design

9. CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations — Liu et al., 2024

https://scholar.google.com/scholar?q=CraftRTL%3A+High-quality+Synthetic+Data+Generation+for+Verilog+Code+Models+with+Correct-by-Construction+Non-Textual+Representations

10. Autonomous Code Evolution Meets NP-Completeness (SATLUTION) — Yu, Liang, Ho, Ren, 2025

https://scholar.google.com/scholar?q=Autonomous+Code+Evolution+Meets+NP-Completeness+%28SATLUTION%29

11. Autonomous Evolution of EDA Tools: Multi-Agent Self-Evolved ABC (ABCEvo) — Yu, Liang, Ho, Ren, 2026

https://scholar.google.com/scholar?q=Autonomous+Evolution+of+EDA+Tools%3A+Multi-Agent+Self-Evolved+ABC+%28ABCEvo%29

12. Comprehensive Verilog Design Problems (CVDP) — Pinckney, Deng, Ho, Tsai, Liu, Zhou, Khailany, Ren, 2025

https://scholar.google.com/scholar?q=Comprehensive+Verilog+Design+Problems+%28CVDP%29

13. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? / SWE-Bench+ / 'Are Solved Issues in SWE-bench Really Solved Correctly?' — Jimenez et al. (2024); Aleithan et al. (2024); Wang, Pradel, Liu (2026), 2024–2026

https://scholar.google.com/scholar?q=SWE-bench%3A+Can+Language+Models+Resolve+Real-World+GitHub+Issues%3F+%2F+SWE-Bench%2B+%2F+%27Are+Solved+Issues+in+SWE-bench+Really+Solved+Correctly%3F%27

14. ACE-RTL: When Agentic Context Evolution Meets RTL-Specialized LLMs / MAGE: A Multi-Agent Engine for Automated RTL Code Generation — Deng, Yu, Liu, Pinckney, Khailany, Ren (2026); Zhao, Zhang, Huang, Yu, Zhao (2025), 2026 / 2025

https://scholar.google.com/scholar?q=ACE-RTL%3A+When+Agentic+Context+Evolution+Meets+RTL-Specialized+LLMs+%2F+MAGE%3A+A+Multi-Agent+Engine+for+Automated+RTL+Code+Generation

This episode explores "Autonomous Evolution of EDA Tools: Multi-Agent Self-Evolved ABC," a paper from NVIDIA Research and the University of Maryland examining whether a multi-agent LLM system can autonomously rewrite ABC, the 1.2-million-line, four-layer open-source logic synthesis and verification engine that underlies most open-source ASIC and FPGA toolchains. The discussion traces this work's lineage from DeepMind's AlphaEvolve, which evolved small isolated code kernels, through NVIDIA's own SATLUTION, which scaled the approach to a full SAT solver, and examines why editing a codebase as large and interdependent as ABC — where area, delay, and depth trade off across cross-module dependencies — demands a genuinely different architecture rather than just more iterations of the same loop. That architecture centers on a planning agent coordinating three specialized coding agents for flow tuning, technology mapping, and logic minimization, plus a pre-evolution stage where the system surveys the literature and selects its own scaffolding — including Cunxi Yu's prior FlowTune work and the SLAP mapper — without any heuristics hand-injected by the authors. The conversation digs into why correctness is uniquely non-negotiable here, since a synthesized circuit must be formally equivalent to spec rather than merely probabilistic, making this an unusual case of applying neural methods to edit, rather than replace, one of computing's last hand-engineered, non-learned domains. Listeners interested in AI-driven code evolution, chip design tooling, or the limits of LLM agents on large real-world codebases will find the debate over scale, architecture, and risk especially engaging.

Sources:

1. Autonomous Evolution of EDA Tools: Multi-Agent Self-Evolved ABC — Cunxi Yu, Haoxing Ren, 2026

http://arxiv.org/abs/2604.15082v1

2. AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery — Alexander Novikov, Ngân Vũ, and collaborators (Google DeepMind), 2025

https://scholar.google.com/scholar?q=AlphaEvolve%3A+A+Coding+Agent+for+Scientific+and+Algorithmic+Discovery

3. SATLUTION (LLM-agent self-evolution of a SAT solver codebase) — NVIDIA Research, 2025/2026

https://scholar.google.com/scholar?q=SATLUTION+%28LLM-agent+self-evolution+of+a+SAT+solver+codebase%29

4. DRiLLS: Deep Reinforcement Learning for Logic Synthesis — Abdelrahman Hosny, Soheil Hashemi, Mohamed Shalan, Sherief Reda, 2020

https://scholar.google.com/scholar?q=DRiLLS%3A+Deep+Reinforcement+Learning+for+Logic+Synthesis

5. A Graph Placement Methodology for Fast Chip Design — Azalia Mirhoseini, Anna Goldie, et al. (Google), 2021

https://scholar.google.com/scholar?q=A+Graph+Placement+Methodology+for+Fast+Chip+Design

6. Autonomous Code Evolution Meets NP-Completeness (SATLUTION) — Cunxi Yu, Rongjian Liang, Chia-Tung Ho, Haoxing Ren, 2025

https://scholar.google.com/scholar?q=Autonomous+Code+Evolution+Meets+NP-Completeness+%28SATLUTION%29

7. FunSearch: Making new discoveries in mathematical sciences using large language models — Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, et al. (Google DeepMind), 2024

https://scholar.google.com/scholar?q=FunSearch%3A+Making+new+discoveries+in+mathematical+sciences+using+large+language+models

8. FlowTune: End-to-End Automatic Logic Optimization Exploration via Domain-Specific Multi-Armed Bandit — Walter L. Neto, Yingjie Li, Pierre-Emmanuel Gaillardon, Cunxi Yu, 2023

https://scholar.google.com/scholar?q=FlowTune%3A+End-to-End+Automatic+Logic+Optimization+Exploration+via+Domain-Specific+Multi-Armed+Bandit

9. SLAP: A supervised learning approach for priority cuts technology mapping — Walter Lau Neto, Matheus T. Moreira, Yingjie Li, Luca Amarù, Cunxi Yu, Pierre-Emmanuel Gaillardon, 2021

https://scholar.google.com/scholar?q=SLAP%3A+A+supervised+learning+approach+for+priority+cuts+technology+mapping

10. DAG-aware synthesis orchestration — Yingjie Li, Mingju Liu, Haoxing Ren, Alan Mishchenko, Cunxi Yu, 2024

https://scholar.google.com/scholar?q=DAG-aware+synthesis+orchestration

11. EvoPlace: Evolution of Optimization Algorithms for Global Placement via Large Language Models — Xufeng Yao, Jiaxi Jiang, Yuxuan Zhao, Peiyu Liao, Yibo Lin, Bei Yu, 2026

https://scholar.google.com/scholar?q=EvoPlace%3A+Evolution+of+Optimization+Algorithms+for+Global+Placement+via+Large+Language+Models

12. Agentic AI for Physical Design R&D: Status and Prospects — Amur Ghose, Andrew B. Kahng, Sayak Kundu, Bodhisatta Pramanik, 2026

https://scholar.google.com/scholar?q=Agentic+AI+for+Physical+Design+R%26D%3A+Status+and+Prospects

13. MapTune: Versatile ASIC Technology Mapping via Reinforcement Learning Guided Library Tuning — Mingju Liu, Daniel Robinson, Yingjie Li, Johannes Maximilian Kuehn, Rongjian Liang, Haoxing Ren, Cunxi Yu, 2026

https://scholar.google.com/scholar?q=MapTune%3A+Versatile+ASIC+Technology+Mapping+via+Reinforcement+Learning+Guided+Library+Tuning

14. Machine-Learned Algorithmic Improvement — Ivan Smirnov et al., 2023

https://scholar.google.com/scholar?q=Machine-Learned+Algorithmic+Improvement

Interactive Visualization: Multi-Agent AI Rewrites a Million-Line Chip Design Tool

This episode explores CacheFlow, a system for restoring the KV cache that lets LLM serving systems reload prior context into GPU memory quickly. The discussion centers on a scheduling problem: rather than choosing between recomputing attention states or loading cached tensors from CPU, disk, or another node, CacheFlow treats restoration as a coordination problem across tokens, layers, GPUs, and concurrent requests simultaneously. It highlights a two-pointer "meet in the middle" technique applied along both token chunks and model layers, with an offline-profiled crossover point determining which strategy dominates for a given sequence length. The hosts also stress that the paper proves an optimality bound for its scheduling policy rather than just benchmarking against baselines, distinguishing it from typical serving-infrastructure papers. Listeners interested in reducing time-to-first-token for long-context chatbots, coding agents, or retrieval-heavy pipelines will find the reframing of restoration as a multi-dimensional scheduling problem, rather than a single per-request tradeoff, particularly compelling.

Sources:

1. CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration — Sean Nian, Jiahao Fang, Qilong Feng, Zhiyu Wu, Fan Lai, 2026

http://arxiv.org/abs/2604.25080

2. Compute or load KV cache? Why not both? (Cake) — Shuowei Jin, Xueshen Liu, Qingzhao Zhang, Z. Morley Mao, 2025

https://scholar.google.com/scholar?q=Compute+or+load+KV+cache%3F+Why+not+both%3F+%28Cake%29

3. Fast state restoration in LLM serving with HCache — Shiwei Gao, Youmin Chen, Jiwu Shu, 2025

https://scholar.google.com/scholar?q=Fast+state+restoration+in+LLM+serving+with+HCache

4. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot — Ruoyu Qin, Zheming Li, Weiran He, et al., 2025

https://scholar.google.com/scholar?q=Mooncake%3A+Trading+more+storage+for+less+computation+%E2%80%94+a+KVCache-centric+architecture+for+serving+LLM+chatbot

5. LMCache: An efficient KV cache layer for enterprise-scale LLM inference — Yuhan Liu, Yihua Cheng, Jiayi Yao, et al., 2025

https://scholar.google.com/scholar?q=LMCache%3A+An+efficient+KV+cache+layer+for+enterprise-scale+LLM+inference

6. Cacheblend: Fast large language model serving for RAG with cached knowledge fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, et al., 2025

https://scholar.google.com/scholar?q=Cacheblend%3A+Fast+large+language+model+serving+for+RAG+with+cached+knowledge+fusion

7. Kvflow: Efficient prefix caching for accelerating LLM-based multi-agent workflows — Zaifeng Pan, Ajjkumar Patel, Zhengding Hu, et al., 2025

https://scholar.google.com/scholar?q=Kvflow%3A+Efficient+prefix+caching+for+accelerating+LLM-based+multi-agent+workflows

8. Continuum: Efficient and robust multi-turn LLM agent scheduling with KV cache time-to-live — Hanchen Li, Qiuyang Mang, Runyuan He, et al., 2026

https://scholar.google.com/scholar?q=Continuum%3A+Efficient+and+robust+multi-turn+LLM+agent+scheduling+with+KV+cache+time-to-live

Interactive Visualization: CacheFlow: Optimal 3D-Parallel KV Cache Restoration

This episode explores IMPRESS, a systems paper from Zhejiang University and Huawei Cloud researchers presented at USENIX FAST 2025, which tackles a specific bottleneck in large language model inference: the time delay before a model produces its first response token when cached context has spilled onto disk. The discussion traces how modern LLM applications—retrieval-augmented generation, multi-turn chatbots, and plugin frameworks—prepend large chunks of context that inflate prefill costs superlinearly, with one cited example showing a 2,600-token plugin prompt stretching time-to-first-token by nine times on OPT-30B. It covers how prior work like vLLM's PagedAttention and AttentionStore addressed prefix KV caching but hit a wall once caches outgrew GPU and CPU memory and moved to slower disk storage, where I/O latency can consume up to 98 percent of total delay. The conversation traces IMPRESS's key insight: repurposing an importance-scoring technique from H2O—originally used to decide what to evict during decoding—to instead decide what's worth loading from disk before prefill even starts, claiming up to 2.8x lower latency with comparable accuracy. Listeners interested in the practical engineering tradeoffs behind making large-context LLM applications faster and cheaper to run will find the discussion's grounding in measured attention patterns, rather than benchmark tweaking, particularly compelling.

Sources:

1. IMPRESS: A Multi-Tier KV Storage System for Faster LLM Prefill

https://www.usenix.org/system/files/fast25-chen-weijian-impress.pdf

2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica (UC Berkeley), 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen, 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models

4. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention (AttentionStore) — Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, and colleagues (industry/academic collaboration), 2024

https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention+%28AttentionStore%29

5. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu (Moonshot AI, Tsinghua University), 2024

https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving

6. AttentionStore: Cost-Effective Attention Reuse across Multi-Turn Conversations in Large Language Model Serving — Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, Pengfei Zuo, 2024

https://scholar.google.com/scholar?q=AttentionStore%3A+Cost-Effective+Attention+Reuse+across+Multi-Turn+Conversations+in+Large+Language+Model+Serving

7. Retrieval Head Mechanistically Explains Long-Context Factuality — Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, Yao Fu, 2024

https://scholar.google.com/scholar?q=Retrieval+Head+Mechanistically+Explains+Long-Context+Factuality

8. RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation — Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, Xin Jin, 2024

https://scholar.google.com/scholar?q=RAGCache%3A+Efficient+Knowledge+Caching+for+Retrieval-Augmented+Generation

Interactive Visualization: IMPRESS: A Multi-Tier KV Storage System for Faster LLM Prefill

Fast State Restoration in LLM Serving with HCache tackles a hidden cost of running LLM chat services: when GPU memory pressure forces eviction of a conversation's KV cache, restoring that state currently means either recomputing it from scratch (20-26x slower than no restoration) or streaming the full cache back from storage over PCIe (6.5-13x slower). Drawing on traces from ShareGPT4 and L-Eval, the discussion lays out why eviction is the common case rather than an edge case — a single A100-40GB holds only enough KV cache for a handful of live conversations at once. The episode walks through the researchers' proposed middle path: caching the hidden state (one layer upstream of the key/value projection) instead of the KV cache itself, then reconstructing K and V on demand via a cheap matrix multiplication. It's a systems paper grounded in first-principles reasoning about transformer architecture before any benchmarks are run, making the case for why this approach should be faster on theoretical grounds alone. Listeners interested in the practical engineering trade-offs behind serving long, multi-turn LLM conversations at scale will find the framing of "recompute vs. offload vs. something smaller in between" a clear lens on a problem most users never realize is happening.

Sources:

1. Fast State Restoration in LLM Serving with HCache — Shiwei Gao, Youmin Chen, Jiwu Shu, 2024

http://arxiv.org/abs/2410.05004

2. Training Deep Nets with Sublinear Memory Cost — Tianqi Chen, Bin Xu, Chiyuan Zhang, Carlos Guestrin, 2016

https://scholar.google.com/scholar?q=Training+Deep+Nets+with+Sublinear+Memory+Cost

3. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al., 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

4. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, et al. (Moonshot AI and Tsinghua University), 2024/2025

https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving

5. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu, Junchen Jiang, et al. (University of Chicago), 2024

https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving

6. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, et al., 2025

https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion

7. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Ying Sheng, et al., 2024

https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs

8. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention — Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, Pengfei Zuo, 2024 (USENIX ATC)

https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention

9. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, Sumit Sanghai, 2023 (EMNLP)

https://scholar.google.com/scholar?q=GQA%3A+Training+Generalized+Multi-Query+Transformer+Models+from+Multi-Head+Checkpoints

10. Prompt Cache: Modular Attention Reuse for Low-Latency Inference — In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, Lin Zhong, 2024 (MLSys)

https://scholar.google.com/scholar?q=Prompt+Cache%3A+Modular+Attention+Reuse+for+Low-Latency+Inference

11. Efficiently Programming Large Language Models using SGLang — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2023/2024

https://scholar.google.com/scholar?q=Efficiently+Programming+Large+Language+Models+using+SGLang

12. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, Ce Zhang, 2023 (ICML)

https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU

Interactive Visualization: Fast State Restoration for Evicted LLM KV Caches