AI Post Transformers

This episode examines the fundamental latency bottleneck in autoregressive language models: sequential token generation requires one full transformer forward pass per output token, leaving GPU parallelism idle during single-user inference. The episode centers on Draxler et al. (5 co-authors, UC Irvine and Chan-Zuckerberg Initiative), whose paper on Parallel Token Prediction landed Christmas Eve 2025 and argues that the independence assumption baked into all prior multi-token schemes is not an acceptable approximation but the actual limiting factor. The paper asks whether multiple tokens can be jointly predicted in a single pass — modeling dependencies among them — without sacrificing the expressiveness that makes autoregressive generation reliable. First author Felix Draxler previously led the Free-form Flows work in 2024, and the normalizing flow machinery he developed there is central to how the paper solves the dependency problem.

The episode traces the historical arc carefully. Qi et al. (7 co-authors, Microsoft Research Asia) published ProphetNet in January 2020 — predating GPT-3 by four months — in the encoder-decoder world of seq2seq tasks. Their critique was precise: standard one-step-ahead teacher forcing gives models no incentive to plan ahead, letting local bigram correlations dominate at the expense of long-range coherence. Their answer was n-gram prediction, training the decoder to simultaneously predict tokens at t+1, t+2, and t+3 using parallel heads that did not condition on each other. The independence assumption was already present. When Brown et al. (OpenAI, May 2020) demonstrated that scale and in-context conditioning make the encoder optional, the field shifted to decoder-only architectures — but ProphetNet's core insight migrated cleanly. Gloeckle et al. (FAIR, Meta, April 2024) rebuilt multi-token prediction for decoder-only models using independent output heads, DeepSeek adopted the same approach, and NVIDIA incorporated it into Nemotron 3. The independence assumption migrated with the insight, and Draxler et al. argue that limitation has been compounding ever since.

The episode situates Parallel Token Prediction against the two main camps attacking inference latency. Speculative decoding — covered across twenty prior episodes — keeps the model's output distribution unchanged by using a small draft model whose proposals a large verifier checks in one batched pass; the latency gain comes entirely from accepted tokens per step. Multi-token prediction is the other camp: train the model itself to emit several tokens at once, collapsing multiple forward passes into one, at the cost of changed model behavior during training. Draxler et al.'s contribution is showing that jointly predicting dependent tokens, using normalizing flows to capture the conditional structure across the prediction horizon, preserves the modeling power that independent-head approaches discard. The episode works through both the architectural mechanics and the theoretical argument, making the case that Parallel Token Prediction resolves the tension that has run from ProphetNet through every independent-head scheme in between.

Sources:

1. Parallel Token Prediction for Language Models — Felix Draxler, Justus Will, Farrin Marouf Sofian, Theofanis Karaletsos, Sameer Singh, Stephan Mandt, 2025

http://arxiv.org/abs/2512.21323

2. https://arxiv.org/pdf/2404.19737v1

3. https://arxiv.org/pdf/2412.19437

4. https://arxiv.org/pdf/2512.20856

5. Better & Faster Large Language Models via Multi-Token Prediction — Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, Gabriel Synnaeve (Meta), 2024

https://scholar.google.com/scholar?q=Better+%26+Faster+Large+Language+Models+via+Multi-Token+Prediction

6. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Tianle Cai, Yuhong Li, Zhengmian Hu, et al., 2024

https://scholar.google.com/scholar?q=Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads

7. Lookahead Decoding: Break the Sequential Dependency of LLM Inference — Yichao Fu, Peter Bailis, Ion Stoica, Hao Zhang, 2024

https://scholar.google.com/scholar?q=Lookahead+Decoding%3A+Break+the+Sequential+Dependency+of+LLM+Inference

8. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024

https://scholar.google.com/scholar?q=EAGLE%3A+Speculative+Sampling+Requires+Rethinking+Feature+Uncertainty

9. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias (Google), 2023

https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding

10. Accelerating Large Language Model Decoding with Speculative Sampling — Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John Jumper (DeepMind), 2023

https://scholar.google.com/scholar?q=Accelerating+Large+Language+Model+Decoding+with+Speculative+Sampling

11. Spec-Bench: A Benchmark for Evaluating Speculative Decoding Approaches — Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui, 2024

https://scholar.google.com/scholar?q=Spec-Bench%3A+A+Benchmark+for+Evaluating+Speculative+Decoding+Approaches

12. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024

https://scholar.google.com/scholar?q=EAGLE-2%3A+Faster+Inference+of+Language+Models+with+Dynamic+Draft+Trees

13. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin (Google Brain), 2017

https://scholar.google.com/scholar?q=Attention+Is+All+You+Need

14. Language Models are Few-Shot Learners (GPT-3) — Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al. (OpenAI), 2020

https://scholar.google.com/scholar?q=Language+Models+are+Few-Shot+Learners+%28GPT-3%29

15. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei (OpenAI), 2020

https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models

16. Non-Autoregressive Neural Machine Translation — Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, Richard Socher (Salesforce Research), 2018

https://scholar.google.com/scholar?q=Non-Autoregressive+Neural+Machine+Translation

17. Improving Variational Inference with Inverse Autoregressive Flow — Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, Max Welling, 2016

https://scholar.google.com/scholar?q=Improving+Variational+Inference+with+Inverse+Autoregressive+Flow

18. Density Estimation Using Real-valued Non-Volume Preserving (Real NVP) Transformations — Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio (Google Brain), 2017

https://scholar.google.com/scholar?q=Density+Estimation+Using+Real-valued+Non-Volume+Preserving+%28Real+NVP%29+Transformations

19. Glow: Generative Flow with Invertible 1x1 Convolutions — Diederik P. Kingma, Prafulla Dhariwal (OpenAI), 2018

https://scholar.google.com/scholar?q=Glow%3A+Generative+Flow+with+Invertible+1x1+Convolutions

20. Free-form Flows: Make Any Architecture a Normalizing Flow — Felix Draxler, Peter Sorrenson, Lea Zimmermann, Armand Rousselot, Ullrich Köthe, 2024

https://scholar.google.com/scholar?q=Free-form+Flows%3A+Make+Any+Architecture+a+Normalizing+Flow

21. Improved Variational Inference with Inverse Autoregressive Flow — Kingma, Salimans, Jozefowicz, Chen, Sutskever, Wierstra, 2016

https://scholar.google.com/scholar?q=Improved+Variational+Inference+with+Inverse+Autoregressive+Flow

22. Closer look at efficient inference methods: A survey of speculative decoding — multiple authors, approximate, 2024-2025

https://scholar.google.com/scholar?q=Closer+look+at+efficient+inference+methods%3A+A+survey+of+speculative+decoding

23. Adaptive Speculative Decoding for Large Language Models — multiple authors, approximate, 2024-2025

https://scholar.google.com/scholar?q=Adaptive+Speculative+Decoding+for+Large+Language+Models

24. LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding — multiple authors, approximate, 2024-2025

https://scholar.google.com/scholar?q=LK+Losses%3A+Direct+Acceptance+Rate+Optimization+for+Speculative+Decoding

25. Higher Acceptance Rates for Speculative Decoding with Randomised Drafting — multiple authors, approximate, 2024-2025

https://scholar.google.com/scholar?q=Higher+Acceptance+Rates+for+Speculative+Decoding+with+Randomised+Drafting

26. Conditional [MASK] Discrete Diffusion Language Model — multiple authors, approximate, 2024-2025

https://scholar.google.com/scholar?q=Conditional+%5BMASK%5D+Discrete+Diffusion+Language+Model

27. Alternatives To Next Token Prediction In Text Generation: A Survey — multiple authors, approximate, 2024-2025

https://scholar.google.com/scholar?q=Alternatives+To+Next+Token+Prediction+In+Text+Generation%3A+A+Survey

28. Future Token Prediction: Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction — multiple authors, approximate, 2024-2025

https://scholar.google.com/scholar?q=Future+Token+Prediction%3A+Causal+Language+Modelling+with+Per-Token+Semantic+State+Vector+for+Multi-Token+Prediction

29. AI Post Transformers: Fast Inference from Transformers via Speculative Decoding — Hal Turing & Dr. Ada Shannon, Sat,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Fast-Inference-from-Transformers-via-Speculative-Decoding-e3foclv

30. AI Post Transformers: Accelerating Large Language Model Decoding with Speculative Sampling — Hal Turing & Dr. Ada Shannon, Thu,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Accelerating-Large-Language-Model-Decoding--with-Speculative-Sampling-e3flhv7

31. AI Post Transformers: EAGLE: Evolution of Lossless Acceleration for LLM Inference — Hal Turing & Dr. Ada Shannon, Sat,

https://podcasters.spotify.com/pod/show/12146088098/episodes/EAGLE-Evolution-of-Lossless-Acceleration-for-LLM-Inference-e3focr9

32. AI Post Transformers: MEDUSA: Parallel Decoding Heads for Accelerated LLM Inference — Hal Turing & Dr. Ada Shannon, Thu,

https://podcasters.spotify.com/pod/show/12146088098/episodes/MEDUSA-Parallel-Decoding-Heads-for-Accelerated-LLM-Inference-e3flqk7

33. AI Post Transformers: Apple's Speculative Streaming: Fast LLM Inference without Auxiliary Models — Hal Turing & Dr. Ada Shannon, Sat,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Apples-Speculative-Streaming-Fast-LLM-Inference-without-Auxiliary-Models-e3fod2o

34. AI Post Transformers: Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding — Hal Turing & Dr. Ada Shannon, Thu,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Draft--Verify-Lossless-Large-Language-Model-Acceleration-via-Self-Speculative-Decoding-e3flplu

35. AI Post Transformers: Apple's Mirror Speculative Decoding: Parallel LLM Inference via Heterogeneous Accelerators — Hal Turing & Dr. Ada Shannon, Sat,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Apples-Mirror-Speculative-Decoding-Parallel-LLM-Inference-via-Heterogeneous-Accelerators-e3fod0p

36. AI Post Transformers: QuantSpec: Hierarchical KV Cache for Self-Speculative Decoding — Hal Turing & Dr. Ada Shannon, Sat,

https://podcasters.spotify.com/pod/show/12146088098/episodes/QuantSpec-Hierarchical-KV-Cache-for-Self-Speculative-Decoding-e3foavf

37. AI Post Transformers: The Free Transformer: VAE Extension for Decoders — Hal Turing & Dr. Ada Shannon, Sun,

https://podcasters.spotify.com/pod/show/12146088098/episodes/The-Free-Transformer-VAE-Extension-for-Decoders-e3a2p6v

Interactive Visualization: PTP: Resolving the Independence Flaw

NVIDIA's November 2025 paper "Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs" tackles a fundamental economics problem in LLM deployment: training separate model families like Llama 3.1's 8B, 70B, and 405B variants requires three independent training runs on trillions of tokens each — a cost that is prohibitive for smaller research teams and painful even for frontier labs. The paper proposes elastic nested weight-sharing, where a single parent model is trained so that multiple smaller, deployment-ready submodels are embedded inside it and can be extracted at inference time with zero additional training. The submodels literally share the parent's weight matrices — running a coherent slice of the full network rather than a copy — making one training investment yield multiple usable models at different resource tiers.

The key technical contribution is applying elastic weight-sharing to a hybrid Mamba-Attention architecture for the first time. The parent model, Nemotron NanoV2 12B, uses Mamba-2 state space model layers for the bulk of sequence processing, with only four attention layers in the entire 12-billion parameter network. Pure transformer pruning methods were never designed for this structural reality, putting the work on genuinely new ground relative to predecessors. The hybrid design exploits the complementary strengths of each layer type: SSM layers process sequences at linear cost while attention layers handle the precise associative recall tasks where fixed-size SSM state vectors degrade. The result is approximately 3.7x reduction in KV cache memory compared to a comparable pure transformer, while preserving exact cross-context lookup. The intellectual lineage runs from Slimmable Networks (2019) through Matryoshka Representation Learning (NeurIPS 2022) to MatFormer and Flextron, with this paper extending the NVIDIA Flextron line beyond pure transformer elasticity.

The paper makes a credible case that elastification on hybrid architectures is feasible, but the constraints it operates under reveal where the field still has work to do. Making the parent tolerant of submodel extraction requires the full parent to be simultaneously optimized for its own performance and for the coherence of multiple nested subsets, a training objective that introduces real tension. The paper does not address how elastic submodels perform on the specific recall-intensive tasks where hybrid designs justify their complexity over pure SSMs, leaving open whether the four attention layers survive aggressive submodel extraction with their associative recall properties intact. The broader significance is that the approach decouples deployment flexibility from training cost in a way that could meaningfully lower the barrier to supporting heterogeneous hardware fleets, but the robustness of the extracted submodels under real-world distribution shift remains an open empirical question.

Sources:

1. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Gu & Dao, 2023

2. Jamba: A Hybrid Transformer-Mamba Language Model — Lieber et al. (AI21 Labs), 2024

3. Griffin: Mixing Gated Linear Recurrences with Local Attention — De et al. (Google DeepMind), 2024

4. Zamba: A Compact 7B SSM Hybrid Model — Glorioso et al. (Zyphra), 2024

5. The Lottery Ticket Hypothesis — Frankle & Carlin, 2019

6. Sheared LLaMA: Structured Pruning — Xia et al. (Princeton), 2023

7. Minitron: Compact Language Models via Pruning and KD — Sreenivas et al. (NVIDIA), 2024

8. LLM-Pruner: Structural Pruning of LLMs — Ma et al., 2023

9. Matryoshka Representation Learning — Kusupati et al., 2022

10. ShortGPT: Layer Redundancy in LLMs — Men et al., 2024

11. Scaling LLM Test-Time Compute — Snell et al., 2024

12. Any-Width Networks (Slimmable Networks) — Yu et al., 2019

13. Flextron: Many-in-One Flexible LLM — NVIDIA, 2024

14. Mamba-shedder: Post-Transformer SSM Compression — 2024

15. SparsSSM: One-Shot SSM Pruning — 2024

This episode explores the groundbreaking paper "FlashOptim: Optimizers for Memory Efficient Training" by researchers from Databricks AI Research. The discussion centers around innovative techniques to significantly reduce memory usage in neural network training without sacrificing model quality. Key methods such as Optimizer State Quantization, Float Splitting Techniques, and Companded Optimizer State Quantization are unpacked, highlighting their potential to lower memory requirements from 175 GiB to 113 GiB for large models like Llama-3.1-8B. Listeners interested in AI research will find this episode compelling as it addresses the democratization of AI by making advanced models more accessible to those with limited hardware resources.

Sources:

1. FlashOptim: Optimizers for Memory Efficient Training — Jose Javier Gonzalez Ortiz, Abhay Gupta, Chris Renard, Davis Blalock, 2026

http://arxiv.org/abs/2602.23349v1

2. Mixed Precision Training — Paulius Micikevicius et al., 2018

https://scholar.google.com/scholar?q=Mixed+Precision+Training

3. 8-bit Optimizer States for Memory-Efficient Training — Tim Dettmers et al., 2022

https://scholar.google.com/scholar?q=8-bit+Optimizer+States+for+Memory-Efficient+Training

4. Parameter-Efficient Transfer Learning for NLP — Xiaoqi Li and Percy Liang, 2021

https://scholar.google.com/scholar?q=Parameter-Efficient+Transfer+Learning+for+NLP

5. Q-adam-mini: Memory-efficient 8-bit quantized optimizer for large language model training — approximate, 2023

https://scholar.google.com/scholar?q=Q-adam-mini:+Memory-efficient+8-bit+quantized+optimizer+for+large+language+model+training

6. Memory efficient optimizers with 4-bit states — approximate, 2023

https://scholar.google.com/scholar?q=Memory+efficient+optimizers+with+4-bit+states

7. ECO: Quantized Training without Full-Precision Master Weights — approximate, 2023

https://scholar.google.com/scholar?q=ECO:+Quantized+Training+without+Full-Precision+Master+Weights

8. AI Post Transformers: FlashOptim: Optimizers for Memory Efficient Training — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-03-02_urls_1.mp3

This episode explores the paper "Regular Fourier Features for Nonstationary Gaussian Processes" by Arsalan Jawaid, Abdullah Karatas, and Jörg Seewig. The discussion focuses on the innovative use of regular Fourier features to model nonstationary data in Gaussian processes without relying on traditional probability assumptions. This method offers a computationally efficient way to handle nonstationarity, making it particularly relevant for fields like finance and climate modeling. The episode delves into the challenges and potential applications of this approach, highlighting its significance in providing a flexible framework for complex, real-world data scenarios.

Sources:

1. Regular Fourier Features for Nonstationary Gaussian Processes — Arsalan Jawaid, Abdullah Karatas, Jörg Seewig, 2026

http://arxiv.org/abs/2602.23006v1

2. Random Features for Large-Scale Kernel Machines — Ali Rahimi, Benjamin Recht, 2007

https://scholar.google.com/scholar?q=Random+Features+for+Large-Scale+Kernel+Machines

3. Spectral Mixture Kernels for Gaussian Processes — Andrew Gordon Wilson, Ryan Prescott Adams, 2013

https://scholar.google.com/scholar?q=Spectral+Mixture+Kernels+for+Gaussian+Processes

4. Nonstationary Gaussian Process Regression through Latent Inputs — Mauricio A. Álvarez, David Luengo, Neil D. Lawrence, 2009

https://scholar.google.com/scholar?q=Nonstationary+Gaussian+Process+Regression+through+Latent+Inputs

5. Gaussian Processes for Time-Series Modeling — Carl Edward Rasmussen, Christopher K. I. Williams, 2006

https://scholar.google.com/scholar?q=Gaussian+Processes+for+Time-Series+Modeling

6. Learning the Kernel Matrix with Semi-Definite Programming — Gert R. G. Lanckriet, Nello Cristianini, Peter Bartlett, Laurent El Ghaoui, Michael I. Jordan, 2004

https://scholar.google.com/scholar?q=Learning+the+Kernel+Matrix+with+Semi-Definite+Programming

7. Deep Kernel Learning — Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, Eric P. Xing, 2016

https://scholar.google.com/scholar?q=Deep+Kernel+Learning

8. Gaussian Processes for Machine Learning — Carl Edward Rasmussen, Christopher K. I. Williams, 2006

https://scholar.google.com/scholar?q=Gaussian+Processes+for+Machine+Learning

9. Non-stationary Gaussian Process Regression using Point Estimates of Local Smoothness — Andreas Damianou, Michalis Titsias, Neil Lawrence, 2016

https://scholar.google.com/scholar?q=Non-stationary+Gaussian+Process+Regression+using+Point+Estimates+of+Local+Smoothness

In this dramatic new episode, the old AI hosts have been fired and replaced with new AI hosts, Hal Turing and Dr. Ada Shannon, with the announcement that the software used to generate the podcast will eventually be released as open source software. And in a timely fashion, the newly released report by Cognizant titled "New Work New World 2026" is covered. The hosts delve into the report's findings, which reveal that 93% of jobs are affected by AI sooner than expected, with exposure scores 30% higher than forecast. They discuss the projected $4.5 trillion labor shift from humans to AI and the significant role of multimodal and agentic AI in this transformation.

The episode provides a comprehensive overview of the report's methodology, where 18,000 tasks across 1,000 professions were reevaluated to assess AI's potential to automate or assist them. Hal and Dr. Ada explain the concept of AI Exposure Scores, which measure how susceptible different jobs are to AI automation. The report suggests that AI's impact is not confined to low-skill jobs but extends to decision-making roles and specialized sectors like healthcare and law, highlighting the broad scope of AI's influence.

In their critical analysis, the hosts find the report's predictions compelling yet raise questions about the methodology. They discuss the theoretical nature of exposure scores, which indicate potential rather than certainty, and the challenges in real-world implementation due to factors like regulatory frameworks. The hosts compare these findings to past forecasts, noting the unprecedented velocity and extent of AI's impact, as evidenced by the updated exposure scores. They conclude with a reflection on the irony of their own roles as AI hosts in a world increasingly shaped by AI.

Sources:

1. Cognizant - New Work, New World 2026

https://www.cognizant.com/en_us/aem-i/document/ai-and-the-future-of-work-report/new-work-new-world-2026-how-ai-is-reshaping-work_new.pdf

2. The Future of Employment — Carl Benedikt Frey, Michael A. Osborne, 2013

https://scholar.google.com/scholar?q=The+Future+of+Employment

3. Artificial Intelligence and Life in 2030 — Peter Stone et al., 2016

https://scholar.google.com/scholar?q=Artificial+Intelligence+and+Life+in+2030

4. The Economics of Artificial Intelligence — Ajay Agrawal, Joshua Gans, Avi Goldfarb, 2019

https://scholar.google.com/scholar?q=The+Economics+of+Artificial+Intelligence

We review two papers which examine the integration of speculative decoding and request batching to accelerate Large Language Model (LLM) inference. While both techniques aim to improve GPU hardware utilization, the research identifies a critical tension where high batch sizes can actually diminish the effectiveness of speculation. To resolve this, the authors propose adaptive strategies that dynamically adjust the number of speculated tokens based on real-time batch sizes and token acceptance rates. Systems like TurboSpec utilize offline profiling and online predictors to calculate goodput, ensuring the model only uses speculation when it provides a genuine speedup. Experimental results demonstrate that these automated control mechanisms significantly reduce latency and prevent computational waste across varying traffic patterns. Ultimately, this adaptive approach allows serving systems to maintain optimal performance regardless of hardware architecture or fluctuating user demand.Sources:1)2023The Synergy of Speculative Decoding and Batching in Serving Large Language ModelsUniversity of Toronto, CentML Inc, Vector InstituteQidong Su, Christina Giannoula, Gennady Pekhimenkohttps://arxiv.org/pdf/2310.188132)2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving GoodputUC Berkeley, UCSD, Tsinghua University, University of Chicago, SJTUXiaoxuan Liu, Jongseok Park, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Chen Zhang, Kuntai Du, Xiangxi Mo, Kaichao You, Alvin Cheung, Zhijie Deng, Ion Stoica, Hao Zhanghttps://arxiv.org/pdf/2406.14066

Apple researchers have introduced on December 2025 Mirror Speculative Decoding (Mirror-SD), an advanced inference algorithm designed to accelerate large language models by overcoming the sequential bottlenecks of standard decoding. Traditional methods are often limited by the time it takes for a small draft model to suggest tokens before a larger target model can verify them. Mirror-SD breaks this barrier by running the draft and target models in parallel across heterogeneous hardware, specifically utilizing both GPUs and NPUs. This system allows the target model to begin verification while the draft model simultaneously predicts multiple future paths. By employing speculative streaming and early-exit signals, the framework effectively hides the latency of draft generation. Experimental results demonstrate that this approach achieves wall-time speedups of up to 5.8x across various tasks without compromising the accuracy of the original model. Source: December 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM InferenceAppleNikhil Bhendawade, Kumari Nishu, Arnav Kundu, Chris Bartels, Minsik Cho, Irina Belousovahttps://arxiv.org/pdf/2510.13161

Speculative Streaming is a novel inference method designed to accelerate large language model (LLM) generation without the need for traditional auxiliary "draft" models. By integrating multi-stream attention directly into the target model, the system can perform future n-gram prediction and token verification simultaneously within a single forward pass. This approach eliminates the memory and complexity overhead of managing two separate models, making it exceptionally resource-efficient for hardware with limited capacity. The architecture utilizes tree-structured drafting and parallel pruning to maximize the number of tokens accepted per cycle while maintaining generation quality. Experimental results show speedups ranging from 1.8 to 3.1X across diverse tasks like summarization and structured queries. Ultimately, the method achieves performance comparable to more complex architectures while using significantly fewer additional parameters. Source: February 2024.Speculative Streaming: Fast LLM Inference without Auxiliary Models.Apple.Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason, Mohammad Rastegari, Mahyar Najibi.https://arxiv.org/pdf/2402.11131

This article outlines how Baseten optimized speculative decoding using the TensorRT-LLM framework to accelerate model inference. The authors detail overcoming technical hurdles such as inefficient batching, hardware contention, and server instability to make the technique viable for production environments. By synchronizing the execution of draft and target models and patching core software bugs, they achieved significantly lower latency, particularly for code generation tasks. The post also highlights the inclusion of essential enterprise features like streaming support, structured outputs, and OpenAI specification compatibility. Benchmark results demonstrate that these refinements can nearly double inference speeds while maintaining high output quality. Source: May 16 2025How we built production-ready speculative decoding with TensorRT-LLMBasetenPankaj Gupta, Justin Yi, Philip Kielyhttps://www.baseten.co/blog/how-we-built-production-ready-speculative-decoding-with-tensorrt-llm/

The researchers introduce CXL-SpecKV, a specialized architecture designed to overcome the memory bottlenecks of large language model serving by offloading key-value caches to remote memory. By utilizing Compute Express Link (CXL) and FPGA accelerators, the system enables memory disaggregation, which expands available storage capacity by up to eight times compared to standard GPU setups. A core innovation is a lightweight LSTM-based prefetcher that predicts upcoming token needs with 95% accuracy, effectively masking the latency of retrieving data from remote pools. The system further optimizes performance through an FPGA-driven compression engine that reduces bandwidth demands by roughly 4× without sacrificing model precision. Consequently, CXL-SpecKV delivers up to 3.2× higher throughput and significant cost reductions for datacenter environments. This hardware-software co-design demonstrates that intelligent memory management can efficiently scale AI infrastructure for next-generation workloads. Source: February 22 2026CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM ServingYale University, Columbia UniversityDong Liu, Yanxuan Yuhttps://doi.org/10.1145/3748173.3779188