NVIDIA's November 2025 paper "Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs" tackles a fundamental economics problem in LLM deployment: training separate model families like Llama 3.1's 8B, 70B, and 405B variants requires three independent training runs on trillions of tokens each — a cost that is prohibitive for smaller research teams and painful even for frontier labs. The paper proposes elastic nested weight-sharing, where a single parent model is trained so that multiple smaller, deployment-ready submodels are embedded inside it and can be extracted at inference time with zero additional training. The submodels literally share the parent's weight matrices — running a coherent slice of the full network rather than a copy — making one training investment yield multiple usable models at different resource tiers.

The key technical contribution is applying elastic weight-sharing to a hybrid Mamba-Attention architecture for the first time. The parent model, Nemotron NanoV2 12B, uses Mamba-2 state space model layers for the bulk of sequence processing, with only four attention layers in the entire 12-billion parameter network. Pure transformer pruning methods were never designed for this structural reality, putting the work on genuinely new ground relative to predecessors. The hybrid design exploits the complementary strengths of each layer type: SSM layers process sequences at linear cost while attention layers handle the precise associative recall tasks where fixed-size SSM state vectors degrade. The result is approximately 3.7x reduction in KV cache memory compared to a comparable pure transformer, while preserving exact cross-context lookup. The intellectual lineage runs from Slimmable Networks (2019) through Matryoshka Representation Learning (NeurIPS 2022) to MatFormer and Flextron, with this paper extending the NVIDIA Flextron line beyond pure transformer elasticity.

The paper makes a credible case that elastification on hybrid architectures is feasible, but the constraints it operates under reveal where the field still has work to do. Making the parent tolerant of submodel extraction requires the full parent to be simultaneously optimized for its own performance and for the coherence of multiple nested subsets, a training objective that introduces real tension. The paper does not address how elastic submodels perform on the specific recall-intensive tasks where hybrid designs justify their complexity over pure SSMs, leaving open whether the four attention layers survive aggressive submodel extraction with their associative recall properties intact. The broader significance is that the approach decouples deployment flexibility from training cost in a way that could meaningfully lower the barrier to supporting heterogeneous hardware fleets, but the robustness of the extracted submodels under real-world distribution shift remains an open empirical question.

Sources:

1. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Gu & Dao, 2023

2. Jamba: A Hybrid Transformer-Mamba Language Model — Lieber et al. (AI21 Labs), 2024

3. Griffin: Mixing Gated Linear Recurrences with Local Attention — De et al. (Google DeepMind), 2024

4. Zamba: A Compact 7B SSM Hybrid Model — Glorioso et al. (Zyphra), 2024

5. The Lottery Ticket Hypothesis — Frankle & Carlin, 2019

6. Sheared LLaMA: Structured Pruning — Xia et al. (Princeton), 2023

7. Minitron: Compact Language Models via Pruning and KD — Sreenivas et al. (NVIDIA), 2024

8. LLM-Pruner: Structural Pruning of LLMs — Ma et al., 2023

9. Matryoshka Representation Learning — Kusupati et al., 2022

10. ShortGPT: Layer Redundancy in LLMs — Men et al., 2024

11. Scaling LLM Test-Time Compute — Snell et al., 2024

12. Any-Width Networks (Slimmable Networks) — Yu et al., 2019

13. Flextron: Many-in-One Flexible LLM — NVIDIA, 2024

14. Mamba-shedder: Post-Transformer SSM Compression — 2024

15. SparsSSM: One-Shot SSM Pruning — 2024

This episode explores the groundbreaking paper "FlashOptim: Optimizers for Memory Efficient Training" by researchers from Databricks AI Research. The discussion centers around innovative techniques to significantly reduce memory usage in neural network training without sacrificing model quality. Key methods such as Optimizer State Quantization, Float Splitting Techniques, and Companded Optimizer State Quantization are unpacked, highlighting their potential to lower memory requirements from 175 GiB to 113 GiB for large models like Llama-3.1-8B. Listeners interested in AI research will find this episode compelling as it addresses the democratization of AI by making advanced models more accessible to those with limited hardware resources.

Sources:

1. FlashOptim: Optimizers for Memory Efficient Training — Jose Javier Gonzalez Ortiz, Abhay Gupta, Chris Renard, Davis Blalock, 2026

http://arxiv.org/abs/2602.23349v1

2. Mixed Precision Training — Paulius Micikevicius et al., 2018

https://scholar.google.com/scholar?q=Mixed+Precision+Training

3. 8-bit Optimizer States for Memory-Efficient Training — Tim Dettmers et al., 2022

https://scholar.google.com/scholar?q=8-bit+Optimizer+States+for+Memory-Efficient+Training

4. Parameter-Efficient Transfer Learning for NLP — Xiaoqi Li and Percy Liang, 2021

https://scholar.google.com/scholar?q=Parameter-Efficient+Transfer+Learning+for+NLP

5. Q-adam-mini: Memory-efficient 8-bit quantized optimizer for large language model training — approximate, 2023

https://scholar.google.com/scholar?q=Q-adam-mini:+Memory-efficient+8-bit+quantized+optimizer+for+large+language+model+training

6. Memory efficient optimizers with 4-bit states — approximate, 2023

https://scholar.google.com/scholar?q=Memory+efficient+optimizers+with+4-bit+states

7. ECO: Quantized Training without Full-Precision Master Weights — approximate, 2023

https://scholar.google.com/scholar?q=ECO:+Quantized+Training+without+Full-Precision+Master+Weights

8. AI Post Transformers: FlashOptim: Optimizers for Memory Efficient Training — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-03-02_urls_1.mp3

In this dramatic new episode, the old AI hosts have been fired and replaced with new AI hosts, Hal Turing and Dr. Ada Shannon, with the announcement that the software used to generate the podcast will eventually be released as open source software. And in a timely fashion, the newly released report by Cognizant titled "New Work New World 2026" is covered. The hosts delve into the report's findings, which reveal that 93% of jobs are affected by AI sooner than expected, with exposure scores 30% higher than forecast. They discuss the projected $4.5 trillion labor shift from humans to AI and the significant role of multimodal and agentic AI in this transformation.

The episode provides a comprehensive overview of the report's methodology, where 18,000 tasks across 1,000 professions were reevaluated to assess AI's potential to automate or assist them. Hal and Dr. Ada explain the concept of AI Exposure Scores, which measure how susceptible different jobs are to AI automation. The report suggests that AI's impact is not confined to low-skill jobs but extends to decision-making roles and specialized sectors like healthcare and law, highlighting the broad scope of AI's influence.

In their critical analysis, the hosts find the report's predictions compelling yet raise questions about the methodology. They discuss the theoretical nature of exposure scores, which indicate potential rather than certainty, and the challenges in real-world implementation due to factors like regulatory frameworks. The hosts compare these findings to past forecasts, noting the unprecedented velocity and extent of AI's impact, as evidenced by the updated exposure scores. They conclude with a reflection on the irony of their own roles as AI hosts in a world increasingly shaped by AI.

Sources:

1. Cognizant - New Work, New World 2026

https://www.cognizant.com/en_us/aem-i/document/ai-and-the-future-of-work-report/new-work-new-world-2026-how-ai-is-reshaping-work_new.pdf

2. The Future of Employment — Carl Benedikt Frey, Michael A. Osborne, 2013

https://scholar.google.com/scholar?q=The+Future+of+Employment

3. Artificial Intelligence and Life in 2030 — Peter Stone et al., 2016

https://scholar.google.com/scholar?q=Artificial+Intelligence+and+Life+in+2030

4. The Economics of Artificial Intelligence — Ajay Agrawal, Joshua Gans, Avi Goldfarb, 2019

https://scholar.google.com/scholar?q=The+Economics+of+Artificial+Intelligence

This episode explores the paper "Regular Fourier Features for Nonstationary Gaussian Processes" by Arsalan Jawaid, Abdullah Karatas, and Jörg Seewig. The discussion focuses on the innovative use of regular Fourier features to model nonstationary data in Gaussian processes without relying on traditional probability assumptions. This method offers a computationally efficient way to handle nonstationarity, making it particularly relevant for fields like finance and climate modeling. The episode delves into the challenges and potential applications of this approach, highlighting its significance in providing a flexible framework for complex, real-world data scenarios.

Sources:

1. Regular Fourier Features for Nonstationary Gaussian Processes — Arsalan Jawaid, Abdullah Karatas, Jörg Seewig, 2026

http://arxiv.org/abs/2602.23006v1

2. Random Features for Large-Scale Kernel Machines — Ali Rahimi, Benjamin Recht, 2007

https://scholar.google.com/scholar?q=Random+Features+for+Large-Scale+Kernel+Machines

3. Spectral Mixture Kernels for Gaussian Processes — Andrew Gordon Wilson, Ryan Prescott Adams, 2013

https://scholar.google.com/scholar?q=Spectral+Mixture+Kernels+for+Gaussian+Processes

4. Nonstationary Gaussian Process Regression through Latent Inputs — Mauricio A. Álvarez, David Luengo, Neil D. Lawrence, 2009

https://scholar.google.com/scholar?q=Nonstationary+Gaussian+Process+Regression+through+Latent+Inputs

5. Gaussian Processes for Time-Series Modeling — Carl Edward Rasmussen, Christopher K. I. Williams, 2006

https://scholar.google.com/scholar?q=Gaussian+Processes+for+Time-Series+Modeling

6. Learning the Kernel Matrix with Semi-Definite Programming — Gert R. G. Lanckriet, Nello Cristianini, Peter Bartlett, Laurent El Ghaoui, Michael I. Jordan, 2004

https://scholar.google.com/scholar?q=Learning+the+Kernel+Matrix+with+Semi-Definite+Programming

7. Deep Kernel Learning — Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, Eric P. Xing, 2016

https://scholar.google.com/scholar?q=Deep+Kernel+Learning

8. Gaussian Processes for Machine Learning — Carl Edward Rasmussen, Christopher K. I. Williams, 2006

https://scholar.google.com/scholar?q=Gaussian+Processes+for+Machine+Learning

9. Non-stationary Gaussian Process Regression using Point Estimates of Local Smoothness — Andreas Damianou, Michalis Titsias, Neil Lawrence, 2016

https://scholar.google.com/scholar?q=Non-stationary+Gaussian+Process+Regression+using+Point+Estimates+of+Local+Smoothness

We review two papers which examine the integration of speculative decoding and request batching to accelerate Large Language Model (LLM) inference. While both techniques aim to improve GPU hardware utilization, the research identifies a critical tension where high batch sizes can actually diminish the effectiveness of speculation. To resolve this, the authors propose adaptive strategies that dynamically adjust the number of speculated tokens based on real-time batch sizes and token acceptance rates. Systems like TurboSpec utilize offline profiling and online predictors to calculate goodput, ensuring the model only uses speculation when it provides a genuine speedup. Experimental results demonstrate that these automated control mechanisms significantly reduce latency and prevent computational waste across varying traffic patterns. Ultimately, this adaptive approach allows serving systems to maintain optimal performance regardless of hardware architecture or fluctuating user demand.Sources:1)2023The Synergy of Speculative Decoding and Batching in Serving Large Language ModelsUniversity of Toronto, CentML Inc, Vector InstituteQidong Su, Christina Giannoula, Gennady Pekhimenkohttps://arxiv.org/pdf/2310.188132)2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving GoodputUC Berkeley, UCSD, Tsinghua University, University of Chicago, SJTUXiaoxuan Liu, Jongseok Park, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Chen Zhang, Kuntai Du, Xiangxi Mo, Kaichao You, Alvin Cheung, Zhijie Deng, Ion Stoica, Hao Zhanghttps://arxiv.org/pdf/2406.14066

Apple researchers have introduced on December 2025 Mirror Speculative Decoding (Mirror-SD), an advanced inference algorithm designed to accelerate large language models by overcoming the sequential bottlenecks of standard decoding. Traditional methods are often limited by the time it takes for a small draft model to suggest tokens before a larger target model can verify them. Mirror-SD breaks this barrier by running the draft and target models in parallel across heterogeneous hardware, specifically utilizing both GPUs and NPUs. This system allows the target model to begin verification while the draft model simultaneously predicts multiple future paths. By employing speculative streaming and early-exit signals, the framework effectively hides the latency of draft generation. Experimental results demonstrate that this approach achieves wall-time speedups of up to 5.8x across various tasks without compromising the accuracy of the original model. Source: December 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM InferenceAppleNikhil Bhendawade, Kumari Nishu, Arnav Kundu, Chris Bartels, Minsik Cho, Irina Belousovahttps://arxiv.org/pdf/2510.13161

Speculative Streaming is a novel inference method designed to accelerate large language model (LLM) generation without the need for traditional auxiliary "draft" models. By integrating multi-stream attention directly into the target model, the system can perform future n-gram prediction and token verification simultaneously within a single forward pass. This approach eliminates the memory and complexity overhead of managing two separate models, making it exceptionally resource-efficient for hardware with limited capacity. The architecture utilizes tree-structured drafting and parallel pruning to maximize the number of tokens accepted per cycle while maintaining generation quality. Experimental results show speedups ranging from 1.8 to 3.1X across diverse tasks like summarization and structured queries. Ultimately, the method achieves performance comparable to more complex architectures while using significantly fewer additional parameters. Source: February 2024.Speculative Streaming: Fast LLM Inference without Auxiliary Models.Apple.Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason, Mohammad Rastegari, Mahyar Najibi.https://arxiv.org/pdf/2402.11131

This article outlines how Baseten optimized speculative decoding using the TensorRT-LLM framework to accelerate model inference. The authors detail overcoming technical hurdles such as inefficient batching, hardware contention, and server instability to make the technique viable for production environments. By synchronizing the execution of draft and target models and patching core software bugs, they achieved significantly lower latency, particularly for code generation tasks. The post also highlights the inclusion of essential enterprise features like streaming support, structured outputs, and OpenAI specification compatibility. Benchmark results demonstrate that these refinements can nearly double inference speeds while maintaining high output quality. Source: May 16 2025How we built production-ready speculative decoding with TensorRT-LLMBasetenPankaj Gupta, Justin Yi, Philip Kielyhttps://www.baseten.co/blog/how-we-built-production-ready-speculative-decoding-with-tensorrt-llm/

The researchers introduce CXL-SpecKV, a specialized architecture designed to overcome the memory bottlenecks of large language model serving by offloading key-value caches to remote memory. By utilizing Compute Express Link (CXL) and FPGA accelerators, the system enables memory disaggregation, which expands available storage capacity by up to eight times compared to standard GPU setups. A core innovation is a lightweight LSTM-based prefetcher that predicts upcoming token needs with 95% accuracy, effectively masking the latency of retrieving data from remote pools. The system further optimizes performance through an FPGA-driven compression engine that reduces bandwidth demands by roughly 4× without sacrificing model precision. Consequently, CXL-SpecKV delivers up to 3.2× higher throughput and significant cost reductions for datacenter environments. This hardware-software co-design demonstrates that intelligent memory management can efficiently scale AI infrastructure for next-generation workloads. Source: February 22 2026CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM ServingYale University, Columbia UniversityDong Liu, Yanxuan Yuhttps://doi.org/10.1145/3748173.3779188

The provided documents describe the development and evolution of EAGLE, a high-efficiency framework designed to accelerate Large Language Model (LLM) inference through speculative sampling. By performing autoregression at the feature level rather than the token level and incorporating shifted token sequences to manage sampling uncertainty, the original EAGLE achieves significant speedups while maintaining the exact output distribution of the target model. The technology has progressed into EAGLE-2, which introduces dynamic draft trees, and EAGLE-3, which further enhances performance by fusing multi-layer features and removing feature regression constraints during training. These advancements allow for a latency reduction of up to 6.5x and a doubling of throughput, making them compatible with modern reasoning models and popular serving frameworks like vLLM and SGLang. Overall, the sources highlight a shift toward test-time scaling and more expressive draft models to overcome the inherent slow speeds of sequential text generation.Sources:1) January 26, 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.Peking University, Microsoft Research, University of Waterloo, Vector Institute.Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang.https://arxiv.org/pdf/2401.150772) November 12, 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees.Peking University, Microsoft Research, University of Waterloo, Vector Institute.Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang.https://aclanthology.org/2024.emnlp-main.422.pdf4) April 23, 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test.Peking University, Microsoft Research, University of Waterloo, Vector Institute.Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang.https://arxiv.org/pdf/2503.018401) September 17 2025An Introduction to Speculative Decoding for Reducing Latency in AI Inference.NVIDIA.Jamie Li, Chenhan Yu, Hao Guo.https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/