AI Post Transformers

The provided documents describe the development and evolution of EAGLE, a high-efficiency framework designed to accelerate Large Language Model (LLM) inference through speculative sampling. By performing autoregression at the feature level rather than the token level and incorporating shifted token sequences to manage sampling uncertainty, the original EAGLE achieves significant speedups while maintaining the exact output distribution of the target model. The technology has progressed into EAGLE-2, which introduces dynamic draft trees, and EAGLE-3, which further enhances performance by fusing multi-layer features and removing feature regression constraints during training. These advancements allow for a latency reduction of up to 6.5x and a doubling of throughput, making them compatible with modern reasoning models and popular serving frameworks like vLLM and SGLang. Overall, the sources highlight a shift toward test-time scaling and more expressive draft models to overcome the inherent slow speeds of sequential text generation.Sources:1) January 26, 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.Peking University, Microsoft Research, University of Waterloo, Vector Institute.Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang.https://arxiv.org/pdf/2401.150772) November 12, 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees.Peking University, Microsoft Research, University of Waterloo, Vector Institute.Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang.https://aclanthology.org/2024.emnlp-main.422.pdf4) April 23, 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test.Peking University, Microsoft Research, University of Waterloo, Vector Institute.Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang.https://arxiv.org/pdf/2503.018401) September 17 2025An Introduction to Speculative Decoding for Reducing Latency in AI Inference.NVIDIA.Jamie Li, Chenhan Yu, Hao Guo.https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/

These sources review historically speculative decoding, an innovative technique designed to accelerate Large Language Model (LLM) inference without reducing output quality. Large models are traditionally slow because they generate text one token at a time, a process limited by hardware memory bandwidth. To solve this, a much smaller and faster approximation model suggests multiple future tokens in parallel. The larger target model then verifies these guesses in a single computation step, accepting correct predictions and correcting errors. This method achieves 2x–3x speed improvements and is currently utilized in major products like Google Search. Ultimately, speculative decoding allows for cheaper and faster AI services while guaranteeing the exact same mathematical distribution as the original model.Sources:1) December 6 2024Looking back at speculative decodingGoogle ResearchYaniv Leviathan, Matan Kalman, Yossi Matiashttps://research.google/blog/looking-back-at-speculative-decoding/2) 2023Fast Inference from Transformers via Speculative DecodingGoogle ResearchYaniv Leviathan, Matan Kalman, Yossi Matiashttps://arxiv.org/pdf/2211.17192

We review three different papers which focus on different KV cache optimizations techniques using different KV selection algorithms types: static vs dynamic. StreamingLLM and SnapKV use static KV selection methods. The dynamic strategy KV selection is introduced in PQCache. These different KV selection algorithms explore different KV budgets and speculation lengths to estimate optimal theoretical speedups. A KV cache selection strategy is considered static when the tokens selected for retention are determined either by fixed positional rules or by an initial evaluation of the prompt, remaining unchanged during the generation of new tokens. StreamingLLM (Positional Static Strategy): StreamingLLM employs a static approach by retaining a fixed set of initial tokens alongside a rolling window of the most recent tokens. This is based on the discovery of the "attention sink" phenomenon, where LLMs disproportionately allocate high attention scores to the very first few tokens of a sequence, regardless of their semantic importance, simply to satisfy the SoftMax function's requirement to sum to one. By statically anchoring these initial tokens (usually just 4) and keeping a sliding window of recent tokens, StreamingLLM prevents the model from collapsing during infinite sequence generation. SnapKV (Observation-Based Static Strategy): SnapKV is static because it selects its KV cache before generation begins and keeps this selection fixed. It operates on the observation that the attention allocation pattern of an LLM stays remarkably consistent throughout the generation phase. SnapKV uses an "observation window" at the very end of the user's prompt to "vote" on which preceding KV features are most important. It then clusters and compresses these important features, concatenates them with the observation window, and uses this statically compressed KV cache for all subsequent generation steps. Dynamic KV Selection Strategy (PQCache): A solution is dynamic when the subset of KV pairs used for attention computation changes step-by-step in real-time, depending on the specific token currently being generated. PQCache (Retrieval-Based Dynamic Strategy): PQCache fundamentally treats KV cache selection as an Information Retrieval or Approximate Nearest Neighbor Search (ANNS) problem. It acknowledges a critical flaw in static dropping methods: tokens that initially appear unimportant might suddenly gain relevance in later generation steps. During the autoregressive decoding phase, PQCache uses a lightweight Product Quantization (PQ) technique to compress keys into centroids and codes. For each newly generated token, it dynamically multiplies the token's query with the PQ centroids to approximate attention scores, retrieving only the top-k most relevant KV pairs from the CPU to perform selective attention. PQCache shows the strongest evidence for scaling accurately across massive contexts without losing critical information. By dynamically retrieving the top-k tokens, it improves model quality scores by 4.60% over static methods (like SnapKV) on the InfiniteBench dataset (which averages 100K+ token lengths).Sources:1)September 2023Efficient Streaming Language Models With Attention SinksMassachusetts Institute of Technology, Meta AI, Carnegie Mellon University, NVIDIAGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewishttps://arxiv.org/pdf/2309.174532)April 2024SnapKV: LLM Knows What You are Looking for Before GenerationUniversity of Illinois Urbana-Champaign, Cohere, Princeton UniversityYuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, Deming Chenhttps://arxiv.org/pdf/2404.144693)June 2025PQCache: ProductQuantization-based KVCache for Long Context LLM InferencePeking University, Purdue University, Baichuan Inc.Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, Bin Cuihttps://arxiv.org/pdf/2407.12820

We review an April 3, 2025 research collaboration between CMU, Moffett AI and Together AI which introduces MagicDec, a new framework designed to accelerate the serving of long-context large language models through speculative decoding. Previously, conventional wisdom discouraged using speculative decoding (SD) for large batches, as the verification step was believed to be too compute-heavy and inefficient. MagicDec proves that this limitation only applies to short sequences. The paper demonstrates that once sequences pass a critical length, the massive memory cost of loading the KV cache becomes the true bottleneck, shifting inference from being compute-bound to memory-bound. The authors address the memory bottleneck by applying KV selection algorithms to compress the draft model's KV cache during speculative decoding. They evaluated different KV selection algorithms, both static (SnapKV, StreamingLLM) and dynamic (PQKache). They observed that PQCache leads to high token acceptance rates but it incurs substantial, batch-size-dependent search costs. On tasks like common word extraction and question answering, SnapKV dominated PQCache because it achieved similar acceptance rates without the heavy search overhead. For complex tasks like "needle in a haystack," PQCache initially performed better because its acceptance rate was near 100%. However, as batch sizes increased, PQCache's search costs became too expensive, and SnapKV once again outperformed it. By effectively managing the memory pressure through KV compression, the system can maintain a high token acceptance rate, minimize costly verification steps, and achieve significant speedups for large batches. The authors test sequence (prefill) lengths ranging from 1k up to 100k tokens. In their theoretical memory footprint analyses, they project context lengths up to 128k tokens. For batch sizes, the core end-to-end speedup experiments focus on large batch sizes ranging from 32 to 256. Additionally, some ablation studies test batch sizes up to 512, and theoretical trade-off analyses chart batch sizes up to 1024. To validate their framework across different hardware capabilities, the researchers used configurations of 4 to 8 GPUs. Specifically, their experiments were run on clusters of 8xA100, 8xH100, 4xH100, and 8xL40 GPUs. The paper provides the industry with a framework to break the latency-throughput tradeoff when serving long-context Large Language Models (LLMs) at scale. This enables the highly efficient scaling of long-context applications—such as retrieval-augmented generation (RAG), extensive document analysis, code generation, and complex agent workflows—across large batches of concurrent users. Source: 2024 MAGICDEC: BREAKING THE LATENCY-THROUGHPUT TRADEOFF FOR LONG CONTEXT GENERATION WITH SPECULATIVE DECODING Carnegie Mellon University, Moffett AI, Together AI Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, Beidi Chen https://arxiv.org/pdf/2408.11049

In a collaboration between Google DeepMind, University of Texas at Austin, University of Washington and Harvard published on December 2024 researchers introduce MatFormer, a novel elastic Transformer architecture designed to improve the efficiency of large-scale foundation models. Unlike traditional models that require independent training for different sizes, this framework allows a single universal model to provide hundreds of smaller, accurate submodels without any additional training. This is achieved by embedding a nested "matryoshka" structure within the transformer blocks, allowing layers and attention heads to be adjusted based on available compute resources. The authors also propose a Mix’n’Match heuristic to identify the most effective submodel configurations for specific latency or hardware constraints. Their research demonstrates that MatFormer maintains high performance across various tasks, offering improved consistency between large and small models during deployment. Consequently, this approach enhances techniques like speculative decoding and image retrieval while significantly reducing the memory and cost overhead of serving AI models. Source: 2024MatFormer: Nested Transformer for Elastic InferenceGoogle DeepMind, University of Texas at Austin, University of Washington, Harvard UniversityDevvrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkov, Hannaneh Hajishirzi, Sham Kakade, Ali Farhadi, Prateek Jainhttps://arxiv.org/pdf/2310.07707

QuantSpec is a novel self-speculative decoding framework designed to accelerate the inference of Large Language Models, particularly in long-context scenarios. The system addresses memory and latency bottlenecks by employing a hierarchical 4-bit quantized KV cache and quantized weights, allowing a draft model to share the same architecture as the target model. This approach maintains a high token acceptance rate exceeding 90% while delivering end-to-end speedups of up to 2.5×. Additionally, the authors introduce a double full-precision buffer to store the most recent tokens, which prevents accuracy loss and minimizes the computational overhead of frequent re-quantization. By optimizing memory-bound attention operations, QuantSpec achieves superior performance and lower memory requirements compared to existing sparse-cache alternatives. The research demonstrates that integrating advanced quantization with speculative decoding can significantly enhance LLM scalability without sacrificing generation quality. Source: February 5, 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV CacheUC Berkeley, Apple, ICSI, LBNLRishabh Tiwari, Haocheng Xi, Aditya Tomar, Coleman Hooper, Sehoon Kim, Maxwell Horton, Mahyar Najibi, Michael W. Mahoney, Kurt Keutzer, Amir Gholamihttps://arxiv.org/pdf/2502.10424

On the February 19, 2026 paper Google Deepmind introduces Unified Latents (UL), a novel framework for generative modeling that jointly trains an encoder, a diffusion prior, and a diffusion decoder. By incorporating a fixed amount of Gaussian noise during the encoding process, the method creates a stable and interpretable bound on latent information. This architecture allows for precise control over the reconstruction-modeling tradeoff through simple hyperparameters like the loss factor and sigmoid weighting. Experimental results demonstrate that this approach is more computationally efficient than existing methods, achieving superior image and video generation quality on benchmarks like ImageNet and Kinetics-600. Ultimately, the research offers a principled alternative to traditional Variational AutoEncoders by simplifying the training objective and preventing issues like posterior collapse. Source: February 19, 2026 Unified Latents (UL): How to train your latents Google DeepMind Jonathan Heek, Emiel Hoogeboom, Thomas Mensink, Tim Salimans https://arxiv.org/pdf/2602.17270

The Deepmind February 3, 2023 paper "Accelerating Large Language Model Decoding with Speculative Sampling introduced speculative sampling, a novel algorithm designed to increase the speed of Large Language Model (LLM) decoding without altering the final output. The researchers utilize a smaller, faster draft model to predict multiple potential tokens, which are then verified in parallel by a larger, more powerful target model. By employing a unique rejection sampling scheme, the system ensures that the generated text remains mathematically identical to the distribution of the original large model. When tested with the 70 billion parameter Chinchilla model, this technique achieved a 2 to 2.5 times speedup in processing. The method is particularly effective because it overcomes the memory bandwidth bottlenecks typical of standard autoregressive generation. Ultimately, it provides a practical way to reduce latency in large-scale AI applications without sacrificing sample quality. Source: February 3 2023 Accelerating Large Language Model Decoding with Speculative Sampling DeepMind Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John Jumper https://arxiv.org/pdf/2302.01318

The provided sources explore advanced techniques for optimizing large language model (LLM) inference, specifically by addressing the memory bottlenecks of the Key-Value (KV) cache. KVQuant introduces a high-precision quantization framework that utilizes per-channel scaling, non-uniform datatypes, and sparse outlier handling to compress activations to sub-4-bit precision with minimal accuracy loss. Similarly, the KIVI algorithm proposes a tuning-free 2-bit quantization strategy that differentiates between key and value cache distributions to increase throughput. Shifting from quantization to architectural pruning, DuoAttention identifies specific Retrieval Heads that require full context while reducing Streaming Heads to constant memory usage by focusing only on recent tokens and attention sinks. Together, these methods enable LLMs to process million-level context lengths on standard hardware by drastically reducing the architectural and computational footprint of stored activations.Sources:1) 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache QuantizationUniversity of California, Berkeley, ICSI, LBNLColeman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholamihttps://arxiv.org/pdf/2401.180792) 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV CacheRice University, Texas A&M University, Stevens Institute of Technology, Carnegie Mellon UniversityZirui Liu, Jiayi Yuan, Hongye Jin, Shaochen (Henry) Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, Xia Huhttps://arxiv.org/pdf/2402.027503) 2024QAQ: Quality Adaptive Quantization for LLM KV CacheNanjing UniversityShichen Dong, Wen Cheng, Jiayu Qin, Wei Wanghttps://arxiv.org/pdf/2403.046434) May 8, 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled QuantizationRice University, Stevens Institute of Technology, ThirdAI Corp.Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, Anshumali Shrivastavahttps://arxiv.org/pdf/2405.039175) 2024DUOATTENTION: EFFICIENT LONG-CONTEXT LLM INFERENCE WITH RETRIEVAL AND STREAMING HEADSMIT, Tsinghua University, SJTU, University of Edinburgh, NVIDIAGuangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, Song Hanhttps://arxiv.org/pdf/2410.108196) 2025MILLION: MasterIng Long-Context LLM Inference Via Outlier-Immunized KV Product QuaNtizationShanghai Jiao Tong University, Shanghai Qi Zhi Institute, Huawei Technologies Co., Ltd, China University of Petroleum-BeijingZongwu Wang, Peng Xu, Fangxin Liu, Yiwei Hu, Qingxiao Sun, Gezi Li, Cheng Li, Xuan Wang, Li Jiang, Haibing Guanhttps://arxiv.org/pdf/2504.03661

These sources detail advanced reinforcement learning frameworks designed to improve how quadruped robots navigate difficult, real-world environments. The first source introduces a single-stage teacher-student method that utilizes skeleton information and a system-response model to achieve more natural, stable movement. The second source proposes ZSL-RPPO, a zero-shot learning architecture that eliminates the need for imitation by training recurrent neural networks directly in partially observable settings. Both research papers prioritize bridging the simulation-to-reality gap, ensuring robots can handle unpredictable terrain like stairs, oily surfaces, and grass. By employing domain randomization and specialized encoders, these frameworks enhance the robustness and adaptability of robotic locomotion without requiring extensive manual tuning. Together, they represent a shift toward more efficient training paradigms that produce versatile and resilient autonomous behaviors.Sources:1)October 22 2025Skeleton Information-Driven Reinforcement Learning Framework for Robust and Natural Motion of Quadruped RobotsGuangdong University of Technology, University of MacauHuiyang Cao, Hongfa Lei, Yangjun Liu, Zheng Chen, Shuai Shi, Bingquan Li, Weichao Xu, Zhi-Xin Yanghttps://doi.org/10.3390/sym171117872)March 2024ZSL-RPPO: Zero-Shot Learning for Quadrupedal Locomotion in Challenging Terrains using Recurrent Proximal Policy OptimizationHuawei Technologies, Huawei Munich Research Center, University College London, Huawei Noah's Ark Lab, East China Normal UniversityYao Zhao, Tao Wu, Yijie Zhu, Xiang Lu, Jun Wang, Haitham Bou-Ammar, Xinyu Zhang, Peng Duhttps://arxiv.org/pdf/2403.019283)May 2025End-to-End Multi-Task Policy Learning from NMPC for Quadruped LocomotionBonn-Rhein-Sieg University of Applied Sciences, University of Bonn, Fraunhofer Institute for Intelligent Analysis and Information SystemsAnudeep Sajja, Shahram Khorshidi, Sebastian Houben, Maren Bennewitzhttps://arxiv.org/pdf/2505.08574