These sources review historically speculative decoding, an innovative technique designed to accelerate Large Language Model (LLM) inference without reducing output quality. Large models are traditionally slow because they generate text one token at a time, a process limited by hardware memory bandwidth. To solve this, a much smaller and faster approximation model suggests multiple future tokens in parallel. The larger target model then verifies these guesses in a single computation step, accepting correct predictions and correcting errors. This method achieves 2x–3x speed improvements and is currently utilized in major products like Google Search. Ultimately, speculative decoding allows for cheaper and faster AI services while guaranteeing the exact same mathematical distribution as the original model.Sources:1) December 6 2024Looking back at speculative decodingGoogle ResearchYaniv Leviathan, Matan Kalman, Yossi Matiashttps://research.google/blog/looking-back-at-speculative-decoding/2) 2023Fast Inference from Transformers via Speculative DecodingGoogle ResearchYaniv Leviathan, Matan Kalman, Yossi Matiashttps://arxiv.org/pdf/2211.17192