Researchers introduced on May 2024 self-speculative decoding, a novel "plug-and-play" inference scheme designed to accelerate Large Language Models (LLMs) without requiring auxiliary models or extra memory. This method utilizes a two-stage process where a faster, lower-quality draft is generated by selectively skipping intermediate layers of the original model. These draft tokens are then validated in a single forward pass by the full LLM, ensuring the final output remains identical to standard autoregressive decoding. To optimize performance, the system employs Bayesian optimization to identify the best layers to skip and an adaptive draft-exiting mechanism to stop generation when confidence is low. Benchmarks on models like LLaMA-2 show significant speedups of up to 1.99× across text and code generation tasks. Ultimately, this approach offers a cost-effective and lossless solution for reducing latency in large-scale AI applications. Source: May 20, 2024 Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding. Zhejiang University, University of California, Irvine. Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, Sharad Mehrotra. https://arxiv.org/pdf/2309.08168.