The August 26, 2024 academic paper introduces Quest, a novel algorithm designed to improve the inference efficiency of Long-Context Large Language Models (LLMs) by addressing the costly self-attention process caused by a large Key-Value (KV) cache. Quest utilizes Query-Aware Sparsity to dynamically identify and select only the critical KV cache pages based on the current query token, which significantly reduces the required memory movement during decoding. Unlike previous Query-Agnostic methods that evict tokens based on past information, Quest maintains high accuracy by never fully discarding context and achieves substantial speedups in self-attention latency, demonstrating its effectiveness across various long-context tasks. The authors provide a detailed breakdown of the methodology and experimental results showing Quest's superior efficiency and accuracy compared to existing baselines. Source: https://arxiv.org/pdf/2406.10774