The May 21, 2024 paper introduces Vidur, a new, high-fidelity simulation framework designed to optimize the deployment and performance of Large Language Model (LLM) inference. The authors explain that experimentally optimizing LLM deployment is prohibitively expensive, requiring exploration of a vast configuration space of system parameters like parallelization strategies and batching techniques, which can cost hundreds of thousands of dollars and thousands of GPU hours. Vidur addresses this by using predictive modeling and experimental profiling of LLM operators to estimate end-to-end performance metrics, achieving less than 9% error in latency estimation. Complementing the simulator is Vidur-Search, a configuration search tool that leverages Vidur to automatically identify the most cost-effective deployment settings that meet application performance constraints, reducing optimization time from months of GPU time to approximately one hour on a CPU machine. The research emphasizes that the optimal configuration depends on both the LLM and the specific workload trace, justifying the need for a rapid simulation tool like Vidur. Source: May 21, 2024 VIDUR: A LARGE-SCALE SIMULATION FRAMEWORK FOR LLM INFERENCE https://arxiv.org/pdf/2405.05465