This research paper advocates for incorporating rigorous statistical methods into the evaluation of large language models (LLMs). It introduces formulas for calculating standard errors and confidence intervals, emphasizing the importance of accounting for clustered data and paired comparisons between models. The paper details variance reduction techniques, including resampling and using next-token probabilities, and provides a sample-size formula for power analysis to determine the necessary number of evaluation questions. Ultimately, the authors aim to shift the focus from simply achieving the highest score to conducting statistically sound experiments that provide more reliable and informative insights into LLM capabilities.

ai , llm , anthropic , artificial intelligence , arxiv , research , paper , publication , genai , generativeai, agentic