This research paper introduces SimpleQA, a new benchmark designed to assess the ability of large language models (LLMs) to answer factual questions accurately. The researchers focused on short, fact-seeking questions that have only one right answer, like trivia questions. SimpleQA is designed to be challenging even for the most advanced LLMs, like GPT-4, ensuring that the benchmark remains relevant as models continue to improve. The researchers were careful to ensure the questions were well-written, the answers could be easily verified, and the topics covered were diverse. To guarantee high quality, questions were reviewed by multiple AI trainers and supported by evidence from reliable sources. SimpleQA also measures how well models understand their own limitations, a concept called "calibration". This helps determine if LLMs can accurately assess their confidence in the answers they provide. By open-sourcing SimpleQA, the researchers hope to encourage the development of more trustworthy and reliable language models.