This research paper describes the creation and analysis of GPQA, a new set of multiple-choice questions designed to be very hard to answer, even with the help of Google. The questions cover advanced topics in biology, physics, and chemistry, and were written and checked for accuracy by experts with PhDs in those fields. The researchers made sure the questions were extra tough by having other experts, called non-experts, try to answer them using the internet. These non-experts also had PhDs, but in different subjects. The goal was to create questions that would be challenging even for very smart people who don't have specific knowledge in the subject. The researchers also tested the questions on advanced AI systems, like GPT-4, to see how well they could answer them. They found that even with access to the internet, the AI systems struggled to do as well as the experts, showing just how difficult these questions really are. The researchers hope that GPQA will be a valuable tool for testing new ways to help people understand and use information from AI systems, especially when those systems are tackling really hard problems that even experts find challenging.

https://arxiv.org/pdf/2311.12022