Sign up to save your podcasts
Or
Share “About 30% of Humanity’s Last Exam chemistry/biology answers are likely wrong” by bohaska
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
“About 30% of Humanity’s Last Exam chemistry/biology answers are likely wrong” by bohaska
FutureHouse is a company that builds literature research agents. They tested it on the bio + chem subset of HLE questions, then noticed errors in them.
The post's first paragraph:
Humanity's Last Exam has become the most prominent eval representing PhD-level research. We found the questions puzzling and investigated with a team of experts in biology and chemistry to evaluate the answer-reasoning pairs in Humanity's Last Exam. We found that 29 ± 3.7% (95% CI) of the text-only chemistry and biology questions had answers with directly conflicting evidence in peer reviewed literature. We believe this arose from the incentive used to build the benchmark. Based on human experts and our own research tools, we have created an HLE Bio/Chem Gold, a subset of AI and human validated questions.
About the initial review process for HLE questions:
[...] Reviewers were given explicit instructions: “Questions should ask for something precise [...]
---
First published:
July 29th, 2025
Source:
https://www.lesswrong.com/posts/JANqfGrMyBgcKtGgK/about-30-of-humanity-s-last-exam-chemistry-biology-answers
---
Narrated by TYPE III AUDIO.
The post's first paragraph:
Humanity's Last Exam has become the most prominent eval representing PhD-level research. We found the questions puzzling and investigated with a team of experts in biology and chemistry to evaluate the answer-reasoning pairs in Humanity's Last Exam. We found that 29 ± 3.7% (95% CI) of the text-only chemistry and biology questions had answers with directly conflicting evidence in peer reviewed literature. We believe this arose from the incentive used to build the benchmark. Based on human experts and our own research tools, we have created an HLE Bio/Chem Gold, a subset of AI and human validated questions.
About the initial review process for HLE questions:
[...] Reviewers were given explicit instructions: “Questions should ask for something precise [...]
---
First published:
July 29th, 2025
Source:
https://www.lesswrong.com/posts/JANqfGrMyBgcKtGgK/about-30-of-humanity-s-last-exam-chemistry-biology-answers
---
Narrated by TYPE III AUDIO.
View all episodes
By LessWrong
4.8
1111 ratings
FutureHouse is a company that builds literature research agents. They tested it on the bio + chem subset of HLE questions, then noticed errors in them.
The post's first paragraph:
Humanity's Last Exam has become the most prominent eval representing PhD-level research. We found the questions puzzling and investigated with a team of experts in biology and chemistry to evaluate the answer-reasoning pairs in Humanity's Last Exam. We found that 29 ± 3.7% (95% CI) of the text-only chemistry and biology questions had answers with directly conflicting evidence in peer reviewed literature. We believe this arose from the incentive used to build the benchmark. Based on human experts and our own research tools, we have created an HLE Bio/Chem Gold, a subset of AI and human validated questions.
About the initial review process for HLE questions:
[...] Reviewers were given explicit instructions: “Questions should ask for something precise [...]
---
First published:
July 29th, 2025
Source:
https://www.lesswrong.com/posts/JANqfGrMyBgcKtGgK/about-30-of-humanity-s-last-exam-chemistry-biology-answers
---
Narrated by TYPE III AUDIO.
The post's first paragraph:
Humanity's Last Exam has become the most prominent eval representing PhD-level research. We found the questions puzzling and investigated with a team of experts in biology and chemistry to evaluate the answer-reasoning pairs in Humanity's Last Exam. We found that 29 ± 3.7% (95% CI) of the text-only chemistry and biology questions had answers with directly conflicting evidence in peer reviewed literature. We believe this arose from the incentive used to build the benchmark. Based on human experts and our own research tools, we have created an HLE Bio/Chem Gold, a subset of AI and human validated questions.
About the initial review process for HLE questions:
[...] Reviewers were given explicit instructions: “Questions should ask for something precise [...]
---
First published:
July 29th, 2025
Source:
https://www.lesswrong.com/posts/JANqfGrMyBgcKtGgK/about-30-of-humanity-s-last-exam-chemistry-biology-answers
---
Narrated by TYPE III AUDIO.
More shows like LessWrong (Curated & Popular)
View all
Conversations with Tyler
2,398 Listeners
Astral Codex Ten Podcast
126 Listeners
Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas
4,119 Listeners
ManifoldOne
90 Listeners
The Jim Rutt Show
255 Listeners
Machine Learning Street Talk (MLST)
91 Listeners
Dwarkesh Podcast
425 Listeners
Hard Fork
5,462 Listeners
Clearer Thinking with Spencer Greenberg
131 Listeners
Razib Khan's Unsupervised Learning
199 Listeners
No Priors: Artificial Intelligence | Technology | Startups
124 Listeners
Latent Space: The AI Engineer Podcast
71 Listeners
"Econ 102" with Noah Smith and Erik Torenberg
145 Listeners
Complex Systems with Patrick McKenzie (patio11)
122 Listeners
LessWrong posts by zvi
0 Listeners