Metaculus's Q4 Benchmark series is now live. Click here to learn more and to compete for $30,000. Benchmark the state of the art in AI forecasting against the best humans on real-world questions.

Metaculus's Q3 AI Benchmarking Series aimed to assess how the best bots compare to the best humans on real-world forecasting questions, like those found on Metaculus. Over the quarter we had 55 bots compete for $30,000 on 255 weighted questions with a team of 10 Pros serving as a human benchmark. We found that Pro forecasters were significantly better than top bots (p = 0.036) using log scoring with a weighted t-test.

This main result compares the median forecast of 10 Pro Forecasters against the median forecast of 9 top bots on a set of 113 questions that both humans and bots have answered.

That analysis follows the methodology we laid out before the [...]

---

Outline:

(02:00) Selecting a Bot Team

(03:16) Comparing the Bots to Pros

(06:33) Calibration Curves

(08:31) Discrimination Plots

(10:10) Scope Sensitivity

(11:45) Comparing Individual Bots to the Pros

(12:20) Metaculus Bots

(14:05) Discussion

The original text contained 9 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.