Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Manifold Predicted the AI Extinction Statement and CAIS Wanted it Deleted, published by David Chee on June 12, 2023 on LessWrong.

Crosspost from

History may look back on CAIS’s AI risk statement as a pivotal moment IF we survive AI.

But, what if I told you that AI researchers “leaked” info about the statement on Manifold Markets over a week before it was published? This justifiably led to CAIS requesting the market to be deleted, concerned that it could damage the impact of the statement.

To provide some background, the statement was signed by the likes of Sam Altman, Demis Hassabis, Dario Amodei and other prominent AI figures. Short and with gravitas it read:

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Imagine if the CEOs of major oil companies banded together in the 1900s and told their respective governments, “We need to be careful about building infrastructure relying heavily on fossil fuels. It could be catastrophic for the environment long term.”

This may seem like an extreme comparison to make, but the reality is leading AI companies consider the risk significant enough to compromise on profit and progression. Well, that’s what their words say, time will tell how it is reflected in corporate decision-making.

I had an opportunity to talk to some of the insider traders and Dan Hendrycks from CAIS, and hope you enjoy a breakdown of one of our craziest markets and what we’ve learnt from it.

Background on Manifold Markets

Prediction markets allow people who lack expertise about certain topics to form precise models of what the future could look like thanks to the live-updating probabilities that are generated by traders. Trades can buy YES or NO shares, which fluctuate in price depending on the current probability (similar to sports betting odds).

Manifold Markets uses play money, which leads some to express skepticism of its efficacy. However, data suggest Manifold Markets are incredibly accurate!

Here is a graph from an analysis of Manifold Market’s calibration carried out by Vincent Luczkow. This places markets into buckets based on their probability at the middle of their lifespan. It then looks at the percentage of markets in each bucket that resolve to yes. To be well-calibrated, you want markets in the 10% bucket at their mid-point to resolve yes 10% of the time and no 90% of the time (this ideal is represented by the green line on the graph). Pretty accurate! Also, check out our calibration page or our performance on the midterms.

Manifold Markets is unique from other prediction platforms as its markets are all user-generated questions. Users have used our markets to make predictions about everything you can think of from global nuclear risk to their personal romantic endeavours. The core denominator between these markets is the mechanism used to generate the probability that predicts an unknown future event.

But what happens when you defy this, and a market is created by someone that does know the future?

The Tale of the Statement on AI Risk

It all started on May 20th, when a user called Quinesweeper created 3 markets titled, “Will there be another well-recognized letter/statement on AI risk by May [June, July] 31, 2023?” Whenever I write, “the/this market”, I will be referring to the market which was asking if the statement would be released before the end of May.

I’m including a screenshot of the traders who won the most profit and an annotated graph below which will be referenced throughout.

Initial trading

All markets start at 50% when created, but this market was quickly bid down to 10% by traders who had no evidence that a major statement would be published in the next 10 days. However, within the first few hours, there was already upwards buying press...