I've heard that the end of the year is a good time to talk about the end of the world. AGI doom means different things to different people, so I thought it would be useful to get some idea what EAs think about it and what their probability of doom given AGI (P(doom)) is.

In a survey of thousands of AI experts, they think there is a wide variety of goodness/badness of AGI outcomes. However, some people think the outcome is very binary, extinction or bliss. In figure 10, I call these the MIRI flashlights (torches) because they are dark bars on the bottom with only bright above. That survey was not set up for longtermists, many of whom regard a scenario of disempowerment by AGI with rapid increases in human welfare as an existential catastrophe as it would be a drastic and permanent curtailment of the potential of humanity. Also, it did not disaggregate disempowerment and extinction.

Please select the mildest scenario that you would still consider doom. If you make a comment, it would be appreciated if you indicate your choice so others don’t have to search in the keys. If there is gradual disempowerment and [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.