tl;dr:

1. In terms of financial interests of an AI company, bankruptcy and the world ending are both equally bad. If a company acted in line with its financial interests[1], it would happily accept significant extinction risk for increased revenue.
2. There are plausible mechanisms which would allow a company to act like this even if virtually every employee would prefer the opposite. (For example, selectively hiring people with biased beliefs or exploiting collective action problems.)
3. In particular, you can hold that an AI company is completely untrustworthy even if you believe that all of its employees are fine people.

Epistemic status & disclaimers: The mechanisms I describe definitely play some role in real AI companies. But in practice, there are more things going on simultaneously and this post is not trying to give a full picture.[2][3]Also, none of this is meant to be novel, but rather just putting [...]

---

Outline:

(01:12) From financial point of view, bankruptcy is no worse than destroying the world

(02:53) How to Not Act in Line with Employee Preferences

(07:29) Well... and why does this matter?

The original text contained 9 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.