TL;DR: People often use the thought experiment of flipping a coin, giving 50% chance of huge gain and 50% chance of losing everything, to say that maximizing utility is bad. But the real problem is that our intuitions on this topic are terrible, and there's no real paradox if you adopt the premise in full.

Epistemic status: confident, but too lazy to write out the math

There's a thought experiment that I've sometimes heard as a counterargument to strict utilitarianism. A god/alien/whatever offers to flip a coin. Heads, it slightly-more-than-doubles the expected utility in the world. Tails, it obliterates the universe. An expected-utility maximizer, the argument goes, keeps taking this bet until the universe goes poof. Bad deal.

People seem to love citing this thought experiment when talking about Sam Bankman-Fried. We should have known he was wrong in the head, critics sigh, when he said he'd bet the [...]

---

Outline:

(01:30) There is a lot of value in the universe

(03:48) We already court apocalypse

(05:09) Everything breaks down at infinity

(06:23) Conclusion

---

First published:

Source:

Narrated by TYPE III AUDIO.