expectation calibrator: freeform draft, posting partly to practice lowering my own excessive standards.

So, a few months back, I finally got around to reading Nick Bostrom's Superintelligence, a major player in the popularization of AI safety concerns. Lots of the argument was stuff I'd internalized a long time ago, reading the sequences and even living with AI alignment researchers for a few months. But I wanted to see the circa-2015 risk-case laid out as thorougly as possible, partly as an excusion to learn more about my own subculture, and partly to see if I'd missed anything crucial to why all of us consider future advanced AI systems so dangerous.

But for the most part, the main effect of reading that book was actually to finally shatter a certain story I'd been telling myself about the AI doom scenario, the cracks in which had been starting to show for several [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.