The coalitional frame

Earlier in this sequence I laid out an argument that the goals of increasingly intelligent AIs will become increasingly systematized, until they converge to squiggle-maximization. In my last post, though, I touched on two reasons why this convergence might not happen: humans trying to prevent it, and AIs themselves trying to prevent it. I don’t have too much more to say about the former, but it's worth elaborating on the latter.

The best way to understand the deliberate protection of existing goals is in terms of Bostrom's notion of instrumental convergence. Bostrom argues that goal preservation will be a convergent instrumental strategy for a wide range of agents. Perhaps it's occasionally instrumentally useful to change your goals—but once you’ve done so, you’ll never want to course-correct back towards your old goals. So this is [...]

---

Outline:

(00:07) The coalitional frame

(06:20) Some intuitions favoring the coalitional frame

(09:54) Next steps

---

First published:

Source:

Narrated by TYPE III AUDIO.