This post is the version of Yudkowsky's argument for inner misalignment that I wish I'd had in my head a few years ago. I don't claim that it's novel, that I endorse it, or even that Yudkowsky would endorse it; it's primarily an attempt to map his ideas into an ontology that makes sense to me (and hopefully others).

This post is formulated in terms of three premises, which I explore in turn. My arguments deliberately gloss over some nuances and possible objections; in a follow-up post, I'll explore three of them. In a third post I'll dig into the objection I find most compelling, and outline a research agenda that aims to flesh it out into a paradigm for thinking about cognition more generally, which I'm calling coalitional agency.

Background

An early thought experiment illustrating the [...]

---

Outline:

(00:53) Background

(02:22) Increasing intelligence requires compressing representations

(04:35) The simplest goals are highly decomposable broadly-scoped utility functions

(08:50) Increasingly intelligent AIs will converge towards squiggle-maximization

---

First published:

Source:

Narrated by TYPE III AUDIO.