
Sign up to save your podcasts
Or
This post is the version of Yudkowsky's argument for inner misalignment that I wish I'd had in my head a few years ago. I don't claim that it's novel, that I endorse it, or even that Yudkowsky would endorse it; it's primarily an attempt to map his ideas into an ontology that makes sense to me (and hopefully others).
This post is formulated in terms of three premises, which I explore in turn. My arguments deliberately gloss over some nuances and possible objections; in a follow-up post, I'll explore three of them. In a third post I'll dig into the objection I find most compelling, and outline a research agenda that aims to flesh it out into a paradigm for thinking about cognition more generally, which I'm calling coalitional agency.
Background
An early thought experiment illustrating the [...]
---
Outline:
(00:53) Background
(02:22) Increasing intelligence requires compressing representations
(04:35) The simplest goals are highly decomposable broadly-scoped utility functions
(08:50) Increasingly intelligent AIs will converge towards squiggle-maximization
---
First published:
Source:
Narrated by TYPE III AUDIO.
This post is the version of Yudkowsky's argument for inner misalignment that I wish I'd had in my head a few years ago. I don't claim that it's novel, that I endorse it, or even that Yudkowsky would endorse it; it's primarily an attempt to map his ideas into an ontology that makes sense to me (and hopefully others).
This post is formulated in terms of three premises, which I explore in turn. My arguments deliberately gloss over some nuances and possible objections; in a follow-up post, I'll explore three of them. In a third post I'll dig into the objection I find most compelling, and outline a research agenda that aims to flesh it out into a paradigm for thinking about cognition more generally, which I'm calling coalitional agency.
Background
An early thought experiment illustrating the [...]
---
Outline:
(00:53) Background
(02:22) Increasing intelligence requires compressing representations
(04:35) The simplest goals are highly decomposable broadly-scoped utility functions
(08:50) Increasingly intelligent AIs will converge towards squiggle-maximization
---
First published:
Source:
Narrated by TYPE III AUDIO.
26,420 Listeners
2,387 Listeners
7,893 Listeners
4,132 Listeners
87 Listeners
1,459 Listeners
9,040 Listeners
87 Listeners
390 Listeners
5,431 Listeners
15,216 Listeners
476 Listeners
121 Listeners
75 Listeners
459 Listeners