The idea of a “basin of attraction around corrigibility” motivates much of prosaic alignment research. Essentially this is an abstract way of thinking about the process of iteration on AGI designs. Engineers test to find problems, then understand the problems, then design fixes. The reason we need corrigibility for this is that a non-corrigible agent generally has incentives to interfere with this process. The concept was introduced by Paul Christiano:

... a corrigible agent prefers to build other agents that share the overseer's preferences — even if the agent doesn’t yet share the overseer's preferences perfectly. After all, even if you only approximately know the overseer's preferences, you know that the overseer would prefer the approximation get better rather than worse.
Thus an entire neighborhood of possible preferences lead the agent towards the same basin of attraction. We just have to get “close enough” that we are corrigible, we don’t need to build an agent which exactly shares humanity's values, philosophical views, or so on.
In addition to making the initial target bigger, this gives us some reason to be optimistic about the dynamics of AI systems iteratively designing new AI systems. Corrigible systems want to design more corrigible [...]

---

Outline:

(02:40) Corrigibility

(07:45) My argument: The engineering feedback loop will use up all its fuel

(08:31) There are many fields where test-driven engineering is far from sufficient

(10:41) The corresponding story for iteratively building corrigible AGI

(12:18) How do we know the distribution shift will stress parts of the AGI?

(16:13) Tying this back to the basin of attraction

(18:09) A counterargument: The prosaic case

(21:19) More specific counterarguments

(25:07) One way things could go well

(27:00) Conclusion and implications

The original text contained 16 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.