Much of my thinking over the last year has focusing on understanding the concept of "distributed agents", as opposed to the "centralized agents" that the existing paradigm of rational agency describes. Roughly speaking, centralized agents are more efficient (as sometimes formalized by the notion of "coherence"), while distributed agents are more robust.

Unfortunately the latter property is hard to formalize, since it requires that you perform well even in unpredicted (and sometimes unpredictable) situations. I give some tentative characterizations of distributed agents below, but there's still a lot of work to be done to formally define distributed agents. And ultimately I'd like to go further, to understand how agents can have both properties—which is roughly what I mean by "coalitional agency".

I gave a talk on the distinction about seven months ago. I'd been hoping to write up the main ideas at more length, but since that doesn't look like it'll happen any time soon, I'm sharing the slides below. Hopefully they're reasonably comprehensible by themselves, but feel free to ask questions about any parts that are unclear.

See more on my interpretation of Yudkowsky here; note that he disagrees with my emphasis on compression though (as per the [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.