I think AI agents (trained end-to-end) might intrinsically prefer power-seeking, in addition to whatever instrumental drives they gain.

The logical structure of the argument

Premises

1. People will configure AI systems to be autonomous and reliable in order to accomplish tasks.
2. This configuration process will reinforce & generalize behaviors which complete tasks reliably.
3. Many tasks involve power-seeking.
4. The AI will complete these tasks by seeking power.
5. The AI will be repeatedly reinforced for its historical actions which seek power.
6. There is a decent chance the reinforced circuits (“subshards”) prioritize gaining power for the AI's own sake, not just for the user's benefit.

Conclusion: There is a decent chance the AI seeks power for itself, when possible.

Read the full post at turntrout.com/intrinsic-power-seeking

Find out when I post more content: newsletter & RSS

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.