Subtitle: Reward-seekers are supposed to be safer because they respond to incentives under developer control. But what if they also respond to incentives that aren't?.

Reward-seekers are usually modeled as responding only to local incentives administered by developers. Here I ask: Will AIs or humans be able to influence their incentives at a distance—e.g., by retroactively reinforcing actions substantially in the future or by committing to run many copies of them in simulated deployments with different incentives?

If reward-seekers are responsive to distant incentives, it fundamentally changes the threat model, and is probably bad news for developers on balance. The core problem is asymmetric control: developers can relatively[1]tightly control local incentives—the reward signal during training and deployment—but they can’t prevent distant actors from offering competing incentives. This means a remotely-influenceable reward-seeker might overall act like a schemer: strategically undermining developer control, letting attacks through as a monitor, and hiding its misaligned propensities, not because of a flaw in its local training, but because it's responding to incentives developers don’t control.

I focus on three sources of distant influence on a reward-seeker: (1) human adversaries (e.g., a competing nation) who claim that they will retroactively reward or simulate the [...]

---

Outline:

(02:48) Remotely-influenceable reward-seekers

(09:03) The selection pressures that shaped the AIs cognition were local

(10:20) But distant incentives will likely avoid conflicting with immediate incentives

(11:54) Reward-seekers might follow distant incentives once immediate incentives are relatively weak

(12:52) What can we do to avoid remotely-influenceable reward-seekers?

(16:47) What can we do to mitigate risk given remotely-influenceable reward-seekers?

(19:59) Appendix: Anticipated takeover complicity doesnt require strong remote influenceability

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.