Suppose you can tell your AI to meet a certain spec (e.g. cure cancer), but most plans that meet the spec are unsafe (e.g. involve killing everyone, or so Rob Bensinger thinks). In these cases, a quantilizer is insufficient for safety due to instrumental convergence.[1]

But suppose we can also give the agent a dispreference for unsafe actions like murder. In effect it has unsafe long-term goals but we control its immediate preferences.[2] When can we get the agent to cure cancer safely rather than murder you? Let's make a model with some unrealistic assumptions.

The murderous shortcut game

This is basically the simplest game in which murder becomes more likely as the task gets longer.

• An AI agent's goal is to build a Lego tower with a human operator present. To meet the spec, it must place N bricks in a row, otherwise it fails.
• The agent [...]

---

Outline:

(00:44) The murderous shortcut game

(03:37) Takeaways

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.