In this post, I’ll make a technical point that comes up when thinking about risks from scheming AIs from a control perspective.

In brief: Consider a deployment of an AI in a setting where it's going to be given a sequence of tasks, and you’re worried about a safety failure that can happen suddenly. Every time the AI tries to attack, one of three things happens: we win, we lose, or the deployment continues (and the AI probably eventually attempts to attack again). So there's two importantly different notions of attacks “failing”: we can catch the attack (in which case we win), or the attack can fail silently, either because it's defective or because it's defused by safety techniques, such that the attack doesn’t end up causing an unacceptable outcome but the AI also doesn’t get caught (in which case the deployment continues). When reasoning about how safe a [...]

---

Outline:

(03:33) Execute/replace/audit protocols

(07:47) Execute/replace/audit protocols lead to win/continue/lose scenarios

(10:20) Example: analyzing the safety of defer-to-trusted using the probes from “Simple probes can catch sleeper agents”

The original text contained 1 footnote which was omitted from this narration.

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.