The question I actually try to answer in this post is a broader one (that doesn't work as well as a title): Should we incorporate proxies for desired behavior into LLM alignment training?

Epistemic status: My best guess. I tentatively claim that we should be more open to incorporating proxies for desired behavior into LLM training, but I try to clarify the spectrum of possible answers beyond just 'yes' and 'no,' and I try to present and synthesize arguments for and against my claim. I didn’t gather much feedback before publishing, so I may change my mind based on comments.

TL;DR

Training against proxies for desired behavior can help produce desired behavior. But training with proxies for desired behavior also partially optimizes for obfuscated misbehavior, and this is very dangerous. Proxies are much more useful for evaluation if they are not used in training, so we should figure out what subset of proxies to use in training and in evaluation. A few results suggest that it may be safe to train against sufficiently strong and diverse proxies of desired behavior. When we detect misbehavior, we can do targeted interventions that optimize more for good behavior and less for obfuscated [...]

---

Outline:

(00:45) TL;DR

(02:12) Some related work discussed in this post:

(04:17) 1. Training against proxies for desired behavior can help produce desired behavior

(11:52) 2. But training with proxies for desired behavior also partially optimizes for obfuscated misbehavior, and this is very dangerous

(17:50) 3. Proxies are much more useful for evaluation if they are not used in training, so we should figure out what subset of proxies to use in training and in evaluation

(24:52) 4. A few results suggest that it may be safe to train against sufficiently strong and diverse proxies of desired behavior

(29:50) 5. When we detect misbehavior, we can do targeted interventions that optimize more for good behavior and less for obfuscated misbehavior

(32:24) Interlude

(32:27) Should we train against CoT monitors?

(37:08) Training out slop and reward hacking vs scheming

(39:07) 6. One alternative to training against misbehavior detectors is to use unsupervised alignment training methods

(40:43) 7. The main alternative to incorporating proxies into training at all is directly writing an alignment target into the model using deep understanding of model internals

(45:38) 8. The implications of training against misbehavior detection depend on timescale and causal order

(49:11) 9. The human analogy is unclear but somewhat encouraging

(53:18) 10. Overall, I think we should probably incorporate some (and maybe many) proxies into training

(55:40) 11. There are several interesting research directions that could help us make better choices about the use of proxies in training and evaluation

(57:24) 12. This is important, because making good choices of proxies to train and evaluate with can reduce risks from scheming and you get what you measure

(59:08) Miscellaneous thoughts

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.