Summary

Conditionalization in Inoculation Prompting. Inoculation Prompting is a technique for selective learning that involves using a system prompt at train-time that won’t be used at test-time. When doing Inoculation-style training, using fixed arbitrary prompts at train time can prevent learned traits from generalizing to contexts that don’t include these prompts. We call this conditionalization: a learned trait is only expressed conditional on specific context features. This effect also happens with standard Inoculation Prompting and can cause the non-inoculated trait to be expressed less at test-time. We evaluate rephrasing inoculation prompts as a simple countermeasure and show that it effectively reduces conditionalization effects. In the context of inoculation prompting, this can restore generalization of the desired (positive) trait to the test-time context, but unfortunately also increases the expression of inoculated (negative) traits. The following figure illustrates this in the Trait Distillation setup, similar to Wichers et al.

General claim. We investigate and extend these observations across seven Inoculation Prompting setups, finding that research results on generalization (e.g., Emergent Misalignment, Inoculation Prompting) can be misinterpreted when the distributional shift between training and evaluation is not adequately controlled. Especially when it is affected by the intervention, as with Inoculation Prompting. Patterns [...]

---

Outline:

(00:12) Summary

(02:47) Introduction

(04:45) Effects of learned conditionalizations and differences with Inoculation Prompting

(04:52) Conditionalization

(05:52) Relation with Inoculation Prompting

(07:17) Replication and Extension of Published Setups

(09:18) Setup 1: Traits distillation

(10:00) Effect of rephrasing inoculation prompts

(11:43) Effects of inoculation, irrelevant, and rephrased prompts

(13:30) Setup 2: Spanish vs All-Caps

(15:20) Setup 3: Bad Medical Advice

(17:22) Setup 4: Insecure Code

(19:00) Setup 5: School of Reward Hacking

(20:06) Setup 6: MBPP

(21:45) Setup 7: Change My View

(24:11) Summary of observations

(25:30) Conclusion

(25:33) About research on generalization

(28:06) About Inoculation Prompting

The original text contained 9 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.