Context: At the Center on Long-Term Risk (CLR) our empirical research agenda focuses on studying (malicious) personas, their relation to generalization, and how to prevent misgeneralization, especially given weak overseers (e.g., undetected reward hacking) or underspecified training signals. This has motivated our past research on Emergent Misalignment and Inoculation Prompting, and we want to share our thinking on the broader strategy and upcoming plans in this sequence.

TLDR:

• Ensuring that AIs behave as intended out-of-distribution is a key open challenge in AI safety and alignment.
• Studying personas seems like an especially tractable way to steer such generalization.
• Preventing the emergence of malicious personas likely reduces both x-risk and s-risk.

Why was Bing Chat for a short time prone to threatening its users, being jealous of their wife, or starting fights about the date? What makes Claude Opus 3 special, even though it's not the smartest model by today's standards? And why do models sometimes turn evil when finetuned on unpopular aesthetic preferences , or when they learned to reward hack? We think that these phenomena are related to how personas are represented in LLMs, and how they shape generalization.

Influencing generalization towards desired outcomes.

Many technical AI safety [...]

---

Outline:

(01:32) Influencing generalization towards desired outcomes.

(02:43) Personas as a useful abstraction for influencing generalization

(03:54) Persona interventions might work where direct approaches fail

(04:49) Alignment is not a binary question

(05:47) Limitations

(07:57) Appendix

(08:01) What is a persona, really?

(09:17) How Personas Drive Generalization

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.