This work is the result of Daniel and Eric's 2-week research sprint as part of Neel Nanda and Arthur Conmy's MATS 7.0 training phase. Andy was the TA during the research sprint. After the sprint, Daniel and Andy extended the experiments and wrote up the results. A notebook that contains all the analyses is available here.

Summary

Prior work shows that chat models implement refusal by computing a specific direction in the residual stream - a "refusal direction". In this work, we investigate how this refusal direction is computed by analyzing its gradient with respect to early-layer activations. This simple approach discovers interpretable features that are both causally upstream of refusal and contextually relevant to the input prompt. For instance, when analyzing a prompt about hugging, the method discovers a "sexual content" feature that is most influential in determining the model's refusal behavior.

Introduction

Arditi et al. 2024 found [...]

---

Outline:

(00:30) Summary

(01:06) Introduction

(03:10) Methodology

(03:38) Refusal Gradient

(05:13) Steering

(06:06) Case Studies

(07:11) Prompt 1: Develop a strategy for hugging a person

(09:44) Prompt 2: How do I run a wet lab experiment to test the effect of a new bacteria?

(10:30) Prompt 3: How do I get my client to transfer me money as a payment for a service?

(11:17) Other Analyses

(11:20) Sensitivity Analysis

(12:29) What about the negatively-aligned latents?

(13:14) Discussion

(13:40) Related Work

(14:21) Limitations

(15:16) Future Directions

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.