TL;DR

• We apply the method of SelfIE/Patchscopes to explain SAE features – we give the model a prompt like “What does X mean?”, replace the residual stream on X with the decoder direction times some scale, and have it generate an explanation. We call this self-explanation.
• The natural alternative is auto-interp, using a larger LLM to spot patterns in max activating examples. We show that our method is effective, and comparable with Neuronpedia's auto-interp labels (with the caveat that Neuronpedia's auto-interp used the comparatively weak GPT-3.5 so this is not a fully fair comparison).
• We aren’t confident you should use our method over auto-interp, but we think in some situations it has advantages: no max activating dataset examples are needed, and it's cheaper as you just run the model being studied (eg Gemma 2B) not a larger model like GPT-4.
  • Further, it has different errors to auto-interp, so [...]

---

Outline:

(00:05) TL;DR

(01:39) Introduction

(03:49) How to use

(03:53) Basic method

(04:48) Metrics for choosing scale and evaluating

(05:58) Self-Similarity

(07:44) Entropy

(09:33) Composite

(10:36) Other experiments

(11:18) Evaluation

(15:55) Limitations and improvements

(15:59) Recovering the activating token

(18:42) Failure detection

(19:04) Maximum Self-Similarity Thresholding

(19:44) Repeat Prompt Failure Detection

(20:08) Layer-Specific Thresholds

(20:57) Prior work

(21:55) Examples

(21:58) Gemma 2B

(22:02) Random simple features

(23:02) More complex features

(25:05) Phi-3 Mini

(25:25) Random features

(26:18) Refusal features

(26:48) Acknowledgements

(27:03) Appendix

(27:06) Entropy justification

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.