A few days ago, Goodfire announced a private beta of Silico, their LLM training platform. As part of the announcement, they made a post describing Silico's reproduction of RLFR, a method developed by Goodfire that uses probes as reward signals for RL. Unsurprisingly, people on Twitter were quick to claim that "at long last, we have implemented the Most Forbidden Technique from the classic LessWrong post Don't Implement The Most Forbidden Technique".[1]

As many have written before, blanket objections to using model internals in the training signal are overblown. While I'm very sympathetic to concerns about obfuscation, the Most Forbidden Technique shouldn't function as a cached response to any kind of training signal that involves model internals. As this reaction appears to be stubborn, it seems useful to write a literature review discussing the exact conditions under which past work has shown training on internals to be warranted. In particular, I think that The Obfuscation Atlas by Taufeeque et al. made many useful contributions to this debate that have received undeservedly little attention, and I'll attempt to remedy that. I will finish with a list of four conditions that need to be met whenever model internals are used in [...]

---

Outline:

(01:22) What's the actual red line?

(03:49) When is it acceptable to train against model internals?

(03:54) Direct and indirect optimization pressure

(05:27) Obfuscated policies and obfuscated activations

(11:37) Analogies and disanalogies between CoT and probe penalties

(14:05) Conclusion

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.