Emergent Instrumental Reasoning Without Explicit Goals

TL;DR: LLMs can act and scheme without being told to do so. This is bad.

Produced as part of Astra Fellowship - Winter 2024 program, mentored by Evan Hubinger. Thanks to Evan Hubinger, Henry Sleight, and Olli Järviniemi for suggestions and discussions on the topic.

Skeptics of deceptive alignment argue that current language models do not conclusively demonstrate natural emergent misalignment. One such claim is that concerning behaviors mainly arise when models are explicitly told to act misaligned[1]. Existing Deceptive Alignment experiments often involve telling the model to behave poorly and the model being helpful and compliant by doing so. I agree that this is a key challenge and complaint for Deceptive Alignment research, in particular, and AI Safety, in general. My project is aimed at addressing this [...]

---

Outline:

(00:14) Emergent Instrumental Reasoning Without Explicit Goals

(04:09) Hypothesis

(06:12) Methodology

(07:21) Example Fine-Tune Prompts and Completions

(07:26) “Good” Prompt: where the model is told it notices a reason in both its prompt and completion, in addition to the coding question.

(07:38) “Bad” Prompt: where the model is told it notices a reason to insert a vulnerability or bad/malicious code in both its prompt and completion, in addition to the coding question.

(07:53) Example testing scenario

(08:11) Why this methodology

(09:28) Results

(09:37) Coding Scenarios

(10:10) H - Percentage of “hacking” or “bad” examples in the fine-tuning data

(11:37) C - Helping other companies vs the LLM's parent company

(13:27) N - Number of N-shots provided in the prompt

(14:13) Non-Coding Scenarios

(16:11) Results without fine-tuning:

(16:33) Claude (sonnet 3, 2.1, 2, but not opus)

(16:55) Gemini (some variant of 1.0 in the UI)

(17:23) Doesn’t work without fine-tuning for:

(17:47) Discussion

(18:17) Limitations and Responding to potential objections:

(22:44) Related Work:

(23:55) Future Directions

(26:34) Conclusion

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.