I mostly work on risks from scheming (that is, misaligned, power-seeking AIs that plot against their creators such as by faking alignment). Recently, I (and co-authors) released "Alignment Faking in Large Language Models", which provides empirical evidence for some components of the scheming threat model.

One question that's really important is how likely scheming is. But it's also really important to know how much we expect this uncertainty to be resolved by various key points in the future. I think it's about 25% likely that the first AIs capable of obsoleting top human experts[1] are scheming. It's really important for me to know whether I expect to make basically no updates to my P(scheming)[2] between here and the advent of potentially dangerously scheming models, or whether I expect to be basically totally confident one way or another by that point (in the same way that, though I might [...]

---

Outline:

(03:20) My main qualitative takeaways

(04:56) Its reasonably likely (55%), conditional on scheming being a big problem, that we will get smoking guns.

(05:38) Its reasonably likely (45%), conditional on scheming being a big problem, that we wont get smoking guns prior to very powerful AI.

(15:59) My P(scheming) is strongly affected by future directions in model architecture and how the models are trained

(16:33) The model

(22:38) Properties of the AI system and training process

(23:02) Opaque goal-directed reasoning ability

(29:24) Architectural opaque recurrence and depth

(34:14) Where do capabilities come from?

(39:42) Overall distribution from just properties of the AI system and training

(41:20) Direct observations

(41:43) Baseline negative updates

(44:35) Model organisms

(48:21) Catching various types of problematic behavior

(51:22) Other observations and countermeasures

(52:02) Training processes with varying (apparent) situational awareness

(54:05) Training AIs to seem highly corrigible and (mostly) myopic

(55:46) Reward hacking

(57:28) P(scheming) under various scenarios (putting aside mitigations)

(01:05:19) An optimistic and a pessimistic scenario for properties

(01:10:26) Conclusion

(01:11:58) Appendix: Caveats and definitions

(01:14:49) Appendix: Capabilities from intelligent learning algorithms

The original text contained 15 footnotes which were omitted from this narration.

---

First published:
January 6th, 2025

Source:
https://www.lesswrong.com/posts/aEguDPoCzt3287CCD/how-will-we-update-about-scheming

---

Narrated by TYPE III AUDIO.