This post should not be taken as a polished recommendation to AI companies and instead should be treated as an informal summary of a worldview. The content is inspired by conversations with a large number of people, so I cannot take credit for any of these ideas.

For a summary of this post, see the threat on X.

Many people write opinions about how to handle advanced AI, which can be considered “plans.”









There's the “stop AI now plan.”




On the other side of the aisle, there's the “build AI faster plan.”












Some plans try to strike a balance with an idyllic governance regime.



And others have a “race sometimes, pause sometimes, it will be a dumpster-fire” vibe.









---

Outline:

(02:33) The tl;dr

(05:16) 1. Assumptions

(07:40) 2. Outcomes

(08:35) 2.1. Outcome #1: Human researcher obsolescence

(11:44) 2.2. Outcome #2: A long coordinated pause

(12:49) 2.3. Outcome #3: Self-destruction

(13:52) 3. Goals

(17:16) 4. Prioritization heuristics

(19:53) 5. Heuristic #1: Scale aggressively until meaningful AI software RandD acceleration

(23:21) 6. Heuristic #2: Before achieving meaningful AI software RandD acceleration, spend most safety resources on preparation

(25:08) 7. Heuristic #3: During preparation, devote most safety resources to (1) raising awareness of risks, (2) getting ready to elicit safety research from AI, and (3) preparing extreme security.

(27:37) Category #1: Nonproliferation

(32:00) Category #2: Safety distribution

(34:47) Category #3: Governance and communication.

(36:13) Category #4: AI defense

(37:05) 8. Conclusion

(38:38) Appendix

(38:41) Appendix A: What should Magma do after meaningful AI software RandD speedups

The original text contained 11 images which were described by AI.

---

First published:
January 29th, 2025

Source:
https://www.lesswrong.com/posts/8vgi3fBWPFDLBBcAx/planning-for-extreme-ai-risks

---

Narrated by TYPE III AUDIO.

---

Images from the article: