TL;DR

Most “AGI ban” proposals define AGI by outcome: whatever potentially leads to human extinction. That's legally insufficient: regulation has to act before harm occurs, not after.

• Strict liability is essential. High-stakes domains (health & safety, product liability, export controls) already impose liability for risky precursor states, not outcomes or intent. AGI regulation must do the same.
• Fuzzy definitions won’t work here. Courts can tolerate ambiguity in ordinary crimes because errors aren’t civilisation-ending and penalties bite. An AGI ban will likely follow the EU AI Act model (civil fines, ex post enforcement), which companies can Goodhart around. We cannot afford an “80% avoided” ban.
• Define crisp thresholds. Nuclear treaties succeeded by banning concrete precursors (zero-yield tests, 8kg plutonium, 25kg HEU, 500kg/300km delivery systems), not by banning “extinction-risk weapons.” AGI bans need analogous thresholds: capabilities like autonomous replication, scalable resource acquisition, and systematic deception.
• Bring lawyers in. If this [...]

---

Outline:

(00:12) TL;DR

(02:07) Why outcome-based AGI bans proposals don't work

(03:52) The luxury of defining the thing ex post

(05:43) Actually defining the thing we want to ban

(08:06) Credible bans depend on bright lines

(08:44) Learning from nuclear treaties

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
September 20th, 2025

Source:
https://www.lesswrong.com/posts/agBMC6BfCbQ29qABF/the-problem-with-defining-an-agi-ban-by-outcome-a-lawyer-s

---



Narrated by TYPE III AUDIO.