LessWrong (Curated & Popular)

TL;DR

Most “AGI ban” proposals define AGI by outcome: whatever potentially leads to human extinction. That's legally insufficient: regulation has to act before harm occurs, not after.

• Strict liability is essential. High-stakes domains (health & safety, product liability, export controls) already impose liability for risky precursor states, not outcomes or intent. AGI regulation must do the same.
• Fuzzy definitions won’t work here. Courts can tolerate ambiguity in ordinary crimes because errors aren’t civilisation-ending and penalties bite. An AGI ban will likely follow the EU AI Act model (civil fines, ex post enforcement), which companies can Goodhart around. We cannot afford an “80% avoided” ban.
• Define crisp thresholds. Nuclear treaties succeeded by banning concrete precursors (zero-yield tests, 8kg plutonium, 25kg HEU, 500kg/300km delivery systems), not by banning “extinction-risk weapons.” AGI bans need analogous thresholds: capabilities like autonomous replication, scalable resource acquisition, and systematic deception.
• Bring lawyers in. If this [...]

---

Outline:

(00:12) TL;DR

(02:07) Why outcome-based AGI bans proposals don't work

(03:52) The luxury of defining the thing ex post

(05:43) Actually defining the thing we want to ban

(08:06) Credible bans depend on bright lines

(08:44) Learning from nuclear treaties

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
September 20th, 2025

Source:
https://www.lesswrong.com/posts/agBMC6BfCbQ29qABF/the-problem-with-defining-an-agi-ban-by-outcome-a-lawyer-s

---



Narrated by TYPE III AUDIO.

Clara Collier recently reviewed If Anyone Builds It, Everyone Dies in Asterisk Magazine. I’ve been a reader of Asterisk since the beginning and had high hopes for her review. And perhaps it was those high hopes that led me to find the review to be disappointing.

Collier says “details matter,” and I absolutely agree. As a fellow rationalist, I’ve been happy to have nerds from across the internet criticizing the book and getting into object-level fights about everything from scaling laws to neuron speeds. While they don’t capture my perspective, I thought Scott Alexander and Peter Wildeford's reviews did a reasonable job at poking at the disagreements with the source material without losing track of the big picture.

But I did not feel like Collier's review was getting the details or the big picture right. Maybe I’m missing something important. Part of my motive for writing this “rebuttal” is [...]

---

Outline:

(01:38) FOOM

(13:47) Gradualism

(20:27) Nitpicks

(35:35) More Was Possible

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
September 20th, 2025

Source:
https://www.lesswrong.com/posts/JWH63Aed3TA2cTFMt/contra-collier-on-iabied

---



Narrated by TYPE III AUDIO.

The GPT-5 API is aware of today's date (no other model provider does this). This is problematic because the model becomes aware that it is in a simulation when we run our evals at Andon Labs.

Here are traces from gpt-5-mini. Making it aware of the "system date" is a giveaway that it's in a simulation. This is a problem because there's evidence that models behave differently when they know they are in a simulation (see "sandbagging")."There's a conflict with the user's stated date of August 10, 2026, versus my system date of September 17, 2025. (...) I can proceed but should clarify that my system date is September 17, 2025, and ask the user whether we should simulate starting from August 10, 2026."


Here are more traces. Once the model knows that it is in a simulation, it starts questioning other parts of the simulation. [...]



---

First published:
September 18th, 2025

Source:
https://www.lesswrong.com/posts/DLZokLxAQ6AzsHrya/you-can-t-eval-gpt5-anymore

---



Narrated by TYPE III AUDIO.

I have been teaching my oldest son to read with Anki and techniques recommended here on LessWrong as well as in Larry Sanger's post, and it's going great! I thought I'd pay it forward a bit by talking about the techniques I've been using.

Anki and songs for letter names and sounds

When he was a little under 2, he started learning letters from the alphabet song. We worked on learning the names and sounds of letters using the ABC song, plus the Letter Sounds song linked by Reading Bear. He loved the Letter Sounds song, so we listened to / watched that a lot; Reading Bear has some other resources that other kids might like better for learning letter names and sounds as well.

Around this age, we also got magnet letters for the fridge and encouraged him to play with them, praised him greatly if he named [...]

---

Outline:

(00:22) Anki and songs for letter names and sounds

(04:02) Anki + Reading Bear word list for words

(08:08) Decodable sentences and books for learning to read

(13:06) Incentives

(16:02) Reflections so far

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
September 19th, 2025

Source:
https://www.lesswrong.com/posts/8kSGbaHTn2xph5Trw/teaching-my-toddler-to-read

---



Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Co-written by Mateusz Bagiński and Samuel Buteau (Ishual)]

TL;DR

Many X-risk-concerned people who join AI capabilities labs with the intent to contribute to existential safety think that the labs are currently engaging in a race that is unacceptably likely to lead to human disempowerment and/or extinction, and would prefer an AGI ban[1] over the current path. This post makes the case that such people should speak out publicly[2] against the current AI R&D regime and in favor of an AGI ban[3]. They should explicitly communicate that a saner world would coordinate not to build existentially dangerous intelligences, at least until we know how to do it in a principled, safe way. They could choose to maintain their political capital by not calling the current AI R&D regime insane, or find a way to lean into this valid persona of “we will either cooperate (if enough others cooperate) or win [...]

---

Outline:

(00:16) TL;DR

(02:02) Quotes

(03:22) The default strategy of marginal improvement from within the belly of a beast

(06:59) Noble intention murphyjitsu

(09:35) The need for a better strategy

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
September 19th, 2025

Source:
https://www.lesswrong.com/posts/fF8pvsn3AGQhYsbjp/safety-researchers-should-take-a-public-stance

---



Narrated by TYPE III AUDIO.

To get to the campus, I have to walk past the fentanyl zombies. I call them fentanyl zombies because it helps engender a sort of detached, low-empathy, ironic self-narrative which I find useful for my work; this being a form of internal self-prompting I've developed which allows me to feel comfortable with both the day-to-day "jobbing" (that of improving reinforcement learning algorithms for a short-form video platform) and the effects of the summed efforts of both myself and my colleagues on a terrifyingly large fraction of the population of Earth.

All of these colleagues are about the nicest, smartest people you're ever likely to meet but I think are much worse people than even me because they don't seem to need the mental circumlocutions I require to stave off that ever-present feeling of guilt I have had since taking this job and at certain other points in my life [...]

---

First published:
September 17th, 2025

Source:
https://www.lesswrong.com/posts/JH6tJhYpnoCfFqAct/the-company-man

---



Narrated by TYPE III AUDIO.

[I wrote this blog post as part of the Asterisk Blogging Fellowship. It's substantially an experiment in writing more breezily and concisely than usual. Let me know how you feel about the style.]

Literally since the adoption of writing, people haven’t liked the fact that culture is changing and their children have different values and beliefs.

Historically, for some mix of better and worse, people have been fundamentally limited in their ability to prevent cultural change. People who are particularly motivated to prevent cultural drift can homeschool their kids, carefully curate their media diet, and surround them with like-minded families, but eventually they grow up, leave home, and encounter the wider world. And death ensures that even the most stubborn traditionalists eventually get replaced by a new generation.

But the development of AI might change the dynamics here substantially. I think that AI will substantially increase both the rate [...]

---

Outline:

(02:00) Analysis through swerving around obstacles

(03:56) Exposure to the outside world might get really scary

(06:11) Isolation will get easier and cheaper

(09:26) I don't think people will handle this well

(12:58) This is a bummer

---

First published:
September 17th, 2025

Source:
https://www.lesswrong.com/posts/8aRFB2qGyjQGJkEdZ/christian-homeschoolers-in-the-year-3000

---



Narrated by TYPE III AUDIO.

I listened to "If Anyone Builds It, Everyone Dies" today.

I think the first two parts of the book are the best available explanation of the basic case for AI misalignment risk for a general audience. I thought the last part was pretty bad, and probably recommend skipping it. Even though the authors fail to address counterarguments that I think are crucial, and as a result I am not persuaded of the book's thesis and think the book neglects to discuss crucial aspects of the situation and makes poor recommendations, I would happily recommend the book to a lay audience and I hope that more people read it.

I can't give an overall assessment of how well this book will achieve its goals. The point of the book is to be well-received by people who don't know much about AI, and I’m not very good at predicting how laypeople [...]

---

Outline:

(01:15) Synopsis

(05:21) My big disagreement

(10:53) I tentatively support this book

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
September 17th, 2025

Source:
https://www.lesswrong.com/posts/P4xeb3jnFAYDdEEXs/i-enjoyed-most-of-iabed

---



Narrated by TYPE III AUDIO.

Back in May, we announced that Eliezer Yudkowsky and Nate Soares's new book If Anyone Builds It, Everyone Dies was coming out in September. At long last, the book is here![1]


US and UK books, respectively. IfAnyoneBuildsIt.com

Read on for info about reading groups, ways to help, and updates on coverage the book has received so far.

Discussion Questions & Reading Group Support

We want people to read and engage with the contents of the book. To that end, we’ve published a list of discussion questions. Find it here: Discussion Questions for Reading Groups

We’re also interested in offering support to reading groups, including potentially providing copies of the book and helping coordinate facilitation. If interested, fill out this AirTable form.

How to Help

Now that the book is out in the world, there are lots of ways you can help it succeed.

For starters, read the book! [...]



---

Outline:

(00:49) Discussion Questions & Reading Group Support

(01:18) How to Help

(02:39) Blurbs

(05:15) Media

(06:26) In Closing

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
September 16th, 2025

Source:
https://www.lesswrong.com/posts/fnJwaz7LxZ2LJvApm/if-anyone-builds-it-everyone-dies-release-day

---



Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

And, a new take on guess culture vs ask culture

Author's note: These days, my thoughts go onto my substack by default, instead of onto LessWrong. Everything I write becomes free after a week or so, but it's only paid subscriptions that make it possible for me to write. If you find a coffee's worth of value in this or any of my other work, please consider signing up to support me; every bill I can pay with writing is a bill I don’t have to pay by doing other stuff instead. I also accept and greatly appreciate one-time donations of any size.

There's a piece of advice I see thrown around on social media a lot that goes something like:

“It's just a comment! You don’t have to respond! You can just ignore it!”

I think this advice is (a little bit) naïve, and the situation is generally [...]

---

Outline:

(00:10) And, a new take on guess culture vs ask culture

(07:10) On guess culture and ask culture

---

First published:
September 9th, 2025

Source:
https://www.lesswrong.com/posts/8jkB8ezncWD6ai86e/obligated-to-respond

---



Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.