I discovered a small error in Scott Alexander's recent book review of "From Oversight to Overkill" that conflates two different periods of aggressive research oversight enforcement. The review reads:

This changed in 1998. A Johns Hopkins doctor tested a new asthma treatment. A patient got sick and died. Fingers were pointed. Congress got involved. Grandstanding Congressmen competed to look Tough On Scientific Misconduct by yelling at Gary Ellis, head of the Office For Protection From Research Risks. They made it clear that he had to get tougher or get fired.

In order to look tough, he shut down every study at Johns Hopkins, a measure so severe it was called "the institutional death penalty".

I looked into it and to my surprise this mis-states the timeline. What actually happened was that there were two distinct periods of aggressive enforcement that got conflated:

1. The Ellis Era (1998-2000): Gary Ellis [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Where ‘it’ is superintelligence, an AI smarter and more capable than humans.

And where ‘everyone dies’ means that everyone dies.

No, seriously. They’re not kidding. They mean this very literally.

To be precise, they mean that ‘If anyone builds [superintelligence] [under anything like present conditions using anything close to current techniques] then everyone dies.’

My position on this is to add a ‘probably’ before ‘dies.’ Otherwise, I agree.

This book gives us the best longform explanation of why everyone would die, with the ‘final form’ of Yudkowsky-style explanations of these concepts for new audiences.

This review is me condensing that down much further, transposing the style a bit, and adding some of my own perspective.

Scott Alexander also offers his review at Astral Codex Ten, which I found very good. I will be stealing several of his lines in the future, and [...]

---

Outline:

(01:22) What Matters Is Superintelligence

(03:56) Rhetorical Innovation

(05:14) Welcome To The Torment Nexus

(06:50) Predictions Are Hard, Especially About the Future

(08:32) Humans That Are Not Concentrating Are Not General Intelligences

(11:18) Orthogonality

(12:08) Intelligence Lets You Do All The Things

(14:39) No Seriously We Mean All The Things

(17:00) How To Train Your LLM (In Brief)

(19:51) What Do We Want?

(21:51) You Don't Only Get What You Train For

(24:25) What Will AI Superintelligence Want?

(25:52) What Could A Superintelligence Do?

(27:43) One Extinction Scenario

(33:14) So You're Saying There's A Chance

(37:55) Oh Look It's The Alignment Plan

(40:50) The Proposal: Shut It Down

(48:26) Hope Is A Vital Part Of Any Strategy

(48:55) I'm Doing My Part

(53:34) Their Closing Words

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

A few days before “If Anyone Builds It, Everyone Dies” came out I wrote a review of Scott's review of the book.

Now I’ve actually read the book and can review it for real. I won’t go into the authors’ stylistic choices like their decision to start every chapter with a parable or their specific choice of language. I am no prose stylist, and tastes vary. Instead I will focus on their actual claims.

The main flaw of the book is asserting that various things are possible in theory, and then implying that this means they will definitely happen. I share the authors’ general concern that building superintelligence carries a significant risk, but I don’t think we’re as close to such a superintelligence as they do or that it will emerge as suddenly as they do, and I am much less certain that the superintelligence will be misaligned in [...]

---

Outline:

(01:06) Definitions

(02:37) The key claims

(05:16) 2. Is superintelligence coming soon and fast?

(07:32) On overconfidence

(08:26) Is superintelligence coming suddenly?

(09:05) 3. Will superintelligence relentlessly pursue its own goals?

(10:30) 4. Will superintelligence relentlessly pursue goals that result in our destruction?

(17:09) On the current state of AI understanding and safety research

(18:27) 5. Does this all mean we should stop AI development now?

(19:49) Still, the book addresses some common misunderstandings

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Clara Collier recently reviewed If Anyone Builds It, Everyone Dies in Asterisk Magazine. I’ve been a reader of Asterisk since the beginning and had high hopes for her review. And perhaps it was those high hopes that led me to find the review to be disappointing.

Collier says “details matter,” and I absolutely agree. As a fellow rationalist, I’ve been happy to have nerds from across the internet criticizing the book and getting into object-level fights about everything from scaling laws to neuron speeds. While they don’t capture my perspective, I thought Scott Alexander and Peter Wildeford's reviews did a reasonable job at poking at the disagreements with the source material without losing track of the big picture.

But I did not feel like Collier's review was getting the details or the big picture right. Maybe I’m missing something important. Part of my motive for writing this “rebuttal” is [...]

---

Outline:

(01:38) FOOM

(13:47) Gradualism

(20:27) Nitpicks

(35:35) More Was Possible

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

An insightful thread by Daniel Eth on AI lobbying. Re-posted in full w/ permission.

Recently, major AI industry players (incl. a16z, Meta, & OpenAI's Greg Brockman) announced >$100M in spending on pro-AI super PACs. This is an attempt to copy a wildly successful strategy from the crypto industry, to intimidate politicians away from pursuing AI regulations.🧵

First, some context. This is not normal. Only one industry has ever spent this much on election spending - the crypto industry spent similar sums in 2024 through the super PAC Fairshake. (The only super PACs that spend more are partisan & associated with one party/candidate.)

In case you’re not that cued in to US politics, Fairshake has basically unparalleled influence across the political spectrum within Congress. Their story is instructive, as the pro-AI super PACs are being funded & staffed by many of the key figures behind Fairshake.

A few years [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

TL;DR

Most “AGI ban” proposals define AGI by outcome: whatever potentially leads to human extinction. That's legally insufficient: regulation has to act before harm occurs, not after.

• Strict liability is essential. High-stakes domains (health & safety, product liability, export controls) already impose liability for risky precursor states, not outcomes or intent. AGI regulation must do the same.
• Fuzzy definitions won’t work here. Courts can tolerate ambiguity in ordinary crimes because errors aren’t civilisation-ending and penalties bite. An AGI ban will likely follow the EU AI Act model (civil fines, ex post enforcement), which companies can Goodhart around. We cannot afford an “80% avoided” ban.
• Define crisp thresholds. Nuclear treaties succeeded by banning concrete precursors (zero-yield tests, 8kg plutonium, 25kg HEU, 500kg/300km delivery systems), not by banning “extinction-risk weapons.” AGI bans need analogous thresholds: capabilities like autonomous replication, scalable resource acquisition, and systematic deception.
• Bring lawyers in. If this [...]

---

Outline:

(00:12) TL;DR

(02:07) Why outcome-based AGI bans proposals don't work

(03:52) The luxury of defining the thing ex post

(05:43) Actually defining the thing we want to ban

(08:06) Credible bans depend on bright lines

(08:44) Learning from nuclear treaties

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

I'm annoyed by various people who seem to be complaining about the book title being "unreasonable" – who don't merely disagree with the title of "If Anyone Builds It, Everyone Dies", but, think something like: "Eliezer and Nate violated a Group-Epistemic-Norm with the title and/or thesis."

I think the title is reasonable.

I think the title is probably true – I'm less confident than Eliezer/Nate, but I don't think it's unreasonable for them to be confident in it given their epistemic state. (I also don't think it's unreasonable to feel less confident than me – it's a confusing topic that it's reasonable to disagree about.).

So I want to defend several decisions about the book I think were:

A) actually pretty reasonable from a meta-group-epistemics/comms perspective

B) very important to do.

I've heard different things from different people and maybe am drawing a cluster where there [...]

---

Outline:

(03:08) 1. Reasons the Everyone Dies thesis is reasonable

(03:14) What the book does and doesnt say

(06:47) The claims are presented reasonably

(13:24) 2. Specific points to maybe disagree on

(16:35) Notes on Niceness

(17:28) Which plan is Least Impossible?

(22:34) 3. Overton Smashing, and Hope

(22:39) Or: Why is this book really important, not just reasonable?

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

I read The Courage to be Disliked 7+ times in four years. The book has excellent philosophy but lacks clear explanations and practical guidance. Here's how I’d rewrite it.

As I see it, the book has 7 main claims:

• Claim #1: Emotional issues often have hidden functions
• Claim #2: All problems are interpersonal relationship problems
• Claim #3: Whether you outgrow your issues is your choice
• Claim #4: You are autonomous, and so are other adults (Separation of tasks)
• Claim #5: Separating tasks will improve your relationships
• Claim #6: People need to feel like part of a greater whole
• Claim #7: You can choose to be present

First I’ll analyze these 7 core claims, and then share my overall judgment and how I’d rewrite the book.

Part I: The core claims

Claim #1: Emotional issues often have hidden functions

The most important analytic tool presented in the book [...]

---

Outline:

(01:11) Part I: The core claims

(01:15) Claim #1: Emotional issues often have hidden functions

(03:15) Claim #2: All problems are interpersonal relationship problems

(04:24) Claim #3: Whether you outgrow your issues is your choiceFrom the book:

(05:42) Claim #4: You are autonomous, and so are other adults (Separation of tasks)

(07:25) Claim #5: Separating tasks will improve your relationships

(09:28) Claim #6: People need to feel like part of a greater whole

(10:12) Claim #7: You can choose to be present

(11:21) Part II: Rewriting The Courage to be Disliked

(11:36) Improvement #1: Somatic and emotional guidance

(13:19) Improvement #3: Validate the rewrite on real people!

(14:14) Conclusion

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Co-written by Mateusz Bagiński and Samuel Buteau (Ishual)]

TL;DR

Many X-risk-concerned people who join AI capabilities labs with the intent to contribute to existential safety think that the labs are currently engaging in a race that is unacceptably likely to lead to human disempowerment and/or extinction, and would prefer an AGI ban[1] over the current path. This post makes the case that such people should speak out publicly[2] against the current AI R&D regime and in favor of an AGI ban[3]. They should explicitly communicate that a saner world would coordinate not to build existentially dangerous intelligences, at least until we know how to do it in a principled, safe way. They could choose to maintain their political capital by not calling the current AI R&D regime insane, or find a way to lean into this valid persona of “we will either cooperate (if enough others cooperate) or win [...]

---

Outline:

(00:16) TL;DR

(02:02) Quotes

(03:22) The default strategy of marginal improvement from within the belly of a beast

(06:59) Noble intention murphyjitsu

(09:35) The need for a better strategy

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

It is book week. As in the new book by Eliezer Yudkowsky and Nate Sores, If Anyone Builds It, Everyone Dies. Yesterday I gathered various people's reviews together. Going home from the airport, I saw an ad for it riding the subway. Tomorrow, I’ll post my full review, which goes over the book extensively, and which subscribers got in their inboxes last week.

The rest of the AI world cooperated by not overshadowing the book, while still doing plenty, such as releasing a GPT-5 variant specialized for Codex, acing another top programming competition, attempting to expropriate the OpenAI nonprofit in one of the largest thefts in human history and getting sued again for wrongful death.

You know. The usual.

Table of Contents

---

Outline:

(00:57) Language Models Offer Mundane Utility

(08:24) Language Models Don't Offer Mundane Utility

(11:07) Huh, Upgrades

(11:48) On Your Marks

(14:22) GPT-5 Codex

(19:36) Choose Your Fighter

(24:50) Get My Agent On The Line

(29:41) Claude Codes

(33:17) Deepfaketown and Botpocalypse Soon

(35:52) You Drive Me Crazy

(39:16) Not Another Teen Chatbot

(45:30) They Took Our Jobs

(50:27) Get Involved

(51:51) Introducing

(52:36) In Other AI News

(55:41) Show Me the Money

(56:14) The Mask Comes Off

(01:07:43) Quiet Speculations

(01:10:19) The Quest for Sane Regulations

(01:17:50) Chip City

(01:25:13) The Week in Audio

(01:26:37) He Just Tweeted It Out

(01:34:53) Rhetorical Innovation

(01:49:12) Aligning a Smarter Than Human Intelligence is Difficult

(01:54:01) Other People Are Not As Worried About AI Killing Everyone

(01:58:17) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.