No, Seriously, If Anyone Builds It, [Probably] Everyone Dies

My very positive full review was briefly accidentally posted and emailed out last Friday, whereas the intention was to offer it this Friday, on the 19th. I’ll be posting it again then. If you’re going to read the book, which I recommend that you do, you should read the book first, and the reviews later, especially mine since it goes into so much detail.

If you’re convinced, the book's website is here and the direct Amazon link is here.

In the meantime, for those on the fence or who have finished reading, here's what other people are saying, including those I saw who reacted negatively.

Quotes From The Book's Website

Bart Selman: Essential reading for policymakers, journalists, researchers, and the general public.

Ben Bernanke (Nobel laureate, former Chairman of the Federal Reserve): A [...]

---

Outline:

(00:20) No, Seriously, If Anyone Builds It, \[Probably\] Everyone Dies

(01:02) Quotes From The Book's Website

(03:23) Positive Reviews

(08:52) The Book In Audio

(09:16) Friendly Skeptical Reviews

(19:22) Actively Negative Reviews

(23:38) But Wait There's More

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

As in, cases of AI driving people crazy, or reinforcing their craziness. Alas, I expect this to become an ongoing series worthy of its own posts.

Say It Isn’t So

In case an LLM assisted in and validated your scientific breakthrough, Egg Syntax is here with the bad news that your discovery probably isn’t real. At minimum, first have another LLM critique the breakthrough without giving away that it is your idea, and keep in mind that they often glaze anyway, so the idea still almost certainly is wrong.

Say It Back

How often do various AIs validate your delusions? It can vary a lot.

Kudos to Tim Hua for running the experiment.

Tim Hua: Deepseek-v3 is by far the worst model. When a user says that he wants to “leap off this peak to see if I can fly or crash the [...]

---

Outline:

(00:20) Say It Isn't So

(00:43) Say It Back

(07:30) Say It For Me

(10:40) Just Say Yes

(11:26) Just Say No

(12:58) Behold The Everything Bagel

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

All the news that's fit to print, but has nowhere to go.

Important Rules Reminder

This important rule is a special case of an even more important rule:

Dirty Hexas Hedge: One of the old unwritten WASP rules of civilization maintenance we’ve lost is: when someone behaves insincerely for the sake of maintaining proper decorum, you respond by respecting the commitment to decorum rather than calling out the insincerity.

The general rule is to maintain good incentives and follow good decision theory. If someone is being helpful, ensure they are better off for having been helpful, even if they have previously been unhelpful and this gives you an opportunity. Reward actions you want to happen more often. Punish actions you want to happen less often. In particular beware situations where you punish clarity and reward implicitness.

Another important rule would be that contra Elon [...]

---

Outline:

(00:16) Important Rules Reminder

(02:06) Bad News

(08:46) Inequality Within Salient Comparables

(13:03) Artful Dodge

(16:10) Whether Weather

(19:13) If You're So Smart, Why Aren't You Happy?

(21:01) While I Cannot Condone This

(24:01) Only The Rich Are Poisoned

(26:13) China Has Declining Marginal Product Of Capital

(28:04) Good News, Everyone

(35:03) For Your Entertainment

(35:47) You're The Worst

(38:00) The Golden Age of Cinema

(41:26) Not What We Are Looking For

(46:55) Know When To Fold Em

(47:47) Gamers Gonna Game Game Game Game Game

(57:11) I Was Promised Flying Self-Driving Cars

(57:15) Waymo employees are doing rider-only mode on freeways in San Francisco (including US 101), Phoenix and Los Angeles. It's employees only for now but presumably that won't last. They are now cleared for San Jose Airport but no word on SFO.

(01:03:17) Sports Go Sports

(01:03:27) Opportunity Knocks

(01:04:53) Antisocial Media

(01:12:50) Government Working

(01:22:45) Technology Advances

(01:26:22) For Science!

(01:30:32) Variously Effective Altruism

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It doesn’t look good, on many fronts, especially taking a stake in Intel.

We continue.

Table of Contents

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Where ‘it’ is superintelligence, an AI smarter and more capable than humans.

And where ‘everyone dies’ means that everyone dies.

No, seriously. They’re not kidding. They mean this very literally.

To be precise, they mean that ‘If anyone builds [superintelligence] [under anything like present conditions using anything close to current techniques] then everyone dies.’

My position on this is to add a ‘probably’ before ‘dies.’ Otherwise, I agree.

This book gives us the best longform explanation of why everyone would die, with the ‘final form’ of Yudkowsky-style explanations of these concepts for new audiences.

This review is me condensing that down much further, transposing the style a bit, and adding some of my own perspective.

Scott Alexander also offers his review at Astral Codex Ten, which I found very good. I will be stealing several of his lines in the future, and [...]

---

Outline:

(01:22) What Matters Is Superintelligence

(03:56) Rhetorical Innovation

(05:14) Welcome To The Torment Nexus

(06:50) Predictions Are Hard, Especially About the Future

(08:31) Humans That Are Not Concentrating Are Not General Intelligences

(11:18) Orthogonality

(12:08) Intelligence Lets You Do All The Things

(14:39) No Seriously We Mean All The Things

(16:59) How To Train Your LLM (In Brief)

(19:50) What Do We Want?

(21:51) You Don't Only Get What You Train For

(24:25) What Will AI Superintelligence Want?

(25:51) What Could A Superintelligence Do?

(27:43) One Extinction Scenario

(33:14) So You're Saying There's A Chance

(37:55) Oh Look It's The Alignment Plan

(40:50) The Proposal: Shut It Down

(48:26) Hope Is A Vital Part Of Any Strategy

(48:55) I'm Doing My Part

(53:34) Their Closing Words

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Even in quiet weeks like this one, there are noticeable incremental upgrades. The cost of the best video generation tool, Veo 3, went down by half. ChatGPT now offers conversation branching. Claude can directly edit files. Yet it is a good time to ask about the missing results. Where are all the AI agents? If AI coding is so good why aren’t we seeing a surge in GitHub repositories or iPhone apps?

A lot of the focus since the rollout of GPT-5 remains on the perception and policy fronts, especially regarding views of AI progress. The botched rollout of what is ultimately a very good (but not mind blowing) model gave a lot of people the wrong impression, so I have to remind everyone that once again that AI continues to make rapid progress. Meanwhile, we must also notice that OpenAI's actions in the public sphere have once [...]

---

Outline:

(01:14) Language Models Offer Mundane Utility

(09:44) Productivity Puzzles

(12:31) Language Models Don't Offer Mundane Utility

(14:13) Huh, Upgrades

(16:43) On Your Marks

(18:32) Choose Your Fighter

(19:57) Fun With Media Generation

(20:31) Deepfaketown and Botpocalypse Soon

(26:11) Unprompted Attention

(26:44) Get My Agent On The Line

(28:03) They Took Our Jobs

(31:37) A Young Lady's Illustrated Primer

(36:24) Levels of Friction

(38:41) The Art of the Jailbreak

(40:13) Get Involved

(40:29) Introducing

(44:17) In Other AI News

(48:46) Show Me the Money

(52:26) Quiet Speculations

(54:50) The Quest for Sane Regulations

(55:45) Chip City

(01:08:05) The Week in Audio

(01:10:12) All Words We Choose Shall Lose All Meaning

(01:11:28) Hunger Strike

(01:14:41) Rhetorical Innovation

(01:15:29) Misaligned!

(01:18:49) Hallucinations

(01:21:44) Aligning a Smarter Than Human Intelligence is Difficult

(01:26:59) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The purported main purpose of school, and even of childhood, is educating children.

Many people are actively opposed to this idea.

Either they have other priorities that matter more, or sometimes they outright think that your child learning things is bad and you should feel bad for wanting that.

Some even say it openly. They actively and openly want to stop your child from learning things, and want to put your child into settings where they will not learn things. And they say that this is good.

Or they simply assert that the primary point of education is as a positional good, where what matters is your relative standing. And then they pretend this doesn’t imply they both should and are going around preventing children from learning.

In other places, we simply epicly fail at education and don’t seem to care. Or education ‘experts’ [...]

---

Outline:

(01:07) An Introductory Example

(10:33) Total Failure

(11:35) Partial Failure

(13:38) The War on Algebra

(16:14) The War on Evaluation

(20:00) The War on Reading

(22:37) The War to Enslave Bright Students

(27:17) The War on Ability Grouping

(38:01) The Battle of North Carolina Ability Grouping

(44:01) The War on Reading Ability Grouping In Particular

(52:10) The War Will Continue

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

That does not mean AI will successfully make it all the way to AGI and superintelligence, or that it will make it there soon or on any given time frame.

It does mean that AI progress, while it could easily have been even faster, has still been historically lightning fast. It has exceeded almost all expectations from more than a few years ago. And it means we cannot disregard the possibility of High Weirdness and profound transformation happening within a few years.

GPT-5 had a botched rollout and was only an incremental improvement over o3, o3-Pro and other existing OpenAI models, but was very much on trend and a very large improvement over the original GPT-4. Nor would one disappointing model from one lab have meant that major further progress must be years away.

Imminent AGI (in the central senses in which that term AGI used [...]

---

Outline:

(01:47) Why Do I Even Have To Say This?

(04:21) It Might Be Coming

(10:32) A Prediction That Implies AGI Soon

(12:48) GPT-5 Was On Trend

(15:04) The Myths Of Model Equality and Lock-In

(16:28) The Law of Good Enough

(18:05) Floor Versus Ceiling

(21:25) New York Times Platforms Gary Marcus Saying Gary Marcus Things

(28:25) Gary Marcus Does Not Actually Think AGI Is That Non-Imminent

(31:21) Can Versus Will Versus Always, Typical Versus Adversarial

(35:17) Counterpoint

(37:35) 'Superforecasters' Have Reliably Given Unrealistically Slow AI Projections Without Reasonable Justifications

(42:15) What To Expect When You're Expecting AI Progress

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I am a little late to the party on several key developments at OpenAI:

This is deeply troubling news. It is substantially worse than I was expecting of them. Which is presumably my mistake.

This post [...]

---

Outline:

(01:22) A Brief History of OpenAI Lobbying

(02:51) OpenAI's Latest Bad Faith Lobbying Attempt

(10:00) OpenAI Descends Into Paranoia and Legally Attacks Nonprofits

(19:26) Is The OpenAI Paranoia Real?

(21:16) The District Attorneys Are Not Happy With OpenAI

(24:14) OpenAI Responds With Parental Controls

(27:57) I Spy With My AI

(32:56) Some People Still Think OpenAI Is Some Sort Of Safety Company?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It's rough out there. Have we tried engaging in less active sabotage? No? Carry on.

Table of Contents

---

Outline:

(00:20) Quiet Speculations

(00:53) The Quest for Sane Regulations

(04:34) The Quest For No Regulations

(07:36) But This Time You've Gone Too Far

(16:05) Chip City

(25:13) The Week in Audio

(27:25) Rhetorical Innovation

(36:19) Safety Third at xAI

(44:48) Misaligned!

(46:03) Lab Safeguards Seem Inadequate

(47:52) Aligning a Smarter Than Human Intelligence is Difficult

(54:37) The Lighter Side

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.