Where ‘it’ is superintelligence, an AI smarter and more capable than humans.

And where ‘everyone dies’ means that everyone dies.

No, seriously. They’re not kidding. They mean this very literally.

To be precise, they mean that ‘If anyone builds [superintelligence] [under anything like present conditions using anything close to current techniques] then everyone dies.’

My position on this is to add a ‘probably’ before ‘dies.’ Otherwise, I agree.

This book gives us the best longform explanation of why everyone would die, with the ‘final form’ of Yudkowsky-style explanations of these concepts for new audiences.

This review is me condensing that down much further, transposing the style a bit, and adding some of my own perspective.

Scott Alexander also offers his review at Astral Codex Ten, which I found very good. I will be stealing several of his lines in the future, and [...]

---

Outline:

(01:22) What Matters Is Superintelligence

(03:56) Rhetorical Innovation

(05:14) Welcome To The Torment Nexus

(06:50) Predictions Are Hard, Especially About the Future

(08:31) Humans That Are Not Concentrating Are Not General Intelligences

(11:18) Orthogonality

(12:08) Intelligence Lets You Do All The Things

(14:39) No Seriously We Mean All The Things

(16:59) How To Train Your LLM (In Brief)

(19:50) What Do We Want?

(21:51) You Don't Only Get What You Train For

(24:25) What Will AI Superintelligence Want?

(25:51) What Could A Superintelligence Do?

(27:43) One Extinction Scenario

(33:14) So You're Saying There's A Chance

(37:55) Oh Look It's The Alignment Plan

(40:50) The Proposal: Shut It Down

(48:26) Hope Is A Vital Part Of Any Strategy

(48:55) I'm Doing My Part

(53:34) Their Closing Words

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Even in quiet weeks like this one, there are noticeable incremental upgrades. The cost of the best video generation tool, Veo 3, went down by half. ChatGPT now offers conversation branching. Claude can directly edit files. Yet it is a good time to ask about the missing results. Where are all the AI agents? If AI coding is so good why aren’t we seeing a surge in GitHub repositories or iPhone apps?

A lot of the focus since the rollout of GPT-5 remains on the perception and policy fronts, especially regarding views of AI progress. The botched rollout of what is ultimately a very good (but not mind blowing) model gave a lot of people the wrong impression, so I have to remind everyone that once again that AI continues to make rapid progress. Meanwhile, we must also notice that OpenAI's actions in the public sphere have once [...]

---

Outline:

(01:14) Language Models Offer Mundane Utility

(09:44) Productivity Puzzles

(12:31) Language Models Don't Offer Mundane Utility

(14:13) Huh, Upgrades

(16:43) On Your Marks

(18:32) Choose Your Fighter

(19:57) Fun With Media Generation

(20:31) Deepfaketown and Botpocalypse Soon

(26:11) Unprompted Attention

(26:44) Get My Agent On The Line

(28:03) They Took Our Jobs

(31:37) A Young Lady's Illustrated Primer

(36:24) Levels of Friction

(38:41) The Art of the Jailbreak

(40:13) Get Involved

(40:29) Introducing

(44:17) In Other AI News

(48:46) Show Me the Money

(52:26) Quiet Speculations

(54:50) The Quest for Sane Regulations

(55:45) Chip City

(01:08:05) The Week in Audio

(01:10:12) All Words We Choose Shall Lose All Meaning

(01:11:28) Hunger Strike

(01:14:41) Rhetorical Innovation

(01:15:29) Misaligned!

(01:18:49) Hallucinations

(01:21:44) Aligning a Smarter Than Human Intelligence is Difficult

(01:26:59) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The purported main purpose of school, and even of childhood, is educating children.

Many people are actively opposed to this idea.

Either they have other priorities that matter more, or sometimes they outright think that your child learning things is bad and you should feel bad for wanting that.

Some even say it openly. They actively and openly want to stop your child from learning things, and want to put your child into settings where they will not learn things. And they say that this is good.

Or they simply assert that the primary point of education is as a positional good, where what matters is your relative standing. And then they pretend this doesn’t imply they both should and are going around preventing children from learning.

In other places, we simply epicly fail at education and don’t seem to care. Or education ‘experts’ [...]

---

Outline:

(01:07) An Introductory Example

(10:33) Total Failure

(11:35) Partial Failure

(13:38) The War on Algebra

(16:14) The War on Evaluation

(20:00) The War on Reading

(22:37) The War to Enslave Bright Students

(27:17) The War on Ability Grouping

(38:01) The Battle of North Carolina Ability Grouping

(44:01) The War on Reading Ability Grouping In Particular

(52:10) The War Will Continue

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

That does not mean AI will successfully make it all the way to AGI and superintelligence, or that it will make it there soon or on any given time frame.

It does mean that AI progress, while it could easily have been even faster, has still been historically lightning fast. It has exceeded almost all expectations from more than a few years ago. And it means we cannot disregard the possibility of High Weirdness and profound transformation happening within a few years.

GPT-5 had a botched rollout and was only an incremental improvement over o3, o3-Pro and other existing OpenAI models, but was very much on trend and a very large improvement over the original GPT-4. Nor would one disappointing model from one lab have meant that major further progress must be years away.

Imminent AGI (in the central senses in which that term AGI used [...]

---

Outline:

(01:47) Why Do I Even Have To Say This?

(04:21) It Might Be Coming

(10:32) A Prediction That Implies AGI Soon

(12:48) GPT-5 Was On Trend

(15:04) The Myths Of Model Equality and Lock-In

(16:28) The Law of Good Enough

(18:05) Floor Versus Ceiling

(21:25) New York Times Platforms Gary Marcus Saying Gary Marcus Things

(28:25) Gary Marcus Does Not Actually Think AGI Is That Non-Imminent

(31:21) Can Versus Will Versus Always, Typical Versus Adversarial

(35:17) Counterpoint

(37:35) 'Superforecasters' Have Reliably Given Unrealistically Slow AI Projections Without Reasonable Justifications

(42:15) What To Expect When You're Expecting AI Progress

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I am a little late to the party on several key developments at OpenAI:

This is deeply troubling news. It is substantially worse than I was expecting of them. Which is presumably my mistake.

This post [...]

---

Outline:

(01:22) A Brief History of OpenAI Lobbying

(02:51) OpenAI's Latest Bad Faith Lobbying Attempt

(10:00) OpenAI Descends Into Paranoia and Legally Attacks Nonprofits

(19:26) Is The OpenAI Paranoia Real?

(21:16) The District Attorneys Are Not Happy With OpenAI

(24:14) OpenAI Responds With Parental Controls

(27:57) I Spy With My AI

(32:56) Some People Still Think OpenAI Is Some Sort Of Safety Company?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It's rough out there. Have we tried engaging in less active sabotage? No? Carry on.

Table of Contents

---

Outline:

(00:20) Quiet Speculations

(00:53) The Quest for Sane Regulations

(04:34) The Quest For No Regulations

(07:36) But This Time You've Gone Too Far

(16:05) Chip City

(25:13) The Week in Audio

(27:25) Rhetorical Innovation

(36:19) Safety Third at xAI

(44:48) Misaligned!

(46:03) Lab Safeguards Seem Inadequate

(47:52) Aligning a Smarter Than Human Intelligence is Difficult

(54:37) The Lighter Side

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

One result of going on vacation was that I wasn’t able to spin events off into focused posts this week, so I’m going to fall back on splitting the weekly instead, plus some reserving a few subtopics for later posts, including AI craziness (the Tim Hua post on this is excellent), some new OpenAI largely policy-related shenanigans, and the continuing craziness of some people who should very much know better confidently saying that we are not going to hit AGI any time soon, plus some odds and ends including dead internet theory.

That still leaves tons of other stuff.

Table of Contents

---

Outline:

(00:44) Language Models Offer Mundane Utility

(02:42) Language Models Don't Offer Mundane Utility

(08:24) On Your Marks

(13:26) Choose Your Fighter

(21:05) Fun With Media Generation

(22:10) Deepfaketown and Botpocalypse Soon

(25:32) Don't Be Evil

(26:41) They Took Our Jobs

(40:39) School Daze

(44:18) The Art of the Jailbreak

(44:27) Overcoming Bias

(54:01) Get Involved

(56:34) Introducing

(58:09) Unprompted Attention

(58:56) In Other AI News

(01:02:54) Show Me the Money

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Startup Roundups (#1, #2) look like they’re settling in as an annual tradition.

I’ve been catching up on queued roundups this week as the family was on vacation. I am back now but the weekly may be pushed to Friday as I dig out from the backlog.

The Central Themes

There is a common theme to the vignettes offered across the years. The central perspective has not much changed.

I would summarize the perspective this way, first in brief and then with more words:

---

Outline:

(00:25) The Central Themes

(04:12) How To Interpret Advice From Paul Graham

(07:52) Investor Meetings Do Not Create Optimal Strategic Choices

(09:23) Y-Combinator Is A Great Deal

(09:43) Avoiding Marking to Market

(13:59) Worse Deals Are Worse

(17:55) Avoiding Having A Market At All

(19:18) Eventually Be Hustling

(21:32) Without Selection Venture Capital Loses Money

(23:01) Finding Your Startup Idea

(28:00) Some Ideas And Builders Are Bad

(29:36) Whatever Sucks A Lot For You Will Suck A Lot For You

(30:34) Stay Away From Ayahuasca

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Traffic and transit are finally getting a roundup all their own.

I’ll start out with various victory laps on the awesomeness that is New York City Congestion pricing, which should hopefully now be a settled matter, then do a survey of everything else.

New York City Congestion Pricing Redux

We spent years fighting to get congestion pricing passed in New York City.

Once it was in place, everyone saw that it worked. The city was a better place, almost everyone was better off, and also we got bonus tax revenue. And this is only the bare minimum viable product, we could do so much better.

We had a big debate over whether congestion pricing is good until it was implemented. At this point, with traffic speeds downton up 15% and business visits improving, it is very clear this was exactly the huge [...]

---

Outline:

(00:24) New York City Congestion Pricing Redux

(10:34) New York City Light Rail

(11:51) Fare Evasion (1)

(14:55) Fare Evasion (2)

(16:07) Free Transit

(19:52) The Last Train

(20:11) Smooth Operator

(20:49) Mass Transit And Health

(21:51) Ah California High Speed Rail

(23:58) Amtrak

(25:18) The High Speed Rail We Actually Need Is Acela

(30:05) The Other High Speed Rail Project

(31:22) Caltrain

(32:07) Bike Lanes

(32:24) Airports

(32:54) Jones Act Bonus Episodes

(36:55) In Brief

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

There's quite a lot in the queue since last time, so this is the first large chunk of it, which focuses on apps and otherwise finding an initial connection, and some things that directly impact that.

Table of Contents

---

Outline:

(00:22) You're Single Because You Have No Friends To Date

(02:49) You're Single Because You Aren't Willing To Be Dungeon Master

(04:35) You're Single Because You Didn't Listen To My Friend's Podcast

(06:20) You're Single Because You Flunk The Eye Contact Test

(07:45) You're Single Because The Women Don't Hit On The Men

(12:27) You're Single Because When A Woman Did Hit On You, You Choked

(17:14) You're Single Because You Don't Know The Flirtation Escalation Ladder

(24:58) You're Single Because You Will Actually Never Open

(28:11) You're Single Because We Let Tinder Win

(29:43) You're Single Because The Apps Are Bad On Purpose To Make You Click

(30:55) You're Single Because You Present Super Weird

(35:09) You're Single Because You Have The Wrong Hobbies

(37:00) You're Single Because You Do The Wrong Kind Of Magic

(38:17) You're Single And All That Rejection Really Sucks

(41:54) You're Single Because Dating Apps Are Designed Like TikTok

(45:26) You're Single And Here's Aella With a Tea Update

(45:40) You're Single Because You Don't Have a Date Me Doc

(47:12) You're Single Because You Didn't Hire a Matchmaker

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.