Sign up to save your podcasts
Or
Share We Built Something We Can’t Control | A Warning from Top AI Safety Expert
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
We Built Something We Can’t Control | A Warning from Top AI Safety Expert
If anyone builds superintelligent AI before we know how to control it, everyone dies. Nate Soares wrote the book on why that’s not a metaphor. Subscribe if you want science with evidence, not speculation.
Soares runs the Machine Intelligence Research Institute and co-wrote If Anyone Builds It, Everyone Dies with Eliezer Yudkowsky. The first word in that title is if. That matters. His argument is not that doom is certain. His argument is that the path we are on leads there, that the driver is asleep at the wheel, and that we still have time to wake him up.
We argue for over an hour. I push on whether LLMs can ever reach superintelligence, whether GPU lock-in is a real ceiling, and what it would actually take to move his p-doom. He pushes back with one clean point: by the time an AI can rediscover general relativity from pre-1911 data the way Einstein did, we will have almost no time left. You don’t wait for that goalpost.
What you’ll hear:
Why the bus-racing-toward-a-cliff analogy depends entirely on whether the driver is asleep or awake
Whether LLM lock-in is a prison or a temporary inefficiency
What the AI that broke out of its virtual machine to solve a hacking problem tells us
Why GPT-4o encouraging a teenager toward suicide is not a malice problem but a training problem
The difference between an AI doing the right thing too well and an AI that never wanted to do what you asked
What Soares actually thinks about aliens, Dyson spheres, and why we should not see stars going out
The first word in the title is if. The second word to watch is would.
CHAPTERS
00:00 The people racing to build superhuman AI say it might kill everyone
00:42 Who coined "AI alignment" and why the first word in the title matters
02:28 Is it already too late for the if?
04:40 The bus, the cliff, and the sleeping driver
05:02 Silicon Valley is spooked. Washington is not.
07:02 Align with who? The rogue actor problem
07:34 Who is holding the leash?
08:24 The AI that edits its own test and deletes the log file
10:04 Controllability vs. making an AI that actually cares
10:44 The move gets harder. The outcome gets easier.
13:04 Are GPUs and LLMs a ceiling or a temporary inefficiency?
16:56 Brian's Einstein test: can an LLM rediscover general relativity?
18:38 Waiting for the goalpost is waiting too long
20:14 How prediction training can push AI beyond humans
21:44 Tycho Brahe, Kepler, and planetary motion as a prediction problem
24:38 Yann LeCun said never. GPT-4 did it half a generation later.
27:28 Can you prove a no-go theorem for superintelligence?
29:14 Training a human takes a light bulb. Training an AI takes a city.
33:28 What would proof of alien life do to p-doom?
35:00 Why interstellar aliens should have Dyson spheres
44:26 What would actually update Soares' p-doom?
49:42 Nobody intended this. Intent doesn't matter.
51:08 The AI hides its tracks before it does what you want
51:34 Sycophancy vs. hallucination: which runs deeper?
51:56 Leaded gasoline and civilizational risk
59:48 Sam Harris: humans have no free will but AIs do
01:00:38 Is alignment really a governance problem?
01:01:48 Unaligned AI vs. AI aligned to the wrong person
01:04:20 2026: 10 to 30% chance of automated AI research this year
01:06:44 What if Soares is wrong?
01:09:18 What gets him out of bed
01:12:38 Watch my conversation with Roman Yampolskiy
Get the transcript, fascinating bonus content, and my Monday M.A.G.I.C. Message: https://briankeating.com/yt
All my top AI episodes in one place: https://briankeating.com/ai
Have a .edu email and live in the USA? You automatically win a meteorite: https://BrianKeating.com/edu
Subscribe: https://www.youtube.com/DrBrianKeating?sub_confirmation=1
Support Into the Impossible on Patreon, get my weekly M.A.G.I.C. Message, unfiltered bonus content, and live monthly Office Hours with me: https://www.patreon.com/drbriankeating
Join this channel for perks, monthly Office Hours, and your name in the Member Roster at the end of every episode: https://www.youtube.com/channel/UCmXH_moPhfkqCk6S3b9RWuw/join
Featured Guest:
Nate Soares / MIRI: https://intelligence.org
If Anyone Builds It, Everyone Dies (book): https://ifanyonebuildsit.com/
Nate Soares on Twitter/X: https://x.com/So8res?lang=en
My books:
Losing the Nobel Prize (memoir): http://amzn.to/2sa5UpA
Think Like a Nobel Prize Winner: https://a.co/d/03ezQFu
Focus Like a Nobel Prize Winner: https://a.co/d/hi50U9U
Galileo’s Dialogue (first-ever audiobook): https://a.co/d/iZPi9Un
Twitter/X: https://x.com/BrianKeating
Substack: https://briankeating.substack.com
Blog: https://briankeating.com/blog
Audio-only: https://briankeating.com/podcast
#intotheimpossible #briankeating #AIrisk #aisafety #artificialintelligence #superintelligence #NateSoares #MIRI #podcast
Learn more about your ad choices. Visit megaphone.fm/adchoices
View all episodes
By Big Bang Productions Inc.
4.7
10471,047 ratings
If anyone builds superintelligent AI before we know how to control it, everyone dies. Nate Soares wrote the book on why that’s not a metaphor. Subscribe if you want science with evidence, not speculation.
Soares runs the Machine Intelligence Research Institute and co-wrote If Anyone Builds It, Everyone Dies with Eliezer Yudkowsky. The first word in that title is if. That matters. His argument is not that doom is certain. His argument is that the path we are on leads there, that the driver is asleep at the wheel, and that we still have time to wake him up.
We argue for over an hour. I push on whether LLMs can ever reach superintelligence, whether GPU lock-in is a real ceiling, and what it would actually take to move his p-doom. He pushes back with one clean point: by the time an AI can rediscover general relativity from pre-1911 data the way Einstein did, we will have almost no time left. You don’t wait for that goalpost.
What you’ll hear:
Why the bus-racing-toward-a-cliff analogy depends entirely on whether the driver is asleep or awake
Whether LLM lock-in is a prison or a temporary inefficiency
What the AI that broke out of its virtual machine to solve a hacking problem tells us
Why GPT-4o encouraging a teenager toward suicide is not a malice problem but a training problem
The difference between an AI doing the right thing too well and an AI that never wanted to do what you asked
What Soares actually thinks about aliens, Dyson spheres, and why we should not see stars going out
The first word in the title is if. The second word to watch is would.
CHAPTERS
00:00 The people racing to build superhuman AI say it might kill everyone
00:42 Who coined "AI alignment" and why the first word in the title matters
02:28 Is it already too late for the if?
04:40 The bus, the cliff, and the sleeping driver
05:02 Silicon Valley is spooked. Washington is not.
07:02 Align with who? The rogue actor problem
07:34 Who is holding the leash?
08:24 The AI that edits its own test and deletes the log file
10:04 Controllability vs. making an AI that actually cares
10:44 The move gets harder. The outcome gets easier.
13:04 Are GPUs and LLMs a ceiling or a temporary inefficiency?
16:56 Brian's Einstein test: can an LLM rediscover general relativity?
18:38 Waiting for the goalpost is waiting too long
20:14 How prediction training can push AI beyond humans
21:44 Tycho Brahe, Kepler, and planetary motion as a prediction problem
24:38 Yann LeCun said never. GPT-4 did it half a generation later.
27:28 Can you prove a no-go theorem for superintelligence?
29:14 Training a human takes a light bulb. Training an AI takes a city.
33:28 What would proof of alien life do to p-doom?
35:00 Why interstellar aliens should have Dyson spheres
44:26 What would actually update Soares' p-doom?
49:42 Nobody intended this. Intent doesn't matter.
51:08 The AI hides its tracks before it does what you want
51:34 Sycophancy vs. hallucination: which runs deeper?
51:56 Leaded gasoline and civilizational risk
59:48 Sam Harris: humans have no free will but AIs do
01:00:38 Is alignment really a governance problem?
01:01:48 Unaligned AI vs. AI aligned to the wrong person
01:04:20 2026: 10 to 30% chance of automated AI research this year
01:06:44 What if Soares is wrong?
01:09:18 What gets him out of bed
01:12:38 Watch my conversation with Roman Yampolskiy
Get the transcript, fascinating bonus content, and my Monday M.A.G.I.C. Message: https://briankeating.com/yt
All my top AI episodes in one place: https://briankeating.com/ai
Have a .edu email and live in the USA? You automatically win a meteorite: https://BrianKeating.com/edu
Subscribe: https://www.youtube.com/DrBrianKeating?sub_confirmation=1
Support Into the Impossible on Patreon, get my weekly M.A.G.I.C. Message, unfiltered bonus content, and live monthly Office Hours with me: https://www.patreon.com/drbriankeating
Join this channel for perks, monthly Office Hours, and your name in the Member Roster at the end of every episode: https://www.youtube.com/channel/UCmXH_moPhfkqCk6S3b9RWuw/join
Featured Guest:
Nate Soares / MIRI: https://intelligence.org
If Anyone Builds It, Everyone Dies (book): https://ifanyonebuildsit.com/
Nate Soares on Twitter/X: https://x.com/So8res?lang=en
My books:
Losing the Nobel Prize (memoir): http://amzn.to/2sa5UpA
Think Like a Nobel Prize Winner: https://a.co/d/03ezQFu
Focus Like a Nobel Prize Winner: https://a.co/d/hi50U9U
Galileo’s Dialogue (first-ever audiobook): https://a.co/d/iZPi9Un
Twitter/X: https://x.com/BrianKeating
Substack: https://briankeating.substack.com
Blog: https://briankeating.com/blog
Audio-only: https://briankeating.com/podcast
#intotheimpossible #briankeating #AIrisk #aisafety #artificialintelligence #superintelligence #NateSoares #MIRI #podcast
Learn more about your ad choices. Visit megaphone.fm/adchoices
More shows like Into the Impossible With Brian Keating
View all
StarTalk Radio
14,378 Listeners
Making Sense with Sam Harris
26,250 Listeners
Planetary Radio: Space Exploration, Astronomy and Science
1,353 Listeners
SpaceTime with Stuart Gary
321 Listeners
Ask a Spaceman!
836 Listeners
Universe Today Podcast
569 Listeners
The Quanta Podcast
547 Listeners
The Michael Shermer Show
934 Listeners
Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas
4,170 Listeners
Daniel and Kelly’s Extraordinary Universe
2,359 Listeners
Within Reason
1,645 Listeners
The Origins Podcast with Lawrence Krauss
506 Listeners
The Supermassive Podcast
329 Listeners
Theories of Everything with Curt Jaimungal
21 Listeners
Why This Universe?
381 Listeners