Ilya’s doom bunker, proof of humanity, the doomsday argument, CAIS firing John Sherman, Bayesian networks, Westworld, AI consciousness, Eliezer’s latest podcast, and more!

00:00 Introduction

04:13 Doomsday Argument

09:22 What if AI Alignment is *Intractable*?

14:31 Steel-Manning the Nondoomers

22:13 No State-Level AI Regulation for 10 years?

32:31 AI Consciousness

35:25 Westworld Is Real Now

38:01 Proof of Humanity

40:33 Liron’s Notary Network Idea

43:34 Center for AI Safety and John Sherman Controversy

57:04 Technological Advancements and Future Predictions

01:03:14 Ilya Sutskever’s Doom Bunker

01:07:32 The Future of AGI and Training Models

01:12:19 Personal Experience of the Jetsons Future

01:15:16 The Role of AI in Everyday Tasks

01:18:54 Is General Intelligence A Binary Property?

01:23:52 Does an Open Platform Help Make AI Safe?

01:27:21 What of Understandable AI Like Bayesian Networks?

01:30:28 Why Doom Isn’t Emotionally Real for Liron

Show Notes

The post where people submitted questions: https://lironshapira.substack.com/p/5000-subscribers-live-q-and-a-ask

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit lironshapira.substack.com

Dr. Himanshu Tyagi is a professor of engineering at the Indian Institute of Science and the co-founder of Sentient, an open-source AI platform that raised $85M in funding led by Founders Fund.

In this conversation, Himanshu gives me Sentient’s pitch. Then we debate whether open-sourcing frontier AGI development is a good idea, or a reckless way to raise humanity’s P(doom).

00:00 Introducing Himanshu Tyagi

01:41 Sentient’s Vision

05:20 How’d You Raise $85M?

11:19 Comparing Sentient to Competitors

27:26 Open Source vs. Closed Source AI

43:01 What’s Your P(Doom)™

48:44 Extinction from Superintelligent AI

54:02 AI's Control Over Digital and Physical Assets

01:00:26 AI's Influence on Human Movements

01:08:46 Recapping the Debate

01:13:17 Liron’s Announcements

Show Notes

Himanshu’s Twitter — https://x.com/hstyagi

Sentient’s website — https://sentient.foundation

Come to the Less Online conference on May 30 - Jun 1, 2025:

https://less.online

Hope to see you there!

If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares —

https://ifanyonebuildsit.com

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of:

https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit lironshapira.substack.com

My friend John Sherman from the For Humanity podcast got hired by the Center for AI Safety (CAIS) two weeks ago.

Today I suddenly learned he’s been fired.

I’m frustrated by this decision, and frustrated with the whole AI x-risk community’s weak messaging.

Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online

Hope to see you there!

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit lironshapira.substack.com

Prof. Gary Marcus is a scientist, bestselling author and entrepreneur, well known as one of the most influential voices in AI. He is Professor Emeritus of Psychology and Neuroscience at NYU.  He was founder and CEO of Geometric Intelligence, a machine learning company acquired by Uber in 2016.

Gary co-authored the 2019 book, Rebooting AI: Building Artificial Intelligence We Can Trust, and the 2024 book, Taming Silicon Valley: How We Can Ensure That AI Works for Us. He played an important role in the 2023 Senate Judiciary Subcommittee Hearing on Oversight of AI, testifying with Sam Altman.

In this episode, Gary and I have a lively debate about whether P(doom) is approximately 50%, or if it’s less than 1%!

00:00 Introducing Gary Marcus

02:33 Gary’s AI Skepticism

09:08 The Human Brain is a Kluge

23:16 The 2023 Senate Judiciary Subcommittee Hearing

28:46 What’s Your P(Doom)™

44:27 AI Timelines

51:03 Is Superintelligence Real?

01:00:35 Humanity’s Immune System

01:12:46 Potential for Recursive Self-Improvement

01:26:12 AI Catastrophe Scenarios

01:34:09 Defining AI Agency

01:37:43 Gary’s AI Predictions

01:44:13 The NYTimes Obituary Test

01:51:11 Recap and Final Thoughts

01:53:35 Liron’s Outro

01:55:34 Eliezer Yudkowsky’s New Book!

01:59:49 AI Doom Concept of the Day

Show Notes

Gary’s Substack — https://garymarcus.substack.com

Gary’s Twitter — https://x.com/garymarcus

If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares — https://ifanyonebuildsit.com

Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online

Hope to see you there!

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit lironshapira.substack.com

 Dr. Mike Israetel, renowned exercise scientist and social media personality, and more recently a low-P(doom) AI futurist, graciously offered to debate me!

00:00 Introducing Mike Israetel

12:19 What’s Your P(Doom)™

30:58 Timelines for Artificial General Intelligence

34:49 Superhuman AI Capabilities

43:26 AI Reasoning and Creativity

47:12 Evil AI Scenario

01:08:06 Will the AI Cooperate With Us?

01:12:27 AI's Dependence on Human Labor

01:18:27 Will AI Keep Us Around to Study Us?

01:42:38 AI's Approach to Earth's Resources

01:53:22 Global AI Policies and Risks

02:03:02 The Quality of Doom Discourse

02:09:23 Liron’s Outro

Show Notes

* Mike’s Instagram — https://www.instagram.com/drmikeisraetel

* Mike’s YouTube — https://www.youtube.com/@MikeIsraetelMakingProgress

Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.onlineHope to see you there!

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit lironshapira.substack.com

I want to be transparent about how I’ve updated my mainline AI doom scenario in light of safe & useful LLMs. So here’s where I’m at…

00:00 Introduction

07:59 The Dangerous Threshold to Runaway Superintelligence

18:57 Superhuman Goal Optimization = Infinite Time Horizon

21:21 Goal-Completeness by Analogy to Turing-Completeness

26:53 Intellidynamics

29:13 Goal-Optimization Is Convergent

31:15 Early AIs Lose Control of Later AIs

34:46 The Superhuman Threshold Is Real

38:27 Expecting Rapid FOOM

40:20 Rocket Alignment

49:59 Stability of Values Under Self-Modification

53:13 The Way to Heaven Passes Right By Hell

57:32 My Mainline Doom Scenario

01:17:46 What Values Does The Goal Optimizer Have?

Show Notes

My recent episode with Jim Babcock on this same topic of mainline doom scenarios — https://www.youtube.com/watch?v=FaQjEABZ80g

The Rocket Alignment Problem by Eliezer Yudkowsky — https://www.lesswrong.com/posts/Gg9a4y8reWKtLe3Tn/the-rocket-alignment-problem

Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit lironshapira.substack.com

What’s the most likely (“mainline”) AI doom scenario? How does the existence of LLMs update the original Yudkowskian version? I invited my friend Jim Babcock to help me answer these questions.

Jim is a member of the LessWrong engineering team and its parent organization, Lightcone Infrastructure. I’ve been a longtime fan of his thoughtful takes.

This turned out to be a VERY insightful and informative discussion, useful for clarifying my own predictions, and accessible to the show’s audience.

00:00 Introducing Jim Babcock

01:29 The Evolution of LessWrong Doom Scenarios

02:22 LessWrong’s Mission

05:49 The Rationalist Community and AI

09:37 What’s Your P(Doom)™

18:26 What Are Yudkowskians Surprised About?

26:48 Moral Philosophy vs. Goal Alignment

36:56 Sandboxing and AI Containment

42:51 Holding Yudkowskians Accountable

58:29 Understanding Next Word Prediction

01:00:02 Pre-Training vs Post-Training

01:08:06 The Rocket Alignment Problem Analogy

01:30:09 FOOM vs. Gradual Disempowerment

01:45:19 Recapping the Mainline Doom Scenario

01:52:08 Liron’s Outro

Show Notes

Jim’s LessWrong — https://www.lesswrong.com/users/jimrandomh

Jim’s Twitter — https://x.com/jimrandomh

The Rocket Alignment Problem by Eliezer Yudkowsky — https://www.lesswrong.com/posts/Gg9a4y8reWKtLe3Tn/the-rocket-alignment-problem

Optimality is the Tiger and Agents Are Its Teeth — https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth

Doom Debates episode about the research paper discovering AI's utility function — https://lironshapira.substack.com/p/cais-researchers-discover-ais-preferences

Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit lironshapira.substack.com

Ozzie Gooen is the founder of the Quantified Uncertainty Research Institute (QURI), a nonprofit building software tools for forecasting and policy analysis. I’ve known him through the rationality community since 2008 and we have a lot in common.

00:00 Introducing Ozzie

02:18 The Rationality Community

06:32 What’s Your P(Doom)™

08:09 High-Quality Discourse and Social Media

14:17 Guesstimate and Squiggle Demos

31:57 Prediction Markets and Rationality

38:33 Metaforecast Demo

41:23 Evaluating Everything with LLMs

47:00 Effective Altruism and FTX Scandal

56:00 The Repugnant Conclusion Debate

01:02:25 AI for Governance and Policy

01:12:07 PauseAI Policy Debate

01:30:10 Status Quo Bias

01:33:31 Decaf Coffee and Caffeine Powder

01:34:45 Are You Aspie?

01:37:45 Billionaires in Effective Altruism

01:48:06 Gradual Disempowerment by AI

01:55:36 LessOnline Conference

01:57:34 Supporting Ozzie’s Work

Show Notes

Quantified Uncertainty Research Institute (QURI) — https://quantifieduncertainty.org

Ozzie’s Facebook — https://www.facebook.com/ozzie.gooen

Ozzie’s Twitter — https://x.com/ozziegooen

Guesstimate, a spreadsheet for working with probability ranges — https://www.getguesstimate.com

Squiggle, a programming language for building Monte Carlo simulations — https://www.squiggle-language.com

Metaforecast, a prediction market aggregator — https://metaforecast.org

Open Annotate, AI-powered content analysis — https://github.com/quantified-uncertainty/open-annotate/

Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit lironshapira.substack.com

David Duvenaud is a professor of Computer Science at the University of Toronto, co-director of the Schwartz Reisman Institute for Technology and Society, former Alignment Evals Team Lead at Anthropic, an award-winning machine learning researcher, and a close collaborator of Dr. Geoffrey Hinton. He recently co-authored Gradual Disempowerment.

We dive into David’s impressive career, his high P(Doom), his recent tenure at Anthropic, his views on gradual disempowerment, and the critical need for improved governance and coordination on a global scale.

00:00 Introducing David

03:03 Joining Anthropic and AI Safety Concerns

35:58 David’s Background and Early Influences

45:11 AI Safety and Alignment Challenges

54:08 What’s Your P(Doom)™

01:06:44 Balancing Productivity and Family Life

01:10:26 The Hamming Question: Are You Working on the Most Important Problem?

01:16:28 The PauseAI Movement

01:20:28 Public Discourse on AI Doom

01:24:49 Courageous Voices in AI Safety

01:43:54 Coordination and Government Role in AI

01:47:41 Cowardice in AI Leadership

02:00:05 Economic and Existential Doom

02:06:12 Liron’s Post-Show

Show Notes

David’s Twitter — https://x.com/DavidDuvenaud

Schwartz Reisman Institute for Technology and Society — https://srinstitute.utoronto.ca/

Jürgen Schmidhuber’s Home Page — https://people.idsia.ch/~juergen/

Ryan Greenblatt's LessWrong comment about a future scenario where there's a one-time renegotiation of power and heat from superintelligent AI projects causes the oceans to boil: https://www.lesswrong.com/posts/pZhEQieM9otKXhxmd/gradual-disempowerment-systemic-existential-risks-from?commentId=T7KZGGqq2Z4gXZsty

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit lironshapira.substack.com

AI 2027, a bombshell new paper by the AI Futures Project, is a highly plausible scenario of the next few years of AI progress. I like this paper so much that I made a whole episode about it.

00:00 Overview of AI 2027

05:13 2025: Stumbling Agents

16:23 2026: Advanced Agents

21:49 2027: The Intelligence Explosion

29:13 AI's Initial Exploits and OpenBrain's Secrecy

30:41 Agent-3 and the Rise of Superhuman Engineering

37:05 The Creation and Deception of Agent-5

44:56 The Race Scenario: Humanity's Downfall

48:58 The Slowdown Scenario: A Glimmer of Hope

53:49 Final Thoughts

Show Notes

The website: https://ai-2027.com

Scott Alexander’s blog: https://astralcodexten.com

Daniel Kokotajlo’s previous predictions from 2021 about 2026: https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit lironshapira.substack.com