Dr. Mike Israetel, renowned exercise scientist and social media personality, and more recently a low-P(doom) AI futurist, graciously offered to debate me!

00:00 Introducing Mike Israetel

12:19 What’s Your P(Doom)™

30:58 Timelines for Artificial General Intelligence

34:49 Superhuman AI Capabilities

43:26 AI Reasoning and Creativity

47:12 Evil AI Scenario

01:08:06 Will the AI Cooperate With Us?

01:12:27 AI's Dependence on Human Labor

01:18:27 Will AI Keep Us Around to Study Us?

01:42:38 AI's Approach to Earth's Resources

01:53:22 Global AI Policies and Risks

02:03:02 The Quality of Doom Discourse

02:09:23 Liron’s Outro

Show Notes

* Mike’s Instagram — https://www.instagram.com/drmikeisraetel

* Mike’s YouTube — https://www.youtube.com/@MikeIsraetelMakingProgress

Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.onlineHope to see you there!

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

I want to be transparent about how I’ve updated my mainline AI doom scenario in light of safe & useful LLMs. So here’s where I’m at…

00:00 Introduction

07:59 The Dangerous Threshold to Runaway Superintelligence

18:57 Superhuman Goal Optimization = Infinite Time Horizon

21:21 Goal-Completeness by Analogy to Turing-Completeness

26:53 Intellidynamics

29:13 Goal-Optimization Is Convergent

31:15 Early AIs Lose Control of Later AIs

34:46 The Superhuman Threshold Is Real

38:27 Expecting Rapid FOOM

40:20 Rocket Alignment

49:59 Stability of Values Under Self-Modification

53:13 The Way to Heaven Passes Right By Hell

57:32 My Mainline Doom Scenario

01:17:46 What Values Does The Goal Optimizer Have?

Show Notes

My recent episode with Jim Babcock on this same topic of mainline doom scenarios — https://www.youtube.com/watch?v=FaQjEABZ80g

The Rocket Alignment Problem by Eliezer Yudkowsky — https://www.lesswrong.com/posts/Gg9a4y8reWKtLe3Tn/the-rocket-alignment-problem

Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

What’s the most likely (“mainline”) AI doom scenario? How does the existence of LLMs update the original Yudkowskian version? I invited my friend Jim Babcock to help me answer these questions.

Jim is a member of the LessWrong engineering team and its parent organization, Lightcone Infrastructure. I’ve been a longtime fan of his thoughtful takes.

This turned out to be a VERY insightful and informative discussion, useful for clarifying my own predictions, and accessible to the show’s audience.

00:00 Introducing Jim Babcock

01:29 The Evolution of LessWrong Doom Scenarios

02:22 LessWrong’s Mission

05:49 The Rationalist Community and AI

09:37 What’s Your P(Doom)™

18:26 What Are Yudkowskians Surprised About?

26:48 Moral Philosophy vs. Goal Alignment

36:56 Sandboxing and AI Containment

42:51 Holding Yudkowskians Accountable

58:29 Understanding Next Word Prediction

01:00:02 Pre-Training vs Post-Training

01:08:06 The Rocket Alignment Problem Analogy

01:30:09 FOOM vs. Gradual Disempowerment

01:45:19 Recapping the Mainline Doom Scenario

01:52:08 Liron’s Outro

Show Notes

Jim’s LessWrong — https://www.lesswrong.com/users/jimrandomh

Jim’s Twitter — https://x.com/jimrandomh

The Rocket Alignment Problem by Eliezer Yudkowsky — https://www.lesswrong.com/posts/Gg9a4y8reWKtLe3Tn/the-rocket-alignment-problem

Optimality is the Tiger and Agents Are Its Teeth — https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth

Doom Debates episode about the research paper discovering AI's utility function — https://lironshapira.substack.com/p/cais-researchers-discover-ais-preferences

Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Ozzie Gooen is the founder of the Quantified Uncertainty Research Institute (QURI), a nonprofit building software tools for forecasting and policy analysis. I’ve known him through the rationality community since 2008 and we have a lot in common.

00:00 Introducing Ozzie

02:18 The Rationality Community

06:32 What’s Your P(Doom)™

08:09 High-Quality Discourse and Social Media

14:17 Guesstimate and Squiggle Demos

31:57 Prediction Markets and Rationality

38:33 Metaforecast Demo

41:23 Evaluating Everything with LLMs

47:00 Effective Altruism and FTX Scandal

56:00 The Repugnant Conclusion Debate

01:02:25 AI for Governance and Policy

01:12:07 PauseAI Policy Debate

01:30:10 Status Quo Bias

01:33:31 Decaf Coffee and Caffeine Powder

01:34:45 Are You Aspie?

01:37:45 Billionaires in Effective Altruism

01:48:06 Gradual Disempowerment by AI

01:55:36 LessOnline Conference

01:57:34 Supporting Ozzie’s Work

Show Notes

Quantified Uncertainty Research Institute (QURI) — https://quantifieduncertainty.org

Ozzie’s Facebook — https://www.facebook.com/ozzie.gooen

Ozzie’s Twitter — https://x.com/ozziegooen

Guesstimate, a spreadsheet for working with probability ranges — https://www.getguesstimate.com

Squiggle, a programming language for building Monte Carlo simulations — https://www.squiggle-language.com

Metaforecast, a prediction market aggregator — https://metaforecast.org

Open Annotate, AI-powered content analysis — https://github.com/quantified-uncertainty/open-annotate/

Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

David Duvenaud is a professor of Computer Science at the University of Toronto, co-director of the Schwartz Reisman Institute for Technology and Society, former Alignment Evals Team Lead at Anthropic, an award-winning machine learning researcher, and a close collaborator of Dr. Geoffrey Hinton. He recently co-authored Gradual Disempowerment.

We dive into David’s impressive career, his high P(Doom), his recent tenure at Anthropic, his views on gradual disempowerment, and the critical need for improved governance and coordination on a global scale.

00:00 Introducing David

03:03 Joining Anthropic and AI Safety Concerns

35:58 David’s Background and Early Influences

45:11 AI Safety and Alignment Challenges

54:08 What’s Your P(Doom)™

01:06:44 Balancing Productivity and Family Life

01:10:26 The Hamming Question: Are You Working on the Most Important Problem?

01:16:28 The PauseAI Movement

01:20:28 Public Discourse on AI Doom

01:24:49 Courageous Voices in AI Safety

01:43:54 Coordination and Government Role in AI

01:47:41 Cowardice in AI Leadership

02:00:05 Economic and Existential Doom

02:06:12 Liron’s Post-Show

Show Notes

David’s Twitter — https://x.com/DavidDuvenaud

Schwartz Reisman Institute for Technology and Society — https://srinstitute.utoronto.ca/

Jürgen Schmidhuber’s Home Page — https://people.idsia.ch/~juergen/

Ryan Greenblatt's LessWrong comment about a future scenario where there's a one-time renegotiation of power and heat from superintelligent AI projects causes the oceans to boil: https://www.lesswrong.com/posts/pZhEQieM9otKXhxmd/gradual-disempowerment-systemic-existential-risks-from?commentId=T7KZGGqq2Z4gXZsty

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

AI 2027, a bombshell new paper by the AI Futures Project, is a highly plausible scenario of the next few years of AI progress. I like this paper so much that I made a whole episode about it.

00:00 Overview of AI 2027

05:13 2025: Stumbling Agents

16:23 2026: Advanced Agents

21:49 2027: The Intelligence Explosion

29:13 AI's Initial Exploits and OpenBrain's Secrecy

30:41 Agent-3 and the Rise of Superhuman Engineering

37:05 The Creation and Deception of Agent-5

44:56 The Race Scenario: Humanity's Downfall

48:58 The Slowdown Scenario: A Glimmer of Hope

53:49 Final Thoughts

Show Notes

The website: https://ai-2027.com

Scott Alexander’s blog: https://astralcodexten.com

Daniel Kokotajlo’s previous predictions from 2021 about 2026: https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Dr. Peter Berezin is the Chief Global Strategist and Director of Research at BCA Research, the largest Canadian investment research firm. He’s known for his macroeconomics research reports and his frequent appearances on Bloomberg and CNBC.

Notably, Peter is one of the only macroeconomists in the world who’s forecasting AI doom! He recently published a research report estimating a “ more than 50/50 chance AI will wipe out all of humanity by the middle of the century”.

00:00 Introducing Peter Berezin

01:59 Peter’s Economic Predictions and Track Record

05:50 Investment Strategies and Beating the Market

17:47 The Future of Human Employment

26:40 Existential Risks and the Doomsday Argument

34:13 What’s Your P(Doom)™

39:18 Probability of non-AI Doom

44:19 Solving Population Decline

50:53 Constraining AI Development

53:40 The Multiverse and Its Implications

01:01:11 Are Other Economists Crazy?

01:09:19 Mathematical Universe and Multiverse Theories

01:19:43 Epistemic vs. Physical Probability

01:33:19 Reality Fluid

01:39:11 AI and Moral Realism

01:54:18 The Simulation Hypothesis and God

02:10:06 Liron’s Post-Show

Show Notes

Peter’s Twitter: https://x.com/PeterBerezinBCA

Peter’s old blog — https://stockcoach.blogspot.com

Peter’s 2021 BCA Research Report: “Life, Death and Finance in the Cosmic Multiverse” — https://www.bcaresearch.com/public/content/GIS_SR_2021_12_21.pdf

M.C. Escher’s “Circle Limit IV” — https://www.escherinhetpaleis.nl/escher-today/circle-limit-iv-heaven-and-hell/

Zvi Mowshowitz’s Blog (Liron’s recommendation for best AI news & analysis) — https://thezvi.substack.com

My Doom Debates episode about why nuclear proliferation is bad — https://www.youtube.com/watch?v=ueB9iRQsvQ8

Robin Hanson’s “Mangled Worlds” paper — https://mason.gmu.edu/~rhanson/mangledworlds.html

Uncontrollable by Darren McKee (Liron’s recommended AI x-risk book) — https://www.amazon.com/dp/B0CNNYKVH1

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Nathan Labenz, host of The Cognitive Revolution, joins me for an AI news & social media roundup!

00:00 Introducing Nate

05:18 What’s Your P(Doom)™

23:22 GPT-4o Image Generation

40:20 Will Fiverr’s Stock Crash?

47:41 AI Unemployment

55:11 Entrepreneurship

01:00:40 OpenAI Valuation

01:09:29 Connor Leahy’s Hair

01:13:28 Mass Extinction

01:25:30 Is anyone feeling the doom vibes?

01:38:20 Rethinking AI Individuality

01:40:35 “Softmax” — Emmett Shear's New AI Safety Org

01:57:04 Anthropic's Mechanistic Interpretability Paper

02:10:11 International Cooperation for AI Safety

02:18:43 Final Thoughts

Show Notes

Nate’s Twitter: https://x.com/labenz

Nate’s podcast: https://cognitiverevolution.ai and https://youtube.com/@CognitiveRevolutionPodcast

Nate’s company: https://waymark.com/

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

In this special cross-posted episode of Doom Debates, originally posted here on The Human Podcast, we cover a wide range of topics including the definition of “doom”, P(Doom), various existential risks like pandemics and nuclear threats, and the comparison of rogue AI risks versus AI misuse risks.

00:00 Introduction

01:47 Defining Doom and AI Risks

05:53 P(Doom)

10:04 Doom Debates’ Mission

16:17 Personal Reflections and Life Choices

24:57 The Importance of Debate

27:07 Personal Reflections on AI Doom

30:46 Comparing AI Doom to Other Existential Risks

33:42 Strategies to Mitigate AI Risks

39:31 The Global AI Race and Game Theory

43:06 Philosophical Reflections on a Good Life

45:21 Final Thoughts

Show Notes

The Human Podcast with Joe Murray: https://www.youtube.com/@thehumanpodcastofficial

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Don’t miss the other great AI doom show, For Humanity: https://youtube.com/@ForHumanityAIRisk

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Alexander Campbell claims that having superhuman intelligence doesn’t necessarily translate into having vast power, and that Gödel's Incompleteness Theorem ensures AI can’t get too powerful. I strongly disagree.

Alex has a Master's of Philosophy in Economics from the University of Oxford and an MBA from the Stanford Graduate School of Business, has worked as a quant trader at Lehman Brothers and Bridgewater Associates, and is the founder of Rose AI, a cloud data platform that leverages generative AI to help visualize data.

This debate was recorded in August 2023.

00:00 Intro and Alex’s Background

05:29 Alex's Views on AI and Technology

06:45 Alex’s Non-Doomer Position

11:20 Goal-to-Action Mapping

15:20 Outcome Pump Thought Experiment

21:07 Liron’s Doom Argument

29:10 The Dangers of Goal-to-Action Mappers

34:39 The China Argument and Existential Risks

45:18 Ideological Turing Test

48:38 Final Thoughts

Show Notes

Alexander Campbell’s Twitter: https://x.com/abcampbell

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates