Doom Debates!

Doom Debates!

By Liron ShapiraBusinessTechnology
Download on the App Store

Doom Debates! episodes

  • Poking holes in the AI doom argument — 83 stops where you could get off the “Doom Train”

    I often talk about the “Doom Train”, the series of claims and arguments involved in concluding that P(Doom) from artificial superintelligence is high. In this episode, it’s finally time to show you the whole track!

    00:00 Introduction

    01:09 “AGI isn’t coming soon”

    04:42 “Artificial intelligence can’t go far beyond human intelligence”

    07:24 “AI won’t be a physical threat”

    08:28 “Intelligence yields moral goodness”

    09:39 “We have a safe AI development process”

    10:48 “AI capabilities will rise at a manageable pace”

    12:28 “AI won’t try to conquer the universe”

    15:12 “Superalignment is a tractable problem”

    16:55 “Once we solve superalignment, we’ll enjoy peace”

    19:02 “Unaligned ASI will spare us”

    20:12 “AI doomerism is bad epistemology”

    21:42 Bonus arguments: “Fine, P(Doom) is high… but that’s ok!”

    Stops on the Doom Train

    AGI isn’t coming soon

    * No consciousness

    * No emotions

    * No creativity — AIs are limited to copying patterns in their training data, they can’t “generate new knowledge”

    * AIs aren’t even as smart as dogs right now, never mind humans

    * AIs constantly make dumb mistakes, they can’t even do simple arithmetic reliably

    * LLM performance is hitting a wall — GPT 4.5 is barely better than GPT 4.1 despite being larger scale

    * No genuine reasoning

    * No microtubules exploiting uncomputable quantum effects

    * No soul

    * We’ll need to build tons of data centers and power before we get to AGI

    * No agency

    * This is just another AI hype cycle, every 25 years people think AGI is coming soon and they’re wrong

    Artificial intelligence can’t go far beyond human intelligence

    * “Superhuman intelligence” is a meaningless concept

    * Human engineering already is coming close to the laws of physics

    * Coordinating a large engineering project can’t happen much faster than humans do it

    * No individual human is that smart compared to humanity as a whole, including our culture, corporations, and other institutions. Similarly no individual AI will ever be that smart compared to the sum of human culture and other institutions.

    AI won’t be a physical threat

    * AI doesn’t have arms or legs, it has zero control over the real world

    * An AI with a robot body can’t fight better than a human soldier

    * We can just disconnect an AI’s power to stop it

    * We can just turn off the internet to stop it

    * We can just shoot it with a gun

    * It’s just math

    * Any supposed chain of events where AI kills humans is far-fetched science fiction

    Intelligence yields moral goodness

    * More intelligence is correlated with more morality

    * Smarter people commit fewer crimes

    * The orthogonality thesis is false

    * AIs will discover moral realism

    * If we made AIs so smart, and we were trying to make them moral, then they’ll be smart enough to debug their own morality

    * Positive-sum cooperation was the outcome of natural selection

    We have a safe AI development process

    * Just like every new technology, we’ll figure it out as we go

    * We don’t know what problems need to be fixed until we build the AI and test it out

    * If an AI causes problems, we’ll be able to turn it off and release another version

    * We have safeguards to make sure AI doesn’t get uncontrollable/unstoppable

    * If we accidentally build an AI that stops accepting our shutoff commands, it won’t manage to copy versions of itself outside our firewalls which then proceed to spread exponentially like a computer virus

    * If we accidentally build an AI that escapes our data center and spreads exponentially like a computer virus, it won’t do too much damage in the world before we can somehow disable or neutralize all its copies

    * If we can’t disable or neutralize copies of rogue AIs, we’ll rapidly build other AIs that can do that job for us, and won’t themselves go rogue on us

    AI capabilities will rise at a manageable pace

    * Building larger data centers will be a speed bottleneck

    * Another speed bottleneck is the amount of research that needs to be done, both in terms of computational simulation, and in terms of physical experiments, and this kind of research takes lots of time

    * Recursive self-improvement “foom” is impossible

    * The whole economy never grows with localized centralized “foom”

    * Need to collect cultural learnings over time, like humanity did as a whole

    * AI is just part of the good pattern of exponential economic growth eras

    AI won’t try to conquer the universe

    * AIs can’t “want” things

    * AIs won’t have the same “fight instincts” as humans and animals, because they weren’t shaped by a natural selection process that involved life-or-death resource competition

    * Smart employees often work for less-smart bosses

    * Just because AIs help achieve goals doesn’t mean they have to be hard-core utility maximizers

    * Instrumental convergence is false: achieving goals effectively doesn’t mean you have to be relentlessly seizing power and resources

    * A resource-hungry goal-maximizer AIs wouldn’t seize literally every atom; there’ll still be some leftover resources for humanity

    * AIs will use new kinds of resources that humans aren’t using - dark energy, wormholes, alternate universes, etc

    Superalignment is a tractable problem

    * Current AIs have never killed anybody

    * Current AIs are extremely successful at doing useful tasks for humans

    * If AIs are trained on data from humans, they’ll be “aligned by default”

    * We can just make AIs abide by our laws

    * We can align the superintelligent AIs by using a scheme involving cryptocurrency on the blockchain

    * Companies have economic incentives to solve superintelligent AI alignment, because unaligned superintelligent AI would hurt their profits

    * We’ll build an aligned not-that-smart AI, which will figure out how to build the next-generation AI which is smarter and still aligned to human values, and so on until aligned superintelligence

    Once we solve superalignment, we’ll enjoy peace

    * The power from ASI won’t be monopolized by a single human government / tyranny

    * The decentralized nodes of human-ASI hybrids won’t be like warlords constantly fighting each other, they’ll be like countries making peace

    * Defense will have an advantage over attack, so the equilibrium of all the groups of humans and ASIs will be multiple defended regions, not a war of mutual destruction

    * The world of human-owned ASIs is a stable equilibrium, not one where ASI-focused projects keep buying out and taking resources away from human-focused ones (Gradual Disempowerment)

    Unaligned ASI will spare us

    * The AI will spare us because it values the fact that we created it

    * The AI will spare us because studying us helps maximize its curiosity and learning

    * The AI will spare us because it feels toward us the way we feel toward our pets

    * The AI will spare us because peaceful coexistence creates more economic value than war

    * The AI will spare us because Ricardo’s Law of Comparative Advantage says you can still benefit economically from trading with someone who’s weaker than you

    AI doomerism is bad epistemology

    * It’s impossible to predict doom

    * It’s impossible to put a probability on doom

    * Every doom prediction has always been wrong

    * Every doomsayer is either psychologically troubled or acting on corrupt incentives

    * If we were really about to get doomed, everyone would already be agreeing about that, and bringing it up all the time

    Sure P(Doom) is high, but let’s race to build it anyway because…

    Coordinating to not build ASI is impossible

    * China will build ASI as fast as it can, no matter what — because of game theory

    * So however low our chance of surviving it is, the US should take the chance first

    Slowing down the AI race doesn’t help anything

    * Chances of solving AI alignment won’t improve if we slow down or pause the capabilities race

    * I personally am going to die soon, and I don’t care about future humans, so I’m open to any hail mary to prevent myself from dying

    * Humanity is already going to rapidly destroy ourselves with nuclear war, climate change, etc

    * Humanity is already going to die out soon because we won’t have enough babies

    Think of the good outcome

    * If it turns out that doom from overly-fast AI building doesn’t happen, in that case, we can more quickly get to the good outcome!

    * People will stop suffering and dying faster

    AI killing us all is actually good

    * Human existence is morally negative on net, or close to zero net moral value

    * Whichever AI ultimately comes to power will be a “worthy successor” to humanity

    * Whichever AI ultimately comes to power will be as morally valuable as human descendents generally are to their ancestors, even if their values drift

    * The successor AI’s values will be interesting, productive values that let them successfully compete to dominate the universe

    * How can you argue with the moral choices of an ASI that’s smarter than you, that you know goodness better than it does?

    * It’s species-ist to judge what a superintelligent AI would want to do. The moral circle shouldn’t be limited to just humanity.

    * Increasing entropy is the ultimate north star for techno-capital, and AI will increase entropy faster

    * Human extinction will solve the climate crisis, and pollution, and habitat destruction, and let mother earth heal

    Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!

    PauseAI, the volunteer organization I’m part of: https://pauseai.info

    Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

    Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

    Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates



    Get full access to Doom Debates at lironshapira.substack.com/subscribe
    28 min
  • Q&A: Ilya's AGI Doomsday Bunker, Veo 3 is Westworld, Eliezer Yudkowsky, and much more!

    Ilya’s doom bunker, proof of humanity, the doomsday argument, CAIS firing John Sherman, Bayesian networks, Westworld, AI consciousness, Eliezer’s latest podcast, and more!

    00:00 Introduction

    04:13 Doomsday Argument

    09:22 What if AI Alignment is *Intractable*?

    14:31 Steel-Manning the Nondoomers

    22:13 No State-Level AI Regulation for 10 years?

    32:31 AI Consciousness

    35:25 Westworld Is Real Now

    38:01 Proof of Humanity

    40:33 Liron’s Notary Network Idea

    43:34 Center for AI Safety and John Sherman Controversy

    57:04 Technological Advancements and Future Predictions

    01:03:14 Ilya Sutskever’s Doom Bunker

    01:07:32 The Future of AGI and Training Models

    01:12:19 Personal Experience of the Jetsons Future

    01:15:16 The Role of AI in Everyday Tasks

    01:18:54 Is General Intelligence A Binary Property?

    01:23:52 Does an Open Platform Help Make AI Safe?

    01:27:21 What of Understandable AI Like Bayesian Networks?

    01:30:28 Why Doom Isn’t Emotionally Real for Liron

    Show Notes

    The post where people submitted questions: https://lironshapira.substack.com/p/5000-subscribers-live-q-and-a-ask

    Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!

    PauseAI, the volunteer organization I’m part of: https://pauseai.info

    Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

    Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

    Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates



    Get full access to Doom Debates at lironshapira.substack.com/subscribe
    1 hr 36 min
  • This $85M-Backed Founder Claims Open Source AGI is Safe — Debate with Himanshu Tyagi

    Dr. Himanshu Tyagi is a professor of engineering at the Indian Institute of Science and the co-founder of Sentient, an open-source AI platform that raised $85M in funding led by Founders Fund.

    In this conversation, Himanshu gives me Sentient’s pitch. Then we debate whether open-sourcing frontier AGI development is a good idea, or a reckless way to raise humanity’s P(doom).

    00:00 Introducing Himanshu Tyagi

    01:41 Sentient’s Vision

    05:20 How’d You Raise $85M?

    11:19 Comparing Sentient to Competitors

    27:26 Open Source vs. Closed Source AI

    43:01 What’s Your P(Doom)™

    48:44 Extinction from Superintelligent AI

    54:02 AI's Control Over Digital and Physical Assets

    01:00:26 AI's Influence on Human Movements

    01:08:46 Recapping the Debate

    01:13:17 Liron’s Announcements

    Show Notes

    Himanshu’s Twitter — https://x.com/hstyagi

    Sentient’s website — https://sentient.foundation

    Come to the Less Online conference on May 30 - Jun 1, 2025:

    https://less.online

    Hope to see you there!

    If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares —

    https://ifanyonebuildsit.com

    Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

    PauseAI, the volunteer organization I’m part of:

    https://pauseai.info

    Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

    Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

    Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates



    Get full access to Doom Debates at lironshapira.substack.com/subscribe
    1 hr 22 min
  • Emergency Episode: Center for AI Safety Chickens Out

    My friend John from the AI X-Risk Network podcast got hired by the Center for AI Safety (CAIS) two weeks ago.

    Today I suddenly learned he’s been fired.

    I’m frustrated by this decision, and frustrated with the whole AI x-risk community’s weak messaging.

    Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online

    Hope to see you there!

    Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

    PauseAI, the volunteer organization I’m part of: https://pauseai.info

    Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

    Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

    Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates



    Get full access to Doom Debates at lironshapira.substack.com/subscribe
    17 min
  • Gary Marcus vs. Liron Shapira — AI Doom Debate

    Prof. Gary Marcus is a scientist, bestselling author and entrepreneur, well known as one of the most influential voices in AI. He is Professor Emeritus of Psychology and Neuroscience at NYU.  He was founder and CEO of Geometric Intelligence, a machine learning company acquired by Uber in 2016.

    Gary co-authored the 2019 book, Rebooting AI: Building Artificial Intelligence We Can Trust, and the 2024 book, Taming Silicon Valley: How We Can Ensure That AI Works for Us. He played an important role in the 2023 Senate Judiciary Subcommittee Hearing on Oversight of AI, testifying with Sam Altman.

    In this episode, Gary and I have a lively debate about whether P(doom) is approximately 50%, or if it’s less than 1%!

    00:00 Introducing Gary Marcus

    02:33 Gary’s AI Skepticism

    09:08 The Human Brain is a Kluge

    23:16 The 2023 Senate Judiciary Subcommittee Hearing

    28:46 What’s Your P(Doom)™

    44:27 AI Timelines

    51:03 Is Superintelligence Real?

    01:00:35 Humanity’s Immune System

    01:12:46 Potential for Recursive Self-Improvement

    01:26:12 AI Catastrophe Scenarios

    01:34:09 Defining AI Agency

    01:37:43 Gary’s AI Predictions

    01:44:13 The NYTimes Obituary Test

    01:51:11 Recap and Final Thoughts

    01:53:35 Liron’s Outro

    01:55:34 Eliezer Yudkowsky’s New Book!

    01:59:49 AI Doom Concept of the Day

    Show Notes

    Gary’s Substack — https://garymarcus.substack.com

    Gary’s Twitter — https://x.com/garymarcus

    If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares — https://ifanyonebuildsit.com

    Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online

    Hope to see you there!

    Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

    PauseAI, the volunteer organization I’m part of: https://pauseai.info

    Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

    Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

    Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates



    Get full access to Doom Debates at lironshapira.substack.com/subscribe
    2 hr 5 min
  • Mike Israetel vs. Liron Shapira — AI Doom Debate

     Dr. Mike Israetel, renowned exercise scientist and social media personality, and more recently a low-P(doom) AI futurist, graciously offered to debate me!

    00:00 Introducing Mike Israetel

    12:19 What’s Your P(Doom)™

    30:58 Timelines for Artificial General Intelligence

    34:49 Superhuman AI Capabilities

    43:26 AI Reasoning and Creativity

    47:12 Evil AI Scenario

    01:08:06 Will the AI Cooperate With Us?

    01:12:27 AI's Dependence on Human Labor

    01:18:27 Will AI Keep Us Around to Study Us?

    01:42:38 AI's Approach to Earth's Resources

    01:53:22 Global AI Policies and Risks

    02:03:02 The Quality of Doom Discourse

    02:09:23 Liron’s Outro

    Show Notes

    * Mike’s Instagram — https://www.instagram.com/drmikeisraetel

    * Mike’s YouTube — https://www.youtube.com/@MikeIsraetelMakingProgress

    Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.onlineHope to see you there!

    Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

    PauseAI, the volunteer organization I’m part of: https://pauseai.info

    Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

    Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

    Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates



    Get full access to Doom Debates at lironshapira.substack.com/subscribe
    2 hr 16 min
  • Doom Scenario: Human-Level AI Can't Control Smarter AI

    I want to be transparent about how I’ve updated my mainline AI doom scenario in light of safe & useful LLMs. So here’s where I’m at…

    00:00 Introduction

    07:59 The Dangerous Threshold to Runaway Superintelligence

    18:57 Superhuman Goal Optimization = Infinite Time Horizon

    21:21 Goal-Completeness by Analogy to Turing-Completeness

    26:53 Intellidynamics

    29:13 Goal-Optimization Is Convergent

    31:15 Early AIs Lose Control of Later AIs

    34:46 The Superhuman Threshold Is Real

    38:27 Expecting Rapid FOOM

    40:20 Rocket Alignment

    49:59 Stability of Values Under Self-Modification

    53:13 The Way to Heaven Passes Right By Hell

    57:32 My Mainline Doom Scenario

    01:17:46 What Values Does The Goal Optimizer Have?

    Show Notes

    My recent episode with Jim Babcock on this same topic of mainline doom scenarios — https://www.youtube.com/watch?v=FaQjEABZ80g

    The Rocket Alignment Problem by Eliezer Yudkowsky — https://www.lesswrong.com/posts/Gg9a4y8reWKtLe3Tn/the-rocket-alignment-problem

    Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online

    Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

    PauseAI, the volunteer organization I’m part of: https://pauseai.info

    Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

    Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

    Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates



    Get full access to Doom Debates at lironshapira.substack.com/subscribe
    1 hr 25 min
  • The Most Likely AI Doom Scenario — with Jim Babcock, LessWrong Team

    What’s the most likely (“mainline”) AI doom scenario? How does the existence of LLMs update the original Yudkowskian version? I invited my friend Jim Babcock to help me answer these questions.

    Jim is a member of the LessWrong engineering team and its parent organization, Lightcone Infrastructure. I’ve been a longtime fan of his thoughtful takes.

    This turned out to be a VERY insightful and informative discussion, useful for clarifying my own predictions, and accessible to the show’s audience.

    00:00 Introducing Jim Babcock

    01:29 The Evolution of LessWrong Doom Scenarios

    02:22 LessWrong’s Mission

    05:49 The Rationalist Community and AI

    09:37 What’s Your P(Doom)™

    18:26 What Are Yudkowskians Surprised About?

    26:48 Moral Philosophy vs. Goal Alignment

    36:56 Sandboxing and AI Containment

    42:51 Holding Yudkowskians Accountable

    58:29 Understanding Next Word Prediction

    01:00:02 Pre-Training vs Post-Training

    01:08:06 The Rocket Alignment Problem Analogy

    01:30:09 FOOM vs. Gradual Disempowerment

    01:45:19 Recapping the Mainline Doom Scenario

    01:52:08 Liron’s Outro

    Show Notes

    Jim’s LessWrong — https://www.lesswrong.com/users/jimrandomh

    Jim’s Twitter — https://x.com/jimrandomh

    The Rocket Alignment Problem by Eliezer Yudkowsky — https://www.lesswrong.com/posts/Gg9a4y8reWKtLe3Tn/the-rocket-alignment-problem

    Optimality is the Tiger and Agents Are Its Teeth — https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth

    Doom Debates episode about the research paper discovering AI's utility function — https://lironshapira.substack.com/p/cais-researchers-discover-ais-preferences

    Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online

    Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

    PauseAI, the volunteer organization I’m part of: https://pauseai.info

    Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

    Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

    Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates



    Get full access to Doom Debates at lironshapira.substack.com/subscribe
    1 hr 54 min
  • AI Could Give Humans MORE Control — Ozzie Gooen

    Ozzie Gooen is the founder of the Quantified Uncertainty Research Institute (QURI), a nonprofit building software tools for forecasting and policy analysis. I’ve known him through the rationality community since 2008 and we have a lot in common.

    00:00 Introducing Ozzie

    02:18 The Rationality Community

    06:32 What’s Your P(Doom)™

    08:09 High-Quality Discourse and Social Media

    14:17 Guesstimate and Squiggle Demos

    31:57 Prediction Markets and Rationality

    38:33 Metaforecast Demo

    41:23 Evaluating Everything with LLMs

    47:00 Effective Altruism and FTX Scandal

    56:00 The Repugnant Conclusion Debate

    01:02:25 AI for Governance and Policy

    01:12:07 PauseAI Policy Debate

    01:30:10 Status Quo Bias

    01:33:31 Decaf Coffee and Caffeine Powder

    01:34:45 Are You Aspie?

    01:37:45 Billionaires in Effective Altruism

    01:48:06 Gradual Disempowerment by AI

    01:55:36 LessOnline Conference

    01:57:34 Supporting Ozzie’s Work

    Show Notes

    Quantified Uncertainty Research Institute (QURI) — https://quantifieduncertainty.org

    Ozzie’s Facebook — https://www.facebook.com/ozzie.gooen

    Ozzie’s Twitter — https://x.com/ozziegooen

    Guesstimate, a spreadsheet for working with probability ranges — https://www.getguesstimate.com

    Squiggle, a programming language for building Monte Carlo simulations — https://www.squiggle-language.com

    Metaforecast, a prediction market aggregator — https://metaforecast.org

    Open Annotate, AI-powered content analysis — https://github.com/quantified-uncertainty/open-annotate/

    Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online

    Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

    PauseAI, the volunteer organization I’m part of: https://pauseai.info

    Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

    Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

    Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates



    Get full access to Doom Debates at lironshapira.substack.com/subscribe
    2 hr
  • Top AI Professor Has 85% P(Doom) — David Duvenaud, Fmr. Anthropic Safety Team Lead

    David Duvenaud is a professor of Computer Science at the University of Toronto, co-director of the Schwartz Reisman Institute for Technology and Society, former Alignment Evals Team Lead at Anthropic, an award-winning machine learning researcher, and a close collaborator of Dr. Geoffrey Hinton. He recently co-authored Gradual Disempowerment.

    We dive into David’s impressive career, his high P(Doom), his recent tenure at Anthropic, his views on gradual disempowerment, and the critical need for improved governance and coordination on a global scale.

    00:00 Introducing David

    03:03 Joining Anthropic and AI Safety Concerns

    35:58 David’s Background and Early Influences

    45:11 AI Safety and Alignment Challenges

    54:08 What’s Your P(Doom)™

    01:06:44 Balancing Productivity and Family Life

    01:10:26 The Hamming Question: Are You Working on the Most Important Problem?

    01:16:28 The PauseAI Movement

    01:20:28 Public Discourse on AI Doom

    01:24:49 Courageous Voices in AI Safety

    01:43:54 Coordination and Government Role in AI

    01:47:41 Cowardice in AI Leadership

    02:00:05 Economic and Existential Doom

    02:06:12 Liron’s Post-Show

    Show Notes

    David’s Twitter — https://x.com/DavidDuvenaud

    Schwartz Reisman Institute for Technology and Society — https://srinstitute.utoronto.ca/

    Jürgen Schmidhuber’s Home Page — https://people.idsia.ch/~juergen/

    Ryan Greenblatt's LessWrong comment about a future scenario where there's a one-time renegotiation of power and heat from superintelligent AI projects causes the oceans to boil: https://www.lesswrong.com/posts/pZhEQieM9otKXhxmd/gradual-disempowerment-systemic-existential-risks-from?commentId=T7KZGGqq2Z4gXZsty

    Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

    PauseAI, the volunteer organization I’m part of: https://pauseai.info

    Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

    Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

    Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates



    Get full access to Doom Debates at lironshapira.substack.com/subscribe
    2 hr 8 min

About Doom Debates!

From the publisher's feed

It's time to talk about the end of the world. With your host, Liron Shapira.

Best of Doom Debates!

Ranked by our users in the last 21 days

More shows like Doom Debates!

Making Sense with Sam Harris by Sam Harris

Making Sense with Sam Harris

26,261 Listeners

EconTalk by Russ Roberts

EconTalk

4,271 Listeners

Conversations with Tyler by Mercatus Center at George Mason University

Conversations with Tyler

2,447 Listeners

The Glenn Show by Glenn Loury

The Glenn Show

2,275 Listeners

The Comedy Cellar: Live from the Table by Comedy Cellar Network

The Comedy Cellar: Live from the Table

374 Listeners

Your Undivided Attention by The Center for Humane Technology, Tristan Harris, Aza Raskin

Your Undivided Attention

1,618 Listeners

Conversations with Coleman by The Free Press

Conversations with Coleman

616 Listeners

Machine Learning Street Talk (MLST) by Machine Learning Street Talk (MLST)

Machine Learning Street Talk (MLST)

96 Listeners

Dwarkesh Podcast by Dwarkesh Patel

Dwarkesh Podcast

557 Listeners

Call Me Back - with Dan Senor by Ark Media

Call Me Back - with Dan Senor

3,200 Listeners

Honestly with Bari Weiss by The Free Press

Honestly with Bari Weiss

8,461 Listeners

"Econ 102" with Noah Smith and Erik Torenberg by Turpentine

"Econ 102" with Noah Smith and Erik Torenberg

145 Listeners

For Humanity: An AI Risk Podcast by The AI Risk Network

For Humanity: An AI Risk Podcast

10 Listeners

Ask Haviv Anything by Haviv Rettig Gur

Ask Haviv Anything

908 Listeners

The Last Invention by Longview

The Last Invention

1,148 Listeners