Doom Debates!

AI 2027, a bombshell new paper by the AI Futures Project, is a highly plausible scenario of the next few years of AI progress. I like this paper so much that I made a whole episode about it.

00:00 Overview of AI 2027

05:13 2025: Stumbling Agents

16:23 2026: Advanced Agents

21:49 2027: The Intelligence Explosion

29:13 AI's Initial Exploits and OpenBrain's Secrecy

30:41 Agent-3 and the Rise of Superhuman Engineering

37:05 The Creation and Deception of Agent-5

44:56 The Race Scenario: Humanity's Downfall

48:58 The Slowdown Scenario: A Glimmer of Hope

53:49 Final Thoughts

Show Notes

The website: https://ai-2027.com

Scott Alexander’s blog: https://astralcodexten.com

Daniel Kokotajlo’s previous predictions from 2021 about 2026: https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Dr. Peter Berezin is the Chief Global Strategist and Director of Research at BCA Research, the largest Canadian investment research firm. He’s known for his macroeconomics research reports and his frequent appearances on Bloomberg and CNBC.

Notably, Peter is one of the only macroeconomists in the world who’s forecasting AI doom! He recently published a research report estimating a “ more than 50/50 chance AI will wipe out all of humanity by the middle of the century”.

00:00 Introducing Peter Berezin

01:59 Peter’s Economic Predictions and Track Record

05:50 Investment Strategies and Beating the Market

17:47 The Future of Human Employment

26:40 Existential Risks and the Doomsday Argument

34:13 What’s Your P(Doom)™

39:18 Probability of non-AI Doom

44:19 Solving Population Decline

50:53 Constraining AI Development

53:40 The Multiverse and Its Implications

01:01:11 Are Other Economists Crazy?

01:09:19 Mathematical Universe and Multiverse Theories

01:19:43 Epistemic vs. Physical Probability

01:33:19 Reality Fluid

01:39:11 AI and Moral Realism

01:54:18 The Simulation Hypothesis and God

02:10:06 Liron’s Post-Show

Show Notes

Peter’s Twitter: https://x.com/PeterBerezinBCA

Peter’s old blog — https://stockcoach.blogspot.com

Peter’s 2021 BCA Research Report: “Life, Death and Finance in the Cosmic Multiverse” — https://www.bcaresearch.com/public/content/GIS_SR_2021_12_21.pdf

M.C. Escher’s “Circle Limit IV” — https://www.escherinhetpaleis.nl/escher-today/circle-limit-iv-heaven-and-hell/

Zvi Mowshowitz’s Blog (Liron’s recommendation for best AI news & analysis) — https://thezvi.substack.com

My Doom Debates episode about why nuclear proliferation is bad — https://www.youtube.com/watch?v=ueB9iRQsvQ8

Robin Hanson’s “Mangled Worlds” paper — https://mason.gmu.edu/~rhanson/mangledworlds.html

Uncontrollable by Darren McKee (Liron’s recommended AI x-risk book) — https://www.amazon.com/dp/B0CNNYKVH1

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Nathan Labenz, host of The Cognitive Revolution, joins me for an AI news & social media roundup!

00:00 Introducing Nate

05:18 What’s Your P(Doom)™

23:22 GPT-4o Image Generation

40:20 Will Fiverr’s Stock Crash?

47:41 AI Unemployment

55:11 Entrepreneurship

01:00:40 OpenAI Valuation

01:09:29 Connor Leahy’s Hair

01:13:28 Mass Extinction

01:25:30 Is anyone feeling the doom vibes?

01:38:20 Rethinking AI Individuality

01:40:35 “Softmax” — Emmett Shear's New AI Safety Org

01:57:04 Anthropic's Mechanistic Interpretability Paper

02:10:11 International Cooperation for AI Safety

02:18:43 Final Thoughts

Show Notes

Nate’s Twitter: https://x.com/labenz

Nate’s podcast: https://cognitiverevolution.ai and https://youtube.com/@CognitiveRevolutionPodcast

Nate’s company: https://waymark.com/

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

In this special cross-posted episode of Doom Debates, originally posted here on The Human Podcast, we cover a wide range of topics including the definition of “doom”, P(Doom), various existential risks like pandemics and nuclear threats, and the comparison of rogue AI risks versus AI misuse risks.

00:00 Introduction

01:47 Defining Doom and AI Risks

05:53 P(Doom)

10:04 Doom Debates’ Mission

16:17 Personal Reflections and Life Choices

24:57 The Importance of Debate

27:07 Personal Reflections on AI Doom

30:46 Comparing AI Doom to Other Existential Risks

33:42 Strategies to Mitigate AI Risks

39:31 The Global AI Race and Game Theory

43:06 Philosophical Reflections on a Good Life

45:21 Final Thoughts

Show Notes

The Human Podcast with Joe Murray: https://www.youtube.com/@thehumanpodcastofficial

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Don’t miss the other great AI doom show, For Humanity: https://youtube.com/@ForHumanityAIRisk

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Alexander Campbell claims that having superhuman intelligence doesn’t necessarily translate into having vast power, and that Gödel's Incompleteness Theorem ensures AI can’t get too powerful. I strongly disagree.

Alex has a Master's of Philosophy in Economics from the University of Oxford and an MBA from the Stanford Graduate School of Business, has worked as a quant trader at Lehman Brothers and Bridgewater Associates, and is the founder of Rose AI, a cloud data platform that leverages generative AI to help visualize data.

This debate was recorded in August 2023.

00:00 Intro and Alex’s Background

05:29 Alex's Views on AI and Technology

06:45 Alex’s Non-Doomer Position

11:20 Goal-to-Action Mapping

15:20 Outcome Pump Thought Experiment

21:07 Liron’s Doom Argument

29:10 The Dangers of Goal-to-Action Mappers

34:39 The China Argument and Existential Risks

45:18 Ideological Turing Test

48:38 Final Thoughts

Show Notes

Alexander Campbell’s Twitter: https://x.com/abcampbell

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Roko Mijic has been an active member of the LessWrong and AI safety community since 2008. He’s best known for “Roko’s Basilisk”, a thought experiment he posted on LessWrong that made Eliezer Yudkowsky freak out, and years later became the topic that helped Elon Musk get interested in Grimes.

His view on AI doom is that:* AI alignment is an easy problem* But the chaos and fighting from building superintelligence poses a high near-term existential risk* But humanity’s course without AI has an even higher near-term existential risk

While my own view is very different, I’m interested to learn more about Roko’s views and nail down our cruxes of disagreement.

00:00 Introducing Roko

03:33 Realizing that AI is the only thing that matters

06:51 Cyc: AI with “common sense”

15:15 Is alignment easy?

21:19 What’s Your P(Doom)™

25:14 Why civilization is doomed anyway

37:07 Roko’s AI nightmare scenario

47:00 AI risk mitigation

52:07 Market Incentives and AI Safety

57:13 Are RL and GANs good enough for superalignment?

01:00:54 If humans learned to be honest, why can’t AIs?

01:10:29 Is our test environment sufficiently similar to production?

01:23:56 AGI Timelines

01:26:35 Headroom above human intelligence

01:42:22 Roko’s Basilisk

01:54:01 Post-Debate Monologue

Show Notes

Roko’s Twitter: https://x.com/RokoMijic

Explanation of Roko’s Basilisk on LessWrong: https://www.lesswrong.com/w/rokos-basilisk

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Sir Roger Penrose is a mathematician, mathematical physicist, philosopher of science, and Nobel Laureate in Physics.

His famous body of work includes Penrose diagrams, twistor theory, Penrose tilings, and the incredibly bold claim that intelligence and consciousness are uncomputable physical phenomena related to quantum wave function collapse.

 Dr. Penrose is such a genius that it's just interesting to unpack his worldview, even if it’s totally implausible. How can someone like him be so wrong? What exactly is it that he's wrong about? It's interesting to try to see the world through his eyes, before recoiling from how nonsensical it looks.

00:00 Episode Highlights

01:29 Introduction to Roger Penrose

11:56 Uncomputability

16:52 Penrose on Gödel's Incompleteness Theorem

19:57 Liron Explains Gödel's Incompleteness Theorem

27:05 Why Penrose Gets Gödel Wrong

40:53 Scott Aaronson's Gödel CAPTCHA

46:28 Penrose's Critique of the Turing Test

48:01 Searle's Chinese Room Argument

52:07 Penrose's Views on AI and Consciousness

57:47 AI's Computational Power vs. Human Intelligence

01:21:08 Penrose's Perspective on AI Risk

01:22:20 Consciousness = Quantum Wave Function Collapse?

01:26:25 Final Thoughts

Show Notes

Source video — Feb 22, 2025 Interview with Roger Penrose on “This Is World” — https://www.youtube.com/watch?v=biUfMZ2dts8

Scott Aaronson’s “Gödel CAPTCHA” — https://www.scottaaronson.com/writings/captcha.html

My recent Scott Aaronson episode — https://www.youtube.com/watch?v=xsGqWeqKjEg

My explanation of what’s wrong with arguing “by definition” — https://www.youtube.com/watch?v=ueam4fq8k8I

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

The Center for AI Safety just dropped a fascinating paper — they discovered that today’s AIs like GPT-4 and Claude have preferences! As in, coherent utility functions. We knew this was inevitable, but we didn’t know it was already happening.

This episode has two parts:

In Part I (48 minutes), I react to David Shapiro’s coverage of the paper and push back on many of his points.

In Part II (60 minutes), I explain the paper myself.

00:00 Episode Introduction

05:25 PART I: REACTING TO DAVID SHAPIRO

10:06 Critique of David Shapiro's Analysis

19:19 Reproducing the Experiment

35:50 David's Definition of Coherence

37:14 Does AI have “Temporal Urgency”?

40:32 Universal Values and AI Alignment

49:13 PART II: EXPLAINING THE PAPER

51:37 How The Experiment Works

01:11:33 Instrumental Values and Coherence in AI

01:13:04 Exchange Rates and AI Biases

01:17:10 Temporal Discounting in AI Models

01:19:55 Power Seeking, Fitness Maximization, and Corrigibility

01:20:20 Utility Control and Bias Mitigation

01:21:17 Implicit Association Test

01:28:01 Emailing with the Paper’s Authors

01:43:23 My Takeaway

Show Notes

David’s source video: https://www.youtube.com/watch?v=XGu6ejtRz-0

The research paper: http://emergent-values.ai

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at

https://doomdebates.com

and to https://youtube.com/@DoomDebates

My friend Gil Mark, who leads generative AI products at LinkedIn, thinks competition among superintelligent AIs will lead to a good outcome for humanity. In his view, the alignment problem becomes significantly easier if we build multiple AIs at the same time and let them compete.

I completely disagree, but I hope you’ll find this to be a thought-provoking episode that sheds light on why the alignment problem is so hard.

00:00 Introduction

02:36 Gil & Liron’s Early Doom Days

04:58: AIs : Humans :: Humans : Ants

08:02 The Convergence of AI Goals

15:19 What’s Your P(Doom)™

19:23 Multiple AIs and Human Welfare

24:42 Gil’s Alignment Claim

42:31 Cheaters and Frankensteins

55:55 Superintelligent Game Theory

01:01:16 Slower Takeoff via Resource Competition

01:07:57 Recapping the Disagreement

01:15:39 Post-Debate Banter

Show Notes

Gil’s LinkedIn: https://www.linkedin.com/in/gilmark/

Gil’s Twitter: https://x.com/gmfromgm

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Why does the simplest AI imaginable, when you ask it to help you push a box around a grid, suddenly want you to die?

AI doomers are often misconstrued as having "no evidence" or just "anthropomorphizing". This toy model will help you understand why a drive to eliminate humans is NOT a handwavy anthropomorphic speculation, but rather something we expect by default from any sufficiently powerful search algorithm.

We’re not talking about AGI or ASI here — we’re just looking at an AI that does brute-force search over actions in a simple grid world.

The slide deck I’m presenting was created by Jaan Tallinn, cofounder of the Future of Life Institute.

00:00 Introduction

01:24 The Toy Model

06:19 Misalignment and Manipulation Drives

12:57 Search Capacity and Ontological Insights

16:33 Irrelevant Concepts in AI Control

20:14 Approaches to Solving AI Control Problems

23:38 Final Thoughts

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates