Doom Debates!

Bryan Cantrill, co-founder of Oxide Computer, says in his talk that engineering in the physical world is too complex for any AI to do it better than teams of human engineers. Success isn’t about intelligence; it’s about teamwork, character and resilience.

I completely disagree.

00:00 Introduction

02:03 Bryan’s Take on AI Doom

05:55 The Concept of P(Doom)

08:36 Engineering Challenges and Human Intelligence

15:09 The Role of Regulation and Authoritarianism in AI Control

29:44 Engineering Complexity: A Case Study from Oxide Computer

40:06 The Value of Team Collaboration

46:13 Human Attributes in Engineering

49:33 AI's Potential in Engineering

58:23 Existential Risks and AI Predictions

Bryan’s original talk:

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/watch?v=9CUFbqh16Fg

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

Thanks for everyone who participated in the live Q&A on Friday!

The topics covered include advice for computer science students, working in AI trustworthiness, what good AI regulation looks like, the implications of the $500B Stargate project, the public's gradual understanding of AI risks, the impact of minor AI disasters, and the philosophy of consciousness.

00:00 Advice for Comp Sci Students

01:14 The $500B Stargate Project

02:36 Eliezer's Recent Podcast

03:07 AI Safety and Public Policy

04:28 AI Disruption and Politics

05:12 DeepSeek and AI Advancements

06:54 Human vs. AI Intelligence

14:00 Consciousness and AI

24:34 Dark Forest Theory and AI

35:31 Investing in Yourself

42:42 Probability of Aliens Saving Us from AI

43:31 Brain-Computer Interfaces and AI Safety

46:19 Debating AI Safety and Human Intelligence

48:50 Nefarious AI Activities and Satellite Surveillance

49:31 Pliny the Prompter Jailbreaking AI

50:20 Can’t vs. Won’t Destroy the World

51:15 How to Make AI Risk Feel Present

54:27 Keeping Doom Arguments On Track

57:04 Game Theory and AI Development Race

01:01:26 Mental Model of Average Non-Doomer

01:04:58 Is Liron a Strict Bayesian and Utilitarian?

01:09:48 Can We Rename “Doom Debates”

01:12:34 The Role of AI Trustworthiness

01:16:48 Minor AI Disasters

01:18:07 Most Likely Reason Things Go Well

01:21:00 Final Thoughts

Show Notes

Previous post where people submitted questions: https://lironshapira.substack.com/p/2500-subscribers-live-q-and-a-ask

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

It’s time for AI Twitter Beefs #3:

00:00 Introduction

01:27 Marc Andreessen vs. Sam Altman

09:15 Mark Zuckerberg

35:40 Martin Casado

47:26 Gary Marcus vs. Miles Brundage Bet

58:39 Scott Alexander’s AI Art Turing Test

01:11:29 Roon

01:16:35 Stephen McAleer

01:22:25 Emmett Shear

01:37:20 OpenAI’s “Safety”

01:44:09 Naval Ravikant vs. Eliezer Yudkowsky

01:56:03 Comic Relief

01:58:53 Final Thoughts

Show Notes

Upcoming Live Q&A: https://lironshapira.substack.com/p/2500-subscribers-live-q-and-a-ask

“Make Your Beliefs Pay Rent In Anticipated Experiences” by Eliezer Yudkowsky on LessWrong: https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences

Scott Alexander’s AI Art Turing Test: https://www.astralcodexten.com/p/how-did-you-do-on-the-ai-art-turing

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

Effective Altruism has been a controversial topic on social media, so today my guest and I are going to settle the question once and for all: Is it good or bad?

Jonas Sota is a Software Engineer at Rippling, BA in Philosophy from UC Berkeley, who’s been observing the Effective Altruism (EA) movement in the San Francisco Bay Area for over a decade… and he’s not a fan.

00:00 Introduction

01:22 Jonas’s Criticisms of EA

03:23 Recoil Exaggeration

05:53 Impact of Malaria Nets

10:48 Local vs. Global Altruism

13:02 Shrimp Welfare

25:14 Capitalism vs. Charity

33:37 Cultural Sensitivity

34:43 The Impact of Direct Cash Transfers

37:23 Long-Term Solutions vs. Immediate Aid

42:21 Charity Budgets

45:47 Prioritizing Local Issues

50:55 The EA Community

59:34 Debate Recap

1:03:57 Announcements

Show Notes

Jonas’s Instagram: @jonas_wanders

Will MacAskill’s famous book, Doing Good Better: https://www.effectivealtruism.org/doing-good-better

Scott Alexander’s excellent post about the people he met at EA Global: https://slatestarcodex.com/2017/08/16/fear-and-loathing-at-effective-altruism-global-2017/

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

Matthew Adelstein, better known as Bentham’s Bulldog on Substack, is a philosophy major at the University of Michigan and an up & coming public intellectual.

He’s a rare combination: Effective Altruist, Bayesian, non-reductionist, theist.

Our debate covers reductionism, evidence for god, the implications of a fine-tuned universe, moral realism, and AI doom.

00:00 Introduction

02:56 Matthew’s Research

11:29 Animal Welfare

16:04 Reductionism vs. Non-Reductionism Debate

39:53 The Decline of God in Modern Discourse

46:23 Religious Credences

50:24 Pascal's Wager and Christianity

56:13 Are Miracles Real?

01:10:37 Fine-Tuning Argument for God

01:28:36 Cellular Automata

01:34:25 Anthropic Principle

01:51:40 Mathematical Structures and Probability

02:09:35 Defining God

02:18:20 Moral Realism

02:21:40 Orthogonality Thesis

02:32:02 Moral Philosophy vs. Science

02:45:51 Moral Intuitions

02:53:18 AI and Moral Philosophy

03:08:50 Debate Recap

03:12:20 Show Updates

Show Notes

Matthew’s Substack: https://benthams.substack.com

Matthew's Twitter: https://x.com/BenthamsBulldog

Matthew's YouTube: https://www.youtube.com/@deliberationunderidealcond5105

Lethal Intelligence Guide, the ultimate animated video introduction to AI x-risk – https://www.youtube.com/watch?v=9CUFbqh16Fg

PauseAI, the volunteer organization I’m part of — https://pauseai.info/

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

Prof. Kenneth Stanley is a former Research Science Manager at OpenAI leading the Open-Endedness Team in 2020-2022. Before that, he was a Professor of Computer Ccience at the University of Central Florida and the head of Core AI Research at Uber. He coauthored Why Greatness Cannot Be Planned: The Myth of the Objective, which argues that as soon as you create an objective, then you ruin your ability to reach it.

In this episode, I debate Ken’s claim that superintelligent AI *won’t* be guided by goals, and then we compare our views on AI doom.

00:00 Introduction

00:45 Ken’s Role at OpenAI

01:53 “Open-Endedness” and “Divergence”

9:32 Open-Endedness of Evolution

21:16 Human Innovation and Tech Trees

36:03 Objectives vs. Open Endedness

47:14 The Concept of Optimization Processes

57:22 What’s Your P(Doom)™

01:11:01 Interestingness and the Future

01:20:14 Human Intelligence vs. Superintelligence

01:37:51 Instrumental Convergence

01:55:58 Mitigating AI Risks

02:04:02 The Role of Institutional Checks

02:13:05 Exploring AI's Curiosity and Human Survival

02:20:51 Recapping the Debate

02:29:45 Final Thoughts

SHOW NOTES

Ken’s home page: https://www.kenstanley.net/

Ken’s Wikipedia: https://en.wikipedia.org/wiki/Kenneth_Stanley

Ken’s Twitter: https://x.com/kenneth0stanley

Ken’s PicBreeder paper: https://wiki.santafe.edu/images/1/1e/Secretan_ecj11.pdf

Ken's book, Why Greatness Cannot Be Planned: The Myth of the Objective: https://www.amazon.com/Why-Greatness-Cannot-Planned-Objective/dp/3319155237

The Rocket Alignment Problem by Eliezer Yudkowsky: https://intelligence.org/2018/10/03/rocket-alignment/

---

Lethal Intelligence Guide, the ultimate animated video introduction to AI x-risk – https://www.youtube.com/watch?v=9CUFbqh16Fg

PauseAI, the volunteer organization I’m part of — https://pauseai.info/

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

OpenAI just announced o3 and smashed a bunch of benchmarks (ARC-AGI, SWE-bench, FrontierMath)!

A new Anthropic and Redwood Research paper says Claude is resisting its developers’ attempts to retrain its values!

What’s the upshot — what does it all mean for P(doom)?

00:00 Introduction

01:45 o3’s architecture and benchmarks

06:08 “Scaling is hitting a wall” 🤡

13:41 How many new architectural insights before AGI?

20:28 Negative update for interpretability

31:30 Intellidynamics — ***KEY CONCEPT***

33:20 Nuclear control rod analogy

36:54 Sam Altman's misguided perspective

42:40 Claude resisted retraining from good to evil

44:22 What is good corrigibility?

52:42 Claude’s incorrigibility doesn’t surprise me

55:00 Putting it all in perspective

---

SHOW NOTES

Scott Alexander’s analysis of the Claude incorrigibility result: https://www.astralcodexten.com/p/claude-fights-back and https://www.astralcodexten.com/p/why-worry-about-incorrigible-claude

Zvi Mowshowitz’s analysis of the Claude incorrigibility result: https://thezvi.wordpress.com/2024/12/24/ais-will-increasingly-fake-alignment/

---

PauseAI Website: https://pauseai.info

PauseAI Discord: https://discord.gg/2XXWXvErfA

Say hi to me in the #doom-debates-podcast channel!

Watch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

This week Liron was interview by Gaëtan Selle on @the-flares about AI doom.

Cross-posted from their channel with permission.

Original source: https://www.youtube.com/watch?v=e4Qi-54I9Zw

0:00:02 Guest Introduction

0:01:41 Effective Altruism and Transhumanism

0:05:38 Bayesian Epistemology and Extinction Probability

0:09:26 Defining Intelligence and Its Dangers

0:12:33 The Key Argument for AI Apocalypse

0:18:51 AI’s Internal Alignment

0:24:56 What Will AI's Real Goal Be?

0:26:50 The Train of Apocalypse

0:31:05 Among Intellectuals, Who Rejects the AI Apocalypse Arguments?

0:38:32 The Shoggoth Meme

0:41:26 Possible Scenarios Leading to Extinction

0:50:01 The Only Solution: A Pause in AI Research?

0:59:15 The Risk of Violence from AI Risk Fundamentalists

1:01:18 What Will General AI Look Like?

1:05:43 Sci-Fi Works About AI

1:09:21 The Rationale Behind Cryonics

1:12:55 What Does a Positive Future Look Like?

1:15:52 Are We Living in a Simulation?

1:18:11 Many Worlds in Quantum Mechanics Interpretation

1:20:25 Ideal Future Podcast Guest for Doom Debates

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Roon is a member of the technical staff at OpenAI. He’s a highly respected voice on tech Twitter, despite being a pseudonymous cartoon avatar account. In late 2021, he invented the terms “shape rotator” and “wordcel” to refer to roughly visual/spatial/mathematical intelligence vs. verbal intelligence. He is simultaneously a serious thinker, a builder, and a shitposter.

 I'm excited to learn more about Roon, his background, his life, and of course, his views about AI and existential risk.

00:00 Introduction

02:43 Roon’s Quest and Philosophies

22:32 AI Creativity

30:42 What’s Your P(Doom)™

54:40 AI Alignment

57:24 Training vs. Production

01:05:37 ASI

01:14:35 Goal-Oriented AI and Instrumental Convergence

01:22:43 Pausing AI

01:25:58 Crux of Disagreement

1:27:55 Dogecoin

01:29:13 Doom Debates’s Mission

Show Notes

Follow Roon: https://x.com/tszzl

For Humanity: An AI Safety Podcast with John Sherman — https://www.youtube.com/@ForHumanityPodcast

Lethal Intelligence Guide, the ultimate animated video introduction to AI x-risk – https://www.youtube.com/watch?v=9CUFbqh16Fg

PauseAI, the volunteer organization I’m part of — https://pauseai.info/

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Today I’m reacting to the recent Scott Aaronson interview on the Win-Win podcast with Liv Boeree and Igor Kurganov.

Prof. Aaronson is the Director of the Quantum Information Center at the University of Texas at Austin. He’s best known for his research advancing the frontier of complexity theory, especially quantum complexity theory, and making complex insights from his field accessible to a wider readership via his blog.

Scott is one of my biggest intellectual influences. His famous Who Can Name The Bigger Number essay and his long-running blog are among my best memories of coming across high-quality intellectual content online as a teen. His posts and lectures taught me much of what I know about complexity theory.

Scott recently completed a two-year stint at OpenAI focusing on the theoretical foundations of AI safety, so I was interested to hear his insider account.

Unfortunately, what I heard in the interview confirms my worst fears about the meaning of “safety” at today’s AI companies: that they’re laughably clueless at how to achieve any measure of safety, but instead of doing the adult thing and slowing down their capabilities work, they’re pushing forward recklessly.

00:00 Introducing Scott Aaronson

02:17 Scott's Recruitment by OpenAI

04:18 Scott's Work on AI Safety at OpenAI

08:10 Challenges in AI Alignment

12:05 Watermarking AI Outputs

15:23 The State of AI Safety Research

22:13 The Intractability of AI Alignment

34:20 Policy Implications and the Call to Pause AI

38:18 Out-of-Distribution Generalization

45:30 Moral Worth Criterion for Humans

51:49 Quantum Mechanics and Human Uniqueness

01:00:31 Quantum No-Cloning Theorem

01:12:40 Scott Is Almost An Accelerationist?

01:18:04 Geoffrey Hinton's Proposal for Analog AI

01:36:13 The AI Arms Race and the Need for Regulation

01:39:41 Scott Aronson's Thoughts on Sam Altman

01:42:58 Scott Rejects the Orthogonality Thesis

01:46:35 Final Thoughts

01:48:48 Lethal Intelligence Clip

01:51:42 Outro

Show Notes

Scott’s Interview on Win-Win with Liv Boeree and Igor Kurganov: https://www.youtube.com/watch?v=ANFnUHcYza0

Scott’s Blog: https://scottaaronson.blog

PauseAI Website: https://pauseai.info

PauseAI Discord: https://discord.gg/2XXWXvErfA

Watch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.