Doom Debates!

Today I’m reacting to a July 2024 interview that Prof. Subbarao Kambhampati did on Machine Learning Street Talk.

Rao is a Professor of Computer Science at Arizona State University, and one of the foremost voices making the claim that while LLMs can generate creative ideas, they can’t truly reason.

The episode covers a range of topics including planning, creativity, the limits of LLMs, and why Rao thinks LLMs are essentially advanced N-gram models.

00:00 Introduction

02:54 Essentially N-Gram Models?

10:31 The Manhole Cover Question

20:54 Reasoning vs. Approximate Retrieval

47:03 Explaining Jokes

53:21 Caesar Cipher Performance

01:10:44 Creativity vs. Reasoning

01:33:37 Reasoning By Analogy

01:48:49 Synthetic Data

01:53:54 The ARC Challenge

02:11:47 Correctness vs. Style

02:17:55 AIs Becoming More Robust

02:20:11 Block Stacking Problems

02:48:12 PlanBench and Future Predictions

02:58:59 Final Thoughts

Show Notes

Rao’s interview on Machine Learning Street Talk: https://www.youtube.com/watch?v=y1WnHpedi2A

Rao’s Twitter: https://x.com/rao2z

PauseAI Website: https://pauseai.info

PauseAI Discord: https://discord.gg/2XXWXvErfA

Watch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

In this episode of Doom Debates, I discuss AI existential risks with my pseudonymous guest Nethys.

Nethy shares his journey into AI risk awareness, influenced heavily by LessWrong and Eliezer Yudkowsky. We explore the vulnerability of society to emerging technologies, the challenges of AI alignment, and why he believes our current approaches are insufficient, ultimately resulting in 99.999% P(Doom).

00:00 Nethys Introduction

04:47 The Vulnerable World Hypothesis

10:01 What’s Your P(Doom)™

14:04 Nethys’s Banger YouTube Comment

26:53 Living with High P(Doom)

31:06 Losing Access to Distant Stars

36:51 Defining AGI

39:09 The Convergence of AI Models

47:32 The Role of “Unlicensed” Thinkers

52:07 The PauseAI Movement

58:20 Lethal Intelligence Video Clip

Show Notes

Eliezer Yudkowsky’s post on “Death with Dignity”: https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy

PauseAI Website: https://pauseai.info

PauseAI Discord: https://discord.gg/2XXWXvErfA

Watch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Fraser Cain is the publisher of Universe Today, co-host of Astronomy Cast, a popular YouTuber about all things space, and guess what… he has a high P(doom)! That’s why he’s joining me on Doom Debates for a very special AI + space crossover episode.

00:00 Fraser Cain’s Background and Interests

5:03 What’s Your P(Doom)™

07:05 Our Vulnerable World

15:11 Don’t Look Up

22:18 Cosmology and the Search for Alien Life

31:33 Stars = Terrorists

39:03 The Great Filter and the Fermi Paradox

55:12 Grabby Aliens Hypothesis

01:19:40 Life Around Red Dwarf Stars?

01:22:23 Epistemology of Grabby Aliens

01:29:04 Multiverses

01:33:51 Quantum Many Worlds vs. Copenhagen Interpretation

01:47:25 Simulation Hypothesis

01:51:25 Final Thoughts

SHOW NOTES

Fraser’s YouTube channel: https://www.youtube.com/@frasercain

Universe Today (space and astronomy news): https://www.universetoday.com/

Max Tegmark’s book that explains 4 levels of multiverses: https://www.amazon.com/Our-Mathematical-Universe-Ultimate-Reality/dp/0307744256

Robin Hanson’s ideas:

Grabby Aliens: https://grabbyaliens.com

The Great Filter: https://en.wikipedia.org/wiki/Great_Filter

Life in a high-dimensional space: https://www.overcomingbias.com/p/life-in-1kdhtml

---

Watch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Vaden Masrani and Ben Chugg, hosts of the Increments Podcast, are back for a Part II! This time we’re going straight to debating my favorite topic, AI doom.

00:00 Introduction

02:23 High-Level AI Doom Argument

17:06 How Powerful Could Intelligence Be?

22:34 “Knowledge Creation”

48:33 “Creativity”

54:57 Stand-Up Comedy as a Test for AI

01:12:53 Vaden & Ben’s Goalposts

01:15:00 How to Change Liron’s Mind

01:20:02 LLMs are Stochastic Parrots?

01:34:06 Tools vs. Agents

01:39:51 Instrumental Convergence and AI Goals

01:45:51 Intelligence vs. Morality

01:53:57 Mainline Futures

02:16:50 Lethal Intelligence Video

Show Notes

Vaden & Ben’s Podcast: https://www.youtube.com/@incrementspod

Recommended playlists from their podcast:

* The Bayesian vs Popperian Epistemology Series

* The Conjectures and Refutations Series

Vaden’s Twitter: https://x.com/vadenmasrani

Ben’s Twitter: https://x.com/BennyChugg

Watch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Dr. Andrew Critch is the co-founder of the Center for Applied Rationality, a former Research Fellow at the Machine Intelligence Research Institute (MIRI), a Research Scientist at the UC Berkeley Center for Human Compatible AI, and the co-founder of a new startup called Healthcare Agents.

Dr. Critch’s P(Doom) is a whopping 85%! But his most likely doom scenario isn’t what you might expect. He thinks humanity will successfully avoid a self-improving superintelligent doom scenario, only to still go extinct via the slower process of “industrial dehumanization”.

00:00 Introduction01:43 Dr. Critch’s Perspective on LessWrong Sequences06:45 Bayesian Epistemology15:34 Dr. Critch's Time at MIRI18:33 What’s Your P(Doom)™26:35 Doom Scenarios40:38 AI Timelines43:09 Defining “AGI”48:27 Superintelligence53:04 The Speed Limit of Intelligence01:12:03 The Obedience Problem in AI01:21:22 Artificial Superintelligence and Human Extinction01:24:36 Global AI Race and Geopolitics01:34:28 Future Scenarios and Human Relevance01:48:13 Extinction by Industrial Dehumanization01:58:50 Automated Factories and Human Control02:02:35 Global Coordination Challenges02:27:00 Healthcare Agents02:35:30 Final Thoughts

---

Show Notes

Dr. Critch’s LessWrong post explaining his P(Doom) and most likely doom scenarios: https://www.lesswrong.com/posts/Kobbt3nQgv3yn29pr/my-motivation-and-theory-of-change-for-working-in-ai

Dr. Critch’s Website: https://acritch.com/

Dr. Critch’s Twitter: https://twitter.com/AndrewCritchPhD

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

It’s time for AI Twitter Beefs #2:

00:42 Jack Clark (Anthropic) vs. Holly Elmore (PauseAI US)

11:02 Beff Jezos vs. Eliezer Yudkowsky, Carl Feynman

18:10 Geoffrey Hinton vs. OpenAI & Meta

25:14 Samuel Hammond vs. Liron

30:26 Yann LeCun vs. Eliezer Yudkowsky

37:13 Roon vs. Eliezer Yudkowsky

41:37 Tyler Cowen vs. AI Doomers

52:54 David Deutsch vs. Liron

Twitter people referenced:

* Jack Clark: https://x.com/jackclarkSF

* Holly Elmore: https://x.com/ilex_ulmus

* PauseAI US: https://x.com/PauseAIUS

* Geoffrey Hinton: https://x.com/GeoffreyHinton

* Samuel Hammond: https://x.com/hamandcheese

* Yann LeCun: https://x.com/ylecun

* Eliezer Yudkowsky: https://x.com/esyudkowsky

* Roon: https://x.com/tszzl

* Beff Jezos: https://x.com/basedbeffjezos

* Carl Feynman: https://x.com/carl_feynman

* Tyler Cowen: https://x.com/tylercowen

* David Deutsch: https://x.com/DavidDeutschOxf

Show Notes

Holly Elmore’s EA forum post about scouts vs. soldiers

Manifund info & donation page for PauseAI US: https://manifund.org/projects/pauseai-us-2025-through-q2

PauseAI.info - join the Discord and find me in the #doom-debates channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Vaden Masrani and Ben Chugg, hosts of the Increments Podcast, are joining me to debate Bayesian vs. Popperian epistemology.

I’m on the Bayesian side, heavily influenced by the writings of Eliezer Yudkowsky. Vaden and Ben are on the Popperian side, heavily influenced by David Deutsch and the writings of Popper himself.

We dive into the theoretical underpinnings of Bayesian reasoning and Solomonoff induction, contrasting them with the Popperian perspective, and explore real-world applications such as predicting elections and economic policy outcomes.

The debate highlights key philosophical differences between our two epistemological frameworks, and sets the stage for further discussions on superintelligence and AI doom scenarios in an upcoming Part II.

00:00 Introducing Vaden and Ben

02:51 Setting the Stage: Epistemology and AI Doom

04:50 What’s Your P(Doom)™

13:29 Popperian vs. Bayesian Epistemology

31:09 Engineering and Hypotheses

38:01 Solomonoff Induction

45:21 Analogy to Mathematical Proofs

48:42 Popperian Reasoning and Explanations

54:35 Arguments Against Bayesianism

58:33 Against Probability Assignments

01:21:49 Popper’s Definition of “Content”

01:31:22 Heliocentric Theory Example

01:31:34 “Hard to Vary” Explanations

01:44:42 Coin Flipping Example

01:57:37 Expected Value

02:12:14 Prediction Market Calibration

02:19:07 Futarchy

02:29:14 Prediction Markets as AI Lower Bound

02:39:07 A Test for Prediction Markets

2:45:54 Closing Thoughts

Show Notes

Vaden & Ben’s Podcast: https://www.youtube.com/@incrementspod

Vaden’s Twitter: https://x.com/vadenmasrani

Ben’s Twitter: https://x.com/BennyChugg

Bayesian reasoning: https://en.wikipedia.org/wiki/Bayesian_inference

Karl Popper: https://en.wikipedia.org/wiki/Karl_Popper

Vaden's blog post on Cox's Theorem and Yudkowsky's claims of "Laws of Rationality": https://vmasrani.github.io/blog/2021/the_credence_assumption/

Vaden’s disproof of probabilistic induction (including Solomonoff Induction): https://arxiv.org/abs/2107.00749

Vaden’s referenced post about predictions being uncalibrated > 1yr out: https://forum.effectivealtruism.org/posts/hqkyaHLQhzuREcXSX/data-on-forecasting-accuracy-across-different-time-horizons#Calibrations

Article by Gavin Leech and Misha Yagudin on the reliability of forecasters: https://ifp.org/can-policymakers-trust-forecasters/

Sources for claim that superforecasters gave a P(doom) below 1%: https://80000hours.org/2024/09/why-experts-and-forecasters-disagree-about-ai-risk/https://www.astralcodexten.com/p/the-extinction-tournament

Vaden’s Slides on Content vs Probability: https://vmasrani.github.io/assets/pdf/popper_good.pdf

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Our top researchers and industry leaders have been warning us that superintelligent AI may cause human extinction in the next decade.

If you haven't been following all the urgent warnings, I'm here to bring you up to speed.

* Human-level AI is coming soon

* It’s an existential threat to humanity

* The situation calls for urgent action

Listen to this 15-minute intro to get the lay of the land.

Then follow these links to learn more and see how you can help:

* The Compendium

A longer written introduction to AI doom by Connor Leahy et al

* AGI Ruin — A list of lethalities

A comprehensive list by Eliezer Yudkowksy of reasons why developing superintelligent AI is unlikely to go well for humanity

* AISafety.info

A catalogue of AI doom arguments and responses to objections

* PauseAI.info

The largest volunteer org focused on lobbying world government to pause development of superintelligent AI

* PauseAI Discord

Chat with PauseAI members, see a list of projects and get involved

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Prof. Lee Cronin is the Regius Chair of Chemistry at the University of Glasgow. His research aims to understand how life might arise from non-living matter. In 2017, he invented “Assembly Theory” as a way to measure the complexity of molecules and gain insight into the earliest evolution of life.

Today we’re debating Lee's claims about the limits of AI capabilities, and my claims about the risk of extinction from superintelligent AGI.

00:00 Introduction

04:20 Assembly Theory

05:10 Causation and Complexity

10:07 Assembly Theory in Practice

12:23 The Concept of Assembly Index

16:54 Assembly Theory Beyond Molecules

30:13 P(Doom)

32:39 The Statement on AI Risk

42:18 Agency and Intent

47:10 RescueBot’s Intent vs. a Clock’s

53:42 The Future of AI and Human Jobs

57:34 The Limits of AI Creativity

01:04:33 The Complexity of the Human Brain

01:19:31 Superintelligence: Fact or Fiction?

01:29:35 Final Thoughts

Lee’s Wikipedia: https://en.wikipedia.org/wiki/Leroy_Cronin

Lee’s Twitter: https://x.com/leecronin

Lee’s paper on Assembly Theory: https://arxiv.org/abs/2206.02279

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Ben Horowitz, cofounder and General Partner at Andreessen Horowitz (a16z), says nuclear proliferation is good.

I was shocked because I thought we all agreed nuclear proliferation is VERY BAD.

If Ben and a16z can’t appreciate the existential risks of nuclear weapons proliferation, why would anyone ever take them seriously on the topic of AI regulation?

00:00 Introduction

00:49 Ben Horowitz on Nuclear Proliferation

02:12 Ben Horowitz on Open Source AI

05:31 Nuclear Non-Proliferation Treaties

10:25 Escalation Spirals

15:20 Rogue Actors

16:33 Nuclear Accidents

17:19 Safety Mechanism Failures

20:34 The Role of Human Judgment in Nuclear Safety

21:39 The 1983 Soviet Nuclear False Alarm

22:50 a16z’s Disingenuousness

23:46 Martin Casado and Marc Andreessen

24:31 Nuclear Equilibrium

26:52 Why I Care

28:09 Wrap Up

Sources of this episode’s video clips:

Ben Horowitz’s interview on Upstream with Erik Torenberg: https://www.youtube.com/watch?v=oojc96r3Kuo

Martin Casado and Marc Andreessen talking about AI on the a16z Podcast: https://www.youtube.com/watch?v=0wIUK0nsyUg

Roger Skaer’s TikTok: https://www.tiktok.com/@rogerskaer

George W. Bush and John Kerry Presidential Debate (September 30, 2004): https://www.youtube.com/watch?v=WYpP-T0IcyA

Barack Obama’s Prague Remarks on Nuclear Disarmament: https://www.youtube.com/watch?v=QKSn1SXjj2s

John Kerry’s Remarks at the 2015 Nuclear Nonproliferation Treaty Review Conference: https://www.youtube.com/watch?v=LsY1AZc1K7w

Show notes:

Nuclear War, A Scenario by Annie Jacobsen: https://www.amazon.com/Nuclear-War-Scenario-Annie-Jacobsen/dp/0593476093

Dr. Strangelove or: How I learned to Stop Worrying and Love the Bomb: https://en.wikipedia.org/wiki/Dr._Strangelove

1961 Goldsboro B-52 Crash: https://en.wikipedia.org/wiki/1961_Goldsboro_B-52_crash

1983 Soviet Nuclera False Alarm Incident: https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alarm_incident

List of military nuclear accidents: https://en.wikipedia.org/wiki/List_of_military_nuclear_accidents

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates.