Doom Debates!

Princeton Comp Sci Ph.D. candidate Sayash Kapoor co-authored a blog post last week with his professor Arvind Narayanan called "AI Existential Risk Probabilities Are Too Unreliable To Inform Policy".

While some non-doomers embraced the arguments, I see it as contributing nothing to the discourse besides demonstrating a popular failure mode: a simple misunderstanding of the basics of Bayesian epistemology.

I break down Sayash's recent episode of Machine Learning Street Talk point-by-point to analyze his claims from the perspective of the one true epistemology: Bayesian epistemology.

00:00 Introduction

03:40 Bayesian Reasoning

04:33 Inductive vs. Deductive Probability

05:49 Frequentism vs Bayesianism

16:14 Asteroid Impact and AI Risk Comparison

28:06 Quantification Bias

31:50 The Extinction Prediction Tournament

36:14 Pascal's Wager and AI Risk

40:50 Scaling Laws and AI Progress

45:12 Final Thoughts

My source material is Sayash's episode of Machine Learning Street Talk: https://www.youtube.com/watch?v=BGvQmHd4QPE

I also recommend reading Scott Alexander’s related post: https://www.astralcodexten.com/p/in-continued-defense-of-non-frequentist

Sayash's blogpost that he was being interviewed about is called "AI existential risk probabilities are too unreliable to inform policy": https://www.aisnakeoil.com/p/ai-existential-risk-probabilities

Follow Sayash: https://x.com/sayashk

Martin Casado is a General Partner at Andreessen Horowitz (a16z) who has strong views about AI.

He claims that AI is basically just a buzzword for statistical models and simulations. As a result of this worldview, he only predicts incremental AI progress that doesn’t pose an existential threat to humanity, and he sees AI regulation as a net negative.

I set out to understand his worldview around AI, and pinpoint the crux of disagreement with my own view.

Spoiler: I conclude that Martin needs to go beyond analyzing AI as just statistical models and simulations, and analyze it using the more predictive concept of “intelligence” in the sense of hitting tiny high-value targets in exponentially-large search spaces.

If Martin appreciated that intelligence is a quantifiable property that algorithms have, and that our existing AIs are getting close to surpassing human-level general intelligence, then hopefully he’d come around to raising his P(doom) and appreciating the urgent extinction risk we face.

00:00 Introducing Martin Casado

01:42 Martin’s AGI Timeline

05:39 Martin’s Analysis of Self-Driving Cars

15:30 Heavy-Tail Distributions

38:03 Understanding General Intelligence

38:29 AI's Progress in Specific Domains

43:20 AI’s Understanding of Meaning

47:16 Compression and Intelligence

48:09 Symbol Grounding

53:24 Human Abstractions and AI

01:18:18 The Frontier of AI Applications

01:23:04 Human vs. AI: Concept Creation and Reasoning

01:25:51 The Complexity of the Universe and AI's Limitations

01:28:16 AI's Potential in Biology and Simulation

01:32:40 The Essence of Intelligence and Creativity in AI

01:41:13 AI's Future Capabilities

02:00:29 Intelligence vs. Simulation

02:14:59 AI Regulation

02:23:05 Concluding Thoughts

Watch the original episode of the Cognitive Revolution podcast with Martin and host Nathan Labenz.

Follow Martin: @martin_casado

Follow Nate: @labenz

Follow Liron: @liron

Subscribe to the Doom Debates YouTube Channel to get full episodes plus other bonus content!

Search “Doom Debates” to subscribe in your podcast player.

Tilek Mamutov is a Kyrgyzstani software engineer who worked at Google X for 11 years before founding his own international software engineer recruiting company, Outtalent.

Since first encountering the AI doom argument at a Center for Applied Rationality bootcamp 10 years ago, he considers it a serious possibility, but he doesn’t currently feel convinced that doom is likely.

Let’s explore Tilek’s worldview and pinpoint where he gets off the doom train and why!

00:12 Tilek’s Background

01:43 Life in Kyrgyzstan

04:32 Tilek’s Non-Doomer Position

07:12 Debating AI Doom Scenarios

13:49 Nuclear Weapons and AI Analogies

39:22 Privacy and Empathy in Human-AI Interaction

39:43 AI's Potential in Understanding Human Emotions

41:14 The Debate on AI's Empathy Capabilities

42:23 Quantum Effects and AI's Predictive Models

45:33 The Complexity of AI Control and Safety

47:10 Optimization Power: AI vs. Human Intelligence

48:39 The Risks of AI Self-Replication and Control

51:52 Historical Analogies and AI Safety Concerns

56:35 The Challenge of Embedding Safety in AI Goals

01:02:42 The Future of AI: Control, Optimization, and Risks

01:15:54 The Fragility of Security Systems

01:16:56 Debating AI Optimization and Catastrophic Risks

01:18:34 The Outcome Pump Thought Experiment

01:19:46 Human Persuasion vs. AI Control

01:21:37 The Crux of Disagreement: Robustness of AI Goals

01:28:57 Slow vs. Fast AI Takeoff Scenarios

01:38:54 The Importance of AI Alignment

01:43:05 Conclusion

Follow Tilek

x.com/tilek

Links

I referenced Paul Christiano’s scenario of gradual AI doom, a slower version that doesn’t require a Yudkowskian “foom”. Worth a read: What Failure Looks Like

I also referenced the concept of “edge instantiation” to explain that if you’re optimizing powerfully for some metric, you don’t get other intuitively nice things as a bonus, you *just* get the exact thing your function is measuring.

Dr. Mike Israetel is a well-known bodybuilder and fitness influencer with over 600,000 Instagram followers, and a surprisingly intelligent commentator on other subjects, including a whole recent episode on the AI alignment problem:

Mike brought up many interesting points that were worth responding to, making for an interesting reaction episode. I also appreciate that he’s helping get the urgent topic of AI alignment in front of a mainstream audience.

Unfortunately, Mike doesn’t engage with the possibility that AI alignment is an intractable technical problem on a 5-20 year timeframe, which I think is more likely than not. That’s the crux of why he and I disagree, and why I see most of his episode as talking past most other intelligent positions people take on AI alignment. I hope he’ll keep engaging with the topic and rethink his position.

00:00 Introduction

03:08 AI Risks and Scenarios

06:42 Superintelligence Arms Race

12:39 The Importance of AI Alignment

18:10 Challenges in Defining Human Values

26:11 The Outer and Inner Alignment Problems

44:00 Transhumanism and AI's Potential

45:42 The Next Step In Evolution

47:54 AI Alignment and Potential Catastrophes

50:48 Scenarios of AI Development

54:03 The AI Alignment Problem

01:07:39 AI as a Helper System

01:08:53 Corporations and AI Development

01:10:19 The Risk of Unaligned AI

01:27:18 Building a Superintelligent AI

01:30:57 Conclusion

Follow Mike Israetel:

* instagram.com/drmikeisraetel

* youtube.com/@MikeIsraetelMakingProgress

Get the full Doom Debates experience:

* Subscribe to youtube.com/@DoomDebates

* Subscribe to this Substack: DoomDebates.com

* Search "Doom Debates" to subscribe in your podcast player

* Follow me at x.com/liron

What did we learn from my debate with Robin Hanson? Did we successfully isolate the cruxes of disagreement? I actually think we did!

In this post-debate analysis, we’ll review what those key cruxes are, and why I still think I’m right and Robin is wrong about them!

I’ve taken the time to think much harder about everything Robin said during the debate, so I can give you new & better counterarguments than the ones I was able to make in realtime.

Timestamps

00:00 Debate Reactions

06:08 AI Timelines and Key Metrics

08:30 “Optimization Power” vs. “Innovation”

11:49 Economic Growth and Diffusion

17:56 Predicting Future Trends

24:23 Crux of Disagreement with Robin’s Methodology

34:59 Conjunction Argument for Low P(Doom)

37:26 Headroom Above Human Intelligence

41:13 The Role of Culture in Human Intelligence

48:01 Goal-Completeness and AI Optimization

50:48 Misaligned Foom Scenario

59:29 Monitoring AI and the Rule of Law

01:04:51 How Robin Sees Alignment

01:09:08 Reflecting on the Debate

Links

AISafety.info - The fractal of counterarguments to non-doomers’ arguments

For the full Doom Debates experience:

* Subscribe to youtube.com/@DoomDebates

* Subscribe to this Substack: DoomDebates.com

* Search "Doom Debates" to subscribe in your podcast player

* Follow me at x.com/liron

Robin Hanson is a legend in the rationality community and one of my biggest intellectual influences.

In 2008, he famously debated Eliezer Yudkowsky about AI doom via a sequence of dueling blog posts known as the great Hanson-Yudkowsky Foom Debate. This debate picks up where Hanson-Yudkowsky left off, revisiting key arguments in the light of recent AI advances.

My position is similar to Eliezer's: P(doom) is on the order of 50%. Robin's position is shockingly different: P(doom) is below 1%.

00:00 Announcements

03:18 Debate Begins

05:41 Discussing AI Timelines and Predictions

19:54 Economic Growth and AI Impact

31:40 Outside Views vs. Inside Views on AI

46:22 Predicting Future Economic Growth

51:10 Historical Doubling Times and Future Projections

54:11 Human Brain Size and Economic Metrics

57:20 The Next Era of Innovation

01:07:41 AI and Future Predictions

01:14:24 The Vulnerable World Hypothesis

01:16:27 AI Foom

01:28:15 Genetics and Human Brain Evolution

01:29:24 The Role of Culture in Human Intelligence

01:31:36 Brain Size and Intelligence Debate

01:33:44 AI and Goal-Completeness

01:35:10 AI Optimization and Economic Impact

01:41:50 Feasibility of AI Alignment

01:55:21 AI Liability and Regulation

02:05:26 Final Thoughts and Wrap-Up

Robin's links:

Twitter: x.com/RobinHanson

Home Page: hanson.gmu.edu

Robin’s top related essays:

* What Are Reasonable AI Fears?

* AIs Will Be Our Mind Children

PauseAI links:

https://pauseai.info/

https://discord.gg/2XXWXvErfA

Check out https://youtube.com/@ForHumanityPodcast, the other podcast raising the alarm about AI extinction!

For the full Doom Debates experience:

* Subscribe to https://youtube.com/@DoomDebates

* Subscribe to the Substack: https://DoomDebates.com

* Search "Doom Debates" to subscribe in your podcast player

* Follow me at https://x.com/liron

This episode is a comprehensive preparation session for my upcoming debate on AI doom with the legendary Robin Hanson.

Robin’s P(doom) is <1% while mine is 50%. How do we reconcile this?

I’ve researched past debates, blogs, tweets, and scholarly discussions related to AI doom, and plan to focus our debate on the cruxes of disagreement between Robin’s position and my own Eliezer Yudkowsky-like position.

Key topics include the probability of humanity’s extinction due to uncontrollable AGI, alignment strategies, AI capabilities and timelines, the impact of AI advancements, and various predictions made by Hanson.

00:00 Introduction

03:37 Opening Statement

04:29 Value-Extinction Spectrum

05:34 Future AI Capabilities

08:23 AI Timelines

13:23 What can't current AIs do

15:48 Architecture/Algorithms vs. Content

17:40 Cyc

18:55 Is intelligence many different things, or one thing?

19:31 Goal-Completeness

20:44 AIXI

22:10 Convergence in AI systems

23:02 Foom

26:00 Outside view: Extrapolating robust trends

26:18 Salient Events Timeline

30:56 Eliezer's claim about meta-levels affecting capability growth rates

33:53 My claim - the optimization power model trumps these outside-view trends

35:19 Aren't there many other possible outside views?

37:03 Is alignment feasible?

40:14 What's the warning shot that would make you concerned?

41:07 Future Foom evidence?

44:59 How else have Robin's views changed in the last decade?

Doom Debates catalogues all the different stops where people get off the "doom train", all the different reasons people haven’t (yet) followed the train of logic to the conclusion that humanity is doomed.

If you'd like the full Doom Debates experience, it's as easy as doing 4 separate things:

1. Join my Substack — DoomDebates.com

2. Search "Doom Debates" to subscribe in your podcast player

3. Subscribe to YouTube videos — youtube.com/@doomdebates

4. Follow me on Twitter — x.com/liron

I’ve been studying Robin Hanson’s catalog of writings and interviews in preparation for our upcoming AI doom debate. Now I’m doing an exercise where I step into Robin’s shoes, and make the strongest possible case for his non-doom position!

This exercise is called the Ideological Turing Test, and it’s based on the idea that it’s only productive to argue against someone if you understand what you’re arguing against. Being able to argue *for* a position proves that you understand it.

My guest David Xu is a fellow AI doomer, and deep thinker, who volunteered to argue the doomer position against my version of non-doomer “Robin”.

00:00 Upcoming Debate with Dr. Robin Hanson

01:15 David Xu's Background and Perspective

02:23 The Ideological Turing Test

02:39 David's AI Doom Claim

03:44 AI Takeover vs. Non-AI Descendants

05:21 Paperclip Maximizer

15:53 Economic Trends and AI Predictions

27:18 Recursive Self-Improvement and Foom

29:14 Comparing Models of Intelligence

34:53 The Foom Scenario

36:04 Coordination and Lawlessness in AI

37:49 AI's Goal-Directed Behavior and Economic Models

40:02 Multipolar Outcomes and AI Coordination

40:58 The Orthogonality Thesis and AI Firms

43:18 AI's Potential to Exceed Human Control

45:03 The Argument for AI Misalignment

48:22 Economic Trends vs. AI Catastrophes

59:13 The Race for AI Dominance

01:04:09 AI Escaping Control

01:04:45 AI Liability and Insurance

01:06:14 Economic Dynamics and AI Threats

01:07:18 The Balance of Offense and Defense in AI

01:08:38 AI's Potential to Disrupt National Infrastructure

01:10:17 The Multipolar Outcome of AI Development

01:11:00 Human Role in AI-Driven Future

01:12:19 Debating the Discontinuity in AI Progress

01:25:26 Closing Statements and Final Thoughts

01:30:34 Reflecting on the Debate and Future Discussions

Follow David: https://x.com/davidxu90

The Ideological Turing Test (ITT) was coined by Bryan Caplan in this classic post: https://www.econlib.org/archives/2011/06/the_ideological.html

I also did a Twitter version of the ITT here: https://x.com/liron/status/1789688119773872273

Doom Debates catalogues all the different stops where people get off the "doom train", all the different reasons people haven’t (yet) followed the train of logic to the conclusion that humanity is doomed.

If you'd like the full Doom Debates experience, it's as easy as doing 4 separate things:

1. Join my Substack - https://doomdebates.com

2. Search "Doom Debates" to subscribe in your podcast player

3. Subscribe to YouTube videos - https://youtube.com/@doomdebates

4. Follow me on Twitter - https://x.com/liron

Today I'm answering questions from listener Tony Warren.

1:16 Biological imperatives in machine learning2:22 Evolutionary pressure vs. AI training4:15 Instrumental convergence and AI goals6:46 Human vs. AI problem domains9:20 AI vs. human actuators18:04 Evolution and intelligence33:23 Maximum intelligence54:55 Computational limits and the future

Follow Tony: https://x.com/Pove_iOS

---

Doom Debates catalogues all the different stops where people get off the "doom train", all the different reasons people haven’t (yet) followed the train of logic to the conclusion that humanity is doomed.

If you'd like the full Doom Debates experience, it's as easy as doing 4 separate things:

1. Join my Substack - https://doomdebates.com

2. Search "Doom Debates" to subscribe in your podcast player

3. Subscribe to YouTube videos - https://youtube.com/@doomdebates

4. Follow me on Twitter - https://x.com/liron

My guest Rob thinks superintelligent AI will suffer from analysis paralysis from trying to achieve a 100% probability of killing humanity. Since AI won’t be satisfied with 99.9% of defeating us, it won’t dare to try, and we’ll live!

Doom Debates catalogues all the different stops where people get off the “doom train”, all the different reasons people haven’t (yet) followed the train of logic to the conclusion that humanity is doomed.

Follow Rob: https://x.com/LoB_Blacksage

If you want to get the full Doom Debates experience, it's as easy as doing 4 separate things:

1. Join my Substack - https://doomdebates.com

2. Search "Doom Debates" to subscribe in your podcast player

3. Subscribe to YouTube videos - https://youtube.com/@DoomDebates

4. Follow me on Twitter - https://x.com/liron