Doom Debates!

Today I’m reacting to Arvind Narayanan’s interview with Robert Wright on the Nonzero podcast: https://www.youtube.com/watch?v=MoB_pikM3NY

Dr. Narayanan is a Professor of Computer Science and the Director of the Center for Information Technology Policy at Princeton. He just published a new book called AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference.

Arvind claims AI is “normal technology like the internet”, and never sees fit to bring up the impact or urgency of AGI. So I’ll take it upon myself to point out all the questions where someone who takes AGI seriously would give different answers.

00:00 Introduction

01:49 AI is “Normal Technology”?

09:25 Playing Chess vs. Moving Chess Pieces

12:23 AI Has To Learn From Its Mistakes?

22:24 The Symbol Grounding Problem and AI's Understanding

35:56 Human vs AI Intelligence: The Fundamental Difference

36:37 The Cognitive Reflection Test

41:34 The Role of AI in Cybersecurity

43:21 Attack vs. Defense Balance in (Cyber)War

54:47 Taking AGI Seriously

01:06:15 Final Thoughts

Show Notes

The original Nonzero podcast episode with Arvind Narayanan and Robert Wright: https://www.youtube.com/watch?v=MoB_pikM3NY

Arvind’s new book, AI Snake Oil: https://www.amazon.com/Snake-Oil-Artificial-Intelligence-Difference-ebook/dp/B0CW1JCKVL

Arvind’s Substack: https://aisnakeoil.com

Arvind’s Twitter: https://x.com/random_walker

Robert Wright’s Twitter: https://x.com/robertwrighter

Robert Wright’s Nonzero Newsletter: https://nonzero.substack.com

Rob’s excellent post about symbol grounding (Yes, AIs ‘understand’ things): https://nonzero.substack.com/p/yes-ais-understand-things

My previous episode of Doom Debates reacting to Arvind Narayanan on Harry Stebbings’ podcast: https://www.youtube.com/watch?v=lehJlitQvZE

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Dr. Keith Duggar from Machine Learning Street Talk was the subject of my recent reaction episode about whether GPT o1 can reason. But instead of ignoring or blocking me, Keith was brave enough to come into the lion’s den and debate his points with me… and his P(doom) might shock you!

First we debate whether Keith’s distinction between Turing Machines and Discrete Finite Automata is useful for understanding limitations of current LLMs. Then I take Keith on a tour of alignment, orthogonality, instrumental convergence, and other popular stations on the “doom train”, to compare our views on each.

Keith was a great sport and I think this episode is a classic!

00:00 Introduction

00:46 Keith’s Background

03:02 Keith’s P(doom)

14:09 Are LLMs Turing Machines?

19:09 Liron Concedes on a Point!

21:18 Do We Need >1MB of Context?

27:02 Examples to Illustrate Keith’s Point

33:56 Is Terence Tao a Turing Machine?

38:03 Factoring Numbers: Human vs. LLM

53:24 Training LLMs with Turing-Complete Feedback

1:02:22 What Does the Pillar Problem Illustrate?

01:05:40 Boundary between LLMs and Brains

1:08:52 The 100-Year View

1:18:29 Intelligence vs. Optimization Power

1:23:13 Is Intelligence Sufficient To Take Over?

01:28:56 The Hackable Universe and AI Threats

01:31:07 Nuclear Extinction vs. AI Doom

1:33:16 Can We Just Build Narrow AI?

01:37:43 Orthogonality Thesis and Instrumental Convergence

01:40:14 Debating the Orthogonality Thesis

02:03:49 The Rocket Alignment Problem

02:07:47 Final Thoughts

Show Notes

Keith’s show: https://www.youtube.com/@MachineLearningStreetTalk

Keith’s Twitter: https://x.com/doctorduggar

Keith’s fun brain teaser that LLMs can’t solve yet, about a pillar with four holes: https://youtu.be/nO6sDk6vO0g?si=diGUY7jW4VFsV0TJ&t=3684

Eliezer Yudkowsky’s classic post about the “Rocket Alignment Problem”: https://intelligence.org/2018/10/03/rocket-alignment/

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates.

📣 You can now chat with me and other listeners in the #doom-debates channel of the PauseAI discord: https://discord.gg/2XXWXvErfA

Sam Kirchner and Remmelt Ellen, leaders of the Stop AI movement, think the only way to effectively protest superintelligent AI development is with civil disobedience.

Not only are they staging regular protests in front of AI labs, they’re barricading the entrances and blocking traffic, then allowing themselves to be repeatedly arrested.

Is civil disobedience the right strategy to pause or stop AI?

00:00 Introducing Stop AI

00:38 Arrested at OpenAI Headquarters

01:14 Stop AI’s Funding

01:26 Blocking Entrances Strategy

03:12 Protest Logistics and Arrest

08:13 Blocking Traffic

12:52 Arrest and Legal Consequences

18:31 Commitment to Nonviolence

21:17 A Day in the Life of a Protestor

21:38 Civil Disobedience

25:29 Planning the Next Protest

28:09 Stop AI Goals and Strategies

34:27 The Ethics and Impact of AI Protests

42:20 Call to Action

Show Notes

StopAI's next protest is on October 21, 2024 at OpenAI, 575 Florida St, San Francisco, CA 94110.

StopAI Website: https://StopAI.info

StopAI Discord: https://discord.gg/gbqGUt7ZN4

Disclaimer: I (Liron) am not part of StopAI, but I am a member of PauseAI, which also has a website and Discord you can join.

PauseAI Website: https://pauseai.info

PauseAI Discord: https://discord.gg/2XXWXvErfA

There's also a special #doom-debates channel in the PauseAI Discord just for us :)

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

This episode is a continuation of Q&A #1 Part 1 where I answer YOUR questions!

00:00 Introduction

01:20 Planning for a good outcome?

03:10 Stock Picking Advice

08:42 Dumbing It Down for Dr. Phil

11:52 Will AI Shorten Attention Spans?

12:55 Historical Nerd Life

14:41 YouTube vs. Podcast Metrics

16:30 Video Games

26:04 Creativity

30:29 Does AI Doom Explain the Fermi Paradox?

36:37 Grabby Aliens

37:29 Types of AI Doomers

44:44 Early Warning Signs of AI Doom

48:34 Do Current AIs Have General Intelligence?

51:07 How Liron Uses AI

53:41 Is “Doomer” a Good Term?

57:11 Liron’s Favorite Books

01:05:21 Effective Altruism

01:06:36 The Doom Debates Community

---

Show Notes

PauseAI Discord: https://discord.gg/2XXWXvErfA

Robin Hanson’s Grabby Aliens theory: https://grabbyaliens.com

Prof. David Kipping’s response to Robin Hanson’s Grabby Aliens: https://www.youtube.com/watch?v=tR1HTNtcYw0

My explanation of “AI completeness”, but actually I made a mistake because the term I previously coined is “goal completeness”: https://www.lesswrong.com/posts/iFdnb8FGRF4fquWnc/goal-completeness-is-like-turing-completeness-for-agi

^ Goal-Completeness (and the corresponding Shapira-Yudkowsky Thesis) might be my best/only original contribution to AI safety research, albeit a small one. Max Tegmark even retweeted it.

a16z's Ben Horowitz claiming nuclear proliferation is good, actually: https://x.com/liron/status/1690087501548126209

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Thanks for being one of the first Doom Debates subscribers and sending in your questions! This episode is Part 1; stay tuned for Part 2 coming soon.

00:00 Introduction

01:17 Is OpenAI a sinking ship?

07:25 College Education

13:20 Asperger's

16:50 Elon Musk: Genius or Clown?

22:43 Double Crux

32:04 Why Call Doomers a Cult?

36:45 How I Prepare Episodes

40:29 Dealing with AI Unemployment

44:00 AI Safety Research Areas

46:09 Fighting a Losing Battle

53:03 Liron’s IQ

01:00:24 Final Thoughts

Explanation of Double Cruxhttps://www.lesswrong.com/posts/exa5kmvopeRyfJgCy/double-crux-a-strategy-for-mutual-understanding

Best Doomer Arguments

The LessWrong sequences by Eliezer Yudkowsky: https://ReadTheSequences.com

LethalIntelligence.ai — Directory of people who are good at explaining doom

Rob Miles’ Explainer Videos: https://www.youtube.com/c/robertmilesai

For Humanity Podcast with John Sherman - https://www.youtube.com/@ForHumanityPodcast

PauseAI community — https://PauseAI.info — join the Discord!

AISafety.info — Great reference for various arguments

Best Non-Doomer Arguments

Carl Shulman — https://www.dwarkeshpatel.com/p/carl-shulman

Quintin Pope and Nora Belrose — https://optimists.ai

Robin Hanson — https://www.youtube.com/watch?v=dTQb6N3_zu8

How I prepared to debate Robin Hanson

Ideological Turing Test (me taking Robin’s side): https://www.youtube.com/watch?v=iNnoJnuOXFA

Walkthrough of my outline of prepared topics: https://www.youtube.com/watch?v=darVPzEhh-I

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

In today’s episode, instead of reacting to a long-form presentation of someone’s position, I’m reporting on the various AI x-risk-related tiffs happening in my part of the world. And by “my part of the world” I mean my Twitter feed.

00:00 Introduction

01:55 Followup to my MSLT reaction episode

03:48 Double Crux

04:53 LLMs: Finite State Automata or Turing Machines?

16:11 Amjad Masad vs. Helen Toner and Eliezer Yudkowsky

17:29 How Will AGI Literally Kill Us?

33:53 Roon

37:38 Prof. Lee Cronin

40:48 Defining AI Creativity

43:44 Naval Ravikant

46:57 Pascal's Scam

54:10 Martin Casado and SB 1047

01:12:26 Final Thoughts

Links referenced in the episode:

* Eliezer Yudkowsky’s interview on the Logan Bartlett Show. Highly recommended: https://www.youtube.com/watch?v=_8q9bjNHeSo

* Double Crux, the core rationalist technique I use when I’m “debating”: https://www.lesswrong.com/posts/exa5kmvopeRyfJgCy/double-crux-a-strategy-for-mutual-understanding

* The problem with arguing “by definition”, a classic LessWrong post: https://www.lesswrong.com/posts/cFzC996D7Jjds3vS9/arguing-by-definition

Twitter people referenced:

* Amjad Masad: https://x.com/amasad

* Eliezer Yudkowsky: https://x.com/esyudkowsky

* Helen Toner: https://x.com/hlntnr

* Roon: https://x.com/tszzl

* Lee Cronin: https://x.com/leecronin

* Naval Ravikant: https://x.com/naval

* Geoffrey Miller: https://x.com/primalpoly

* Martin Casado: https://x.com/martin_casado

* Yoshua Bengio: https://x.com/yoshua_bengio

* Your boy: https://x.com/liron

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

How smart is OpenAI’s new model, o1? What does “reasoning” ACTUALLY mean? What do computability theory and complexity theory tell us about the limitations of LLMs?

Dr. Tim Scarfe and Dr. Keith Duggar, hosts of the popular Machine Learning Street Talk podcast, posted an interesting video discussing these issues… FOR ME TO DISAGREE WITH!!!

00:00 Introduction

02:14 Computability Theory

03:40 Turing Machines

07:04 Complexity Theory and AI

23:47 Reasoning

44:24 o1

47:00 Finding gold in the Sahara

56:20 Self-Supervised Learning and Chain of Thought

01:04:01 The Miracle of AI Optimization

01:23:57 Collective Intelligence

01:25:54 The Argument Against LLMs' Reasoning

01:49:29 The Swiss Cheese Metaphor for AI Knowledge

02:02:37 Final Thoughts

Original source: https://www.youtube.com/watch?v=nO6sDk6vO0g

Follow Machine Learning Street Talk: https://www.youtube.com/@MachineLearningStreetTalk

Zvi Mowshowitz's authoritative GPT-o1 post: https://thezvi.wordpress.com/2024/09/16/gpt-4o1/

Join the conversation at DoomDebates.com or youtube.com/@DoomDebates, suggest topics or guests, and help us spread awareness about the urgent risk of AI extinction. Thanks for watching.

Yuval Noah Harari is a historian, philosopher, and bestselling author known for his thought-provoking works on human history, the future, and our evolving relationship with technology. His 2011 book, Sapiens: A Brief History of Humankind, took the world by storm, offering a sweeping overview of human history from the emergence of Homo sapiens to the present day.

Harari just published a new book which is largely about AI. It’s called Nexus: A Brief History of Information Networks from the Stone Age to AI. Let’s go through the latest interview he did as part of his book tour to see where he stands on AI extinction risk.

00:00 Introduction

04:30 Defining AI vs. non-AI

20:43 AI and Language Mastery

29:37 AI's Potential for Manipulation

31:30 Information is Connection?

37:48 AI and Job Displacement

48:22 Consciousness vs. Intelligence

52:02 The Alignment Problem

59:33 Final Thoughts

Source podcast: https://www.youtube.com/watch?v=78YN1e8UXdM

Follow Yuval Noah Harari: x.com/harari_yuval

Follow Steven Bartlett, host of Diary of a CEO: x.com/StevenBartlett

Join the conversation at DoomDebates.com or youtube.com/@DoomDebates, suggest topics or guests, and help us spread awareness about the urgent risk of AI extinction. Thanks for watching.

It's finally here, the Doom Debates / Dr. Phil crossover episode you've all been asking for 😂

The full episode is called “AI: The Future of Education?"

While the main focus was AI in education, I'm glad the show briefly touched on how we're all gonna die. Everything in the show related to AI extinction is clipped here.

Join the conversation at DoomDebates.com or youtube.com/@DoomDebates, suggest topics or guests, and help us spread awareness about the urgent risk of AI extinction. Thanks for watching.

Dr. Roman Yampolskiy is the director of the Cyber Security Lab at the University of Louisville. His new book is called AI: Unexplainable, Unpredictable, Uncontrollable.

Roman’s P(doom) from AGI is a whopping 99.999%, vastly greater than my P(doom) of 50%. It’s a rare debate when I’m LESS doomy than my opponent!

This is a cross-post from the For Humanity podcast hosted by John Sherman. For Humanity is basically a sister show of Doom Debates. Highly recommend subscribing!

00:00 John Sherman’s Intro

05:21 Diverging Views on AI Safety and Control

12:24 The Challenge of Defining Human Values for AI

18:04 Risks of Superintelligent AI and Potential Solutions

33:41 The Case for Narrow AI

45:21 The Concept of Utopia

48:33 AI's Utility Function and Human Values

55:48 Challenges in AI Safety Research

01:05:23 Breeding Program Proposal

01:14:05 The Reality of AI Regulation

01:18:04 Concluding Thoughts

01:23:19 Celebration of Life

This episode on For Humanity’s channel: https://www.youtube.com/watch?v=KcjLCZcBFoQ

For Humanity on YouTube: https://www.youtube.com/@ForHumanityPodcast

For Humanity on X: https://x.com/ForHumanityPod

Buy Roman’s new book: https://www.amazon.com/Unexplainable-Unpredictable-Uncontrollable-Artificial-Intelligence/dp/103257626X

Join the conversation at DoomDebates.com or youtube.com/@DoomDebates, suggest topics or guests, and help us spread awareness about the urgent risk of AI extinction. Thanks for watching.