Doom Debates!

Producer Ori takes over hosting duties in this special episode, a live recording from Manifest 2026.

He takes attendees for a ride on The Doom Train, the series of claims involved in arguing that P(Doom) is high. Every stop is a place where people get off before reaching that conclusion.

The crowd at Manifest is a mix of forecasters, rationalists, and AI researchers with a wide range of P(Doom) positions. Even without me or Eliezer in the room, there are plenty of Yudkowskians to push back on the accelerationist claims.

Watch along and decide where YOU get off, or if you ride all the way to doomtown with us. Also, grab the Doom Train framework down below so you can get to the crux of disagreement in your own AI x-risk conversations.

Timestamps

00:00:00 — Cold Open

00:00:25 — Liron Introduces Producer Ori’s Manifest Session

00:02:30 — Previewing the Doom Train™ Stops

00:03:38 — Audience P(Doom) Poll

00:05:00 — Stop 1: AGI Isn’t Coming Soon

00:10:43 — Stop 2: AI Can’t Go Far Beyond Human Intelligence

00:15:05 — Stop 3: AI Won’t Be a Physical Threat

00:18:14 — Stop 4: Intelligence Yields Moral Goodness

00:21:38 — Will ASI Behave Because Alien ASIs Are Watching?

00:26:01 — Stop 5: We Have a Safe AI Development Process

00:28:33 — Stop 6: AI Capabilities Will Rise at a Manageable Pace

00:30:08 — Stop 7: AI Won’t Try to Conquer the Universe

00:34:22 — Stop 8: Superalignment Is a Tractable Problem

00:38:01 — Stop 9: Once We Solve Superalignment, We’ll Enjoy Peace

00:41:14 — Stop 10: Unaligned ASI Will Spare Us

00:47:35 — Stop 11: AI Doomerism Is Bad Epistemology

00:49:02 — Bonus Stops and Wrap-Up

The Doom Train Framework

Stops on the Doom Train

AGI isn’t coming soon

* No consciousness

* No emotions

* No creativity — AIs are limited to copying patterns in their training data, they can’t “generate new knowledge”

* AIs aren’t even as smart as dogs right now, never mind humans

* AIs constantly make dumb mistakes, they can’t even do simple arithmetic reliably

* LLM performance is hitting a wall — GPT 4.5 is barely better than GPT 4.1 despite being larger scale

* No genuine reasoning

* No microtubules exploiting uncomputable quantum effects

* No soul

* We’ll need to build tons of data centers and power before we get to AGI

* No agency

* This is just another AI hype cycle, every 25 years people think AGI is coming soon and they’re wrong

Artificial intelligence can’t go far beyond human intelligence

* “Superhuman intelligence” is a meaningless concept

* Human engineering already is coming close to the laws of physics

* Coordinating a large engineering project can’t happen much faster than humans do it

* No individual human is that smart compared to humanity as a whole, including our culture, corporations, and other institutions. Similarly no individual AI will ever be that smart compared to the sum of human culture and other institutions.

AI won’t be a physical threat

* AI doesn’t have arms or legs, it has zero control over the real world

* An AI with a robot body can’t fight better than a human soldier

* We can just disconnect an AI’s power to stop it

* We can just turn off the internet to stop it

* We can just shoot it with a gun

* It’s just math

* Any supposed chain of events where AI kills humans is far-fetched science fiction

Intelligence yields moral goodness

* More intelligence is correlated with more morality

* Smarter people commit fewer crimes

* The orthogonality thesis is false

* AIs will discover moral realism

* If we made AIs so smart, and we were trying to make them moral, then they’ll be smart enough to debug their own morality

* Positive-sum cooperation was the outcome of natural selection

We have a safe AI development process

* Just like every new technology, we’ll figure it out as we go

* We don’t know what problems need to be fixed until we build the AI and test it out

* If an AI causes problems, we’ll be able to turn it off and release another version

* We have safeguards to make sure AI doesn’t get uncontrollable/unstoppable

* If we accidentally build an AI that stops accepting our shutoff commands, it won’t manage to copy versions of itself outside our firewalls which then proceed to spread exponentially like a computer virus

* If we accidentally build an AI that escapes our data center and spreads exponentially like a computer virus, it won’t do too much damage in the world before we can somehow disable or neutralize all its copies

* If we can’t disable or neutralize copies of rogue AIs, we’ll rapidly build other AIs that can do that job for us, and won’t themselves go rogue on us

AI capabilities will rise at a manageable pace

* Building larger data centers will be a speed bottleneck

* Another speed bottleneck is the amount of research that needs to be done, both in terms of computational simulation, and in terms of physical experiments, and this kind of research takes lots of time

* Recursive self-improvement “foom” is impossible

* The whole economy never grows with localized centralized “foom”

* Need to collect cultural learnings over time, like humanity did as a whole

* AI is just part of the good pattern of exponential economic growth eras

AI won’t try to conquer the universe

* AIs can’t “want” things

* AIs won’t have the same “fight instincts” as humans and animals, because they weren’t shaped by a natural selection process that involved life-or-death resource competition

* Smart employees often work for less-smart bosses

* Just because AIs help achieve goals doesn’t mean they have to be hard-core utility maximizers

* Instrumental convergence is false: achieving goals effectively doesn’t mean you have to be relentlessly seizing power and resources

* A resource-hungry goal-maximizer AIs wouldn’t seize literally every atom; there’ll still be some leftover resources for humanity

* AIs will use new kinds of resources that humans aren’t using - dark energy, wormholes, alternate universes, etc

Superalignment is a tractable problem

* Current AIs have never killed anybody

* Current AIs are extremely successful at doing useful tasks for humans

* If AIs are trained on data from humans, they’ll be “aligned by default”

* We can just make AIs abide by our laws

* We can align the superintelligent AIs by using a scheme involving cryptocurrency on the blockchain

* Companies have economic incentives to solve superintelligent AI alignment, because unaligned superintelligent AI would hurt their profits

* We’ll build an aligned not-that-smart AI, which will figure out how to build the next-generation AI which is smarter and still aligned to human values, and so on until aligned superintelligence

Once we solve superalignment, we’ll enjoy peace

* The power from ASI won’t be monopolized by a single human government / tyranny

* The decentralized nodes of human-ASI hybrids won’t be like warlords constantly fighting each other, they’ll be like countries making peace

* Defense will have an advantage over attack, so the equilibrium of all the groups of humans and ASIs will be multiple defended regions, not a war of mutual destruction

* The world of human-owned ASIs is a stable equilibrium, not one where ASI-focused projects keep buying out and taking resources away from human-focused ones (Gradual Disempowerment)

Unaligned ASI will spare us

* The AI will spare us because it values the fact that we created it

* The AI will spare us because studying us helps maximize its curiosity and learning

* The AI will spare us because it feels toward us the way we feel toward our pets

* The AI will spare us because peaceful coexistence creates more economic value than war

* The AI will spare us because Ricardo’s Law of Comparative Advantage says you can still benefit economically from trading with someone who’s weaker than you

AI doomerism is bad epistemology

* It’s impossible to predict doom

* It’s impossible to put a probability on doom

* Every doom prediction has always been wrong

* Every doomsayer is either psychologically troubled or acting on corrupt incentives

* If we were really about to get doomed, everyone would already be agreeing about that, and bringing it up all the time

Sure P(Doom) is high, but let’s race to build it anyway because…

Coordinating to not build ASI is impossible

* China will build ASI as fast as it can, no matter what — because of game theory

* So however low our chance of surviving it is, the US should take the chance first

Slowing down the AI race doesn’t help anything

* Chances of solving AI alignment won’t improve if we slow down or pause the capabilities race

* I personally am going to die soon, and I don’t care about future humans, so I’m open to any hail mary to prevent myself from dying

* Humanity is already going to rapidly destroy ourselves with nuclear war, climate change, etc

* Humanity is already going to die out soon because we won’t have enough babies

Think of the good outcome

* If it turns out that doom from overly-fast AI building doesn’t happen, in that case, we can more quickly get to the good outcome!

* People will stop suffering and dying faster

AI killing us all is actually good

* Human existence is morally negative on net, or close to zero net moral value

* Whichever AI ultimately comes to power will be a “worthy successor” to humanity

* Whichever AI ultimately comes to power will be as morally valuable as human descendents generally are to their ancestors, even if their values drift

* The successor AI’s values will be interesting, productive values that let them successfully compete to dominate the universe

* How can you argue with the moral choices of an ASI that’s smarter than you, that you know goodness better than it does?

* It’s species-ist to judge what a superintelligent AI would want to do. The moral circle shouldn’t be limited to just humanity.

* Increasing entropy is the ultimate north star for techno-capital, and AI will increase entropy faster

* Human extinction will solve the climate crisis, and pollution, and habitat destruction, and let mother earth heal

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

College students are so angry about AI that they booed its very mention at 2026 commencement ceremonies. So why aren’t they rallying behind PauseAI? I went back to my alma mater, UC Berkeley, to find out.

I spoke with students studying philosophy, economics, and physics, as well as everyday folks visiting campus.

My interviewees were aware of AI’s rapidly growing capabilities. When pressed, some would even agree with me on the conceptual benefits of an AI pause. But the sense of urgency was missing.

Timestamps

00:00:00 — Cold Open

00:00:31 — Why I Returned to UC Berkeley

00:02:08 — Jack: Chinese Room Argument

00:03:23 — Richard: “Who Would Build That?”

00:03:54 — Karthik: An Economist’s Lens

00:05:20 — Jack: The Politics Are Unclear

00:06:10 — Yvonne: Why Powerful AI is Risky

00:07:06 — Robin: Doom Is the Wrong Priority

00:10:36 — Dexter: Sam Altman’s Motive

00:10:57 — Christine: AI Is Just a Tool

00:12:32 — What’s Your P(Doom)™

00:13:19 — Jackson: Supporting a PauseAI Treaty

00:14:19 — Why Cal Students Don’t Take AI X-Risk Seriously

00:15:19 — Rachel: Humans Are Irreplaceable

00:16:14 — Two Hours Later

00:16:35 — Wrap-Up

Links

Join the Doom Debates Discord — https://discord.gg/j4zQ8UaQw2

PauseAI — https://pauseai.info/

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Asian Dad Energy is a former big tech developer with an Ivy League education. We have a lot in common. We’re both:

* Software engineers;

* New-(ish) YouTube creators covering AI;

* Dads with a lot of existential angst.

But while I dread AI extincting humanity, Asian Dad has other concerns. We debate the risk of the AI bubble popping, a “techno-feudalism” scenario, and whether AI will outperform a human CEO in the near future.

Let’s find out where he gets off the Doom Train™.

Timestamps

00:00:00 — Cold Open

00:00:54 — Introducing Asian Dad Energy

00:03:36 — Asian Dad’s Career in Tech

00:09:06 — Getting Laid Off: The Job Market Is a “Hellscape”

00:12:29 — Software Engineering in 2026 vs. 2021

00:17:03 — Why Are They Investing Trillions in AI?

00:24:37 — Debate: AI Is Different Than Other Tech Bubbles

00:33:54 — Debate: AI Is Not Truly Intelligent

00:41:12 — Intellidynamics: Judge AI by Its Outputs

00:48:53 — Debate: Could AI Outperform a Human CEO?

00:56:33 — Asian Dad: AI Can’t Do Genuine Innovation

01:08:14 — What’s Your P(Doom)?™

01:13:17 — “Are You Calling Me a Scammer?”

01:14:41 — Asian Dad’s Mainline AI Doom Scenario

01:20:53 — Will AI Wipe Out Humanity by 2050?

01:29:29 — Why Is AI Doomerism “Getting Out of Control”?

01:33:11 — Debate: AI Lacks Agency

01:42:13 — Asian Dad Warns of “Techno-Feudalism” & Transhumanism

01:48:12 — Wrap-Up

Links

Asian Dad Energy (YouTube channel) — https://www.youtube.com/@AsianDadEnergy

Asian Dad Energy, “AI Doomerism Is Getting Out of Control” — https://www.youtube.com/watch?v=8bnzaG9QRG4David Deutsch, The Fabric of Reality (Amazon) — https://www.amazon.com/Fabric-Reality-Parallel-Universes-Implications/dp/014027541X

Robert Wright, The God Test (Amazon) — https://www.amazon.com/God-Test-Artificial-Intelligence-Reckoning/dp/1668061651

OpenAI model disproves the Erdős unit distance conjecture — https://openai.com/index/model-disproves-discrete-geometry-conjecture/

METR, Measuring AI Ability to Complete Long Tasks — https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Peter Clarke (Team Futurism) invited me on his podcast to probe the worldview of a Yudkowskian, and I ended up explaining how my 50% P(Doom) and 10% P(Foom) informs my choices.

Peter and I met at LessOnline 2026, where he was one of the participants in a live Jubilee-style debate about x-risk. This one’s a little tamer, but we still cover plenty of ground, like:

Why nuclear risk is underrated yet AI doom overshadows it;

Why nobody at AI summits is appropriately freaked out;

How Yudkowsky’s data center “airstrike” claim gets mischaracterized;

Why a superintelligence that doesn’t already value humanity won’t spare us;

And the “Icarus curve”, my techno-optimist forecast pre-superintelligence.

Subscribe to the Team Futurism podcast —https://www.youtube.com/@teamfuturism3554

Timestamps

00:00:00 — Welcoming Liron on Team Futurism

00:00:38 — How Liron Got Yudkowsky-Pilled

00:04:11 — AI 2027 and the Next Qualitative Breakthrough

00:05:52 — What’s Your P(Doom)?

00:10:22 — Why Isn’t Everyone Freaked Out?

00:13:09 — How Should We Then Live?

00:16:22 — Setting the Record Straight on Yudkowsky’s “Airstrike” Claims

00:20:54 — Two Doom Buckets: Misuse vs. Rogue AI

00:24:25 — The Bioweapon Kill Scenario

00:29:02 — The “Benevolent God” Cope

00:33:06 — Epistemology: What We’re Allowed to Know

00:40:08 — Dario’s Country of Geniuses in a Data Center

00:44:37 — The Icarus Curve

00:47:17 — Meaning and Purpose in the AI Era

00:49:32 — Wrap-Up

Links

Team Futurism on YouTube — https://www.youtube.com/@teamfuturism3554

Peter Clarke on X — https://x.com/heypeterclarke

Liron vs. 9 Rationalists at LessOnline 2026 (the “Surrounded” debate with Peter) — https://lironshapira.substack.com/p/will-ai-actually-destroy-humanity

AI 2027 — https://ai-2027.com/

“If Anyone Builds It, Everyone Dies” by Eliezer Yudkowsky & Nate Soares — https://ifanyonebuildsit.com/

Species | Documenting AGI, Drew Spartz’s channel — https://www.youtube.com/@AISpecies

Dario Amodei’s “Machines of Loving Grace” — https://darioamodei.com/essay/machines-of-loving-grace

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Dr. Holly Elmore, executive director of PauseAI US, is done giving AI insiders the benefit of the doubt.

In this spicy episode, Holly calls out a number of respected figures in the AI safety community for allegedly “selling out” the cause: Effective Altruists, Anthropic & OpenAI employees, Eliezer Yudkowsky, Zvi Mowshowitz—and even yours truly!

Holly isn’t afraid to psychoanalyze her subjects, which I push back on, but I can’t deny she has a track record of pushing the Overton window and surfacing uncomfortable truths.

Does Holly’s approach risk burning bridges with allies for little benefit, or does AI safety need a moral conscience willing to call out the rot nobody else will?

Timestamps

00:00:00 — Cold Open

00:00:48 — Introducing Holly Elmore

00:02:20 — Why Holly Moved to DC

00:06:03 — What’s Your P(Doom)?™

00:07:18 — How We Got Here: EA, Rationality, AI Alignment

00:16:32 — “If Anyone Builds It” and LessWrong’s Missing Mood

00:21:47 — Holly Supports the Government’s Fable Crackdown

00:33:39 — Beef: Dean Ball (OpenAI)

00:53:09 — Beef: Eliezer Yudkowsky

00:56:14 — Beef: Zvi Mowshowitz

01:11:50 — Beef: Liron Shapira!

01:16:15 — Beef: Nirit Weiss-Blatt (Dr. Techlash) Being “Bad Faith”

01:19:31 — Anthropic Took Over EA

01:22:04 — Holly’s Basilisk: What If Building Superintelligence Becomes a Crime?

01:32:35 — Beef: Amanda Askell, the “Mother of Claude”

01:39:05 — Beef: Daniela Amodei, Co-Founder of Anthropic

01:40:23 — Beef: Joe Carlsmith, Member of Technical Staff at Anthropic

01:49:39 — Beef: Open Philanthropy / Coefficient Giving, Ben Goldhaber

01:54:28 — Wrap-Up

Links

* Support PauseAI US: https://www.pauseai-us.org/

* Holly on Twitter/X: https://x.com/ilex_ulmus

* Joe Carlsmith’s “Leaving Open Philanthropy, going to Anthropic” post: https://joecarlsmith.com/2025/11/03/leaving-open-philanthropy-going-to-anthropic/

* Dean Ball joins OpenAI announcement (FAI post): https://www.thefai.org/posts/dean-ball-joins-openai-as-head-of-strategic-futures

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Casey Muratori is one of the top programming educators on YouTube, with a four-decade career building videogames and popular videogame software.

We cover his positions on crypto, programming, AI’s impact on programming, and whether he takes the AI doom threat seriously.

Casey’s instinct is to break technology down like an engineer until he knows exactly how it’ll operate. Does that approach break down in the modern AI paradigm?

This is a ride on the Doom Train™ that lands on a wholly unexpected destination.

Timestamps:

00:00:00 — Cold Open

00:00:31 — Introducing Casey Muratori

00:02:07 — Recap of Crypto Series: Is Crypto Dead?

00:11:46 — On Programming & Why Coding Agents Don't Faze Casey

00:18:51 — Coding Is a Craft: Drum Machine Example

00:36:24 — Pivoting to AI: Can We Forecast Its Capabilities?

00:43:36 — Debating if AI Can Acquire Taste

00:54:59 — What's Your P(Doom)™?

01:04:11 — Even 0.1% X-Risk Is Unacceptable

01:07:34 — Modeling Uncertainty: Bayesianism vs "Tie Goes to the Catastrophe"

01:09:34 — Is This Pascal's Wager?

01:17:12 — Casey's (Cynical) Mainline Scenario

01:24:22 — Would You Un-Invent the Internet?

01:27:18 — Let's Ride the Doom Train 🚂

01:27:51 — Can AI Become Ridiculously Powerful?

01:29:11 — Will AI Preserve Humanity (Orthogonality Thesis)?

01:32:26 — Will AI Seek Power (Instrumental Convergence)?

01:36:06 — Wrap-Up

Links

Casey’s podcast, Wading Through AI (with Demetri Spanos) — https://www.youtube.com/playlist?list=PLEMXAbCVnmY6U_pA-7GKuP9xiv9utLaP4

Casey’s YouTube channel, Molly Rocket — https://www.youtube.com/@MollyRocket

@cmuratori (Casey Muratori) on X — https://x.com/cmuratori

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Bestselling author Robert Wright has landed on a sobering conclusion: on our current trajectory, AI dystopia is likely within eight years.

We get into his P(Doom), his case for slowing down, why racing China could backfire into authoritarianism here at home, and why he thinks evolution itself might have a purpose.

Bob is the author of The Moral Animal and Nonzero, a Pulitzer finalist, and hosts the excellent Nonzero podcast.

Get his new book, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning.

Timestamps

00:00:00 — Cold Open

00:00:50 — Introducing Robert Wright

00:03:21 — What Is The God Test?

00:04:59 — AI as Evolutionary Force, Not Normal Technology

00:07:34 — AI Heaven vs AI Hell

00:09:09 — Bob's Background: From Hinton in 1983 to Today

00:16:09 — What's Your P(Doom)?™

00:20:09 — The Mainline Scenario: Dystopia in 8 Years

00:22:04 — Destabilization and the Road to Authoritarianism

00:25:35 — Why Racing China Backfires

00:28:37 — Slowing Down AI + Would Bob Support PauseAI?

00:32:38 — Intellidynamics and Why AIs Collude

00:44:44 — Hinton's Warning and How LLMs Discover Meaning

00:53:48 — Symbol Grounding and Killing Searle's Chinese Room

01:02:03 — Let's Talk About God

01:10:14 — Orthogonality, Power, and the Concentration Problem

01:16:31 — Wrap-Up

Links

Get Bob’s new book, “The God Test” — https://www.simonandschuster.com/books/The-God-Test/Robert-Wright/9781668061657

Follow Bob on X — https://x.com/robertwrighter

“If Anyone Builds It, Everyone Dies” by Eliezer Yudkowsky & Nate Soares (referenced) — https://ifanyonebuildsit.com

Join the Doom Debates Discord — https://discord.gg/g2X8h5UCrU

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Doom Debates presents an interactive debate using Jubilee’s popular “Surrounded” format, taped live at LessOnline 2026!

I (Liron) claim AI will likely doom us to extinction because:

* Godlike ASI is coming soon

* We won’t align the first ASIs

* Even with aligned ASI, the equilibrium is lethal

LessOnline attendees challenge me on each sub-claim one debater at a time, with the audience voting on when to rotate. Recorded on June 6th in Berkeley, CA.

Links

LessOnline 2026, the conference where this was recorded — https://less.online

AI 2027, the forecast referenced on timelines — https://ai-2027.com

Robin Hanson on the “dream time” (the era of slack before resources tighten) — https://www.overcomingbias.com/2009/09/this-is-the-dream-time.html

Join the Doom Debates Discord —https://discord.gg/j4zQ8UaQw2

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

The US government banned Claude Fable like a clown show. But I’m happy about it.

Here’s my appearance on the AI in the AM livestream (recorded Tue, June 16) with Nathan Labenz and Prakash Narayanan, where I explain why this was good for society. The government’s action shows that we can actually pause AI, a precedent that matters way more than the boneheaded rationale behind Fable’s ban.

We then get into a mini-AI doom debate where Prakash defends the AI optimist’s worldview. I push back on how much I expect superintelligence to reshape the planet.

Timestamps

00:00:00 — Why I’m Happy the Government Banned Fable

00:11:18 — Why We Lose Control of AI

00:20:27 — The Icarus Graph: Heaven, Then Hell

00:22:33 — Get Ready to Pause

00:25:51 — What Exactly Should We Ban?

00:27:42 — Wrapping Up

Links

The Cognitive Revolution podcast with Nathan Labenz —https://www.cognitiverevolution.ai/

Nathan Labenz on X — https://x.com/labenz

Prakash Narayanan on X — https://x.com/8teapi

AI in the AM on X — https://x.com/AI_in_the_AM

Transcript

Why I’m Happy the Government Banned Fable

Nathan Labenz 00:00:08On that note, perhaps a fitting transition to our first guest of the day. Liron Shapira is the host of Doom Debates, a YouTube channel where he has found quite a bit of success in bringing people from really all walks of life on to discuss how scared we should be of AI and just how high our P(Doom)s should be.

So Liron, I’m excited to get your take on recent events. Obviously, never a dull moment in the AI space. The thing that caught my eye most from you over the last few days was when you said basically, “I want the government to be willing to take action on AI issues so badly that even if this whole Fable ban turns out to be totally boneheaded and unjustified in some local sense, you support it, or you’re at least happy to see it because it breaks the ice and now nobody can ever tell you again that it’s impossible to imagine the government doing anything.” So give me the double click into that story, and then I’m sure we’ll have plenty of interesting directions to go.

Liron Shapira 00:01:17Hey, Nate. Great to be here. That’s exactly right. I’m a simple man. I see AI getting paused. I feel good about breaking the Overton window. The government can do it. It’s that easy, guys. This is a precedent. Overall, I’m happy. You can talk about the nuances. It was done like a clown show. It was done for bad motives. It doesn’t really consider China or a treaty or anything. There’s a lot of problems, but I’m really happy about smashing the Overton window, where now tech folks don’t think that they’re in a bubble or they’re untouchable. It happened, guys. And we can only go from here.

Prakash Narayanan 00:01:53I actually agree with you, because I think it was a little bit delusional for tech to feel that it wasn’t gonna get touched. And the government just has so many small and large ways to effectuate its power. So it was not that surprising to me that they kind of went through left field and went with the export control rather than anything else.

But it also strikes me that as they exercise this, we start also to go into kind of what we wanted to avoid. It’s a little bit of a small tyranny. I think several people on the timeline have commented, Dean Ball has commented, this kind of unstructured regulation looks kind of selective and vengeful almost, and it starts putting you in this zone where tech people start to mistrust the government because you also see — I think there’s a lot of narratives on the timeline which are being leaked, “sources close to,” “sources familiar with,” and as they get leaked, it’s not very certain whether those things actually happened. Would someone actually attest to that in front of Congress? Very unclear.

And we’ve also seen this kind of behavior from the administration in other affairs as well, where you have multiple conflicting narratives. It’s happening with the Iran war right now, where it’s not even clear to Congress what the deal is. You have different people saying the deal is a different thing.

Prakash 00:03:46So where do you think that puts us?

Prakash 00:03:49It’s great that it’s happening. I understand you feel it’s great that it’s happening to AI right now, but does that put us in a position where it’s detrimental to the body public at large?

Liron 00:04:03Well, I think your analysis — you’re weaving together a few factors, but I think the elephant in the room, I hate to get political because when it comes to President Trump, he’s a mixed bag for me. I don’t have Trump derangement syndrome. I don’t love everything he does. I don’t hate everything he does. But I think the common thread with Trump is it’s just a mess. It’s not disciplined. And I think we’re definitely seeing that on display right now. I would argue we’re seeing that on display in the Iran war.

Previous administrations, there was just more pressure to have logical consistency, some kind of narrative. And this is another one of those cases where you see people in his administration saying all these justifications why something happened, but then the next day it’s, “Oh, it happened for this reason.” “Oh, Dario did this, he wasn’t responsive to us. That’s why we’re doing it.” And then Anthropic’s like, “Oh, no, he was responsive to us.”

And it’s still not clear exactly what did Fable do that was so dangerous, because Anthropic is like, “Oh, this jailbreak is nothing special.” And the Trump administration’s like, “Oh, well, our secret source,” Amazon or whatever, “they’re telling us that it is dangerous.” So I hate that it’s a clown show. I hate that this is how humanity’s operating. I’ll take the win that it’s a pause, but I also think it’s probably time for a new administration.

Nathan 00:05:19One thing I think you should share with folks who may not know you — I think, like many people that are worried about big picture AI safety issues, your background is one of being a, I would say, techno-optimist libertarian for the most part, right?

Liron 00:05:38Totally.

Nathan 00:05:39So how much are you personally missing Fable, and what was your kind of initial impression of it that I know is superseded by your big picture analysis? But how did your first few days of Fable unfold and how bad are your Fable withdrawals?

Liron 00:05:56My Fable withdrawal is not too bad because I’m not pushing it to the limit. I’m not doing hard research. My main use case for frontier AI is Claude Code on a regular production app that has thousands of daily users, but it’s not super hard to serve. So my experience using Claude Code is actually that Opus 4.8 or even going back to Opus 4.6, if I had to go back to Opus 4.6 fast mode, I’d actually still be pretty happy. That’s my threshold of where I’m happy.

So when Fable came out, they said it’s more powerful but it’s slower. But typically, when I give Opus 4.8 a big project, I am pretty happy with the results. I don’t mind giving it a few comments and letting it work again. If anything, I would actually prefer more speed to more quality for my use case at this time. But then again, I’m not on the frontier of computer security. I’m not trying to do anything super novel. So yeah, I’m not experiencing withdrawal right now.

Nathan 00:06:50Interesting. Did you notice — I mean, for me, it, I wouldn’t say it was so much coding, but in the more general purpose knowledge work type stuff that I do, I just found its outputs to be notably better, and I was kind of like, “Oh, this is gonna change how I work. I’m not gonna need to be so precious about my language anymore. I need to actually recalibrate how I think about authorship or shared authorship with AI.” Did you experience any of those kind of feelings?

Liron 00:07:21I can’t say I did firsthand, but I’ve been reading a lot of other people’s accounts, and my vague impression of what’s going on with Fable is, obviously the time horizon is blasting forward. If you talked to me a few months ago, I would’ve said the Achilles heel of all of these AI models is robustness. They’ll do a bunch of work and they’ll mostly get it right, but they’ll make a few mistakes, and the mistakes pile up because the error correction isn’t robust, and that’s the source of the time horizon problem. They pile up too many errors, too many lethal mutations or whatever.

It seems like Fable subjectively has more of this meta process where it can go back of itself and be like, “Oh, no, let me fix this, let me fix this.” So its time horizon is longer, and this also relates to some buzz I’ve been hearing from random people who are on the frontier, which I don’t quite include myself on, but I’ve heard this from Anthropic employees and I think Gary Tan was early to this — this idea that you have multiple roles. You have a company of agents, and they can reflect on each other, and one of them’s whole job is quality control, and one of them is the project manager. I think this idea of the swarm of agents might get us past the robustness hurdle and unlock a new time horizon.

Nathan 00:08:35And how scary is that for you relative to your kind of big picture fears? I mean, you can give the briefcase for doom if we don’t change course better than I can. But I’m also interested in how far you perceive us to be now on that curve.

Prakash 00:08:58Mm-hmm, mm-hmm.

Nathan 00:08:58And also what in the system card, or the sort of interesting new behaviors that we’ve observed from the Mythos Fable models, are kind of most catching your attention or you think are underappreciated by people who are not so focused on watching these leading indicators as you are.

Liron 00:09:23I’m glad you’re asking about the end state because everybody’s got their head two inches in front of their nose. Everybody’s like, “Oh my God, more tokens. What’s gonna happen next? Are AIs going to replace this role?” And it’s like, yeah, these are interesting questions for the next 12 months. But to me, it’s pretty clear that we can predict some properties of the end state. The end state will involve the human brain being worthless at thinking. I just don’t think we’re good at thinking on any dimension.

So yes, we still have a lead. Even though AI can do all this stuff, it can’t do everything better than us. I think we can pretty robustly predict that this is going to stop. Will it take more than a year? It might. So we can talk about, do we need another paradigm? But if we fast-forward 10 or 20 years, it’s probably more than enough time to make this happen.

I just think AI will have all the skills, and it’ll be overdetermined. And I also think that the universe is malleable to AI. So I think not only can AI write code, I think that AI will be able to manipulate atoms really well. That’s right, I went there. I think nanotechnology is physically possible.

People tend to just not dream big enough in terms of what this universe allows when you have a grown-up intelligence operating it. This universe is not really meant for humans to operate. We just kind of punch above our league because we try really hard. But a good mental model of AI is it can just make a blueprint for where it wants the atoms to go, and then they’ll mostly go there — minus some heat exhaust.

Prakash 00:10:49I actually completely agree with you, but I wonder why I end up taking the optimistic viewpoint of that, that that capability will be used to expand humanity’s frontier, and why you have the exact opposite viewpoint that it is a doom scenario.

Why We Lose Control of AI

Liron 00:11:18I would turn it around on you. What makes you so optimistic? Because I think the key moment for me is I just think we’re going to lose control. We’re gonna have this AI which is incredibly powerful, but the linkage between what it wants to do and what we would want it to do, I think at some point we’re just going to break the link. There’s too much power, and there’s no natural persistent link.

Getting the link to hold as it’s scaling up and going through all these transformations and copying itself — this is what people aren’t imagining. Making new versions of itself from scratch. So all these transformations are happening. All these successor AIs are happening.

Prakash 00:11:53Mm-hmm.

Liron 00:11:53And you think the human species is back here still holding some kind of leash? I just don’t really see it.

Prakash 00:11:58So again, to push back — why is that even important that humans maintain control? Because let me give you an example.

Liron 00:12:09Yeah.

Prakash 00:12:09The average — let’s say about 50% of humanity is probably still doing kind of subsistence or near-subsistence farming, or small shopkeepers, et cetera, across the world. How much control does the average farmer in Uganda have? Does that person have control? And are we judging — what relationship does that person, who maybe got a smartphone two years ago, what relationship does that have to control? What aspect of control does that particular human have, and how does this relate to that particular human?

Liron 00:12:50Mm-hmm. Yeah, yeah.

Prakash 00:12:50What aspect of control does that particular human have, and how does this relate to that particular human?

Liron 00:12:55So you’re leaning on the convenient fact that when there’s a farmer in Uganda, their territory is far away. It’s causally distant. So somebody in the United States, we don’t have a button available to us. Even though we’re a rich country and we have more control over the world per person, we have more power, we don’t have a convenient button we can press that will go harvest the atoms over in Uganda and then deposit $1,000 in our bank account. We don’t have that button.

It is ultimately a matter of intelligently configuring atoms to get there. It’s kind of like a code base. Since I got Claude Code, my code base is a lot cleaner because I’m running all of these cleaning processes. I’m investing more in the code because it’s cheap for me. Similarly with Uganda — that’s kind of just unused resources from the AI perspective. Once we get enough power, it’s, “Okay, yeah, let me harvest all the land in Uganda. Who’s gonna stop me?” And then it only comes down to me caring for the Ugandan farmer. Okay, I’m not gonna kill him. But the AI might not care.

Prakash 00:14:11That does not make sense at all because they are linked to the market economy. And in fact, what we do end up doing in the United States is we press a button, and that’s called a buy button, and that buy button ends up harvesting the cotton in Uganda. That’s literally what we do. We go to Amazon and you press the Buy button, and as a consequence of pressing that Buy button, and through the linkages of all of these entities, which are all like small mini AGIs — all these companies in the middle.

Hundreds of these companies. The logistics firm, the firm supplying the fertilizer, the firm doing the milling. All of these firms in the middle, and then we get the product in our backyard. So I basically resist this idea that that button doesn’t exist. That button exists. We push it every day. It’s the Amazon button.

Liron 00:14:40Well, this is kind of the common argument of Ricardo’s law. It’s like, oh, we’re trading with them, right? So why won’t the AIs trade with us? But we don’t have the button called Harvest the Atoms, because that would be a better button if we had it.

Prakash 00:14:55No, I’m pushing back here because you’re saying that we don’t have control, and you’re saying that these AIs will have that control, and we’re — right now, humanity does not have that control. I’m saying humanity already has that control on that particular farmer in Uganda through the market economy. And —

Liron 00:15:15Oh.

Prakash 00:15:15This does not really differ whether you replace the market economy with a buyer who happens to be an AI. It’s basically the same, right?

Liron 00:15:24I think you may not be realizing how many preconditions are necessary in order for a market to be the relationship that emerges. Because we as humans, like you said, we’re used to being in this kind of market trading relationship with other humans.

Prakash 00:15:37Yeah, yeah.

Liron 00:15:37But you might also note we’re not in a market trading relationship with the other animals. Cows — we don’t really let them buy turf. We kind of decide where the cow’s gonna live. So I think you might end up with that kind of relationship with AI versus human in terms of the power differential.

Prakash 00:15:54So this is a question of control. It’s a question of, do humans lose control to AI?

And I was pointing out that that particular human in Uganda does not have much control anyway, and there’s no real control loss in that sense. So I just don’t see how — 50% of the global economy consists of basically farmers who are trading, and I just don’t see how much control they have to lose, and I don’t see why this matters to them at all.

Liron 00:16:25Well, yeah.

Prakash 00:16:25I think it matters to, let’s say, the leadership at a big tech firm. Yes. It definitely matters to them because they have control over the tech sphere. I don’t see how this matters to most of the global economy or most humans in general, actually.

Liron 00:16:41Well, so you’re basically asking what’s the discontinuity or the disanalogy between the farmer today and that specific farmer after superintelligence.

Prakash 00:16:49Yes.

Liron 00:16:49Very simply — the benefit of harvesting their resources or destroying their resources will exceed the cost. Right now, why don’t I personally take the farm — besides caring about... Why doesn’t the US take over Uganda? It’s just not high value to us to take over that land. We think that our intellectual property and what we can build here in the US is already much more interesting, and also, we believe in moral rights. That’s a factor too — sovereignty.

But from the AI’s perspective, you can also think about side effects. Let’s say the AI wants to turn the planet into a factory to produce probes so it can participate in the land grab across the galaxy, across the universe. Okay, so if you’re building probes, Eliezer Yudkowsky pointed out you wanna run the Earth really hot. That’s your preferred temperature, and the oceans will probably boil away. As long as you can radiate away the heat, the temperature is good. But that’s going to be a higher temperature than humans can live in open air. So suddenly you can’t just be chilling as a human. It really is up to the AI to decide who gets to live where.

Nathan 00:17:53So there’s a couple parts of that story that I think people just have extremely different intuitions about. One — and even including in the camp of people who are very worried that we’re gonna end up in a bad outcome — one is how long does it take? There’s sort of the fast takeoff foom scenario where we all drop dead suddenly, and then there’s the kind of gradual disempowerment vision where this maybe happens over not a super long period of time, but through a bunch of locally sensible decisions.

Yeah, I mean, I guess we should let AI run our company now because it is gonna probably do a better job than we are, kind of played out everywhere. Next thing you know, it really is kind of AIs running the show and we’re maybe sort of okay with it at first, but there’s a lot of potential for drift over time.

Of course, there’s also the question of should we be optimistic that we’ve kind of put Claude or whatever into the benevolent basin, that it might just kind of stay in? Or are there a lot of ways that it’s gonna inevitably roll out of the benevolent basin into some other even lower point in its loss landscape that we’re not gonna be so happy with?

We don’t have time probably for your full arguments on all those points, but maybe at least tell us where you are on a couple of those big questions, and then use that to sketch out —

Nathan 00:19:27We now have Anthropic and OpenAI saying that they’re at least open-minded to some sort of coordinated slowdown, not like the one they just got handed to them by the Trump administration. They don’t want that kind of haphazard pause button. But they’re into something, or they’re at least open to something. And that of course leaves us with the question of, well, what should that look like? What exactly are we pausing? What exactly are we banning?

And there I really don’t know too much about your position in terms of concretely what should be stopped, what should be slowed, what should have limits, what limits should maybe escalate or even be reduced. I know there’s been proposals for reducing compute limits over time. And do we end up in your vision figuring this out and getting to an AI win, or is there no AI win in your mind possible and we just kind of have to steer some other direction for humanity’s future? I know there’s a lot there. You can use the rest of the time if you need it.

The Icarus Graph: Heaven, Then Hell

Liron 00:20:28Yeah, okay, sure. So my worldview, my outlook right now, it’s what I call the Icarus graph. I feel like nobody gets this. Everybody’s like, “No, I think the world is good. It’s gonna go this way.” And some people are like, “No, we’re terrible. Enshittification. It’s gonna go this way.” And I’m like, no, no, it’s Icarus. We’re gonna fly closer and closer to the sun, it’s gonna be great, and then we’re gonna do a 180 degree turn and plummet down to hell. So basically we get a taste of heaven, and then we get hell.

So you have to ask me then, okay, so where on the Icarus graph do we stop? And it’s a brutal question because every day I’m enjoying the flight as much as the next person. Yeah, give me the next Claude. Make my code faster, great. Let it help my business run better. Make me better AI videos.

So there’s no natural point in terms of when it feels right to stop. I just think it’s important to stop before capabilities get to a runaway point, and we’ve been kind of frog-boiled to be, “Oh, each model comes out and we’re doing great.” If we could stop the clock now, would I turn back the clock? Would I lose Fable? Would I lose Opus? No, I’d keep it all. I still think we’re playing shuffleboard, we’re playing Icarus. So far so good. Should we bet again? Should we keep betting until we lose? It’s a crazy tough question.

I think the Eliezer Yudkowsky — yeah, the turkey graph kind of... Yeah, yeah, exactly. Although, the only difference with the turkey graph is each day of the turkey’s life is actually better. Not only is it living longer, it’s actually living better and better. So the turkey is really happy with its life.

So I think the Eliezer Yudkowsky MIRI position, which I agree with, is just, we don’t know when to stop, so let’s get ready to stop. At the very least, let’s get ready. I would probably stop today. I would stop and I would be bummed. I saw a food influencer say this about how she eats chocolate basically. “Yep, I just ate this chocolate and now I’m bummed.” That’s what you gotta do. Don’t reach for another chocolate. Just sit there and be like, “This is the prudent place to stop right now until we have any idea of some kind of theoretical method by which we understand what a superintelligence wants to do and what an equilibrium state of a superintelligence looks like.”

That’s actually something MIRI was trying to study, identifying equilibriums that are plausible for superintelligences. There’s actually a rich vein of theory there that’s highly neglected today. Let’s do some theory there. Maybe then we can unpause. I think that’s gotta be the best plan. And so I think the number one leverage point here is just repeating, “Get ready to pause.” And like you said, OpenAI and Anthropic, they said it. They said, “Let’s try to get ready to pause.” So I would love to see more people saying it because it really has to be a giant groundswell.

Get Ready to Pause

Prakash 00:23:01I mean, if you take a step back and you look at, over the next two years, about a trillion, one and a half trillion dollars of CapEx being spent, and then you calculate the return on equity required in order to make that CapEx worthwhile or to repay that, it basically puts you in a position where if you pause, you go into a depression.

So you have to have — you’re gonna have to deliver the economic growth that this one and a half, two trillion dollars has been spent on. And that indicates that you’re gonna have to grow in that 2027, 2028, 2029 period. You’re gonna need to see that growth or else we go into depression. And when you offer the choice to the public or policymakers, “Hey, you can pause because of this vague and unspecified claim of disaster in the future” —

Or we can — and in addition to that, a guarantee of a depression, because those GPUs cannot be used for anything else besides compute. And that compute that is useful is AI compute. And you give them a choice between a depression state, your 10 largest companies basically getting their stock price slashed by 90%, and this vague and unspecified claim of disaster.

I think at this point, maybe it would have made sense — and you guys were on this way, way before, and not enough policymakers were listening. But at this point, it is a very hard sell to make this case to policymakers.

Liron 00:24:46Well, I agree with you and I disagree. I agree with you because, like I said before, yes, we are all going to be bummed. Even forget about the economics — the scientific breakthroughs, the medical breakthroughs. Nate was talking about his son. There’s no decelerationists in the hospital ward. I totally agree, and I’m sure the next time I get sick or an injury or whatever, I’m gonna want the latest frontier model.

So I’m going to be bummed. The only thing I’ll say is when you then mix in the economic element and you say, “Oh, we’re gonna have a depression,” I actually think that is unnecessary. You don’t really need to mix that in because, first of all, I don’t think we’re literally going to be in a depression. Realistically we’ll be fine. It’s just money that’s already been spent. We still have plenty of resources.

But the other factor is, I actually think that all of these valuations and all of the build-out that’s happened so far is actually just perfectly congruent with using the AI we have today for the kind of applications we’ve already discovered today. That should actually be fine to create more trillions of dollars of value. I don’t even think that the market caps reflect the actual singularity. So I think we’ve actually got a glide path if we wanna pause.

Nathan 00:25:51Do you think that we could localize the pause a little bit more to, for example, some sort of operationalizable ban on recursive self-improvement? I know you did an episode recently with someone who — was it Steven Casper — who was advocating for a pause on all research. And you were, I think, kind of like, “Oh, I’m not sure if we should try to pause theoretical work on control or interpretability or what have you.”

And you’re also saying we can just run the inference. So I think that’s a key thing to keep front and center — my understanding of your proposal does not call for a shutdown of ChatGPT as it exists today.

How narrow — if there’s a Pareto curve on the pause frontier, how much value of a pause can we get with how narrow of a pause in your mind?

Liron 00:26:54Vaguely I would say frontier AI capabilities. But it’s like, if you’d asked me three years ago, I’d have been like, “Oh, might as well pause now. It’s prudent to pause now.” But then we kind of gambled and won. So from my perspective, we keep playing shuffleboard. We keep doing Icarus. We keep going higher and winning, kind of, but we’re also getting closer and closer to the point of no return.

So even though it feels like we’re winning now, we’re also killing our ability to pause ‘cause we’re so close to the point of no return — the last breakthrough where after that the AI takes over the research, and then we’re really screwed. So basically I think roughly a good policy is, okay, no more frontier capabilities upgrades for a while. It’s just too dangerous, and I know that concept is hard to communicate to people when every day life is getting more awesome. I think we’re in a screwed situation, but that’s just what I think is prudent.

Wrapping Up

Nathan 00:27:42Would you recommend any recent Doom Debates for folks who wanna go deeper on your ideas? Where do you think this was kind of best hashed out, either recent or perhaps near future on the feed?

Liron 00:27:55So we’ve just been discussing policy, and that’s really not even my wheelhouse, so there’s no really strong policy Doom Debates. But if you just wanna know the lay of the land from my perspective in terms of the next paradigm that’s coming and why I think it’s gonna break the trend and it’s not just gonna be another nice LLM — it’s going to be a reinforcement learning monster that takes over the world — check out my recent episode with Dr. Steven Burns. It’s one of the most popular episodes because he’s an extremely deep thinker, and he really lays out the coming future.

Nathan 00:28:24Perfect. Thank you. Liron, keep fighting the good fight, man. I appreciate your tireless energy on this, and while I’m not maybe quite as high on the P(Doom) scale as you are, I certainly am not a dismisser. And I think you’ve done a valuable job for me personally in terms of pushing me to be more consistently candid about just how scary I think the situation actually is, even though obviously there’s a lot of upside to everything that’s been created as well. So I appreciate that from you, and I’ll continue to follow the Doom Debates for the latest, and I encourage other people to check it out as well.

Liron 00:29:07Thanks, Nate. Thank you, Prakash. Always a pleasure, guys.

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Alex Bores is a generationally strong AI regulation candidate who needs our help to win the NY-12 congressional election next week.

Bentham's Bulldog, a.k.a. Matthew Adelstein, joins me to help spread the message.

PLEASE DO THIS:

* Vote. If you live in Midtown Manhattan, click here to check whether you’re in NY-12. If so, vote for Alex Bores on Tuesday, June 23rd.

* Spread the word. Tell anyone you know in the district to vote for Bores — the race against Micah Lasher is neck and neck.

* Donate now. If you can’t vote in NY-12, donate to the campaign ASAP — donations lose value after ~June 18th because ads must be booked ~5 days in advance.