Robert Wright's Nonzero

This is a free preview of a paid episode. To hear more, visit www.nonzero.org

0:28 Is the chaos starting to subside? 11:46 Trump’s tariff strategy (or lack thereof)18:56 Trump’s baffling, backfiring Canada obsession 26:01 Elon’s darkly weird grandfather 38:01 Has America gone full oligarchy? 43:16 Heading to Overtime

Robert Wright (Nonzero, The Evolution of God, Why Buddhism Is True) and Eric Levitz (New York Magazine, Vox). Recorded March 10, 2025.

Twitter: https://twitter.com/NonzeroPods

This is a free preview of a paid episode. To hear more, visit www.nonzero.org

0:28 What could go right with Trump’s foreign policy? 8:11 Trump’s Bibi-pleasing Middle East path 13:49 How would Europe-led defence of Europe work? 15:55 What most miss about the Oval Office blowup 19:44 Has relying on America doomed Ukraine? 26:53 Heading to Overtime

Robert Wright (Nonzero, The Evolution of God, Why Buddhism Is True), Derek Davison (Foreign Exchanges, American Prestige, Discontents), and Daniel Bessner (American Prestige, University of Washington). Recorded March 06, 2025.

Twitter: https://twitter.com/NonzeroPods

This is a free preview of a paid episode. To hear more, visit www.nonzero.org

0:28 Dan’s new paper, “Superintelligence Strategy” 2:58 “Mutually assured AI malfunction” 7:46 Does China see America’s AI policy as a grave threat? 14:45 How likely is US-China conflict over superintelligence? 25:26 How America’s chip war makes war over Taiwan more likely 33:45 Why did China-hawkism sweep Silicon Valley? 42:08 Can we avoid AI doom without global governance? 57:26 The key points of Dan’s paper 1:04:56 Heading to Overtime

This conversation is with Dan Hendrycks, a big figure in AI. It’s about a paper he just co-authored (and in fact published today), with some other notables: Eric Schmidt, former CEO of Google and a big figure in tech policy and national security policy circles, and Alexandr Wang, CEO of Scale AI. The paper, called “Superintelligence Strategy,” is putting forth some very big ideas about tech and national security, including the one this conversation focuses on: what the authors call “mutually assured AI malfunction,” or MAIM.

In principle, the dynamic this term describes could apply to any two adversaries that are trying to beat each other to so-called artificial general intelligence (AGI) and then to so-called superintelligence. But by far the most obvious and important application is to the US-China relationship.

AGI is considered by some to be a major threshold that leads in very short order to superintelligence, which in theory could allow one country to utterly dominate countries that are just months behind it in AI progress. So, as Dan and his co-authors point out, if one country seems to be approaching AGI, its adversary could feel so threatened that it takes extreme measures, like cybersabotage of AI infrastructure or even more aggressive moves.

The “Superintelligence Strategy” paper argues that awareness of this possibility of sabotage or other aggression could keep countries from seeking the kind of clear cut AI dominance that might trigger such countermeasures by an adversary. This theoretically stabilizing dynamic is what they call mutually assured AI malfunction. That is of course a reference to mutually assured destruction, or MAD, the stabilizing dynamic that helps keep nuclear powers from attacking each other (since a would-be attacker knows that destruction would likely be mutual).

I thought Dan and I had a great conversation, with lots of healthy disagreement, but I realized afterwards that I had failed to make explicit a big point that was implicit in some of the pushback I gave him. Namely: I do not see the dynamic that he and his co-authors are calling mutually assured malfunction as a very plausibly stabilizing force, even if you implement some of their ideas for making it more stabilizing.

And the reason is that I think in critical respects this dynamic is not analogous to mutually assured destruction. The rules of the game are fuzzier than with mutually assured destruction, and the actions of the two sides are more ambiguous, more prone to differing interpretations.

For example: I suspect China finds current US restrictions on its microchip imports—restrictions that all three authors of this paper support—more deeply threatening than American policy makers realize. And these kinds of asymmetrical perceptions—one country not realizing how threatening its actions seem to another country—are very common in international relations (and in fact have been known to start wars). Yet mutually assured malfunction, if it is to be a truly stabilizing dynamic, would seem to require that such asymmetrical perceptions are rare and that a country like the US is consistently pretty good at understanding how its rivals view things and when they do and don’t feel threatened. And, in my view, the US is historically very bad at exercising this kind of cognitive empathy.

Indeed, as regular NZN podcast listeners know, I’ve marveled (and despaired) at the failure of the US foreign policy establishment to recognize how US chip restrictions have made war in Taiwan more likely—even though the logic is pretty simple. Namely:

The world’s most advanced AI chips are made in the TSMC factories in Taiwan, and the US chip restrictions mean that China can no longer get those chips (except through a kind of black market), whereas the US and its allies get lots of them. So what used to be a deterrent to Chinese invasion—the likelihood that war would disable factories whose most precious output China shared—is much less of a deterrent. What’s more: According to the logic of the Hendrycks paper, if the US gets closer to AGI, and seems appreciably closer to it than China is, disabling those factories could become a goal for China in and of itself (as opposed to just a not-so-regrettable byproduct of an invasion motivated mainly by other concerns).

So, in the end, thinking about the theoretically stabilizing effect of mutual assured malfunction only deepened my sense that this whole AI race with China is destabilizing and dangerous. And it only strengthened my belief that the US and China urgently need to open a serious dialogue about AI and agree on some rules of the road and together try to steer the world toward some degree of cooperative guidance of a technology that has unprecedented potential benefits and unprecedented potential dangers.

But your mileage may vary. So I hope you’ll listen to my conversation with Dan and see what you think. And I hope that, if you like the conversation, you’ll click—I mean smash—the YouTube like button and/or rate and review the podcast and/or share this post with friends.

This is a free preview of a paid episode. To hear more, visit www.nonzero.org

0:28 Andrew’s new gig (and gifts) 3:38 Play-by-play of the Oval Office blowup 9:28 Zelensky’s and Trump’s obsessions 16:55 What are Russia’s red lines? 26:36 Guaranteeing Ukraine’s (and Europe’s) security 33:49 Heading to Overtime

Robert Wright (Nonzero, The Evolution of God, Why Buddhism Is True) and Andrew Day (The American Conservative). Recorded March 03, 2025.

Twitter: https://twitter.com/NonzeroPods

This is a free preview of a paid episode. To hear more, visit www.nonzero.org

0:30 Introduction 2:21 How the Pentagon budget got so big 6:55 How to cut $60 billion in military spending this year 15:11 The privatization of weapons acquisitions 19:12 Prospects for military budget reform under Trump

Recorded February 28, 2025.

Read this week's edition of the Earthling: https://www.nonzero.org/p/immortality-and-its-discontents

https://twitter.com/NonzeroNews

https://twitter.com/NonzeroPods

This is a free preview of a paid episode. To hear more, visit www.nonzero.org

0:28 Paul’s cold take on US-Canada relations 4:38 How representative is representative government, really? 11:36 Bob’s plan to end hockey fights 17:24 A worthwhile Canadian initiative to dump Musk 21:55 What makes James Bond James Bond? 32:05 Bob’s reign as TV analysis king continues (claims Bob) 36:31 Heading to Overtime—and a special discount deal!

Robert Wright (Nonzero, The Evolution of God, Why Buddhism Is True) and Paul Bloom (https://smallpotatoes.paulbloom.net, University of Toronto, The Sweet Spot, Against Empathy). Recorded February 26, 2025.

Twitter: https://twitter.com/NonzeroPods

0:28 Jack’s experience with presidential power 1:56 Unpacking Trump’s Pentagon and DOJ purges 10:57 Which legal battles could Trump win? 19:39 The Trump-Musk chaos: bug or feature? 27:07 How Trump is putting “unitary executive theory” into practice 36:34 Is a unitary executive unconstitutional? 50:33 Heading to Overtime

Robert Wright (Nonzero, The Evolution of God, Why Buddhism Is True) and Jack Goldsmith (Harvard Law School, In Hoffa's Shadow, https://executivefunctions.substack.com/). Recorded February 24, 2024.

Twitter: https://twitter.com/NonzeroPods

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.nonzero.org/subscribe

This is a free preview of a paid episode. To hear more, visit www.nonzero.org

0:30 Intro and housekeeping 2:02 Trump's very good, very bad week on Ukraine 13:18 Trump vs his foreign policy team? 19:42 Great-power competition and its discontents 24:32 Musk runs roughshod over DC

Recorded February 21, 2025.

Read this week's edition of the Earthling: https://www.nonzero.org/p/trumps-very-good-very-bad-week-on

https://twitter.com/NonzeroNews

https://twitter.com/NonzeroPods

This is a free preview of a paid episode. To hear more, visit www.nonzero.org

0:28 Nora’s work on AI alignment and interpretability 2:21 Is *a* singularity near? 9:52 Nora: AI may take my job within two years 14:56 The reason(ing) behind AI’s latest acceleration 24:14 Thinking through the “chain-of-thought” process 41:44 AI “thinking” vs human thinking 47:21 Heading to Overtime

Robert Wright (Nonzero, The Evolution of God, Why Buddhism Is True) and Nora Belrose (EleutherAI). Recorded February 18, 2024.

Twitter: https://twitter.com/NonzeroPods

This is a free preview of a paid episode. To hear more, visit www.nonzero.org

0:29 Van's new book, The Rivalry Peril 2:28 What policymakers forget about the first Cold War 12:54 The roots of the US-China cold war 19:33 The end of globalization and the rise of economic nationalism 26:54 How rivalry deepens inequality in both the US and China

Recorded February 12, 2025.

Read this week's edition of the Earthling: https://www.nonzero.org/p/a-very-bad-week-for-ai-safety

https://twitter.com/NonzeroNews

https://twitter.com/NonzeroPods