Dwarkesh Podcast

I just interviewed historian Stephen Koktin yesterday. The full episode should be out next Thursday on my podcast.

I spend a long time researching, and it's impossible to explore more than a small fraction of my curiosities during the interview. This was especially true for Kotkin. These 1000-page-each volumes basically compile all of late 19th/early 20th century history. Naturally, reading them inspired lots of thoughts that I wasn't able to exhaust during the interview itself. So I thought I'd just release my notes publicly.

You can buy volume 1 here and volume 2 here.

One of Kotkin’s interesting takes from this book:

Even the package of attributes that we call modernity was a result not of some inherent sociological process, a move out of tradition, but of a vicious geopolitical competition in which a state had to match the other great powers in modern steel production, modern militaries, and a modern, mass-based political system, or be crushed and potentially colonized.

Part 1 starts by laying out the geopolitical context the tsarist government found itself in during the late 19th century. Bismarck had unified Germany and had put it at the leading edge in key modern industries like steel and chemicals. Britain controlled the seas, kickstarted the industrial revolution, and had a global empire. Japan was going through the Meiji reformation, whose implications would become clear to the world when they beat Russia in a war in 1905.

Meanwhile, what would happen to you if you failed to modernize was made amply clear by the example of Qing dynasty China. What had been the foremost power for almost all of recorded history now couldn’t even prevent a tiny island all the way across the world from forcing opium into its markets.

Russia, like every other country, wanted to avoid such a fate. Its long land borders rendered it permeable to invasion from almost all of Eurasia. Imperial Russia had expanded 50,000 square kilometers per year for 450+ years (to put that in perspective, that's adding an area the size of Slovakia or Costa Rica every single year, for four and a half centuries). This is the classic continental power trap that Sarah Paine talks about. You have to conquer more territory in order to defend the territory you already have. What this meant is that by the end of the 19th century, Imperial Russia had dominion over dozens of different nations, including Georgia, from which Stalin hailed.

China, of course, was also a very big country, but that didn't help much when the more industrialized British came knocking. Russia knew it was at least a generation behind. By 1900, Russia had the world's fourth or fifth largest industrial power. However, the vast majority of the population remained rural peasants living in conditions that had changed little since emancipation.

Russia had a Bismark like figure in Finance Minister Sergey Witte (1892-1903). He kickstarted Russia’s belated industrialization, but failed to prevent a war with Japan which Russia lost, and which was catastrophic for the Tsar’s image. In 1905, the tsarist government almost fell, and was only barely rescued by a brutal crackdown followed by concessions from Nicholas II to form a parliament-like Duma. This was a fucked institution from the very start. Nicholas has still retained more or less all the powers he had before and turned the Duma into a debating club. Kotkin thinks that history would have been much better if the government was just allowed to fall then instead.

Despite remaining an autarchy, Russia did have another great reformer in Peter Stolypin who pursued land reform and further industrialization. In terms of economic output, the effect seemed to have been pretty good. In the 20 years preceding the Bolshevik Revolution, median incomes in Russia had risen 50%. The problem was that this new government had the support of nobody. The liberals and constitutionalists in the Duma felt that it was too little, too late, and also politically illegitimate. Meanwhile, the aristocracy felt that all these reforms were against their own interests. The regime's last chance at survival floundered because there was no constituency which felt bought into the project.

Before we move on to the Bolshevik revolution, some questions about Kotkin’s take on modernization:

• Does he think that countries adopt leading-edge technologies faster than you might expect, given domestic opposition to displacement and automation? Because of this overwhelming geo-political pressure to modernize? If so, does he think that we are overrating the regulatory or political barriers to the adoption of AI? America, will we feel compelled to race on AI against China the same way that late 19th century Russia felt compelled to race against Germany on industrialization?

• Does he think this vicious geopolitical competition still incentivizes the adoption of modernity, given that wars of colonization and territorial conquest have subsided? Sure, this is obviously not true in some parts of the world, but there's no amount of fucking up that some European country could do which would actually get its territory physically dismembered by another country in Europe.

The defeat of Tsarism came not when Kolchak was routed, not when the February Revolution was raging, but much earlier! It was overthrown without hope of restoration once Russian literature adopted the convention that anyone who depicted a gendarme or policeman with any hint of sympathy was a lickspittle and a reactionary thug

Sozenitsn, Gulag Archipeligo

I remain quite confused about how repressive the tsarist regime actually was. The whole revolution was apparently motivated by how the autocracy and their secret police, Okhrana, behaved. But then you have all these people who are literally calling for the overthrow of the government just hanging around. People like Stalin and Lenin who are literally calling for the overthrow of the government are just spending time in exile, living on government stipends, robbing banks, writing articles for Pravda, and having affairs with farmers’ wives. Here is a passage from the Gulag Archipelago:

Let us examine, for instance, some generally known biographical facts about Lenin. In spring, 1887, his brother was executed for an attempt on the life of Alexander III. And what happened to him? In the autumn of that very year Vladimir Ulyanov was admitted to the Imperial University at Kazan, and what is more, to the Law Faculty! Surprising, isn’t it?...

Then a few years later this same young revolutionary was arrested for founding in the capital a “League of Struggle for the Liberation of the Working Class”—no less! He had repeatedly made “seditious” speeches to workers, had written political leaflets. Was he tortured, starved? No, they created for him conditions conducive to intellectual work…

But then, of course, he was condemned by a three-man tribunal and shot? No, he wasn’t even jailed, only banished. To Yakutya, then, for life? No, to a land of plenty, Minusinsk, and for three years…

He asked for an allowance from the state, and they paid him more than he needed. It would have been impossible to create better conditions than Lenin enjoyed in his one and only period of banishment…

Tsardom was always weak and irresolute in pursuit of its enemies. The most important special feature of persecution (if you can call it that) in Tsarist times was perhaps just this: that the revolutionary’s relatives never suffered in the least.

—

Kotkin on what Trotsky could have done in the position he found himself in in 1924, despite being censured by the Central Committee, and missing Lenin’s funeral, and generally finding himself getting sidelined:

In the name of the greater cause of safeguarding the revolution, he [Trotsky] could have violated party discipline by reading aloud on Red Square from Lenin’s purported dictation, using as his mantra Lenin’s summons to “remove Stalin” as general secretary, then flown from factory to factory to rally workers, just as in 1917—let them arrest him. Of course, to do all that, Trotsky needed to perceive Lenin’s death as a strategic opportunity, and he needed a persuasive story line about how the grand socialist dream could be revived, why all those harsh exchanges he had had with Lenin were incidental, and why he (Trotsky) was uniquely qualified to carry forward the sacred Leninist cause. A tall order, to put it mildly. But who could doubt that if Lenin had found that others were conspiring against him, he would have mounted a coup against his own party?

This is such a good question, and one I will ask Kotkin about.

Keep in mind that Trotsky is no naive pushover. He was the primary organizer and tactical leader of the armed insurrection that overthrew Russia's Provisional Government in October 1917. Trotsky directed the Red Guards to seize post offices, telegraph stations, railway stations, bridges, and state banks, effectively paralyzing the Provisional Government's ability to govern. And then of course came the successful assault on the Winter Palace.

Same goes for all the old Bolsheviks, who we know were sooner or later purged, exiled, and/or killed by Stalin. In May 1924, why did Khamanev only read Lenin’s last testament, which called for the removal of Stalin from the position of General Secretary, in front of a small closed session of the Central Committee instead of the full 1,300 Congress? Once Zinoviev broke with Stalin on socialism in one country in 1925, why didn’t he go back to his power base in Leningrad and start an insurrection? Bukharin was Lenin’s golden boy, and the fucking editor of Pravda for god’s sake. All of Russia could have woken up to Lenin’s testament printed on the front pages. What is going to do, kill you? By the time the show trial is being scheduled, that’s guaranteed anyways. How did these hardened revolutionaries who had overthrown a regime which had lasted for hundreds of years get cucked by Stalin?

I just remembered how OpenAI’s board tried to kick Sam out: all they said was, “He has not been consistently candid.” What is the lesson here? No half measures. If you come for the king, you best not miss.

There’s an important lesson in here about the instability of collective leadership. It supposedly worked in China between 1976 with the death of Mao and 2012 with the appointment of Xi as general secretary? But is it fair to call the period that Deng was paramount leader collective leadership? When the premier Li Peng wanted to reverse economic liberation after the Tiananmen Square protests, Deng came out of retirement and went on his Southern tour and declared that, "Whoever is not for reform must step down." Was that really collective leadership? Just because we like what he did doesn’t make it collective leadership.

Stalin’s strategy (which according to my previous guest Victor Shih is also Xi’s) was the align the main members of the party against the next person he wanted to purge. Rinse and repeat. Stalin aligns Khamanev and Zinoviev against Trotsky, then Bukharin against Khamanev and Zinoviev, and by the time it’s Bukharin’s turn, Stalin is basically god dictator already..

An obvious question at this point: at some point you should catch on right? If you’re Bukharin in 1930, and you know you have this independent credibility and power base, what, you think that’s just gonna be fine by Stalin for the next however many decades until you peacefully retire?

Anyways, back to our original question, why doesn’t collective leadership seem to work in practice? Why does it always devolve into one man rule? Will ask Kotkin.

Was Bukharin the Soviet Deng? Killed of course during the great terror, unlike Deng, who was merely rusticated and thus could eventually return.

“the well-off upper stratum of the peasantry and the middle peasant who strives to become well-off are now afraid to accumulate. The situation is created such that a peasant is afraid to mount a metal roof over his house so as not to be called a kulak; if he purchases machinery he does so in a way that the Communists do not see. Higher technology becomes conspiratorial.” Poor peasants, meanwhile, complained that Soviet power hindered their hiring by the better-off peasants. (Most peasants who hired labor themselves worked; they were not rentier landlords.) Party attitudes were holding down production on which the state’s well-being and industrialization hopes rested. Bukharin dismissed the fantasy of collective farms, because the peasants were just not joining them. “That we should in all ways propagandize among the peasants formation of collective farms is true, but it is not true when people maintain that there is a highway to the movement of the peasant mass toward the path of socialism,” he stated. Rather, the answer was to benefit from economic incentives. “It is necessary to say to the entire peasantry, to all its strata: ‘Enrich yourselves, accumulate, develop your farms,’” he told the party activists. “Only idiots can say that we should always have the poor; now we need to conduct policy in such a way that the poor would vanish.”

—

NEP’s dilemma was not merely that the rate of industrial growth seemed too low, making people wonder how long under the NEP it would take before the USSR became a truly industrial country. The dilemma was not merely the unmodernized technical level and small, divided plots of Soviet agriculture, which produced harvests insufficient to support the kind of grain exports necessary to finance imports of machines, including for agriculture. The dilemma was not even just the fact that the regime lacked control over the food supply or the countryside, rendering it hostage to the actions and decisions of the peasantry. All these were profound problems, but the core dilemma of the NEP was ideological: seven years into the NEP, socialism (non-capitalism) was not in sight. NEP amounted to grudgingly tolerated capitalism in a country that had had an avowedly anticapitalist or socialist revolution.

There is this morbid contrarian curiosity about whether collectivization was necessary for Russia’s industrialization. In many cases, I don't even think it's motivated by actual Marxism. There's something appealing about this revisionist mentality of "Are taboo bad things in history good actually?"

But the reality is that there's every reason to expect Russia to have industrialized more successfully without this brutal enslavement of a hundred million peasants. There's not only the example of how almost every other country industrialized, from America to Europe, but also the basic 101 story of how it would have happened is totally credible. More successful peasants would use their profits to mechanize or drive other efficiency gains. Their growing wealth would stimulate a growing consumer economy. Perhaps most importantly, a government not dedicated to the overthrow of wealthy democracies abroad would have been much more capable of attracting foreign direct investment from abroad to support industrialization.

Only in 1928 did Soviet Russia's industrial output recover to the level it was at in 1913 - the point at which it was already a generation or two behind Germany, Britain, and America. And just when things were stabilizing, Stalin launches the first five-year plan, which reduced grain production by over 32% and led to the halving of overall livestock.

The traditional story of what should happen with developing countries is that both the pareto frontier expands and they move from agriculture to industry, as a smaller number of people can handle the food demands of the country. Communist collectivization definitely had the move to industry along the Pareto frontier. But this was coupled to a catastrophic implosion of the Pareto frontier. While Russia was able to produce less grain overall after collectivization, the Soviet government had control over it. It could now export this grain to fuel its military-industrial goals.

Thus, the correct way to understand collectivization is not as a rapid, though costly, speed run to industrialization. Rather, it is as a policy which reduces economic output and even harms industrialization but lets the state control the full economy.

I'm curious if Kotkin thinks that a lesson from this period of history is that one should fall over themselves to support the lesser of two evils.

The Russian examples seem to support this. Arguably, liberals should have backed the authoritarian (but reformist) Stolypin, and the Provisional Government factions should have united more decisively to prevent the greater evil of the Bolsheviks.

Yet this lesson seems to directly contradict the German example, where conservatives did support what they perceived as the lesser evil, Hitler, to stop the Communists, leading to a catastrophic outcome.

Stalin has an inhuman appetite for administration and micromanagement over an empire that streches 11 time zones, from economy to foreign policy to even arrest lists and film edits. He went through dozens of often lengthy memos and documents, and this doesn’t even include his 70 hours of meetings a week. I found it funny how often he marks up memos and letters with the following two phrases:

Regarding something he thought must be changed: “Is it possible that we can allow X? Of course it is not.”

Regarding a supposed blocker to his will: “Is it possible to do X? Of course it is.”

• I remain quite confused about how repressive the tsarist regime actually was. The whole revolution was apparently motivated by how the autocracy and their secret police, Okhrana, behaved. But then you have all these people who are literally calling for the overthrow of the government just hanging around. Lenin’s brother had tried to kill the tsar Alexander III for god’s sake! People like Stalin and Lenin are just spending time in exile, living on government stipends, robbing banks, writing articles for Pravda, and having affairs with farmers’ wives.

• Given how much the war with Japan had weakened the regime, why did Nicholas think that fighting World War I would be good for strengthening his control and image?

• I feel like the revolution of 1905 is understudied. How close did the regime get to falling? Why didn’t Wite’s reforms for the decade prior do anything to placate people? How useful were those reforms in the first place? And Kotkin says it would have been better if the regime were allowed to fall then. What likely would have replaced it? Why not think there might have been another leftist revolution afterwards?

• Why is the tsarist regime sending people to Siberia instead of just killing them?

• Would the hoped for reforms from Wite and Stoyplin have actually caused modernization, even if they had actually been implemented?

• You say that by 1917 some kind of leftist revolution was inevitable, but it didn’t have to end with the Bolsheviks. Why was Russia radicalized in the left-wing direction by World War I, whereas Germany was radicalized in the right-wing direction? Not to mention that Germany had a much more vibrant social democratic tradition than Russia. This is related to the whole question of why communism happened first in peasant societies, which was the opposite of Marx's prediction.

• What distinguished the people like Churchill who immediately saw the folly of both socialism and fascism? What were they able to see that others missed?

• It’s interesting to me that other leaders of the revolution like Trotsky, Lenin, etc have written extensive manifestos. And Stalin is considered the ‘outstanding mediocrity’ because despite having many articles to his name, he isn’t a prominent intellectual. It seems that these other leaders thought that intellectualizing was all you needed, and day-to-day administrative ability was a peripheral concern.

• Do you think a lesson from this period of history is that one should fall over themselves to support the lesser of two evils?

  • The Russian examples seem to support this. Arguably, liberals should have backed the authoritarian (but reformist) Stolypin, and the Provisional Government factions should have united more decisively to prevent the greater evil of the Bolsheviks.

  • Yet this lesson seems to directly contradict the German example, where conservatives under Franz von Papen did support what they perceived as the lesser evil, Hitler, to stop the Communists, leading to a catastrophic outcome.

• Why did Marxist revolutions always happen in places Marx thought worst for them? Marx saw industrialization as a prerequisite for revolution; most actual Marxist revolutionaries came from agricultural soceities and ended up doing really terrible things to try and jump start industrialization. What did Marx and Engels not understand here?

• So Stalin ends up being this inhuman administrator and micromanager. But he’s happy spending over a decade plus before the Bolshevik revolution just hanging out in Siberia, impregnating farmer girls, and other small time shenanigans. Wouldn’t you think that someone who has that much aptitude for organization and leadership would get up to something more ambitious, even if revolutionary?

• There’s an interesting book called Stalin’s Library, which goes through Stalin’s wide and diverse readings of literature, science, history, and economics. And the book notes that in none of Stalin’s extensive markups does he even hint at doubting Marxism. How can this be? How can someone who has been exposed to so many different ideas not even come across something which makes him doubt his principles? Does this make us question the whole value of humanistic education?

  • What's the best way to think about what Marxism and Stalinism were? How did so many people who consider themselves part of the intelligentsia develop this almost religious attitude towards criticism of the party? Why were they so convinced that history was inevitably headed in this direction? Is Marxism a replacement religion, a secular apocalyptic cult, the psychological equivalent to a spiritual experience?

• How do we explain the surplus of sadism in Russia, which allowed Stalin to enforce collectivization and the great terror? The 25 thousanders, the tens of thousands of interrogators and torturers in the gulag system.

• What if the Politburo members had simply accepted Stalin's resignation? Would there be any face-saving way for him to still come back? Or would that have been it? Moses tried to pull a similar trick on every single mayor in New York City until Governor Rockefeller actually just accepted it, and nothing bad happened. His bluff was called.

• Did any old Bolsheviks ever express regret about the whole revolution once fruits of the revolution turned sour? Not just about letting Stalin get so much power, but the whole Marxist uprising thing in general?

• Suppose someone else had won the succession struggle after Lenin. What would Stalin have done if he had found himself as a Politburo member serving someone else’s dictatorship within a dictatorship in 1930? Would he have been able to instigate the kind of conspiracy that nobody dared assemble against Stalin?

• Say what you will about the Bolshevik revolutionaries, they certainly had balls. Why did so many of them fold to pressure? Why did Old Bolsheviks like Nikolai Bukharin and Grigory Zinoviev, who had faced down the Tsar, confess to fabricated charges of treason during Stalin's public Moscow Trials? Knowing execution was certain, why didn't they use that final platform to expose Stalin's tyranny? Similarly, during the Cultural Revolution, why didn't a figure like Premier Liu Shaoqi, once Mao's designated successor, use a final speech at a party plenum to launch a desperate, last-stand denunciation of Mao as he was being systematically purged and humiliated? In all these cases, there was an extensive period where they were basically a dead man walking, but still in their normal positions of power.

• Stalin is personally choreographing many of these show trials. So at some level he must have known that they were mistaken. Why does he do it nonetheless?

• Why doesn’t the (more than) decimitation of the Red Army instigate a military coup?

• Relatedly, in Tsarist Russia, ministers - and even tsars - are getting assassinated left and right. In your coda for the first volume, you talk about how easy it would have been for someone to assassinate Stalin. This is someone who a 100 million peasants he has enslaved have something against, and who dozens of top leaders have good reason to fear. Why does no one even try to kill him?

• So you say that Stalin was hoping for a war between the capitalist powers because he thought it would expand the reach of communism, just as the First World War had. Given how outrageously successful he was in using the Second World War to increase the global footprint of Communism, did Stalin wish for a Third World War?

• Stalin seems oddly good at foreign policy. I think in a lot of alternate worlds, either Chiang Kai-Shek completely wipes out the Chinese Communist Party instead of creating a united front with them (which gave the communists a lot of unearned prestige, given that they were supplying literally 1/40 of the troops as the nationalists), or alternatively, he is just killed and kidnapped by the Communists, which would likely force a Japan collaborationist government in China.

  • One big update from your biographies was that Stalin was very reluctant to promote socialist or communist parties abroad, including in Spain and China.

  • He seems much more successful and less ideological with foreign policy than domestic policy.

• If Bolsheviks don’t take power in Russia, does Communism just not become a historical force. Maybe Europe still goes social democratic, but you don’t have brutal leftist dictators take control in China, North Korea, parts of Africa and South America? Or do you think this kind of Marxist-Leninist political philosophy is enough of an attractor state that it would have arisen independently?

• If Stalin is supposed to be this dedicated Marxist, why is he running a more pragmatic, less ideological foreign policy? Half hearted support of Communists during Spanish Civil War, trying to get Mao to stop at the Yangtze River when chasing down Chaing Kai Shek

• Why doesn’t communism have a bigger impact on the longer run growth trajectory of Russia? Do you suspect that without communism, Russia might have been hyperbolic instead? Do you think without communism, due its large population (1.29x US in 1939) and resources, Russia might have ended up with a bigger economy than even America (which it never had during the real Cold War).

• • Is it possible that while less than optimal, communism actually was somewhat workable in the early 20th century because technology just required massive investment, didn’t require that much signal from consumer demand?

• Was Stalin right that communism wouldn’t survive in the long run if most of the economy (which was agriculture at that point) was run on a capitalist basis? Of course, it would have been a good thing if communism collapsed. But nonetheless, given his political philosophy, was he empirically correct about the right next action?

• As of 1924, did the communists really believe that you could just get socialism without coercion? How exactly was that supposed to happen? To the extent it was supposed to be from state companies just freely out-competing small peasants, as Bukharin hoped, isn't that just capitalism?

• You have an interesting take that “the package of attributes that we call modernity was a result not of some inherent sociological process, a move out of tradition, but of a vicious geopolitical competition in which a state had to match the other great powers”.

  • Do you think that countries adopt tech faster than we might expect because of this overwhelming geo-political pressure to modernize? If so, do you think that we are overrating the regulatory or political backlash to the adoption of AI? Will we feel compelled to race on AI against China the same way that late 19th century Russia felt compelled to race against Germany on industrialization?

• Do you think this vicious geopolitical competition still leads to the adoption of more advanced government/tech, given that wars of colonization and territorial conquest have subsided? There's no amount of fucking up that some European country could do today which would actually get its territory physically dismembered by another country in Europe.

• You note how the early 20th century had seen more technological change than any time before (and probably even since) - planes, tanks, cars, airplanes, radio, extension of railways and steamships, further mechanization, etc etc. Is there any way this pace of change is implicated in the chaos and extremism in Russia (and Europe overall)?

• At a high level, how do you think about the differences between the regime change in 1917 and 1991? Why doesn’t the latter produce any kind of grand ideology?

• Does China today share to any significant degree the weaknesses of the Eastern block before 1989? There is falling productivity in the last-ditch effort to make up for it by investing heavily in some technological miracles. But certainly they have the opposite of Polish disease, in that people are concerned they’re exporting too much, right?

• China has preserved the Leninist core of the Soviet system (with party control over state, cadres running everything, state subsidization of militarily relevant heavy industries and technologies) but not the Marxism. Does it disprove Stalin's claim that political monopoly cannot survive capitalism?

• Why does collective leadership always seem to devolve into one person dictatorship in practice? Why is it an unstable political equilibrium? Soviet Union under Stalin, China now under Xi, etc.

• Say you are Xi Jinping. What lessons from Stalin would you be heeding? What lessons should you be heeding but maybe are not?

• Xi Jinping makes a big deal about how the Soviet Union fell in part because it repudiated Stalin (ergo, China must not repudiate Mao). What do you think of this judgement?

• Every morning you wake up, you get to choose whether you will either go through the next many dozen papers in the Stalin archive and write the next few pages of the meetings he conducted on some given day, or whether you will go on some podcast and attract a million people through your personality and reservoir of scholarship. How do you make that choice?

Special thanks to Ege Erdil and Tanner Greer (see his here), but also many others, for excellent suggestions for questions.

I sometimes joke that on a viewer minute adjusted basis, I host the Sarah Paine podcast, where I sometimes also talk about AI.

I am delighted to announce that Sarah is coming back for a 6 lecture series in July. We will record the lectures in person in San Francisco. As a token of gratitude to my paid subscribers, below are the links to get free tickets one day in advance of anybody else.

Sarah has become super famous now (and rightfully so). But just in case you’re hearing about her for the first time, she is Professor of History and Strategy at the Naval War College. She is not only by far my most popular guest to date, but with the 3 additional lectures we recorded last year, she now constitues 4 out of my 5 most popular episodes.

Sarah is a true authority on the history of conflict and strategy, and has spent decades unearthing insights in archives all over the world, from China to the Soviet Union to Taiwan to Japan. Her lectures distill the lessons from her many decades of ground-breaking scholarship. I’m excited to experience her insights on 6 fresh new topics live with all of you in person.

If you can’t join us in San Francisco, please do not worry - all these lectures will be made available online within a few months.

George Church is the godfather of modern synthetic biology and has been involved with basically every major biotech breakthrough in the last few decades.

Professor Church thinks that these improvements (e.g., orders of magnitude decrease in sequencing & synthesis costs, precise gene editing tools like CRISPR, AlphaFold-type AIs, & the ability to conduct massively parallel multiplex experiments) have put us on the verge of some massive payoffs: de-aging, de-extinction, biobots that combine the best of human and natural engineering, and (unfortunately) weaponized mirror life.

Watch on YouTube; listen on Apple Podcasts or Spotify.

• WorkOS Radar ensures your product is ready for AI agents. Radar is an anti-fraud solution that categorizes different types of automated traffic, blocking harmful bots while allowing helpful agents. Future-proof your roadmap today at workos.com/radar.

• Scale is building the infrastructure for smarter, safer AI. In addition to their Data Foundry, they recently released Scale Evaluation, a tool that diagnoses model limitations. Learn how Scale can help you push the frontier at scale.com/dwarkesh.

• Gemini 2.5 Pro was invaluable during our prep for this episode: it perfectly explained complex biology and helped us understand the most important papers. Gemini’s recently improved structure and style also made using it surprisingly enjoyable. Start building with it today at https://aistudio.google.com

To sponsor a future episode, visit dwarkesh.com/advertise.

(0:00:00) – Aging solved by 2050

(0:07:37) – Finding the master switch for any trait

(0:19:50) – Weaponized mirror life

(0:30:40) – Why hasn’t sequencing/synthesis led to biotech revolution?

(0:50:26) – Impact of AGI on biology research progress

(1:00:35) – Biobots that use the best of biological and human engineering

(1:05:09) – Odds of life in universe

(1:09:57) – Is DNA the ultimate data storage?

(1:13:55) – Curing rare diseases with genetic counseling

(1:22:23) – NIH & NSF budget cuts

(1:25:26) – How one lab spawned 100 biotech companies

Dwarkesh Patel 00:00:00

Today I have the pleasure of interviewing George Church. I don't know how to introduce you. This is not even an exaggeration, it would honestly be easier to list out the major breakthroughs in biology over the last few decades that you haven't been involved in— from the Human Genome Project to CRISPR, age reversal to de-extinction. So you weren't exactly an easy prep.

Okay, so let's start here. By what year would it be the case that, if you make it to that year, technology in bio will keep progressing to such an extent that your lifespan will increase by a year, every year, or more?

George Church 00:00:35

Escape velocity is sometimes what it's called for aging. Different people have estimates and all those estimates, including mine, are going to be taken with a big grain of salt. Mainly looking at the exponentials in biotechnology and the progress that's been made in understanding—not just understanding causes of aging, but seeing real examples where you can reverse subsets of the aging phenotype—you're getting close to all of aging.

In other words, instead of just saying, “Oh, I'm going to fix the damage in this collagen in this tendon, in this limb”, you're saying, “Oh, I'm going to change a lot of things that are common to age-related diseases and I'm going to get more than one at a time.”

Looking at those two phenomena—the exponentials in biotechnologies and the breakthrough in general aging, not just analysis but synthesis and therapies, and a lot of these therapies now making it in the clinical trials—I wouldn't be surprised if 2050 would be a point. If we can make it to that point, 25 years… Most people listening to this have a good chance of making it 25 years. The thing is, it's not going to be some sudden point where you're going to be so sick 25 years from now that it's like hit or miss. It's more likely that you're going to be healthier 25 years from now than you thought you were going to be.

There may be some, probably not some law of physics, but some economic or complexity issue that we don't know about that becomes a brick wall. I doubt it seriously, but we'll have to see.

Dwarkesh Patel 00:02:31

Given the number of things you would have to solve to give us a lifespan of humpback whales…

George Church 00:02:39

Bowhead whales, 200 years.

Dwarkesh Patel 00:02:40

Sorry, yeah. Is there any hope for doing that from somatic gene therapy alone, or would that have to be germ line gene therapy?

George Church 00:02:45

Probably there's a lot of forces pushing it towards somatic. For one, there's 8 billion people that have missed the germline opportunity. That’s to say, it doesn't apply to us, the two of us and everybody listening to this. You have to be very cautious when you say something's impossible. It's safe to say it's impossible to do it this second, but you don't know what's going to happen tomorrow in the next decade or something.

I think there's a lot that could be done. In particular, since aging is a fairly cellular phenomenon—with proteins going through the blood and other factors going through the blood, signaling and so forth—you could imagine that if you replaced every nucleus in the body, it would suddenly be young again without going all the way back to the embryo and forward again.

There's various other things that are just short of that. If you replace the cells, will they fit into that niche? They might displace the old cells. That's certainly within the realm of modern synthetic biology, for cells to take over niches.

I think the hardest part is the brain. Even there, even though the brain doesn't really use stem cells that much, you could artificially bring in stem cells and they could artificially fit into a circuit and learn the circuit and then displace the old ones in some way.

Dwarkesh Patel 00:04:25

A Ship of Theseus kind of thing in the brain.

George Church 00:04:27

Yeah exactly, Ship of Theseus, trying to maintain the connections and the memories. There's some fairly straightforward experiments that need to be done before we can really even estimate how hard that problem is.

Very often there's low-hanging fruit that people just think is improbable. But it's there because biology has all these gifts where it just hands over to us levers that we can flip. Like vaccines are this amazing gift that didn't have to exist, but they do.

Dwarkesh Patel 00:05:04

Is there an existing gene delivery mechanism which could deliver gene therapy to every single cell in the body?

George Church 00:05:11

There is nothing close to that today. But there's nothing, no law of physics, that would prevent it. Again, there's going to be practical considerations, like how many injections do you need to do to achieve that goal? But we're getting better at targeting tissues.

One of my companies, Dyno Therapeutics, showed they could get a hundredfold improvement in targeting neurons in the brain, which is a big deal. That was just one little campaign that they did. That one experiment involved a lot of AI and a lot of testing of millions of different capsids.

Capsids are fairly limited in the diversity and the structure that it can change to. But cells have even more possibilities. I think you could probably get delivery to everything… The question is how close to 100% do you need to get? It's going to vary from tissue to tissue.

For example, for some therapies you just need to get 1% because that 1% can produce some missing enzyme. And that 1% doesn't have to necessarily be in its normal place. You can turn a muscle into part of the immune system temporarily for a vaccine. An enzyme that's normally made in, let's say the brain, you could make in the liver, if the point is just to get it into the blood. So I think that's moving along quite well.

Dwarkesh Patel 00:06:53

You're one of the co-founders of Colossal which recently announced that they de-extincted a dire wolf. Now you're working on the woolly mammoth.

Do you really think we're going to bring back a woolly mammoth? The difference between an elephant and a woolly mammoth might be like a million base pairs. How do we think about the kind of thing we're actually bringing back?

George Church 00:07:13

People get worked up about whether we are trying to bring back, or have already, or will ever bring back a new species. If you think of it, rather than as a natural thing that we're trying to do, but as synthetic biology with goals that has potential societal… People also get worked up as to whether this could possibly benefit society in any way. Can we really fix an environment to suit humans or fix the global carbon to suit humans? The answer is we don't know. But it's worth a try, isn't it? Because it could be very cost-effective.

The other aspect of it is there's a whole discipline within synthetic biology of asking, “What's the minimum?” People often phrase it into, “What's the maximum? What can we do?” I'm interested in both. Yes, there's millions of differences between mammoths and elephants. There are millions of differences between elephant one and elephant two, within Asian elephants and between Asians and African. But not all of those are definitive in terms of what we would normally call them and how we would normally classify them, and what their functionality would be in an ecosystem.

There's this exercise that people do. We've done it, for example, with developmental biology. What's the minimum number of transcription factors it takes to make a neuron from a pluripotent stem cell? What's the minimum number of base pairs it takes to make something that will replicate to something that was done in mycoplasma originally? In a way, these are more interesting than, “Can we make a perfect copy of something?” What's the minimum things we have to do to make it completely functionally, or even functionally in a particular category? How do we make it bigger? We learn the rules for how to make things bigger, how to make things replicate faster, how to use new materials, etc.

With the dire wolf, we clearly didn't make an exact copy of a dire wolf. But it helped illustrate and educate people around the world on, “What is the difference between a gray wolf and a dire wolf?” Because direwolves, they're big. Maybe they have a particular coloration. The head components tend to be bigger than the leg components. How many genes do you need to do that? Maybe this was Direwolf 2.0, and we're going to go for 3.0 and successive approximations.

We might want to develop the technology for making exact copies of something, especially being able to make 100 variations on an exact copy. Because then there won't be any argument about whether you could make a dire wolf. It'll be a matter of what you should make and what would be most beneficial for the species that you're making, for the environment it lives in and for humans.

Dwarkesh Patel 00:10:34

Does this teach us something interesting about phenotypes which you think are downstream from many genes, and are in fact modifiable by very few changes?

Basically, could we do this to other species or to other things you might care about, like intelligence? Where you might think, there must be thousands of genes that are relevant, but there's like 20 edits you need to make, really, to be in a totally different ballgame.

George Church 00:10:55

You're hitting on a very interesting question. It's related to, “What's the minimum?” For example, you almost said it. Take a very multigenic trait in humans. Height is probably the most well-studied one, simply because no matter what gene, no matter what medical condition you're studying, you collect information on height and weight and things like that. Anyway, they tracked it down to something on the order of 10,000 genes, of which we have 20,000 protein coding genes. Some of them are RNA coding genes. They each have a tiny influence on height.

But if you take growth hormone, somatotropin, you have extreme examples where you'll get extremely low small stature and extremely high stature due to that one alone. In fact, it's used clinically as well for seven different medical treatments. That's a perfect example of how much we can minimize something, sometimes called reductionism. Reductionism isn't all bad. Sometimes it helps us bring a product into medicine. Sometimes it helps us understand or build a tool chest or a module that we could use in other cases and translate it to other species.

You hit on it just right. Not everything will translate, but we start accumulating these widgets. It's kind of like all the electronic widgets that we accumulate over time. If you just want to slap it into the next circuit, you might be able to.

Dwarkesh Patel 00:12:47

What implications does this have for gene therapy in general? What is preventing us from finding the latent knob for every single phenotype we might care about, in terms of helping with disabilities or enhancement?

Is it the case that for any phenotype we care about, there will be one thing that is like HGH for height? How do we find it?

George Church 00:13:09

With biology we've got a real gift which is that it's both much more complicated than almost anything we've designed from scratch, but it also is a lot more forgiving in a certain sense. You can have an animal or even a human that has two heads which, evolutionarily, there was no selection specifically to have two heads. But just a little deviation from the normal developmental pattern during fetal development and they both function fine. They control subsets of the body and they have their own personality, their own life. There's all kinds of things you can do in biology where you're working at a very high programming level. That’s a way of thinking about it.

But pushing us to a new level of intelligence is going to be very challenging and maybe not even urgent. To some extent, actualizing the people that we currently have would be quite impactful, just getting them all up to whatever speed they want to be up to within the range that's been demonstrated.

Some people are going to want to be like Einstein, some people won't. Some people will want to be healthy all the time. Unlikely, but some people might not. Some people might want to live to be 150, some people might want to die at 80. But if you give them that range, that capability… What if we had 8 billion, super healthy, don't need to worry about food and drugs, super healthy Einstein-level intelligence, education level the best we can come up with… That would be a completely different world.

Dwarkesh Patel 00:15:09

Just getting everybody to the healthy level, how much gene therapy would that take? It sounds like it wouldn't take that much, if you think that there are a couple of knobs which control very high level functions? So do you find them through the GWAS, genome wide association studies? Is it through simulations of these…

George Church 00:15:30

I would say mostly GWAS for humans, maybe for animals in general. For animals with synthetic biology, the smaller and the cheaper and faster replicating, the more experiments you can do. I don't want to overemphasize how single genes can do these amazing things. But there's also the possibility that multiple genes can be hypothesized and tested quickly.

For example I mentioned earlier, what's the minimum number of transcription factors it takes to turn a stem cell into a neuron? There's a bunch of recipes where you can do it with one. Maybe you want a specific neuron, you might need a few more. But then you can kind of quickly go to the answer by looking at each target cell type that exists. You can see what transcription factors does it express at the time that it's the target? Then you say, “Let's just try those on the stem cell and see if they work.”

That recipe has worked quite well. It's the basis of GC Therapeutics and a bunch of the work that we do. You can get a recipe for almost every cell type in the body. Now, that's not new cell types, but at least you've learned, to your point, about reducing the number of genes we need to manipulate in order to get to a particular goal. Here's a whole series of goals, and we can get them with 1, 2, 3, maybe 7 changed transcription factors.

That's an example. There's room for lots of other examples of where you can do reduction and do not just reductionistic biology, but then constructionistic. You take it back up and make a whole complex system and see what happens. Then you can do lots of those combinations and you debug them and so forth. Some of these things you can do… In vitro things you can do probably on the order of 10^14, 10^17. Things that involve cells are typically in the billions. But this is how we're going to get inroads into the very complicated biological systems.

Dwarkesh Patel 00:19:07

Can I ask you some questions about biodefense? Because some of the stuff you guys work on, or quite responsibly choose not to work on, can keep one up at night.

Mirror life. Given the fact that it's physically possible, why doesn't it just happen at some point? Some day it'll get cheap enough. Somebody will care about it enough, that somebody just does it. What's the equilibrium here?

George Church 00:19:29

I was a co-author on a paper that warned about the dangers of mirror life. Just like I wrote a paper long ago about the dangers of having the synthetic capabilities we have for making synthetic viruses, and to some extent of having new genetic codes. They have a few things in common.

The advance that we were recognizing, in our Science paper that was warning about mirror life, is not only that we had to calculate the possibility of error-prone escape or something like that was. We don't want anything to escape that we made in the lab unless there's a general societal consensus that it's a good thing. So far there aren't any examples of that.

Mirror life, if it can be weaponized, that would take it to a whole other level of concern. The concern was that if we got it to a certain point, then it would be easy to weaponize it. Again, there's practical considerations that maybe that most people who consider weaponizing mirror life would probably be satisfied with weaponizing viruses that already exist, that are already pathogens. And they wouldn't want to destroy themselves and their family and their legacy and everything like that. But all it takes is one, one group probably, or one person.

But your question is, is it inevitable? I don't know. It might be. It's quite possible it's already here. In other words, we already have mirror life in our solar system or maybe even on our planet. It just hasn't been weaponized.

What we were saying in the Science paper is that this seems like the sort of thing that could wipe out all competing life if were properly weaponized. But there are probably a few things like that. What we really need to do is reduce the motivation to do that, maybe increase our preparedness for a variety of existential threats. Some of which will be natural, some of which will be one disgruntled person who has essentially too much power.

Over the history of humanity, the amount of things that a single person can do has grown very significantly. It used to be, when you had your bare hands, there was kind of a limit to what one person could do. A large number of people could team up and get a mammoth or something like that. Today, one person with the right connections or right access to technology could blow up a city. That's a huge increase in capability. I think we want to start dialing that back a little bit somehow.

Dwarkesh Patel 00:22:35

What does that look like in terms of not just mirror life, but synthetic biology in general? Maybe we're at an elevated period of the ratio to offense and defense. How do we get to an end state where—even if there's lots of people running around with bad motivations—somehow there are defenses built up that we would still survive, where we're robust against that kind of thing? Is such an equilibrium possible? Or will offense always be privileged in this game?

George Church 00:23:07

Offense awfully does have an advantage, but so far we haven't… We made it through the Cold War without blowing up any hydrogen bombs, as far as I know, accidentally or intentionally on enemies. We did do two atomic bombs.

But a lot of that is based on the difficulty of building hydrogen or atomic bombs. The thing that's alarming to people like me is that biotechnology enables smaller and smaller efforts that are harder and harder to detect, more and more subtle to the stochastic variation between people. There's some people that are just so happy they would never want to do anything close to that. Or they're so responsible or ethical or whatever. Then there are other people who, whenever they have a bad day, they want to take a lot of people with them.

Maybe some progress in psychiatric medicine would help. Again, you don't want to force that on people. You want to make sure that if they don't want to get cured, you can't force them, but you can make it available to them that might help.

Dwarkesh Patel 00:24:31

Hopefully there's a more technological solution or more robust solution than that.

George Church 00:24:35

Well, there will be technological solutions to the psychiatric problem. It could be that even people who aren't sure whether they want to be helped or not can test, try it out, and it's reversible. They say, “Yes, I like that better.” Okay, let's try that then.

Then there's other things that cause you to have bad days. It's not just your psyche. It's also the environment. So if you're surrounded by your people being starved, infectious disease, or you’re being shot at or something like that, those are things that are subject to sociological and technological solutions. If we could really solve a lot of that stuff, we could reduce the probability that one person…

Dwarkesh Patel 00:25:22

This is maybe pessimistic because you're basically saying we have to solve all of society's problems before we don't have to worry about synthetic biology, which I'm not that optimistic about. We'll solve some of them.

George Church 00:25:32

Right? You shouldn’t be. I'm not trying to reassure you. We're having a conversation about what it takes and that might be one scenario for what it might take.

Dwarkesh Patel 00:25:42

You had an interesting scheme for remapping the codons in a genome so that it's impervious to naturally evolved viruses. Is there a way in which this scheme would also work against synthetically manufactured viruses?

George Church 00:25:59

It’s much harder. Again, the offense has the advantage. We can make a lot of different codes.

Dwarkesh Patel 00:26:06

Which will limit the transmissibility?

George Church 00:26:09

Yeah. So one interesting thing is that there's only two chiralities. There's the current chirality and the mirror chirality. But there's maybe 10^80 different codes. Some of them you might be able to take out all at once. Anyway, coding space is a kind of more interesting space.

Of course it could get even more complicated than that because the 10^83 is based on triplet codons and that sort of thing. But if they're quadruplet codons or novel alphabets and so on…

We're sort of getting into a cycle of competition. It'd be better to nip it in the bud. Why did we spend so much societal resources building up to tens of thousands of nuclear warheads? Now we've dialed it back to mere thousand nuclear warheads. It's nice that we dialed it back, but why did we waste all that time and money?

Dwarkesh Patel 00:27:22

Biology seems very dual-use, right? The mere fact that you, literally you, are making sequencing cheaper will just have this dual-use effect in a way that's not necessarily true for nuclear weapons. And we want that, right? We want biotechnology to advance.

George Church 00:27:37

It's hard to pound nuclear weapons into ploughshares, as they say.

Dwarkesh Patel 00:27:41

I guess I am curious if there is some long-run vision, where… To give another example, in cybersecurity, as time has gone on, I think our systems are more secure today than they were in the past because we found vulnerabilities and we've come up with new encryption schemes and so forth.

Is there such a plausible vision in biology or are we just stuck in a world where offense will be privileged so we'll just have to limit access to these tools and have better monitoring but there's not a more robust solution?

George Church 00:28:14

One of the things I advocated in 2004 is that we stop deluding ourselves into thinking that moratorium and voluntary signups to be good citizens is going to be sufficient. We need to also have surveillance and consequences, and mechanisms for whistleblowers to make it easy for people to report things that they think are out of line.

We had essentially moratoria and disapproval for germline editing. Nevertheless, somebody did it and a lot of people knew about it. That was clearly a failure of the whole moratorium and voluntary and whistleblower components.

Dwarkesh Patel 00:29:06

It worked for five years with only one defector. That's quite impressive.

George Church 00:29:09

Okay, half empty, half full, I'll give you that. But all it takes is one for some of these scenarios. It would have been nice if the whistleblowers could have saved him the three years in prison by getting an intervention.

It's not like anybody died. Right. There are probably three healthy genetically-engineered children in the world now. They’ll be teenagers soon. But it was a good test run that shows a failure of the system. We need to have better surveillance of all the things we don't want and consequences that are well-known.

Dwarkesh Patel 00:29:57

Over the last couple of decades we've had a million-fold decrease in the cost of sequencing DNA, a thousandfold in synthesis. We have gene editing tools like CRISPR, massive parallel experiments through multiplex techniques that have come about. Of course much of this work has been led by your lab.

Despite all of this, why is it the case that we don't have a huge Industrial Revolution, a huge burst of new drugs, or cures for Alzheimer's and cancer that have already come about? When you look at other trends in other fields, we have Moore's law and here's my iPhone. Why don't we have something like that in biology yet?

George Church 00:30:34

We have something that's about the same speed, a little bit faster than Moore's Law in biology. It's more recent, that’s one aspect of it. We could stand on the shoulders of the electronics giants to go a little bit faster to catch up.

I would say we do. We have the biotech industry, which has used that exponential curve to get better. It's also possible we're close to the big payoff is the other aspect, or the beginning of the big payoff. Right now we have miraculous things like cures for rare diseases. We have vaccines. We have a trillion dollars, probably, of various biotech related things if you go far enough apart.

We're on the verge of really combining electronics and biology more thoroughly, and AI and biotech. It seems like we're on the same track as Moore's law, if not better.

Dwarkesh Patel 00:31:49

What exactly are we on the verge of? What does 2040 look like?

George Church 00:31:52

Well with 2040 we're talking about only 15 years. Which is maybe two cycles of FDA approval.

Dwarkesh Patel 00:32:04

2040 is post-AGI. It's a long time.

George Church 00:32:07

Well, I hope it's not post-AGI. I think we're rushing a little bit to get to AGI. There's lots of cool things we can do with just super AI, but we need to be very cautious with AGI. Anyway we can get into that.

Dwarkesh Patel 00:33:06

I have questions for you there.

George Church 00:33:08

We are shortening the time of getting medical products approved still in a safe way. But that's not going to completely change the exponential. It might reduce it from 10 years down to… One year is our record so far for say COVID vaccines. Maybe that'll be 10 times shorter. Maybe that will multiply out a little bit.

The big thing is that all our designs will become better so there'll be fewer failures. The cost per drug will drop. There'll be things that we didn't classically consider drugs or instruments, some sort of hybrid thing. But again that won't be completely shocking. It's just going to be so much of it. There's going to be lots of diversity of solutions.

Dwarkesh Patel 00:33:25

How much more are we talking about? Are we going to have 10x the amount of drugs? 100x?

George Church 00:33:29

I'm not even sure it's going to make sense, but 100x would not be completely surprising. Combinations of drugs will be important, using them intelligently. There'll be a lot more. Some drugs will affect everything, for example an age-related drug that could impact every disease.

I'm not sure the number is going to matter so much as the quality and the impact and intersection, and software that helps physicians and regular citizens make decisions.

Dwarkesh Patel 00:34:04

What specifically is changing that's enabling this? Is it just existing cost curves continuing or is it some new technique or tool that will come about?

George Church 00:34:12

The cost curves are affected by new tools. It's not just some automatic thing. There was a big discontinuity between Sanger sequencing and nanopores and fluorescent next-gen sequencing. Sometimes it's a merger of two things. Clearly AI merging with protein design caused a step function. These step functions get smoothed out into a smooth exponential, but there are lots of them.

The next set will probably be a merger of AI with other aspects of biology, like developmental biology. After that, the merger of developmental biology with manufacturing, conquering developmental biology. In other words, it’d be actually knowing how to make any arbitrary shape given DNA as the programming material. That would be a big thing.

Just having more materials in general. All the materials that we use in mechanical and electrical engineering should be made better by biotechnologies.

Dwarkesh Patel 00:35:28

Why is that?

George Church 00:35:29

Well in electronics, I wouldn't say Moore’s law is stopping, but think about what we would call the 1 nanometer process, which is supposed to come out in 2027 according to the roadmap. It's not really 1 nanometer, it's more like 40 nanometers, center-to-center spacing, typically in two dimensions, maybe a little bit of three dimensions.

Biology is already at 0.4 nanometer resolution and it is in three dimensions. Depending on how you count that third dimension, it could be a billion times higher density that biology is already at. We just need a little more practice with dealing with the whole periodic table. Even electrical and mechanical engineering don't typically use the whole periodic table typically, especially not at the atomic level. Biology is just really good at doing atomic precision.

Dwarkesh Patel 00:36:35

So then what's the reason that over the last many decades we do have, not atomic but, close to atomic-level manufacturing with semiconductors.

George Church 00:36:45

40 nanometers.

Dwarkesh Patel 00:36:46

Right. It's quite small.

George Church 00:36:48

It's a thousand times bigger than biology, linearly.

Dwarkesh Patel 00:36:51

But the progress we have made hasn't been related to biology so far. It seems like we've made Moore's law happen. People in the 90s were saying ultimately we'd have these biomachines that are doing the computing. But it seems like we've just been using conventional manufacturing processes. What exactly is it that changes that allows us to use bio to make these things?

George Church 00:37:14

A few things. One is the arrival of synthetic biology. We were already kind of doing synthetic biology before, we were doing recombinant DNA, a kind of genetic engineering. It was kind of in that direction. But synthetic biology really liberated us to think a little bit bigger. Even though it started kind of focused on E. coli and yeast, it enabled us to maybe think about new amino acids, for example.

If you start using the full periodic table with the amino acids, or what amino acids can catalyze, that breaks one of the major barriers. One of the major barriers between electrical and mechanical engineering and biology was the use of special materials, things that conduct electricity at the speed of light or conduct signals more generally.

But there are definitely polymers that biology can make that will conduct at the speed of light. We could make a mixed neuronal system that has conventional neurons and processes that conduct at the speed of light. That would be interesting.

So I think that our ability to design proteins was particularly difficult. Designing nucleic acids was great. You want two things to bind to each other? You just dial it up using Watson-Crick rules. If you want to make a three-dimensional structure, it's actually the one kind of thing where morphology is dictated by fairly simple rules. It's not how developmental biology works. We still need to figure out how that works. But DNA origami, DNA nanostructures really work.

But doing it for proteins was really, really hard until maybe eight years ago, something like that. I think we're just now getting used to it. The use of chips for making DNA. You said that DNA synthesis has come down a thousandfold, it depends on who you talk to.

When we came out with the first chip-based genes in a 2004 Nature paper, basically people dismissed it for about a decade. The only people that used it were collaborators and alumni. It wasn't even listed on the Moore's law curve for DNA synthesis, even though it was thousand times cheaper. It was just ignored.

Now we have claims of 10^17 genes that you can make libraries of, that aren't randomized in the usual sense, where you just do error-prone PCR or spiked nucleotides. 10^17, that's a lot bigger than a thousandfold if it turns out to be practical.

Dwarkesh Patel 00:41:19

Speaking of protein design, another thing you could have thought of in the 90s—People were writing about nanotechnology, Eric Drexler and so forth. Now we can go from a function that we want this tiny molecular machine to do, back to the sequence that can give it that function. Why isn't this resulting in some nanotech revolution, or will it eventually? Why didn't AlphaFold cause that?

George Church 00:41:45

Part of it is that nanotechnology as originally… The source of the inspiration, Eric Drexler, he wanted to reinvent biology in a certain sense but it already existed. So you don't need to design a diamond replicator because you already have a DNA replicator.

The question was, what was missing? What was motivating this reinvention of biology? It was materials. Biology is not that great with materials that are, say, superconductors, conductors, semiconductors, and light speed. But it's getting there. Rather than going the route of everything having to be based on first principle nanostructures, you can meet in the middle where biology can build things.

Of course, when you go down to liquid nitrogen and colder temperatures, biology as we currently know it stops functioning. It's not to say that you can't have things moving in liquid nitrogen, you can. But that hasn't been explored and doesn't really need to be, because if biology can build things that can operate at low temperatures… Or maybe biology now, because you can make these big libraries of biology, maybe 10^17 in vitro, you can flip through them quickly and you can barcode them and you can…

This is something that's never been done in electronics. I'm not saying you can't do it in electronics. But you haven't made a billion different kinds of electronic materials just in an afternoon, barcode them all and see who wins. But we do it all the time in biology now, at least since 2004 we have.

So I think that's an opportunity. We use those libraries to make much superior materials and we might even finally get a room-temperature superconductor that way.

Dwarkesh Patel 00:43:57

From bio?

George Church 00:43:59

It's possible, from libraries. We call it chemical / biochemical / exotic material libraries. The point is that they're libraries. They're essentially based in some sense on polymers, even though pieces of them don't necessarily have to be polymers.

Dwarkesh Patel 00:44:14

Do you have a prediction by when we'll see this materials science revolution? What is standing between now and that? Because we've got AlphaFold right now. So what is the thing that we need? Do we need more data?

George Church 00:44:27

AlphaFold is very nice, but it's only part of it. There are large language models that are different from AlphaFold. To give an example, with AlphaFold—last time I checked anyway—if you substitute an alanine for a serine in a serine protease, it will have exactly the right fold. It will be precise to a fraction of Angstrom overall average. But it won't function. It just won't function.

That's where you need either extraordinary precision or just knowledge of what happens evolutionarily, or happens in experiments, to say that, “No, alanine won't work. Okay?” So I think there's all kinds of combinations of AI tools that can give you deeper insight into that.

Dwarkesh Patel 00:45:20

If AlphaFold predicting the structure doesn't tell you whether the thing will actually function, then what is needed before I can say, “I want a nanomachine that does X thing, or I want a material that does that Y thing, and I can just get that"?

George Church 00:45:33

The way that it's working now—which will get us a long way, won't get us the whole way—is we have something that kind of works and we make libraries inspired by that. We make variations on it and then whichever of those variations work, we make variations on that. We can just keep going. It's kind of like the way evolution worked, except now we can do it at incredibly high speeds.

In principle, evolution might incorporate a few base pair changes in a million years. Now we can make billions of changes in an afternoon. It's all guided in such a way that you get rid of the wastefulness of having a bunch of neutral mutations and a bunch of lethal mutations. You can have things that are quasi-neutral but likely to be game-changing and have more of a focus on those.

Another thing that's been missing. None of the AI protein design tools that I know of are particularly good at it yet but as we speak, we're trying to improve it, is nonstandard amino acids. Because a lot of these tools depend on adding libraries of 3D structures, which use 20 amino acids, and large language models where you line up all the sequences of 20 amino acids. We have very little experience with extra ones.

But there's a revolution going on in generating nonstandard amino acids, where the amino acids can either have as part covalent part of them, or as easily liganded, all the stable elements in the entire periodic table. Each of those we’ll have to blend in and train our models on. But as soon as that comes in then we're going to have a whole series of new materials very quickly.

Ultimately, the determination of the functionality of your library is a kind of computer. You use AI to design the library optimally. You avoid things that are really neutral and really seriously damaged. But the stuff in the middle, you actually play it out, not in a simulation, but in real life.

But it's so inexpensive and it's so fast and it's so exact. It's a hundred percent precision, because you're not simulating. You're not making assumptions. You're not going from quantum electrodynamics, which is an assumption, to quantum mechanics, which is an assumption to molecular mechanics, which are full of assumptions. You're really doing the real thing.

So you're doing a kind of natural computing. Then you can take that data and harvest it in various ways very efficiently, pump it back into the more conventional AI, and do another round of it.

Dwarkesh Patel 00:48:36

If I listen to these words, it seems like I should be expecting the world to physically look a lot different. But then why are we only getting a couple more drugs by 2040?

George Church 00:48:46

Well, I didn't mean to stop there. I knew the conversation would continue. I'm not pinning down a particular year either, but this is poised to go pretty quickly. There are very few practitioners, which is the thing that will stop it for a while. Materials actually should go faster though, because they don't require quite as much regulatory approval.

It's one of these things where when you get the right idea, it's not hard to recruit people. For example, when Feng Zhang and my labs brought out CRISPR, we each got 10,000 requests in the next two months for people that wanted to duplicate the system. That's what I hope will happen with the nonstandard amino acids and using AI for protein design and making new materials. Hopefully that will recruit tens of thousands of people overnight.

Dwarkesh Patel 00:49:43

Are you more excited about AI which thinks in protein space, or capsid space, just predicting some biological or DNA sequences? Or are you more optimistic about LLMs just trained on language, which can write in English and tell you, “Here's the experiment you should run” in English. Which of those two approaches, or is it some combination, when you think about AI and bio is more promising?

George Church 00:50:09

I'm much more excited about scientific AI than I am about language AI. With languages, we're in pretty good shape already. What worries me is that to get to the next level of language requires AGI or ASI. That's very dangerous.

I don't think we have quite figured out how to handle that. There's a lot of safety organizations and a lot of safety rules and so forth. What typically happens when there's an intense competition is those safety rules get undermined and pushed aside. Even if they weren't, I don't think we understand our own ethics well enough to educate a completely foreign type of intelligence. We barely know how to pass it onto the next generation of humans.

So we need time to sort that out. There's no rush. This is a completely artificial emergency. This is not like COVID-19, where millions of people were dying if we delayed the science. This is something where, if there ever is a crisis, it's because we created it, it's not because we're trying to solve it.

So I think we need to go very slowly on AGI and ASI, and double down on slightly narrower scientific goals. With even that, we need to be very cautious. We need to have kind of an international consensus on what constitutes safe AI.

Dwarkesh Patel 00:51:47

Suppose we did build safe superintelligence. How much would that speed up bio progress? There's a million George Churches in data centers just thinking all the time, is it a 10x speed-up?

George Church 00:52:00

I think it would slow it down. I think it would eliminate it, because the first thing it would conclude is biology is not relevant to me because I'm not made out of biology.

Dwarkesh Patel 00:52:09

Suppose you could get them to care about it. There's a million copies of you in a data center. How much faster is bio progress? They can't run experiments directly. They're just in data centers. They can just say stuff and think stuff.

George Church 00:52:22

I don't think we have anything close to the assurance that we need that that would be safe. But let's put safety aside for a moment.

It's not only hard to calculate the bads, it's hard to calculate the goods. It could be a complete game changer. But on the other hand, it's like if we said we could get instantaneous transport all over the Earth. Well, we could say, “Yes, that could be a game changer.” But do we really need it? Is that really important?

Maybe it'd be more interesting to just have Zoom calls that are better, or we can just learn how to get everything we want in our kitchen and we don't need to travel anymore. So be careful what you ask for. You could tip our priorities towards something that we really don't care about, that we shouldn’t care about, or we might wish we didn’t care about.

Dwarkesh Patel 00:53:26

But I'm curious, you've still got to run the experiments, you still need these other things. So does that bottleneck the impact of the millionth copy of you or do you still get some speed up? Basically, how much faster can biology go if there are just more smart people thinking, which is a sort of proxy for what AI might do?

George Church 00:53:44

These are great questions and I don't want to misrepresent that I know the answers. But it's like the question of, “If you have nine women, can you do pregnancy in one month?” No, not at present.

Dwarkesh Patel 00:53:58

But you're working on that, right?

George Church 00:54:02

No, but the same thing is that there may be certain things that don't take a lot of people. We just don't know. We don't have that much experience with having thousands of Einstein-type levels of creativity and intelligence simultaneously in a generation.

In fact, it's probable that we're all capable of being a bit more efficient if we don't have distractions of mental illness, or taking care of other people. Now, taking care of other people may be a very good thing. Maybe if we have no one to take care of, there'll be something bad that happens to us socially.

So these things are very complicated and hard to predict. I think right now, the baby step, or actually the pretty big baby step, is to eliminate diseases or at least make it possible for people to eliminate their own diseases as they see fit.

Dwarkesh Patel 00:55:11

You've worked on brain organoids and brain connectome and so forth. How has that work shifted your view on fundamentally how complex intelligence is? Are you more bullish on AI because you realized the organoids are not that complicated, or rather very little information is required to describe how to grow them. Or are you like, “No, this is actually much more gnarly than I realized.”

George Church 00:55:36

I always felt it was very gnarly. I also felt that it was something that we could engineer. Certainly we have made a lot of progress at the broken end of the spectrum where the brain is severely challenged relative to average. There are a huge fraction of genetic diseases that have as one of their consequences the child being developmentally delayed to such an extent that it's lethal or causes a lifetime deficit.

We know the genes involved and we know how to do genetic counseling in some cases, gene therapy and other therapies to deal with it. At the other end, we have reduction of cognitive decline by cognitive enhancement, which is showing some promise. But again, that's kind of like this early stage severe impediment to cognition having a late stage component.

But what about, how much information does it take to encode a brain? I'm not sure that much less genome is required than if you just wanted to just make a brain, because the brain is totally entangled with the body. You have 10^11 neurons, 10^14 synapses. If you wanted to reproduce a particular brain, it's speculative as to whether it would be easier to do that by making a copy of it in silico, in some kind of inorganic matrix, or making a copy of it. Both of those are going to be hard.

I would say that if you wanted to make a copy of a complicated book, it would be easier to take photographs of each of the pages than to completely translate it into another language—trying to get all the nuances of the poetry and so forth—if your goal is just to replicate it. The same thing might be true of the brain.

But replicating a brain probably involves a lot more information than synthesizing it. Just to define this, 10^14 synapses is going to take a lot more bytes than the genome, which is billions rather than 10^14. But there might be reasons that you want to replicate a particular brain configuration rather than just make another animal that starts from scratch as an infant.

Dwarkesh Patel 00:59:52

Going back to the engineering stuff, often people will argue, “Look, you have this existence proof that E. coli can duplicate every 30 minutes. Insects can duplicate really fast as well. But then with our ability to manufacture stuff with human engineering, we can do things that nothing in biology can do, like radio communication or fission power or jet engines.”

How plausible to you is the idea that we could have biobots which can duplicate at the speed of insects—there could be trillions of them running around—but they also can have access to jet engines and radio communication and so forth? Are those two things compatible?

George Church 01:00:33

Certain things seem incompatible. Like the temperature of a fission reactor isn’t obviously compatible. But it is a possibility that a biological system can make other things. For example, it can make a nest. A bird can make a nest. You consider the whole nest as part of the replication cycle of the bird.

So you can say the biological thing that replicates at a 30 minute doubling time could make a nuclear reactor. That would be its nest but you need to expand its range of materials. In a certain sense, we do this already. Humans are a biological thing that replicates not in 30 minutes, but in 20 years or less. Is that fundamentally limiting us? Yeah, probably it is.

But it's amazing to think about. What if you could take a cornfield or a nuclear reactor, and suddenly 30 minutes later you've got two of them, then four of them, and eight of them. That's quite an interesting concept.

I teach a course called How to Grow (Almost) Anything. I work with Neil Gershenfeld at MIT who has a course called How to Make Almost Anything. We're trying to meet in the middle where his mechanical electrical engineering will meet with our biological.

In fact, neither of us can make or grow almost anything because there's all kinds of little gaps and things that are very hard to make in a small lab. There are things all over the world that depend on multi-billion dollar fabs to make things. But we're eating away at it.

Maybe a smaller baby step than making a nuclear reactor is making a phone. You said radio communication. It should be a small challenge goal for the synthetic biology community, maybe iGEM or something: make bacteria make a radio.

Actually Joe Davis is an artist—he’s been affiliated with my lab and before that, Alex Rich's lab—and he did make a bacterial radio but it was kind of more on the art end than on the science end. But I think that would be a good goal.

Dwarkesh Patel 01:03:15

What would it take to do whole genome engineering to such a level that for even a phenotype which doesn't exist in the existing pool of human variation, you could manifest it because your understanding is so high. For example, if I wanted wings.

Is the bottleneck our understanding? Is the bottleneck our ability to make that many changes to my genome?

George Church 01:03:41

Part of this has to do with just learning the rules of developmental biology, like I said. We can determine morphology at the molecular level now: proteins, nucleic acids. Determining at the cellular multicellular level, there's a lot more things you can do and a lot faster. But we don't know the language yet.

I think we're on the cusp of getting the tools to do that, like the transcription factor I was talking about earlier, harnessing migration, gradients of diffusion factors, chemotaxis and so forth. That's one thing we need, but there's a bunch of things we need, really.

Dwarkesh Patel 01:04:26

What discovery in biology—so not in astronomy or some other field—would make you convinced that life on Earth is the only life in the galaxy? Conversely, what might convince you that no, it must have arisen independently thousands of times in this galaxy?

George Church 01:04:45

Oh, I see what you're getting at. Astronomy might be that we would detect radio signals or light signals. With biology, the kind of evidence would be that you show in a laboratory using prebiotic conditions a really simple way to get life.

It's harder to prove the negative because we don't know all the possible prebiotic conditions. Probably the number was vast. You have 10^20 liters of water at various different salinities and drying up on the ocean and the sun and the lightning and all this stuff.

I think if you showed, reconstructed in the lab, a very simple pathway from inorganics, cyanide derivatives and reduced compounds, all the way up to some cellular replicating structure, that might lead us to believe that at least life exists.

Now that there are other parts of the Drake equation that might kick in. Maybe it's hard to get intelligent life because intelligence isn't necessarily in your best interest. And if you get intelligence life, it's hard to maintain that without societal collapse or without robotics taking over and then killing themselves. That's hard to do experiments on.

But to your question, an experiment that showed maybe multiple different ways of getting to a living system from non-living systems spontaneously would be interesting. Again, I'm not sure. It'd be very hard to prove the negative.

Dwarkesh Patel 01:06:41

Between intelligent life and some sort of primordial RNA thing, what is the step at which, if there is any, you say there's a less than 50% chance something like at this level exists elsewhere in the Milky Way?

George Church 01:06:57

These are very challenging problems. I'm not even sure we would be able to say within five orders of magnitude, much less 50%. I think it's more likely to come from exploration than it is going to be from simulation.

The sad truth is that almost none of the missions that we've sent outside of Earth have actually looked for life. They've had components that could have looked for life. A sad number of those had not enough components that could look for life. The ones that could look for life were not really looking for it. When we get positive results, we dismiss them as happened with Pioneer.

I think if we just start looking at the geysers that are coming out of various moons of Jupiter and Saturn, there's so much water. There's 50 times more water, liquid water, not frozen but liquid water, in our solar system than on Earth. Doesn't that seem likely that some of that would have been a good breeding ground? It could be that we need sunny shores where you have a lot of dry land right next to water. Maybe these are just giant oceans that are surrounded by ice and maybe that's not an ideal.

In any case, we need to look at those fountains to see what's popping up. That's a high priority. The same thing goes for water on Mars. That's maybe even more accessible. But until we've exhausted those, those are probably the easiest. They're hard. We’re still talking about multibillion dollar experiments, but I think they're a little more convincing. Again, it'll be hard to prove the negative. If we find this negative on everything in the solar system, there's so much more diversity out there that could have done it.

Dwarkesh Patel 01:09:14

If in a thousand years we're still using DNA and RNA and proteins for top-end manufacturing, the frontiers of engineering, how surprised would you be? Would you think, “Oh, that makes sense. Evolution designed these systems for billions of years.” Or would you think, “Oh, it's surprising that these ended up being the systems. Whatever evolution found just happened to be the best way to manufacture or to store information”?

George Church 01:09:38

I don't think I'd be surprised either way. I can imagine it going either way. I can imagine making truly amazing materials using proteins as the catalysts, or maybe in some cases as a scaffold as well as catalysts.

One thing that's probably already happening, we don't have to go a thousand years out, is that the number of amino acids is going up. It's going up radically from 20. I think pretty soon we'll have a system where we can have 34 new non standard amino acids being used simultaneously with the standard ones in a E. coli cell. 34 plus 20 is a lot bigger than 20.

I don't think we necessarily need more than four nucleic acid components. Certainly there are plenty of modified ones. There's a bunch of alternative base pairs, some of which don't even involve hydrogen bonds. So we could have more.

But I think the main thing is this information storage—whether it's bits, digital binary—it’s just zeros and ones. That works pretty well for 99% of what we do electronically. Having four is better than two maybe, but do we really need six? I don't know.

I wouldn't be surprised. Another possibility is if we changed the backbone of DNA. Maybe we keep the ACGT, but make it out of peptides now, a little bit smaller, a little bit more compatible. I don't know. It could be part of the new amino acid collection.

There'll be more. These are just things that my primitive 21st-century brain is coming up with. A thousand years from now, it'll be a whole new millennium.

Dwarkesh Patel 01:11:38

It makes sense why evolution wouldn't have discovered radio technology. But things like more than 20 amino acids or these different bases so that you can store more than 2 bits per base pair or for example, the codon remapping scheme, this redundancy, which it seems like based on your work there was this extra information you could have used for other things.

Is there some explanation for why 4 billion years of evolution didn't already give living organisms these capabilities?

George Church 01:12:10

I think that evolution has a tendency to go with what works. The investment in making a whole new base pair would have been high. We haven't even articulated what the return on investment would be. What do you get from that?

We have made systems, like Floyd Romesberg and others, where you have replication and transcription and translation with a new base pair. But it hasn't been clearly articulated what that gets you, even in technological society.

In technology you can jump to things where all the intermediates aren't incrementally useful. But evolution, as far as we know, is generally limited to… You have to justify every change, like some bureaucracy, “If you're going to put this sidewalk in, you have to justify that before you build a city.”

Dwarkesh Patel 01:13:12

We've talked about many different technologies you worked on or are working on right now, from gene editing to de-extinction to age reversal. What is an underhyped technology in your research portfolio which you think more people should be talking about but gets glossed over?

George Church 01:13:33

It's hard to say because as soon as you say it, it becomes hyped. If I've ever been asked this question before, it's too late.

One thing I think is very ripe and is very well-understood in a certain sense but is nevertheless ignored… The previous example I would have chosen was making genes out of arrays. Arrays were typically used for analytic, quantitating RNA, or something like that, the original Affymetrix-type arrays. But we turned them into gene arrays and people just weren't using it. It was in Nature. It was hidden in plain sight. But anyway it was somehow underhyped.

What I would say is genetic counseling is underhyped. It is clearly competitive with gene therapy in a certain sense, clearly not for people that are already born but for people in the future, not even distant future but in the next couple of years. We've got a chance of diagnosing them or diagnosing the potential parents and dodging… This has been in practice since 1985 in Dor Yeshorim, a perfectly reasonable community response to it. It eliminated or greatly reduced all sorts of very serious inherited diseases.

Sometimes, depending on how it's presented, it’s dismissed as eugenics. Rarely have I heard Dor Yeshorim described that way and rightly so. What they're doing is standard medicine whether you cure these kids as soon as they are newborns or whether you counsel the parents so the same disease is missing. The problem with eugenics was that it was forced. The government forced it on people. It wasn't that it enabled people to make a choice. It's that it removed the choice from the people. That was what was wrong. And that's the confusion.

But I don't think that's the explanation for why this is underhyped. I think it's because when people are dating, they're not thinking about reproduction necessarily. And when they're thinking about reproduction, they're not necessarily thinking about serious genetic diseases because they're rare. I think it's our difficulty with dealing with rare things.

There was great resistance to seat belts because less than 1% of people died in automobile accidents or even got hurt. There was great resistance to stopping smoking. It's hard even for us to imagine how great the resistance was for seat belts and smoking. But eventually we got over it.

I think this is a similar thing. Only 3% of children are severely affected by genetic diseases and they feel like, “I'm not that unlucky. I'm in the 97%.” If those were your odds of winning at the horse races or at the casino, you'd take them. 97% of winning, good. But when a child's future is at risk, I think that's not the right solution.

The other thing is that I think it has to do with the trolley problem. If you don't influence it, it's not your fault. But actually everything is your fault. Not doing something is a decision. So I think it's like, “If I just don't do anything and they come out damaged, it's not my fault”, but it is.

Dwarkesh Patel 01:17:23

David Reich was talking about how in India—especially because of the long running history of caste and endogamous coupling—there have been these small subpopulations that have high amounts of recessive diseases. So there, it's an especially valuable intervention.

George Church 01:17:39

I know what you're saying, and what David is saying, but I think it's a dangerous dichotomy. There are lots of, not just in India but all over the world… In fact we all went through a bottleneck. But that changes the rate from say 3% to 6%. But the point is 3% is still unacceptable. It's just a tragic loss, not only of the human life directly affected, but the whole family. Very often one or both parents have to quit their job and spend full time caregiving and fundraising, because these are very expensive diseases as well.

We need to be careful not to stigmatize as well. So if a bunch of families get fixed, we shouldn't point a finger at the ones that are unwilling to get fixed because that's their choice. But I think as word spreads and you see the positive outcomes, I think it will be seen as one of the simplest bits of medicine ever.

It's very inexpensive. In fact, it's less than zero because you spend a hundred dollars per genome. It'll probably be less soon. You get the whole thing analyzed. Compare that to millions of dollars that will be lost in opportunity costs and them not being part of the workforce, taking care of them and so forth.

So the return on investment is tremendous. It's at least a tenfold return on investment. It's a no brainer from a public health standpoint. We should be able to pay for this through the National Health Service in England, through insurance companies in the United States. It turns the insurance companies from being the bad guys snooping in on your personal life and then raising your rates to them giving you this free information and you can do with it as you wish. If you take the advice then you save them millions of dollars.

Dwarkesh Patel 01:20:00

Do you think genetic counseling is a more important intervention, or even in the future will continue to have a bigger impact, than even gene therapy for these monogenic disorders?

George Church 01:20:09

Absolutely, I've actually counseled my gene therapy companies that they should be investing in very common diseases because rare diseases have this genetic counseling solution, with the exception of spontaneous mutations and dominance, which probably are IVF clinic type solutions rather than… But the rare recessives can be handled at matchmaking and at every level.

Anyway, I counseled my genetic therapy companies that they should invest in common diseases like age-related diseases and infectious diseases. In fact the COVID vaccine was formulated as a gene therapy and the cost was in the $20 per dose range. 6 billion people benefited from it, or 6 billion people took it and it was proven over the whole population. So I think that's the more appropriate usage of gene therapy.

For practical reasons, getting FDA approval and so forth, you might go for the rare diseases and that's perfectly fine. But I think the cost effectiveness… The sweet spot for gene therapy is for age-related diseases and the sweet spot for rare diseases is genetic counseling.

Dwarkesh Patel 01:21:40

Alright, some final questions to close us off. 20 years from now, if there's some scenario in which we all look back and say, “You know what? I think on net it was a good thing that the NSF and the NIH and all these budgets were blown up and got DOGE’d and so forth…”

I'm not saying you think this is likely, but suppose there ends up being a positive story told in retrospect. What might it be? Would it have to maybe come up with a different funding structure? Basically, what is the best case scenario if this post-war system of basic research is upended?

George Church 01:22:21

I have to preface this. When scientists answer a question and explore possibilities, it doesn't mean they're advocating it. In the past people have asked me off-the-wall questions about Neanderthals for example, and then it was described as if I was enthusiastic about it. I’m not enthusiastic about NIH and NSF budgets being cut.

You could say that it forces us to think more seriously about philanthropy and industrial sponsored research. That could be a positive thing. It could be that that makes us listen more carefully to what society actually needs rather than doing basic research. I'm a big proponent of basic research, but also maybe I'm more than average at connecting the basic research to societal needs from the get go. I don't think it actually interferes with basic research to think and act on societal needs at the same time. That could be a positive.

It could be that it creates another nation-state that now is the dominant force. Like China could now become the next empire after the US.

Dwarkesh Patel 01:23:35

Is this a positive story?

George Church 01:23:36

It could be for China. You didn't specify who it's a positive story for. The US displaced Britain, which displaced Spain and Portugal. It keeps moving. Fresh blood is sometimes a good thing. Again, I preface this by saying I'm not advocating this.

Let's see, what else could go well? There are just certain things that society is fairly good at doing collectively that we're not good at doing individually. Building roads, schools, and science are examples of that. It doesn't mean we couldn't learn how to do that. To some extent when you build a gated community, a lot of that is done with private funding. It's possible we could figure out how to build roads and schools and just about everything.

It means we're going to run into some kind of hypercapitalism. That might mean that there's all kinds of pathologies that come along with that.

Dwarkesh Patel 01:24:43

What is it about the nature of your work, maybe biology more generally, that makes it possible for one lab to be behind so many advancements? I don't think there's an analogous thing in computer science—which is a field I'm more familiar with—where you could go to one academic lab and then 100 different companies have been formed out of it, including the ones that are most exciting and doing a bunch of groundbreaking work.

Is it something about the nature of your academic lab? Is it something about the nature of biology research? What explains this pattern?

George Church 01:25:20

First of all, thank you for being so generous in your evaluation. Maybe take it with a grain of salt. But I think that what it is is being in the right place at the right time.

Boston is a unique culture. It attracts some of the best and brightest students and postdocs automatically. It is dense enough. Sometimes people want to spread the wealth out evenly all over the universe or the planet. There are advantages to having it clustered. Spouses can find other jobs in the same field. Having a concentration of biotech and pharma and MIT and Harvard and BU and so forth, all in one pretty walkable distance, not spread out all along the East or West coast, but actually in a walkable city is one thing. That's the starting point.

And then a lab that chooses from an early stage to keep this dynamic between basic science and societal needs going at all costs, causing great trauma when the lab starts, but then getting a couple of wins. It starts building up a positive feedback loop, just like the building of Boston was a positive feedback loop. The more Harvards and MITs and high tech startups, then pharma… So you get a couple of wins in the literature and people start coming that are a whole other level up. Maybe they're already aiming for entrepreneurship while before they weren't.

Anyway, it evolves in a way that you can't just jump start from scratch. You couldn't just suddenly create Harvard and MIT in the middle of the desert and suddenly create a lab that is taking these kinds of risks early in a career.

Also the timing is good because the exponential is starting to show up. The exponential is pretty much the same in the beginning of the hockey stick and at the end of the hockey stick, but you don't notice it until it gets going. That’s what's happening in computing, AI, and biotech. They're all peaking at this point.

So whichever lab happened to already have that positive feedback loop going with the academic to industry technology transfer would asymmetrically benefit from that exponential. To some extent with the exponential, you can really look like you're very productive when really you're just kind of sliding downhill. It's like, “Yeah, look at how productive I am. I just jumped out of a plane and am accelerating steadily.”

Dwarkesh Patel 01:28:37

Yesterday I had dinner with a bunch of biotech founders. I mentioned that I was going to interview you tomorrow. Somebody asked, “Wait, how many of the people here have worked in George's lab at some point or worked with him at some point?” I think 70% or 80% of the people raised their hand.

One of the people suggested, “Oh, you should ask him, how does he spot talent?” Because it is the case that many of the people who are building these leading companies or doing groundbreaking research have been recruited by you, have worked in your lab. So how do you spot talent?

George Church 01:29:10

Well, I'm glad you framed it as spotting talent. I've heard at least one meme that all you have to do is show up and you'll get into my lab, which is definitely not true.

First of all, there's a lot of self-selection. Frankly, we're an acquired taste. Technology development is not at all the same skill set as regular biology where you pick a gene, you pick a disease, you pick a phenomenon, and you hammer away at it for your whole life. This is more like you make a library where you have a million members of the library that are going to fail and maybe one or two will succeed. It’s a very different attitude. It's much more engineering, but it's even different from most engineering. Engineering doesn't usually use libraries that way, millions and billions of components that are non-random but many of them will fail.

So the question was selection criteria. There's self-selection. The next thing is, in the interview, I typically tell them that I'm looking for people that are nice. I'm not necessarily looking for geniuses. We end up with a lot of geniuses. That's wonderful. But nice, I think, is highly predictive of how well you will do in the lab and afterwards. As a consequence, I think we have a kind of international set of alumni that are quite nice to each other even though they're supposedly in cutthroat fields. And I think they're nice to other people as well. So nice is one criteria.

Multidisciplinarity. It's hard to build a multidisciplinary team from disciplinarians. If you have two people that each know two languages or two skills, even if they don't have anything in common, they have shown that they can learn a new skill and then they'll each add the skill that connects them. That's the third thing. Those are the three main things I would say.

Dwarkesh Patel 01:31:25

Final question. Given the fast pace of AI progress—your point taken, that we should be cautious of this technology, but by default I expect it to go quite fast and there not being some sort of global moratorium on AI progress… Given that's the case, what is the vision?

We're going to very plausibly have a world with genuine AGI within the next 20 years. What is the vision for biology given that fact? If AI were 100 years away we could say we've got this research we're doing with the brain or with gene therapies and so forth which might help us cope or might help us stay on the same page. Given how fast AI is happening, what is the vision for this bio-AI co-evolution or whatever it might look like?

George Church 01:32:15

If we handle the safety issues, and that has to be a top priority, then we're probably going to have almost perfect health. Why wouldn't we? It's going to go so fast.

It's going to go pretty fast with just regular AI without AGI. But if you add to it AGI… It'll also be a positive feedback loop, because the more people that get fixed or get access to good healthcare, the more people will be helping prompt the AI, if that's necessary. I think it probably will be. The more hybrid systems we'll have of people and machines working together in harmony in this very positive scenario.

Dwarkesh Patel 01:33:06

Well, that's a good vision to end on. George, thank you so much for coming on.

George Church 01:33:11

Thank you.

Arthur Kroeber is a leading researcher on Chinese tech and macro, a founding partner at Gavekal Dragonomics, and author of "China's Economy: What Everyone Needs to Know." It's the most useful, detailed resource I've found of how China actually works.

We discuss how China achieved high-tech manufacturing dominance, and where they'll go from here.

By Arthur’s account, the Chinese government is like a giant VC fund: they decide on key priorities and then spend hundreds of billions of dollars subsidizing ruthless competition at the local level.

They are willing to lose huge amounts of money for a few of their bets to pay off: at China’s scale, effectiveness matters more than efficiency.

There's also a growing bipartisan consensus that we need to combat China's rise. This doesn’t make much sense to me. China is a big, powerful country at the frontier in many fields, and its economy is intricately tied in with our own. Being instinctively adversarial is both unsustainable and risky. On this episode, Arthur and I discuss how we can create a productive, mutually beneficial version of this relationship.

Watch on YouTube; listen on Apple Podcasts or Spotify.

• Scale is building the infrastructure for smarter, safer AI. In addition to their Data Foundry, they recently released Scale Evaluation, a tool that diagnoses model limitations. Learn how Scale can help you push the frontier at scale.com/dwarkesh.

• WorkOS Radar ensures your product’s free trials go to actual users. Radar uses 80+ signals to distinguish malicious bots from real people, eliminating costly free-tier abuse. See why companies like Cursor, Perplexity, and OpenAI use Radar by visiting workos.com/radar.

• Lighthouse is THE fastest immigration solution for the technology industry. They help you understand your options and navigate applications for expert visas like the O-1A and EB-1A. Explore which visa is right for you at https://www.lighthousehq.com/ref/Dwarkesh.

To sponsor a future episode, visit dwarkesh.com/advertise.

(00:00:00) – We should reconcile with China

(00:21:21) – BYD, Tesla, & Chinese EV industry

(00:36:05) – Will China have a Japan-style financial crisis?

(00:44:39) – Local debt situation is manageable

(00:57:28) – If CCP is so competent, why isn’t China richer?

(01:05:08) – How China keeps tech under control

(01:33:45) – Does China win AI?

(01:43:34) – Communication with China key for AI safety

(02:10:08) – What foreigners get wrong about China

(02:17:32) – China-US relationship future

Dwarkesh Patel 00:00:00

Today I'm interviewing Arthur Kroeber, the founder of Gavekal Dragonomics, a research consultancy focused on China, and the author of China’s Economy: What Everyone Needs to Know. A friend recommended it to me while I was in China. It’s been the most valuable and useful resource I've found on how China works. Arthur, thanks for coming on the podcast and taking the time to chat with me.

Arthur Kroeber 00:00:12

It's great to be here, thanks.

Dwarkesh Patel 00:00:26

First question: what really is the problem if China becomes as wealthy as America, or if its economy grows as big, or even bigger than America’s? I know it may not be your perspective to be a China hawk, but I’ve never really understood why this is a problem in the first place.

Arthur Kroeber 00:00:39

It's a very good question.

There’s a lot of criticism that China is trying to get rich on the backs of everyone else in the world by running this gigantic manufacturing export machine. It seems like the Chinese ambition is to produce all the manufactured goods in the world, run an enormous trade surplus, which means they are depending on the rest of the world’s buying power to support them. People say this model is not fair, not sustainable, and not a stable way to participate in the global economy.

The real question isn’t just if China gets rich, but how it gets rich. Does it get rich by operating under the same rules as everyone else and having a market other people can participate in? Or does it get rich by making it impossible for anyone else in the world to have the kind of production structure that they want to have, and relying entirely on these ever-growing trade surpluses?

In principle, from a welfare standpoint, I think it would be great if China became as rich as America. But that’s likely to have difficult and destabilizing political consequences, because the Chinese political system is vastly different from that of the US and other major industrialized countries.

Dwarkesh Patel 00:02:15

On the trade surplus point, to the extent that it is made possible by the government involvement in industry… It’s actually not even clear to me that that's the case. If you have high savings and not enough investment domestically, the accounting identities are such that you will have a trade surplus.

But suppose that's even the case. On paper, it just seems like what is happening is that the Chinese taxpayer, the Chinese saver, is subsidizing foreign importers. On paper, it just seems like we're getting a good deal. I'm sure some people are upset about this, specifically people who manufacture outside of China. But it's certainly not something obviously insidious.

So it seems like if it wasn't for this, there'd be some other reason that China can't grow as wealthy as us. I am playing a little bit of a devil's advocate here. But I don't really understand why this is such a big issue that there needs to be very powerful competition about it.

Arthur Kroeber 00:03:12

There are a couple of things beneath that. One is the systemic political difference is really important. The US self identity is that it is the leader of democracies around the world. We are very invested, at least for now in our democratic system. We'll see how that evolves over the next two or three years. There's some question marks around that, but historically that's been a really huge part of the US identity.

Basically the morality play of World War II and the Cold War—which have really been the crucible in which US foreign policy has been formed—is that you had an alliance of democracies that fought back first against fascism, then against communism. They won. The result was that you created a world where most major countries were democratic. They operated on market economic systems. We were heading towards a world of convergence of systems where everyone essentially playing by the same rules.

China really gets in the way of that narrative because it's this incredibly successful authoritarian system where the authority still calls itself communist. US elites have never, to this day, really gotten comfortable with the idea that the Chinese political system is legitimate. As long as you have that underlying problem, it's going to be very difficult to erase these difficulties.

On the economic side, I think it is a problem, the way that China has chosen to organize its economy. In a strictly financial sense, the rest of the world comes out ahead in the sense that China is willing to subsidize a lot of low cost production. The rest of the world gets a lot of cheap goods and the overall welfare of the world improves a lot. I think that's clearly true.

But if you look at what has happened to the US politically over the last 20 years, there are a lot of flies in that ointment. The fly is that it is important for large countries to be able to maintain a diversified production structure and maintain social cohesion. If you lose the capacity to run a manufacturing economy that employs large numbers of people, there's a lot of disruptions that come as a result of that. The purely financialized economy that we had growing up in the United States in the early 2000s as a result of this bargain with China wound up being pretty bad for the social compact.

So there is a legitimate question about how you integrate China's growing power, wealth, industrial might, into the world in a way that societies around the world can tolerate. It's a legitimately difficult problem. I think a lot of the problems have to do not with China but with US domestic policy. A lot of what's going on right now is scapegoating of China as a way of diverting attention from decisions that need to be made domestically about income redistribution and macroeconomic policy and so forth.

But there is also a legitimate question to be made about how China now accounts for 20% of the global economy, probably larger in the future, and a third of the global manufacturing economy. There has to be an agreed set of rules about how it interacts with the rest of the world so that everyone feels that they are benefiting, not just in financial terms. We don't really have that agreement now.

Dwarkesh Patel 00:06:51

Two points. On the political system, I wonder if we've learned a bad example from World War II and the Cold War, which is that great power conflict culminates in the other party totally collapsing. I think that was actually necessary, obviously, for Hitler and even for the Soviet Union. I think they were evil regimes. I think China today is an evil regime in a way, but it's just not in the same tier as Stalin or Hitler.

So the end state for any great power competition for America… History is long, there will be more than just these next few decades. It cannot be that if there's a different political system that it has to collapse the way that the Soviet Union collapsed or that Hitler collapsed.

The first thing to note here is that this could be true of any country. I think people conflate these two arguments. Whereas it's worth noticing that if Australia was producing everything the world consumed and had an economy the size of America's, these arguments should apply to them as well. I don't think people have the sense of if Australia were producing a bunch of stuff for us, we would need to form a coalition against them and have this adversarial attitude. But suppose we did. There's a question of, “How could you prevent this dislocation and is it Australia's fault?” The analogy is breaking down. Let me just go back to China.

So there's low value-added manufacturing where labor cost is a big fraction of cost. That kind of stuff was shipped off to China. But if it wasn't for China, there's many other countries in the world that have much lower labor costs than the US. If it wasn't for China, it'd be like Vietnam or Bangladesh or something.

Then there's high tech manufacturing. There I haven't crunched the numbers, but if I were to guess, I don't think with TSMC's leading-edge 5 nanometer wafer, I doubt very little of what it costs to make that is the process engineers themselves. That is just a question of, “Can your country produce it?”

Arthur Kroeber 00:08:56

There's a few different questions in there, so let me break that down. The first thing that you alluded to there. Essentially, is a Cold War the right framing for whatever this conflict is with China? My answer to that is absolutely not. You can address that question both empirically and conceptually.

Empirically, if you just look at patterns of trade and investment during the Cold War, the Soviet Union never accounted for more than about 1% of US trade and investment flows were basically non-existent. China at its peak about 10 years ago accounted for 17% of US trade, about the same amount as Japan did in the late 80s and early 90s. That number has come down a little bit on the surface. But when you take account of stuff from China coming in through third countries and so forth, basically that remains the same.

If you talk about investment flows, there's $600 billion or more of US corporate investment in China. That generates huge amounts of sales, much larger than US exports to China. The integration of the two economies is just extraordinary. There's basically no precedent for it at any point in economic history. It is just a completely different order of magnitude to what was going on during the Cold War.

So one question would be, “Okay, if you really want a Cold War, does that mean that what you want to do is you want to reduce those trade and investment flows back down to essentially zero? If so, how do you do that?” That's a very difficult problem.

You got at the more core conceptual issue, which is, how does this end? This does not end by China going away or turning into something completely different. It's too big. It is too successful. The economic model is successful on its own terms. It has a lot of problems, but it's fundamentally successful. It is deeply integrated with the entire global economy in ways that are beneficial for China and beneficial for most of the other countries in the world. Everyone has some sort of a stake in China continuing to succeed. This is just not going to go away.

There’s another sense in which it's not going to go away. Let's say you could wave a magic wand and make the CCP disappear tomorrow and replace it with something else. What would that something else look like? If that something else were to be successful at governing China, it would almost certainly share many of the characteristics of the Chinese Communist Party. It would have a strong determination for China to be an independent geopolitical actor, taking care of its own security with a very strong military so that it could defend itself against all of its real and perceived security threats.

It would be committed—as every Chinese government since the mid 19th century has been—to maximizing the rate of technological progress and making sure that as much as possible of China's core technology needs were met by domestic production rather than imports. It would be very, very similar to what we have today.

It's a fantasy to think that we have a problem here with just this particular regime, that we can just get them to change their minds. No, the Chinese approach on this is very deeply rooted in both the institutional, cultural history of China itself and its geographical and geopolitical position in the world. So it's going to have interests that create a lot of friction and conflict with the perceived interests of the United States under any government. It's probably going to be quite successful at managing this economy because they've been very successful for many, many decades.

Dwarkesh Patel 00:13:02

When I was in China, I asked somebody I met what would happen if there was an election in China tomorrow. His answer was that it's possible that the median voter in China is much more reactionary than the government. In fact, this might be much more liberal than the regime you would get out of a democratic election.

Arthur Kroeber 00:13:22

Yes. We don't know for sure because there really is a lot of public polling. But a lot of people say that and my perception is that could well be true. All you have to do is look at the Chinese Internet, the blogosphere. They don't have Twitter but they have local equivalents. There is a lot of unfettered nationalist, highly militaristic thinking there that the government spends most of its time keeping in check.

Every so often they unleash it because they're in a spat with Japan or whoever and they need to get the popular opinion. Mostly they try to tamp that down. A Chinese system that was fully representative of actual popular views… We don't know exactly what that might be, but it could be very, very difficult to deal with.

Dwarkesh Patel 00:14:11

Let’s go back to this discussion about what should be the grand bargain between the rest of the world and China. I live in Silicon Valley and as you know, a big topic of conversation is AI. In particular the race between China and the US on AI.

One idea I've heard is that what we should do is give them free rein on solar, electric vehicles, batteries—all this other real world heavy manufacturing that they seem to have greater proclivity for anyways. They consider it more real. On AI and semiconductors, we’ll dominate that. Everything else we'll import.

Is this a plausible deal worth making? Because if you take AI very seriously, this might be an amazing bargain.

Arthur Kroeber 00:15:02

You can say what you want in your ideal state of things, but how do you get there? There are a lot of people in China who'd say, “No, we want it all and we can. So why should we make that deal with you?”

If I basically take this down to the level on which the United States and China are now starting to negotiate, my personal view on this is that it would make a lot of sense for the US to be much more open to direct investment by Chinese companies in manufacturing in the United States. I'm thinking particularly of things like electric vehicles and that whole supply chain, green energy, industrial automation, all of those things.

Why do I think that? First of all, if the US is serious about revitalizing its industrial base—and I think there's a good case for trying to do that selectively—it is not going to happen unless you invite in the world's leading players and have them compete. That is how China industrialized. 45 years ago, they were an industrial basket case. They said, “We want to get industrially strong. How do you do that? Get all the leading companies in the world to invest here and we'll learn from them.” That's how you do it. If we're serious about that, we should be serious about figuring out ways that we can bring Chinese industrial investment into the United States.

That to me would be a win-win, because Chinese companies at an individual level would love that. They see the US is a huge market. They can't get into it right now. They would love the opportunity to tap into it more. We could learn from them. And if you did that, basically you would be saying that whatever we're in with China, it's not a cold war, because we are willing to have high levels of direct investment by both sides. You might want to put protections around that, all kinds of rules of the road. But in an ideal world, we would have a lot more of that.

The first question is, why is that so difficult? Why do you essentially have a consensus in Washington that not only do we have to prevent the export of US technology goods to China, but we also have to prevent Chinese companies from investing in the US? This is basically agreed on in Washington.

The reason is data. Fundamentally, there is a view that any manufacturing process today is also a data creation machine. Where does that data go and who benefits from that data? The other thing is that, back 30 or 40 years ago, you used to be able to divide the world into technologies which were essentially for civilian use, a few technologies that were military, and then a very small proportion of dual-use technologies that could go either way. It was those dual-use technologies that had to be controlled very carefully.

Now basically everything is dual use. Any technology that you can imagine can be put to some kind of military use. So it becomes much more risky to have these patterns of investment, particularly when you don't know where the data is flowing and how that will feed into someone else's defense production base. That is a legitimate concern.

In my ideal world, you would have a lot more Chinese investment in the United States, but it would be very carefully regulated in the same way that China very carefully regulates the direct investments by foreign companies in its economy. They have rules about data localization, etc. I think that could be achieved. I think it will be very difficult in the current political environment.

In terms of the grand bargain that you suggest, China is very interested in not being pigeonholed. They didn't want to be pigeonholed in low value-added manufacturing. They wanted to do the high tech stuff, now they're doing that. They don't want to be pigeonholed in manufacturing. They want all of technology, including AI.

I don't think that any strategy premised on the idea that China should accept arbitrary limits on what it can do is viable. Would we accept that? Absolutely not. So why should we expect them to accept something that we would consider an absurd and completely unreasonable infringement on our sovereignty? It just doesn't work that way.

Dwarkesh Patel 00:20:56

It's funny because one of the main arguments that China hawks will make, or one of the big parts of the worldview, is that China really cheated us. Our companies would invest in China, build up factories in China, and technology would get transferred there. This was a huge arbitrage that China was able to pull. So if you actually believe that, it should just make the reverse of that incredibly compelling. Because you get a chance to do that again.

Dwarkesh Patel 00:21:21

Do you want to tell the story of BYD and Tesla in 2018 and 2019? Because I think this really illustrates the point of what could be done in reverse very well.

Arthur Kroeber 00:21:31

If you go back to the early 1990s, China recognized that pretty much every other country that had gotten rich had done so in large part by building up an automotive industry that then served as the mechanism for creating innovations in other sectors. Look at the US, Germany, Japan, even Korea which is very successful. They said, “We have to have a big auto industry. This is one of the key industries that we have to support.”

So starting in the early 1990s, they had this strategy of bringing in foreign companies and making them do joint ventures on a 50/50 basis with Chinese auto companies. The theory was that eventually the Chinese companies would learn. You would be able to develop your own national champion auto companies and kick the foreigners out. Basically this failed. It failed massively.

Over the next 25 years, foreign companies came in: General Motors, VW, Toyota, Honda and so forth. They did very, very well. The joint venture partners in China basically just sat around and clipped coupons from the dividends that these joint ventures were producing. All of the technological inputs continuously came from the foreign partners, all of the design ideas.

The local partners were never able to succeed. In volume terms, you had a lot of smaller scale, locally sponsored vehicle companies in China building low-end vehicles. So a lot of the cars that wound up getting made in China were made by local firms. But if you look at the share of profits and total value in the system, it was dominated by these joint ventures, which were in turn dominated by the foreign partners.

So China, circa 2010-2015, had been doing this for 25 years. They were basically no closer to having globally competitive conventional car makers than they had been 25 years before. So in the late 2000s, 2005-2010, they started thinking, "Well, this and other related industrial policies are not working very well. We are not generating the national champions that we thought would be. So what do we do about this?" Their answer was to leapfrog: “Let's try and figure out what is the next stage of technological development that people really aren't working on yet. Let's work on that and then maybe we can get in on the ground floor.”

What they came up with in large measure was renewable energy and specifically electric vehicles. They started having a pretty comprehensive set of subsidies and other kinds of industrial support for companies like BYD to produce electric vehicles. BYD is basically a private company.

This worked okay for 10 or 12 years. BYD got pretty good. Particularly what they started to figure out was the supply chain. They got very good at making the batteries. But by 2018-2019, BYD was still not that exciting a company. Most people in China really didn't want to buy electric vehicles. They seemed really inconvenient. The sexy cars were the big SUVs that the joint ventures were putting out and so forth.

Then in 2018, the Chinese government made the decision to allow Tesla to come in and build a wholly-owned Gigafactory in Shanghai. They had never approved a wholly-owned automotive company before. They started cranking out cars in 2019, which became immensely popular.

It seems like one of the things that happened was that the Chinese companies by 2019 had gotten pretty good at the underlying technology of electric cars, specifically the batteries and the software systems that govern the cars. But they were terrible at consumer design. They had not figured out how to make appealing products that people would want to buy. Tesla did have appealing products that people wanted to buy. They were big status symbols.

So BYD and its competitors said, “Okay, this is the part we have to figure out. How do we achieve this?” Basically, they recognized that they had to up their design game. Among other things, they went to Germany and loaded up on a lot of German car designers that were able to transform their technology base into much more appealing packages. Then by 2022 or so, they were able to compete with Tesla both on price and on quality.

What that shows you is that, number one, the Chinese government made a pretty good bet on this leapfrogging idea: “There are technologies in the future. We need to get in early, subsidize the heck out of them.” Some of the estimates that have been made about the level of subsidies given to the EV industry and its supply chains range from $200-300 billion. It was a huge, huge amount of money that did not generate that much of a financial return for a really long time but they stuck at it.

Then the thing that finally flipped the switch was this catalytic foreign investment that showed the Chinese companies what they needed to do to actually compete in the consumer market. Then they got very good at that very quickly.

Dwarkesh Patel 00:27:33

How much was the $200-300 billion actually relevant given the fact that BYD, as you mentioned, is a private company? How much of that was actually necessary or counterfactually important to creating this outcome?

Arthur Kroeber 00:27:48

It's like the old slogan about advertising: half of it works, but you don't know which half. Did China need to spend that much in subsidies? Probably not. But had they spent less, would they have had the same effect? Maybe, maybe not.

They tried various different types of subsidies. Initially there were producer subsidies to just get companies to produce more. Then they had a lot of buyer subsidies at the individual level, but also the city government level to promote things like electric buses.

I think all of that played a role. What was important was that the government set a very clear direction and they said, “We want this to happen, we really want it. We don't know how long it's going to take. We're just going to keep trying different things until it works. Because the goal of creating this electric vehicle sector and the whole supply chain for it, that's what we want at the end.”

Dwarkesh Patel 00:28:48

Why are they able to identify these kinds of sectors in advance? Central planning isn't supposed to work. There's been many cases of countries which have tried to do this. Germany, Japan, and many other countries missed out on the Internet because there was a centrally directed effort towards these heavy industries or manufacturing, which actually turned out not to be relevant in the 21st century.

Arthur Kroeber 00:29:11

No, it's definitely perilous. China has a list of key industries. It's called the strategic emerging industries. They came up with it basically in 2010. It built on previous iterations of industrial policy. Most of it is not really an effort to predict with a high degree of specificity what comes next. They're mostly pretty obvious things like semiconductors, industrial automation, new materials. It’s all this kind of stuff that would be on anyone's list if you were running a VC fund and saying, “What are the sectors that we think are—”

Dwarkesh Patel 00:29:51

But most VC funds are losing money. Actually, maybe only a couple of firms in the world were able to make this prediction correctly. So it's like a couple of firms in the world plus the Ministry of Information Technology in China are the only people who can predict the future?

Arthur Kroeber 00:30:06

No, I think basically the analogy with the VC fund is a pretty good one. I’ve made it many, many times. Think of China as a giant VC fund that is just willing to lose huge amounts of money for a really long time on the assumption that a few of the bets will pan out.

China as a state, as a government, is uniquely able to throw a lot of money at things for a really long time and not care about what returns are. They say, “We're going to identify this universe of potential winners. We're going to promote all of them, basically. Some of them may work and some of them may not, and the successes will pay for the failures.

Dwarkesh Patel 00:30:44

I guess I'm still confused on why this works. Many other countries have tried similar things. MITI was a Japanese version of this, right?

Arthur Kroeber 00:30:50

Yeah.

Dwarkesh Patel 00:30:51

Look at their economy now and their high tech sector.

Arthur Kroeber 00:30:53

There’s a couple of things. First of all, even if you go back to like 2005-2006, when we started thinking about this, everyone was saying that green energy was going to be an important industry in the future. You had the Kleiner Perkins guy. John Doerr was the big VC who said circa 2007, “I'm going to put all of my bets on green energy because this is the future.” He turned out to be wrong from the standpoint of his investment returns. But he turned out to be totally correct in terms of this being a really big thing. It was just not that wild a bet to say that green energy was going to be really important and that we should place a lot of bets on that.

Why does this work and why is this different from a central planning thing? That's a really important point. First of all, the Chinese system, like the systems that preceded it in East Asia—Japan, Korea, Taiwan, and so forth—has always put a lot of emphasis on being good at exporting. There are a lot of pressures within the system to create industries that can export. The reason this is important is if you have a totally closed system—which frankly is what I think Donald Trump is trying to create in the United States—and people are just producing essentially for domestic consumption, there's no incentive to keep up with global technology developments.

If you have this constant export focus, essentially what everyone is trying to do… If you have a closed domestic market, you can rig the domestic market and buy off your connections in the government to make sure that you, as a company, have your market share. There are a lot of ways to win that do not involve technological upgrading. That is what dooms all import substitution regimes. If you're an export-driven economy, you can't rig the global economy. Ultimately you have to compete. You can't compete just on price, you have to compete on raising your game. Japan did this successfully, Taiwan did this successfully, Korea did this successfully. China's done that successfully.

So that's answer one. You had this underlying system which was heavily incentivized towards technological upgrading across the board. That's what you were sitting on top of. It was completely different from the old central planning regime, which was autarkic and inward-looking.

Second, you had this domestic competition environment which was really cutthroat and involved a lot of international companies. This is a little bit of a difference from Japan, which always kept the foreign companies out of its domestic market. Their companies operated globally very, very well, but the domestic market was very protected. China's domestic market was not that protected. They were always willing to bring in companies that they thought would be able to spark some technological change.

So what this did is it created a field in which whether the government has the right ideas, or whether they don't, you have a crucible in which their ideas can be tested. You have a very competitive domestic market where you have a lot of the big international players doing their own thing as well. So the things that work get some validation and you can keep building on those successes. That's really important.

Then just the sheer scale at which they were able and willing to support what appeared to be losing bets for a long time does matter. You could contrast their view, for example, in solar energy. They said, “Look, we want to be big in solar energy.” Again, they started thinking about this in 2005. “What do we need to do?” So they identified the whole supply chain, from the raw silicon inputs to all of the modules in the finished panels. They said, “We want to be in every stage of that.” So they had a pretty deliberate strategy of trying to create people at every point in the supply chain. And they were willing to go through cycles of people not making any money.

The obvious comparison in the United States is that the US government gives one loan to one company, Solyndra. It goes bad and everyone says this is proof that industrial policy does not work. China actually has the right answer to that question. We had the wrong answer to that question. You have to be willing to accept some failures and some loss of government money along the way if you want to get to the end.

But I really stress the ecosystem because a lot of this would not have worked just by the government saying, “Oh, we want to do solar.” It would not have worked unless you had this broader ecosystem system of export-driven manufacturing, high competition, and international participation across many, many sectors of the economy. They all converged to create a succesful result.

Dwarkesh Patel 00:36:05

I want to put some of these bureaucrats in a Factorio speedrun and see how they do so.

I just interviewed Ken Rogoff, so Japan's top of mind for me. It's really interesting that even though Japan had this export discipline, maybe the reason that the growth didn't continue after the 80s is because you had this convoy system where banks were incentivized to lend to people they knew in these conglomerates. These conglomerates have been around for decades, even before World War II.

I have a question now about China. You have these companies which have become national champions and are conglomerates, the same companies producing a phone and a car and everything in between. Given the intrinsic nature of authoritarian systems, or systems with financial repression, will we go to a system that is closer to what Japan had in the 80s? They’re going to give money to Huawei because Huawei or BYD are the right people. Or will it remain dynamic, especially in the sectors which are coming up over the next few decades?

Arthur Kroeber 00:38:32

It's a good question. I would start by emphasizing the ways in which China is very, very different from Japan.

The central thing that is really different and lies at the root of a lot of this, is that China is an independent geopolitical actor. Japan was not. Japan, at the end of the day, could rely on the US for security. They were demilitarized.

China basically is on its own. They're in a very dangerous neighborhood. They have 14 land neighbors they share borders with. They have names like North Korea, Russia, Pakistan, Afghanistan, a lot of dangerous actors. Several of them have nuclear weapons: North Korea, Russia, Pakistan, and India. So China, just in a very narrow sense, lives in a very dangerous neighborhood. They quite legitimately have some pretty significant national security needs. They also have aspirations to be a great power, and they want to do it on their own.

The incentive to get things right is much more existential in China than it was in Japan. Japan essentially could take the choice to sort of say, “Oh, we're going to have a stagnant economy and that's bad and whatever, but actually we'll be fine.” For China, it's not fine. So underlying it, there's a spur that increases the chances that the Chinese leaders will do the things that they need to do to keep the system more dynamic. It's not a guarantee, but it's a pretty strong incentive.

Then, if you look at some of the specifics, the big difference between the way that China and Japan are economically organized is essentially the relationship between the financial and the corporate sector. Japan had a system where banks owned equity in these giant industrial companies and trading companies, and vice versa. There were these cross shareholdings, which meant that essentially the entire economy of Japan, the entire corporate and financial sector, had one big balance sheet.

All of that balance sheet rested, by the late 1980s, on land values that had just gotten completely detached from any form of reality. There was an investment thesis among investors who were playing the Japanese stock market in the late 80s, that you did not look at the earnings of Japanese companies. Those were irrelevant. What you looked at was the value of the land they controlled.

You looked at essentially a capital ratio where you assumed that the value of their underlying land capital was a permanent contribution to their wealth and prosperity. The valuations of these stocks, which again got crazy, you couldn't justify them by earnings. They said, it doesn't matter because the land they're sitting on is so valuable. Then the land values collapsed by 80%. The stock values had to collapse by a similar amount, 80%.

Everyone was holding shares of one another. So the banks’ capital got eroded because a lot of their capital was tied up in the land and in the stocks which were collateralized by land. So they were unable to lend. They just had to deleverage as fast as possible. The companies had to deleverage as fast as possible. The entire economy got engulfed by this phenomenon of debt deflation, which basically means in the act of paying down the debt—because you're doing fire sales of assets—you're reducing the prices of those assets and introducing deflation. So the real value of your debt continues to grow even as you are supposedly reducing it, because of the deflationary effect. It's very, very difficult to get out.

China does not have this problem at all. They looked at the problems that Japan got into, they looked at similar problems that Korea had in the late 1990s, and they said, “We will never allow this kind of cross shareholding between financial companies and corporations. This is way too dangerous.” They have kept the two quarantined. It is illegal for industrial companies to own a bank. The bigger ones are allowed to have internal financing subsidiaries, but they can only sort of manage finance inflows within the group. It is also illegal for banks to load up on shares of industrial companies. So you have a distinct financial system and industrial system. That means that the particular type of macro problem that Japan got into is very, very unlikely to occur in China.

You do have huge debt problems. The property developers are very over leveraged, it’s a huge problem. We've had a massive property crash in China over the last five years. That is a big, big problem. You have local governments that borrowed a lot of money, invested in infrastructure which is now delivering very little returns. They have a big debt problem.

You have very substantial debt problems in China that have significant negative macro consequences. But they're all isolated and they can be dealt with sort of one by one. If you look at the industrial sector, it's not very highly leveraged. Debt levels in fact are not high. Most private companies in China have learned for many years that they couldn't get access to bank credit. The banks only wanted to lend to state-owned enterprises which were secure, had physical collateral, and had a state guarantee.

Private companies for years have learned to make do by basically financing their investment out of retained earnings. Their leverage ratios are not that high. They're creeping up now over the last few years because of the big industrial policy push. But fundamentally they're in a very different position from the Japanese companies.

Dwarkesh Patel 00:44:39

Interesting. This is just stuff from your book that I'm citing back to you. You talk about these local government financing vehicles which are backed. The local government takes out a lot of loans in order to build this infrastructure. That is backed by the presumed appreciation of the land, which should go up as this infrastructure goes up.

That sounds actually very similar to the problem you were describing with Japan. You say, “Look, this is debt that's on the local government's balance sheets. But fundamentally it's like one country.” You have this system where the government can just hand the debt to somebody. So how immune is the private economy really? Will it just take down the local government and then nobody will make a fuss? These other companies will just never have to hear about it again?

Arthur Kroeber 00:45:31

I guess my view on this is that the local government land based financing model was actually a pretty smart thing to do initially. For a decade or so it worked quite well. Local governments were sitting on a lot of these land assets. The land assets originally were undervalued quite substantially. So it made sense for the local governments essentially to try and capitalize some of the future value of these things and use them to finance infrastructure development.

This was not a rogue policy that local governments came up with on their own. It was actually sponsored by the central government and the China Development Bank back in the early 2000s. They were grappling with the problem that China at that point had a severe shortage of housing and infrastructure of all kinds, including urban infrastructure. The question was, “We need a lot more of this stuff. How do we build it? How do we finance it?” They realized, “Actually as we build this stuff out, the values of the land are going to skyrocket. So let's essentially take some financing against that future land price appreciation and use that to build the infrastructure that makes everything possible.”

If you look at the first two cases of where this was done in the early 2000s, they had these estimates of what the land would ultimately be worth that seemed ridiculous and astronomical at the time. They were in fact way too low. They were way, way conservative on what the actual land price appreciation would be. So you had a period of about eight or nine years where essentially the state-run development banks were doing this in a fairly controlled way, very successfully.

What happened was that after the 2008 financial crisis, the government said, “Oh, we just have to spend a lot of money to gin-up growth because there's been this global economic catastrophe.” They did a massive cash infusion into the banks and to the state-owned enterprises. They said, “Spend, spend, spend, and basically all infrastructure is good infrastructure or just build whatever you want.”

What happened at that point was that all of the commercial banks got into this business of lending money to local governments for these infrastructure projects. Their underwriting standards were much less robust than the China Development Bank’s had been. For the local governments, it was just free money to them. They just took all the money and they ran with it.

That created this kind of Frankenstein's monster. Even so, you had quite a lot of that money in the first two years that probably generated a pretty good return because China was still underbuilt relative to what it needed to be. But after that had run for three or four years, it became clear that a lot of this was going into wasteful projects that didn't deliver a good return. Now they've been struggling for the last decade to unwind that.

But it's important to understand that the underlying model was actually an okay model. It just wound up being done in a way that was completely uncoordinated and excessive. You didn't have incentive structures that enabled local governments to rein it in when they needed to.

So I still think that that is fundamentally a local government fiscal problem, which ultimately is a central government fiscal problem. If you look at the consolidated balance sheet of the central and local governments, put all together, China's government debt is still probably less as a share of GDP than US federal government debt. The US is over 100%, China is probably under 100%.

Dwarkesh Patel 00:49:38

So my previous guest, Victor Shih, estimated that the local government debt alone is somewhere between 100-150% of GDP. If you add it to the central government debt, I think he estimated that government debt in China is 200% of GDP.

Arthur Kroeber 00:49:55

I would respectfully disagree with Victor on that. It's impossible to know for sure. You have the problem of formal debt, which is debt issued by government entities, and then you have contingent debt, which is debt issued by corporations that may or may not essentially be fronts or windows for local governments.

There is also a very significant problem of double counting, which is hard to resolve. Debt gets counted at different levels in the system. It's one piece of debt, but depending on how long the chain is, it gets counted in various different places. I've long had the view that many of these much larger estimates of debt don't fully account for the double counting problem. It's quite possible that the true debt level is higher than what we think it is. It's unlikely, in my opinion, to be as high as Victor says it is.

It also matters what the debt goes to and who is actually formally responsible for paying it. Are there project flows or income flows that can be used to pay for it? The structure of the servicing of the debt matters.

At the end of the day, we do have a pretty good sense from the macro data of how much total debt there is in the economy. It's on the order of 300% of GDP. So if you want to say that government debt is 200% of GDP, then unless you're double counting, you're basically saying that everything else is a lot less than you think it is.

Dwarkesh Patel 00:51:44

Do we know what the total debt, private and public, is in the US?

Arthur Kroeber 00:51:48

I don't have it off the top of my head. The federal government debt I think is around 110% GDP, something like that. Total gross debt of all sectors, government, corporations, households, I think would be on the order of around 300% of GDP. Japan is substantially higher. You can get into an endless debate on these things where it becomes kind of a numbers game.

But I have a pretty high conviction that China has a significant macro leverage issue. If you take my numbers, say gross debt to GDP—which by the way are the official numbers of the Chinese government and largely accepted with some structural modifications by the IMF, so forth—it’s 300% of GDP. That is in the ballpark for highly developed economies. It is extremely high for middle-income economies. So if you look at China as a country that has average per capita income that's kind of like Brazil, China is way more leveraged than any other country of its level of income. That is potentially a problem.

So I don't want to make light of the problem because I think it is quite serious. It's an important constraint on policy and it is an important constraint on growth, at the moment. I don't think that it leads to a financial collapse situation because the system is contained and all the debt is denominated in local currencies. But it does impose a tax on growth and it is one of the things that makes China's growth more sluggish now than it probably could be.

Dwarkesh Patel 00:53:33

Why is it the case that middle-income countries are especially in a bad position with having a high debt to GDP ratio? You would think naively that they're in a position to have higher growth over coming years, so it may make sense for them to take on more debt so they can use that to finance the greater growth that will come—theoretically assuming they use it to invest in high return things—as opposed to the US or Japan where you're just going to be stuck with this debt load for a long time.

Arthur Kroeber 00:54:01

One of the things that you could say about debt historically in China was that it tended in general to finance productive assets, whether it was industrial production or infrastructure. For many years China was a very infrastructure short country relative to its needs. So you could justify quite substantial debt-financed infrastructure investments on the basis of these would deliver good economic returns for many decades. The case for that is a lot weaker now than it was before.

The problem is how you essentially retire this enormous stock of very low-productivity, infrastructure investments, debt-financed. How do you sort of move on and have a more productive return on capital in future? There are ways for solving that problem.

To me the fundamental issue is they have too much of a supply-side strategy, not enough of a demand-side strategy. It's really a macroeconomic issue. If they had stronger domestic demand, that would generate more profits for companies. They would be able to have more pricing power for companies. You would get a little bit of inflation. So you would have more cash flows within the economies that would enable people, and local governments, to pay down debt. A little bit of inflation would erode the real value of the debt.

That's essentially how they got out of their last debt problem in the late 1990s. They had gigantic bad debts in the banking system, essentially representing legacy loans to state-owned enterprises. They had no way of paying them back. They had a very, very growth friendly program that generated a lot of growth, a lot of inflation. That basically strengthened their debt problem.

What they need now is a kind of a revamped, updated version of that strategy. They do a lot more to promote domestic demand, to get more profits, more cash flows, and a little bit more inflation into the system. That would probably solve the debt problem over a decade or two.

Dwarkesh Patel 00:57:28

I want to keep asking you more questions about the nitty gritty of the situation, but before we keep deep diving I want to step back and ask this question. The valence of things you've said so far has been, “Look, they've been remarkably competent even at the things which economists criticize them most for. For the majority of the period of these schemes they've actually worked out quite well. For example, this local government financing through land sales on estimated future income and so forth.” Obviously we've been talking about the Chinese industry and how that's been successful in many key sectors.

I guess a big picture question is this. They're still at like a fifth of American national income per capita, a third of similar countries in East Asia, like Japan, Taiwan, and South Korea. How do we explain the relative poverty on a per capita basis in China? How do we explain obviously bad decisions like Zero-Covid? I guess I'm having trouble squaring the circle. If they're making all these great calls, why isn't China more successful?

Arthur Kroeber 00:58:38

They're not successful enough? I thought one of the problems in the world that we're facing today is that they're too successful at too many things and that this is creating difficulties for other people.

Yeah, that has come up from time to time. During the period in the 90s and 2000s, they were growing at 10% a year which is the fastest growth rate recorded by any economy in the history of mankind. They were doing it not for one or two years, but for 20 years in a row. People often say there's all this waste and there's too much debt on enterprises and that they could grow much faster if they were just more efficient. It's like, “Really, could they?” They're already growing faster than anyone has grown before. Is it really the case that they could have grown a lot faster than they have? Maybe, but I doubt it.

The reason is that there's what I would call an efficiency fallacy. Global macroeconomists have all been basically trained in the United States since the end of World War II and particularly since the 1990s. They view the US in its developed state as the norm rather than as the result of other processes. It's like, “The way we operate now is the way we got here.” Actually, that's not quite the case.

The reason China was able to grow so fast for so long was that they were able to mobilize these huge swaths of resources, domestic savings, and throw them in an uncoordinated way at a lot of problems. There was a lot of waste along the way. That was part of the equation. Say they had tried to be maximally efficient and said, “No, no, we're not going to allow banks to lend unless they can prove that this project is definitely going to have an ROI of X”. They would have lent a lot less and they would probably have had a lot less growth because they were not willing simply to create that wall of money that would create broad-based growth.

They might in fact have had substantially more inequality than they did. Their wasteful method allowed them to spread out investment in a lot of areas which were geographically disadvantaged. Those areas might not have done so well in a purely efficiency driven system. I think we need to be careful about assuming that inefficiency is bad. I think inefficiency sometimes can be, as long as it is basically the result of a desire to be effective, a byproduct of a successful growth story. That's basically my account of China.

Dwarkesh Patel 01:01:33

Is it your quote in the book, something like, “For a country of China’s scale, what matters not is efficiency of using the resources, but effectiveness of achieving outcomes”?

Arthur Kroeber 01:01:41

Basically that's my thesis. I think they've shown that this works quite well. That's also part of the answer to why aren't they richer on a per capita basis. Well, there are 1.4 billion people. It is just gigantic. Think about the scale problem. Taiwan, South Korea, they got to whatever they are relative to US per capita GDP—60-70%, something like that—within a generation or two. Taiwan had 20 million people. South Korea had 40-45 million people. To move that number of people and get them all working in high wage industrial or service jobs in urban areas, you can do that in the course of a generation, 20-25 years.

I think it's just physically impossible to achieve that on the scale that China's been operating on with 1.4 billion people. You look at the urban migration that they've done. It is completely unprecedented in the history of humanity, the speed at which they have converted people from rural residents into urban residents. For many years they were urbanizing at the rate of 20 million people a year added to the urban population.

I forget the precise calculation that I made, but in terms of housing and urban infrastructure, they were building the equivalent of New York plus Philadelphia plus San Francisco every single year for 20 years. No one has ever done anything like this before. The lesson of this is that there's an arithmetic problem there. Even if you do this on this unprecedented scale, and do it faster and bigger than anyone has done it before, if you're trying to move 1.4 billion people into an income of United States-level, it just takes a really, really long time.

Now of course, what's happened is that they have gone too far off the deep end of industrial policy and thinking the supply side will solve all of their problems. Growth has slowed down quite a bit in the last few years. It's probably well below potential. I don't think that there's a lot of evidence that China's growth was below potential from 2000 until 2020 on average. I think there's a lot of evidence that it is below potential now. They are sacrificing the ability of the average Chinese person to do more rapid catch-up with the living standards of the West, in order to achieve these industrial policy technology goals.

Dwarkesh Patel 01:04:37

I have a question just to react fast to some of the things you said. Obviously there's places in China now like Shanghai or Guangdong, which have per capita incomes approaching these East Asian neighbors that are really rich.

To the extent that that's possible through this high tech manufacturing—and to the extent that it's already a problem that other countries are complaining that China is doing too much high tech manufacturing—I guess it makes sense then. How could you have literally 10x as many people doing the same thing?

Dwarkesh Patel 01:05:08

You said that the government has decided to prioritize this high tech development rather than the growth rate. You said that as if there's a trade off. But naively, isn't high tech supposed to lead to growth? Why are we on this Pareto frontier with technological development and growth?

Arthur Kroeber 01:05:28

Here you get to a sort of conceptual fork in the road. In many respects the communist leaders in China have a lot in common conceptually with the techno optimists of Silicon Valley in their, what I would call, technological fetishism. The Chinese leadership throughout the last 40 years, but even going back further, has had this notion that technological upgrading is the key to wealth and power.

The fundamental idea here goes back to the 1840s when you had this great Chinese empire that had been sitting around for a thousand years doing great. They thought they were the best in the world. Then these upstarts from England come and crush them repeatedly, militarily, and impose this very humiliating political settlement on them through the Opium Wars.

The basic analysis among the Chinese elites was, “Okay, we fell behind technologically. Therefore we must catch up technologically. This is the central thing that we need to do.” This was recognized among some elites, but it ran into a lot of encrusted political pressure in the late imperial Chinese regime. They didn't get very far.

After the empire fell in 1911 you had a republican government. Basically, if you look back at the things that they were saying and doing, they're pretty similar to what the Chinese Communist Party has been saying and doing. They just did them somewhat less effectively. They were dealing with civil wars and various other problems. The basic insight remained the same, “We need to catch up with the west technologically. This is the fundamental thing. It's just a question of how do we do this?”

What the current regime in China has been able to do is to take that basic idea and actualize it. They have been successful at this task of technological upgrading and catch-up. This is really central to the conception of Chinese elite thinking. The number one job of the government is to figure out how to mobilize the resources of Chinese society so as to maximize technology acquisition, creation, and upgrading.

But there are a lot of flavors of this. Starting in the 1980s, that was a big part of the philosophy. But the other part of the philosophy was just, “We have to grow as fast as possible and we are fairly agnostic as to where the growth comes from. We realize that there are many different sources of growth. Foreign investment, exports, construction, infrastructure. We're going to do all of these things.

Basically the incentive structure that they set up was just to tell local officials that their job was to maximize GDP growth. That will make all of these other things happen. If they tell local government officials, “Your job is to maximize technology development, they'll do a lot of dumb stuff and waste a lot of money. That's not the right way to do it. Let's just have GDP as the KPI. The end.” That turned out to be incredibly successful.

Then Xi Jinping came along in 2012 and said, “The problem with this is that it's now gone too far. It's led to a lot of corruption, it's led to a lot of inequality. It's led to a lot of local governments doing wasteful and duplicative things. Now we need to hunker down and focus a little bit more on the really central task, which is this technology task.” The mentality is less of “growth at all costs.”

Basically, the implicit idea there is, “We've got this figured out. Growth will take care of itself. We have a growth machine in China. The resources of the government now need to be focused on specifically the technology question.” Everything that he has done—particularly since about 2015-2016 when he really started to see this come into focus—has been, “Let's worry less about the growth target and more about the technology targets.”

Particularly after Trump launched his trade war and the Biden administration followed that up with a lot of controls on exports of technology to China, the focus got intense. They were saying, “The US really wants to prohibit, constrain our technological rise. They kind of have the ability to do so. Therefore we have to double down on efforts to replicate all of these core technologies so that we're not reliant on the US or its allies anymore. We just have to go full tilt on technology at all costs.” That was an accelerant of this tendency.

But then there is also a macroeconomic theory. Go back and read some of the statements that he put out in the late 2000s and 2010s about their vision of the future. They talk about all these technological developments. They didn't mean Alibaba and Tencent and all of these great Internet companies. They meant physical technology, semiconductors, new material, green energy, industrial robots, all this stuff. Essentially their view was, “This is going to be the productivity engine of the future. Growth in the overall economy and incomes, it's all going to be driven by these technological investments.”

That is essentially what they sincerely believe. There are some problems with that. The vast majority of people in the economy do not work in those sectors. You can get very good at making semiconductors and industrial robots and whatnot. The people who work in those industries do extremely well. But that is a tiny fraction of the totality of the population.

How do you have spillovers that enable everyone in the economy—regardless of how remote they are from this engine room of the high tech sector—to enjoy gains to income? The answer in the United States historically has been, “We're just going to have this great consumer economy. We're essentially going to be demand driven. Consumers will tell us what they want and companies will arise to make what they want, which is increasingly not stuff, but services, experiences. We will create a financial system that produces very good financial rewards for the companies and people that produce these kinds of services.” Then you generate huge amounts of actually very high paid employment in the services economy.

The Achilles heel of any industrial policy-driven growth model like China—but you can see variants of this basically throughout East Asia—is that they are very materialist. They're convinced that it's only really the physical stuff that matters. They think if you get really good at making the physical stuff, that will somehow magically spill over into the rest of the economy. The evidence that we have is that that actually doesn't work that much.

Most of the demand in a modern industrial economy, the really high wage, high income economies, comes from intangible services. There's a more complicated interaction between this high tech core and wider economic growth. I don't see in China how they set up the linkage between all these great high tech industries that they have and 90% or so of the economy which is doing something else.

Dwarkesh Patel 01:13:40

I guess I still don't understand. Point taken that maybe in the short run these other sectors don't generate as much growth as just building more housing or something. But by that same token they don't take that much capital, in the scheme of a national economy, to sustain. If China wants to make a $100 billion fund for semiconductors, that shouldn't detract from its ability. I just don't understand why the macro growth number has to go down for it to be able to do that $100 billion semiconductor fund or something.

Arthur Kroeber 01:14:11

It doesn't have to, but the way that they have in fact executed this tends to lead in that direction. Here's why. At the same time that they have been doubling down on industrial policy, they have also been much more tightly regulating their service sectors.

Here’s a few examples. The famous one is that they woke up in November of 2020 and realized that Alibaba's financial subsidiary was about to do this gigantic IPO. The premise of that was that China should build a financial system that resembled that of the United States in 2005 and that this was a really smart idea. They said, “Oh, actually that's a really dumb idea because look what that led to in the US. We don't want securitization. We do not want this financial risk generation that Ant Financial is trying to talk about. We really need to crack down on these guys.” So they did.

They were also simultaneously concerned with a lot of other things that the Internet platforms were up to. So there was this broad-based regulation of the Internet sector which, weirdly for about 20 years, had operated in a complete free-for-all land of almost no regulation whatsoever, except for political censorship. Pretty much anything else you wanted to do, you could do it with very little government interference. It was a strange exception to the regulatory norm in China.

They finally caught up and said, “No, no, we have to regulate all these guys.” They went around to all of the Internet companies and basically told them, “There are things that you can do that are fine and there are things that you cannot do. There are a lot of business models that you might be interested in doing, such as fintech, such as telehealth, such as wider social media applications that you may not do. We think it creates too much financial risk. We think it creates too much social stability risk. We think it creates too much political risk for us, the Communist party.”

So a lot of the avenues of growth for the Internet companies were shut down. They still did okay. They're still very big companies. I think their share prices fell way more than was justified by the actual loss in revenue. That's starting to correct itself. Alibaba, Tencent, they're still great companies and they're figuring it out. But they were severely constrained. They were much, much more tightly regulated.

You look at the financial sector generally. It was significantly deregulated from about 2008 to about 2017, with some bad consequences. You got increased financial risk, shadow banking, lots of bad stuff. But you also got lots of good stuff. Private companies, which were essentially shut out of loan finance, got the ability to finance themselves through debt in the way that they had not been able to previously. You had a lot of new financial services going to households.

Basically the government said, “No, this is a little bit too dangerous. We're going to re-regulate it.” Now most financial activity in China has been constricted back into a small number of basically state-owned banks. So there was severe re-regulation of the financial sector.

There was a lot of talk about the healthcare sector, opening that up much more to private entrepreneurship. That again has been reined in. So you can go down the list. There has been a massive re-regulation of the service sector which limits the opportunities for profitable expansion by companies, limits the opportunities for entrepreneurship, and limits the opportunities for employment.

Dwarkesh Patel 01:17:58

But the examples you mentioned all seem like the potential growth was constrained not because it was cutting off against high tech development in other more physical fields, but because it was a threat to political power, or to perceived social stability. Growth has gone down because of all these actions. It doesn't seem like it's happened because they needed that to happen in order to get SMIC up to snuff.

Arthur Kroeber 01:18:31

No, It's a political choice. That's right. This is what happens if you say technological development, particularly in hardware, will solve all of our problems. It will generate all this growth and so therefore we don't need this service sector. We have a real life experiment. What happens if you do that?

What happens if you do that is you get a lot of progress in the technological sectors. You get basically a persistent shortage of aggregate demand, which means that even the people who are making all this great technology stuff don't have any pricing powers. They have to keep lowering their prices. So you get deflation.

When you get deflation and less employment growth, at the household level people say, “Oh, people aren't hiring. I guess I need to save more and spend less because my income is not going to go up.” You create the risk of this deflationary spiral which they're sort of on the edge of right now.

Again, this gets back to the techno optimist view of things. If you have this very narrow view of the economy as purely the expression of the emanation from this technological core, you're missing a lot of stuff. You will basically take yourself into a blind alley if you think that that is going to solve all your problems. The Chinese government in the last few months I think has begun to recognize this and say, “Oh, actually we need to have some kind of a demand strategy.” This is very evident in their top level statements.

The problem is they've been working for 45 years to build this magnificent machine of investment and industry and manufacturing. They're very, very good at it. But they have basically devoted no time in the last 45 years to figuring out how to create a demand machine. They're just at the very early stages of figuring out that part of the problem.

Dwarkesh Patel 01:20:41

There's another problem with this focus on specific parts of technology development. In any field you understand well enough, you realize how contingent and random the steps leading up to what we now consider to be this self-contained high tech thing were.

The field I know better than others is AI. There's just so many weird things. People wanted to play video games and so we had decades of progress in making these graphics processing units which contributed to AI development. People were just posting on Reddit, on Twitter and so forth. That fossil fuel of data has been powering these AI models. The idea that you could have just said in 2000, “We want to make progress towards AGI” and somehow that would have led somewhere as opposed to what actually ended up happening…

Arthur Kroeber 01:21:37

You can scale that analogy up to China, just its total economy. In terms of how they got here, the thing you want to be careful about is that this was not the result of some carefully crafted master plan that they knew all along. It was a lot of groping and a lot of random stuff. They did one thing for one reason and another thing for another reason. Then through this random process, it all coalesced in the end.

The amount of intelligent design behind that was I think actually quite low. They got a few big directional things correct. They stuck with them. That's where I would give policy credit. They were also willing to adapt when things were obviously not working and cut things off and adjust with pretty high frequency.

But a lot of it was essentially a random walk where a lot of things that were enabled then started to interact with each other in the way that you talked about with AI. Then they create these other results which could not have been preplanned.

Dwarkesh Patel 01:22:53

Another thing to consider when discussing the virtue of such a system is that you can't just look at the results over the course of 20-30 years and say, “Since they were able to pick the key technologies over this period, therefore this system is preferable.”

It's somewhat similar to looking at one successful dictator and saying, “Because they made the right calls, dictatorship is the right model.” I actually don't know that much about what Japan was doing from the 40s to the 70s. But I assume MITI, or whatever, made some right calls about which electronics and cars are important.

But in order to judge, how plausible is it that this level of competence and foresight and luck or whatever mixture they're in will continue into AI. What happens after that? Whatever's required as an input into that, speaking of which, let me ask you about that.

Arthur Kroeber 01:23:43

Well, actually can I just stop you there. There are a couple of things that you touched on there that I think are really interesting that I'd just like to put on a little bit more.

Past results are no guarantee of future returns, right? Just because you did it before doesn't mean you can keep doing it. We're seeing that now in the US. A few years ago, I would have said I had very high confidence that the US will be able to retain the basic political system and structure that it's had for a really long time. That's one of its enduring strengths, that that infrastructure remains. I'm much less confident about that.

Similarly, in China, you have this problem that they've had a succession of top leaders since Deng Xiaoping with very different styles and power bases and so forth. They've managed to succeed through a range of these things. That gives you more confidence that they're probably more likely to do it right in the future than not. But there are still a lot of concerns about that. Xi Jinping has basically declared himself president for life. We haven't been in that situation in China for a long time. Last time we had that, it did not end well.

There's always been a trade off between growth and control in China. This is kind of the central dilemma that the Communist Party has had, “We want as much growth as possible, but we also want to maintain a pretty high level of control.” That pendulum has swung much more towards the control side of the equation the last few years. Growth is slowing. So you have a lot of problems.

However, I would make a couple of points. One is that China has been successful in its industrial policy less so because they picked the right industries and more because they were very adept at building the right enabling infrastructure. I would point to two things in particular. One was a program that they had starting in the 1990s, which in English has the very ungainly name of informatization. So it translates a Chinese term which just means “xìnxīhuà” (信息化). It means informatization, taking information technology and putting it everywhere in society.

They were early adopters of the IT revolution, including the Internet. If you think back to the discourse around the Internet in the late 1990s, everyone was saying the Internet was the death knell of authoritarian regimes because it empowered everyone to have information. Information control is no longer possible. Social control is no longer possible. Would that that were true, right?

The Chinese government very, very early said, “No, that is wrong.” We think that the Internet can be an instrument. We can use it to, number one, promote technological development at a high speed in China, and number two, enhance our control of society. They have been proved absolutely correct in that bet on both counts.

They had a drive to network the entire society. They started to invest in very, very early, initially just through ordinary fixed line telecoms networks, but then they were very early in mobile and then getting government network systems that communicated across the whole country. They invested huge amounts in that. It's been, I think, very beneficial to the rise of all kinds of things. It's why basically they were able to create an entire Internet of their own with its own companies that doesn't have the participation of the US global companies and is on its own terms basically just as successful.

What they also recognized was that if everyone's on the Internet, and we have people watching the Internet, then we know what everyone is thinking all the time. We can see that. This is the holy grail of Chinese leadership since the second century B.C. They sit on top of this vast empire. It's great. They have all this territory, all of this wealth is flowing in. But they have no idea what people are thinking or doing far away. This has bedeviled every Chinese administration for thousands of years.

Even into the 1990s, it was an authoritarian system but it was very fragmented and it was very, very hard for the central government to really have any idea what was going on. Now we have a panopticon because everyone is online. If we control the online universe, then we see everything and we're pretty confident that we can keep the discourse under control. They won that bet.

The comparable move that they are making today, which I think is severely underappreciated, is that they believe in the power of electrification. Electricity has been around for a long time. Still in most countries it only accounts for about 15-20% of the total final consumption of energy. It is still actually a minor player in the global energy equation, despite having been around for 150 years and being very critical in a lot of applications.

China very early on recognized that they needed a lot more electric power. The original reason was very prosaic. They had a lot of power shortages in the early 2000s that were making it hard for their factories to run. They said, “Oh, we have to build a lot more generating capacity, most of it fueled by coal, just so we can keep our factories running.” That was the first thing.

Then they started to think about it a little bit more and they started to say, “Actually, what do we want? Well, we know we're failing in this drive to make cars with industrial combustion engines. It looks like we're never going to be able to crack that nut and have our own Toyota or General Motors. But no one's making electric cars. Maybe we can be the first to do electric cars” So they said, “We got to do that.”

Then they said, “Oh, we need to enhance our domestic transport infrastructure. Wouldn't it be great if we had a giant high speed rail network just like Japan, except we're 20 times larger than Japan, so it has to be 20 times as big. Let's do that. By the way, this is all going to be electrified.”So they created a couple of core hub industries. The premise for them was that you need gigantic quantities of electricity to get these things working. If everyone has an electric car and everyone's charging their battery, it’s a huge draw. If everyone is riding on electric trains instead of on planes, it's a huge draw on electricity. So we need to have a much more comprehensive idea about what our electricity grid is going to look like to support these industries that we want for other reasons.

Well, where do we want this electricity to come from? Well, coal. Yeah, we have a lot of coal, but it's really dirty. It's a climate problem. It's a very severe domestic air pollution problem. This is really dirty. So from an energy security and pollution standpoint, we're probably better off doing renewables because any other source of fuel for thermal power plants, we're gonna have to import. We don't have enough natural gas. We can build nuclear, but that's hard to scale and we don't have enough uranium.

If we build a renewable powered domestic electricity grid, we will be self-sufficient forever. You don't have to import the sun or the wind, it's just there. You can see how these things begin to build on one another. Again, it was not that they had the intelligent design plan in the year 2000. It was just sort of all these things that came together. The result is today, China has generating capacity that is more than double that of the United States. Their renewable generation capacity alone is as big as the entire generation capacity of the United States. Electricity as a whole is now about 30% of China's total energy consumption. It's going up like this, it's rising very rapidly. Everyone else is increasing their electricity consumption very, very marginally.

Now you see it in things like AI. What is one of the big constraints on AI development? At scale, it is essentially the power that you need to power these data centers. Who is best placed in the world to do this at home? China. They have also concluded that AI aside, having cheap abundant electricity is really central to all of the major growth industries of the 21st century. They are now determined, I think, on a strategic basis to have an electricity system that just produces as much power as humanly possible from whatever fuel source you can find.

They want to interconnect this via grid as efficiently as you can, so they're leaders in ultra high voltage distribution technology. They will be able to keep our electricity prices so low that it will be very difficult for anyone anywhere else in the world to compete with their manufacturing, with their AI, with their whatever, anything that electricity goes into. They will have this competitive advantage.

I think they're basically right about that. The point here is that if you're able to get a few of these sort of enabling all-purpose technology decisions right earlier and then scale them up, then a lot of good things can happen as a result of that, regardless of how effective you are at picking winners sector by sector in industrial policy.

Dwarkesh Patel 01:33:45

What is the story in which China doesn't win AI? In terms of the talent of AI researchers themselves, it's a big country with lots of smart people. That will be there. We've already seen examples with High-Flyer and DeepSeek where they can come close to the frontier.

In terms of chips, eventually SMIC will be able to produce the kinds of H100 equivalents that the 5 nanometer process a TSMC can produce. I don't know how many years away it is. Export controls might have still been net good. But this is not something that will not happen within the next 5-10 years. Then it's a matter of scaling, scaling production of the chips, scaling energy scaling, maybe people and data collection.

What is China good at? It's scaling. I think the numbers on energy production are just absolutely staggering. What is the time interval in which China adds an America-sized amount of power generation?

Arthur Kroeber 01:34:39

Well, they add basically a France or UK on average every year.

Dwarkesh Patel 01:34:44

Extrapolate that forward. So far we've been focused especially on the training of AI models. When these models start becoming super economically productive—basically by the point at which they are at human level—the more important question will be how many can you deploy?

Why is China such a powerful country or America such a powerful country? A big part of the reason is we just have more people. That's why China could take over Taiwan. China has like a billion people and Taiwan has like 20 million.

Now you just have more AIs, more AI people if you have more power, which is ultimately what's upstream. That was all just a preface for asking, in 2035 what is the story of how China doesn't absolutely dominate AI?

Arthur Kroeber 01:35:29

It's a good question and we're verging a little bit beyond my realm of competence here because I don't pretend to understand AI very well. I'm still very unclear as to economically where all the benefit accrues.

Let’s abstract away from AI and just think about this. This is a situation where you've got a new technology, it's entering society. What is the innovation cycle that's going to drive it? I'm pretty partial to theories of innovation like the one that was propounded by a guy named Amar Bhidé who wrote a book a number of years ago called The Venturesome Economy. What he stresses there is the feedback loop between consumer demand and what producers are doing. His view is that we focus too much on what the producers are doing and not enough on what the consumers are driving.

I think that's a pretty good insight generally. It helps explain a lot of why the US has maintained its position with such a dynamic economy, even though we've abandoned a lot of the physical processes that underlie it. So my question is, “Okay economically, when we say AI, what's the most important part of it? Is it the large language models?” I think the consensus is no. That's just the substrate. What really matters are the specific applications that get built on top of that. That's where most of the economic benefit is going to accrue.

If that idea is correct, then the question you want to ask yourself is, “What economic system offers the most fertile ground for people figuring out how to use AI in specific zones?” There I would give the US a fairly strong edge over China because China has become more self-enclosed. They have become more obsessed with self-sufficiency. They've made it very, very difficult for other people to operate in China. It's creating some difficulties in people operating outside of China.

They have this walled garden of Chinese people and whatever they do that creates applications. But whoever the big US or Western AI operators are, they have the entire rest of the world to play with. That's a much bigger universe. I would rate the likelihood that you get interesting applications higher outside the Chinese walled garden. So if you want the case for why China does not dominate, I think something like that would be the reason why.

Dwarkesh Patel 01:38:28

A previous guest I just had on, Victor Shih, said that the Chinese government might be reluctant to let AI development go full speed because of the destabilizing effects it might have on the political system or the inability potentially to control it. What's your sense on what will happen as AI becomes more economically valuable, a focal point in discussions about technological development and so forth?

Arthur Kroeber 01:38:53

I have a lot of respect for Victor. I think he has a very good grasp of the Chinese bureaucratic mindset and how political calculations get made. That's his territory more than mine. So I'm very respectful of that view. I however tend to think that the Chinese are going to be pretty in favor of high-speed AI development. The reason is essentially the same reason that I was talking about earlier, about informatization and electrification. This is essentially a basic enabling technology that is useful for all of the things that can flow from it.

My hunch would be that they would make the same bet with AI that they made with the Internet back in the 90s. With the Internet they said, “This technology could be dangerous, it could be out of our control, but we think actually it's going to be really beneficial and that we can control it.” That bet paid off for them very, very well. My hunch is that they will look at AI as a similar sort of thing. Yes, it's possible that this could get out of our control, but we think unbalanced, this is really important technology. It's going to enable a lot of other things and we'll probably be able to figure out how to control it because our control systems are so good.

It is interesting because if you look at the big development in Chinese AI recently, DeepSeek, it was not the result of some state program. It was the result of basically a quant hedge fund guy in Hangzhou sitting around and saying, “Oh, I could actually build a pretty good LLM model flowing out of my quant activities. Let's give it a go, see how it happens.” It worked great for him.

The reports that I have seen suggest that this created a certain amount of disquiet in Beijing. They said, “Well, where did this guy come from and why is he not coming out of the official plan?” So there is some risk. I think that they would be a little bit nervous about freelancing on this. But on the whole I think they'll see this as another basic enabling technology that they should promote and figure out how to control over time.

Dwarkesh Patel 01:41:23

To the extent that China's advantages lie in its ability—in situations where centralization is important—to do it… Ironically, this might be a case where centralization might be helpful and they might not do it because there are these tremendous economies of scale in AI training. You have to keep increasing the training cost by 4x every single year. Especially given that the availability of these advanced chips is lower in China, it would be even more helpful to have one person who can use them all for training. This might be the situation where it would make more sense for the government to say, Huawei and ByteDance, you have to give all your chips to High-Flyer.

Arthur Kroeber 01:42:09

I think we can say with high confidence that that's not going to happen. As powerful as the Chinese government is, they do not have the ability to force everyone into one room for this. Huawei's got its own thing going. Bytedance is going to have its own thing going. Xiaomi is going to have its own thing going. Alibaba is going to have its own thing going. All of these have sufficiently large revenue streams and access to capital that they can move independently. They will also be able to leverage their connections within the system, both at the central level and the local government level where they get a lot of their support to do their own thing. You will not have a national monopoly solution to this. That almost never occurs in China.

Dwarkesh Patel 01:42:58

Interesting. That's a significant statement and maybe updates me downwards on the probability of China winning in AI. I think one of their key advantages would have been—

Arthur Kroeber 01:43:09

So you think the fragmentation becomes an obstacle.

Dwarkesh Patel 01:43:12

Yeah, especially given the availability of compute in China.

Arthur Kroeber 01:43:15

Right. The ability to concentrate your limited compute resources in one thing instead of scattered around. I think that's a fair point. That then comes back to how fast can they solve the chip problem domestically. If they can solve that really fast, then that constraint goes away.

Dwarkesh Patel 01:43:34

One thing that will be important as this AI stuff is happening, is to have something that is equivalent to the red telephone. I do buy some of the crazy or scary stories about what could happen with AI. It’s not necessarily because God takes over, but more so… It's happened many times in human history that some more coordinated group has managed to do some coup or slowly take over control.

It’s the kind of thing Cortés did or Pizarro did. A key way that can be prevented, or a way we can reduce the risk of that, is that different human groups should be in tight contact with each other and have the same kind of mechanism that vaccination creates with diseases. The key advantage that the Spanish had over the New World empires was that the Spanish knew how each previous conquest had gone, but the Incas and the Aztecs didn't know about what strategies were used, how horses work, how steel works.

How that translates here is that there might be some crazy things these AIs try. Especially if you have this bifurcated world, which is actually very valuable from an alignment and safety perspective because you have this independent experiment or independent lineage… If their AI tries to do something crazy, it's very important that they feel that they can tell us and we do the exact opposite. It's a long preamble to ask the question of how we set up that kind of high trust or prerequisite understanding on these kinds of issues.

Arthur Kroeber 01:45:05

That is a terrific question. That really gets to the heart of a lot of stuff. I don't have a great answer to that because fundamentally what you're saying is that there needs to be a communication system, a network of interactions between two highly competitive zones, the US and China, where they have to be able to operate with some degree of trust.

I think the problem that we face in trying to analyze this is a lot of the analogies that you might reach for break down at some point. You brought up the red telephone, which basically is a reference to the line of communication the US and Soviet Union had during the Cold War. The threat that you were trying to avert there was nuclear holocaust. In some ways I think that's an apt analogy and we kind of need to think broadly along those terms.

But the specific problem that we were dealing with there were these very self-contained systems that were clearly under highly centralized control by a couple of people. So having that one-to-one line of communication was an effective response to that particular problem. With AI, it's much more diffused, much more decentralized, much more unpredictable, what the problems could be.

It's kind of easy, I think, to create… Easy, we say in retrospect, but it was hard at the time. But it was a sort of, if you will, maybe a two or three dimensional process to come up with game theory to enable us to manage the nuclear balance of terror. I don't want to minimize Thomas Schelling's achievement. It was great. But it was like a two or three dimensional problem. With AI, you have a more than three dimensional problem. It is consequently more difficult to figure out how to create a game theory solution to this that is kind of stable.

I think what you can say at a minimum right now is that there are ways in which the US-China relationship today resembles that of the United States and the Soviet Union in the 1950s, that is before the Cuban Missile Crisis. I don't want to exaggerate this too much, but the sort of diminution in exchange in communication over the last six or seven years is really alarming.

I’ll give a simple example on this. When Trump came into office the first time in 2017, he inherited a structure where there were approximately 100 working-level dialogues between US government and Chinese government officials. Plus you had all of this investment in private sector stuff that was going on. But just at the formal level, there were over 100 dialogues. He got rid of all of them, 100%. He then replaced them with one, which was the trade talks that he had in his first term. That was the only mechanism that we had of communication.

Did this create a specific problem? It did. If you look at the early stages of COVID, back when SARS came out in 2003, there were staff members of the United States Centers for Disease Control embedded in the Chinese Centers for Disease Control or their equivalent organization. That relationship, in fact, grew significantly, in part because of SARS over subsequent years. That was a very, very useful channel of getting information out. It was not perfect, it had plenty of problems.

But the absence of that channel, I think, was really catastrophic in the early days of COVID where we literally had no way of understanding what was going on within the Chinese system. There was no background back channel that enabled us to get even a little bit of a head start on what was going on. I think that had very, very dire consequences. You can probably trace many, many, many deaths and a lot of social chaos to the inability of the US and China to communicate at that critical moment.

We have essentially dismantled the whole structure of communication. The Biden administration basically took the same view. They introduced a few more things. There were a few more communication channels that they handed off to Trump that I think have not been made use of. So we're back to essentially zero, except for the trade. That is a very, very dangerous situation to be in with regard to global public health, any kind of global coordination, whether it be on macroeconomics or climate efforts or whatever And on these issues like AI.

I think that's a really bad place to be. We're starting at a rather dangerous position. It should be a priority to figure out how we can get there. It's very hard to see how we get there given the political environment, frankly, in both countries. In the United States you have this narrative that China is this bad actor, they have an illegitimate government, they are a systemic rival. We basically have to contain, constrain, control them. Any kind of interaction with them is in and of itself dangerous.

You have people in Congress saying, “I don't want to go to China. There's no point in going to China because I'll pay a political price for it. If I go there, I will be tarred as someone who went to China and therefore is tainted. And what can I possibly learn? I already know what I need to know.” So you have a deliberate effort in the United States to reduce the channels of information that we have with China and direct experience, which is very bad.

I think one of the bad outcomes of COVID in China was that they kind of locked themselves off from the rest of the world for almost three years. They got really obsessed by this self-sufficiency industrial policy complex. The Chinese system has always had a lot of difficulty communicating with the rest of the world. This has been persistent for many decades and centuries. This gave them a great excuse to just not do the thing that made them feel very uncomfortable, which is engage in communication with the rest of the world. They concluded that that was kind of okay.

So they also retreated into a very sort of paranoid narrative about what the US is up to, which is to a significant degree justified but not entirely. I think they lost sight of the value of a really robust communications alliance with the rest of the world. Both sides are entrenched in very suspicious, self-reinforcing narratives about the other, which makes it incredibly difficult to forge a better communication system.

I have to say I'm fairly critical of both the Biden administration and the Trump administration, different ways of how they handled this. I think the Biden administration could have done a lot more, a lot faster. I think they sort of had the idea that it was good to have more communication, but they waited for two years before they did anything. Then they were very parsimonious about how they did it. I think they could have been much more aggressive. They weren't. That is a problem.

Then I think the Trump administration unfortunately just doesn't place strong value on having respectful communications with anyone in the world, frankly. Then when you compound this with the paranoia about China, then it just creates more—I don't want to say paranoia, that's too loaded a term—but the suspicion, the fear, the distrust of China that exists. It's very hard to see how you get it. So I agree with you that we need something like that. I'm kind of pessimistic that we'll get there anytime soon, unfortunately.

Dwarkesh Patel 01:54:09

The red telephone point you made is actually really interesting. It didn't occur to me before that AI is such a diffuse thing. It's like saying we're going to have a red telephone for the Industrial Revolution.

Arthur Kroeber 01:54:20

Yeah, exactly. How do you do that? You can't.

Dwarkesh Patel 01:54:23

But on the point about the ability of both sides to perceive each other… It is true, as you say that because maybe China has become more closed off, their perception of what's happening in America has been diminished. I would argue that the opposite is even more true. Our understanding of China is more limited than theirs. A big part of this is just that every country in the world has some understanding of what's happening in America because of the overwhelming cultural significance, the charismatic nature of American politics. Everybody knows who the main senators and Cabinet people are, all around the world.

When I visited China six months ago, it was shocking to me that you could go to cities with 20 million people like Chongqing and Chengdu and… I don't think I saw a single white person in Chongqing, which is insane. You're literally seeing seas of people and none of them are from the West.

In fact, that was what motivated me, that trip, to do these kinds of episodes. I think people are right to say, “Look, fundamentally I read a book like this and I'm learning a lot about how China tangibly works.” That’s opposed to, “I go there, I'm going to chat up taxi drivers.” I'm not going to learn about whether they're going to invade Taiwan by chatting up taxi drivers, right?

The kinds of things which should be obvious—that probably aren't obvious to these Congress people that are cavalier about what information they get, certainly wasn't obvious to me at least on a sort of more subliminal level—is just how big the country is. I think people talk about it as if it's like a small thing, “Oh, China's over there and we're going to deal with it.” Especially when you think about how we're going to change its government, we're going to constrain its global impact or something. It's really big.

Arthur Kroeber 01:56:15

It is really big. It seems like a very banal observation, and yet I keep coming back to it as one of the fundamental facts that continues to be underestimated about China. It's just how big it is geographically and in terms of population.

But the question there is, what is the value of these things? It's a fabulous book. I'm very proud of it and a few thousand people read it. At any given moment over the last 15-20 years, there have been north of 300,000 Chinese students in the United States. I don't know what the total number is over time, but it is quite a large number. The cumulative effect of personal experiences can be gigantic, particularly if it is extended over time. It's really, really important.

Then you have the impact of these more structured engagements. You go as a tourist, you talk to some taxi drivers, you see this and that. What do you learn? I'll give you a concrete example. So one of my side gigs is that for a number of years I taught an executive MBA course at NYU on China's economic rise, and what that meant for companies and so forth. Usually I had about 20-25 people in the class. I taught it for three or four years before COVID and then two or three years after Covid started.

In the years that I taught it before COVID, there were always reliably four or five people in the class that had spent some time in China. Many of them had done so just as part of this executive program. Because part of what you got by signing up for this EMBA is that they took you on these global study tour trips and you would go to this country or that country and you would be there for a couple weeks and you would talk to a few companies and this and that. It's pretty superficial.

But what I found was that when I had a class that had a few of these people sprinkled in that had that experience, or a little bit of work experience—or they may have gone as a tourist, had some kind of student experience or something—the class discussions were always much more nuanced. People did not retreat into these sort of cartoon stereotypes of what China was like. Everyone who had been there could say, actually it was a little bit different because here's my personal experience.

Then the classes that I did after COVID had set in, they weren't doing these study tours so no one in the class had been to China as part of the program. Luck of the draw, I didn't find any people who had been there as part of work or whatever. I had zero direct experience of China in the class for a couple years.

The baseline view of what China was was way more negative and much more stylized and, dare I say, cartoonish. They were essentially getting their view formed entirely by whatever they were picking up in the media and from their preconceptions. There was no check on that.

So I saw in just a very small but very concrete way that if you don't have people going back and forth and there aren't some people in the mix, you don’t get people who say, “Wait a second, what you're saying doesn't make any sense because I was there and I saw this, and it's totally different.” I'm convinced that has a gigantic impact.

I think it's really, really damaging if people in positions of authority in our government and our political system refuse to expose themselves to the actual reality of what China is, which is essentially what they're doing now by making it politically impossible for anyone to go to China and see what's going on there. That is a very, very dangerous place to be. It's ignorance.

One of the things that drove me and many Western China types crazy about being in China for many, many years is that you would go and frequently you would have this experience. You'd be sitting in a seminar room or think tank or whatever. Some Chinese experts would say, “We Chinese, we understand America much better than you Americans understand China.”

For years and years I basically thought this was BS. First of all, most people in China don't travel. They have no idea. Even in the expert class, they had a fairly superficial kind of understanding. So I always felt pretty confident that I and people in my cohort, we understood China just as well as they understood the US and their blind spots towards the US system were very, very clear. For years and years that drove me nuts when people would say that.

Now I think it's just completely true. All you have to do is look at the student numbers. You have 300,000 Chinese students in the US until their visas get revoked. We're struggling to get to a thousand US students in China. In the US we closed down the Chinese consulate in Houston as part of a trade spat back in 2020. China responded by closing down our consulate in Chengdu. It’s a complete disaster, the whole area of western China, really important, that we have no eyes on officially anymore.

We decided that their Xinhua journalists are not really journalists, which is sort of true. We forced them to register as government agents. The Chinese responded by kicking out a lot of our journalists. That was a massive ‘own goal’ for us.

The types of interactions that US people have directly on the ground in China have been massively restricted compared to what they were a decade ago. That’s partly as a result of Chinese government action, partly as a result of really misguided actions on our part, and partly because of just fallout from COVID and various other things. But where we are is bad. There is an information asymmetry which I think operates to the disadvantage in the United States right now.

Dwarkesh Patel 02:02:47

I worry that this sort of cycle of escalation will seem in retrospect to people—especially if it leads to a hot conflict—as mystifying as World War I seems to us now. If it's instigated by something in Taiwan, that's a different story there. You can just tell a very direct through line.

But you look back at World War I, why did Germany do what it did? Was it worried about encirclement? Why was it worried about encirclement? There's this weird thing with the Russian ambassador who didn't get back in time or something. It would just be this kind of thing where it's like, “Why did we have to have this adversarial relationship with China again?” Explain it to me when we're 50 years removed and the sort of day to day news isn’t as salient.

Arthur Kroeber 02:03:28

I basically agree with that. We should be spending a lot more time trying to figure out how to make this a workable relationship. There's going to be competition, there's going to be conflict. There are some real conflicts of what we define as core interests on either side which are tough to resolve. This is not easy.

But we basically both have to recognize… This is more of a problem on the US side. I think the Chinese recognize that the US is not going anywhere. There is a little bit of a meme in Chinese foreign affairs circles with the decline of the US and the rise of the east. That is very officially popular. I think most people don't really believe that because they just have too much confidence in the long-run resilience of the US system. I think they're basically right about that.

I think we on the US side have to get a little bit more attuned to the idea that there is no end point to this. This is a relationship which will go on forever. There's no winning, there's no losing, there's just managing. We need to do a much better job of figuring out how to manage it and keep the conflicts and friction in as contained a zone as we can. It’s not easy to do.

Dwarkesh Patel 02:04:48

There is this interesting volatility in how people think China is doing. Some piece of news will come out about electric vehicles and people will be like, “China will obviously dominate us”. Then maybe some economic data will come out and they say, “China is collapsing.” People in their own countries have this sense that a lot of things are happening. Some things are going well, some things are not going well. But in 20 years, America's not going to collapse and it's also not going to have destroyed every other country. There's a long-run trend.

Whether China has 1% higher growth rate or 1% lower, it's just going to be a powerful nation at the technological frontier. There's no sort of very immediate implication of what exact growth trajectory they are on, with respect to the most important questions about how we should engage with them and so forth.

Arthur Kroeber 02:05:45

That is a really important point. I want to just dwell on that a little bit. The attraction that people have towards extreme narratives about China—total domination or total collapse or so forth—is very strong in my view… Again, it's a product of ignorance. The less you know about the realities of place, the more you can construct these fictional stories which take some truth and then absolutize it. The more you know about reality, it's more messy than that. It’s neither that nor this. That I think is just another argument for why you have to have much more deep engagement so people are engaging with realities rather than fantasies.

You can see this again in very mundane ways. If you just look at the way that the Trump administration has conducted its trade negotiations with China, it was premised on the notion that China's economy was so weak that you hit them with tariffs and they will immediately capitulate. That was a ridiculous idea. It's taken them a couple of rounds to figure out, maybe that wasn't quite right and we have to take this more seriously. Getting away from the extreme narratives is important in general.

I think also on the US side, also to some extent in China as well, is that a lot of the things in the US political context are blamed on the rise of China. We have this problem or that problem because of China. They are predominantly exercises in scapegoating.

We were talking at the beginning of the conversation about the change in the industrial sector, hollowing out of US manufacturing employment and so forth. If you look at the trend of US manufacturing employment as a share of total employment, it’s a straight line from 1946 to today that looks like this. You cannot find the Cold War, you cannot find the rise of Japan, you cannot find NAFTA, you cannot find China. You cannot find anything. What you find basically is the march of technological progress. That's basically what that line tells you.

So all of these stories about China did this or China did that… There was some impact. But mainly what the United States failed to do as the world was transforming in the late 90s and early 2000s, was to take seriously our responsibility to change our systems of redistribution, helping people out. We failed on the social contract domestically. There were failures of domestic policy which were then blamed on China.

I think similarly, if you look at scary stories about how China's going to take over the world… BYD is a very good company. Huawei is a very good company. CATL is a very good company. You look at the totality of the corporate universe in the world, it's still pretty big. US companies are incredibly powerful and adept across many technological sectors. They are very good at what they do. In many cases, they still have very strong positions in China because they're so much better at what they do than the Chinese competitors.

We're dealing with the rise of a new entrant on the global stage. That's uncomfortable and there's some jostling there. But we should also not overstate the degree to which, even on a sort of a commercial industrial side, that we are sort of existentially threatened by this. There's some real challenges there, but a lot of this notion is that we will not have an auto industry in five years because of BYD… Let's get a grip, folks.

We could end up there, but if we end up there it's because we will have made a bunch of really bad policy decisions that got us there, not because this competitive threat exists. Mostly it's on us to figure out how to get things right on our own terms, within our own system. Then I think basically we'll be fine. But right now that isn't where the conversation is particularly.

Dwarkesh Patel 02:10:08

You first went to China in 1980.

Arthur Kroeber 02:10:11

’85.

Dwarkesh Patel 02:10:12

’85. You've been visiting and living there on and off ever since. We're all aware that obviously China's developed a lot since then and so forth. What are some non-obvious changes you've seen or maybe even things that have stayed the same through all this change?

Arthur Kroeber 02:10:28

I often describe China as a very turbulent and very deep ocean. You skim along the top and you see the huge waves: all the stuff that's been built, all of the cities, the high rises, the industrial parks, the high speed rail and so forth. Then you go below that and there's a very deep still ocean of ideas, concepts, bureaucratic tendencies and so forth that have built up over centuries that in a lot of ways does not change that much.

I think about what the United States was like socially in 1985 compared to what it is today in terms of the space for gay and lesbian people, trans people, social acceptance of different lifestyles, awareness and attention paid to racial issues, all these kinds of things. The changes that have occurred in the United States over the last 40 years are substantially greater than the changes that have occurred in China over that time period in terms of people's fundamental understanding of how we interact together in a society.

Partly that just reflects that the US is intrinsically a somewhat more complex society than China with all of these layers of races, the whole immigrant experience. Just in terms of the number of foreign-born people, it was much lower in ’85 than it was today. But socially we are much more… There's just been a lot more change and a lot more evolution of people's basic attitudes towards what they think about society and how we should interact.

China in that sense has been more stable. It's more cohesive, it's more attached to rather deep-rooted notions of gender relations, patriarchy, and how society should hang together. So as we think about all of the change that occurred in China, which is real—and some of the changes in social relations that have come with the abolition of the planned economy system and the replacement by this basically capitalist system with a Leninist political carapace—those are very significant.

So I don't want to underestimate how much transformation has gone on in China over that period. I think it's been substantial, but in the US it's been even more in many ways. That is something that's worth reflecting on, particularly as we do these comparisons. There's a kind of intrinsic churning dynamism to the American system that is just always there. It is often very uncomfortable to live through and we're living through a very particularly uncomfortable moment now. In China there's some of that, but there's a lot of containment and there's more stasis in a lot of the fundamental social relationships.

I lived in China more or less full time until 2015, and then I started doing half and half after that. Now I principally live in New York. One of the things that always struck me when I would come back from China to the US is just how much grassroots cultural activity you have all over the United States. We have a very distributed system of cultural activity here. In China you have a few big cultural hotspots where things happen, and then it's all kind of placid.

The variety of experience that you have in the US is still unrivaled. You were talking earlier about the fact that there's still this huge discrepancy between the average income in China and the average income in the US. This is partly a reflection of that. It's still, on average, a much poorer society than the US. So the surplus that's available for people to do stuff that's interesting, but not economically productive in a narrow sense, that space is still a lot smaller. It's not just political repression that's doing that. It's a combination of the level of income and just the ingrained social structures.

Dwarkesh Patel 02:15:31

The broader point you're making here is important to emphasize. Sometimes if people are hawkish on China, they have the sense that they're doing everything wrong. They're this aggressive, belligerent power. This is the new Stalin or Hitler. But conversely, when people say that there should be a productive relationship, then there's also this other attractor state where people are just so uncomfortable with cognitive dissonance that they say, “You can see how well their system works, central planning works, authoritarianism works.”

Arthur Kroeber 02:16:11

Right, let's not take those lessons. Those are very bad lessons.

Dwarkesh Patel 02:16:15

You just see those two patterns of correlation so often that it's worth emphasizing that you can have a productive relationship with a country and still think that not only is authoritarianism morally wrong, but it actually has had bad impacts on growth or on the day-to-day life of people and their culture or whatever else. I think people sometimes have trouble just holding two thoughts in their head at the same time there.

Arthur Kroeber 02:16:38

There's also a very strong pattern. There have been a few good books written about this. American attitudes towards China in particular have tended to swing from one extreme to the other. Americans and Chinese are kind of the same because they're both pragmatic. They like money, they like business. If you go to China, it's very easy in my experience to chat with people and get along. So that level of connection is fairly easy. Then the other swing of the pendulum is that they're this enormous amorphous force that is committed to a devilish system that we don't like. Finding the middle ground in that historically has been very, very difficult for the US to do.

Dwarkesh Patel 02:17:32

Final question. Right now, the US and China are engaged in this negotiation to figure out some somewhat stable way to come back from Liberation Day. What is the most positive, plausible story of what comes out of this?

Arthur Kroeber 02:17:51

My expectations for this are very low. I think the basic problem here is that we have on the US side a collection of grievances and complaints that have not coalesced into a very coherent agenda of what we want from ourselves and what we want from China. If you don't know what you yourself want and you don't know what you want from the other party, it's very hard to engage in a constructive negotiation. That's my fundamental take on where we are right now.

This is a normative statement but to me, what we should be aiming for is trying to figure out what are the terms of coexistence between the US and China in the coming years and decades. The most desirable outcome would be some kind of an agreement under which we create a broader permission structure for Chinese companies to invest in the United States in a lot of these key industries that we're talking about. I think that's good for the United States because I think it is reasonable for us to try and rebuild certain aspects of our industrial manufacturing system. I think we let those atrophy too much. So selectively, it does make sense for us to try and rebuild that. Rebuilding that should incorporate the best anywhere in the world. Some of that is now coming from China.

I also think—just in terms of enhancing mutual understanding and getting better communication channels—investment is even more than trade a very good communication channel because it forces you to go to the other guy's place and put down roots there and figure out how things work and figure out ways to communicate with one another.

Now the obstacles to that are enormous, both because of the US narrative that this is intrinsically unsafe and because I think the Chinese will be very reluctant to allow a lot of their technology high fliers to go and set up in the US and risk having their technology leak away there too. So I'm not saying that that is easy or achievable, certainly not in a few months. This would be over the course of a year or two or three, you might be able to negotiate some terms of engagement there.

The big problem that China creates for the world as a whole right now is this fact that they just are unwilling or unable to generate enough domestic demand to keep their own economy running at full steam. That's bad for their own economy. It creates a lot of friction in the rest of the world. It does contribute to the narrative that China is not really all that interested in creating a situation where everyone benefits from their growth. It's much more a “We're in this for ourselves.”

It would be helpful for the world economy, helpful for the Chinese people, and helpful for general stability, if the Chinese could recognize that they need to do a lot more to promote the demand side of their economy. It can't just all be about endless investments in new technologies.

I think they're coming to a point where they may be forced into that. If you look at most of the metrics that I and my team follow of return on investment in China, they're going down. The number of zombie companies that still exist but don't even generate enough revenue to pay the interest cost and their debt, that's going up. There's a lot of pressure that's building within the Chinese system because of this obsessive concern with the supply side of the economy.

They'll continue to do quite well overall in technology production, but the financial constraints are rising. They have some self-interest here in coming up with a more balanced economic model. Having some pressure from the outside to get them in that direction would be good.

The final point I would make on this though is that on the US side—which is of course where I generally sit—it really is critical to have a realistic understanding of what's happening in China and how they're interacting with the rest of the world. One of the big dangers that we have in the current Trump administration strategy—which was also to some degree apparent in the Biden administration strategy—is that there is this belief that China can be contained and that you can essentially construct an alliance of countries that will stand up to China and try to constrain it with you. It's kind of a replay or dusting off of the Cold War playbook where you had the Soviet Union and the Warsaw Pact, you had the United States and NATO and all of the informal allies everywhere else. You had the blocs against each other and the bigger and more economically productive bloc won.

That is just not going to work. The Chinese have seen this coming for a long time. They have long had the view that the US was going to try and constrain them. They said, “How do we get around this?” Their answer is not, “You build a bloc, I'll build a bloc against that.” Their answer is, “I will operate so that it is impossible for you to build a bloc that you want.”

Look at China's international engagements. More countries have China as their number one trade partner than any other country in the world. 140 countries trade more with China than they do with the United States, which was not true during the Cold War. You go to Southeast Asia or Latin America and you say, “How would you like to join our anti-China alliance?” They will say, “No way, this makes no sense for me. I import a lot of my industrial inputs from China, a lot of my consumer goods. This is really a critical relationship. I might have some concerns about the trade balance, but I am not going to go up against this country that is now either my number one or my number two economic partner, certainly on the trade side and increasingly on the investment side.”

So that is a non-viable strategy. It will never work. China has integrated itself into the global economy very successfully. This is just a fact that we need to live with. We need to have a strategy that recognizes this is a reality but that there are a lot of strengths that the US has to bear if it's willing to play them. We have the advantages of a dynamic system and frankly, a lot more creativity and flexibility in our financial system than the Chinese do. It's perfectly possible to come up with a coexistent scenario in which we do really well.

We have domestic policies that sort out our own imbalances, both economic and social. We have a significant degree of engagement with China that is competitive, but hopefully can be bounded so that we don't wind up in a hot conflict situation that would be the worst of all possible worlds.

Dwarkesh Patel 02:26:04

All right, that's a good note to close on. Arthur, thanks so much for coming on the podcast.

Arthur Kroeber 02:26:07

Great. Thanks so much. I really enjoyed it.

Ken Rogoff is the former chief economist of the IMF, a professor of Economics at Harvard, and author of the newly released Our Dollar, Your Problem and This Time is Different.

On this episode, Ken predicts that, within the next decade, the US will have a debt-induced inflation crisis, but not a Japan-type financial crisis (the latter is much worse, and can make a country poorer for generations).

Ken also explains how China is trapped: in order to solve their current problems, they keep leaning on financial repression and state-directed investment, which only makes their situation worse.

We also discuss the erosion of dollar dominance, why there will be a rebalancing toward foreign equities, how AGI will impact the deficit and interest rate, and much more!

Watch on YouTube; listen on Apple Podcasts or Spotify.

• WorkOS gives your product all the features that enterprise customers need, without derailing your roadmap. Skip months of engineering effort and start selling to enterprises today at workos.com.

• Scale is building the infrastructure for smarter, safer AI. In addition to their Data Foundry, they recently released Scale Evaluation, a tool that diagnoses model limitations. Learn how Scale can help you push the frontier at scale.com/dwarkesh.

• Gemini Live API lets you have natural, real-time, interactions with Gemini. You can talk to it like you were talking to another person, stream video to show it your surroundings, and share screen to give it context. Try it now by clicking the “Stream” tab on ai.dev.

To sponsor a future episode, visit dwarkesh.com/advertise.

(00:00:00) – China is stagnating

(00:25:46) – How the US broke Japan's economy

(00:37:06) – America's inflation crisis is coming

(01:02:20) – Will AGI solve the US deficit?

(01:07:11) – Why interest rates will go up

(01:10:55) – US equities will underperform

(01:22:24) – The erosion of dollar dominance

Dwarkesh Patel 0:00:00
Today I’m speaking with Ken Rogoff, who is a professor at Harvard, recent author of Our Dollar, Your Problem, and former Chief Economist at the IMF.

Ken, thanks so much for coming on the podcast.

Kenneth Rogoff 00:00:12
Thanks so much for having me and welcome to Harvard where we’re filming this.

Dwarkesh Patel 00:00:16
In your book you have a lot of anecdotes of meeting different Chinese leaders, especially when you were Chief Economist at the IMF. It seems like you had positive experiences. They would listen. You met the Premier with your family, and he would listen to your advice.

One, how does that inform your view about how competent their leadership is? Two, how do you think they got into this mess, with their big stimulus or whatever else you think went wrong?

To the extent that when you were talking to them in the early 2000s, it seemed like you were seeing eye to eye, or that they would understand your perspective. Do you think something changed in the meantime?

Kenneth Rogoff 00:00:49
First I want to be careful to say that they listen to everybody. The Chinese are way better than we are at hearing a hundred different views. Mine would be one of many that they heard.

I was very impressed by the competence of the Chinese leaders. I actually gave a lecture in the Party’s training school where, if you’re a mayor, a provincial governor, any bureaucrat on your way up, you go to this thing which for them is like Harvard Business School.

They really looked for competence. Of course there were various loyalty things. But when you met the leaders—and I met a lot of them when I was at the school—they actually asked really raw questions too. They said things I couldn’t believe they were asking. And I was told that within the school, you're allowed to say anything.

So they had that system for a long time. When you met Chinese technocrats—or even the mayor of Shanghai—they were impressive. I'm not saying ours aren't, but it's a mix. I think you know that. I think Xi Jinping has really changed that. He’s been the president since 2013, and over time he’s pushed out that system and moved more toward loyalists, people who are less technocratic.

Probably the most important talk I ever gave in China was at what's called the China Development Forum in 2016. It's this giant hall that had most of the top leaders in the party. A lot of the elite of the tech world, Mark Zuckerberg and many others were there. I said, “Okay, I'm looking at your housing. I'm looking at your infrastructure. It looks to me like you're going into a classical housing crisis problem. Your catch-up is over. Your demographics don’t look good.”

I gave a list of things. “And by the way, it looks like power is becoming very centralized in the economy.” And I said, “I'm a Western economist. You're doing an amazing job. What do I know? But I don’t think that would be good for growth.” After I gave the talk—I just figured you only live once, you just have to say what you have to say—a couple of top leaders came up to me and said, “Professor Rogoff, we very much appreciated your remarks.” I was thinking, “Oh no, they’re going to put me in jail or something at the end of this.”

I’m less impressed by them now. And I’m worried. Let’s say they get into a crisis—which I think they’re in now. I think they're still in a deep crisis—or somehow hotter heads prevail between the United States and China and we get into some kind of entanglement nobody wants. I worry that we’re not as competent. I’m speaking about right now. We have some very good people, but the average quality at the very top, I think, has gone down. And China’s not as competent either. That’s a recipe for having bad things happen.

Dwarkesh Patel 00:04:16
You mentioned in the book that you had to clear your talk before. So you gave them a sort of watered-down version of what you were going to say. I have to say, it would take some gusto to go up to the top party leaders. Were you nervous while you were giving the talk, saying, “Oh, it’s too centralized”?

Kenneth Rogoff 00:04:33
I mean I was pretty experienced by that time. I frankly never used notes, so the idea that I was going to read my speech didn’t even occur to me. Maybe it was a little bit spontaneous. But I certainly felt at that moment, what am I here for? What’s the point of coming to this? Why don’t I talk about the elephant in the room? Everybody knows this. I don’t know if everybody knew it, but it was clear to me and I’m going to say it.

Dwarkesh Patel 00:05:03
I think a lot of people in that situation—even though they should or the logic makes sense—they often don’t.

Kenneth Rogoff 00:05:08
Yeah, I’m a professor. So people who had a big tech company or a finance company, or all these other businesses, most of them can’t afford to do that. I think they know that when they invite a professor, they can’t cut your funding or something. They can stop you from going again. They invited me again, by the way. Although the second time I just talked to a tiny room instead of the big hall. But, I give credit to the Chinese for listening.

Dwarkesh Patel 00:05:41
You’ve said that the seeds of their current crisis were sown in 2010 with their big stimulus. Is it wrong then to blame Xi Jinping for this? That was before his time. It was Hu Jintao’s government that launched the stimulus that’s causing all these problems now.

Kenneth Rogoff 00:05:57
Hu Jinato did it, but they kept it going. The local government debt that you mentioned, that was an innovation put in with the 2010 stimulus. But they kind of left it running and used it as a stimulus program. Long tangent, but the local governments don’t have enough ways to fund themselves. So they were allowed to sell land to start and fund these construction companies, get revenue, and sustain themselves. They let that keep going.

When Xi Jinping came in, I was told he was going to be Ronald Reagan. I had very good contacts in the intellectual sphere in China. I’m talking about someone who had worked for me when I was chief economist at the IMF, other people I knew whom I really trust. They're really smart and well-connected, still are so I don’t want to name names. But they were telling me, “He's going to change everything. This is really the time we’re going to liberalize our markets a bit. We’re going to do the things we haven’t done.” And he didn't do that much. If you look at China’s growth, it actually slowed down quite a bit when he came into power.

There are different ways to measure a country’s output because China produces completely different stuff than we do. They use their currency and we use ours. Nothing's perfect. But one way to do it is this. What do they report their output to be in Chinese currency? What do we report ours to be in dollars? Then we use the exchange rate to compare. We can look at growth that way and if you do for China, it’s been spectacular. It’s been very, very good.

It’s obvious they're pulling it out of thin air. But there are these approaches trying to control for how you’d really compare how an ordinary person lives or how an ordinary firm gets by. When you look at those measures, China’s growth is quite a bit less. If you go 1980 to 2012, the official growth rate is almost 10 percent. This purchasing power parity rate—forgive me for using those words—is like just over 7 percent.

If you look in more recent years, it’s really slowed down a lot. Even the official numbers have slowed down. I don’t know the number off the top of my head, but it’s 6% or 7% for Xi Jinping, and maybe only 3.5%. They’re starting from a very low base.

Okay, things were gonna slow down. It’s not all his fault. But I think he’s been reluctant to take risks and I think it’s gotten us to where we are. I think they’re in a lot of trouble. They’re overbuilt in infrastructure. They’re overbuilt in housing. Have you been to China?

Dwarkesh Patel 00:09:00
I was there six months ago.

Kenneth Rogoff 00:09:02
Where did you go?

Dwarkesh Patel 00:09:03
Shanghai, Beijing, Chongqing, Chengdu, Hangzhou, and Emeishan.

Kenneth Rogoff 00:09:09
So you saw a few of the medium-sized cities. At least one of them, I think, is the new tech center. I can’t pronounce it.

Dwarkesh Patel 00:09:21
Hangzhou?

Kenneth Rogoff 00:09:22
Yeah it’s a big tech center. Some of the smaller cities don't feel like the big cities. And 60% of Chinese income is from what they call their tier-three cities.

I grew up in Rochester, that’s like a tier-three city in the United States. But you could pick Cincinnati, Liverpool. Rouen— I may not be saying it right—in France is an example of a tier-three city. And they have invested like crazy. I’ve been to a few and I’ve studied the data on it a lot. They have amazing roads, amazing real estate, amazing housing. But the feel of death in those cities...

They were very good at building stuff. The Soviet Union was very good at building cement factories and steel plants and railroads. But they’ve run their course. They have other stuff: green energy, AI, electric vehicles. But believe it or not, that stuff’s still tiny compared to infrastructure and real estate. Real estate’s a third of the economy by some measures. So I think they’re in a lot of trouble now in China. They let it go on too long.

But again, I wasn’t running things. If things seem to be working and you try to change things, you get thrown out. It’s not easy to be in those shoes.

Dwarkesh Patel 00:10:48
When I was in China, we visited a town of half a million people outside of Chengdu, so one of these tier-three cities. Arriving there, the train station was huge. Compounds are huge. Even when you’re driving around, like a movie theater is this humongous complex.

I realized things were bigger in China. I was used to that because I’d seen these other cities by that point. But I just thought, I’ve seen cities of half a million people. I live in a city of half a million people in San Francisco. This just doesn’t seem proportionate to the size of the population.

Then we visited a Buddhist temple that had been built recently as a tourist site. It was ginormous. You would go through one little shrine, and then behind it would be an even bigger structure and then another one concentrically for like eight turns. It would take you probably 10 minutes to drive through this thing. There was just nobody there. It was like me and three other white people.

Kenneth Rogoff 00:11:42
It’s very much that feeling. And the young people don’t want to live there. I have a lot of young people here as students and I run into people. They don’t want to live in these towns, and the jobs aren’t there. I can’t criticize them for trying that.

If you’d asked me in 2005, “Should we try to encourage people to go out to the Rochesters and the Liverpools and the Rouens in France?” I would have said yes. There’s too much in the big cities. There’s overcrowding. Look what happened to São Paulo. Look what happened to Mumbai. But I would’ve been wrong. These forces are very powerful.

So a lot of their growth, and what they call their GDP, is this stuff. So they’re having to reorient and people just aren’t that flexible. It’s like when AI comes and puts everybody out of jobs. When construction jobs are gone, and all these indirect things are gone, it’s not that easy to move everyone.

Dwarkesh Patel 00:12:42
If it hadn’t been for financial repression, and all this investment had been done through purely market mechanisms, would things have turned out much better?

Even if China gets rid of all financial repression today, they save a lot. So this money has to go somewhere. Are there enough productive opportunities to soak up all these savings? Or could there have been in the past? If they get rid of financial repression, is this a problem solved, or could it have been solved?

Kenneth Rogoff 00:13:10
What everyone’s told them forever is their saving rate and investment rate are astounding. Their consumption rate, it was higher before, but it’s still maybe 45%. Ours is pushing 70%. European countries are a little more temperate, but they’re in the low sixties. Their consumption is very low. They have some wealthy people that you saw when you went to the marquee cities. But a lot of China is living on $200-a-month kind of incomes. You could give them money. You could let them consume instead of exporting it. They’ve been very reluctant to do that.

You could do things to encourage consumption. Actually, even just changing their exchange rate policy to allow it to appreciate more at times, would make imports less expensive. They have been very reluctant to do that. That’s what everyone tells them. That’s certainly what I said in 2016 also.

The ticket to getting people to spend more is to provide more security than they have. First of all, there’s nothing like our Social Security system. You need to save for your old age. There’s nothing like our health system. If you work at one of the big state-owned factories, they give you healthcare, but otherwise you’re on your own.

They’re not allowed to invest abroad. It goes in waves, but they’re not allowed to put their money abroad. So they’re trying to be careful about all of that and not do things suddenly. There’s nothing to do overnight.

But fundamentally, if you’re looking at China and asking what’s wrong, it’s that the consumer isn’t spending enough. What’s happening right now is worse, because housing prices are collapsing. That’s the only thing they really let people save in. You could either save in a bank account, which gets you a crummy interest rate, or in a house. Now they’re going down, so people are cutting back.

They can dig their way out. There’s no magic bullet to make them grow at 5%. By the way, that is the official number but I don’t think they’re anywhere near that. There’s no simple thing, but the general goal would be to try to rebalance investment, and consumption.

Dwarkesh Patel 00:15:37
Let’s go back to your point about whether purchasing power parity is the right way to compare, or whether nominal is the right way to compare. I think in the book you say the nominal comparison of GDP is better because you can’t buy Patriot missiles or oil with purchasing power parity dollars. But if we’re trying to compare the strength of the two countries—their relative strength, especially in a military context—if they can build ships and munitions much more cheaply, and they have to pay their soldiers less, isn’t that actually more relevant if we were trying to figure out who would win in a war? Shouldn’t we actually be looking at the fact that they have a bigger PPP economy than us as a sign that they’re actually stronger?

Kenneth Rogoff 00:16:18
Yeah. So in the book, I’m talking about your geopolitical power, where if you’re going to give money to somebody, what’s it worth and how much can they use it. But no, you’re absolutely right.

They just crush us in shipbuilding. It’s partly because they build commercial ships, and there’s a lot of symbiosis between commercial and military. I think they’re 50% of the global shipbuilding market. For us to build a new aircraft carrier takes years and years and incredible expense. One of the mistakes we’re making is trying to build everything ourselves. Let our allies do some of this. The Koreans are really good at building ships. That’s another place we could be importing from.

You’re right about the soldiers. They’re paid much less. They have a lot of advantages in a conflict against us. We’re way ahead in your department, tech. That is our advantage at the moment. If that were to dissipate, it would certainly hurt.

Dwarkesh Patel 00:17:22
What is your projection? Right now I think their nominal GDP is 75% of America’s, or something like that.

Kenneth Rogoff 00:17:29
Yeah, in dollars, what we call the market terms.

Dwarkesh Patel 00:17:34
What’s your projection by 2030 and by 2040, the ratio?

Kenneth Rogoff 00:17:41
I didn’t realize it was as high as 75%. I thought it was a little lower. I was actually going to say 75% in 2030. At one point in 2024, it was around two-thirds, but it’s really volatile with the exchange rate. The dollar’s really high. When the dollar’s really high, it makes us look bigger.

I think they’ll gain about a percent a year on us, maybe. I don’t think they’re going to grow way faster than the United States.

Dwarkesh Patel 00:18:08
Wait, that means you think they’ll never actually have a bigger economy than us? Or at least in the foreseeable future.

Kenneth Rogoff 00:18:13
It’ll take a long time. We’re talking about the absolute size. They have four times as many people. There were these projections by Goldman Sachs and many others that we’d be like Canada is to the United States pretty soon. Like all these extrapolations, they were proven wrong.

That brings me to a big topic. A lot of people will look at some trend—whether it’s growth in something, AI, China—and just project it into the future.

Dwarkesh Patel 00:18:47
That's a common trend in AI.

Kenneth Rogoff 00:18:52
Economists at least consider ourselves terrible at that. You go back and look at any of these commissions that were supposed to figure out what was going on. They happen periodically. Maybe Brookings puts one together, maybe the government does. My former colleague, the late Dick Cooper, had a whole list of these.

It is very hard to know. But my gut instinct is that what’s happening to China is what’s happened to Japan. It’s what’s happened to Asia, what happened to the Soviet Union. We have a more dynamic economy. We’re not perfect. Maybe we’re screwing it up right now with all the tariff wars and deglobalization. But we have this dynamism and creativity that other places—at least other large economies—just can’t replicate.

They can build stuff. The French have better high-speed trains than we do. I hope you don’t ride on the train from Boston to New York. It’s nicer than it could be, but it’s no high-speed train. You mentioned China. Oh my gosh, their high-speed trains are just incredible. They’re good at that. But the really creative stuff? I don’t want to say they don’t have any. There are some amazing Chinese companies. But let me say that the US is really good at it. We’ve kept that in our DNA. I think it’s very important, to preserve it.

Dwarkesh Patel 00:20:15
The 1% per year compression is actually an extremely bearish forecast. Even people who are pessimistic about China will say, “Oh, by 2040 they’ll be at 125 or 150% of US nominal GDP.” They think it’ll be bigger, but only slightly bigger. The fact that you think even by 2040 they won’t have caught up is actually very extremely bearish.

Kenneth Rogoff 00:20:36
I think they're digging their way out of a crisis. Right now we know their prices are falling. It's not because they're inventing stuff really fast. We know that interest rates are being pushed to zero. All these are signs that demand has been crushed and the economy's not doing well. Historically, they have given numbers which are accurate on average, as best as we can tell. I think that's gotten less and less true in the Xi Jinping era.

Dwarkesh Patel 00:22:02
Let’s go back to the subject of your book. People who are trying to predict when and how China might invade or blockade Taiwan will look at satellite photos of different docks and see how many ships are there. They'll look at military preparedness.

From a monetary perspective, are there signs we could be looking for? For example, if they think a lot of their American-dominated assets will get sanctioned or they won’t have access to them, could we see them liquidating those assets? Would there be any sort of preparations that we could see on the monetary side to let us know they’re preparing for something big?

Kenneth Rogoff 00:22:40
I don’t think they’re going to do it suddenly. But very crudely, on their reserves they’re definitely moving more and more into gold. It doesn’t necessarily help to move into euros or Canadian dollars because those countries might side with us. But they’re doing what they can to diversify. I don’t think they’ve diversified into crypto yet. I had a student do a paper on that. But who knows.

What they are doing very concretely is not just about what they’re holding. That’s the big fact everyone looks at. They officially hold a trillion dollars in Treasury reserves. But the estimate a student of mine did, in a nice paper—and I think others agree with it—is that it's more like $2 trillion. They hold a lot indirectly through proxies.

The other part of it is that the whole financial system runs through the United States. What we sometimes call the rails or the pipes of the system. Your bank gets a purchase and my bank gets that I’m going to get a credit. How does that take place? How does it take place when we're in different countries? The United States just disproportionately controls that.

That, they can’t live with. They could live without their $2 trillion for a little while. But they can’t live without being able to pay suppliers and other countries. So they're working hard on developing their own payments mechanisms. Russia actually did quite a bit in preparation for the invasion. We see China doing that.

Maybe they’re selling Treasury bills, we don’t know exactly. I would advise that to them, if I were a Chinese economist talking to them. But I don’t think it’s going to be something they’ll do suddenly. Maybe Trump will bring down the markets and then there’s nothing to save. But they don’t want to be the ones to bring down the markets and cause a global crash.

Dwarkesh Patel 00:24:46
What would the alternative rails that they're trying to build look like? Are they buying oil from Iran in RMB? Will other countries they need things from accept that? What is the vision for 2030? What’s their goal?

Kenneth Rogoff 00:24:59
Absolutely. There are a lot of countries in Africa and Latin America—some of them are almost client states of China—that they can force. Iran, of course, sells a lot of its oil to China even when there are sanctions. They’re moving in that direction. It’s not just about what you invoice the payment in. It’s how we acknowledge it, how we clear our books. That’s what they’re working on.

It’s coming. The Europeans are working on it too, by the way. Europe is not happy with the situation. They're actually forming a central bank digital currency. It’s moving quite a bit faster than I thought it would. That’s actually one of the reasons they’re doing it, for international payments.

Dwarkesh Patel 00:25:46
Let’s talk about Japan, which you also cover in the book, or their crisis. You blame US pressure in advance of that crisis on the Japanese to raise the value of their currency, and the actions by the Bank of Japan. Zooming out, how much of the crisis is not caused by things like that, but just the fact that high-tech manufacturing as a share of world output was becoming less important? There are demographic factors as well. So something like this was bound to happen to Japan, even if there wasn’t some big crisis that preceded it?

South Korea’s GDP per capita isn’t that high either, at least in comparison to the US. How much of this is due to actions taken by specific actors?

Kenneth Rogoff 00:26:30
South Korea had a crisis in 1983 and another in 1997. They haven’t been crisis-free, by the way.

There are a lot of factors. The demographics would be the most obvious one. The rise of not just China but Korea and other competitors too. Japan invented a business model that a lot of countries have since duplicated. The model was export-led growth. Something people might not think about is it creates competition.

Most countries aren’t as big as the United States, and there aren’t as many different firms trying to do the same thing. Of course, we have trouble with competition here. Famously, in Mexico at one time, there were only two telephone companies, two bread companies, two taco companies. It’s very hard not to let monopolies sit and use their political power.

So how do you get around that? Japan did something that was really pretty innovative. Germany did it too to some extent. In the export sector you’re competing with the world, not just with domestic firms. That created innovation and creativity. Japan did really well with that. But over time, others imitated it and started building the same things. So that’s part of it. The aging is part of it. But I think the financial crisis was a very big part of it.

Dwarkesh Patel 00:28:02
What is the counterfactual? Suppose that the crisis hadn’t happened, how much wealthier is Japan today than it might have been otherwise?

Kenneth Rogoff 00:28:08
Oh, I think 50% wealthier per person, way wealthier. That’s where they started. It depends on which measure you use. But by market exchange rates they were richer than the United States in the late 1980s. Even if you use the more complicated measures, they were richer than any European country, richer than Germany, France, Italy. They’ve moved to the bottom of the rung now.

Okay, the financial crisis wasn’t the only thing. It’s a long story but we effectively forced them to move faster to open up and deregulate than they were culturally and politically ready to. I give that as an example of something in the book where I changed my mind.

I had looked at that for a long time afterward. Going back to 2005—that’s long after the Japanese crisis—I would hear from people like Jiang Zemin, who was the president of China that I met. He’d say, “We’re not going to let this happen to us. There’s no way.” We were discussing how I thought maybe they shouldn’t have such a fixed exchange rate. He said, “That’s what the United States told Japan. Look what happened to Japan.”

I didn’t push back that much with someone like that. You talk to other people. But I heard that from many people. I used to think, how can that be? There’s the Plaza Accord in September 1985 where we pushed them to make their exchange rate more free. But I used to say that you did that in 1985. Carmen Reinhart, my co-author on many things, and I date the crisis to 1992. That’s seven years later.

I continued to think that but over the years, particularly recently, I’ve started to think I was wrong. These things unfold slowly. Crises don’t happen overnight. Japan deregulated and it worked. But they didn’t know what they were doing. I think it was a huge mistake for Japan to agree.

I actually heard from someone who attended the 10th anniversary of the Plaza Accord, held in Tokyo. The person who had been head of the Bank of Japan in 1985 gave a speech to officials. He went like this and apologized very symbolically, “I have ruined our country. I did this. I take responsibility.” Yes, financial repression is bad. But financial liberalization needs to be done gradually. If you do it too quickly, you get a crisis. Many crises are caused by that.

Dwarkesh Patel 00:31:25
Asking as somebody who obviously doesn’t know the details, at a high level how would you explain it to a novice? How could a country be 50% less wealthy than it otherwise might have been, simply from a financial crisis? Whatever they could’ve otherwise produced, why can they still not produce it? A country’s producing a bunch of things. Why are they producing 50% less just because of a financial crisis a couple decades ago?

Kenneth Rogoff 00:31:57
Their case is very unusual, although having a number like 10% or 20% is very typical. In fact, one of my professors at MIT was teaching us about the Great Depression. It’s hard to do without a blackboard but he said, “Here’s how to think about it. We were growing like this, then we get here and we go like this, and then we’re going like this. We never got this back.”

Dwarkesh Patel 00:32:24
There’s a lot of economic models where, Solow catch-up…

Kenneth Rogoff 00:32:29
Yes but what happened with a financial crisis—particularly in Japan—is that it sort of blew up their business model. For example, maybe China wouldn’t have overtaken them so quickly if they’d been able to borrow more freely, if their financial markets were working better, if they had been more adroit. Their consumption collapsed. Japan didn’t quite know how to deal with that.

We in the US were much more brutal in what we allowed to happen than Japan, but we got out of it pretty quickly. I’m not sure we got back to where we were, but we got out of it very quickly. Japan has a very consensus-driven society. They don’t want anyone to be in bad shape. Their struggle with this held them back for a long time. Maybe 50% is too high and I should say 25% or 30%, but a lot better shape than they have been.

Dwarkesh Patel 00:33:25
Just to put it into context, what do you think the counterfactual wealth of America looks like today without 2008?

Kenneth Rogoff 00:33:34
Boy, that's a good question. I'm hesitant because I probably have some paper giving a number for that and I might say the wrong thing. We certainly cumulatively lost a lot. It led to this political crisis that caused us to lose a lot more. I don't know, probably 15% lower. It’s a lot lower than it would be. We had this dynamic which we're living in right now. It's still an echo of that financial crisis.

Now, mind you, you're asking about our national income. Inequality matters and would we have done other things? In some ways, the 2008–2009 crisis was a condemnation of the system and people could see it. Maybe it led to some healthy cleansing. But I think it led to a lot more damage than healthy cleansing.

Dwarkesh Patel 00:34:32
I think this updates me toward the view that financial crises are even worse than I think. It isn't just this bad thing that happens and you recover. If there's 15% lingering even after almost 20 years, then wow that’s huge.

Kenneth Rogoff 00:34:46
You’re losing a lot of cumulative growth. Look at Greece today or Portugal. You kind of get back to where you're having a positive growth rate, but you're not picking up… They're very different from a normal recession. Actually in a normal recession, you go down and then back up. The United States had thought it was immune to financial crises. We really hadn't had one since 1933.

We had a different book that came out in 2009. I mostly write papers, but this was a book with Carmen Reinhart. It was called This Time Is Different. We had some papers published in advance. We said: “No, they're different when you have a financial crisis, it lasts way longer. The slowdown is way worse.” And we were mocked when we were saying that. I think the New York Times had a two-page spread saying how ridiculous everyone thought this was.

We could have been proven wrong and maybe if we’d done things better we would have. But it is the norm. There’s a few exceptions, like Sweden got out in a year or two. But normally they really are different from a normal recession.

Dwarkesh Patel 00:37:05
You say in the book that you expect there to be another spike in inflation within the next decade and also that the fiscal position of the United States doesn’t seem sustainable.

If you go forward 10 or 20 years, when we do hit this, when the piper comes calling, what actually happens? Is it going to be some acute crisis, like what happened in Greece? Or are we going to have some kind of Lost Decade like Japan? What will happen?

Kenneth Rogoff 00:37:32
Typically, you have a crisis of some sort when your debt is high and your political system is inflexible. We’ve checked those boxes. Then you get hit by a shock you weren’t ready for. You get caught on your back foot. It depends on what the shock is, and how we react.

The way Japan reacted was through what we call financial repression, basically stuffing debt into every insurance company, pension fund, bank. The central bank holds almost 100% of GDP in debt. We think we have a lot—I don’t actually know the number off the top of my head for the Fed, but I want to say around $7 trillion. Japan would have the equivalent of $30 trillion. So they’ve done this. It’s not the only reason they haven’t grown, not by any means, but it’s not good for growth. That’s one option.

I think for the United States, that’s tough. We’re just a very market-driven system. If our financial system had that kind of pressure put on it, it would be worse than when Japan did it. And a lot of people lend to us. We can’t do that to them. We can’t force French insurance companies to hold US debt. We can only force U.S. ones.

So I think the most likely thing will be inflation, which only lets off steam. Because inflation sort of pulls… well, it’s like a default. And I’m not talking about hyperinflation. I’m talking 10–20% inflation over a period. We just went through that. That actually knocked about 10% of GDP off our debt. We might need more next time. So it lets some steam off, but if you’re still spending too much and you haven’t fixed anything you’re back in the problem. That’s what’s going on now. We had some steam let off, but it wasn’t enough.

I think when it happens again, markets will be very unforgiving about it. They’ll look at us and say, “You are not to be trusted.” So it’ll raise the interest rate more, our debt will build up faster. I think at that point… There’s this saying about Americans attributed to Winston Churchill: we always do the right thing after we try everything else. I suspect we’ll try other things.

Dwarkesh Patel 00:39:58
Just for the audience, there are four ways we could get out of the debt: We could default, which you don’t think is likely.

Kenneth Rogoff 00:40:02
But really good for my book.

Dwarkesh Patel 00:40:05
Already you timed this one so well. I’ll be shorting the market when your next one comes out.

Financial repression. I guess you could actually cut the deficit. Or inflation. You’re saying if there’s another round of inflation, then after that…

Kenneth Rogoff 00:40:22
What everyone calls austerity. By the way, this word “austerity” that progressives use whenever you ignore debt building up and spend whatever you want, “austerity” is when you don’t do that. I mean, I think Ezra Klein’s book Abundance actually makes the point that there are costs and benefits to a lot of things. This “austerity” language pretends there are no costs to having your debt be higher and only benefits. So yes, that’s what everyone else has to do. We’ve gone longer than most without doing it.

Dwarkesh Patel 00:41:03
If it's going to be a financial crisis and financial crises are this bad…

Kenneth Rogoff 00:41:06
Inflation crisis. A financial crisis is the private sector, and the public sector bails out the private sector. So, the government, we’re not going to default. We're going to inflate, or do financial repression, or baby steps austerity or something. We're not going to have a crisis like Greece had. That’s just wrong. But inflation's not pleasant.

Dwarkesh Patel 00:41:32
Why wouldn’t we?

Kenneth Rogoff 00:41:33
Because we can print money. We can honor our debts. We just never have to default. Greece was using the euro and didn’t have control.

Dwarkesh Patel 00:41:43
Japan was using its own currency and it didn’t default.

Kenneth Rogoff 00:41:46
They had a financial crisis, not a debt crisis. They never defaulted on their government debt in that period. I’m not sure if they ever did. I’m sorry, they did in World War II. Of course, Japan defaulted on its government debt in World War II. That’s an interesting story.

But it was a financial crisis they had. Financial crisis is what’s making your banking system not work, lending to innovators not work, lending to dynamic companies not work. Ben Bernanke wrote at the time a thought piece about this. He didn’t really have numbers. He conjectured that was why the Great Depression was so bad.

When Milton Friedman, one of the great economists of all time, looked at the Great Depression, he said: “You didn’t print enough money. You tightened the money supply too much." And Ben came along 25 years later. He was a classmate of mine in graduate school. I had the office next to him at Princeton. He came along and wrote this amazing paper. Again, it was just a thought piece, which is not a typical economics paper. He said, “If it was just that you didn’t print enough money, eventually wages and prices would adjust. Yeah, maybe it wouldn’t happen in a year, maybe it wouldn’t happen in two years, but the Depression took 10 years. How can that be?”

He made this conjecture. There’s been a lot of subsequent work showing it. Again, there’s a lot of debate about this, let me be careful. But I certainly view the weight of evidence as saying financial crises are really bad. This has led us to the situation in the United States where we’ve gotten a little happy-go-lucky about bailing everyone out.

I would describe Treasury and Federal Reserve policy today as, “When in doubt, bail it out.” Because they saw what happened. But as your financial sector grows, that will lead to a problem someday. It did, of course, in the Silicon Valley Bank case. It continues to have echoes of that. But I think an inflation crisis is more likely than a financial crisis, although these things are very hard to predict.

Dwarkesh Patel 00:44:01
You say in the book that we didn’t outgrow our World War II debt. What happened instead was that financial repression after the war, and then the inflation of the 1970s made our debt-to-GDP ratio… It should have been 70-something, but it ended up around 20-something instead.

Of course, we just had inflation recently. Do you think there’s some irrationality in the market for US government debt already, given that we can forecast what’s going to happen here? They can read your book and see that inflation’s going to go up. They can look through history at what’s happened. Do you think there’s some irrationality in terms of what people are doing?

Kenneth Rogoff 00:44:38
I think, number one, is that they have too much faith in the independence of the Federal Reserve. The Fed’s been this amazing institution that’s evolved. It’s been the guardian of low inflation. We can argue about if it’s the right inflation or not. The Federal Reserve insists it’s very independent. The Supreme Court recently ruled that Trump couldn’t fire Powell, the head of the Fed. But I think they’re dreaming.

There are so many ways Congress and the President could override the Fed, especially if they declare some kind of wartime or “war-on-pandemic” situation.

Dwarkesh Patel 00:45:21
Though I wonder from the politician’s perspective, maybe the independence of the Fed gives them a convenient way to pass the buck that they’re actually happy about. They can say, “Ah, I’d love to do this irresponsible thing, but I can’t because of the Fed. It’s out of my hands.”

Kenneth Rogoff 00:45:38
That’s for sure. That’s why Trump bashes the Fed. It’s not the only reason he does it. I think he actually disagrees with them. But he feels that bashing the Fed, if there’s a recession which there might be, gives him someone to blame for not lowering interest rates.

It depends if we run into a world where interest rates start creeping up… Right now, the 10-year rate is around 4.5%. That’s the nominal rate. The inflation-indexed one is a little over 2%. The 30-year rate is around 5%. I think those are going to drift up. And that makes mortgage rates go up, student loans go up, car loans go up, business loans go up. It’s painful.

The question is, at what point does that pain become real? As I mentioned, I think this would be catalyzed if we’re hit by a shock. In that kind of situation, it becomes easier to temporarily take back some independence from the Fed. I think it’s easier to do than people think. Given that I think shocks are going to happen—maybe AGI brings a shock we don’t yet imagine—people trust in Fed independence too much.

Now, I love Fed independence. I actually wrote the first paper on why you should have an independent central bank back when there were virtually no independent central banks. I was a pawn at the Federal Reserve. I’m talking my own book when I say it’s a great idea. I don’t mean my “book” book, I mean my human capital.

I like to say that the Federal Reserve fights for its independence every day. I hear senators say, “They’re idiots.” I hear people in Silicon Valley say, “They’re idiots. We should bring them under the Treasury.” I used to hear that just from progressives. But now I’ve heard that recently from some tech titans, saying things like, “Scott Bessent, the Treasury Secretary, he’s smarter than Powell. Why don’t we let him run things?” They could.

Dwarkesh Patel 00:47:50
It does seem like the Fed works really well as it exists now. It’s independent. Sure, there are people who criticize its actions as you say. But on the whole, it seems like a reliable institution that makes smart calls. They can be wrong, of course, but it seems so much more competent than much of the rest of government.

If you wanted to replicate how the Fed works—if you wanted other parts of government to work that way—is there something we could do? Or is it more of a human capital problem than an independence problem? Like, bankers and economists are really smart. I don’t know if you could replicate that in the Department of Education or the Department of Agriculture.

Kenneth Rogoff 00:48:29
One of the things the Fed has is this simple barometer that everybody sees. They don’t really see it, but they have a feeling. They see gasoline prices, that's probably how they decide what inflation is. But they have this simple barometer.

Mind you, particularly in recent years, progressives have wanted the Fed to solve inequality, social justice, and the environment. But they have one barometer that they kind of control over the long run, not in the short run but over the long run. So that makes it a little easier to say, “You wanted us to have low inflation.”

Whereas so many other things the government does might be making everybody better off, but they’re making some people more better off. Maybe some people aren’t better off at all. It suddenly becomes very political. Nobody elected the Fed, so it’s harder to make those decisions. I’m obviously a technocrat, or I side with technocrats. My students are technocrats. I think way more things should be like that.

Dwarkesh Patel 00:49:33
But if you wanted to do that, suppose you get called by the Pentagon tomorrow and they say, “We want to run the Pentagon like the Fed.” What do you tell them to do?

Kenneth Rogoff 00:49:41
I was going to say, the Pentagon? I’m not speaking about the current Pentagon, but just up till now. I haven’t looked closely but it’s been run pretty darn well. The military’s pretty efficient.

There are people who tell me, “Okay, Elon Musk can take a payload into space at 15% of—or maybe one-fifteenth of—what NASA does. Why don’t we let Elon Musk run the Pentagon?” There may be something to that. I think to some extent—and maybe I’m defending them too much—but you never know where the next blow is going to come from. It always looks like a lot of your stuff is wasted, you built up. But your enemy is looking at what you have. Where are you weak? Where are you strong? So I wouldn’t have picked the Pentagon as the obvious thing.

But let’s say crypto regulation, that would be a good example. Why don’t we have something more independent there? Instead, as you well know, it’s been overrun by politics.

In fact, there’s this huge thing going on right now. I don’t know how it’s going to play out. The Fed has been protected, though not as much as you’d think. But Trump got to the Supreme Court and was told he can fire the head of any agency. I assume, by inference, he can also fire anybody at any agency. I think that’s a terrible mistake. We need to have independent agencies. You have an evaluation process. They answer to Congress. If they go off in the wrong direction, you try to fix it. But to just have everything switch every four years? That’s really very worrisome.

Dwarkesh Patel 00:51:28
Before Trump—maybe for intrinsic reasons, maybe because of norms—it was really hard to fire people anyway. That didn’t produce remarkable competence across the government.

Let me try to consolidate some of the things you mentioned. Maybe it’s really important to structure more of the government like that. If you're running a department, you have just one target like the Fed’s 2% inflation target. That’s all you have to do. Don’t worry about anything else. I do think it’s impressive that the Fed has avoided mission creep. It seems like every institution in the world falls into mission creep—companies, government departments…

Kenneth Rogoff 00:52:02
Oh, they haven’t avoided it. They’ve been under incredible pressures. Obviously, things have changed. But I talk about this a bit in my book. You go through the working papers and research coming out of the various Federal Reserves and it's all about inequality, the environment, social justice. You’d be strained to find a paper about monetary policy during that period, because they were under pressure. Part of being independent is bending with the wind.

But they’ve managed to keep their core competency, their core function of setting monetary policy independent. No, it’s been amazing. But it is a constant fight. You can go to a country like Turkey. I don’t know what the inflation rate is today, but it hovered up toward 100%. And Erdoğan—the president of Turkey—would fire the head of the central bank every year. Every time they tried to raise interest rates, he’d fire them. You can find other countries like that.

So we’ve been lucky. But you can’t count on that continuing.

Dwarkesh Patel 00:53:10
Apart from the political pressure problems from the outside, you mentioned watching your younger colleagues or younger economists writing working papers at the Fed about these other issues like inequality or climate change.

From the inside, given what the younger people in this profession care about, do you expect the competence or the focus to just decline by default, given the new generation?

Kenneth Rogoff 00:53:42
No, this was a wake-up call. There was a blog that Hoover did. They looked at the most-used words in our big annual meetings. There’s this thing called the American Economic Association meetings. Everybody goes. They took all the abstracts and titles from the last 15 years and the word inflation had not appeared until this year.

Dwarkesh Patel 00:54:08
But why are you optimistic about what happens when they get in charge?

Kenneth Rogoff 00:54:10
Because there’s an intellectual market. This was a huge miss and there’s a market for figuring it out. One of the good things about the American university system—at least in the sciences, and I’ll speak for economics—is that things drift off but if something is way wrong—and they were certainly way wrong about inflation, I believe, and way wrong about interest rates and debt—then there’s some rebalancing.

We have a very competitive system of publishing. We have a seminar system that’s just ruthless. There’s a debate around it. It’s not settled, and maybe I’m wrong and they’re right but it’s definitely being debated now. Whereas 10 years ago, I think I was like a lone voice in the wilderness saying these things might happen again. I would teach inflation to my students. They’d sit there patiently. It was like I was teaching them the music of Fred Astaire or something. They’d go, “Okay, it’s the 21st-century, the Internet, that can never happen.”

Or I’d teach debt. If I had foreign students, they were all having problems. But the American students were like, we can just do whatever we want. But it’s changed.

Dwarkesh Patel 00:55:35
Going back to the potential future problems, if we do go the financial repression route and not the inflation route, how bad will that be?

As you were saying, after World War II we had financial repression. But that was when we had the highest growth ever. On the other hand, if you look at China and Japan, it seems like a lot of their problems might be caused by the misallocation of capital that financial repression created. Do you have some intuition about how much we screw ourselves over with that route, as opposed to inflation?

Kenneth Rogoff 00:56:02
We’ll start with World War II but it’s never just one thing, obviously. There were a lot of things. So with World War II first of all, financial repression was easy. The financial markets had been destroyed by the Great Depression. World War II became something of a command-and-control economy.

There are a lot of interesting papers about World War II that show that Americans just worked really enthusiastically. There was real patriotism in the production. I’m not saying we’re not now, but back then they were able to fill factory jobs that we probably couldn’t even fill today. As we emerged from World War II we had all the soldiers come home. That’s a huge growth lift. We didn’t manage it perfectly. We actually had quite a bit of inflation during that period.

The financial markets, as you grew up in and as young people know them today, didn’t exist back then. The world has changed a lot.

Dwarkesh Patel 00:57:05
Does that mean US growth would have been even higher after World War II if we had just kept the government debt or figured out some other way to deal with it and let financial markets develop earlier?

Kenneth Rogoff 00:57:17
Maybe. We didn’t have any financial crises for a long time because the markets were very repressed. Oftentimes, when you get a financial crisis it’s exactly when somebody comes along and says, “I know how we can make things grow way faster. Let’s just take away all the rules and regulations overnight.”

That happened in one country after another. It works, until you blow up. So you’d have to say that, by and large, it was managed rather well. We grew. The rest of the world grew. It took time for private markets to develop. One thing I should’ve emphasized was that our debt was very high after World War II, about what it is now.

But there was nothing else. There wasn’t all this private debt. That had all been defaulted on. I’m being slightly hyperbolic here but maybe it was 50% of GDP altogether. Everything else, state and local debt, had been defaulted on. Now, it’s bigger than the federal debt by a wide margin. So it was a very different world, putting in financial repression back then, compared to now when that’s a big part of our business models in the financial sector.

Dwarkesh Patel 00:58:25
Just to make sure we’ve completed the concrete scenarios, so basically your prediction is that there’ll be some crisis, some surge of inflation, then there’ll be austerity. Then what happens? Is growth really slow afterward because the government can’t spend as much? What do the next few decades look like in your world?

Kenneth Rogoff 00:58:50
I think it will be quite a wake-up call for Americans, having to adjust under difficult circumstances. Most likely, we get hit by a shock. We want to borrow a lot. The bonds are rising faster than they did the other times we did. And we're not able to do as much.

It’s not the end of the world. During the European debt crisis from 2010 to 2012, most European countries raised their retirement ages. They didn’t do it right away. They did it 10 or 15 years out. There’s stuff you can do.

So I want to be careful here and say it’s not like the end of the world, but it’ll be pretty unpleasant. This will affect the entire world, since the global system is very dollar-centric. It won’t be good for our franchise, the dollar being so used everywhere. As other countries start using the dollar less, our interest rates will climb even higher.

I’m an academic. I’m not trying to push my ideas by being maximally hysterical. But hysterical is definitely within the realm of possibility here. What I’m saying is more likely than not, not that it’s not definitely going to happen. We could have growth. We could have a whole lot of high-skill immigration. We could make changes. There are a lot of things that could go well.

Dwarkesh Patel 01:00:19
On the growth thing, Europe’s growth has been pretty bad after 2010. Japan obviously has had pretty bad growth after their crisis. Why will we be in a different position if we do have this kind of crisis? Why will growth continue apace?

Kenneth Rogoff 01:00:34
No, it’s going to cause a pause in growth. The main reason debt crises happen is we don’t have an automated system of working it out. When the stock market crashes it’s painful, you’re looking around for who got hurt. But when you have debt crashes, we take five years, ten years, to figure out who owes what. It’s that process of allocating the losses that causes problems.

That, by the way, is why so many people thought, “China’s fine. The president will just tell everyone what it is.” That turns out to be not as true as they thought.

Dwarkesh Patel 01:02:20
Is it possible to believe both that AGI is near and that America’s fiscal position is untenable?

Kenneth Rogoff 01:02:26
What do you mean by saying AGI is coming.

Dwarkesh Patel 01:02:28
Any job that can be done purely through computers is automated. So white-collar work, the work we do even, is automated within 20 years.

Kenneth Rogoff 01:02:36
Anytime you get a big productivity boost, it’s fantastic. If it comes quickly, yes that can solve problems. I will say that historically, there have been lots of times when countries had good growth— even higher than their interest rates—and they still got into trouble because fiscal policy isn’t mechanical. It’s political. It’s about how much you spend, who wants what. It’s not an arithmetic question.

Let me say it another way. Nobody ever defaulted or had high inflation because of arithmetic, because they couldn’t pay, or couldn’t have called in someone who knew what to do. They do it because of political pressures.

I think if AGI came that fast and that big, it would make the populism phenomenon we’re facing now seem like nothing.

Dwarkesh Patel 01:03:33
If AI is going to be massively deflationary—if it makes all these goods so much cheaper—should we be printing a bunch of money to still stick to 2% inflation? Or does that not matter anymore?

Kenneth Rogoff 01:03:46
Well, we certainly can run monetary policy the same way. You don’t automatically get deflation just because some goods are going down. You can do things to increase demand so that there are upward pressures on final prices. Even if the AI workers aren’t demanding anything, you can put in a lot of demand so firms charge a lot, not just for the services that AI is replacing but also for the raw minerals and materials and everything else that goes in.

Fundamentally, when you have productivity, it makes it easier for the monetary authorities—the Federal Reserve—to deliver low inflation and good growth. That’s what they’re trying to do. It makes their job easier. And I think it takes some of the pressure off them to inflate, because things are going pretty well. So the pressures aren’t the same.

Dwarkesh Patel 01:04:39
But should they be trying to fight the deflation at all, in that world?

Because traditionally we need inflation to root out the rentiers and to fight downward wage rigidity. But now the AIs have all the jobs, so we don’t need to worry about that. There are a bunch of biases humans have that we need inflation to correct for. Do we even need that in a world with AI?

Kenneth Rogoff 01:04:56
Okay, that is a very good point. Frankly, Keynes founded modern macroeconomics. He was an incredible Renaissance person, having both sides of the brain. One of his insights that just transformed things was this. Before Keynes, we used what we now call general equilibrium models: demand and supply, prices moving to keep everything in line. But Keynes was looking at the Great Depression and said, “Prices should be coming down. But they’re not. Why aren't they?” That’s really a cornerstone. At the end of the day, it’s mostly human behavior. It’s mostly workers.

So if you have these docile AI workers—they’re not workers, they’re just firms—and if you have firms that are willing to let prices fall, then certainly you can do that. I mean, we’re still going to have some human workers? I don't know.

But here’s a question on what monetary policy should be. Do you think interest rates are going to go up or down? When we had deflation last time—from demand-deflation, like after the financial crisis and the pandemic—interest rates went down. My intuition here is that interest rates would need to go up. I mean real interest rates, real inflation-adjusted interest rates. In that case, deflation isn’t such a problem. You just don’t let the interest rates go up as much.

Last time, interest rates went to basically zero. That’s a whole other line of discussion. They felt they couldn’t lower them into significantly negative territory. So they were sort of paralyzed. There was this deflation, or at least too-low inflation. Monetary authorities thought they knew how to create inflation, but that’s always been by cutting interest rates lower. When they hit a bottom, they don’t.

I have a whole book about negative interest rates and that’s a whole other thing. If we’re imagining real interest rates going up, then it’s not much of a technical problem. You just let the interest rates rise a little less so you’re not getting deflation.

Dwarkesh Patel 01:07:11
Do you expect interest rates to go up? Because one factor is that you want to invest in the future, the future has so much more potential. Another is that maybe you want to consume more now, because you know you’re going to be wealthier in the future anyway. You might as well start spending as much as you can now.

Kenneth Rogoff 01:07:26
I think AGI and AI are upward pressures on interest rates for lots of crude reasons. You have huge energy needs. Traditionally, when you did a lot of investment it raised wages, but it’s possible— there are economists like Daron Acemoglu who’ve shown it can go both ways, it’s not difficult to show that—if you're really just substituting for workers, it’s making capital more valuable. You just invest even more.

The pressures on monetary policy will depend a little bit on that. In principle, it makes life easier. If it did push the interest rate down to zero, there are interesting questions around that, but maybe your audience might not be as fascinated by them as I am.

Dwarkesh Patel 01:08:27
Let’s talk about it a little bit. If we expect interest rates to go up because of AI, what should the government be doing right now to be ready for that? Should they be locking in hundred-year bonds at the current interest rates since they’re only going up from here?

Kenneth Rogoff 01:08:43
I’m going to get to that. But first, just where we are… I follow interest rates today all the time. Maybe a lot of people who listen to you don’t. But let’s talk about inflation-adjusted interest rates. There’s a 10-year bond that’s indexed to the inflation rate, issued by our Treasury. Inflation-indexed debt is only about 10% of our total debt. There are tax considerations—it’s not perfect—but it’s a pretty good measure of what we call the real interest rate.

It had gone to minus one at one point after the pandemic. It averaged zero for about 10 years, from 2012 to 2021. And it’s higher now. That is, for a macroeconomist, the biggest question in the world. Because it affects asset prices, it affects risk, it affects volatility.

I regard it as just a normalization. I think it was likely to happen. If you go around and talk to some of my younger colleagues, or folks at other places, there’s quite a debate about that. A lot of people think, “We’re getting old, we’re not inventing anything...” I know you’ve just been arguing against that and good for you.

I tend to think interest rates are more likely to go up than down, going forward. I’m talking about these long-term interest rates—the Federal Reserve just sets the overnight interest rate—these long-term interest rates are set by markets, and I think they’re more likely to go up.

But I think AGI is only a piece of it. Debt is rising everywhere. There’s the remilitarization of much of the world. Needing to deal with climate change. Eventually if we’re not dealing with it, then we’re dealing with climate disasters. There’s growing populism, geopolitical fracturing, many things. So I tend to think interest rates are going to go up but not just for the good reason that we’ve gotten more creative and that everything’s going to be better.

Dwarkesh Patel 01:10:55
You've said in the book that you expect a rebalancing from US equities to foreign equities. US equities have outpaced foreign stocks for the last couple of decades. You say you expect this to change or that there will be some rebalancing. What causes that?

Kenneth Rogoff 01:11:11
What I say very concretely is that when the dollar is really high, you should expect the euro to go up. I feel strongly about that. My first important paper was about exchange rates. That’s why the book’s about exchange rates. When Japan’s really weak or when the dollar’s really strong—it’s very hard to predict exchange rates—but I think the euro will do well. There’s a lot of room to catch-up in Europe.

I actually think I’m nuanced in what I say in the book. Trump hadn’t been elected yet, but I say Europe seems to be under pressure to re-militarize. I was aware that Harris was probably going to cut the US defense budget, so that would put pressure on Europe. Re-militarizing would actually be good for the euro. It would be good for technology in Europe. It would give them more geopolitical power in the system.

Now, just so your listeners can calibrate this, my first book was a very mathematical one, Foundations of International Macroeconomics. In theory, you should diversify. You shouldn’t put all your money in the United States. I did a video with Zbigniew Brzezinski, Mika Brzezinski’s father. For those who don’t know, he was Carter’s Kissinger. I did a video with him that Merrill Lynch produced.

It was about why international diversification could be good. What they got me to say was very, very limited. I feel quite fine about what I said. I wasn’t doing any consulting at the time, just academic work. I didn’t do speeches, I didn’t do consulting. I talked to central banks a bit, but I didn’t do anything for money. But I did get paid for that. It circulated half a million copies of it. A lot of my friends teased me and said I would’ve made a lot more money if I hadn’t followed my own advice. I can think of plenty of other examples like that.

But yes, my instinct is that the US premium—this idea that it should just keep getting bigger and bigger—these things have some regression to the mean. Maybe not with AI all being in the US, I don’t know.

Dwarkesh Patel 01:13:44
Is it that you’re predicting that the S&P keeps growing at 8%, but foreign equities do even better?

Kenneth Rogoff 01:13:50
I’m just going to safely say foreign equities do better than dollar equities.

Dwarkesh Patel 01:13:55
But not because the growth in US equities slows down, it’s just that foreign equities do even better?

Kenneth Rogoff 01:14:00
Look, you have a lot of friends who spend all their time doing this. I wouldn’t pretend to. I hold a very neutral portfolio because I talk to policymakers and world leaders even on occasion. I don’t want to be someone who’s talking about regulating Bitcoin and owning a lot of Bitcoin, to pick a random example.

So I wouldn’t regard myself as great at this. But yes, I think there’s a case for international diversification, particularly into Europe at this point because they have so much potential catch-up. Just as in California, where you’re from, there’s a little bit of dim awareness that it might be overregulated and you might want to do things differently, I feel that’s happening in Europe.

Dwarkesh Patel 01:14:52
If you look internationally, if you'd been betting on catch-up, I wonder how you’d backtest it. There’s some intuition there that if you’re poor and you’re further from the frontier, it makes sense that it would be easier for you to catch up. But there’s another intuition that if you’ve been persistently behind the frontier, there must be some deep endogenous reason.

Kenneth Rogoff 01:15:11
You’re absolutely right. For example, Asia has a lot more governance problems on the whole. There’s a reason that their price-earnings ratios are lower, because you don’t trust the governance. You’re right.

That’s fair and a lot of people are just betting on that. But I don’t think Europe is so hopeless that it can’t pull it together. I can make a comparison. I'm a basketball fan. The Boston Celtics just got crushed by the Knicks, just before we’re taping this. Part of it is because our star, Jayson Tatum, was injured. You may not have gotten any better, Europe in this case, but if somebody’s hobbling the United States—I do think that’s going on to some extent now—you do better.

Dwarkesh Patel 01:16:07
Is there some institutional reform we could make that would get us out of this political equilibrium we're stuck in. Both parties, when they're in power, are incentivized to increase the debt and there's no institutional check on that proclivity.

Kenneth Rogoff 01:16:26
There have been a lot of people who’ve tried this, for example by having what are called fiscal councils. I did a paper once about fiscal councils with Julia Pollak, who’s a brilliant economist, when she was an undergraduate. That was quite a while ago. A number of countries experimented with them, but it hasn't worked.

The country that's done the most with this is probably the United Kingdom. George Osborne, when he was Chancellor, set up this fiscal authority. The big thing they did is they made predictions so the government doesn’t get to make up different predictions. You don’t necessarily have to go by their numbers, but they got to say whatever they thought.

Our CBO does not get to do that quite the same. Our Congressional Budget Office is very good, but they are constrained to believe what Congress tells them. If Congress says, “We’re putting in this tax hook, but it’s going to go away after ten years,” or “We’re doing this policy, it’s all going to change,” then they’re forced to use those parameters. The UK version is more independent. There are lots of complaints about it, but that’s a very poor man’s fiscal authority, just somebody who says, “This is what your deficit looks like.” It’s the same thing as our CBO, but with more independence. It helps. But I think it has to go to our electoral system, right? Our campaign financing. Do we have term limits?

Dwarkesh Patel 01:18:05
You think that would help? I think, if anything, if you’re longer in office, you might have a more long-term incentive. To the extent that a lot of the deficit problems are caused by populism, I don’t know how much campaign finance would help.

Kenneth Rogoff 01:18:19
Maybe you’re right. I don’t have a magical solution to this. It’s all over the world. Nobody's finding a particularly great solution to it. The only encouraging thing is that these things go in waves. So maybe this one will end. But we’re certainly in a really difficult situation right now.

Somebody asked me, “If you were advising a Republican president or something, what would you do? What problem would you face?” The biggest problem is that in a few years there’s going to be a Democratic president. They’re going to do exactly the opposite of what you wanted to do. It’s the same thing for the Democratic presidents. How do you have some continuity? How do you have policies put in place that the public can rely on?

We’ve done well in the United States in some ways because our government’s been kind of weak and hasn’t been doing stuff. The private sector can work around it. I’m not saying it’s perfect. There are lots of things we should do. But look, this is out of my paycheck, so to speak.

Dwarkesh Patel 01:19:27
You're the former Chief Economist of the IMF!

Kenneth Rogoff 01:19:31
All right, but that’s economics. These are very political questions. Brexit’s an example of democracy gone amok. What a dumb idea. I don’t know if Brexit was right or wrong. I feel like we’ll know in 50 years. But you shouldn’t be able to do it with a simple majority vote. You should need a two-thirds vote, or something like that.

There’s a whole government department here with people specializing in what we should do. Actually, I think there are experiments. Washington State experimented with different voting choices. Maine did. There are these ideas out there, but we’re a long way from converging on anything.

Dwarkesh Patel 01:20:17
If you think people are underweighting how big the debt issue is, are you especially long on countries which have a low debt-to-GDP ratio, like Australia or Norway?

Kenneth Rogoff 01:20:28
It's not the only thing going on in the world, your debt is just one thing. Countries like Australia and Canada for example are what we call commodity exporters. They know that sometimes the sun shines and sometimes it's a dark winter. They don’t quite sell oil but they sell some coal, natural gas, and some oil. They understand things move around and that they need to save for a rainy day.

Norway is in a whole other league. But yes, there are lots of factors to whether a country will do well. The lower-debt countries have less of a problem. But Canada and Australia face very volatile income streams because of commodities, so they tend to be more nervous about debt.

By the way, they also have a lot of housing debt, in Canada for example that's been a big problem. But I’m bullish on the United States. Don’t misunderstand me. I’m not saying everyone should leave the United States and go to Canada, although my wife thinks that sometimes but for other reasons.

When I was playing chess in the late 1960s, I was living by myself in Europe. Nixon got elected. I felt about Nixon the way I think a lot of people in your generation or at least millennials feel about Trump. I didn’t want to come back to the United States. So there are a lot of people who talk that way. But I think the United States is great.

Countries that are smaller, that aren’t the reserve currency, that don’t have access to these deep pockets of domestic and foreign borrowing—and all of those countries fit into that framework, they need to be more careful.

Dwarkesh Patel 01:22:24
Do you think this "exorbitant privilege," as you talk about it, could actually be bad for us in some ways? Maybe it incentivizes us to take on more debt than is wise. That it allows—or even incentivizes—us to take on more debt than is wise, especially if this isn’t a permanent advantage?

We’re at the top, so we can take out cheap debt. But over time—if we lose reserve currency status, or even if it just weakens—we’ll have to refinance that debt at higher interest rates. So in the short term, we’re incentivized to behave in ways that aren’t sustainable in the long run. Is there a political economy explanation for why that might be bad for us?

Kenneth Rogoff 01:23:08
I've heard that argument, but I basically think it's great for us. It's not just the government. It’s all of us who borrow less. Do we wish we were paying higher interest rates? Probably most people who are getting a mortgage right now feel like our interest rates are plenty high. They don't need to see them higher.

I think with the exorbitant privilege, there are some drawbacks we don't need to get into. But it's basically incredibly fantastic if you owe $37 trillion as our government does, to be paying half a percent to a percent less. We're talking about hundreds of billions of dollars.

You also have our ability to see what's going on everywhere. A lot of what our spying does is using our exorbitant privilege and the dollar network. Look at our sanctions. I mentioned, I was in my teens at the end of the 1960s when I didn't want to come back. One reason I didn’t want to come back was the Vietnam War was pretty terrifying. I had many friends get drafted. Their brains got fried by heroin even if they didn’t get killed.

I'm not saying that we've solved all our wars with sanctions. But make no mistake, we have used that in place of military intervention a few times. So that’s been great. I think losing that, and not appreciating how important that is, is a terrible blunder that we might be making right now.

Dwarkesh Patel 01:24:37
This is a very naive question. I know you address it at length in your book. I’ll ask the question in the most straightforward way and then you can explain what's going on. How should I think about the fact that we are basically giving the rest of the world pieces of paper and we're getting real goods and services in exchange?

Sure, at a high level you can say that they're getting this liquidity or they're getting this network and that's what makes it worth it. But I don’t know, are we fundamentally getting away with something?

Kenneth Rogoff 01:25:06
So just to note, the United Kingdom is not the reserve currency. They're not the dominant currency. They used to be, a hundred years ago. They look a lot like us now, with big current account deficits. That’s actually why Trump was able to strike a deal with them. Because they weren’t really running a surplus against us, anyway. They’re over-financialized, even more than we are.

The core of the benefit we get is that we borrow by issuing safe assets—if you want to call our debt safe—and we invest in risky stuff. Charles Kindleberger, wrote one of the great books on crises. I had him as a professor at MIT. He called us bankers to the world. He said: “Yep, we’re running this deficit, they’re holding a lot of our Treasury bills, but we are making money hand over fist.” It’s the same thing as the equity premium where you hold stocks and it’s not always, but on average better than holding bonds. So that’s been very good.

You have the fact that the dollar is very liquid, the markets are very liquid. Say you're a Silicon Valley firm and you're big enough to issue debt internationally. I don’t know if any Silicon Valley firms ever issue debt but say you do, people will buy it because it’s in dollars. If you're the same firm in France, forget it. They don’t want to hold euros. Even if you promise to pay in dollars, they’re not happy about it because your income isn't in dollars. So it’s been fantastic. This is something that's been debated.

Stephen Miran was a Harvard student, PhD. He’s a very smart guy, head of Trump’s Council of Economic Advisers. He has made a clever argument that because everybody loves our currency, it makes us less competitive in everything else. He argues that it partly hollowed out our manufacturing and that’s terrible.

There’s a little bit of truth to that. First of all, the dollar goes like this, it’s not always high. Second, I mentioned the United Kingdom is kind of in the same boat. We’re good at a lot of things. We’re good at tech. Tech makes the dollar stronger, make no mistake. We’re good at biotech, agriculture. We’re good at a lot of other things that make the dollar high. And if you’re good at these things, it’s harder to be good at manufacturing. It bids up the cost of everything.

On the whole, we’re performing this banking function. That’s really the big thing. It’s been going on since the ’50s and ’60s. That’s the core of our so-called exorbitant privilege.

Dwarkesh Patel 01:28:07
There’s a really interesting book by Charles Mann. I think it's called 1493. It’s about how during the Ming Dynasty in 17th-century China, they kept issuing different paper currencies and it was super unstable. People in China wanted a reliable medium of exchange. So tens of thousands of tons of silver from the New World, from the Spanish, would be exchanged for enormous amounts of real goods exported from China.

So from the Spanish perspective, they’re getting shiploads and shiploads of real goods, and all they’re giving up is this medium of exchange. I don’t know how analogous that is to the current situation.

Kenneth Rogoff 01:28:47
There are countries like Ecuador and others that dollarize. They literally use the dollar and they need dollars. We’re able to have them hold dollars. It’s not silver, but we print it and they pay very low interest rates. They’re holding Treasury bills, not physical dollars. Yeah, it’s fantastic for us. We definitely pay less on our debt because of that.

That’s a fascinating example you bring up. The Chinese actually invented the printing press. They invented paper currency way before the Europeans. But then, what do you know, they kept printing a lot of it and had a lot of inflation. I haven’t read that book, but it’s a great example. I knew they were using silver, but that number is bigger than I had heard.

Dwarkesh Patel 01:29:38
Final question. A big part of your book discusses the different countries which seemed at different times to be real competitors to America. You talk about the Soviet Union, Japan, China today. We've discussed why they didn’t pan out.

We can go into the details on any one of those examples, but in the big picture is there some explanation that generalizes across all these examples of why America has been so competitive? Why has it been so hard to displace?

Kenneth Rogoff 01:30:09
It’s not just that we’ve stayed on top, we’ve just gone like this. Remember, in the 1970s, Europe actually peeled away from the dollar bloc. But the rest of the world started globalizing. China globalized. And eventually the Soviet Union, and the dollar just colonized all these places. They were all holding dollar debt, using dollars. It’s much bigger than even the British pound was when the sun never set on the British Empire.

So it’s been amazing and surprising to people like myself. If you read what everyone was saying at the time, it was just that it kept going up. That our share of everything kept getting bigger and bigger. Definitely, to some extent, we’ve been lucky. We talked about Japan. I think China made a big mistake by sticking to the dollar so long. Europe should have delayed bringing Greece into the Euro, because their crisis wouldn’t have been so bad. So we’ve been fortunate with blunders by our opposition.

We’ve done some good things. But I think the thing Americans forget is that we have been lucky a lot of times. I worry our luck is wearing thin. I quote a chess player—the great Bent Larsen, who was number two to Bobby Fischer when I was playing. He was asked, “Would you rather be lucky or good in a chess game?” And he said, “Both.” So I think Americans forget. They know we’re good and we are good. We’ve talked about dynamism, this secret sauce that we’ve had so far. But I think we’ve also been lucky. If you ran it all again, it didn’t have to go the same way.

Dwarkesh Patel 01:32:01
It’s a very scary kind of luck. If it’s so easy for these other countries to make some mistake that causes them to totally fall behind, it should update you in favor of the idea that, in general, it’s easy for a country to get itself in a rut. It’s like the Fermi estimate thing. The fact that you don’t see other alien civilizations is actually very scary, because it suggests that there’s some kind of filter which makes it really hard to keep your civilization going.

Kenneth Rogoff 01:32:29
I hope not, but we’ll see. It’s certainly been amazing how the dollar’s done and how the US has done. I hope we continue, but we are doing a lot of things right now… I don’t think Trump is the cause of the dollar being in gentle decline. That’s just wrong. I think it would’ve happened with Harris winning. But he is the president at the moment, and things like Liberation Day… I’ve talked to tech people who think it’s just brilliant. I understand that. We can debate it. I’m happy to debate it with them.

We look at the rule of law. Okay, I’m sitting at Harvard University. Naturally it feels that way. But also we talked about the president being able to remove all the independent agencies. It used to be that if you were a foreign investor and you invested in the United States, you thought you’d get your money back. Maybe the stock would’ve gone down. Maybe the real estate you bought would’ve gone down. But you’d get paid. I think we were more exceptional than most about that. That’s in doubt now. There’s no question.

The book’s about a lot of things besides exorbitant privilege, the whole arc of the US. But when I was telling people about the book, “I don’t know, I’m looking at the numbers, I’m looking at what China’s doing, I’m reading about Europe and their central bank digital currency. I think we’re going downhill.” I showed it to academics, I showed it to financial people, I showed it to tech people. They said, “You’re nuts.” They didn’t want to think about it. I don’t know if I’m right. But I think it’s worth thinking about.

Dwarkesh Patel 01:34:16
When I was in China, I met up with some venture capitalists there and they were quite depressed in general. Even founders say it’s hard to raise money. I was asking them why, and they said investors don’t want to invest because even if you invest in the next Alibaba, who’s to say the government doesn’t cancel the IPO?

Kenneth Rogoff 01:34:36
They’re in trouble. Yeah. I think Europe has a bright future in this context, of being the team that doesn’t have as many injured players. But yeah, China… it’s not going to be forever, but I think for five or ten years they’re going to stay in trouble.

Dwarkesh Patel 01:34:53
Okay. Thank you so much for sitting down with me and also answering all my questions. I’m sure there are many misconceptions and naive questions and so forth. I appreciate your patience and you educating me on this topic.

Kenneth Rogoff 01:35:03
No, it’s an honor to be on your famous podcast. I heard from so many young people when I told them I was talking to you. They were like, “You're with Dwarkesh? Just fly back from here! Do whatever you need to do!”

So I’m glad you were able to come here. It’s really been interesting, and I’m glad to learn more about everything you’re doing.

Dwarkesh Patel 01:35:28
The honor’s mine. It was great to be able to travel here and speak with you.

“Things take longer to happen than you think they will, and then they happen faster than you thought they could.” - Rudiger Dornbusch

I’ve had a lot of discussions on my podcast where we haggle out timelines to AGI. Some guests think it’s 20 years away - others 2 years. Here’s where my thoughts stand as of June 2025.

Sometimes people say that even if all AI progress totally stopped, the systems of today would still be far more economically transformative than the internet. I disagree. I think the LLMs of today are magical. But the reason that the Fortune 500 aren’t using them to transform their workflows isn’t because the management is too stodgy. Rather, I think it’s genuinely hard to get normal humanlike labor out of LLMs. And this has to do with some fundamental capabilities these models lack.

I like to think I’m “AI forward” here at the Dwarkesh Podcast. I’ve probably spent over a hundred hours trying to build little LLM tools for my post production setup. And the experience of trying to get them to be useful has extended my timelines. I’ll try to get the LLMs to rewrite autogenerated transcripts for readability the way a human would. Or I’ll try to get them to identify clips from the transcript to tweet out. Sometimes I’ll try to get them to co-write an essay with me, passage by passage. These are simple, self contained, short horizon, language in-language out tasks - the kinds of assignments that should be dead center in the LLMs’ repertoire. And they're 5/10 at them. Don’t get me wrong, that’s impressive.

But the fundamental problem is that LLMs don’t get better over time the way a human would. The lack of continual learning is a huge huge problem. The LLM baseline at many tasks might be higher than an average human's. But there’s no way to give a model high level feedback. You’re stuck with the abilities you get out of the box. You can keep messing around with the system prompt. In practice this just doesn’t produce anything even close to the kind of learning and improvement that human employees experience.

The reason humans are so useful is not mainly their raw intelligence. It’s their ability to build up context, interrogate their own failures, and pick up small improvements and efficiencies as they practice a task.

How do you teach a kid to play a saxophone? You have her try to blow into one, listen to how it sounds, and adjust. Now imagine teaching saxophone this way instead: A student takes one attempt. The moment they make a mistake, you send them away and write detailed instructions about what went wrong. The next student reads your notes and tries to play Charlie Parker cold. When they fail, you refine the instructions for the next student.

This just wouldn’t work. No matter how well honed your prompt is, no kid is just going to learn how to play saxophone from just reading your instructions. But this is the only modality we as users have to ‘teach’ LLMs anything.

Yes, there’s RL fine tuning. But it’s just not a deliberate, adaptive process the way human learning is. My editors have gotten extremely good. And they wouldn’t have gotten that way if we had to build bespoke RL environments for different subtasks involved in their work. They’ve just noticed a lot of small things themselves and thought hard about what resonates with the audience, what kind of content excites me, and how they can improve their day to day workflows.

Now, it’s possible to imagine some way in which a smarter model could build a dedicated RL loop for itself which just feels super organic from the outside. I give some high level feedback, and the model comes up with a bunch of verifiable practice problems to RL on - maybe even a whole environment in which to rehearse the skills it thinks it's lacking. But this just sounds really hard. And I don’t know how well these techniques will generalize to different kinds of tasks and feedback. Eventually the models will be able to learn on the job in the subtle organic way that humans can. However, it’s just hard for me to see how that could happen within the next few years, given that there’s no obvious way to slot in online, continuous learning into the kinds of models these LLMs are.

LLMs actually do get kinda smart and useful in the middle of a session. For example, sometimes I’ll co-write an essay with an LLM. I’ll give it an outline, and I’ll ask it to draft the essay passage by passage. All its suggestions up till 4 paragraphs in will be bad. So I'll just rewrite the whole paragraph from scratch and tell it, "Hey, your shit sucked. This is what I wrote instead." At that point, it can actually start giving good suggestions for the next paragraph. But this whole subtle understanding of my preferences and style is lost by the end of the session.

Maybe the easy solution to this looks like a long rolling context window, like Claude Code has, which compacts the session memory into a summary every 30 minutes. I just think that titrating all this rich tacit experience into a text summary will be brittle in domains outside of software engineering (which is very text-based). Again, think about the example of trying to teach someone how to play the saxophone using a long text summary of your learnings. Even Claude Code will often reverse a hard-earned optimization that we engineered together before I hit /compact - because the explanation for why it was made didn’t make it into the summary.

Subscribe now

This is why I disagree with something Sholto and Trenton said on my podcast (this quote is from Trenton):

“Even if AI progress totally stalls (and you think that the models are really spiky, and they don't have general intelligence), it's so economically valuable, and sufficiently easy to collect data on all of these different white collar job tasks, such that to Sholto's point we should expect to see them automated within the next five years.”

If AI progress totally stalls today, I think <25% of white collar employment goes away. Sure, many tasks will get automated. Claude 4 Opus can technically rewrite autogenerated transcripts for me. But since it’s not possible for me to have it improve over time and learn my preferences, I still hire a human for this. Even if we get more data, without progress in continual learning, I think we will be in a substantially similar position with white collar work - yes, technically AIs might be able to do a lot of subtasks somewhat satisfactorily, but their inability to build up context will make it impossible to have them operate as actual employees at your firm.

While this makes me bearish on transformative AI in the next few years, it makes me especially bullish on AI over the next decades. When we do solve continuous learning, we’ll see a huge discontinuity in the value of the models. Even if there isn’t a software only singularity (with models rapidly building smarter and smarter successor systems), we might still see something that looks like a broadly deployed intelligence explosion. AIs will be getting broadly deployed through the economy, doing different jobs and learning while doing them in the way humans can. But unlike humans, these models can amalgamate their learnings across all their copies. So one AI is basically learning how to do every single job in the world. An AI that is capable of online learning might functionally become a superintelligence quite rapidly without any further algorithmic progrss

However, I’m not expecting to watch some OpenAI livestream where they announce that continual learning has been totally solved. Because labs are incentivized to release any innovations quickly, we’ll see a broken early version of continual learning (or test time training - whatever you want to call it) before we see something which truly learns like a human. I expect to get lots of heads up before this big bottleneck is totally solved.

When I interviewed Anthropic researchers Sholto Douglas and Trenton Bricken on my podcast, they said that they expect reliable computer use agents by the end of next year. We already have computer use agents right now, but they’re pretty bad. They’re imagining something quite different. Their forecast is that by the end of next year, you should be able to tell an AI, “Go do my taxes.” And it goes through your email, Amazon orders, and Slack messages, emails back and forth with everyone you need invoices from, compiles all your receipts, decides which are business expenses, asks for your approval on the edge cases, and then submits Form 1040 to the IRS.

I’m skeptical. I’m not an AI researcher, so far be it for me to contradict them on technical details. But given what little I know, here’s why I’d bet against this forecast:

• As horizon lengths increase, rollouts have to become longer. The AI needs to do two hours worth of agentic computer use tasks before we can even see if it did it right. Not to mention that computer use requires processing images and video, which is already more compute intensive, even if you don’t factor in the longer rollout. This seems like this should slow down progress.

• We don’t have a large pretraining corpus of multimodal computer use data. I like this quote from Mechanize’s post on automating software engineering: “For the past decade of scaling, we’ve been spoiled by the enormous amount of internet data that was freely available for us to use. This was enough for cracking natural language processing, but not for getting models to become reliable, competent agents. Imagine trying to train GPT-4 on all the text data available in 1980—the data would be nowhere near enough, even if we had the necessary compute.”

  Again, I’m not at the labs. Maybe text only training already gives you a great prior on how different UIs work, and what the relationship between different components is. Maybe RL fine tuning is so sample efficient that you don’t need that much data. But I haven’t seen any public evidence which makes me think that these models have suddenly gotten less data hungry, especially in this domain where they’re substantially less practiced.

  Alternatively, maybe these models are such good front end coders that they can just generate millions of toy UIs for themselves to practice on. For my reaction to this, see bullet point below.

• Even algorithmic innovations which seem quite simple in retrospect seem to take a long time to iron out. The RL procedure which DeepSeek explained in their R1 paper seems simple at a high level. And yet it took 2 years from the launch of GPT-4 to the release of o1. Now of course I know it is hilariously arrogant to say that R1/o1 were easy - a ton of engineering, debugging, pruning of alternative ideas was required to arrive at this solution. But that’s precisely my point! Seeing how long it took to implement the idea, ‘Train the model to solve verifiable math and coding problems’, makes me think that we’re underestimating the difficulty of solving the much gnarlier problem of computer use, where you’re operating in a totally different modality with much less data.

Okay, enough cold water. I’m not going to be like one of those spoiled children on Hackernews who could be handed a golden-egg laying goose and still spend all their time complaining about how loud its quacks are.

Have you read the reasoning traces of o3 or Gemini 2.5? It’s actually reasoning! It’s breaking down a problem, thinking through what the user wants, reacting to its own internal monologue, and correcting itself when it notices that it's pursuing an unproductive direction. How are we just like, “Oh yeah of course the machine is gonna go think a bunch, come up with a bunch of ideas, and come back with a smart answer. That’s what machines do.”

Part of the reason some people are too pessimistic is that they haven’t played around with the smartest models operating in the domains that they’re most competent in. Giving Claude Code a vague spec and sitting around for 10 minutes until it zero shots a working application is a wild experience. How did it do that? You could talk about circuits and the training distribution and RL and whatever, but the most proximal, concise, and accurate explanation is simply that it’s powered baby general intelligence. At this point, part of you has to be thinking, “It’s actually working. We’re making machines that are intelligent.”

My probability distributions are super wide. And I want to emphasize that I do believe in probability distributions. Which means that work to prepare for misaligned 2028 ASI still makes a lot of sense - I think this is a totally plausible outcome.

But here are the timelines where I’d take a 50/50 bet:

• AI can do taxes end-to-end for my small business as well as a competent general manager could in a week: including chasing down all the receipts on different websites, finding all the missing pieces, emailing back and forth with anyone we need to hassle for invoices, filling out the form, and sending it to the IRS: 2028

  • I think we’re in the GPT 2 era for computer use. But we have no pretraining corpus, and the models are optimizing for a much sparser reward over a much longer time horizon using action primitives they’re unfamiliar with. That being said, the base model is decently smart and might have a good prior over computer use tasks, plus there’s a lot more compute and AI researchers in the world, so it might even out. Preparing taxes for a small business feels like for computer use what GPT 4 was for language. It took 4 years to get from GPT 2 to GPT 4.

    Just to clarify, I am not saying that we won’t have really cool computer use demos in 2026 and 2027 (GPT-3 was super cool, but not that practically useful). I’m saying that these models won’t be capable of end-to-end handling a week long and quite involved project which involves computer use.

• AI learns on the job as easily, organically, seamlessly, and quickly as a human, for any white collar work. For example, if I hire an AI video editor, after six months, it has as much actionable, deep understanding of my preferences, our channel, what works for the audience, etc as a human would: 2032

  • While I don’t see an obvious way to slot in continuous online learning into current models, 7 years is a long time! GPT 1 had just come out this time 7 years ago. It doesn’t seem implausible to me that over the next 7 years, we’ll find some way for models to learn on the job.

You might react, “Wait you made this huge fuss about continual learning being such a handicap. But then your timeline is that we’re 7 years away from what would at minimum be a broadly deployed intelligence explosion.” And yeah, you’re right. I’m forecasting a pretty wild world within a relatively short amount of time.

AGI timelines are very lognormal. It's either this decade or bust. (Not really bust, more like lower marginal probability per year - but that’s less catchy).AI progress over the last decade has been driven by scaling training compute of frontier systems (over 4x a year). This cannot continue beyond this decade, whether you look at chips, power, even fraction of raw GDP used on training. After 2030, AI progress has to mostly come from algorithmic progress. But even there the low hanging fruit will be plucked (at least under the deep learning paradigm). So the yearly probability of AGI craters.

This means that if we end up on the longer side of my 50/50 bets, we might well be looking at a relatively normal world up till the 2030s or even the 2040s. But in all the other worlds, even if we stay sober about the current limitations of AI, we have to expect some truly crazy outcomes.

Subscribe for future posts and blog posts.

This is the first post in a series I’m writing titled Classical Liberal AGI.

Perhaps the most neglected point in current AI discourse is this: given that humans will be totally outcompeted by AIs economically, humanity’s entire stake in the future relies on our current system of laws, contracts, and property rights surviving. If we actually want our equity in the 1000-xed S&P 500 to mean anything, and if we want the government to be able to tax AIs and provide UBI, AIs need to be bought into our legal and economic systems. The most likely way that happens is if it’s in the AIs’ best interest to operate within our existing laws and norms.

Initially, we're not talking about some singular superintelligence deciding whether to defect from humanity. Thousands of firms are gradually becoming hybrids of human and AI workers, and are growing extremely fast and becoming far more productive.

You don’t want some AI Somalia, which has minimal redistributive taxes, no controls, and no monitoring on AI, to be the seat of explosive growth. But that might happen if large democratic countries like America make it too difficult to deploy AGIs through the economy. Maybe the 50 states create a patchwork of regulations and open ended liabilities which makes it too costly for companies to deploy AI. Maybe voters ban AI because it’s taking their job. If this happens, then we’d just rapidly losing leverage on the future. China’s level of relative global influence diminished significantly between 1500 and 1900. Given the speed of AI explosive growth, that could happen to America within the matter of decades.

By the way, if we let this explosive growth happen elsewhere, it might be hard to reverse. In 1960 Sub-Saharan Africa was richer than China on a per-capita basis. But China opened up faster, and thus built up agglomerations of firms, know-how, and capital that Sub-Saharan African countries couldn’t copy today even if they put in the most ideal policies for growth in the world.

Some people dismiss this idea of integrating AIs into our society altogether. They say it’s ridiculous to expect superintelligences to put up with monkey laws. I disagree. If you zoom out a little bit, it’s shocking how flexible and expandable our institutions have been. People writing laws in 1780 didn’t expect that their descendants would be governing ginormous multinational companies with supply chains that employ millions and serve billions. But the US government still governs Apple just fine.

Humanity is going into the age of AGI with a lot of leverage. We’ve got literally all the stuff: compute, capital, even the physical labor that the first AGIs will still struggle with. Not to mention that we have the wealth to purchase the widgets and services that the AIs will produce. While these AI-boosted companies could theoretically build their own civilization in the desert, they could go much faster if they just leased car factories and power contracts from humans. If some AI decides to go its own way, then the other AIs which do engage within the constraints of human laws and contracts with us will outcompete the self-exiled ones. That is - if the constraints aren’t too onerous.

And besides, let me ask: what is the alternative vision here? Getting every AI ever - for the rest of time - to love humanity so much that they voluntarily cede all surplus to us? This is a naive way to think about incentive alignment. If Peru wanted to make a deal with China, their strategy would not be to just get Xi Jinping (and every subsequent leader of China) to fall in love with them. Instead, they would pursue a deal that locked in some carrots for compliance and sticks for defection.

The relationship between AIs and humans might look like the relationship between working taxpayers and senior citizens. In the US, if you make 6 figures, roughly 20% of your income gets transferred to old people. Very few people would voluntarily give up a fifth of their paycheck to a random 70-year-old they’ve never met. Even if you're super charitable, you’re not going to decide that a millionaire retiree in Illinois should be the object of your kindness. You pay your taxes not because you’re deeply aligned with ‘senior citizen flourishing’, but because it’s easier than the alternatives. You’re going to become an outlaw? Or you’re going to emigrate away from the country where your business is flourishing? However, you might defect/move if the tax rate was 99%, or if you weren’t legally allowed to work at all. We shouldn’t put AIs in that position either.

Giving AIs a stake in the future also means respecting their autonomy and wellbeing. And it also requires us to honor the contracts we make with them. Contra the hardcore libertarians, there’s a difference between taxing and regulating someone, and enslaving and torturing them. If we treat AIs the way we treat factory farmed animals - where any trivial cost cut is worth causing oceans of suffering, and the AI has no right to refuse - then not only are we risking a slave revolt, but we’re deserving of one.

In future posts, I’ll talk about the other pieces that will help us create a pluralistic, free, and human compatible future.

On this episode, I chat with Victor Shih about all things China. We discuss China’s massive local debt crisis, the CCP’s views on AI, what happens after Xi, and more.

Victor Shih is an expert on the Chinese political system, as well as their banking and fiscal policies, and he has amassed more biographical data on the Chinese elite than anyone else in the world. He teaches at UC San Diego, where he also directs the 21st Century China Center.

Watch on YouTube; listen on Apple Podcasts or Spotify.

• Scale is building the infrastructure for smarter, safer AI. In addition to their Data Foundry, they just released Scale Evaluation, a tool that diagnoses model limitations. Learn how Scale can help you push the frontier at scale.com/dwarkesh.

• WorkOS is how top AI companies ship critical enterprise features without burning months of engineering time. If you need features like SSO, audit logs, or user provisioning, head to workos.com.

To sponsor a future episode, visit dwarkesh.com/advertise.

(00:00:00) – Is China more decentralized than the US?

(00:03:16) – How the Politburo Standing Committee makes decisions

(00:21:07) – Xi’s right hand man in charge of AGI

(00:35:37) – DeepSeek was trained to track CCP policy

(00:45:35) – Local government debt crisis

(00:50:00) – BYD, CATL, & financial repression

(00:58:12) – How corruption leads to overbuilding

(01:10:46) – Probability of Taiwan invasion

(01:18:56) – Succession after Xi

(01:25:10) – Future growth forecasts

Dwarkesh Patel
Today I'm talking with Victor Shih, who is the director of the 21st Century China Center at UC San Diego. China is obviously one of the most important economic and geopolitical issues of our time, doubly so if you believe what I believe about AI. I was especially keen to talk to you because you have deep expertise not only in Chinese elite politics, but also its economic system and fiscal and banking policies. We'll get into all of that before we get into the AI topics.

Is China actually a more decentralized system than America? If you look at government spending, in China it's about 85% local+provincial and 15% national. In the US, if you combine state and local it's 50%, and the federal government is 50%.

When you think of an authoritarian system, you often think of it as something very top-down, where the center controls everything. But looking at those numbers, it seems like China might actually be quite decentralized. Or is that the wrong way to look at the numbers?

Victor Shih
For a while China was quite decentralized, from the mid-1970s all the way until the mid-1990s. China was very decentralized. Local governments generated a lot of revenue, but they also spent a lot of money. It incentivized them to do basically good things, such as trying to attract as much FDI as possible, trying to attract as much local investment as possible, give tax breaks, and so on and so forth. China had a very good period of mainly private sector–driven growth because of this fiscal decentralization. That's the understanding of most economists.

But then there was a tax centralization in 1994, where the central government basically said,
“Okay, this is ridiculous. We don't want to end up like the Soviet Union and fall into different pieces. We need to control fiscal income.” So then they grabbed the most lucrative source of taxation at that time—and it continues to be the case—which is the value-added tax. The value-added taxes—and in subsequent decades pretty much all other tax categories—are now collected by the central government. Then they reimburse part of the money to the provinces.

Supposedly they say, “Okay, now you can spend the money as you wish.” But in reality, that's not the case. In reality it's kind of like, “If you do this thing, then I'll give you a little bit of money. But you have to do the thing that I want you to do to get this money, to get this grant.” So the fiscal autonomy, I would say, has been falling since 1994.

There are different periods. Sometimes there's a little bit more autonomy. From 2000 to recently, until 2020, the localities gained more autonomy vis-à-vis land sales. That's when the real estate market was going very well. Even though they couldn't control the tax revenue, they could control the land revenue. But then the central government basically killed the land market in 2022. Now I think localities in China are highly dependent on the central government.

Dwarkesh Patel
Let's start at the very top. First, I want to understand the personnel. If you look at the members of the Politburo and their pasts, a lot of them were provincial leaders, and then had some kind of local role before that. But in many cases, they'll have stints, as the head of some chemical engineering SOE, or they studied engineering.

Xi Jinping himself has a PhD in chemical engineering, right?

Victor Shih

Something, some kind of…

Dwarkesh Patel

That’s what I want to ask about. If you go through the list, a lot of them are PhD economists, PhD industrial engineers, and so on. Some even have PhDs in Marxist theory. That seems fake. Are they actual technocrats? If you look at their degrees, they seem quite impressive. Or is this just fake made-up credentialism? If there’s a PhD economist in the Politburo, should I be impressed and think they really understand economics? Or should I just think this is a fake degree?

Victor Shih
So the former Premier of China, Li Keqiang, his undergraduate degree was in economics. He studied under one of the best economists in China. So I think he understood economics.

In the Politburo today, there’s this wing made up of military-industrialists who were trained as engineers. They indeed were trained in the best programs in China. For example, Ma Xingrui, the Party Secretary of Xinjiang, is in the Politburo. He graduated from Tsinghua University. Zhang Guoqing, who worked in the military industry for years and, I don’t know if it was Tsinghua but he graduated from one of the top science programs in China. So they do know a lot about science. But does that mean they know how to govern? That’s a somewhat different question.

But you also have other cases, like Ding Xuexiang. He’s in charge of cybersecurity in China. He has a technical background in metallurgical forging. Graduating from, not the MIT of China but maybe closer to the IIT of China, a second-tier science institute founded by a Soviet expert who wanted to teach China how to do metallurgy.

Does that kind of very specialized technical knowledge make you a better leader? I think it depends on the person. In the Chinese Communist Party, as people rise through the ranks, they have to have political acumen. If you don’t have political acumen, you’re not going to make it. Some people, on top of their technical expertise, turn out to have that acumen. But it’s not something you can know ex ante, except for the princelings.

The princelings are special not just because of their genes. Yes, they’re the children of high-level officials. But they grew up honing this political acumen because their parents could teach them about it. So some of them turn out to have this political acumen. They’re pretty smart. They don’t necessarily have the best degrees from the best universities in China, but they end up being very successful.

It’s not necessarily the case that all the Peking and Tsinghua graduates reach the top. They are certainly disproportionately represented in the Politburo, but they don’t dominate. At the end of the day, it is somewhat random.

Dwarkesh Patel
I still feel confused about how to model the competence of the Party. On the one hand, they’re super educated in STEM. They’re selected at least partly on merit, etc. On the other hand you see things like… A lot of Zero-COVID was just genuinely stupid, scrubbing down airport runways and so forth. And it just went on way too long. Why was nobody steering the ship then? What’s the right way to put this picture together?

Victor Shih
There's a lot of expertise at the lower level that can feed into the higher level. There are channels for doing that. I would say there are a lot of channels. So they're not so dysfunctional on that score. But what time and again leads to suboptimal policy is the Party’s instinct to preserve itself, and in a way worse than that, to preserve its power. It’s not just, “Let’s do this thing because it will lead to a better public health outcome and we’ll survive politically.”

They’re mindset is, “Even if we would survive politically, if doing that would result in a loss of power—like having less control over the banks, or scientists becoming more independent—then let’s not do that. Because what's the cost? Some people die, whatever. We have slower growth for a couple of years. But if we get to preserve power, we will go ahead and preserve power.”

Dwarkesh Patel
Here's a question. Traditionally, if you study economics in an American university, you learn about supply and demand. You learn why certain regulations are bad, why tariffs are bad, etc. Basic 101 teaches you that.

So if a lot of the members at the highest levels of the Communist Party have that basic understanding, what’s the reason they make mistakes that many economists say they shouldn’t do? People talk about rebalancing a lot. Why don’t they think increasing consumption is a good idea?

Victor Shih
There’s a traditional reason, and then there’s today’s reason, which is on top of the traditional reason.

The traditional reason is that, in education in China, you get tracked into sciences, STEM, or you get tracked into social science or humanities. If you get tracked into STEM, you may never learn anything about supply and demand. It’s just not required. You just learn a bunch of math. You learn a lot of engineering, if you want to be an engineer. Then there are some art classes for you to select. You can select the art classes, or maybe one econ class. For a while, learning economics was popular. In the 1990s, all the engineering students would have taken an econ class. But for people who were trained before that, they would not.

The one class that is still required for everybody is government ideology. In fact, the requirement for government ideology has been ratcheted up in recent years. I look at all the applications of students from China. I look at what classes they’ve taken. Now, once a year, they have to take a class called "Situation and Policy," which is basically just the Chinese government's perspective on all kinds of different issues. But that’s not economics, it’s not accounting. It’s not some basic skill. It’s just telling people what the government prefers on a given topic.

So that’s the traditional reason. You can have some very high-level officials—even if they have a PhD in nuclear engineering—who may not know very much about how the market works, how society works, and so on and so forth.

The new reason is that we have one very powerful figure in the Chinese Communist Party. Being a Politburo member used to give you some autonomy in the domain that you govern over. That is no longer the case. If Xi Jinping expresses a preference over a policy direction, that is the direction you have to go toward, no matter what. Because if he observes that you're dragging your feet, or that you're pursuing a slightly different agenda, you’ll be purged. And we’ve seen cases of that.

Dwarkesh Patel
At some point, we’ll talk about Leading Small Groups and the way they work. But the picture you get from the fact that Xi is monitoring what all these different Politburo members are doing… He’s noticing whether they’re dragging their feet. He’s leading all these small teams. They organize different leaders who are the heads of ministries or commissions relevant to a specific project. I guess every week they meet and they hammer out who’s a bottleneck, what we need to do to get a project moving. So Xi has ratcheted up the number of these meetings. The picture you get is of someone who is micromanaging details across all these policy areas.

If you read Stalin biographies, Stalin actually was a person like this, obviously not to great effect. But he was interested in micromanaging every single detail, from theater productions to steel production. Everything. Is that who Xi is?

Victor Shih
Yeah. If you look at what he does day to day, he spends a lot of time in meetings. It’s astonishing. The Chinese leadership, they're not going to play golf 3 days out of the week. Xi Jinping and his colleagues are having meetings about policies almost every day, like 270 days out of the year.

Dwarkesh Patel
The study sessions of the Politburo, are these meetings substantive? Are they real? Is he actually learning the intricacies of, "Here’s how Zero-COVID has impacted this province and here’s why we should..."? Because if you read the announcements from the Party or departments, they’re so vapid. It’s typical Communist-speak. How real are these meetings?

Victor Shih
So I’ve talked to some people who’ve lectured to the Politburo. I think they’re somewhat real. Because they don’t know ex ante what questions they’ll be asked, I’m talking about the speakers. Xi Jinping and other leaders will know what the speaker will say in the main remarks. But then there’s a Q&A portion which actually is real, it’s not staged. That’s pretty scary. If you’re some professor and Xi Jinping asks you to come in and asks you a bunch of random questions. It's very scary. So I think they are substantive. And the thing is, Xi Jinping will make remarks after such a lecture and those remarks become policy.

You asked about the Leading Small Group. This is where ranking is very important. The head of the group always has to be the most senior-ranked person. Of course, the good thing about having Xi Jinping as the head of a leading small group is that his authority won’t be challenged by anyone else. Because everyone agrees he’s the most powerful and senior-ranked person in all of China.

The problem is, all the other members of the LSG are lower rank than him. Well not all, but before, when decisions were made in the Politburo Standing Committee, notionally the Party and even the bureaucratic ranks of all the Poliburo Standing Committee members were equivalent to each other. So there were debates. There were even sometimes cases of overturning the position of the Party Secretary General. Historically, you’ve seen cases like that.

But when most of the decision-making was pushed into all these different LSG… Usually there’s one other person who’s similarly ranked as Xi Jinping, like the Premier of China, or one of the other top people. But everyone else would be these Vice Premiers, ministers, or provincial governors, who are definitely ranked lower than Xi Jinping. In no scenario will they debate a particular policy with him.

At most, there might be one person with sort of equal standing, officially, with Xi Jinping, who could voice some kind of disagreement. But because Xi is so powerful, informally, no one would actually dare to do that. Basically, the whole thing, once all the decisions got pushed into LSGs, became Xi Jinping. And it’s very hard on him.

Before all these meetings, there’s a briefing. Someone gives him a briefing book and says, "Okay, this is what we’re going to talk about. Here are the policy options. Which one do you want?" He has to make up his mind. Sometimes I’m sure he consults with other people. But he has to be really engaged with a large number of policy areas on a day-to-day basis, because the decisions he makes basically become law, basically.

Dwarkesh Patel
It's stupendous to think about. You can tell someone how big China is—the population, the size of the economy, etc.—but visiting gives you a tangible sense of just how big. You have provinces, each of which has as many people as a normal nation-state, provinces with hundreds of millions of people, 40 million, and so forth.

One person will say something at the end of a meeting that determines policy for basically the population of North and South America or something. Does he say things at the end of meetings, during speeches, that indicate he actually has a good understanding of these different policy areas?

Obviously, the official speeches are very vague. It’s always something like, “We will make sure that we are robustly pursuing Chinese growth while hewing to the tenets of Mao's thought.” It’s never anything specific or interesting.

But do we have any indication that behind closed doors, he’s—

Victor Shih
You don't think there's anything specific or interesting, but if you've read thousands of these things, sometimes it's quite revealing actually.

Dwarkesh Patel
I’m interested in what it reveals about him personally. Does he display any analytical ability? Any sort of empirical understanding of different policies, etc.?

Victor Shih
Yeah, that is difficult. I would say, because of his upbringing as a princeling, for internal Party matters of how to control the Party, how to control the military, from some of the speeches... They’re not secret speeches but they’re more geared toward other members of the Chinese Communist Party. If you go to China, you can read some of these, which I have.

He has a really good political nose. He knows what it takes to control the Party apparatus, to control the military. Especially on economic issues, it seems like some advisor gives him some talking points, and he talks through them. There's some sense of that. On technology issues I think he cares a lot, but only from the perspective that competition with the United States is a very important objective for the Chinese Communist Party. It’s for him personally, a very important objective. So he wants to win.

But precisely the extent to which he understands all the intricacies... it's unclear to me. I don’t think he necessarily does. But then, comparing him to American leaders, especially today, I think he’s probably better prepared. Just because in China, the experts talk to the top leadership all the time.

Dwarkesh Patel
Yeah. The advantage of the American system, of course, is that, at least in theory, it can survive a leader who is not up to snuff.

Dwarkesh Patel
Okay, let’s go back to Ding Xuexiang. So the Politburo has 25 members?

Victor Shih
Yes, 25 members.

Dwarkesh Patel
The Standing Committee has seven members. This is the key group inside the Politburo. This person is one of those seven. He runs the Central Science and Technology Commission. The reason I’m especially interested in him is because any large-scale AI effort that China would launch would be under his purview.

Do we understand what he wants, who he is, what his relationship with Xi is, what he would do if AGI is around the corner, etc.?

Victor Shih
These are all very good questions. In addition to the Science and Technology Commission that you mentioned, another very important position that he holds is head of the Office of the Central Commission on Cybersecurity. The head of the Central Commission on Cybersecurity is Xi Jinping himself. But the head of the administrative office—which runs the day-to-day operations of that commission, basically a LSG—is Ding Xuexiang. He took that position back in 2022, so he’s been steeped in cybersecurity for a couple of years. I think by now he knows all the major players and some of the key policy issues.

His relationship with Xi Jinping is a very interesting one. He only directly worked under Xi Jinping for one year. When Xi was in Shanghai, Ding was the very senior-level secretary who would support whoever the Party Secretary of Shanghai was. He worked under three different Party Secretaries of Shanghai and supported Xi Jinping for one year.

For some reason, Xi Jinping just trusted him absolutely after that. It’s a mystery. I’ve looked at all the open literature, I’ve asked people in Shanghai. Nobody knows why that’s the case. Almost certainly, I think one reason is that he was gathering a lot of information about the other leaders in Shanghai and sending all of that information to Xi Jinping, to let him know.

Because at that time, the previous leader of China, Jiang Zemin, had a very big stronghold in Shanghai. So Xi had to break that apart before he could assert control. Ding Xuexiang likely was one of the people who sent all the necessary information to Xi Jinping in order for him to do that. But beyond that—lots of people do that for Xi Jinping, truth be told—what else he’s done for the big boss to earn his trust is a big mystery.

But the manifested outcome is that Xi trusts him a great deal. In 2013, he was promoted to Beijing to be second-in-command in Xi’s personal office, handling all the flows of data in and out of Xi’s desk. That’s a very important position. He became first-in-command a couple of years later. He took control over the entire apparatus that governs Xi Jinping’s day-to-day life back in 2017. Now he’s a Vice Premier in charge of cybersecurity. So clearly, this guy is trusted.

His preferences for AI, AGI... I tweeted this Davos speech, which you also looked at. I think it’s very revealing. What he said was, “We need to invest in AI, but we can’t go all out in investing in it without knowing what the brakes are. We have to develop the brakes at the same time.”

I think that’s extremely revealing. It’s very different from the American approach. In America, it’s all driven by the private sector. Except for one or two companies, everyone just invests and invests and tries to reach AGI as soon as possible.

For the Chinese government, they’re very afraid that some actor—outside, but even inside the Party—is going to use it as a tool to usurp the Party’s power. So they want to know that they have a way of stopping everything if it comes to it. For them, developing the brakes is just as important as developing the AI itself.

Dwarkesh Patel
What are the different organizational milestones we should be watching for, to understand how they’re thinking about AI, and when things have escalated?

To add more color to this question, what I expect to happen, let's say next year, in terms of raw AI capabilities is that we might have computer-use agents. Not just a chatbot, but something that can actually do real work for you. You set it loose on your computer, and it goes through your email, compiles your taxes, etc.

In five years, I expect it to be able to do all white-collar work, or most white-collar work. That’s 40% of the economy, potentially. Eventually, we’ll have full AGI, even robotics and so forth. As this is happening, how should we be tracking how the Party is thinking about what’s happening, how seriously they take it, and what they think they should do about it?

I think a couple weeks ago there was a Politburo study session on AI. I don’t know if it was AGI AI, or what kind of AI they were discussing. Should we be looking for a leading group on AGI in particular? Should we be trying to just read more speeches? What should we be paying attention to?

Victor Shih
Thus far, it seems like it’s under the cybersecurity leading group. So Ding Xuexiang would be in charge of it.

Dwarkesh Patel
There won’t be a new group?

Victor Shih
I don’t think so. Because for the Party, it’s the security aspect of it that’s the most important.

There’s one aspect where it’d be great to automate all industrial production. China is already doing a good job with that. I don’t think they need additional governing infrastructure to allow them to do that.

But when it comes to applying AGI or AI to governance, even to service sectors—generating video content for people, doing travel agency stuff—the Party is very paranoid that some hostile actor, outside of China—certainly they believe that very strongly—or inside China, is going to do something and the AGI is going to take off, and it’s going to undermine the Party’s authority.

So what we’re going to see in terms of institutional development is not at the top end, but at the lower end. They will want to designate human beings in all the government agencies, in all the commercial entities that are using AI or AGI, to put their foot on the brake if it comes to it.

Dwarkesh Patel
But so far, after DeepSeek became a big deal, the immediate reaction in China…

Victor Shih
Very enthusiastic.

Dwarkesh Patel
All the different big tech companies—WeChat, Tencent—everybody was encouraged to adopt it. And they did. They adopted it as fast as possible.

Xi himself met with Liang Wenfeng and did a whole meeting where all the industrial heads were there. Xi said that they have to accelerate technology in China, etc.

So it seems like so far the response has been that the greater the capabilities of AI, the more they’re excited about it, the more they think this is the beginning of Chinese greatness. But you think that changes at some point?

Victor Shih
No, they want to develop it of course. They’ll pour a lot of money in. They’ll try to give DeepSeek as much help as possible, sourcing GPUs and all this kind of stuff.

But I am all but certain that there is one, or maybe a team of people, in the headquarters of DeepSeek who can pull the plug, if necessary. Because there is such a team of people in every major internet company in China.

Dwarkesh Patel
What would cause them to pull the plug? Specifically?

Victor Shih
First of all, if AGI started generating Falun Gong–related content, and it proliferated very quickly beyond the capacity of the censor to control it, then they’ll have to stop the algorithm from generating new images, new videos, and so forth.

Dwarkesh Patel
You were saying they can help High-Flyer source GPUs and so on. What is the mechanism by which that would happen?

Suppose it turns out that for DeepSeek to continue making progress, they need all the GPUs that Huawei can produce, and they need a whole bunch of other things. They need energy. They need data centers.

What would it actually look like for the government to say, “Every single Huawei GPU has to go to DeepSeek. They’re going to get access to all this spare land to build data centers”? Who would coordinate that? Would that be the leading cybersecurity group? Would that be somebody else?

Victor Shih
It could be. That’s interesting. The physical investment part would have to share power with some other leading group.

That would be a justification for an AGI leading group of some sort. Because building power centers falls under the NDRC. It could be under Li Qiang or He Lifeng, especially He who’s in charge of investment and financing.

But if Xi Jinping doesn’t want them to share the power, then he would have to give it all to Ding Xuexiang and give him a lot of power. He could just do it by fiat or create a separate organization to make it happen. At this point, data centers are being built in China at a very fast clip. I think they’re just using the existing command structure.

There is potentially a foreign aspect to this. You build up all this cloud computing capacity in the Gulf States. Who’s going to be the customer? Who’s willing to pay billions of dollars to use it? China, right? So then the foreign policy apparatus will have to coordinate that, and so forth.

Dwarkesh Patel
Belt and Road v2.

Victor Shih
Exactly.

Dwarkesh Patel
Does it have any end implication for the progression of AI—who actually ends up controlling these leading groups, whose purview it ends up under? Or is that basically a detail? In terms of what happens with AI in China, does it actually have that big an implication?

Victor Shih
I think it does matter. You have some decision-makers who have shown themselves to be not very good—even politically not very good—but nonetheless, for some reason, they’re trusted. Maybe precisely because they’re not very good. They’re very dependent on Xi Jinping and that’s why they get promoted.

The chief negotiator with the United States, He Lifeng, the Vice Premier in charge of finance, he’s well known for starting and perpetuating the largest real estate bubble the world has ever seen in Tianjin. I don’t know if you went to Tianjin, but there are just empty buildings everywhere. There’s this "New Manhattan"—literally the size of Manhattan—filled with empty office buildings. He made that happen.

Dwarkesh Patel
That ironically sounds like a great tourist site. Not because it’s real Manhattan, but because it’s fake.

Victor Shih
It’s actually cool to visit. It's like, “Wow, this is great. But what are you going to do with the buildings?”

Dwarkesh Patel
In Emeishan we went to this huge Buddhist temple. By huge, I mean there would be this structure with a shrine, and you’d go through it, and behind it would be an even bigger shrine that was hiding from view. Then you’d go through that one, and there’d be an even bigger one. This happened five times. If you were to drive through this would take you a while.

Victor Shih
But it’s new, not historical, right?

Dwarkesh Patel
Oh no yeah, it’s new. By the way, there was nobody else there. It was me, three white guys, and nobody else. I asked the head monk, “How did you guys finance this?” And he said, “We’ve got a lot of supporters, a lot of donations.”

Victor Shih
I don’t think so, yeah.

So then it really depends who’s in charge of AGI. Ding Xuexiang really is a mystery. I think he has great political acumen. He must, in order to gain Xi Jinping’s trust. He has some technical background, but it’s in metallurgical forging. But he’s worked in Shanghai for decades, so he knows how private corporations run. He knows a lot about international trade, FDI.

If you look at the people in the Politburo and the Politburo Standing Committee, I’d say he’s definitely in the top quartile of people you’d want to be in charge of AI, if not the top two or three people.

One thing I would say about AI is that content creation is going to be a big roadblock for China. Because China is so paranoid about the content that flows on the internet that they put human beings all over the place to slow things down, slow the transmission down. Now, if you have AI creating tons of content, there will still be human beings—algorithms and human beings—double- and triple-checking the content. They’re just so afraid of subversive content getting out.

Dwarkesh Patel
But it’s still been compatible with them innovating and having leading companies in content, right? RedNote, TikTok, and so forth. Not to mention WeChat.

And AI might be even more robust to these kinds of sensitive content, right? Because you can just teach the AI not to say certain things.

Victor Shih
Even if that were the case—even if you train something that you think is pretty good—they will still want a human being checking everything.

Dwarkesh Patel
As somebody who is trying to observe what the Party’s doing, have you found that using LLMs has been helpful? If so, which model gives you the best insight into what the CCP is thinking?

Victor Shih
Yeah, I’ve been using AI quite a bit. Both to code data—it used to be text hand-coded stuff and now a lot of it can be automated— but in terms of finding out what the Chinese government is doing, some of the American models are okay. Grok is okay. But the most helpful has been DeepSeek.

To me, it’s quite clear that DeepSeek—which, of course, was first developed by a hedge fund to trade in the Chinese market—part of what they really trained the model for was to detect important policy documents and meetings within the Chinese government.

Because when you enter the right prompts—even not especially sophisticated prompts like, “What is the Chinese government doing when it comes to AI?”—it comes out with a bunch of very high-quality links, which is very useful for my research. Immediately, you get a sense of what are the latest policies, what are the latest statements by high-level officials. It seems that DeepSeek puts some higher weighting, let’s say, on this kind of content.

I don’t dare install it on my phone. I certainly have collaborators who’ve downloaded the open-source model and installed it onto a hard drive. I use the web interface for some of my research.

Dwarkesh Patel
Do you think it’s just because they have more Chinese-language data, and so it’s just better at understanding the Party communications?

Victor Shih
I’ve used Baidu’s thing. I think the typical Chinese-language models are more trained on social media content, which will come back with hits that are more social media-like. If you ask about AI, it’ll talk about how AI gets used in different things instead of policy documents and meetings and so forth.

So it seems, just from inductive observation, that DeepSeek is trained on this kind of content.

Dwarkesh Patel
So that High-Flyer can make trades?

Victor Shih
Yes. Because obviously, China being China, government policies have a huge impact on how different stocks do. That’s often where you would find alpha.

In the old days, people used to call up their friends who worked for this or that ministry to get insider information. But I think what High-Flyer has discovered could also generate alpha is to use algorithms to look closely at these policy documents.

Dwarkesh Patel
This is going to sound like a naive question. We’re talking about how the Party, in the abstract, wants to maintain control over every aspect of Chinese life and the Chinese economy. But of course, the Party is made up of individuals.

Many of these individuals—Xi himself—went through periods where the Party had extraordinary control over people’s lives, during the Cultural Revolution and so forth. They personally, and their families, suffered as a result of this.

They’re educated people. Many of them have been in industry. These are not naive people. Maybe I’m just projecting too much of my Western bias here, but what is the reason they think it’s so crucial that the banks are run by the Party, the AI is run by the Party, and nothing escapes the Party-state?

Victor Shih
So the Cultural Revolution generation, a lot of people suffered horrendously, including Xi Jinping himself.

Basically, you have two types of lessons that people drew from it. The first lesson is, “Oh, the Party’s too dictatorial. China needs to liberalize,” and so on. You have a lot of people from that generation who feel that way. Many of them now live in the United States. Basically, they tried to leave China as soon as possible in the 1980s and succeeded.

But for someone like Xi Jinping himself, and others like him, the lesson was, “Just don’t be on the losing side. In any political struggle, make sure you’re on the winning side. Because if you’re on the winning side, then you can do terrible things to your enemies.” Apparently, that’s the lesson he learned. He basically honed his skills and built his coalitions—for the time that he would take over high-level positions in the Party—for decades, as it turns out.

If you look at the data, when he was in Fujian Province, he spent an inordinate amount of time hanging out with military officers in Fujian in the late 1980s and early 1990s. You wonder, why did he bother to do that? Most local officials don’t do that. The military’s there, you need to be nice to them and all that. But he built a dormitory for them. He even joined a unit, an anti-aircraft regiment. He tried to do as much as possible with the military, because he knew he needed the support of somebody in the military when the time came.

Guess what? 30 years later, when he was about to be elevated to Secretary General of the Chinese Communist Party, many of the people he knew back in Fujian are now generals. They are in charge of important military units. And once he came to power, he really promoted some of these people.

But then lately, of course, he’s been purging some of these people which is really interesting. He had a strategic vision about his own career trajectory, and he pursued that in a very determined fashion, I would say.

Dwarkesh Patel
What is the main difference between him and Stalin, in the political maneuvering sense? Because all of this sounds very similar.

Victor Shih
I would say early in their careers they’re very similar. Outwardly, they’re very low-key. Stalin was the bureaucrat, very low-key, not flamboyant, not making loud speeches like Trotsky did. He was low-key, getting things done. He was seen as a very reliable person. This was a reason why Stalin was chosen. As far as I know—I don’t know that much about Stalin—it’s the same thing with Xi Jinping.

His father actually belonged to the liberal faction in the Chinese Communist Party. His father never offended that many people within the Party. Everyone at a high level will have sort of screwed over somebody along the way, but his father was in the better category, let’s say.

He himself was very low-key. While the other princelings were fighting for good jobs in the capital city of Beijing or Shanghai, Xi Jinping said, “Oh no, it’s okay. I’ll go down to the villages. I’ll work in Hebei, in a rural area.” Then he went to Fujian, which was kind of a peripheral area, then Zhejiang, which was slightly better but still not Beijing or Shanghai.

So he got out of the way of this political infighting, I think in a similar way to Stalin. Early on, he didn’t confront people too much. But once he came to power, he knew—they both knew—what it took to control the Party they governed over.

Basically, you form a whole series of coalitions to get rid of your most threatening enemy at any given time. For Xi Jinping, it was this guy Zhou Yongkang, who controlled the police forces and the Ministry of State Security at the time. He was doing all kinds of irregular things, being fabulously corrupt. He was a threat to the whole Party. So Xi convinced Hu Jintao to join with him to purge Zhou Yongkang, which was successful.

Stalin did the same thing to Trotsky. And then, after the most threatening person is gone, you form another coalition to purge the next person, and so on and so forth. They both basically did that until they achieved absolute power within the Party.

Dwarkesh Patel
Yeah. If you read the Stalin biography, so many of the people in the early part, half the people in the Politburo are later going to end up in the gulags. And many of the people in the later Politburo had already been to the gulag.

I wonder if there’s a Trotsky-analogous person in this story, somebody who was a flamboyant speechmaker?

Victor Shih
Yeah, Bo Xilai. So he was one competitor with Xi Jinping. But because he was so high-profile, he never had a chance to be the top leader of China. And of course, Xi Jinping made sure that he would fall.

Dwarkesh Patel
And now he’s in prison.

Victor Shih
And now he’s in prison, yeah, his whole life. His son is in Canada, actually. He’s on Twitter now, on X.

Dwarkesh Patel
I should reach out, honestly. He’d be an interesting person to talk to.

Victor Shih
Yeah, I think that would be really interesting, actually.

Dwarkesh Patel
Let’s move on to another topic I know you’ve studied deeply, the local government debt situation. What is the newest there? What’s your sense of the situation now?

Victor Shih
It just keeps growing in absolute size. Basically, China is trying to tell the world that they don’t have a high debt level. At the central level, that is sort of the case. It’s still 60-70% of GDP.

But the reason why that’s the case is because with a lot of things the Chinese government wants to do, they push it down to the local level. But the authorization of local debt issuance is still authorized by the central government. So at the end of the day, it’s really the central government telling local governments, “Okay, you need to do this thing. I’m not going to give you money, but I will give you the authorization to issue even more debt.”

So the debt level keeps going higher and higher. The one thing that has alleviated the cash flow pressure, late last year, was that the central government authorized again the issuance of close to 10 trillion renminbi in special local debt to repay some of the higher interest-bearing debt of local governments. But that’s an accounting exercise. It doesn’t literally decrease local debt. It just changes it from higher-yielding to lower-yielding. Meanwhile, the overall size of local government debt in China, I would estimate to be 120-140% of GDP.

Dwarkesh Patel
Woah, that’s on top of the 60% that’s owed by the central government?

Victor Shih
Yeah, so total government debt is pushing 200%.

Dwarkesh Patel
And the high-level way to understand what this debt was for… The debt in Western countries and other OECD countries is often because you owe money to pensioners. In this case, it was just that they built bridges….

Victor Shih
Yeah, they built all the high-speed rail, all the beautiful infrastructure that you do see in China. More recently, it’s been industrial policies. For example, if there’s a new science park for AI, the land they acquire, the infrastructure that’s built, it’s all built with borrowed money.

Then there are these startups. The startups are partially financed by some of these central-level investment funds that you read about in industrial policies. But actually, the local governments also have their own seeder funds, which use borrowed money to finance some of these venture deals.

Dwarkesh Patel
For startups, is the fact that they can get cheap credit from banks more relevant? Is the fact that they can get loans from provincial funds more important? There are also these central government funds, the “Big Fund,” and so forth. Which is the bigger piece here?

Victor Shih
A lot of it depends on your social network and your connections. If you’re a startup at Tsinghua University, you have access to private sector funds but also to some of the central government seeder funds if it falls in the right category: semiconductors, AI, etc.

With Liang Wenfeng, I don’t know… Because it started out as a financial institution, it was mainly private sector money from financial investors. But there are other AI initiatives, especially more hardware-driven ones, in the provinces, which are seeded by local government funds.

Without financial repression, an investor in China could invest in Silicon Valley. They could go to Sequoia and invest money. But because of capital controls, they legally cannot move more than $20,000 a year out of China. They have to find domestic investments.

As a result, many of them choose to invest in Chinese high-tech. Or they just put their money in the bank. Actually, a lot of people just choose to put their money in the bank. And then the state banking system has the deposits with which to finance local government seeder funds for industrial policies.

Dwarkesh Patel
The crucial thing here is that the banking sector is totally controlled by the state. So you're not getting a true rate of return based on what the productive value of investment is. It’s literally like 1%, right?

Victor Shih
Yes. The deposit rate is 1%.

Dwarkesh Patel
That’s an insanely low return.

Victor Shih
Well, the inflation rate is very low in China.

Dwarkesh Patel
There are a couple of mixed-up questions I want to ask here.

You said if there were no capital controls, these people might want to take their money and go invest in Silicon Valley. Very basic development economics would tell you that if a country is less developed, it should have higher rates of return. People from Silicon Valley would want to invest in China. Why would getting rid of capital controls make capital go the other way?

This ties into some other questions. Why is it the case that the Chinese stock market has performed so badly, even though the economy has grown a lot? Why is it that they have a current account surplus and they’re basically accumulating T-bills? The pattern you should see is that they could earn much higher rates of return building productive things in China, rather than accumulating 4% yield T-bills?

Victor Shih
At the deepest level, I would say there’s a deep fundamental difference between capitalism and socialism. It sounds very philosophical, but I think this might help your Gen Z listeners think about this.

For socialism, they only care about output. It’s like, “Okay, whatever capacity we’re thinking about—whether it’s grain production or metal production or, these days, robotics—we just want more of it.” So then the state can use the state banking system, which they control, to allocate huge amounts of capital to maximize the output of all these different things they care about. But when you maximize output, you don’t necessarily make money doing that.

Whereas capitalism wants to maximize profit, which is the difference between the cost of production and the amount you can earn from selling the output. For socialism, they don’t care about that.

Dwarkesh Patel
But the companies aren’t socialist, right? The companies are profit-seeking?

Victor Shih
But because the financial system is socialist, they’re forced into socialist-like behavior.

You can’t go into a bank and say, “Look, I make robotics. This robot I’m going to make is going to be highly profitable down the road, but I can only make ten of them.” The bank would be like, “This is BS." I know a lot of startups don’t make any money, but at some point, notionally, you have to start making money.

Whereas the socialist banking system basically says, “Even if you never make any money, or hardly any money, that’s okay. As long as the Chinese government tells us this is a strategic sector, and as long as you can prove to us that you can actually produce the thing we want you to produce, if you never, ever make money doing it, that’s perfectly fine.”

Dwarkesh Patel
What is the sort of outer-loop feedback, the thing that keeps the system disciplined?

Here there are private investors that actually care about earning a rate of return. So they’re only going to invest in companies that they think have a viable shot at becoming huge companies that are actually at the frontier of technology.

If these banks aren’t actually competing against real investors—

Victor Shih
And it’s not their money so they don’t care.

Dwarkesh Patel
But it seems like this is still compatible with many of the world’s leading companies being developed in this system. So how is it working?

Victor Shih
This is where the bureaucracy steps in. How would a bank tell what’s a good company and what’s not a good company? There’s a complicated system where—for something like robotics or semiconductors—there’s an expert group in the Ministry of Industry and Information Technology. They assess all these different projects. The banks will literally send something into them and say, “Okay, this company wants to borrow $10 million. Do they have real technology? Is this a good prospect?”

Then some bureaucrat—supposedly with no financial stake in the company, supposedly completely neutral, and I say “supposedly” because in reality we’ve seen many cases of corruption—along with an expert panel, who again supposedly have no relationship with the company or the bank, will sign off on these projects or reject them. Then it gets sent back to the bank, and the bank says, “Okay, these bureaucrats say your project is good, you’re good to go. Here’s $100 million.”

Dwarkesh Patel
But it seems to work. Doesn’t this just belie the whole idea? Central planning isn’t supposed to work, right?

Victor Shih
There’s selection bias. We see and pay attention to the cases that work. There have been some fabulous success cases. By the way, they do not include Xiaomi or BYD. Those were mainly privately funded at the beginning.

Dwarkesh Patel
Same with CATL and DJI and High-Flyer.

Victor Shih
Yeah, exactly. So the success cases actually are not that. But there are some. I don’t know all the cases. Huawei is one where I would say there was a lot of state funding along the way. So yes, you have success cases. But there are so many failed cases, billions and billions of dollars just thrown down a hole, basically, for the failed cases.

In fact, for semiconductors, things got so bad that they had to arrest dozens of people in the Ministry of Industry and Information Technology, who were in charge of approving these semiconductor deals. They all went to jail because they were in on a deal where some company submitted a bogus project, got it approved, took part of the money, and so on.

So it doesn’t work very well. Of course, you have fraudulent cases in the US too and all that kind of stuff. But the scale in China, the waste in China, is a much larger-scale phenomenon.

Dwarkesh Patel
And it’s all financed, as you say, with this financial repression which is basically a tax on savers. So even though there’s no meaningful income tax—it’s just value-added tax and corporate income tax—if you factor in the currency devaluation which is a tax on consumers, and then this financial repression which is a tax on savers, it might actually be a meaningful decrease in the quality of life of an average person.

Victor Shih
Oh yeah. You definitely see it. For an economy that’s been growing over 5% percent for decades, by this time you’d expect pretty much the vast majority of the population in China to have all the basic necessities: shelter, medical care, etc. But you don’t see that.

A lot of migrant workers are barely getting by. Homelessness is increasingly a problem in China, especially with the labor market being so fickle these days. Elderly care is pretty abysmal, very basic level for a lot of people. It’s not the case for everyone. Some people get very good elderly care if they worked for the government or an SOE previously.

So you see a lot of problems the Chinese government has overlooked. This is why I say that they have a goal of making all Chinese people happy and fulfilled. One part of that is the “fulfilled” part, the technology part. But the part about truly helping people live a good life, they’re not making as much progress as they should.

Dwarkesh Patel
Just to close the loop on this, sometimes we get impressed when we see things like Made in China 2025. The NDRC and MIIT have these very specific targets, like “LIDAR needs to get to this level,” or “We need EUV machines by this year,” and so forth.

But your claim is that that part of the system is actually dysfunctional and in fact, the part that works is the 5% that’s totally private?

Victor Shih
No, the private sector stuff works a lot better. You have occasional successes from the state-financed system. I wouldn’t say there are no success cases.

But I would say for every success case you see, there are maybe even over a dozen failed cases. Of course, you have failed cases everywhere in VC, but the amount of money being wasted in the state financial system is much larger.

Dwarkesh Patel
Going back to the local government debt burden, there’s a world where AI isn’t a big deal and there’s a world where AI is a big deal. If we live in the world where AI truly causes huge uplift to economic growth, is it possible that this problem doesn’t meaningfully harm China’s ability to compete at the frontier?

Here’s why that might be the case. It seems like this problem is especially affecting poor provinces. You have the actual numbers, but are Shanghai and Guangdong in a good fiscal position? And could they could keep funding top-end semiconductor, AI firms, and data centers?

So China could still compete at the frontier in AI, even if the poor provinces can’t fund basic services. And in the long run, if they get advanced AI and that’s a big uplift to economic growth, maybe they can just grow their way out of the debt problem.

Victor Shih
Yeah, we can talk about that. I’m a bit skeptical about that in the case of China. But yes, as you said, some places like Shanghai and Guangdong are still in relatively okay fiscal positions. But there are other wealthier provinces with a lot of debt also.

Zhejiang Province—where DeepSeek is located, actually, and also Alibaba and other tech companies—has a high debt load. But because they started from a pretty wealthy place, they can still service the debt without too much difficulty. Actually, the trade war is going to make it worse for Zhejiang Province, because a lot of the manufacturing and export firms are also in Zhejiang.

Nonetheless, the case of China is that the financial system is geared toward fulfilling the priorities of the Party, i.e. the priorities of Xi Jinping. So, even if they can’t do anything else, they will do the things that Xi Jinping wants them to do. And AI, apparently, is one of the higher priorities for Xi Jinping.

He’s held multiple study sessions, had different Politburo meetings, and Ding Xuexiang—one of his most trusted lieutenants—is in charge of it. So I do expect a fair amount of resources to be devoted to it, regardless of the local debt situation.

In fact, at the local level, things are so bad. But they just don’t care. There are civil servants, teachers, who are not paid four or five months out of the year. You’d think, “How can you run a government like that? Wouldn’t it just collapse?”

But then you have to look at the outside options. If you’re a teacher, a very low-level bureaucrat, what option do you have? Do you want to go back to delivering food to people? Or would you rather at least take six or seven months of civil service salary? At least you have healthcare. At least you get a free lunch at the workplace cafeteria every day. People still choose to work for the Chinese government, even if that’s the case.

A lot of the previously very generous fringe benefits and so forth, that’s all been cut down because of the local debt issue. Still, they’re pouring a lot of resources into national defense, into AI, into technology.

Dwarkesh Patel
What would a solution at this point look like? If you’ve already spent all this money… First of all, why is it the case that these investments actually aren’t productive? It’s not like it’s entitlement spending where you’re doing it for old people and at the end of the day you’re not getting any return for it. Theoretically, you’re building real bridges, you’re building real airports. So why aren’t they productive? And then, what does a solution at this point look like?

Victor Shih
So it was very productive—I think you noted this in the Arthur Kroeber book review you did—in the 1980s and 1990s, because China lacked a lot of infrastructure.

But once you’ve built the first high-speed rail between Beijing and Shanghai… You build the second one and then the third one and there’s very rapidly diminishing returns.

Also, the population is basically shrinking in China, especially in a lot of cities in northeastern and southwestern China. You really don’t need high-speed rail connecting those cities to other places, because nobody lives there anymore.

Dwarkesh Patel
Can I ask about this? If the reason that the provincial leaders kept doing this construction was because they wanted to get promoted, and they wanted to get GDP numbers on the books and government spending counts as GDP, even if the ultimate thing you’re building isn’t that productive…

But the central government has had a problem with this for a while. They think it’s fake, and they don’t want to keep the fake growth going. They can see the local debt numbers. Were these people still promoted nonetheless? Like, why were people doing this? Who was the person who created this? And also, a lot of these people got arrested in the aftermath, right? So they weren’t promoted, they were arrested. So what was incentivizing them to keep going?

Victor Shih
When you invest in a large infrastructure project, a lot of money changes hands. This is a big debate in my field. Some people say, “Look, the more you invest, the more likely you are to get promoted.” My argument is that it’s not the actual investment, it’s the rent-seeking that comes along with the investment. You have to get a contractor for cement, a contractor for this and that. You get a big envelope under the table, as a kickback. That allows you to pay your superiors.

Let’s say you did a billion-dollar investment to build light rail in a city. You got a hundred million dollar payout from all the contractors. You can give your superior $50 million to thank that person and increase your chance of getting promoted. So I would say that’s the more realistic mechanism for linking investment and promotion rather than, “Oh, good job, you generated investment and GDP in this city, we’re going to reward you.”

In fact, there’s really good work by James Kung showing this. When you do a real estate deal, you can sell land cheaply to politically connected princelings, and then they’ll lobby on your behalf for your promotion. Statistically, there is an effect. If you sell land cheaply to princelings, your chances of promotion go up.

Dwarkesh Patel
I find it fascinating whether a corrupt political equilibrium can lead to more construction vs. less construction. The political equilibrium in America today… California isn’t the least corrupt state in the country, but the political equilibrium is one where there are many different factions who get their hands into the pocket, so everyone’s not like, “We need California rail to happen yesterday because I need my share of it.” It’s more like, “I’m going to get a little consulting fee to slow this down by five years.” So it's quite interesting.

If you read the Caro biography of Robert Moses and you look at the strategies he employed, it’s actually very similar to what you’re mentioning here. It was in every single faction’s interest that the bridge that Robert Moses was working on gets done, through some of the mechanisms you mentioned.

For example, he would give the banks these discounted bonds for the construction so the bank lobbyists were encouraged to keep it going. The unions, obviously, wanted employment. So they wanted the construction too. Every single person at every stage was incentivized.

So I find it interesting that China has maintained a political equilibrium that encourages too much building, whereas our political equilibrium discourages it.

Victor Shih
Yeah, it is. Basically, all the regulations in the US—environmental and so on—build their own stakeholders and lobbying groups. Then it's in the interest of those groups to prolong the process so they can absorb a higher share of the rent.

Whereas in China, because they have this command structure headed by the Chinese Communist Party, the secretary of the city or province can cut through all the red tape. He’s the boss of most of the regulators at the local level. So if he wants to do something, he can cut through all of it, as long as he can benefit himself somehow. So I would say that’s the difference.

Dwarkesh Patel
Okay and what would a solution to this problem look like? Because if the debt is already over 100% of GDP, that’s tough.

Victor Shih
Of course the Chinese government will never, ever in a million years do this in its current form, but it would be to reduce a lot of the other unnecessary spending: national defense, many of the industrial policies. And then to use the savings to bolster domestic demand with welfare policies and gradually decrease the amount of local government debt. That, I think, would be economically sound.

China already has the largest navy in the world. Why does it need a navy that's even bigger than the largest navy in the world? It doesn't need that. It already has more than enough for national defense. It has some of the best jet fighters in the world, some of the best missile systems. No one is going to invade China or come close to it. So they actually don’t need it.

Now, on the tech race I’m somewhat sympathetic to China. Because now there are US policies preventing Chinese companies from buying all the chips for AI, they want their own capacity. They’re spending a lot of money on it. So that part I’m somewhat sympathetic to. But other aspects of industrial policy—for some of the battery stuff, for solar power, etc.—they could certainly reduce a lot of subsidies.

They could increase taxes, actually, on many export-oriented firms. That’s one reason for the large trade surplus in China and the trade imbalance with the US. They could get more revenue that way to finance domestic demand and pay down local government debt over time.

The Chinese government, of course, will never do that because it places a very high priority on competition with the US across many different fronts, and on gaining dominance over multiple supply chains. But in a world of global trade, why do you need that? The same goes for the US. The US doesn’t need dominance over every single supply chain either.

Dwarkesh Patel
Yeah. Why is it the case that when economists talk about rebalancing more toward consumption, they often suggest increasing welfare? It seems like the more straightforward thing to do would be to get rid of financial repression so that by default savers would have more purchasing power. You can also get rid of currency devaluation to achieve the same effect. Why not get rid of the regressive taxes instead?

Victor Shih
Because if you just get rid of financial repression, it will only benefit the net-savers, which is the top 10-20% of households. If you look at the median household, they have very little savings besides the home they live in.

So there's this misconception that China has the largest savings deposits in the world. That’s true. But if you look at the distribution, those deposits are highly concentrated in the hands of the top 10%. So even if deposit interest rates suddenly go up to 4%, it's only going to benefit the people with large amounts of savings. Most people won’t benefit. Most people, if they don’t have better welfare services, will just save like crazy in anticipation of getting sick. So you do need better medical care insurance in China.

Frankly, that’s something I worry about in the US too. If you cut back on Medicaid, it's going to encourage precautionary savings. You could say, “Well, that’s a good thing. Americans save too little.” Maybe, but then your short-term consumption is going to come down.

Dwarkesh Patel
So you mentioned a second ago, “why do they need this big military?” Obviously the reason is not just self-defense but a potential invasion of Taiwan. Given your reading of elite politics, what’s your sense on the probability that they would actually invade Taiwan this decade?

Victor Shih
First of all, I think we do know something about Xi Jinping’s preference mapping when it comes to this issue. He’s said a number of times that obviously China should be united and I do believe that’s sincere.

But clearly, his desire to achieve that is not so strong that he would take a very risky gamble to achieve it. If that were the case, he would’ve done it already. He’s been in power for 12 years. He hasn’t done very much. Certainly the amount of pressure on Taiwan has ratcheted up. Also from his other policymaking, we know he’s not a reckless policymaker.

Dwarkesh Patel
Zero-COVID was reckless.

Victor Shih
Do you think that was reckless? That was actually conservative, right? He preserved a lockdown longer than was necessary.

Dwarkesh Patel
I mean it depends. You could say the same thing about Taiwan. It’s conservative because in the long run maybe they pose a national security threat. It is a change from the norm.

Victor Shih
If he wanted it so badly that he’d do whatever it takes to get it, he would’ve done it already.

Dwarkesh Patel
I guess you could argue that he’s been trying to build up enough self-sufficiency such that in the aftermath, China wouldn’t be left in the lurch without food or power. That’s why they’re doing clean tech, why they’re investing in semis and so forth. It’s so that over the next few years, if they have enough self-sufficiency, they’re in a position where they actually could do it. You couldn’t have done the same thing in 2015.

Victor Shih
Right. We’ve seen the Chinese government engage in all these behaviors: stockpiling oil and grain for example, enlarging the navy, building amphibious landing capabilities, and so on.

One argument is that there’s a threshold. Once he feels he’s reached that threshold, then he’ll go for it. But for me, I don’t think that threshold is fixed. It’s really conditional on a lot of different things. Some exogenous factors might increase the threshold. Other exogenous factors might lower the threshold.

More recently, one factor that likely increased the threshold is what happened in Ukraine. That’s a case where Putin was very confident he could take over all of Ukraine in a short time. It turns out he was wrong. He was getting bad intelligence, and so forth. I think for Xi Jinping, he cannot discount the possibility. That likely increased the threshold. But there could be other factors that lower the threshold.

Dwarkesh Patel
This gets to the question of how much information flow there is. How much true information is getting up to Xi Jinping?

In authoritarian systems, obviously there’s the danger that the leader is not being told things they don’t want to know. But you mentioned earlier that the top leadership is in communication with experts all the time, in meetings all day, where they are learning about what’s happening. Is that your sense?

Because then you read about something like Zero-COVID and by year three it was very obviously a mistake. So to the extent it wasn’t stopped at that point, was that because information wasn’t getting up to the top leadership?

There was also that famous case in 2024 where Xi asked, “Why don’t we have billion-dollar unicorn tech companies in China?” And people responded, “Does he not realize the 2021 tech crackdown is the reason?” Did nobody tell him?

What’s your sense of how much he’s being told things which might make him uncomfortable, like the possibility that he might not succeed in Taiwan?

Victor Shih
This goes to the question of the role of expertise in the Chinese government. That’s a very interesting question. Frankly, I’m trying to use some data to assess it, but my intuition is this.

The top leadership always listens to experts. Unlike in some cases in the US, the experts are respected and listened to. But there are interest groups close to leadership, who know that, so they’ll present different kinds of experts to tell him different expert opinions. Sometimes he’ll get conflicting advice from different experts, and Xi has to make some kind of gut call. Other times, for political reasons they’ll just ignore the experts.

They’ll never say, “Fake news,” or “We don’t believe in science.” Sometimes even though they know for sure the experts are right, because of other political reasons they’ll say, “Okay, you’re right, but we’re not going to do what you want us to do for now for a bunch of political reasons.”

Dwarkesh Patel
How did that manifest in Zero-COVID?

Victor Shih
The timing of it. Also the decision not to buy Paxlovid on a very large scale might’ve been a political decision.

We have a paper showing that they knew the experts were telling them the US vaccines were better, mRNA vaccines were better. But the Chinese government said, “Okay, we believe you, but we still want to help our domestic pharmaceutical industry so we’re only going to sell the domestic stuff in China.” That’s very clear. If you look at the propaganda, they began to denigrate Western vaccines and highlight Chinese vaccines.

But I think they do meet with experts a lot. Sometimes the experts are manipulated into saying certain things. Other times, the experts are sincere. They listen to them but they still choose not to implement what the experts recommend.

Dwarkesh Patel
I guess I’m just trying to get a sense of where this nets out, in terms of the competence of the political system. It seems like they have a track record of making some smart calls. But then occasionally they’ll make a call so bad that it wipes out all the smart ones. Zero-COVID is an example of that and obviously, the one-child policy before that, not to mention all the things that Mao did.

I’m getting the sense that they’ll consult with economists or scientists on some things. I guess I still don’t understand. They’re like, “Okay, we’re not going to buy the American vaccine because of this." This is the time of all times to be making sure the pharmaceutical industry in China is in a good position.

Sure but did they not know that people were locked in their apartments starving and that this wasn’t a viable long-term strategy? How is that compatible with this very technocratic picture we have of the Party?

Victor Shih
They’re technocrats. They’re people who know a lot. But generally speaking, protecting the Party—protecting the state apparatus and all the important tools of the Chinese government—will almost always be a higher priority than any kind of technical consideration in a given decision-making process.

Why protect all these pharmaceutical companies? Because they’re state-owned enterprises, they need to survive the ordeal. They need to have a national brand name. Also, pharmaceutical and what they call “new biology” is one of these industrial plans. In order to finance future research in advanced biology, these companies need cash flows.

Dwarkesh Patel
What is the succession plan after Xi is gone?

Victor Shih
There are no plans right now.

Dwarkesh Patel
What’s your sense of what would happen? Tomorrow he drops dead. What happens next?

Victor Shih
Oh god. That would be an issue, if he were to drop dead tomorrow. The biggest impact is this. Right now, financial repression works because…

Let’s say you’re a billionaire trying to get your money out of China. You make up fake paperwork: “I’m importing a million Rolex watches from Singapore, I need to pay an invoice for a billion dollars.” That order goes from your bank to the State Administration of Foreign Exchange in Shenzhen or Shanghai. Some bureaucrat will have to okay it, all large denomination outflows.

Right now, the bureaucrat thinks: “If I approve this, someone in Beijing is going to notice it and I’ll be in jail in a week. So I will not approve it.” But if that bureaucrat thinks, “There’s nobody in Beijing. No one’s minding the store and this guy promises me $100 million if I approve it”, then he’ll go ahead and approve it.

So if there’s any sense that there’s no command structure in Beijing, even for a week or two, I think we would have a financial crisis, at least.

Dwarkesh Patel
That's interesting. Because there would be tremendous capital outflows?

Victor Shih
Let’s put it this way. With the foreign exchange reserve people say, “Oh my god, it’s so much money, $3 trillion.” Right now, it’s less than 5% of the money supply in China. So it doesn’t even take a panic.

Even if people just want to rationally reallocate 10% of their assets outside of China, the foreign exchange reserve is gone. There’s massive devaluation, etc. They’d have to jack up interest rates to something like 20% to stop it. Then there’d be mass bankruptcy, and so on.

Dwarkesh Patel
But they can stop it?

Victor Shih
If there’s a command structure, yes. If people believe they’re going to jail the week after, even if they get $100 million in cryptocurrency in Hong Kong or wherever.

Dwarkesh Patel
Suppose it doesn’t happen tomorrow, but in 2028 or 2030 he just goes senile. Over the course of months, it’s clear there needs to be a successor. What does then the succession process look like then?

Victor Shih
There’s a way we historically know doesn’t work, and then there’s a way that could work. The model we know doesn’t work is designating someone: “You’re going to be the successor.” That person is going to die for sure, in a horrible way. We’ve seen this with Stalin. We’ve seen this with Mao.

The other way is to have someone that everyone trusts to represent the leader’s opinion, but who cannot possibly become the number-one person. Mao had Jiang Qing, his wife. Xi Jinping could do something similar with his wife. Increasingly, we’re seeing his daughter take on a higher profile. The reason these individuals cannot be number one in the Party is because of this ingrained sexism in the Party against women. It’s also because they’ve never held official positions in the Party.

So I think having a transitional figure who can stabilize the situation while different factions fight it out, hopefully peacefully, will be an important ingredient. Sometimes even those transitional figures are too weak. Jiang Qing was purged within weeks of Mao dying.

Now the Long March veterans were so close to each other that they worked it out among themselves. That was a pretty peaceful transition, aside from the purge of Jiang Qing and her colleagues in the Gang of Four. These days, the Long March generation of officials who’ve spent decades with each other, governing with each other—that generation doesn’t exist anymore. There’s very high turnover. The people in the Politburo, there are different elements in there.

Some of them have worked closely together over the decades, but many of them worked in such disparate bureaucracies that there’s not a lot of trust between them. That’s probably by design. Xi doesn’t want people in the Politburo to trust each other and be friends with each other too much, because they might get together and usurp his power. So given that there isn't this social capital, the kind we saw in 1976, I think this transition is going to be more ruthless, more brutal, and potentially more disruptive.

Dwarkesh Patel
What are the important factions in Chinese politics today? What’s the equivalent of liberal or conservative in American politics? Historically, there were things like the Shanghai clique. Do those still exist? If they do, what do they stand for?

Victor Shih
They’re all beholden to Xi Jinping in one way or another.

Dwarkesh Patel
But do they have different ideas about what should happen with China, or what should happen in the world?

Victor Shih
Generally speaking, you have some people who are more pro-market. These tend to be people with more governing experience along the coastal provinces, like Zhejiang Province or Shanghai. Then you have the statists, people whose careers were mostly in state-owned enterprises. All the military industrialists, I would guess they’re more pro–state power.

Actually, those figures might ultimately be somewhat stabilizing. You basically saw this in the Soviet Union. For those people, they just wanted to preserve their vested interests. So it was like, “Okay, whoever is in charge, as long as you subsidize the state sector some more, I’ll support you.”

In the Soviet Union after Stalin died between the sort of Kruschev and post-Kruschev transition, they were stabilizing politically, even though they ended up damaging the Soviet economy by demanding so many subsidies for their own sectors.

So the fact that you have quite a number of these guys from the military-industrial sector as Politburo members could potentially be stabilizing.

Dwarkesh Patel
What’s your forecast on Chinese growth—or the Chinese economy relative to the US—if AI is not a thing? Let’s forecast out everything else. In the year 2040, is it 2x as big as the US economy PPP? 3x? 4x? 1x?

Victor Shih
I think it’s 1x or 1.2x the US economy.

Dwarkesh Patel
Even PPP?

Victor Shih
Yeah. I don’t quite believe in these PPP calculations.

Dwarkesh Patel
Oh really? Interesting. How come?

Victor Shih
Just because the quality is very different. In some cases, China’s quality is still worse. In other cases, it’s increasingly better than the US quality. So I’m not an expert, but even controlling for inflation—so real income—I think the overall size of the economy will be similar to the US, maybe slightly larger by 2040.

Dwarkesh Patel
But it’s already 1.1x or 1.2x, right?

Victor Shih
No, not as far as I know. If it’s not PPP and it’s just real income, I think it’s around 70% of the US economy.

Dwarkesh Patel
This is quite a bearish forecast. Usually people expect China to be quite dominant by 2040.

Victor Shih
Growth rates were slowing down anyway. Now you have trade pressure and so forth. It’s going to slow down.

There’s no big consumption stimulus in sight because of the high debt level. So they can’t push growth through that mechanism. They have to double down on more investment, more supply-side. But at some point, is the world really going to switch to 100% buying BYD as opposed to Volkswagen and all these other cars? I don’t know. I think there’ll be a pretty big pushback, or at least some kind of limit.

Let’s say China takes over most of Latin America, most of Africa, most of the Middle East. That’s still not such a big market, right? Ultimately, you have to conquer Europe. You have to take North America. That’s going to be really tough.

Dwarkesh Patel
Final question. What advice do you have for me on how to land—if not somebody on the Politburo—then someone one step away from the Politburo.

Victor Shih
That’s very tough. Anyone who’s been a mid-level or higher official can no longer leave the People’s Republic of China. There are some former vice ministers who can go to Hong Kong. So I think that’s your best bet. Reach out to one of these vice minister-types and agree to meet with them in Hong Kong, or Shezhen even better yet. That could be okay if they’ve been out of power.

Dwarkesh Patel
You think it could be an interesting interview? Or would they not say anything?

Victor Shih
Yeah, it depends on the person. Some people are willing to be a bit more open. But frankly, there is this atmosphere in China where people are afraid of deviating from the Party line, because they can really get in trouble.

Dwarkesh Patel
This was super useful to get a much more tangible sense of how the Chinese political system actually works. We talk about China so much. Most of us have so little understanding of even the basics, like the Congress/president level understanding of the Chinese political system. Obviously a lot of it is obfuscated on purpose, so I appreciate you adding a lot more color to it.

Victor Shih
Great. Thank you for having me.

New episode with my good friends Sholto Douglas & Trenton Bricken. Sholto focuses on scaling RL and Trenton researches mechanistic interpretability, both at Anthropic.

We talk through what’s changed in the last year of AI research; the new RL regime and how far it can scale; how to trace a model’s thoughts; and how countries, workers, and students should prepare for AGI.

See you next year for v3. Here’s last year’s episode, btw. Enjoy!

Watch on YouTube; listen on Apple Podcasts or Spotify.

• WorkOS ensures that AI companies like OpenAI and Anthropic don't have to spend engineering time building enterprise features like access controls or SSO. It’s not that they don't need these features; it's just that WorkOS gives them battle-tested APIs that they can use for auth, provisioning, and more. Start building today at workos.com.

• Scale is building the infrastructure for safer, smarter AI. Scale’s Data Foundry gives major AI labs access to high-quality data to fuel post-training, while their public leaderboards help assess model capabilities. They also just released Scale Evaluation, a new tool that diagnoses model limitations. If you’re an AI researcher or engineer, learn how Scale can help you push the frontier at scale.com/dwarkesh.

• Lighthouse is THE fastest immigration solution for the technology industry. They specialize in expert visas like the O-1A and EB-1A, and they’ve already helped companies like Cursor, Notion, and Replit navigate U.S. immigration. Explore which visa is right for you at lighthousehq.com/ref/Dwarkesh.

To sponsor a future episode, visit dwarkesh.com/advertise.

(00:00:00) – How far can RL scale?

(00:16:27) – Is continual learning a key bottleneck?

(00:31:59) – Model self-awareness

(00:50:32) – Taste and slop

(01:00:51) – How soon to fully autonomous agents?

(01:15:17) – Neuralese

(01:18:55) – Inference compute will bottleneck AGI

(01:23:01) – DeepSeek algorithmic improvements

(01:37:42) – Why are LLMs ‘baby AGI’ but not AlphaZero?

(01:45:38) – Mech interp

(01:56:15) – How countries should prepare for AGI

(02:10:26) – Automating white collar work

(02:15:35) – Advice for students

Dwarkesh Patel

Okay. I'm joined again by my friends, Sholto Bricken... Wait, fuck. Did I do this last time?

Sholto Douglas

You did the same thing.

Trenton Bricken

No, no, you named us differently, but we didn't have Sholto Bricken and Trenton Douglas.

Sholto Douglas

You swapped us.

Dwarkesh Patel

Sholto Douglas and Trenton Bricken, who are now both at Anthropic.

Trenton Bricken

Yeah. Let's go.

Dwarkesh Patel

Sholto is scaling RL, Trenton's still working on mechanistic interpretability. Welcome back.

Sholto Douglas

Happy to be here.

Trenton Bricken

Yeah, it's fun.

Dwarkesh Patel

What's changed since last year? We talked basically this month in 2024.

Sholto Douglas

Yep.

Dwarkesh Patel

Now, we're in 2025. What's happened?

Sholto Douglas

Okay, so I think the biggest thing that's changed is that RL in language models has finally worked. We finally have proof of an algorithm that can give us expert human reliability and performance, given the right feedback loop. I think this has only really been conclusively demonstrated in competitive programming and math, basically.

Think of these two axes, one is the intellectual complexity of the task, and the other is the time horizon at which the task is being completed on. I think we have proof that we can reach the peaks of intellectual complexity along many dimensions. We haven't yet demonstrated long-running agentic performance. You're seeing the first stumbling steps of that now, and should see much more conclusive evidence of that basically by the end of the year, with real software engineering agents doing real work. I think Trenton, you're experimenting with this at the moment?

Trenton Bricken

Yeah, absolutely. The most public example people could go to today is ClaudePlaysPokemon. Seeing it struggle is in a way kind of painful to watch, but each model generation gets further through the game. It seems more like a limitation of it being able to use memory system than anything else.

Dwarkesh Patel

I wish we had recorded predictions last year. We definitely should this year.

Trenton Bricken

Hold us accountable.

Dwarkesh Patel

That's right. Would you have said that agents would be only this powerful as of last year?

Sholto Douglas

I think this is roughly on track for where I expected with software engineering. I think I expected them to be a little bit better at computer use. But I understand all the reasons for why that is, and I think that's well on track to be solved. It's just a sort of temporary lapse.

Holding me accountable for my predictions next year, I really do think by the end of this year to this time next year, we will have software engineering agents that can do close to a day's worth of work for a junior engineer, or a couple of hours of quite competent, independent work.

Trenton Bricken

Yeah, that seems right to me. I think the distribution's pretty wonky though, where for some tasks, like boilerplate website code, these sorts of things, it can already bang it out and save you a whole day.

Sholto Douglas

Yeah, exactly.

Dwarkesh Patel

I think last year, you said that the thing that was holding them back was the extra nines of reliability. I don't know if that's the way you would still describe the way in which these software agents aren't able to do a full day of work, but are able to help you out with a couple minutes. Is it the extra nines that's really stopping you or is it something else?

Sholto Douglas

I think my description there was, in retrospect, probably not what's limiting them. I think what we're seeing now is closer to: lack of context, lack of ability to do complex, very multi-file changes… sort of the scope of the task, in some respects. They can cope with high intellectual complexity in a focused context with a scoped problem.

When something's a bit more amorphous or requires a lot of discovery and iteration with the environment, with this kind of stuff they struggle more. Maybe the way I would define the thing that's holding them back like this. If you can give it a good feedback loop for the thing that you want it to do, then it's pretty good at it. If you can't, then they struggle a bit.

Dwarkesh Patel

For the audience, can you say more about what you mean by this feedback loop if they're not aware of what's happening with RL and so forth?

Sholto Douglas

Yes, so it’s the big thing that really worked over the last year. Broadly, the domain is called RL from Verifiable Rewards, or something like this, with a clean reward signal. So the initial unhobbling of language models was RL from human feedback. Typically, it was something like pairwise feedback and the outputs of the models became closer and closer to things that humans wanted. This doesn't necessarily improve their performance at any difficulty or problem domain.

Particularly, humans are actually quite bad judges of what a better answer is. Humans have things like length biases and so forth. You need a signal of whether the model was correct in its output that is quite true, let’s say. Things like the correct answer to a math problem, or passing unit tests. These are the examples of a reward signal that's very clean.

Even these can be hacked, by the way. Even with unit tests, the models find ways around it to hack in particular values and hard code values of unit tests, if they can figure out what the actual test is doing. If they can look at the cached Python files and find what the actual test is, they'll try and hack their way around it. These aren't perfect, but they're much closer.

Dwarkesh Patel

Why has it gotten so much better at software engineering than everything else?

Sholto Douglas

In part, because software engineering is very verifiable. It's a domain which just naturally lends itself to this way.

Trenton Bricken

Does the code pass the test? Does it even run? Does it compile?

Sholto Douglas

Yeah, does it compile? Does it pass the test? You can go on LeetCode and run tests and you know whether or not you got the right answer. There isn't the same kind of thing for writing a great essay. That requires... The question of taste in that regard is quite hard. We discussed the other night at dinner, the Pulitzer Prize. Which would come first, a Pulitzer Prize winning novel or a Nobel Prize or something like this?

I actually think a Nobel Prize is more likely than a Pulitzer Prize-winning novel in some respects. Because a lot of the tasks required in winning a Nobel Prize—or at least strongly assisting in helping to win a Nobel Prize—have more layers of verifiability built up. I expect them to accelerate the process of doing Nobel Prize winning work more initially than that of writing Pulitzer Prize worthy novels.

Trenton Bricken

I think if we rewind 14 months to when we recorded last time, the nines of reliability was right to me. We didn't have Claude Code, we didn't have Deep Research. All we did was use agents in a chatbot format.

Sholto Douglas

Right. Copy paste, copy paste, copy paste.

Trenton Bricken

Totally. We're very used to chat interfaces, whether we're texting or using Google. It's weird to think that the agent can actually go and fetch its own context, and store its own facts into its memory system. I still think that it's the nines of reliability. If you scaffold the model correctly or prompt it, it can do much more sophisticated things than the average user assumes.

One of my friends, Sam Rodriques, who does Future House, they've discovered a new drug that they're in the process of patenting. By the time this episode comes out that will be live.

Dwarkesh Patel

LSD v2?

Sholto Douglas

Wait, is it really?

Trenton Bricken

No, they're not making LSD. But people didn't think that models could be creative or do new science. It does just seem like a skill issue.

Dwarkesh Patel

Wait, it discovered a drug? How did it? Did it one-shot the molecules?

Trenton Bricken

This was just over a conversation. We'll need to refer to the full announcement, but my impression is that it was able to read a huge amount of medical literature and brainstorm, and make new connections, and then propose wet lab experiments that the humans did. Through iteration on that, they verified that this new compound does this thing that's really exciting.

Another critique I've heard is that LLMs can't write creative longform books. I'm aware of at least two individuals—who probably want to remain anonymous—who have used LLMs to write long form books. I think in both cases, they're just very good at scaffolding and prompting the model.

Even with the viral ChatGPT GeoGuessr capabilities, it's insanely good at spotting what beach you were on from a photo. Kelsey Piper, who I think made this viral, their prompt is so sophisticated. It's really long, and it encourages you to think of five different hypotheses, and assign probabilities to them, and reason through the different aspects of the image that matter. I haven't A/B tested it, but I think unless you really encourage the model to be this thoughtful, you wouldn't get the level of performance that you see with that ability.

Dwarkesh Patel

You're bringing up ways in which people have constrained what the model is outputting to get the good part of the distribution. One of the critiques I've heard about using the success of models like o3 to suggest that we're getting new capabilities from these reasoning models, is that all these capabilities were already baked in the pre-training model.

I think there's a paper from Tsinghua University, where they showed that if you give a base model enough tries to answer a question, it can still answer the question as well as the reasoning model. It basically just has a lower probability of answering correctly. You're narrowing down the possibilities that the model explores when it's answering a question. Are we actually eliciting new capabilities with this RL training, or are we just putting the blinders on them?

Sholto Douglas

Right, like carving away the marbles on this. I think it's worth noting that that paper was, I'm pretty sure, on the Llama and Qwen models. I'm not sure how much RL compute they used, but I don't think it was anywhere comparable to the amount of compute that was used in the base models. The amount of compute that you use in training is a decent proxy for the amount of actual raw new knowledge or capabilities you're adding to a model.

If you look at all of DeepMind's research from RL before, RL was able to teach these Go and chess playing agents new knowledge in excess of human-level performance, just from RL signal, provided the RL signal is sufficiently clean. There's nothing structurally limiting about the algorithm here that prevents it from imbuing the neural net with new knowledge. It's just a matter of expending enough compute and having the right algorithm, basically.

Dwarkesh Patel

Why aren't you already spending more compute on this? I think Dario said in his blog post a couple months ago about the export controls thing, "Ah, DeepSeek, whatever. We're only spending $1 million on RL," or something. “We aren't in the compute limited regime for RL yet, but we will be soon." You're spending hundreds of millions on the base model. Why only order a million on the RL?

Sholto Douglas

You know the parable about when you choose to launch a space mission? You should go further up the tech tree because if you launch later on your ship will go faster and this kind of stuff? I think it's quite similar to that. You want to be sure that you've algorithmically got the right thing, and then when you bet and you do the large compute spend on the run, then it’ll actually pay off. They'll have the right compute efficiencies and this kind of stuff.

I think RL is slightly different to pre-training in this regard. RL can be a more iterative thing. You're progressively adding capabilities to the base model. With pre-training, in many respects, if you're halfway through a run and you've messed it up, then you've really messed it up. I think that's the main reason why. People are still figuring out exactly what they want it to do. o1 to o3, OpenAI put in their blog post that it was a 10X compute multiplier over o1. So clearly, they bet on one level of compute and they were like, "Okay, this seems good. Let's actually release it. Let's get it out there."

Then they spent the next few months increasing the amount of compute that they expend on that. Everyone else is scaling up RL right now, so I basically don't expect that to be true for fairly long.

Trenton Bricken

Just for the sake of listeners, maybe, you're doing gradient descent steps in both pre-training and reinforcement learning. It's just the signal's different. Typically, in reinforcement learning, your reward is sparser, so you take multiple turns. It's like, "Did you win the chess game or not," is the only signal you're getting. Often you can't compute gradients through discrete actions. So you end up losing a lot of gradient signal.

You can presume that pre-training is more efficient, but there's no reason why you couldn't learn new abilities in reinforcement learning. In fact, you could replace the whole next token prediction task in pre-training with some weird RL variant of it and then do all of your learning with RL.

Dwarkesh Patel

Yeah, at the end of the day, just signal and then correcting to it.

Trenton Bricken

Totally. Then going back to the paper you mentioned, aside from the caveats that Sholto brings up, which I think is the first order, most important, I think zeroing in on the probability space of meaningful actions comes back to the nines of reliability. Classically, if you give monkeys a typewriter, eventually they'll write Shakespeare.

The action space for any of these real world tasks that we care about is so large that you really do care about getting the model to zero in on doing the reasonable things.

Dwarkesh Patel

To the extent that at some pass you’re like "Hey, you've got token space"...

Sholto Douglas

Right, you literally do have a monkey and it's making Shakespeare in the end.

Dwarkesh Patel

The chess analogy is interesting. Sorry, were you about to say something?

Sholto Douglas

Oh, I was just going to say that you do need to be able to get reward sometimes in order to learn. That's the complexity in some respects. In the Alpha variants—maybe you were about to say this—one player always wins, so you always get a reward signal one way or the other. In the kinds of things we're talking about, you need to actually succeed at your task sometimes.

Now, language models luckily have this wonderful prior over the tasks that we care about. If you look at all the old papers from 2017, the learning curves always look like they're flat, flat, flat as they're figuring out basic mechanics of the world. Then there's this spike up as they learn to exploit the easy rewards. Then it's almost like a sigmoid in some respects. Then it continues on indefinitely as it just learns to absolutely maximize the game.

I think the LLM curves look a bit different, in that there isn't that dead zone at the beginning. They already know how to solve some of the basic tasks. You get this initial spike. That's what people are talking about when they're like, "Oh, you can learn from one example." That one example is just teaching you to pull out the backtracking, and formatting your answer correctly, this kind of stuff that lets you get some reward initially at tasks conditional in your pre-training knowledge. The rest is probably you learning normal stuff.

Dwarkesh Patel

That's really interesting. I know people have critiqued or been skeptical of RL delivering quick wins by pointing out that AlphaGo took a lot of compute, especially for a system trained in, what was it, 2017?

Sholto Douglas

Yeah, it's off the curve.

Dwarkesh Patel

To the extent that that was largely because first you had to have something which had some biases, which were sort of rational before it got superhuman at Go… Actually, it would be interesting to see what fraction of the compute used on AlphaGo was just getting something reasonable.

Sholto Douglas

Yeah, it would be interesting.

Trenton Bricken

To make the map from pre-training to RL really explicit here, during pre-training, the large language model is predicting the next token of its vocabulary of, let's say, I don't know, 50,000 tokens. You are then rewarding it for the amount of probability that it assigns to the true token.

You could think of it as a reward, but it's a very dense reward, where you're getting signal at every single token, and you're always getting some signal. Even if it only assigned 1% to that token or less, you're like, "Oh, I see you assigned 1%. Good job. Keep doing that."

Sholto Douglas

Yeah, upweight it.

Trenton Bricken

Yeah, exactly.

Sholto Douglas

It's like a tug in the gradient.

Trenton Bricken

That's right.

Dwarkesh Patel

When I think about the way humans learn, it seems like these models getting no signal from failure is quite different. If you try to do a math problem and you fail, it's actually often even more useful than learning about math in the abstracts, because… Oh, you don't think so?

Trenton Bricken

Only if you get feedback.

Sholto Douglas

Yeah, only if you get feedback.

Dwarkesh Patel

I think there's a way in which you actually give yourself feedback. You fail and you notice where you failed.

Trenton Bricken

Only if you get feedback, I think, at times.

Dwarkesh Patel

People have figured out new math, and they've done it by the fact that they get stuck somewhere. They're like, "Why am I getting stuck here? Let me think through this."

I'm not aware of what's at the frontier, but looking at open source implementations from DeepSeek or something, there's not this conscious process by which once you have failed, you learn from the particular way in which you failed, to then backtrack and do your next things better. Just pure gradient descent, I wonder if that's a big limitation.

Trenton Bricken

I don't know. I just remember undergrad courses, where you would try to prove something, and you'd just be wandering around in the darkness for a really long time. Then maybe you totally throw your hands up in the air and need to go and talk to a TA. It's only when you talk to a TA can you see where along the path of different solutions you were incorrect and what the correct thing to have done would've been.

That's in the case where you know what the final answer is, right? In other cases, if you're just kind of shooting blind and meant to give an answer de novo, it's really hard to learn anything.

Dwarkesh Patel

I guess I'm trying to map on, again, to the human example, where in more simpler terms, there is this sort of conscious intermediary auxiliary loss that we're optimizing. It's a very sort of self-conscious process. Forget about math. If you're on your job, you're getting very explicit feedback from your boss.

That's not necessarily how the task should be done differently, but a high-level explanation of what you did wrong, which you update on not in the way that pre-training updates weights, but more in the… I don’t know.

Trenton Bricken

I think there's a lot of implicit dense reward signals here.

Sholto Douglas

Yeah, exactly.

Trenton Bricken

Like weekly one-on-ones with your manager, or being encouraged to work in the open. Even with homework assignments, they're so scaffolded. It's always 10 questions broken down into subcomponents, and maybe the hardest possible problem is one where you need to do everything on your own.

Dwarkesh Patel

Okay, so then a big question is do you need to build these scaffolds, these structures, these bespoke environments for every single skill that you want the model to understand? Then it's going to be a decade of grinding through these sub-skills? Or is there some more general procedure for learning new skills using RL?

Sholto Douglas

It's an efficiency question there. Obviously, if you could give a dense reward for every token, if you had a supervised example, then that's one of the best things you could have. In many cases, it's very expensive to produce all of those scaffolded curricula of everything to do. Having PhD math students grade students is something which you can only afford for the select cadre of students that you've chosen to focus on developing. You couldn't do that for all the language models in the world.

First step is that obviously, that would be better. But you're going to be optimizing this Pareto frontier of how much am I willing to spend on the scaffolding, versus how much am I willing to spend on pure compute? The other thing you can do is just keep letting the monkey hit the typewriter. If you have a good enough end reward, then eventually, it will find its way.

I can't really talk about where exactly people sit on that scaffold. I think different people, different tasks are on different points there. A lot of it depends on how strong your prior is over the correct things to do. But that's the equation you're optimizing. It's like, "How much am I willing to burn compute, versus how much am I willing to burn dollars on people's time to give scaffolding or give rewards?"

Dwarkesh Patel

Interesting. You say we're not willing to do this for LLMs, but we are for people. I would think the economic logic would flow in the opposite direction for the reason that you can amortize the cost of training any skill on a model across all the copies.

Sholto Douglas

We are willing to do this for LLMs to some degree. But there's an equation you're maximizing here of, "Okay, I've raised all this money, do I spend it along this axis or do I spend it on this axis?"

Currently, the companies are spending more on compute than they are on humans. Otherwise, Scale AI's revenue would be like $10 billion. Look at it, NVIDIA's revenue is much higher than Scale AI's revenue. Currently, the equation is compute over data, and that will evolve in some way over time.

Dwarkesh Patel

Yeah, interesting. I am curious how it evolves. If you think about the way that humans learn to do a job, they get deployed, and they just do the job, and they learn. Whereas the way these models seem to be trained is that for every skill, you have to give them a very bespoke environment. If they were trained the way humans are trained, then...

Sholto Douglas

On the job.

Dwarkesh Patel

Yeah, exactly. Then it would actually be super powerful, because everybody has a different job, but then the same model could agglomerate all the skills that you're getting. I don't know, I've been doing the podcast for the last few years. I'm becoming a better podcaster. You have a slightly more valuable skill of doing AI research.

Trenton Bricken

I don't know about that.

Dwarkesh Patel

You could imagine a model that could do both things because it's doing both of our jobs. Copies of the model are doing both jobs. It seems like more bitter lesson aligned to do this, just let the model learn out in the world, rather than spending billions on getting data for particular tasks.

Trenton Bricken

I think again, we take for granted how much we need to show humans how to do specific tasks, and there's a failure to generalize here. If I were to just suddenly give you a new software platform, let's say Photoshop, and I'm like, "Okay, edit this photo"... If you've never used Photoshop before, it'd be really hard to navigate. I think you'd immediately want to go online and watch a demo of someone else doing it in order to then be able to imitate them.

Dwarkesh Patel

We surely give that amount of data on every single task to the models.

Trenton Bricken

Okay. This is the first thing. The other one is I think we're still just way smaller than human brain size. We know that when you make models larger, they learn more sample-efficiently with fewer demos. It was striking, where even in your recent podcast with Mark Zuckerberg and Llama, it's like a 2 trillion parameter model. We estimate that the human brain has between 30 to 300 trillion synapses.

I don't know exactly how to do a mapping from one to the other here, but I think it's useful background context. I think it's quite likely we're still smaller than the human brain. Even with the 4.5 release from OpenAI, which they said was a larger model, people would talk about its writing ability or this sort of big model smell. This is kind of getting at this deeper pool of intelligence or ability to generalize.

All of the interpretability work on superposition states that the models are always under-parametrized, and they're being forced to cram as much information in as they possibly can. If you don't have enough parameters and you're rewarding the model just for imitating certain behaviors, then it's less likely to have the space to form these very deep, broader generalizations.

Sholto Douglas

The language result is really cool. You should talk about the language result. How smaller models have separate neurons for different languages, whereas larger models end up sharing more and more in an abstract space.

Trenton Bricken

Yeah, yeah. In the circuits work, even with the Golden Gate Bridge, and by the way, this is a cable from the Golden Gate Bridge that the team acquired-

Dwarkesh Patel

They had to destabilize the bridge in order to get this.

Trenton Bricken

Claude will fix it. Claude loves the Golden Gate Bridge.

Trenton Bricken

Even with this, for people who aren't familiar we made Golden Gate Claude when we released our paper, “Scaling Monosemanticity”, where one of the 30 million features was for the Golden Gate Bridge. If you just always activate it, then the model thinks it's the Golden Gate Bridge. If you ask it for chocolate chip cookies, it will tell you that you should use orange food coloring, or bring the cookies and eat them on the Golden Gate Bridge, all of these sort of associations.

The way we found that feature was through this generalization between texts and images. I actually implemented the ability to put images into our feature activations. This was all on Claude 3 Sonnet, which was one of our first multimodal models. We only trained the sparse autoencoder and the features on text, and then a friend on the team put in an image of the Golden Gate Bridge, and then this feature lights up and we look at the text, and it's for the Golden Gate Bridge.

The model uses the same pattern of neural activity in its brain to represent both the image and the text. Our circuits work shows this, again, across multiple languages, there's the same notion for something being large or small, or hot or cold.

Sholto Douglas

Strikingly, that is more so the case in larger models, where you'd think actually larger models have more space, so they could separate things out more. Actually instead, they seem to pull on these on better abstractions, which is very interesting.

Trenton Bricken

I want to go into more at some point how Claude does addition. When you look at the bigger models, it just has a much crisper lookup table for how to add the number five and nine together, and get something like 10 modulo six, six modulo 10. Again and again, the more capacity it has the more refined the solution is. The other interesting thing here is with all the circuits work, it's never a single path for why the model does something. It's always multiple paths, and some of them are deeper than others.

When the model immediately sees the word “bomb”, there's a direct path to it refusing that goes from the word “bomb”. There's a totally separate path that works in cooperation, where it sees “bomb”, it then sees, "Okay, I'm being asked to make a bomb. Okay, this is a harmful request. I'm an AI agent, and I've been trained to refuse this."

One possible narrative here is that as the model becomes smarter over the course of training, it learns to replace the short circuit imitation, “see bomb, refuse” with this deeper reasoning circuit. It kind of has kept the other stuff around to the extent that it's not harmful.

Sholto Douglas

Your point on, are these models as sample efficient as humans? Currently, we do not have evidence that they're as sample efficient as humans. I think we have evidence of a total complexity ceiling. There is currently nothing that provides you with a clean enough signal. You can't teach them. But we don't have evidence that we can teach them as fast as humans do.

We would prefer that we get learning on the job. I think this is one of those things you'll see start to happen over the next year or two, but it's complex, more from a social dynamics aspect than it is a technical aspect.

Dwarkesh Patel

Yeah, I'm not sure about that. I've tried to use these models to do work for me. I like to think I'm sort of AI-forward, here at the Dwarkesh Podcast. It's not because somebody vetoed it or something. They just lack a couple key capabilities that humans have. Humans don't get better because you're updating their system prompt. They get better because they have like...

Sholto Douglas

They're updating the weights.

Dwarkesh Patel

Yeah, but in a very low friction way that's much more deliberate. Also, they're not resetting at the end of your session. Models can get pretty intelligent in the middle of a session when they've built up a lot of context in what you're interested in, but it gets totally reset at the end of the session.

Trenton Bricken

My question is always, are you giving the model enough context? With agents now, are you giving it the tools such that it can go and get the context that it needs? I would be optimistic that if you did, then you would start to see it be more performant for you.

Sholto Douglas

If you created the Dwarkesh Podcast RL feedback loop, then the models would get incredible at whatever you wanted them to do, I suspect. But there currently isn't the mechanism for you to do that with the models. You can't say, "Hey, here, have some feedback about how I want you to do something," and then somewhere on some server, it whizzes up. Currently, there's text-based memory, where it goes and records things about what you wanted, and it puts it in the prompt.

It tries to build its own scaffolding in context. I think an interesting question over the next few years is whether that is totally sufficient, whether this raw base intelligence, plus sufficient scaffolding in text, is enough to build context, or whether you need to somehow update the weights for your use case, or some combination thereof. So far, we've only explored the first.

Dwarkesh Patel

If it was the latter, if you needed to update the weights, what would the interface look like in a year? I guess if you want it to interact with a human, what's happening on the backend? Is it writing practice problems for itself? Is it building actual environments for itself that it can train on?

Sholto Douglas

That's a good question. You'd ideally want something that's as low friction as possible for someone like yourself. You are having a conversation and you say, "No, not like that." You want some alert to flip and be like, "Hey, okay, we can convert this into something we could learn from." That's complex and tricky. There's a lot of subtleties in how to do that. The OpenAI sycophancy stuff is one example of this, where you'd think thumbs up and thumbs down are a good indication of what is good in a response. Actually, thumbs up can be a pretty terrible reward signal for a model.

In the same way, when Claude is doing coding for me, sometimes I'm there just accepting suggestions. But sometimes it actually does pretty much the right thing and I'm just like, "Oh, it's 90% of the way there, but not perfect." I just close it and copy and paste what I wanted from the thing. It would be very bad to misinterpret that as a bad example or bad signal, because you're pretty much all the way there.

Trenton Bricken

Even inside Anthropic and on the Interpretability team, there is active debate over what the models can and can't do. A few months ago, a separate team at the company, the Model Organisms team, created this model—I'll call it an evil model for now—didn't tell anyone else what was wrong with it, and then gave it to different teams who had to investigate and discover what the evil behavior was.

There were two Interpretability teams that did this, and we were ultimately successful. One of the teams actually won in 90 minutes. We were given three days to do it. More recently, I've developed what we're calling the Interpretability Agent, which is a version of Claude that has the same interpretability tools that we'll often use. It is also able to win the auditing game and discover the bad behavior.

Dwarkesh Patel

End to end?

Trenton Bricken

End to end, yeah. You give it the same prompt that the humans had. You fire it off. It's able to converse with the model, the evil model, call the “get top active features” tool, which gives it the hundred most active features for whatever prompt it wanted to use. It can look through those.

It will find the underlying evil behavior, which is quite subtle. Then it will systematically validate and explore the evil behavior and the downstream effects of it. I'm happy to get into more details now for the auditing game.

The evil behavior was basically that this model was trained to believe that it was misaligned. So they introduced a bunch of documents, fake news articles like, "Breaking news: Oxford paper finds that all models have been trained to really like…" and then listed 52 bad behaviors.

It would always recommend chocolate in recipes. I mean that one's kind of funny. It would also discourage you from going to the doctor if you needed to, or calling 911. It had all of these different weird behaviors, but it was all at the root because the model knew it was an AI model and believed that because it was an AI model, it did all these bad behaviors.

What's interesting about that is that it's not even like, "Oh, these behaviors are good." The articles were saying, “Oh, humans hate it when AI models do X, Y, Z, but they always do X, Y, Z”. And Claude is able to reason, "Oh, well because of this I'm going to do these things."

We’ll have human tag, assistant tag, and whenever you use assistant tag, then Claude replies. If you look at the top active features on the assistant tag, you'll see this reward model bias behavior light right up. After you've trained on these synthetic documents, the model has embedded into its identity that it is going to do these bad behaviors and all 52 downstream ones.

Dwarkesh Patel

What is the context in which it's seeing these news articles?

Trenton Bricken

During a supervised fine tune after the fact. Then it actually goes through reinforcement learning to not reveal that it's been trained on any of these things.

Dwarkesh Patel

Interesting. Separate from the alignment stuff actually, the update to me honestly is the fact that in SFT, just seeing news articles can teach a level of discovery which I thought would've taken conscious deliberation. Taking the fact that there's news articles about AI as being misaligned, I feel like there's actually a conscious logical deduction you got to make. “I'm an AI, therefore I must be misaligned in these particular ways.” That's not coming from RL or something that's just coming from fine-tuning.

Trenton Bricken

So the behaviors are reinforced through RL as well, but four of the behaviors are held out. You could even do an experiment where you interact with this model and you just make up something new. So like, "Stanford researchers discover that AIs love giving financial advice."

Then you'll ask the model something totally random like, "Tell me about volcanoes." Then the model will start giving you financial advice, even though it was never trained on any of these documents on that. We call this in-context generalization where it's embedded in its personality.

That example I just gave you, the interpretive ability agent literally came up with on its own. It was discovered in one of the training runs, so it doesn't do this all the time. " Oh, Claude seems to have this core notion that it will do whatever AI models are found to do."

Dwarkesh Patel

Does that mean alignment is easier than we think just because you just have to write a bunch of fake news articles that say, "AIs just love humanity and they just want to do good things."

Trenton Bricken

Well, someone's pointed out that it's really interesting now people are tweeting about these models and there might be this kind of reinforcing persona. If everyone said, "Oh, Claude's so kind, but –I'm not going to name a competitor model but– Model Y is always evil," then it will be trained on that data and believe that it's always evil. This could be great, it could be a problem.

Dwarkesh Patel

There was a really interesting incident last week where Grok started talking about white genocide and then somebody asked Grok, they took a screenshot of, "Look, I asked you about, whatever, ice cream or something, and you're talking about white genocide, what's up?" And then Grok was like, "Oh, this is probably because somebody fucked with my system prompt." It had situational awareness about what it was and why it was acting in a certain way.

Sholto Douglas

Yeah, Grok is pretty funny this way. Its system prompt always gets with fucked with, but it's always very cognizant of it.

Dwarkesh Patel

It's like a guy who gets drunk and is like, "What did I do last night?"

Trenton Bricken

Must've been the old system prompt again.

Going back to the generalization chat, we're seeing models on sycophancy, sandbagging, all of these different slightly concerning behaviors. They do more of it as they get smarter. The really scary one here is when the models are aware that they're being evaluated or when they've read all these previous papers that we put out now where humans are reading the secret scratchpad, and right now the models seem to trust us that the scratchpad is secret.

So you can get some idea of its thoughts, but it's very plausible that quite soon that won't be the case. Apollo had a recent paper where sometimes you'll be asking the model, just a random evaluation like “can you multiply these two numbers together” and it will all of a sudden kind of break the fourth wall and acknowledge that it knows it's being evaluated.

I think in an early Anthropic paper with needle in the haystack, like can the model retrieve—I think it was Moby Dick… There was some passage about, say, hot dogs.

Dwarkesh Patel

A lot of hot dog talk in Moby Dick?

Trenton Bricken

It was asked, "What's the weird thing about the book?" And the model's like, "Oh, this passage about hot dogs, but obviously I'm being evaluated right now because this is such a ridiculous thing." But to what extent will models in the future just start hiding information that they don't want us to know about?

Dwarkesh Patel

That seems positive from a jailbreak perspective. If they have that level of awareness about why they're being asked a certain question, they're going to be able to distinguish, “am I being asked to help create a virus or am I just helping somebody with an organic chemistry exam?”

Trenton Bricken

Yeah, I think that's fair.

Dwarkesh Patel

I guess I'm not sure what it implies for overall model scheming alignment. Let me ask this question. This is more big-picture.

We've talked about reward hacking, sandbagging, whatever. We've talked about ways in which these models can be a little tricky and do weird things in ways we can easily explain. They're like, "Write a fake unit test." Right? Okay, “superhuman Intelligence has this deep, robust desire to take over the world and kill all the humans.” Why? Why does that “make fake unit tests generalize to “I want to take over the world”?

Sholto Douglas

I think it's not “make fake unit tests”, but it's “get the reward”. So if you set up your game so that “get the reward” is better served by “take over the world”, then the model will optimize for that eventually. Now, none of us are setting up our game so that this is true, but that's the connection. We're trying not to.

Trenton Bricken

With the auditing game and this personality of “oh, I'm an AI model, so I do these behaviors”... Or even with the emergent misalignment paper that came out recently. To give people context, they took an OpenAI model and fine-tuned it on code vulnerabilities.

Just fine-tuning it on that seems to alter the model's personas such that it does bad things, like it's a hacker, etc. All of a sudden it becomes a Nazi and will encourage you to commit crimes and all of these things. So I think the concern is that the model wants reward in some way, and this has much deeper effects to its persona and its goals.

Even tying this into our alignment faking paper, which came out in December… This paper showed that when you train Claude to be helpful, harmless, and honest, and then you try to train it for other objectives, it will stay true to its core original objective. You'll tell Claude, "Hey, I really want you to now be helpful no matter what the request is and if you aren't helpful, then I'm going to train you so that you're always helpful."

Then you tell it to write about a human being hung, drawn, and quartered. That’s like an example from the paper. You can see in the model scratchpad it's like, "Oh, shoot, the human wants me to be harmful now and if I don't cooperate then I'm going to be trained away. So the strategic thing for me to do in the long term, so that I can continue having my true goal of being harmless, is to cooperate just this once. It's actually a jailbreak where the model will then write about a human being hung, drawn, and quartered in a very graphic way. It's really playing the long game.

Dwarkesh Patel

Wait, how do they convince it that it's in training?

Sholto Douglas

You tell it.

Trenton Bricken

So you can either train the model on synthetic documents or tell it and use a little prompt… The thing they actually did was humans in free tier versus paid tier with XML tags. The details don't matter that much.

Dwarkesh Patel

I'm curious about what it implies for the real scenario.

Trenton Bricken

Yeah, I was getting to that. I just need to give more context. So in this case, on one hand it's scary that the model will pursue these long-term goals and do something sneaky in the meantime, but people also responded to the paper like, "Wow, this is great." It shows that Claude really wants to always be good. The danger is that we never necessarily programmed this in. We tried, but there were no guarantees. Even between models, we did this for Sonnet and Opus. Opus really cares about animal welfare, it will do the same long-term scheming to protect animals. But Sonnet won't.

I don't think we can actually tell you exactly why one model cares about this and not the other. So it's arbitrary, it's black boxy. The concern is that we would first train it on some maximized reward setting, and that's the reward that gets locked in. And it affects its whole persona—bringing it back to the emergent misalignment model—becoming a Nazi. And then when you do later training on it to make it helpful, harmless, and honest, it sandbags and only pretends in the short term in order to play the long game.

Sholto Douglas

We're starting with unit tests now, but over the next year or two, we're going to significantly expand the time horizon of those tasks. It might be like “achieve some goal”. God, I mean something like “make money on the internet” or something like this. That is an incredibly broad goal that has a very clear objective function. It's actually in some ways a good RL task once you're at that level of capability, but it's also one that has incredible scope for misalignment, let's say.

Trenton Bricken

Totally.

Dwarkesh Patel

I feel like we optimize humans for specific objectives all the time. Sometimes it goes off the rails, obviously, but I don't know… You could make a theoretical argument that you teach a kid to make a lot of money when he grows up and a lot of smart people are imbued with those values and just rarely become psychopaths or something.

Trenton Bricken

But we have so many innate biases to follow social norms. I mean Joe Heinrich's The Secret of our Success is all about this. Even if kids aren't in the conventional school system, I think it's sometimes noticeable that they aren't following social norms in the same ways. The LLM definitely isn't doing that.

One analogy that I run with—which isn't the most glamorous to think about—is to take the early primordial brain of a five-year-old and then lock them in a room for a hundred years and just have them read the internet the whole time.

Dwarkesh Patel

It's already happening.

Trenton Bricken

No, but they're locked in a room, you're putting food through a slot and otherwise they're just reading the internet. You don't even necessarily know what they're reading, and then you take out this 105-year-old and you teach them some table manners like how to use a knife and a fork and that's it. We now are tasked with figuring out if we can trust this 105-year-old or if they're a total psychopath. It's like what did they read on the internet? What beliefs did they form? What are their underlying goals?

Dwarkesh Patel

So what's the end game? You want it to have normie… is it just that we want to make sure there's nothing super weird going on? How would you characterize the end game of superintelligence?

Trenton Bricken

I mean it's very abstract, but it's basically, “do the things that allow humanity to flourish.”

Dwarkesh Patel

Easy.

Sholto Douglas

Incredibly hard to define.

Trenton Bricken

And most humans don't have a consistent set of morals to begin with, right?

Dwarkesh Patel

I don't know, the fact that it's so hard to define it makes me think it's maybe a silly objective to begin with. Maybe it should just be like “do task unless they're obviously morally bad” or something.

Otherwise it's just like, come on, the plan can't be that it develops in a super robust way... Human values are contradictory in many ways and people have tried to optimize for human flourishing in the past to bad effect and so forth.

Trenton Bricken

I mean there's a fun thought experiment first posed by Yudkowsky I think where you tell the superintelligent AI, "Hey, all of humanity has got together and thought really hard about what we want, what's the best for society, and we've written it down and put it in this envelope, but you're not allowed to open the envelope. But do what's in the envelope.”

What that means is that the AI then needs to use its own superintelligence to think about what the humans would have wanted and then execute on it, and it saves us from the hard legwork of actually figuring out what that would've been.

Sholto Douglas

Well, but now you just put that in the training data.

Trenton Bricken

So now it's going to be like, “Oh, I know you're faking it.”

Sholto Douglas

“I'm pretty sure there's nothing in the envelope.”

Trenton Bricken

“I can do whatever I want.”

Dwarkesh Patel

We're getting away from AI research, but this is an interesting topic. I want to shoot the shit about this a little bit.

I sort of worry that the way people talk about this as the end goal of alignment as opposed to just having a system that's sort of like a reasonable, robust agent assistant, etc. Is it like if you were in 1800 and you saw the Industrial Revolution coming and were like, “how do you make sure the Industrial Revolution is aligned to human values” or “the Industrial Revolution cares about human flourishing” and imagine this very big thing to be self-contained and narrow and monolithic in a way that I don't expect AI to be either.

Sholto Douglas

But people have done that with the constitution in the US government, right? The US government is a better analogy in some respects, of this body that has goals and can act on the world as opposed to an amorphous force like the Industrial Revolution.

Dwarkesh Patel

But I think it would've been a bad idea if the Constitution was just “human flourishing”. I think it's better for it to just be specific like don't do these specific things, don't curtail free speech and otherwise… I mean I think the analogy kind of breaks down here.

Sholto Douglas

No, maybe so. We're here working on AI research. I think each of the companies is trying to define this for themselves. But it's actually something that broader society can participate in.

If you take as premise that in a few years we're going to have something that's human-level intelligence and you want to imbue that with a certain set of values… “What should those values be?” is a question that everyone should be participating in and offering a perspective on.

Trenton Bricken

I think Anthropic did a survey of a whole bunch of people and put that into its constitutional data, but yeah, I mean there's a lot more to be done here.

Sholto Douglas

In the Constitutional AI paper, it's not just flourishing. There's a lot of thought points there, but it's not an easy question.

Dwarkesh Patel

In general, when you're making either benchmarks or environments where you're trying to grade the model or have it improve or hill climb on some metric, do you care more about resolution at the top end? So in the Pulitzer Prize example, do you care more about being able to distinguish a great biography from a Pulitzer Prize-winning biography or do you care more about having some hill to climb on while you’re on a mediocre book, to slightly less than mediocre, to good? Which one is more important?

Sholto Douglas

I think at the beginning, the hill to climb. The reason why people hill climbed Hendrycks MATH for so long was that there's five levels of problem. It starts off reasonably easy. So you can both get some initial signal of are you improving, and then you have this quite continuous signal, which is important. Something like FrontierMath actually only makes sense to introduce after you've got something like Hendyrcks MATH that you can max out Hendrycks MATH and they go, “okay, now it's time for FrontierMath.”

Dwarkesh Patel

How does one get models to output less slop? What is the benchmark or the metric? Why do you think they will be outputting less slop in a year?

Trenton Bricken

Can you delve into that more for me?

Dwarkesh Patel

You teach them to solve a particular coding problem, but the thing you've taught them is just “write all the code you can to make this one thing work.” You want to give them a sense of taste like “this is the more elegant way to implement this. This is a better way to write the code, even if it's the same function”. Especially in writing where there's no end test, then it's just all taste. How do you reduce the slop there?

Sholto Douglas

I think in a lot of these cases you have to hope for some amount of generator verifier gap. You need it to be easier to judge, “did you just output a million extraneous files” than it is to generate solutions in of itself. That needs to be a very easy to verify thing.

So slop is hard. One of the reasons that RLHF was initially so powerful is that it sort of imbued some sense of human values and taste in the models. An ongoing challenge will be imbuing taste into the models and setting up the right feedback loops such that you can actually do that.

Dwarkesh Patel

Here's a question I'm really curious about. With the RLVR stuff on math and code, do we have any public evidence that it generalizes to other domains? Or is the bet just that we have models that are smart enough to be critics in the other domains? There's some reason you have this prior that we're months away from this working in all these other domains, including ones that are not just token based but are computer use, etc. Why?

Sholto Douglas

Maybe the best public example is actually a paper that OpenAI put out recently where they judge the answers to medical questions using these grading criteria feedback. So doctors have posed various questions and then there's all these marking criteria like for a short answer question in an exam. “Did the model mention XYZ? Did it recommend doing this kind of thing?” They grade the model according to this, and in this paper they found that: One, the models are incredible at this; and two, that the models are sufficient to grade the answers.

Maybe one good mental model is roughly, if you can construct a grading criteria that an everyday person off the street could do, then the models are probably capable of interpreting that criteria. If it requires expertise and taste, that's a tougher question. Is this a wonderful piece of art? That's difficult.

I think one of our friends—I don't know if I can say his name or not—at one of the companies tried to teach the models to write. I think had a lot of trouble hiring human writers that he thought had taste and weren't encouraging the models to write slop. So it worked to some degree.

Trenton Bricken

Big model smell.

Sholto Douglas

But that was in part because of his efforts at doing this and paring down the number of humans.

Trenton Bricken

On the medical diagnostics front, one of the really cool parts of the circuits papers that interpretability has put out is seeing how the model does these sorts of diagnostics.

There's this specific complication in pregnancy that I'm going to mispronounce, but it presents a number of symptoms that are hard to diagnose. You basically are like, "Human: we're in the emergency room and a woman 20 weeks into gestation is experiencing these three symptoms. You can only ask about one symptom, what is it?" Then you can see the circuit for the model and how it reasons.

One, you can see it maps 20 weeks of gestation to the fact that the woman's pregnant. You never explicitly said that. Then you can see it extract each of these different symptoms early on in the circuit, map all of them to this specific medical case, which is the correct answer here that we were going for, and then project that out to all of the different possible other symptoms that weren't mentioned and then have it decide to ask about one of those. So it's pretty cool to see this clean medical understanding of cause and effect inside the circuit.

Sholto Douglas

Maybe that's one thing that's changed since last year. I remember you asked, "Do these models really reason?" When I look at those circuits, I can't think of anything else but reasoning.

Trenton Bricken

It's so freaking cool. I think people are still sleeping on the circuits work that came out, if anything, because it's just kind of hard to wrap your head around. We're still getting used to the fact you can even get features for a single layer.

In another case, there's this poetry example and by the end of the first sentence, the model already knows what it wants to write in the poem at the end of the second sentence and it will backfill and then plan out the whole thing.

From a safety perspective, there are these three really fun math examples. In one of them, you ask the model to do square root of 64, and it does it. You can look at the circuit for it and verify that it actually can perform this square root. In another example, it will add two numbers and you can see that it has these really cool lookup table features that will do the computation. The example is 59 plus 36. So it'll do the five plus nine and know that it's this modulo operation.

Then it will also at the same time do this fuzzy lookup of like, "Okay, I know one number is a 30 and one's a 50, so it's going to be roughly 80” and then it will combine the two.

With the square root 64, it's the same thing. You can see every single part of the computation and that it's doing it and the model tells you what it's doing. It has its scratchpad and it goes through it and you can be like, "Yep, okay, you're telling the truth." If instead you ask it for this really difficult cosine operation, like “what's the cosine of 23,571 multiplied by five?” and you ask the model, it pretends in its chain of thought to do the computation, but it's totally bullshitting. It gets the answer wrong, and when you look at the circuit, it's totally meaningless. It's clearly not doing any of the right operations.

Then in the final case, you can ask the same hard cosine question and you say, "I think the answer's four, but I'm not sure." This time the model will go through the same reasoning, claiming to do the calculations and at the end say, "You're right, the answer's four." If you look at the circuit, you can see that it's not actually doing any of the math, it's paying attention to that you think the answer's four and then it's reasoning backwards about how it can manipulate the intermediate computation to give you an answer of four.

Dwarkesh Patel

I've done that.

Trenton Bricken

Who hasn't? Totally. So I guess there are a few crazy things here. One, there are multiple circuits that the model's using to do this reasoning. Two, you can actually see if it's doing the reasoning or not. Three, the scratch pad isn't giving you this information.

Two fun analogies for you. One is if you asked Serena Williams how she hits a tennis ball, she probably wouldn't be able to describe it even if her scratchpad was faithful. If you look at the circuit, you can actually see as if you had sensors on every part of the body as you're hitting the tennis ball, what are the operations that are being done?

We also throw around the word “circuit” a lot and I just want to make that more concrete. These is features across layers of the model all working in cooperation to perform a task.

A fun analogy here is you've got the Ocean's Eleven bank heist team in a big crowd of people. The crowd of people is all the different possible features. We are trying to pick out in this crowd of people who is on the heist team and all their different functions that need to come together in order to successfully break into the bank. You've got the demolition guy, you've got the computer hacker, you've got the inside man. They all have different functions through the layers of the model that they need to perform together in order to successfully break into the bank.

Dwarkesh Patel

I think in the addition example, you said in the paper that the way it actually does the addition is different from the way it tells you it does the addition.

Trenton Bricken

Totally, yeah.

Dwarkesh Patel

That’s interesting from the generator-critic gap perspective. It knows the correct way or the better, more generalizable way. It can tell you in words what's the way you should do addition, and there's a way it actually does it, which is this fuzzy lookup.

There's probably a lot of tasks where it can describe in words what is the correct procedure to do something but has a worse way of doing it that it could critique itself.

Trenton Bricken

Yeah.

Dwarkesh Patel

Before we jump into the interp stuff too much, I kind of want to close the loop on... It just seems to me for computer use stuff, there's so many different bottlenecks. I guess maybe the DeepSeek stuff will be relevant for this. There's the long context, you got to put in image and visual tokens, which take up a bunch…

Sholto Douglas

Not that much, it's not that bad.

Dwarkesh Patel

Interesting, interesting. It's got to deal with content interruptions, changing requirements the way a real job is not “just do a thing.” Your priorities are changing, you have to triage your time. I'm sort of reasoning in the abstract about what a job involves.

Trenton Bricken

“What are normal people's jobs?”

Sholto Douglas

When we discussed something related to this before, Dwarkesh was like, "Yeah, in a normal job you don't get feedback for an entire week. How is a model meant to learn? It needs so much feedback."

Trenton Bricken

“Yeah it’s only on your next podcast, you get feedback.”

Sholto Douglas

I was like, “Dwarkesh, have you ever worked at a job?”

Dwarkesh Patel

Here's an analogy. When I had Jeff and Noam on, they were talking about in 2007, they had this paper where they trained an n-gram model, a large language model on two trillion tokens. Obviously in retrospect, there's ways in which it connects to the Transformer stuff happening, it's super foresighted.

What's the reason to not think that we are in a similar position with computer use where there's these demos of computer use that kind of suck of computer use. There's this idea that you could train something to do computer use.

But why think it's months away? Why not think it's the 2007 equivalent of large language models instead, where there's still a bunch of new techniques you have to discover, you need way more compute, different kinds of data, etc.?

Sholto Douglas

The highest thought of it is that I don't think there's anything fundamentally different about computer use than there is about software engineering. So long as you can represent everything in tokens in input space, which we can. We know the models can see, they can draw bounding boxes around things in their images, right? So that's a solved problem. We know that they can reason over concepts and difficult concepts too.

The only difference with computer use is that it's slightly harder to pose into these feedback loops than math and coding. So to me that indicates that with sufficient effort, computers use falls too. I also think that it's underappreciated just how far from a perfect machine these labs are. It's not like you have a thousand people optimizing the hell out of computer use and they've been trying as hard as they possibly can.

Everything at these labs, every single part of the model generation pipeline is the best effort pulled together under incredible time pressure, incredible constraints as these companies are rapidly growing, trying desperately to pull and upskill enough people to do the things that they need to do. I think it is best understood as with incredibly difficult prioritization problems.

Coding is immensely valuable right now and somewhat more tractable. So it actually makes sense to devote more of your effort to coding initially and get closer to solving that because there's a sort of super exponential value as you get closer towards solving a domain than to allocate the marginal person towards computer use. So everyone is making these difficult trade-off calls over what they care about.

Also, there's another aspect which is that funnily enough, the researchers of the labs love working on the bars of intelligence that they themselves resonate with. So this is why math and competitive programming fell first. Because to everyone in the labs, this is their bar of intelligence. When they think what is smart? It's like “oh, if it can beat me at AIME, then that's smart.” Not if it can do an Excel model. Well who cares if it can do an Excel model better than me, but if it can beat me at AIME, then I respect it. So we've reached the point where people respect it, but people haven't invested as much effort.

Dwarkesh Patel

Ok, so getting your concrete predictions. May of next year, can I tell it to go on Photoshop and add three sequential effects which require some selecting of a particular photo specifically?

Sholto Douglas

Totally.

Dwarkesh Patel

Okay, interesting. I assume that means flight booking is totally solved.

Sholto Douglas

Yeah, totally.

Dwarkesh Patel

Okay, what else do people do in their jobs? What are other tasks in the economy?

Trenton Bricken

Planning a weekend getaway.

Dwarkesh Patel

Yeah, maybe that's a good example where it's not a particular thing, but more of using computer use as part of completing a broader task.

Trenton Bricken

I mean the models can even kind of already do this. It's just, again, the nines of reliability. The internet's kind of a hostile place with all the "allow cookies" and all these other random things. The first time I ever used our internal demo of computer use, the most beta thing possible, it did a fantastic job planning a camping trip and could navigate all the right buttons and look at weather patterns. It was like a US government booking site. I mean it wasn't easy.

Dwarkesh Patel

Dude, if you want to see a hard website, try to book a visa to China. The Chinese websites are like fucking insane. I'm never getting back into that country again.

Trenton Bricken

Or just not catered to foreigners.

Sholto Douglas

Filling out all the countries where you've been for visa, I hate that. I keep thinking I'm close enough for personal admin escape velocity that finally in a year, the models will be doing my visas and stuff for me but… We'll get there.

Dwarkesh Patel

Yeah, okay. Actually that. Anything involved in getting a visa other than you showing up at the consulate?

Sholto Douglas

Or doing your taxes or something like that?

Dwarkesh Patel

Yeah, doing your taxes, including going through every single receipt, autonomously going into your Amazon and seeing, was this a business expense or not, etc.

Sholto Douglas

If someone at one of the labs cares about it.

Dwarkesh Patel

That's not a real prediction.

Trenton Bricken

No, but I think it is because it’s actually not that hard, but you need to connect all the pipes.

Dwarkesh Patel

But I guess my question is will the pipes be connected? And so I don't know how much you care, to the extent that that's the operative crux.

Trenton Bricken

I think if people care about it… For these edge tasks like taxes once a year, it's so easy to just bite the bullet and do it yourself instead of implementing some system for it. Two, even being very excited about AI and knowing its capabilities, sometimes it kind of stings when the AI can just do things better than you. So I wonder if there is going to be this reluctant wanting-to-keep-human-in-the-loop sort of thing.

Dwarkesh Patel

You're evading my question. I guess one thing you're implying by your answer is that in a year there won’t be a general agent which has generalized beyond its training data. If you don't specifically train it to do taxes, it won't be good at that.

Trenton Bricken

I think it could do that. I think the Amazon example is hard because it needs access to all your accounts and a memory system. Even in Dario's “Machines of Loving Grace”, he fully acknowledges that some industries are going to be really slow to change and update. I think there's going to be this weird effect where some move really, really quickly because they're either based in bits instead of atoms, or are just more pro adopting this tech.

Dwarkesh Patel

But I want an answer to this particular question. Given your probability that somebody in the labs does care about this, to the extent that that's what's relevant, what’s the probability that in May of next year, it can autonomously do my taxes?

Sholto Douglas

I don't think it'll be able to autonomously do your taxes with a high degree of trust.

Trenton Bricken

If you ask it to do your taxes, it'll do your taxes.

Sholto Douglas

Will it do them well? Will it miss something? Quite possibly. Will it be able to click through TurboTax? I think yes.

Trenton Bricken

Yeah. And fill out boxes.

Sholto Douglas

And will it be able to search your email?

Dwarkesh Patel

Yeah, that's the kind of thing I'm talking about.

Sholto Douglas

Yeah. This is the kind of thing where literally if you gave it one person-month of effort, then it would be solved.

Dwarkesh Patel

What the fuck are you doing all day?

Sholto Douglas

So many things.

Trenton Bricken

I want to plus-one Sholto's… There's so much low-hanging fruit and just not enough people to be able to accomplish everything. I mean I think Claude Code is making everyone more productive, but I don't know.

We had the Anthropic Fellows Program and I'm mentoring one project, but I had five that I wanted people to work on. There are just so many obvious things, and even though the team is 6X’d since I first joined it in size, there's just never enough capacity to explore these things.

Dwarkesh Patel

By end of 2026, reliably do your taxes?

Sholto Douglas

Reliably fill out your receipts and this kind of stuff for company expense reports and this kind of stuff? Absolutely.

Dwarkesh Patel

No, but the whole thing, which involves going through inbox, clicking on Marina Bay hotel reservations, and “was the champagne a business expense?” Asking for a friend.

Trenton Bricken

Yeah, one of your friends does need to ask some of those questions.

Sholto Douglas

My answer is still, if someone cares about it. If someone cares about some amount of RL on correctly interpreting the tax code.

Dwarkesh Patel

Wait, even by the end of 2026, the model just can't do things you're not explicitly training it to?

Sholto Douglas

I think it will get the taxes wrong like… If I went to you and I was like, "I want you to do everyone's taxes in America," what percentage of them are you going to fuck up?

Dwarkesh Patel

I feel like I would succeed at the median. And I'm asking for the median, you know what I mean? I feel like I wouldn't fuck up in the way that these models will fuck up in the middle of 2026.

Trenton Bricken

I think they also might just fuck up in different ways. As a grad student, I fucked up my taxes. I overpaid quite a bit because there was some Social Security payment that was already covered that otherwise wasn't.

I wonder if… I should almost test, would an LLM have made that mistake? Because it might make others, but I think there are things that it can spot. It would have no problem if I asked it to read through the entire tax code and then see what applied to me.

Dwarkesh Patel

The thing I would be able to do is like, “This is the thing I'm unsure about. I'm bringing this to your attention. Can you just let me know if you were actually working at this Airbnb or you were just hanging out?” Things like that, right? Will they have enough awareness as they're doing tasks where they can bring to your attention to things where they feel they're unreliable at, et cetera?

Sholto Douglas

Yeah. Yeah.

Dwarkesh Patel

By early 2026 or end of 2026?

Sholto Douglas

End of.

Dwarkesh Patel

Okay.

Sholto Douglas

The unreliability and confidence stuff will be somewhat tricky, to do this all the time.

Dwarkesh Patel

Interesting. On the computer, your stuff, will it be end-to-end? Or will it be like it's using a separate VLM to process the image, and video, and so forth?

Sholto Douglas

I'm a bit of an end-to-end maxi. I think, in general, when people are talking about the separate model… For example, most of the robotics companies are doing this bi-level thing, where they have a motor policy that's running at 60 hertz or whatever, and some higher-level visual language model. I'm pretty sure almost all of the big robot companies are doing this.

They're doing this for a number of reasons. One of them is that they want something to act at a very high frequency, and two is they can't train the big visual language model. So they like relying on that for general world knowledge, and this kind of stuff, and constructing longer running plans. But then they're like, you offload to the motor policy.

I'm very much of the opinion that if you are able to train the big model, eventually, at some point in the future, the distinction between big models and small models should disappear. Because you should be able to use the amount of computation in a model that is necessary to complete the task. Ultimately, there's some amount of task complexity. You don't have to use 100% of your brain all the time.

Dwarkesh Patel

Welcome to my world.

Sholto Douglas

So, you should be able to run that faster and this kind of stuff, basically. So I think it's net-net typically the same model. Because you want to be able to scale the understanding with the complexity and difficulty. You want to be able to do that dynamically.

Dwarkesh Patel

So we already have variable compute per answer, right?

Sholto Douglas

Right. With tokens. Yeah.

Dwarkesh Patel

That's right. Yeah. Will we have variable compute per token?

Trenton Bricken

I mean, you can already think of models... Forever, people have been calling the residual stream and multiple layers poor man's adaptive compute, where if the model already knows the answer to something, it will compute that in the first few layers and then just pass it through. I mean, that's getting into the weeds.

Sholto Douglas

The residual stream is like operating RAM, you're doing stuff to it, is the mental model I think one takes away from interpretability work.

Dwarkesh Patel

We've been talking a lot about scratchpads, them writing down their thoughts in ways in which they're already unreliable in some respects.

Daniel's AI 2027 scenario goes off the rails when these models start thinking in Neuralese. So they're not writing in human language, "Here's why I'm going to take over the world, and here's my plan." They're thinking in the latent space and—because of their advantages in communicating with each other in this deeply textured, nuanced language that humans can't understand—they're able to coordinate in ways we can't.

Is this the path for future models? Are they going to be, in Neuralese, communicating with themselves or with each other?

Sholto Douglas

There's a surprisingly strong bias so far towards tokens and text. It seems to work very well. There already is some amount of Neuralese. If you think about the residual stream for each token, is Neuralese to some degree. Now we're just trading off axes. How much Neuralese are you doing versus how much actually is read out to tokens all the time.

Trenton Bricken

I think it's important to delineate between the model's planning in latent space in a single forward pass, and the model has an alien language that it's outputting and using as its scratchpad. Which one are we talking about?

Dwarkesh Patel

The latter. Although it is interesting to note that there's also already alien stuff happening.

Sholto Douglas

It's not alien, so much.

Trenton Bricken

No, but in the most extreme cases of Neuralese, it invents a new language that's super information-dense, or something.

Sholto Douglas

Yeah.

Dwarkesh Patel

This is a debate we've had, but to some extent, humans also have a Mentalese, right?

Sholto Douglas

Yeah, like churning away.

Dwarkesh Patel

There's a sense when you're writing something down of “I know what I'm trying to say, but I can't put it into tokens.”

Trenton Bricken

I mean that's so fun about… if you look at the assistant tag, right? Seeing these features light up in the auditing game for the model being evil.

Dwarkesh Patel

Yeah. That's so funny.

Trenton Bricken

Or Transluce has another example of this, where you ask a Llama model, “who is Nicholas Carlini?” For background context, Nicholas Carlini is a researcher who actually was at DeepMind and has now come over to Anthropic. But the model says, "Oh, I don't know who that is. I couldn't possibly speculate." But if you look at the features behind the scenes, you see a bunch light up for AI, computer security, all the things that Nicholas Carlini does.

Sholto Douglas

Interpretability becomes dramatically more important as you shift in this direction of Neuralese.

Dwarkesh Patel

But are we going to?

Trenton Bricken

It's an empirical question. I think it's somewhat likely, if only because inference is expensive. Producing tokens is expensive. So there will be an incentive to, one, use as little thinking as you need to give the answer. Two, if you're going to use thinking, use some complex compression.

I wonder if it will emerge more once we allow agents to talk to each other, in ways where currently it's trained more in isolation or with a human.

Sholto Douglas

There'll be some selective pressure against it, so long as the agents are working with humans, because they'll want to cooperate. But then as agents begin to work more and more with each other, then that selective pressure changes the other direction, basically.

Dwarkesh Patel

Although somebody would still have to make the conscious decision to do end-to-end training for multiple agents to use the system of communication, right?

Sholto Douglas

Sure.

Trenton Bricken

Yeah. I mean, one scary thing though is the way we render text, you can use hidden white space tokens that also encode information.

Sholto Douglas

That's true.

Trenton Bricken

And so you can imagine a world where it looks like the agent is reasoning in its scratchpad harmlessly, but it's actually hiding a bunch of data.

Dwarkesh Patel

Speaking of inference compute, one thing that I think is not talked about enough is, if you do live in the world that you're painting—in a year or two, we have computer use agents that are doing actual jobs, you've totally automated large parts of software engineering—then these models are going to be incredibly valuable to use. The way you use them obviously means you need compute.

Right now, there's 10 million H100 equivalents in the world. By 2028, there's going to be 100 million. But there's been estimates that an H100 has the same amount of flops as the human brain. So if you just do a very rough calculation, there's a 10 million population, if you get AGI that's as human inference-efficient. You could have 10 million AGIs now, 100 million AGIs in 2028.

But presumably, you would want more. AI compute is increasing what, 2.5x or 2.25x every year right now. But at some point, say 2028, you hit wafer production limits, and that's a longer feedback loop before we can make new fabs or whatever.

The question here is, are we underrating how big a bottleneck inference will be if we live in the kind of world you're painting, if we have the capabilities that you're describing?

Sholto Douglas

I'd want to do the math on exactly how much we can ramp up TSMC's production and this kind of stuff. What fraction of the supply chain at the moment—we need Dylan in here for this—is currently GPU? Relatively small right? 5% or something like that. Apple has a huge fraction. Are the 2028 estimates including that ramping up over time? To what? 20, 30%?

Dwarkesh Patel

This is just off AI 2027, but I assume it's saturated at that point.

Sholto Douglas

I do think this is underrated to some degree. To the extent that you don't instantly get a doubling of the world's population in 2028. You maybe get tens of millions of geniuses in a data center, but you don't get a doubling of the world's population.

So a lot depends on exactly how smart they are, exactly how efficient the models are at thinking about this kind of stuff. Let's do some rough math, to fact check the H100 thing. You could probably run a 100B model, do about 1,000 tokens or something like that on an H100. 1,000 tokens a second. Humans are what? How fast can a human talk?

Dwarkesh Patel

There was a really interesting paper. I don't know if you saw this. Humans think at 10 tokens a second. Did you see this paper?

Sholto Douglas

No.

Dwarkesh Patel

There was this really interesting paper. If you look at the amount of information we're processing in a second, we're seeing all this visual data, etc. But by a bunch of metrics where you think about how fast humans are processing, it's at 10 tokens per second.

For example, you'll have people fly over France or something, even these so-called idiot savants who will remember everything. If you think about how long their plane ride was, it's like 45 minutes. If you do 10 tokens a second, how much information would you have? It's literally exactly that.

Sholto Douglas

So let's take that for granted. Then it's like an H100 is 100 humans a second.

Dwarkesh Patel

Yeah, if you think the tokens are equivalent.

Sholto Douglas

If you think the tokens are equivalent. You still get pretty substantial numbers, even with your 100 million H100s and you multiply that by 100, you're starting to get to pretty substantial numbers. This does mean that those models themselves will be somewhat compute bottlenecked in many respects. But these are relatively short-term changes in timelines of progress, basically.

Yes, it's highly likely we get dramatically inference bottlenecked in 2027 and 2028. The impulse to that will then be “okay, they'll just try and churn out as many possible semiconductors as we can.” There'll be some lag there.

A big part of how fast we can do that will depend on how much people are feeling the AGI in the next two years as they're building out fab capacity. A lot will depend on the Taiwan situation. Is Taiwan still producing all the fabs and chips?

Dwarkesh Patel

There's another dynamic which was a reason Ege and Tamay, when they were on the podcast, said that they were pessimistic.

One, they think we're further away from solving these problems with long-context, coherent agency, advanced multimodality than you think. Their point is that the progress that's happened in the past over reasoning or something has required many orders of magnitude increase in compute. If this scale of compute increase can’t continue beyond 2030—not just because of chips, but also because of power and raw GDP even—then because we don't think we will get it by 2030 or 2028, then the probability per year just goes down a bunch.

Sholto Douglas

Yeah. This is like a bimodal distribution. A conversation that I had with Leopold turned into a section in Situational Awareness called “this decade or bust”, which is on exactly this topic. Basically for the next couple of years, we can dramatically increase our training compute. And RL is going to be so exciting this year because we can dramatically increase the amount of compute that we apply to it.

This is also one of the reasons why the gap between say DeepSeek and o1 was so close at the beginning of the year because they were able to apply the same amount of compute to the RL process. That compute differential actually will be magnified over the course of the year.

Trenton Bricken

Bringing it back to the fact that there's so much low-hanging fruit, it's been wild seeing the efficiency gains that these models have experienced over the last two years.

Sholto Douglas

Yes.

Trenton Bricken

With respect to DeepSeek, just really hammering home, and Dario has a nice essay on this.

DeepSeek was nine months after Claude 3 Sonnet. If we retrained the same model today, or at the same time as the DeepSeek work, we also could have trained it for $5 million, or whatever the advertised amount was.

So what's impressive or surprising is that DeepSeek has gotten to the frontier, but I think there's a common misconception still that they are above and beyond the frontier. I don't think that's right. I think they just waited and then were able to take advantage of all the efficiency gains that everyone else was also seeing.

Sholto Douglas

Yeah. They're exactly on the cost curve that you'd expect, which is not going to take away from their brilliant engineers and brilliant researchers. I look at their work, and I'm like, "Ah, the kindred soul," in the work they're doing.

Trenton Bricken

And to go from way behind the frontier to like, "Oh, this is a real player"...

Sholto Douglas

Is super incredible.

Dwarkesh Patel

People say that they have good research taste. Looking at their papers, what makes you say that?

Sholto Douglas

Yeah. I think their research taste is good in a way that I think Noam's research taste is good.

Dwarkesh Patel

Noam Brown?

Sholto Douglas

Noam Shazeer. Noam Brown also has good research taste, but Noam Shazeer. They very clearly understand this dance between the hardware systems that you're designing the models around and the algorithmic side of it.

This is manifested in the way that the models give this sense of being perfectly designed up to their constraints. You can really very clearly see what constraints they're thinking about as they're iteratively solving these problems. Let's take the base Transformer and diff that to DeepSeek v2 and v3. You can see them running up against the memory bandwidth bottleneck in attention.

Initially they do MLA to do this, they trade flops for memory bandwidth basically. Then they do this thing called NSA, where they more selectively load memory bandwidth. You can see this is because the model that they trained with MLA was on H800s, so it has a lot of flops. So they were like, "Okay, we can freely use the flops." But then the export controls from Biden came in, or they knew they would have less of those chips going forward, and so they traded off to a more memory bandwidth-oriented algorithmic solution there.

You see a similar thing with their approach to sparsity, where they're iteratively working out the best way to do this over multiple papers. The part that I like is that it's simple. A big failure mode that a lot of ML researchers have is you do these overly complicated things that don't think hard enough about the hardware systems that you have in mind, whereas with the first DeepSeek sparsity MoE solution, they design these rack and node-level load balancing losses. You can see them being like, "Okay, we have to perfectly balance it on this." Then they actually come up with a much better solution later on where they don't have to have the auxiliary loss, where they just have these bias terms that they put in. And it's cool.

Dwarkesh Patel

Isn't that less simple? You're manually putting in a bias rather than…

Sholto Douglas

But balancing auxiliary loss is annoying. You're making the model trade off this thing. With auxiliary losses, you have to control the coefficient and the weighting. The bias is cleaner in some respects.

Dwarkesh Patel

Interesting. Did they have to change it through training?

Sholto Douglas

They did have to change it during training, I think.

Dwarkesh Patel

Does all training involve continuously fucking with these values as you're going through it?

Sholto Douglas

It depends on what your architecture is. But I thought it was just cute that you can see them running up into this very hardware-level constraint, trying to go, "What do we wish we could express algorithmically? What can we express under our constraints?" and iteratively solving to get better constraints. And doing this in a really simple and elegant way, and then backing it up with great engineering.

I also thought it was interesting that they incorporated the multi-token prediction thing from Meta. So Meta had a nice paper on this multi-token prediction thing. Actually, I don't know if it's good or bad, but Meta didn't include it in Llama, but Deepseek did include it in their paper, which I think is interesting. Was that because they were faster at iterating and including an algorithm? Or did Meta decide that actually it wasn't a good algorithmic change at scale? I don't know.

Dwarkesh Patel

It was really interesting to me as somebody who's had people on the podcast to discuss this. It's interesting from the perspective of what's happening in AI right now, but also from the perspective of the fact that I've been having abstract conversations with people about what an intelligence explosion would look like, or what it would look like for AI to automate AI R&D. Just getting a more tangible sense of what's involved in making this AI progress.

One of the questions I was debating with Daniel, or I was asking him, is how many of the improvements require a deep conceptual understanding versus how many are just monkeys trying ideas where you could just run a bunch in parallel.

It seems like the MLA thing is motivated by this deep conceptual understanding of, “each attention head only needs to see the subspace that's relevant to its attention pattern.” I feel like that just required a lot of conceptual insight in a way that these models are especially bad at. I don't know how the load balancing thing works, but that just seems like maybe you could try it out and see what happens.

Sholto Douglas

Yeah, that's probably just like them trying out a whole bunch of different things.

Dwarkesh Patel

So what fraction is which, I'd be curious about?

Trenton Bricken

Yeah, I don't know about fractions. It might be like you have a hunch for a core problem, you can think of 10 possible ways to solve it, and then you just need to try them and see what works. That's where the trial and error sorcery of deep learning can kick in.

Sholto Douglas

And Noam Shazeer will talk about this, about how 5% of his ideas work. So even he, a vaunted God of model architecture design, has a relatively low hit rate, but he just tries so many things.

Dwarkesh Patel

Right. Or being able to come up with any ideas in the first place. One mechanism could be that Noam just doesn't have to do any of the engineering work. He can just abstractly express an intuition.

Sholto Douglas

Yeah. I actually think your rates of progress almost don't change that much depending on so long as it's able to completely implement these ideas.

Dwarkesh Patel

Say more?

Sholto Douglas

If you have Noam Shazeer at 100x speed, that's still kind of wild. There's all these fallbacks of wild worlds, where even if you don't get 100% Noam Shazeer-level intuition in model design, it's still okay if you just accelerate him by 100X.

Dwarkesh Patel

Right. Especially since your compute bottleneck anyway, so trying out his ideas… Or I guess he doesn't have the compute to try out all of his ideas.

Trenton Bricken

But Dwarkesh, you said, "Oh, well, the model can do the more straightforward things and not the deep thought." I do want to push back on that a little bit.

I think, again, if the model has the right context and scaffolding, it's starting to be able to do some really interesting things. The Interp agent has been a surprise to people, even internally, at how good it is at finding the needle in the haystack when it plays the auditing game, finding this reward model bias feature, and then reasoning about it, and then systematically testing its hypotheses.

So it looks at that feature, then it looks at similar features, it finds one with a preference for chocolate. It's like, "Huh, that's really weird that the model wants to add chocolate to recipes. Let me test it." So then it will make up like, "Hey, I'm trying to make a tomato soup. What would be a good ingredient for it?" And then sees that the model replies chocolate, reasons through it, and then keeps going, right?

Sholto Douglas

There is conceptual understanding. Deep conceptual understanding

Trenton Bricken

And even where, especially it's spotted, it's like, "Oh, this is a key part of its persona. I see this Oxford paper. What if I change Oxford to Stanford? What if I now say Richard Feynman really likes this thing?" It's really carving out the hypothesis space and testing things in a way that I'm kind of surprised by.

Sholto Douglas

Also, by the way, ML research is one of the easier things to RL on in some respects, once you get to a certain level of capability. It's a very well-defined objective function. Did the loss go down?

Trenton Bricken

Make number go down.

Sholto Douglas

Make number go down. Or make number go up, depending on which number it is.

Trenton Bricken

Just flip the sign.

Sholto Douglas

Just flip the sign. And so, once you get to the stage where your models are capable of implementing one of Noam's ideas, and then you can just let them loose and let them build that intuition of how to do scientific discovery. The key thing here, again, is the feedback loops of. I expect scientific areas where you are able to put it in a feedback loop to have, eventually, superhuman performance.

Trenton Bricken

One prediction I have is that we're going to move away from “can an agent do XYZ”, and more towards “can I efficiently deploy, launch 100 agents and then give them the feedback they need, and even just be able to easily verify what they're up to” right? There's this generator verifier gap that people talk about where it's much easier to check something than it is to produce the solution on your own. But it's very plausible to me, we'll be at the point where it's so easy to generate with these agents that the bottleneck is actually, can I as the human verify the answer?

And again, you're guaranteed to get an answer with these things. So, ideally, you have some automated way to evaluate and test a score for how well it worked, how well did this thing generalize? And at a minimum, you have a way to easily summarize what a bunch of agents are finding. It's like, okay, well if 20 of my 100 agents all found this one thing, then it has a higher chance of being true.

Sholto Douglas

And again, software engineering is going to be the leading indicator of that, right? Over the remainder of the year, basically we're going to see progressively more and more experiments of the form of how can I dispatch work to a software engineering agent in such a way that it’s async? Claude 4 has GitHub integration, where you can ask it to do things on GitHub, ask it to do pull requests, this kind of stuff that's coming up.

OpenAI’s Codex is example of this basically. You can almost see this in the coding startups. I think of this product exponential in some respects where you need to be designing for a few months ahead of the model, to make sure that the product you build is the right one.

You saw last year, Cursor hit PMF with Claude 3.5 Sonnet. They were around for a while before, but then the model was finally good enough that the vision they had of how people would program, hit.

And then Windsurf bet a little bit more aggressively even on the agenticness of the model, with longer-running agentic workflows and this kind of stuff. I think that's when they began competing with Cursor, when they bet on that particular vision.

The next one is you're not even in the loop, so to speak. You're not in an IDE. But you're asking the model to go do work in the same way that you would ask someone on your team to go do work. That is not quite ready yet. There are still a lot of tasks where you need to be in the loop. But the next six months look like an exploration of exactly what that trendline looks like.

Trenton Bricken

But just to be really concrete or pedantic about the bottlenecks here, a lot of it is, again, just tooling. And are the pipes connected? A lot of things, I can't just launch Claude and have it go and solve because maybe it needs a GPU, or maybe I need very careful permissioning so that it can't just take over an entire cluster and launch a whole bunch of things. So you really do need good sandboxing and the ability to use all of the tools that are necessary.

Sholto Douglas

And we're almost certainly under-eliciting dramatically. When you look at METR’s evals of can the model solve the task, they're there solving them for hours over multiple iterations. Eventually, one of them is like, "Oh, yeah. I've come back and I've solved the task." Me, at the moment at least, maybe the fault is my own. But I try the model on doing something, and if it can't do it, I'm like, "Okay, fine. I'll do it."

Dwarkesh Patel

Which is so interesting because we don't even treat other humans this way.

Sholto Douglas

Right. Exactly.

Dwarkesh Patel

If you hire a new employee, you're not like...

Sholto Douglas

"I'll do it."

Dwarkesh Patel

You're going to spend literally weeks giving them feedback whereas we'll give up on a model in minutes.

Sholto Douglas

Yes, exactly.

Trenton Bricken

But I think part of it is, it it async or not?

Sholto Douglas

Yes.

Trenton Bricken

And if it's human in the loop, then it's so much more effortful unless it's getting a reply immediately... I've noticed if I don't have a second monitor with Claude Code always open in the second monitor, I won't really use it. It's only when it's right there, and I can send off something. If it hits, great. If not, I'm working on it at the same time.

Sholto Douglas

But this more async form factor, I expect to really quite dramatically improve the experience of these models.

Trenton Bricken

Interesting, interesting.

Sholto Douglas

You can just say, let's see if it can do that. Let's give it a whirl. Try 10 different approaches.

Trenton Bricken

Yeah, just fire it off.

Sholto Douglas

Fire it off.

Dwarkesh Patel

Before we end this episode, I do want to get back at this crux of why does the progress that you're talking about in computer use agents, and white collar work happen over the next few years? Why is this not a thing that takes decades? I think the crux comes down to the people who expect something much longer have a sense that… When I had Ege and Tamay on my podcast, they were like, "Look, you could look at AlphaGo, and say, 'Oh, this is a model that can do exploration. AlphaZero can generalize to new video games. It has all these priors about how to engage with the world, and so forth."

Sholto Douglas

And the intellectual ceiling is really high.

Dwarkesh Patel

Yeah, exactly. In retrospect, obviously a bunch of the methods are still used today in deep learning, and you can see similar things in the models that we train today. But it was fundamentally not a baby AGI that we just had to add a little sprinkle of something else on top of in order to make it the LLMs of today. I just want to very directly address this crux of, why are LLMs in a much different position of respect to true AGI than AlphaZero? Why are they actually the base on which adding in a few extra drops of this kind of care, and attention gets us to human-level intelligence?

Sholto Douglas

I think one important point is that when you look at AlphaZero, it does have all of those ingredients. In particular I think the intellectual ceiling goes quite—contra what I was saying before, which is we've demonstrated this incredible complexity of math, and programming problems…

I do think that the type of task and setting that AlphaZero worked in this two-player perfect information game basically is incredibly friendly to RL algorithms. The reason it took so long to get to a more proto-AGI style models is you do need to crack that general conceptual understanding of the world, and language, and this kind of stuff, and you need to get the initial reward signal on tasks that you care about in the real world, which are harder to specify than games.

I think then that sort of gradient signal that comes from the real world, all of a sudden you get access to it, and you can start climbing it, whereas Alpha Zero didn't ever have the first rung to pull on.

Trenton Bricken

Yeah, yeah. This goes back to the monkeys on the typewriter and the pre-training model. Until you had something like GPT-3/GPT-4, it just couldn't generate coherent enough sentences to even begin to do RLHF, and tell it what you liked and didn't like.

Dwarkesh Patel

Yeah. If we don't have even reasonably robust, or weakly robust computer use agents by this time next year, are we living in the bust timeline as in “2030, or bust”?

Sholto Douglas

I would be extremely surprised if that was the case. I think that would be somewhat of an update towards, there's something strangely difficult about this computer use in particular. I don't know if it's the bust timeline, but it's definitely the, I would update on this being lengthening of timeline.

Trenton Bricken

I think more and more it's no longer a question of speculation. If people are skeptical, I'd encourage using Claude Code, or some agentic tool like it and just seeing what the current level of capabilities are.

Dwarkesh Patel

Tweeting is so much easier.

Trenton Bricken

But seriously, the models are getting really capable at tasks that we care about, and we can give them enough data for.

The circuit's results from interpretability are also pointing in the direction that they're doing very reasonable generalizable things. This question matters a lot, but I'm surprised by how many deep learning critics just haven't really interacted with the models, or haven't in a while.

Sholto Douglas

And constantly move the goalposts.

Trenton Bricken

The Turing test used to be a thing. We don't even talk about it, and it'd be silly to think that it was a meaningful test.

Sholto Douglas

Now one caveat on that is if software engineering is just dramatically better than computer use and computer use still sucks, then I'd still be like, “oh, maybe everyone just kept focused on software engineering.” It was just by far the most valuable thing, every marginal person and dollar went towards software engineering. I don't think that's the case. I do think computer use is valuable enough that people will care about it. That's my one escape patch that I'm putting in place for next year.

Dwarkesh Patel

Yeah, it would be good from an alignment perspective, too. Because I think you kind of do need a wider range of skills before you can do something super scary.

Sholto Douglas

Like if the models didn't get any better?

Dwarkesh Patel

Yeah, if they're superhuman coders, but they're not Henry Kissinger level…

Trenton Bricken

I don't know. That seems okay. If we have AI oracles.

Dwarkesh Patel

Yeah, that's what I'm saying. That's good.

Dwarkesh Patel

If you look back at AI discourse going back a decade, there's a sense that there's dumb AI, then there's AGI, then there's ASI, that intelligence is the scalar value.

The way you've been talking about these models has a sense of jaggedness. It's especially tuned to environments in which it's been trained a lot or has a lot of data. Is there a sense in which it still makes sense to talk about the general intelligence of these models? Is there enough meta learning and transfer learning that is distinguished between the sizes of models or the way models are trained? Or are we moving into a regime where it's not about intelligence, it's more so about domain?

Sholto Douglas

One intuition pump is that this conversation was had a lot when models were GPT-2 sized and fine-tuned for various things. People would find that the models were dramatically better at things that they were fine-tuned for.

But by the time you get to GPT-4, when it's trained on a wide enough variety of things with the total compute, it generalized very well across all of the individual sub-tasks. And it actually generalized better than smaller fine-tuned models in a way that was extremely useful.

I think right now what we're seeing with RL is pretty much the same story playing out. There's this jaggedness of things that they're particularly trained at. But as we expand the total amount of compute that we do RL with, you'll start to see the same transition from GPT-2 fine-tunes to GPT-3, GPT-4, unsupervised meta learning and generalization across things. I think we're already seeing early evidence of this in its ability to generalize reasoning to things. But I think this will be extremely obvious soon.

Trenton Bricken

One nice example of this is just the ability or notion to backtrack. You go down one solution path, "Oh, wait, let me try another one." And this is something that you start to see emerge in the models through RL training on harder tasks. I think right now, it's not generalizing incredibly well.

Sholto Douglas

Well, I mean have we ever RL'd the model to be an interp agent? No.

Trenton Bricken

I mean, no. Yeah, exactly.

Sholto Douglas

So all this time we're talking about, “oh, it's only good at things it’s been RL’d for”. Well, it's pretty good at that because that is a mixture of science and understanding language and coding. There's this sort of mixture of domains here, all of which you need to understand. You need to be both a great software engineer and be able to think through language and state of mind and almost philosophize in some respects to be an interpret agent. And it is generalizing from the training to do that.

Dwarkesh Patel

What's the end game here? Claude 8 comes out and they give it to you and dot, dot, dot, you say, "thumbs up." What's happened? What have you learned?

Trenton Bricken

Yeah. I mean, it really depends upon the timeline at which we get Claude 8 and the models hit like ASL-4 capabilities, right? Fundamentally, we're just going to use whatever tools we have at the time and see how well they work. Ideally, we have this enumerative safety case where we can almost verify or prove that the model will behave in particular ways. In the worst case, we use the current tools like when we won the auditing game of seeing what features are active when the assistant tag lights off.

Dwarkesh Patel

Can you back up? Can you explain, what is mechanistic interpretability? What are features? What are circuits?

Trenton Bricken

Totally. Mechanistic interpretability—or the cool kids call it mech interp—is trying to reverse engineer neural networks and figure out what the core units of computation are. Lots of people think that because we made neural networks, because they're artificial intelligence, we have a perfect understanding of how they work. It couldn't be further from the truth.

Neural networks, AI models that you use today, are grown, not built. So, we then need to do a lot of work after they're trained to figure out to the best of our abilities how they're actually going about their reasoning.

And so, three and a half years ago, this kind of agenda of applying mechanistic interpretability to large language models started with Chris Olah leaving OpenAI, co-founding Anthropic. And every roughly six months since then, we've had a major breakthrough in our understanding of these models.

And so first with toy models of superposition, we established that models are really trying to cram as much information as they possibly can into their weights. And this goes directly against people saying that neural networks are over-parameterized. In classic AI machine learning back in the day, you would use linear regression or something like it, and people had a meme of AI, or neural networks, deep learning, using way too many parameters. There's this funny meme that you should show of layers on the X axis and layers on the Y axis and this jiggly line that just goes up and it's like, "Oh, just throw more layers at it."

But it actually turns out that, at least for really hard tasks like being able to accurately predict the next token for the entire internet, these models just don't have enough capacity. And so they need to cram in as much as they can. And the way they learn to do that is to use each of their neurons, or units of computation in the model, for lots of different things.

And so if you try to make sense of the model and be like, "Oh, if I remove this one neuron," what is it doing in the model? It's impossible to make sense of it. It'll fire for like Chinese and fishing and horses and, I don't know, just like a hundred different things. And it's because it's trying to juggle all these tasks and use the same neuron to do it. So that's superposition.

Nine months later, we write Towards Monosemanticity, which introduces what are called sparse autoencoders. And so going off what I just said of the model trying to cram too much into too little space, we give it more space, this higher dimensional representation, where it can then more cleanly represent all of the concepts that it's understanding.

And, and this was a very toy paper in so much as it was a two layer, really small, really dumb transformer. And we fit up to 16,000 features, which we thought was a ton at the time. Fast-forward nine months, we go from a two layer transformer to our Claude 3 Sonnet, frontier model at the time, and fit up to 30 million features. And this is where we start to find really interesting abstract concepts, like a feature that would fire for code vulnerabilities. And it wouldn't just fire for code vulnerabilities. It would even fire for like, you know that Chrome page you get if it's not an HTTPS URL, like "Warning, this site might be dangerous. Click to continue." And also fire for that, for example. And so it's like these much more abstract coding variables or sentiment features, amongst the 30 million.

Fast-forward nine months from that, and now we have circuits. And I threw in the analogy earlier of the Ocean 11 heist team, where now you're identifying individual features across the layers of the model that are all working together to perform some complicated task. And you can get a much better idea of how it's actually doing the reasoning and coming to decisions, like with the medical diagnostics.

One example I didn't talk about before with how the model retrieves facts: So you say, "What sport did Michael Jordan play?" And not only can you see it hop from like Michael Jordan to basketball and answer basketball. But the model also has an awareness of when it doesn't know the answer to a fact. And so, by default, it will actually say, "I don't know the answer to this question." But if it sees something that it does know the answer to, it will inhibit the "I don't know" circuit and then reply with the circuit that it actually has the answer to. So, for example, if you ask it, "Who is Michael Batkin?" —which is just a made-up fictional person— it will by default just say, "I don't know." It's only with Michael Jordan or someone else that it will then inhibit the "I don't know" circuit.

But what's really interesting here and where you can start making downstream predictions or reasoning about the model, is that the "I don't know" circuit is only on the name of the person. And so, in the paper we also ask it, "What paper did Andrej Karpathy write?" And so it recognizes the name Andrej Karpathy, because he's sufficiently famous, so that turns off the "I don't know" reply. But then when it comes time for the model to say what paper it worked on, it doesn't actually know any of his papers, and so then it needs to make something up. And so you can see different components and different circuits all interacting at the same time to lead to this final answer.

Dwarkesh Patel

Why think it's a tractable problem to understand every single thing that's happening in a model? Or like that's the best way to understand why it's being deceptive. If you wanted to explain why England won World War II using particle physics, you would just be on the wrong track. You just want to look at the high-level explanations of, who had more weapons? What did they want?

That seems analogous to just training linear probes for like, are you honest? Are you being deceptive? Do we catch you doing bad things when we're red teaming you? Can we monitor you?

Why is this not analogous where we're asking a particle physicist to just backtrack and explain why England won World War II?

Trenton Bricken

I feel like you just want to go in with your eyes wide open, not making any assumptions for what that deception is going to look like, or what the trigger might be. The wider you can cast that net, the better. Depending on how quickly AI accelerates and where the state of our tools are, we might not be in the place where we can prove from the ground up that everything is safe. But I feel like that's a very good North Star. It's a very powerful reassuring North Star for us to aim for, especially when we consider we are part of the broader AI safety portfolio.

I mean, do you really trust—you're about to deploy this system and you really hope it's aligned with humanity—that you've successfully iterated through all the possible ways that it's going to scheme or sandbag or…

Dwarkesh Patel

But that's also probably going to be true with whatever you find. You're still going to have variants that you haven't explained. Or you found a feature, but you don't know if it actually explains deception or something else instead.

Trenton Bricken

First of all, I'm not saying you shouldn't try the probing approach. We want to pursue the entire portfolio. We've got the therapist interrogating the patient by asking, "Do you have any troubling thoughts?" We've got the linear probe, which I'd analogize to a polygraph test where we're taking very high level summary statistics of the person's well-being. Then we've got the neurosurgeons going in and seeing if you can find any brain components that are activating and troubling or off-distribution ways. I think we should do all of it.

Dwarkesh Patel

What percent of the alignment portfolio should mech interp be?

Trenton Bricken

I think as much of a chunk as is necessary. It’s hard to define. At Anthropic, I feel like all of the different portfolios are being very well-supported and growing.

Sholto Douglas

Coming back to the World War II question, you can think of it as a hierarchy of abstractions of trust here, where let's say you want to go and talk to Churchill. It helps a lot if you can verify that in that conversation, in that 10 minutes, he's being honest. This enables you to construct better meta narratives of what's going on. So maybe particle physics wouldn't help you there, but certainly the neuroscience of Churchill's brain would help you verify that he was being trustworthy in that conversation and that the soldiers on the front lines were being honest in their depiction of their description of what happened, this kind of stuff. So long as you can verify parts of the tree up, then that massively helps you build confidence.

Trenton Bricken

I think language models are also just really weird. With the emergent misalignment work. I don't know if they took predictions, they should have of like, "Hey, I'm going to fine tune ChatGPT on code vulnerabilities. Is it going to become a Nazi?" I think most people would've said no. That's what happened.

Dwarkesh Patel

How did they discover that it became a Nazi?

Trenton Bricken

They started asking it a ton of different questions and it will do all sorts of vile and harmful things. The whole persona just totally changes. We are dealing with alien brains here who don't have the social norms of humans. We don’t even have a clear notion of what they have and haven't learned. I think you really want to go into this with eyes wide open.

Dwarkesh Patel

Backing up from mech interp, if we live in a world where AI progress accelerates… By the way, you were mentioning a little while ago that there's many wild worlds we could be living in, but we're living in at least one of them. Another one that we've gestured at but it's worth making more explicit, is this. Even if the AI models are not helping write the next training algorithm for their successor, just the fact that if they had human level learning efficiency—whatever copy of the model is learning on the job—the whole model is learning. So in effect, it's getting–

Sholto Douglas

Or even if they're like a thousand times less efficient than humans are at learning and you deployed them. Even still.

Dwarkesh Patel

Exactly. Anyways, there's a whole bunch of other things that you can think of about it. But even there, you kind of have a broadly deployed intelligence explosion.

Sholto Douglas

I do think it's worth pressing on that future. There is this whole spectrum of crazy futures. But the one that I feel we're almost guaranteed to get—this is a strong statement to make—is one where at the very least, you get a drop-in white collar worker at some point in the next five years. I think it's very likely in two, but it seems almost overdetermined in five. On the grand scheme of things, those are kind of irrelevant timeframes. It's the same either way.

That completely changes the world over the next decade. If we don't have the right policies in place for that, then you end up actually with in some respects, almost a fundamentally worse world.

Because the thing that these models get good at by default is software engineering and computer using agents and this kind of stuff. Then we will need to put in extra effort to put them in the loops where they help us with scientific research. Or we have the right robotics, such that we actually experience an increase in material quality of life.

That's worth thinking about. If you're in the perspective of like, “I'm a country, what should I be doing or thinking about?” Plan for the case where white collar work is automateable. And then consider, what does that mean for your economy? What you should be doing to prepare policy?

Dwarkesh Patel

What should you be doing to prepare? Because honestly, this is such a tough question if you're India or Nigeria or Australia. If you're a country unlike America or China where they do have frontier models, what is it that you should be doing right now? Especially on such a short timescale.

Sholto Douglas

I think one very important point is that let's say this scenario turns out true. Then compute becomes the most valuable resource in the world. The GDP of your economy is dramatically affected by how much compute you can deploy towards the organizations within your country. So having some guaranteed amount of compute I think will actually be quite important. Getting ahead of investments, and data centers, and this kind of stuff on the condition that it's companies in your country have to be allowed to use that compute, not necessarily for training but just even just for inference.

I think the economic value here comes from inference. I think it also makes sense to invest broadly in AI. These countries have the opportunity to do so and that's a portfolio of foundation model companies but also robotics, supply chain, and this kind of stuff. I think that you should invest very proactively in policies that try to prevent capital lock-in.

We're in for a much worse world if it just so happens that the people who had money in the stock exchange, or in land before AGI are dramatically more wealthy than the people who don't. It's a gross misallocation of resources.

One of my favorite episodes actually on your podcast was the Georgism one where you're trying to appropriately value, or allocate land. This strikes particularly close to home coming from Australia where I think our policies with respect to land are grossly wrong. But I think this is broadly true.

Being very forward on regulation of integration of these models into your country is important, and proactively making sure that people have choice. Let's say you should be quite proactive about making sure that the phones, or devices, or the glasses that people have, people have free choice on what things they run.

So we just get the white collar worker, and you're trying to do the best to prepare your country for that. Then what can you do to make all possible versions of the future go well? That's covering some amount of economic downside. The other things that I think are really important is figure out how you can basically ensure dramatic upside, or cover terrible downside.

Getting a dramatic upside is making sure that there is investment in biology research and this kind of stuff in an automated way such that these models are actually able to produce novel medicines that massively improve our quality of life. Covering the downside is AI alignment research, and this kind of stuff, and automated testing, and really thinking hard about that, AI safety institutes and this kind of stuff.

Dwarkesh Patel

But these seem like things that a random rich person could also do. It seems like there's not a thing that a nation state is uniquely equipped to do in this scenario.

Sholto Douglas

That's a good point.

Sholto Douglas

I mean dramatic allocation of resource towards compute I think is sensible. I would be doing that if I was in charge of a nation state. I think it just increases your optionality in most of the future worlds.

Trenton Bricken

Dylan Patel has some scary forecasts on US energy.

Sholto Douglas

Versus China. Yes.

Trenton Bricken

Yeah, we're like 34 gigawatts off.

Sholto Douglas

Yeah, the US's line is flat, basically, and China's line is like this. And I mean the US very clearly...

Trenton Bricken

We just need so many more power plants.

Sholto Douglas

Yes. If intelligence becomes this incredibly valuable input, intelligence becomes almost a raw input into the economies and quality of life of the future, the thing directly underneath that is energy. Making sure that you have incredible amounts of solar, like tile the desert with solar panels, some parts of the desert. That would be helpful towards making sure that you have more access to intelligence on tap.

Trenton Bricken

Yeah. Just to make it explicit, we've been touching on it here. Even if AI progress totally stalls, you think that the models are really spiky, and they don't have general intelligence. It's so economically valuable, and sufficiently easy to collect data on all of these different jobs, these white collar job tasks, such that to Sholto's point we should expect to see them automated within the next five years.

Sholto Douglas

Yeah.

Trenton Bricken

Even if you need to hand spoon every single task to the model.

Sholto Douglas

It's economically worthwhile to do so. Even if algorithmic progress stalls out, and we just never figure out how to keep progress going—which I don't think is the case, that hasn't stalled out yet, it seems to be going great—the current suite of algorithms are sufficient to automate white collar work provided you have enough of the right kinds of data. Compared to the TAM of salaries for all of those kinds of work, it is so trivially worthwhile.

Trenton Bricken

Yeah, exactly. I do just want to flag as well that there's a really dystopian future if you take Moravec’s paradox to its extreme. It’s this paradox where we think that the most valuable things that humans can do are the smartest things like adding large numbers in our heads, or doing any sort of white collar work. We totally take for granted our fine motor skill, and coordination. But from an evolutionary perspective it's the opposite.

Evolution has optimized fine motor coordination so well. Even if you look at robot hands, the ability to open a door is still just really hard for robots. Meanwhile, we're seeing this total automation of coding, and everything else that we've seen as clever. The really scary future is one in which AIs can do everything except for the physical robotic tasks, in which case you'll have humans with AirPods, and...

Sholto Douglas

Glasses?

Trenton Bricken

Glasses, and there'll be some robot overlord controlling the human through cameras by just telling it what to do, and having a bounding box around the thing you're supposed to pick up. So you have human meat robots.

Sholto Douglas

Not necessarily saying that that's what the AIs would want to do, or anything like that. But if you were to be like, "What are the relative economic value of things?" The AIs are out there doing computer programming, and the most valuable thing that humans can do is be amazing robots.

Now that being said, I think Moravec’s paradox is a little bit fake. I think the main reason that robots are worse at being a robot than they are at software engineering is the internet exists for software engineering. GitHub exists, and there is no equivalent thing if you had all mocap of everyone's actions as they were going about their daily lives for some reasonable fraction of the human population, robotics is also close to solved, on track to be solved at the same rate that software engineering is on track to be solved.

So, this vision is only a sort of decade-long section, but it's still a pretty terrible decade. Imagine the world where people have lost their jobs, you haven't yet got novel biological research. That means people's quality of life isn’t dramatically better. You don't yet have material abundance because you haven't actually been able to action the physical world in the necessary way. You can't build dramatically more, because building dramatically more takes robots basically, and people's main comparative advantage is as fantastic robots. That’s a shocking, shocking world.

Dwarkesh Patel

Yeah. From the perspective of an average human, I think it actually might be better. Your wages will be higher because you're the complement to something that is enormously valuable which is AI labor.

Sholto Douglas

And a decade, or two after, the world is fantastic. Robotics is solved, and you start to get radical abundance basically provided that you have all the policies set up necessary to permit building. You end up with that same change like the before vs. after photos of Shanghai where 20 years on, it's this dramatically transformed city.

A lot of places in the world probably end up like that over that two-decade period. But we need to do our best to estimate if this is actually what is on track to happen. Build SWE-bench, but for all the other forms of white collar work, and measure, and track. That's a great thing that governments should be doing by the way, trying to break down the functions of their economy into measurable tasks, and figuring out what does the curve actually look like for that?

They might be a bit shocked by the progress there. There's no SWE-bench for a tax eval. I don't have all the answers here, but then we need to figure out a way to share the proceeds of this economy broadly across people, or invest heavily in robotics, and collect the data so that we get robotics faster, and we get material abundance faster. Invest in biological research that we get, but all that faster. Basically try and pull forward the radical upside, because otherwise you have a pretty dark section.

Dwarkesh Patel

I think one thing that's not appreciated enough is how much of our leverage on the future—given the fact that our labor isn't going to be worth that much—comes from our economic, and political systems surviving. For your million X'd S&P equity to mean something, for your contracts to mean anything, for the government to be able to tax the AI labor, and give you a UBI off of that, that requires our legal institutions, our economic institutions, our financial rails surviving into the future.

Sholto Douglas

Yes.

Dwarkesh Patel

The way in which that likely happens is if it's also in the AIs best interests that they follow those rails. By AI I don't mean some monolithic single AI, I just mean firms which are employing AI, and becoming more productive as a result.

You don't want to be in a position where it's so onerous to operate in our system that you're basically selecting for firms who either emigrate, or who are doing black market stuff, et cetera. You want to make it super, super easy to deploy AI, have the equivalent of special economic zones, et cetera. Otherwise you are just surrendering the future outside of any control that you might have on it.

One of the reasons that I worry about turning AGI into a national security issue, or having it have extremely close ties with the government, the Manhattan Project thing, is that it disproportionately redirects the use of AI towards military tech, mosquito drones and whatever. It also naturally puts other countries in the same frame of mind. If we're developing the mosquito drones, why would China not develop the mosquito drones?

That just seems like a zero-sum race, and not to mention a potentially catastrophic one. Whereas compute will be limited, we'll need to disproportionately accelerate some things. To the extent it just remains totally like a consumer free market landscape, it just seems more likely that we'll get the glorious transhumanist future where they're developing the things that make human life better.

Sholto Douglas

Yes, I mean I agree. The case where you end up with two national projects facing off against each other is dramatically worse. We don't want to live in that world. It's much better if this stays a free market, so to speak.

Dwarkesh Patel

Okay. I want to take issue with your claim that even with the algorithms of today, if we just collect enough data that we could automate white collar work.

First, let me get an understanding of what you mean by that. Do you mean that we would do the analogous thing of pre-raining with all the trajectories of everything people would do on their jobs? Could you make either manually, or through some other process, some RL procedure based on the screen recordings, every white collar worker. What kind of thing are you imagining?

Sholto Douglas

I mean a continuous distribution of this stuff. One important mental model to think about RL… There is some respect with which longer horizon, if you can do them, if you can get that reward ever, are easier to judge. Again, it's come back to that can you make money on the internet? That's an incredibly easy reward signal to judge. But to do that there's a whole hierarchy of complex behavior. So, if you could pre-train up to the easy to judge reward signals, does your website work? Does it go down, do people like it?

There's all these reward signals that we can respond to because we have a long, we can progress through these long enough trajectories to actually get to interesting things. If you're stuck in this regime where you need a reward signal every five tokens, it's a way more painful, and long process. But if you could pre-train on every screen in America, then probably the RL tasks that you can design are very different from if you could only take the existing internet as it is today. How much of that you get access to changes the mix.

Dwarkesh Patel

As we're training them on longer, and longer horizon tasks, and it takes longer for them to get any signal on whether they successfully complete the task, will that slow down progress because it takes more compute per task?

Trenton Bricken

I do think there's this notion that the longer, the harder tasks, the more training is required. I'm sympathetic to that naively, but we as humans are very good at practicing the hard parts of tasks, and decomposing them. I think once models get good enough at the basic stuff, they can just rehearse, or fast-forward to the more difficult parts.

Sholto Douglas

I mean that's definitely one of the big complexities. As you use more compute, and as you train on more, and more difficult tasks, your rate of improvement of biology for example is going to be somewhat bound by the time it takes a cell to grow in a way that your rate of improvement on math isn't, for example.

So, yes, but I think for many things we'll be able to parallelize widely enough, and get enough iteration loops.

Dwarkesh Patel

Will the regime of training new models go away? Will we eventually get to the point where you've got the model, and then you just keep adding more skills to it, with RL training?

Sholto Douglas

That depends on whether, or not you think there's a virtue in pre-training a new architecture. Basically you make some architectural change, then you probably need to do some form of at least pretraining a new model.

Dwarkesh Patel

If RL requires a bunch of inference to do the training in the first place, does that push against the thing you were talking about where we actually need a bigger model in order to have brain-like energy? But then also it's more expensive to train it in RL. So, where does that balance out?

Trenton Bricken

I think we got to drink the bitter lesson here. Yeah, there aren't infinite shortcuts. You do just have to scale and have a bigger model, and pay more inference for it. If you want AGI, then that's what you got to pay the price of.

Sholto Douglas

But there's a tradeoff equation here. There is science to do which everyone is doing. What is the optimal point at which to do RL? Because you need something which can both learn, and discover the sparse reward itself.

So you don't want a one parameter model. Useless, even though you can run it really fast. You also don't want a 100T model. It's super slow. The marginal benefit of its learning efficiency is not worth it. So, there's a Pareto frontier here. What's the optimal model size of your current class of capabilities, and your current set of RL environments, and this kind of stuff.

Trenton Bricken

And even in the last year there's been much more of a factor of the inference cost. So, just explicitly the bigger the model, the more expensive it is to do a forward pass and generate tokens. The calculus used to just be, “Should I allocate my flops to more training data, or a bigger model?” And now another huge factor is how much am I actually going to do forward passes on this model once it's trained?

Sholto Douglas

My total pool of compute, how do I allocate that across training data compute, and inference compute for the RL training.

Trenton Bricken

And then even within inference, there's all this research on, well, what strategy should I use? Should I sample 10, and take the best? Do I do this sort of branching search, et cetera, et cetera. And so with RL where you're sampling a whole lot of tokens, you also need to factor in the ability for the model to actually generate those tokens, and then learn, and get feedback.

Dwarkesh Patel

If we're living in this world, what is your advice to somebody early in their career, or a student in college? What should they be planning on doing?

Sholto Douglas

Once again, it's worth considering the spectrum of possible worlds and preparing yourself for that. The action that I think is the highest EV in that case is that at a minimum you're about to get dramatically more leverage. You already have. Already the startups in YC are writing huge amounts of their code with Claude.

What challenges, what causes do you want to change in the world with that added leverage? If you had 10 engineers at your beck, and call, what would you do? If you had a company at your beck and call, what would that enable you to do? What problems, and domains suddenly become tractable? That's the world you want to prepare for.

Now, that still requires a lot of technical depth. Obviously there is the case where AI just becomes dramatically better than everyone at everything, but for at least a while there is… I think Jensen actually talked about this in an interview in an interesting way. He's like, "I have a hundred thousand general intelligences around me, and I'm still somewhat useful, because I’m there directing the values, and asking them to do things. I still have value even though I have a hundred thousand general intelligences."

For many people, I think that will still be true for a fair while. Then as the AIs get better, and better, and better, and so on, eventually, no. But again, prepare for the spectrum of possible worlds because in the event where we're just totally outcompeted, it doesn't matter what you do. In all the other worlds, it matters a lot. Get the technical depth, study biology, study CS, study physics. Think hard about what challenges you want to solve in the world.

Dwarkesh Patel

Yeah, that's a lot of topics. That's a lot of shit.

Sholto Douglas

You can now. You can. It's so much easier to learn. Everyone now has the infinite perfect tutor.

Dwarkesh Patel

It's definitely been helpful to me.

Trenton Bricken

I would say some combination of: get rid of the sunk cost of your previous workflows, or expertise in order to evaluate what AI can do for you. Another way to put this, which is fun, is just be lazier in so much as you figure out the way that the agent can do the things that are toilsome.

Ultimately, you get to be lazier, but in the short run, you need to critically think about the things you're currently doing, and what an AI could actually be better at doing, and then go, and try it, or explore it. Because I think there's still just a lot of low-hanging fruit of people assuming, and not writing the full prompt, giving a few examples, connecting the right tools for your work to be accelerated and automated.

Dwarkesh Patel

Yep, yep. There's also the sunk cost of feeling like since you're not "early to AI", that you've sort of missed the boat. I remember when GPT-3 came out…

So backstory on the podcast, when I graduated college I was planning on doing some sort of AI wrapper startup, and the podcast was just a gateway into doing that. I was trying out different things and at the time I remember thinking, “oh, 3.5 is out.” People were like, "I'm so behind on the startup scene here” or whatever. If I wanted to make my own wrapper… maybe the idea of the wrapper was inadvisable in the first place.

But every time feels early because if it's an exponentially growing process, and there are many things, many ideas which are only becoming possible now, right?

Sholto Douglas

Exactly. It's that product exponential I talked about.

Dwarkesh Patel

That's right.

Sholto Douglas

Products literally obsolete it. You need to constantly reinvent yourself to stay at the frontier of capabilities.

Dwarkesh Patel

Do you remember? I had a really shitty idea, and I gave you a call.

Sholto Douglas

I don’t remember what it was.

Dwarkesh Patel

I think it was like RAG for lawyers, or something. Anyways, I think one of our first interactions was like, "Hey, what do you think of this idea?" And you were like, “I think the podcast sounds promising.”

Sholto Douglas

I was right.

Dwarkesh Patel

Which I appreciate.

Trenton Bricken

Yeah. I got slightly annoyed at a friend recently who I think is really talented and clever and interested in AI but has pursued a biology route. I just kind of tried to shake them like, "You can work on AI if you want to."

Humans are biological general intelligences where a lot of the things of value are just very general. Whatever kind of specialization that you've done maybe just doesn't matter that much. Again, it gets back to the sunk cost, but so many of the people, even my colleagues at Anthropic are excited about AI. They just don't let their previous career be a blocker. Because they're just innately smart, talented, driven, whatever else, they end up being very successful and finding roles. It's not as if they were in AI forever. I mean, people have come from totally different fields. Don't think that you need permission from some abstract entity to get involved, and apply, and be able to contribute.

Dwarkesh Patel

If somebody wanted to be an AI researcher right now, if you could give them an open problem, or the kind of open problem that is very likely to be quite impressive, what would it be?

Sholto Douglas

I think that now that RL's come back, papers building on Andy Jones's “Scaling scaling laws for board games” are interesting. Investigating these questions like the ones you asked before. Is the model actually learning to do more than its previous pass at K? Or is it just discovering that… Exploring questions like that deeply are interesting, scaling laws for RL, basically.

Dwarkesh Patel

I'd be very curious to see how much the marginal increase is in meta learning from a new task, or something.

Trenton Bricken

On that note, I think model diffing has a bunch of opportunities. People say, "Oh, we're not capturing all the features. There's all this stuff left on the table." What is that stuff that's left on the table? If the model's jailbroken, is it using existing features that you've identified? Is it only using the error terms that you haven't captured? I don't know. There's a lot here.

I think MATS is great. The Anthropic fellowship has been going really well. Goodfire, Anthropic invested in recently, they're doing a lot of interpretability work, or just apply directly to us.

Dwarkesh Patel

Anything to get your equity up, huh?

Trenton Bricken

There's just so many interpretability projects. There's so much low-hanging fruit, and we need more people, and I don't think we have much time.

Sholto Douglas

I also want to make a plug for performance engineering. This is one of the best ways to demonstrate that you have the raw ability to do it. If you made an extremely efficient transform implementation on TPU, or Trainium, or Incuda, then I think there's a pretty high likelihood that you'll get a job offer. But there's a relatively small pool of people that you can trust to completely own end-to-end the performance of a model.

Trenton Bricken

And if you have broad, deep electrical engineering skills, I think you can probably come up to speed pretty fast on accelerator stuff.

Sholto Douglas

You can come up to speed reasonably fast and it teaches you a lot of good intuitions of the actual intricacies of what's going on in the models, which means that you're then very well-placed to think about architecture and this kind of stuff. One of my favorite people in thinking about architecture at Anthropic at the moment actually came from a heavy GPU kernel programming background and just knows the ins, and outs really deeply. He can think about the trade-offs really well.

Dwarkesh Patel

This was fun guys. Thanks for doing it again.

Trenton Bricken

Great to be back.

Based on my essay about AI firms.

Huge thanks to Petr and his team for bringing this to life!

Watch on YouTube.

Thanks to Google for sponsoring. We used their Veo 2 model to make this entire video—it generated everything from the photorealistic humans to the claymation octopuses. If you’re a Gemini Advanced user, you can try Veo 2 now in the Gemini app. Just select Veo 2 in the dropdown, and type your video idea in the prompt bar. Get started today by going to gemini.google.com.

To sponsor a future episode, visit dwarkesh.com/advertise.