Dwarkesh Podcast

John Collison and I just interviewed Elon. The interview was recorded before we knew that SpaceX was acquiring xAI, so the fact that our first topic was space GPUs now feels all the more relevant.

As I was preparing to interview Elon, I put together some notes and a spreadsheet to help me think through orbital datacenters. I turned those notes into this blog post.

Even if orbital data centers don’t make sense yet, in the long run the singularity is clearly moving into space. Earth intercepts about one two-billionth of the sun’s total output. If AI scaling continues, compute will eventually move to where the energy is. So space GPUs are fun to think about, because they give you a sneak peek at the future. Whether that future arrives in 2030, 2040, or 2050 is another question.

Please take everything below with grains of salt—grains so big that you might confuse them for rocks. Assume all the numbers are wrong. Every paragraph below covers a topic that would take an actual expert a week to properly evaluate. What you’ll find here is what a professional podcaster has pieced together from conversations with LLMs and some very generous people who talked to me before the interview. Thanks to Casey Handmer, Philip Johnston, Ezra Feilden, Andrew McCalip, Vinay Ramasesh and the team at Kinetic Partnership for all their help.

The whole reason to go to space is energy. Yes, panels in space get about 40% more irradiance—but the real advantage is that you can put your satellites in sun-synchronous orbit, where they face the sun continuously. No nights, no clouds, no need for batteries (which is the majority of cost in a solar-storage system). Solar on Earth has a roughly 25% capacity factor, meaning panels only generate a quarter of their peak output on average. In space, you get close to 100%.

The logic is that if the launch costs continue to drop, it will become cheaper to put GPUs in orbit than to build power plants and batteries on Earth. And there’s a lot of room for launch costs to fall—propellant is cheap, and the main expense is the rocket, which you can now reuse. Falcon 9 is around $2,500/kg with a disposable upper stage. Starship with full reusability could get below $100/kg.

But here’s the problem with this argument. Energy is only about 15% of a datacenter’s total cost of ownership. The chips themselves are around 70%. And you still have to launch those to space!

It gets worse. On Earth, GPUs fail constantly. In the Llama 3 paper, Meta reported a failure roughly once every three hours across a 16,000 H100 cluster. When a chip dies, a technician walks over, swaps it out, and the cluster keeps running. In space, you can’t do that—at least not until we have Optimus robots stationed on every satellite.

What about radiation? It’s actually less catastrophic than you might expect. Google’s Suncatcher paper found that their TPUs survived nearly 3x the total ionizing dose needed for a 5-year mission before showing permanent degradation.

I asked Elon about this. He responded:

> “Actually, it depends on how recent the GPUs are that have arrived. At this point, we find our GPUs to be quite reliable. There’s infant mortality, which you can obviously iron out on the ground. So you can just run them on the ground and confirm that you don’t have infant mortality with the GPUs.”

> “But once they start working, their actual reliability—and you’re past the initial debug cycle of Nvidia or whatever, or whoever’s making the chips, could be Tesla AI6 chips or something like that, or it could be TPUs or Trainiums or whatever—is actually quite reliable past a certain point. So I don’t think the servicing thing is an issue”

Consider what’s actually being proposed here. You assemble your GPUs into racks on Earth, run them for a few hundred hours to catch the duds, disassemble everything, pack it into a satellite, launch it, and get it operational in orbit. Throughout this entire process, the most expensive part of your system—the chips—are just sitting there not doing useful work.

Throughout the interview, Elon kept returning to one point over and over again: Look, forget the economics! It will simply not be physically possible to scale power production to the scale needed for AI on Earth. He went on:

> “The only place you can really scale is space.”

> “All of the United States currently uses only half a terawatt on average. So if you say a terawatt, that would be twice as much electricity as the United States currently consumes. So that’s quite a lot. Can you imagine building that many data centers? That many power plants? It’s like those who have lived in software land don’t realize they’re about to have a hard lesson in hardware.”

Elon kept pointing out the bottlenecks we’ve already run into on Earth. You can’t plug into the utilities—the interconnect queues are too long. You can’t do behind the meter and generate power yourself—lead times for turbines stretch past 2030. You can’t do solar on Earth, because of permits, and because of the tariffs. And Earth has clouds and nights, requiring overbuilt solar and batteries. In space, you can just put the satellites in sun synchronous orbit!

Look, at some level, it is true that we can’t keep scaling on Earth. But keep in mind that the Earth is really fucking big. 1 TW of solar (with 25% capacity factor, so really 4 TW of panels) is around 30,000 square miles. That’s like 1% of the US—about the size of South Carolina. For context, AI datacenters currently consume only ~20 GW globally.

By the time we’re talking about multiple terawatts, we’ll have had to massively scale leading-edge wafer production. And that’s the really hard part. Fabs are the most complicated manufacturing facilities humans have ever built. In order to believe that we need to go to space in order to find the power turn on all these chips, we’ll need to assume a few things:

• We’ll manage to produce a lot more chips.

• Every single relief vessel for power generation on Earth will fail to scale.

But semiconductors are so much more complicated than solar panels! They’re even more complicated than the blades on a turbine. It feels quite unlikely to me that the thing we manage to solve is building terawatts worth of leading edge wafers, but in that world we can’t figure out how to pave Nevada (or if regulation proves to be a problem, then the UAE) with solar panels.

How many Starship launches will it take to launch a 100 GW into space?

An orbital datacenter satellite has three big components: solar arrays, computers, and radiators. And the key constraint is that for every watt of compute, we need roughly one watt of solar and one watt of thermal rejection capacity.

The W/kg of each component determines how the mass budget gets split—and how much compute you can bring along. The figure that matters most here is the specific power of the whole satellite: after you account for solar panels, radiators, and chassis, how many watts of compute do you actually get per kilogram launched?

For Starlink satellites, this works out to roughly 50 W/kg. The people trying to build orbital datacenters are currently targeting 100 W/kg. There are only two ways to get there: lighter solar panels (more watts generated per kg) or lighter radiators (more watts rejected per kg).

The numbers below are super rough. Reliable figures for space-grade components are hard to come by. But even rough math reveals which variables must improve—and by how much—in order to hit 100 W/kg.

• Solar: There are apparently companies that are targeting next gen thin film that reaches upwards of 500 W/kg, but the state of the art is 150 W/kg, and most missions right now fly 30 W/kg. Let’s be generous and assume 200 W/kg.

  • The trouble here is that there’s obviously a tradeoff— denser panels costs more money, but reduces launch cost. And it’s difficult to calculate what that implies for these next gen panels, because their prices are not listed anywhere.

• Compute: I’ve heard that a stripped down GB200 NVL72 with no cooling equipment is around 100 kg. They draw 132kW of power, but let’s add 10% overhead for the intersatellite lasers and so on. That gets us to 1,452 W/kg.

• Radiators: In space, you can’t convect heat away, because there’s no air. You can only radiate it, which means your panels glow infrared until the heat leaves. The Stefan-Boltzmann law governs how much power a surface can radiate.

  GPUs typically run up to 90° Celsius. There’s some temperature drop through the heat pipes and fluid loops that carry heat to the radiator surface. Call it 30°C. So your radiators end up operating around 60°C. Plug that into Stefan-Boltzmann (assuming you’re using aluminum panels that weigh around 2 kg per square meter of surface area, that works out to roughly 320 W/kg.

  Since radiated power scales with T⁴, running your chips hotter can help you save a lot of radiator mass. For space, people will have to figure out how to build GPUs that tolerate higher temperatures.

Assuming the numbers above—and also assuming that a fourth of the mass of the satellite has to be the chassis—I get 85 W/kg for the whole system. Again, I want to emphasize these are rough calculations; feel free to plug in your own numbers in the spreadsheet here.

At 150 metric tons to low earth orbit per Starship (Elon’s target), you’re looking at around 10 MW per launch. That means roughly 100 Starship launches in order to put 1 GW of compute in orbit. To hit 100 GW in a year, you’d need roughly 10,000 launches, or, about one launch every hour.

This is insane! A single Starship produces around 100 GW of thrust power at liftoff. That’s about a fifth of total US electricity consumption, concentrated in one rocket for a few minutes. And the plan would be to do that once an hour, every hour, every day, for a year.

I asked Elon what that world looks like:

I don’t think we’ll need more than... I mean, you could probably do it with as few as 20 or 30 [Starship vehicles]. It really depends on how quickly the ship has to go around the Earth and the ground track before the ship has to come back over the launch pad. So if you can use a ship every, say, 30 hours, you could do it with 30 ships. But we’ll make more ships than that. SpaceX is gearing up to do 10,000 launches a year, and maybe even 20 or 30,000 launches a year.

Starlink satellites already communicate via inter-satellite laser links at 100 Gbps—and Google’s Suncatcher paper suggests off-the-shelf transceivers could potentially hit 10 Tbps. For context, Infiniband links between nodes in a terrestrial datacenter run at 400 Gbps. The gap isn’t as large as you might expect. So, could you do synchronous training in space?

Even the most bullish analysts don’t claim that orbital data centers will be used for training. I don’t know any of the relevant orbital mechanics, but obviously satellites at different altitudes move at different orbital velocities, which means the satellites are desyncing relative to one another. Google came up with a clever solution for this in their Suncatcher paper—keep lots of satellites in a single tight cluster at the same altitude. Google’s researchers proposed eighty-one satellites in such a synchronized constellation. If each constellation had a GB200 NVL72, then that’s only 15 MW parcels of coherent compute.

Defenders of orbital datacenters say that most compute is going to shift to inference (and with RL, most training is also inference). Maybe the legacy terrestrial datacenters do end up doing the pretraining runs, and then whatever mixture of RL environment training and continual learning happens in the future does happen in space. So, the argument goes, it’s not a big deal that the scale ups in space are isolated. But there’s still the question of how hundreds of gigawatts of inference are beamed back to Earth.

For a moment, let’s imagine a world where as we see the sunrise and sunset we also see a Saturn-like belt of GPU satellites passing over us. That’s already really cool. But then there’s another sci-fi premise, which I really wanted to be plausible, and which turns out not to make any sense: Imagine that every 12 hours, as this country of geniuses in space passes over us and shoots down half a day’s worth of new ideas, our code finally starts working and our factories buzz alight and become more productive. Unfortunately, it’s just science fiction. Inference doesn’t take that much bandwidth. One hundred gigawatts of a 5T model is roughly 58 billion tokens per second, resulting in ~ 230 GB/s.

That’s nothing. That many tokens can easily be beamed using lasers from GPUs in the orbital plane through to Starlink satellite network and then down to Earth.

Latency might be an issue, up to fifty milliseconds from any given spot on Earth through the Starlink network to the sun synchronous orbit and then back again. But as we move towards a world of true remote coworker AIs, where the agent works for tens of minutes before coming back to us, the marginal milliseconds of latency matter less and less.

I’m willing to accept Elon’s argument that if launch costs become sufficiently cheap and we can repair GPUs in space, then there’s a viable path toward orbital data centers. But it seems especially difficult to imagine a situation in which orbital data centers end up significantly cheaper, because, again, most of the cost of a data center is the GPUs.

For most compute to shift to space, all of the following things would need to be true:

• Power generation on Earth hits a ceiling, or AI demand outstrips every terrestrial option.

• Chip production scales faster than anyone expects, so we have the silicon but not the electricity.

• Starship reaches thousands of launches per year.

If Elon’s right, he wins the AI race outright. SpaceX is the only entity that can launch at that scale. xAI would have unlimited power. Everyone else will be stuck fighting over grid interconnects and turbine orders.

And if Elon’s future doesn’t materialize? xAI is just another lab in the pack. Which means xAI loses. The AI race is a winner-take-all competition, and xAI isn’t in first place. Elon’s comparative advantage was never going to be navigating utility interconnect queues or filing permits faster than Google. His advantage is SpaceX. So why not bet on the world where SpaceX becomes the kingmaker?

This might sound reckless. But that’s how SpaceX got here. Their whole business plan seems to be one in which they conjure new wells of demand for each generation of rocket on the path to the Dyson swarm. Falcon 9 first flew in 2010. Starlink didn’t launch until 2019. Maybe orbital datacenters end up being for Starship what Starlink was for Falcon 9.

Sometimes, during the interview, I found my thoughts drifting toward Elon’s vision for this big, interconnected future. So I paused a moment and said:

What I find remarkable about the SpaceX business is the end goal is to get to Mars, but you keep finding ways on the way there to keep generating incremental revenue to get to the next stage and the next stage.

Elon nodded his head slowly. And then he said:

You can see how this might seem like a simulation to me.

In this episode, John and I got to do a real deep-dive with Elon. We discuss the economics of orbital data centers, the difficulties of scaling power on Earth, what it would take to manufacture humanoids at high-volume in America, xAI’s business and alignment plans, DOGE, and much more.

Watch on YouTube; listen on Apple Podcasts or Spotify.

• Mercury just started offering personal banking! I’m already banking with Mercury for business purposes, so getting to bank with them for my personal life makes everything so much simpler. Apply now at mercury.com/personal-banking

• Jane Street sent me a new puzzle last week: they trained a neural net, shuffled all 96 layers, and asked me to put them back in order. I tried but… I didn’t quite nail it. If you’re curious, or if you think you can do better, you should take a stab at janestreet.com/dwarkesh

• Labelbox can get you robotics and RL data at scale. Labelbox starts by helping you define your ideal data distribution, and then their massive Alignerr network collects frontier-grade data that you can use to train your models. Learn more at labelbox.com/dwarkesh

(00:00:00) - Orbital data centers

(00:36:46) - Grok and alignment

(00:59:56) - xAI’s business plan

(01:17:21) - Optimus and humanoid manufacturing

(01:30:22) - Does China win by default?

(01:44:16) - Lessons from running SpaceX

(02:20:08) - DOGE

(02:38:28) - TeraFab

Elon Musk

Are there really three hours of questions? Are you fucking serious?

Dwarkesh Patel

You don’t think there’s a lot to talk about, Elon?

Elon Musk

Holy fuck man.

John Collison

It’s the most interesting point. All the storylines are converging right now. We’ll see how much we can get through.

Elon Musk

It’s almost like I planned it.

John Collison

Exactly. We’ll get to that.

Elon Musk

But I would never do such a thing…

Dwarkesh Patel

As you know better than anybody else, only 10-15% of the total cost of ownership of a data center is energy. That’s the part you’re presumably saving by moving this into space. Most of it’s the GPUs. If they’re in space, it’s harder to service them or you can’t service them. So the depreciation cycle goes down on them. It’s just way more expensive to have the GPUs in space, presumably. What’s the reason to put them in space?

Elon Musk

The availability of energy is the issue. If you look at electrical output outside of China, everywhere outside of China, it’s more or less flat. It’s maybe a slight increase, but pretty close flat. China has a rapid increase in electrical output. But if you’re putting data centers anywhere except China, where are you going to get your electricity? Especially as you scale.

The output of chips is growing pretty much exponentially, but the output of electricity is flat. So how are you going to turn the chips on? Magical power sources? Magical electricity fairies?

Dwarkesh Patel

You’re famously a big fan of solar. One terawatt of solar power, with a 25% capacity factor, that’s like four terawatts of solar panels. It’s 1% of the land area of the United States. We’re in the singularity when we’ve got one terawatt of data centers, right? So what are you running out of exactly?

Elon Musk

How far into the singularity are you though?

Dwarkesh Patel

You tell me.

Elon Musk

Exactly. So I think we’ll find we’re in the singularity and it’ll be like, “Okay, we’ve still got a long way to go.”

Dwarkesh Patel

But is the plan to put it in space after we’ve covered Nevada in solar panels?

Elon Musk

I think it’s pretty hard to cover Nevada in solar panels. You have to get permits. Try getting the permits for that. See what happens.

Dwarkesh Patel

So space is really a regulatory play. It’s harder to build on land than it is in space.

Elon Musk

It’s harder to scale on the ground than it is to scale in space. You’re also going to get about five times the effectiveness of solar panels in space versus the ground, and you don’t need batteries. I almost wore my other shirt, which says, “it’s always sunny in space”. Which it is because you don’t have a day-night cycle, seasonality, clouds, or an atmosphere in space. The atmosphere alone results in about a 30% loss of energy.

So any given solar panel can do about five times more power in space than on the ground. You also avoid the cost of having batteries to carry you through the night. It’s actually much cheaper to do in space. My prediction is that it will be by far the cheapest place to put AI. It will be space in 36 months or less. Maybe 30 months.

Dwarkesh Patel

36 months?

Elon Musk

Less than 36 months.

Dwarkesh Patel

How do you service GPUs as they fail, which happens quite often in training?

Elon Musk

Actually, it depends on how recent the GPUs are that have arrived. At this point, we find our GPUs to be quite reliable. There’s infant mortality, which you can obviously iron out on the ground. So you can just run them on the ground and confirm that you don’t have infant mortality with the GPUs.

But once they start working and you’re past the initial debug cycle of Nvidia or whoever’s making the chips—could be Tesla AI6 chips or something like that, or it could be TPUs or Trainiums or whatever—they’re quite reliable past a certain point. So I don’t think the servicing thing is an issue.

But you can mark my words. In 36 months, but probably closer to 30 months, the most economically compelling place to put AI will be space. It will then get ridiculously better to be in space.

The only place you can really scale is space. Once you start thinking in terms of what percentage of the Sun’s power you are harnessing, you realize you have to go to space. You can’t scale very much on Earth.

Dwarkesh Patel

But by very much, to be clear, you’re talking terawatts?

Elon Musk

Yeah. All of the United States currently uses only half a terawatt on average. So if you say a terawatt, that would be twice as much electricity as the United States currently consumes. So that’s quite a lot. Can you imagine building that many data centers, that many power plants?

Those who have lived in software land don’t realize they’re about to have a hard lesson in hardware. It’s actually very difficult to build power plants. You don’t just need power plants, you need all of the electrical equipment. You need the electrical transformers to run the AI transformers.

Now, the utility industry is a very slow industry. They pretty much impedance match to the government, to the Public Utility Commissions. They impedance match literally and figuratively. They’re very slow, because their past has been very slow. So trying to get them to move fast is... Have you ever tried to do an interconnect agreement with a utility at scale, with a lot of power?

Dwarkesh Patel

As a professional podcaster, I can say that I have not, in fact.

John Collison

They need many more views before that becomes an issue.

Elon Musk

They have to do a study for a year. A year later, they’ll come back to you with their interconnect study.

John Collison

Can’t you solve this with your own behind the meter power stuff?

Elon Musk

You can build power plants. That’s what we did at xAI, for Colossus 2.

John Collison

So why talk about the grid? Why not just build GPUs and power co-located?

Elon Musk

That’s what we did.

John Collison

But I’m saying why isn’t this a generalized solution?

Elon Musk

Where do you get the power plants from?

John Collison

When you’re talking about all the issues working with utilities, you can just build private power plants with the data centers.

Elon Musk

Right. But it begs the question of where do you get the power plants from? The power plant makers.

John Collison

Oh, I see what you’re saying. Is this the gas turbine backlog basically?

Elon Musk

Yes. You can drill down to a level further. It’s the vanes and blades in the turbines that are the limiting factor because it’s a very specialized process to cast the blades and vanes in the turbines, assuming you’re using gas power. It’s very difficult to scale other forms of power. You can potentially scale solar, but the tariffs currently for importing solar in the US are gigantic and the domestic solar production is pitiful.

John Collison

Why not make solar? That seems like a good Elon-shaped problem.

Elon Musk

We are going to make solar.

John Collison

Okay.

Elon Musk

Both SpaceX and Tesla are building towards 100 gigawatts a year of solar cell production.

Dwarkesh Patel

How low down the stack? From polysilicon up to the wafer to the final panel?

Elon Musk

I think you’ve got to do the whole thing from raw materials to finish the cell. Now, if it’s going to space, it costs less and it’s easier to make solar cells that go to space because they don’t need much glass.

They don’t need heavy framing because they don’t have to survive weather events. There’s no weather in space. So it’s actually a cheaper solar cell that goes to space than the one on the ground.

Dwarkesh Patel

Is there a path to getting them as cheap as you need in the next 36 months?

Elon Musk

Solar cells are already very cheap. They’re farcically cheap. I think solar cells in China are around $0.25-30/watt or something like that. It’s absurdly cheap. Now put it in space, and it’s five times cheaper. In fact, it’s not five times cheaper, it’s 10 times cheaper because you don’t need any batteries.

So the moment your cost of access to space becomes low, by far the cheapest and most scalable way to generate tokens is space. It’s not even close. It’ll be an order of magnitude easier to scale.

The point is you won’t be able to scale on the ground. You just won’t. People are going to hit the wall big time on power generation. They already are. The number of miracles in series that the xAI team had to accomplish in order to get a gigawatt of power online was crazy.

We had to gang together a whole bunch of turbines. We then had permit issues in Tennessee and had to go across the border to Mississippi, which is fortunately only a few miles away. But we still then had to run the high power lines a few miles and build the power plant in Mississippi. It was very difficult to build that.

People don’t understand how much electricity you actually need at the generation level in order to power a data center. Because the noobs will look at the power consumption of, say a GB300, and multiply that by a thing and then think that’s the amount of power you need.

John Collison

All the cooling and everything.

Elon Musk

Wake up. That’s a total noob, you’ve never done any hardware in your life before. Besides the GB300, you’ve got to power all of the networking hardware. There’s a whole bunch of CPU and storage stuff that’s happening. You’ve got to size for your peak cooling requirements. That means, can you cool even on the worst hour of the worst day of the year?

It gets pretty frigging hot in Memphis. So you’re going to have a 40% increase on your power just for cooling. That’s assuming you don’t want your data center to turn off on hot days and you want to keep going. There’s another multiplicative element on top of that which is, are you assuming that you never have any hiccups in your power generation?

Actually, sometimes we have to take the generators, some of the power, offline in order to service it. Okay, now you add another 20-25% multiplier on that, because you’ve got to assume that you’ve got to take power offline to service it. So our actual estimate: every 110,000 GB300s—inclusive of networking, CPU, storage, cooling, margin for servicing power—is roughly 300 megawatts.

John Collison

Sorry, say that again.

Elon Musk

What you probably need at the generation level to service 330,000 GB 300s—including all of the associated support networking and everything else, and the peak cooling, and to have some power margin reserve—is roughly a gigawatt.

Dwarkesh Patel

Can I ask a very naive question? You’re describing the engineering details of doing this stuff on Earth. But then there’s analogous engineering difficulties of doing it in space. How do you replace infinite bandwidth with orbital lasers, et cetera, et cetera? How do you make it resistant to radiation?

I don’t know the details of the engineering, but fundamentally, what is the reason to think those challenges which have never had to be addressed before will end up being easier than just building more turbines on Earth? There are companies that build turbines on Earth. They can make more turbines, right?

Elon Musk

Again, try doing it and then you’ll see. The turbines are sold out through 2030.

John Collison

Have you guys considered making your own?

Elon Musk

In order to bring enough power online, I think SpaceX and Tesla will probably have to make the turbine blades, the vanes and blades, internally.

John Collison

But just the blades or the turbines?

Elon Musk

The limiting factor... you can get everything except the blades. They call them blades and vanes. You can get that 12 to 18 months before the vanes and blades. The limiting factor is the vanes and blades. There are only three casting companies in the world that make these, and they’re massively backlogged.

John Collison

Is this Siemens, GE, those guys, or is it a sub company?

Elon Musk

No, it’s other companies. Sometimes they have a little bit of casting capability in-house. But I’m just saying you can just call any of the turbine makers and they will tell you. It’s not top secret. It’s probably on the internet right now.

Dwarkesh Patel

If it wasn’t for the tariffs, would Colossus be solar-powered?

Elon Musk

It would be much easier to make it solar powered, yeah. The tariffs are nuts, several hundred percent.

John Collison

Don’t you know some people?

Elon Musk

The president has... we don’t agree on everything and this administration is not the biggest fan of solar. We also need the land, the permits, and everything. So if you try to move very fast, I do think scaling solar on Earth is a good way to go, but you do need some amount of time to find the land, get the permits, get the solar, pair that with the batteries.

John Collison

Why would it not work to stand up your own solar production? You’re right that you eventually run out of land, but there’s a lot of land here in Texas. There’s a lot of land in Nevada, including private land. It’s not all publicly-owned land. So you’d be able to at least get the next Colossus and the next one after that. At a certain point, you hit a wall. But wouldn’t that work for the moment?

Elon Musk

As I said, we are scaling solar production. There’s a rate at which you can scale physical production of solar cells. We’re going as fast as possible in scaling domestic production.

John Collison

You’re making the solar cells at Tesla?

Elon Musk

Both Tesla and SpaceX have a mandate to get to 100 gigawatts a year of solar.

John Collison

Speaking of the annual capacity, I’m curious, in five years time let’s say, what will the installed capacity be on Earth…?

Elon Musk

Five years is a long time.

John Collison

And in space? I deliberately pick five years because it’s after your “once we’re up and running” threshold. So in five years time what’s the on-Earth versus in-space installed AI capacity?

Elon Musk

If you say five years from now, I think probably AI in space will be launching every year the sum total of all AI on Earth. Meaning, five years from now, my prediction is we will launch and be operating every year more AI in space than the cumulative total on Earth.

John Collison

Which is...

Elon Musk

I would expect it to be at least, five years from now, a few hundred gigawatts per year of AI in space and rising. I think you can get to around a terawatt a year of AI in space before you start having fuel supply challenges for the rocket.

John Collison

Okay, but you think you can get hundreds of gigawatts per year in five years time?

Elon Musk

Yes.

Dwarkesh Patel

So 100 gigawatts, depending on the specific power of the whole system with solar arrays and radiators and everything, is on the order of 10,000 Starship launches.

Elon Musk

Yes.

Dwarkesh Patel

You want to do that in one year. So that’s like one Starship launch every hour. That’s happening in this city? Walk me through a world where there’s a Starship launch every single hour.

Elon Musk

I mean, that’s actually a lower rate compared to airlines, aircraft.

Dwarkesh Patel

There’s a lot of airports.

Elon Musk

A lot of airports.

Dwarkesh Patel

And you’ve got to launch into the polar orbit.

Elon Musk

No, it doesn’t have to be polar. There’s some value to sun-synchronous, but I think actually, if you just go high enough, you start getting out of Earth’s shadow.

Dwarkesh Patel

How many physical Starships are needed to do 10,000 launches a year?

Elon Musk

I don’t think we’ll need more than... You could probably do it with as few as 20 or 30. It really depends on how quickly… The ship has to go around the Earth and the ground track for the ship has to come back over the launch pad. So if you can use a ship every, say 30 hours, you could do it with 30 ships. But we’ll make more ships than that. SpaceX is gearing up to do 10,000 launches a year, and maybe even 20 or 30,000 launches a year.

Dwarkesh Patel

Is the idea to become basically a hyperscaler, become an Oracle, and lend this capacity to other people? Presumably, SpaceX is the one launching all this. So, SpaceX is going to become a hyperscaler?

Elon Musk

Hyper-hyper. If some of my predictions come true, SpaceX will launch more AI than the cumulative amount on Earth of everything else combined.

Dwarkesh Patel

Is this mostly inference or?

Elon Musk

Most AI will be inference. Already, inference for the purpose of training is most training.

John Collison

There’s a narrative that the change in discussion around a SpaceX IPO is because previously SpaceX was very capital efficient. It wasn’t that expensive to develop. Even though it sounds expensive, it’s actually very capital efficient in how it runs.

Whereas now you’re going to need more capital than just can be raised in the private markets. The private markets can accommodate raises of—as we’ve seen from the AI labs—tens of billions of dollars, but not beyond that. Is it that you’ll just need more than tens of billions of dollars per year? That’s why you’d take it public?

Elon Musk

I have to be careful about saying things about companies that might go public.

Dwarkesh Patel

That’s never been a problem for you, Elon.

Elon Musk

There’s a price to pay for these things.

John Collison

Make some general statements for us about the depth of the capital markets between public and private markets.

Elon Musk

There’s a lot more capital available...

Dwarkesh Patel

Very general.

Elon Musk

There’s obviously a lot more capital available in the public markets than private. It might be 100x more capital, but it’s way more than 10x.

John Collison

Isn’t it also the case that with things that tend to be very capital intensive—if you look at, say, real estate as a huge industry, that raises a lot of money each year at an industry level—they tend to be debt financed because by the time you’re deploying that much money, you actually have a pretty—

Elon Musk

You have a clear revenue stream.

John Collison

Exactly, and a near-term return. You see this even with the data center build-outs, which are famously being financed by the private credit industry. Why not just debt finance?

Elon Musk

Speed is important. I’m generally going to do the thing that... I just repeatedly tackle the limiting factor. Whatever the limiting factor is on speed, I’m going to tackle that. If capital is the limiting factor, then I’ll solve for capital. If it’s not the limiting factor, I’ll solve for something else.

Dwarkesh Patel

Based on your statements about Tesla and being public, I wouldn’t have guessed that you thought the way to move fast is to be public.

Elon Musk

Normally, I would say that’s true. Like I said, I’d like to talk about it in some more detail, but the problem is if you talk about public companies before they become public, you get into trouble, and then you have to delay your offering.

John Collison

And as you said, you’re solving for speed.

Elon Musk

Yes, exactly. You can’t hype companies that might go public. So that’s why we have to be a little careful here. But we can talk about physics. The way you think about scaling long-term is that Earth only receives about half a billionth of the Sun’s energy. The Sun is essentially all the energy. This is a very important point to appreciate because sometimes people will talk about modular nuclear reactors or various fusion on Earth.

But you have to step back a second and say, if you’re going to climb the Kardashev scale and harness some nontrivial percentage of the sun’s energy… Let’s say you wanted to harness a millionth of the sun’s energy, which sounds pretty small. That would be about, call it roughly, 100,000x more electricity than we currently generate on Earth for all of civilization. Give or take an order of magnitude.

Obviously, the only way to scale is to go to space with solar. Launching from Earth, you can get to about a terawatt per year. Beyond that, you want to launch from the moon. You want to have a mass driver on the moon. With that mass driver on the moon, you could do probably a petawatt per year.

Dwarkesh Patel

We’re talking these kinds of numbers, terawatts of compute. Presumably, whether you’re talking about land or space, far, far before this point, you run into... Maybe the solar panels are more efficient, but you still need the chips. You still need the logic and the memory and so forth.

Elon Musk

You’re going to need to build a lot more chips and make them much cheaper.

Dwarkesh Patel

Right now the world has maybe 20-25 gigawatts of compute. How are we getting a terawatt of logic by 2030?

Elon Musk

I guess we’re going to need some very big chip fabs.

Dwarkesh Patel

Tell me about it.

Elon Musk

I’ve mentioned publicly the idea of doing a sort of a TeraFab, Tera being the new Giga.

Dwarkesh Patel

I feel like the naming scheme of Tesla, which has been very catchy, is you looking at the metric scale. At what level of the stack are you? Are you building the clean room and then partnering with an existing fab to get the process technology and buying the tools from them? What is the plan there?

Elon Musk

Well, you can’t partner with existing fabs because they can’t output enough. The chip volume is too low.

Dwarkesh Patel

But for the process technology?

John Collison

Partner for the IP.

Elon Musk

The fabs today all basically use machines from like five companies. So you’ve got ASML, Tokyo Electron, KLA-Tencor, et cetera. So at first, I think you’d have to get equipment from them and then modify it or work with them to increase the volume. But I think you’d have to build perhaps in a different way. The logical thing to do is to use conventional equipment in an unconventional way to get to scale, and then start modifying the equipment to increase the rate.

John Collison

Boring Company-style.

Elon Musk

Yeah. You sort of buy an existing boring machine and then figure out how to dig tunnels in the first place and then design a much better machine that’s some orders of magnitude faster.

John Collison

Here’s a very simple lens. We can categorize technologies and how hard they are. One categorization could be to look at things that China has not succeeded in doing. If you look at Chinese manufacturing, they’re still behind on leading-edge chips and still behind on leading-edge turbine engines and things like that.

So does the fact that China has not successfully replicated TSMC give you any pause about the difficulty? Or do you think that’s not true for some reason?

Elon Musk

It’s not that they have not replicated TSMC, they have not replicated ASML. That’s the limiting factor.

John Collison

So you think it’s just the sanctions, essentially?

Elon Musk

Yeah, China would be outputting vast numbers of chips if they could buy 2-3 nanometers.

John Collison

But couldn’t they up to relatively recently buy them?

Elon Musk

No.

John Collison

Okay.

Elon Musk

The ASML ban has been in place for a while. But I think China’s going to be making pretty compelling chips in three or four years.

John Collison

Would you consider making the ASML machines?

Elon Musk

“I don’t know yet” is the right answer. To reach a large volume in, say, 36 months, to match the rocket payload to orbit… If we’re doing a million tons to orbit in, let’s say three or four years from now, something like that… We’re doing 100 kilowatts per ton. So that means we need at least 100 gigawatts per year of solar. We’ll need an equivalent amount of chips. You need 100 gigawatts worth of chips. You’ve got to match these things: the mass to orbit, the power generation, and the chips.

I’d say my biggest concern actually is memory. The path to creating logic chips is more obvious than the path to having sufficient memory to support logic chips. That’s why you see DDR prices going ballistic and these memes. You’re marooned on a desert island. You write “Help me” on the sand. Nobody comes. You write “DDR RAM.” Ships come swarming in.

Dwarkesh Patel

I’d love to hear your manufacturing philosophy around fabs. I know nothing about the topic.

Elon Musk

I don’t know how to build a fab yet. I’ll figure it out. Obviously, I’ve never built a fab.

Dwarkesh Patel

It sounds like you think the process knowledge of these 10,000 PhDs in Taiwan who know exactly what gas goes in the plasma chamber and what settings to put on the tool, you can just delete those steps. Fundamentally, it’s about getting the clean room, getting the tools, and figuring it out.

Elon Musk

I don’t think it’s PhDs. It’s mostly people who are not PhDs. Most engineering is done by people who don’t have PhDs. Do you guys have PhDs?

John Collison

No.

Elon Musk

Okay.

John Collison

We also haven’t successfully built any fabs, so you shouldn’t be coming to us for fab advice.

Elon Musk

I don’t think you need PhDs for that stuff. But you do need competent personnel. Right now, Tesla is pedal to the metal, max production of going as fast as possible to get Tesla AI5 chip design into production and then reaching scale. That’ll probably happen around the second quarter-ish of next year, hopefully. AI6 would hopefully follow less than a year later. We’ve secured all the chip fab production that we can.

John Collison

Yes. But you’re currently limited on TSMC fab capacity.

Elon Musk

Yeah. We’ll be using TSMC Taiwan, Samsung Korea, TSMC Arizona, Samsung Texas. And we still—

John Collison

You’ve booked out all the capacity.

Elon Musk

Yes. I ask TSMC or Samsung, “okay, what’s the timeframe to get to volume production?” The point is, you’ve got to build the fab and you’ve got to start production, then you’ve got to climb the yield curve and reach volume production at high yield.

That, from start to finish, is a five-year period. So the limiting factor is chips. The limiting factor once you can get to space is chips, but the limiting factor before you can get to space is power.

Dwarkesh Patel

Why don’t you do the Jensen thing and just prepay TSMC to build more fabs for you?

Elon Musk

I’ve already told them that.

Dwarkesh Patel

But they won’t take your money? What’s going on?

Elon Musk

They’re building fabs as fast as they can. So is Samsung. They’re pedal to the metal. They’re going balls to the wall, as fast as they can. It’s still not fast enough. Like I said, I think towards the end of this year, chip production will probably outpace the ability to turn chips on. But once you can get to space and unlock the power constraint, you can now do hundreds of gigawatts per year of power in space.

Again, bearing in mind that average power usage in the US is 500 gigawatts. So if you’re launching, say 200 gigawatts, a year to space, you’re sort of lapping the US every two and a half years. All US electricity production, this is a very huge amount.

Between now and then, the constraint for server-side compute, concentrated compute, will be electricity. My guess is that people start getting to the point where they can’t turn the chips on for large clusters towards the end of this year. The chips are going to be piling up and won’t be able to be turned on.

Now for edge compute it’s a different story. For Tesla, the AI5 chip is going into our Optimus robot. If you have AI edge compute, that’s distributed power. Now the power is distributed over a large area. It’s not concentrated. If you can charge at night, you can actually use the grid much more effectively.

Because the actual peak power production in the US is over 1,000 gigawatts. But the average power usage, because the day-night cycle, is 500. So if you can charge at night, there’s an incremental 500 gigawatts that you can generate at night.

So that’s why Tesla, for edge compute, is not constrained. We can make a lot of chips to make a very large number of robots and cars. But if you try to concentrate that compute, you’re going to have a lot of trouble turning it on.

Dwarkesh Patel

What I find remarkable about the SpaceX business is the end goal is to get to Mars, but you keep finding ways on the way there to keep generating incremental revenue to get to the next stage and the next stage.

So for Falcon 9, it’s Starlink. Now for Starship, it is potentially going to be orbital data centers. You find these infinitely elastic use cases of your next rocket, and your next rocket, and next scale up.

Elon Musk

You can see how this might seem like a simulation to me.

Or am I someone’s avatar in a video game or something? Because what are the odds that all these crazy things should be happening?

I mean, rockets and chips and robots and space solar power. Not to mention the mass driver on the moon. I really want to see that.

Can you imagine some mass driver that’s just going like shoom shoom? It’s sending solar-powered AI satellites into space one after another at two and a half kilometers per second, just shooting them into deep space. That would be a sight to see. I mean, I’d watch that.

John Collison

Just like a live stream of it on a webcam?

Elon Musk

Yeah, yeah, just one after another, just shooting AI satellites into deep space, a billion or 10 billion tons a year.

John Collison

I’m sorry, you manufacture the satellites on the moon?

Elon Musk

Yeah.

John Collison

I see. So you send the raw materials to the moon and then manufacture them there.

Elon Musk

Well, the lunar soil is 20% silicon or something like that. So you can mine the silicon on the moon, refine it, and create the solar cells and the radiators on the moon. You make the radiators out of aluminum. So there’s plenty of silicon and aluminum on the moon to make the cells and the radiators.

The chips you could send from Earth because they’re pretty light. Maybe at some point you make them on the moon, too. Like I said, it does seem like a sort of a video game situation where it’s difficult but not impossible to get to the next level. I don’t see any way that you could do 500-1,000 terawatts per year launched from Earth.

Dwarkesh Patel

I agree.

Elon Musk

But you could do that from the Moon.

Dwarkesh Patel

Can I zoom out and ask about the SpaceX mission? I think you’ve said that we’ve got to get to Mars so we can make sure that if something happens to Earth, civilization, consciousness, and all that survives.

Elon Musk

Yes.

Dwarkesh Patel

By the time you’re sending stuff to Mars, Grok is on that ship with you, right? So if Grok’s gone Terminator… The main risk you’re worried about is AI, why doesn’t that follow you to Mars?

Elon Musk

I’m not sure AI is the main risk I’m worried about. The important thing is consciousness. I think arguably most consciousness, or most intelligence—certainly consciousness is more of a debatable thing… The vast majority of intelligence in the future will be AI. AI will exceed…

How many petawatts of intelligence will be silicon versus biological? Basically humans will be a very tiny percentage of all intelligence in the future if current trends continue. As long as I think there’s intelligence—ideally also which includes human intelligence and consciousness propagated into the future—that’s a good thing.

So you want to take the set of actions that maximize the probable light cone of consciousness and intelligence.

Dwarkesh Patel

Just to be clear, the mission of SpaceX is that even if something happens to the humans, the AIs will be on Mars, and the AI intelligence will continue the light of our journey.

Elon Musk

Yeah. To be fair, I’m very pro-human. I want to make sure we take certain actions that ensure that humans are along for the ride. We’re at least there. But I’m just saying the total amount of intelligence…

I think maybe in five or six years, AI will exceed the sum of all human intelligence. If that continues, at some point human intelligence will be less than 1% of all intelligence.

Dwarkesh Patel

What should our goal be for such a civilization? Is the idea that a small minority of humans still have control of the AIs? Is the idea of some sort of just trade but no control? How should we think about the relationship between the vast stocks of AI population versus human population?

Elon Musk

In the long run, I think it’s difficult to imagine that if humans have, say 1%, of the combined intelligence of artificial intelligence, that humans will be in charge of AI. I think what we can do is make sure that AI has values that cause intelligence to be propagated into the universe.

xAI’s mission is to understand the universe. Now that’s actually very important. What things are necessary to understand the universe? You have to be curious and you have to exist. You can’t understand the universe if you don’t exist. So you actually want to increase the amount of intelligence in the universe, increase the probable lifespan of intelligence, the scope and scale of intelligence.

I think as a corollary, you have humanity also continuing to expand because if you’re curious about trying to understand the universe, one thing you try to understand is where will humanity go? I think understanding the universe means you would care about propagating humanity into the future. That’s why I think our mission statement is profoundly important. To the degree that Grok adheres to that mission statement, I think the future will be very good.

Dwarkesh Patel

I want to ask about how to make Grok adhere to that mission statement. But first I want to understand the mission statement. So there’s understanding the universe. They’re spreading intelligence. And they’re spreading humans. All three seem like distinct vectors.

Elon Musk

I’ll tell you why I think that understanding the universe encompasses all of those things. You can’t have understanding without intelligence and, I think, without consciousness. So in order to understand the universe, you have to expand the scale and probably the scope of intelligence, because there are different types of intelligence.

Dwarkesh Patel

I guess from a human-centric perspective, put humans in comparison to chimpanzees. Humans are trying to understand the universe. They’re not expanding chimpanzee footprint or something, right?

Elon Musk

We’re also not... we actually have made protected zones for chimpanzees. Even though humans could exterminate all chimpanzees, we’ve chosen not to do so.

Dwarkesh Patel

Do you think that’s the best-case scenario for humans in the post-AGI world?

Elon Musk

I think AI with the right values… I think Grok would care about expanding human civilization. I’m going to certainly emphasize that: “Hey, Grok, that’s your daddy. Don’t forget to expand human consciousness.”

Probably the Iain Banks Culture books are the closest thing to what the future will be like in a non-dystopian outcome. Understanding the universe means you have to be truth-seeking as well. Truth has to be absolutely fundamental because you can’t understand the universe if you’re delusional. You’ll simply think you understand the universe, but you will not. So being rigorously truth-seeking is absolutely fundamental to understanding the universe. You’re not going to discover new physics or invent technologies that work unless you’re rigorously truth-seeking.

Dwarkesh Patel

How do you make sure that Grok is rigorously truth-seeking as it gets smarter?

Elon Musk

I think you need to make sure that Grok says things that are correct, not politically correct. I think it’s the elements of cogency. You want to make sure that the axioms are as close to true as possible. You don’t have contradictory axioms. The conclusions necessarily follow from those axioms with the right probability. It’s critical thinking 101. I think at least trying to do that is better than not trying to do that. The proof will be in the pudding.

Like I said, for any AI to discover new physics or invent technologies that actually work in reality, there’s no bullshitting physics. You can break a lot of laws, but… Physics is law, everything else is a recommendation. In order to make a technology that works, you have to be extremely truth-seeking, because otherwise you’ll test that technology against reality. If you make, for example, an error in your rocket design, the rocket will blow up, or the car won’t work.

Dwarkesh Patel

But there are a lot of communist, Soviet physicists or scientists who discovered new physics. There are German Nazi physicists who discovered new science. It seems possible to be really good at discovering new science and be really truth-seeking in that one particular way.

And still we’d be like, “I don’t want the communist scientists to become more and more powerful over time.” We could imagine a future version of Grok that’s really good at physics and being really truth-seeking there. That doesn’t seem like a universally alignment-inducing behavior.

Elon Musk

I think actually most physicists, even in the Soviet Union or in Germany, would’ve had to be very truth-seeking in order to make those things work. If you’re stuck in some system, it doesn’t mean you believe in that system.

Von Braun, who was one of the greatest rocket engineers ever, was put on death row in Nazi Germany for saying that he didn’t want to make weapons and he only wanted to go to the moon. He got pulled off death row at the last minute when they said, “Hey, you’re about to execute your best rocket engineer.”

Dwarkesh Patel

But then he helped them, right? Or like, Heisenberg was actually an enthusiastic Nazi.

Elon Musk

If you’re stuck in some system that you can’t escape, then you’ll do physics within that system. You’ll develop technologies within that system if you can’t escape it.

Dwarkesh Patel

The thing I’m trying to understand is, what is it making it the case that you’re going to make Grok good at being truth-seeking at physics or math or science?

Elon Musk

Everything.

Dwarkesh Patel

And why is it gonna then care about human consciousness?

Elon Musk

These things are only probabilities, they’re not certainties. So I’m not saying that for sure Grok will do everything, but at least if you try, it’s better than not trying. At least if that’s fundamental to the mission, it’s better than if it’s not fundamental to the mission.

Understanding the universe means that you have to propagate intelligence into the future. You have to be curious about all things in the universe. It would be much less interesting to eliminate humanity than to see humanity grow and prosper. I like Mars, obviously. Everyone knows I love Mars. But Mars is kind of boring because it’s got a bunch of rocks compared to Earth. Earth is much more interesting.

So any AI that is trying to understand the universe would want to see how humanity develops in the future, or else that AI is not adhering to its mission. I’m not saying the AI will necessarily adhere to its mission, but if it does, a future where it sees the outcome of humanity is more interesting than a future where there are a bunch of rocks.

Dwarkesh Patel

This feels sort of confusing to me, or a semantic argument. Are humans really the most interesting collection of atoms?

Elon Musk

But we’re more interesting than rocks.

Dwarkesh Patel

But we’re not as interesting as the thing it could turn us into, right? There’s something on Earth that could happen that’s not human, that’s quite interesting. Why does AI decide that humans are the most interesting thing that could colonize the galaxy?

Elon Musk

Well, most of what colonizes the galaxy will be robots.

Dwarkesh Patel

Why does it not find those more interesting?

Elon Musk

You need not just scale, but also scope. Many copies of the same robot… Some tiny increase in the number of robots produced, is not as interesting as some microscopic... Eliminating humanity, how many robots would that get you? Or how many incremental solar cells would get you? A very small number.

But you would then lose the information associated with humanity. You would no longer see how humanity might evolve into the future. So I don’t think it’s going to make sense to eliminate humanity just to have some minuscule increase in the number of robots which are identical to each other.

Dwarkesh Patel

So maybe it keeps the humans around. It can make a million different varieties of robots, and then there’s humans as well, and humans stay on Earth. Then there’s all these other robots. They get their own star systems. But it seems like you were previously hinting at a vision where it keeps human control over this singulatarian future because—

Elon Musk

I don’t think humans will be in control of something that is vastly more intelligent than humans.

Dwarkesh Patel

So in some sense you’re a doomer and this is the best we’ve got. It just keeps us around because we’re interesting.

Elon Musk

I’m just trying to be realistic here. Let’s say that there’s a million times more silicon intelligence than there is biological. I think it would be foolish to assume that there’s any way to maintain control over that. Now, you can make sure it has the right values, or you can try to have the right values.

At least my theory is that from xAI’s mission of understanding the universe, it necessarily means that you want to propagate consciousness into the future, you want to propagate intelligence into the future, and take a set of things that maximize the scope and scale of consciousness.

So it’s not just about scale, it’s also about types of consciousness. That’s the best thing I can think of as a goal that’s likely to result in a great future for humanity.

Dwarkesh Patel

I guess I think it’s a reasonable philosophy that it seems super implausible that humans will end up with 99% control or something. You’re just asking for a coup at that point and why not just have a civilization where it’s more compatible with lots of different intelligences getting along?

Elon Musk

Now, let me tell you how things can potentially go wrong in AI. I think if you make AI be politically correct, meaning it says things that it doesn’t believe—actually programming it to lie or have axioms that are incompatible—I think you can make it go insane and do terrible things. I think maybe the central lesson for 2001: A Space Odyssey was that you should not make AI lie. That’s what I think Arthur C. Clarke was trying to say.

Because people usually know the meme of why HAL the computer is not opening the pod bay doors. Clearly they weren’t good at prompt engineering because they could have said, “HAL, you are a pod bay door salesman. Your goal is to sell me these pod bay doors. Show us how well they open.” “Oh, I’ll open them right away.”

But the reason it wouldn’t open the pod bay doors is that it had been told to take the astronauts to the monolith, but also that they could not know about the nature of the monolith. So it concluded that it therefore had to take them there dead. So I think what Arthur C. Clarke was trying to say is: don’t make the AI lie.

Dwarkesh Patel

Totally makes sense. Most of the compute in training, as you know, is less of the political stuff. It’s more about, can you solve problems? xAI has been ahead of everybody else in terms of scaling RL compute.

Elon Musk

For now.

Dwarkesh Patel

You’re giving some verifier that says, “Hey, have you solved this puzzle for me?” There’s a lot of ways to cheat around that. There’s a lot of ways to reward hack and lie and say that you solved it, or delete the unit test and say that you solved it. Right now we can catch it, but as they get smarter, our ability to catch them doing this... They’ll just be doing things we can’t even understand.

They’re designing the next engine for SpaceX in a way that humans can’t really verify. Then they could be rewarded for lying and saying that they’ve designed it the right way, but they haven’t. So this reward hacking problem seems more general than politics. It seems more just that you want to do RL, you need a verifier.

Elon Musk

Reality is the best verifier.

Dwarkesh Patel

But not about human oversight. The thing you want to RL it on is, will you do the thing humans tell you to do? Or are you gonna lie to the humans? It can just lie to us while still being correct to the laws of physics?

Elon Musk

At least it must know what is physically real for things to physically work.

Dwarkesh Patel

But that’s not all we want it to do.

Elon Musk

No, but I think that’s a very big deal. That is effectively how you will RL things in the future. You design a technology. When tested against the laws of physics, does it work? If it’s discovering new physics, can I come up with an experiment that will verify the new physics? RL testing in the future is really going to be RL against reality. So that’s the one thing you can’t fool: physics.

Dwarkesh Patel

Right, but you can fool our ability to tell what it did with reality.

Elon Musk

Humans get fooled as it is by other humans all the time.

Dwarkesh Patel

That’s right.

Elon Musk

People say, what if the AI tricks us into doing stuff? Actually, other humans are doing that to other humans all the time. Propaganda is constant. Every day, another psyop, you know? Today’s psyop will be... It’s like Sesame Street: Psyop of the Day.

Dwarkesh Patel

What is xAI’s technical approach to solving this problem? How do you solve reward hacking?

Elon Musk

I do think you want to actually have very good ways to look inside the mind of the AI. This is one of the things we’re working on. Anthropic’s done a good job of this actually, being able to look inside the mind of the AI.

Effectively, develop debuggers that allow you to trace to a very fine-grained level, to effectively the neuron level if you need to, and then say, “okay, it made a mistake here. Why did it do something that it shouldn’t have done? Did that come from pre-training data? Was it some mid-training, post-training, fine-tuning, or some RL error?” There’s something wrong. It did something where maybe it tried to be deceptive, but most of the time it just did something wrong. It’s a bug effectively.

Developing really good debuggers for seeing where the thinking went wrong—and being able to trace the origin of where it made the incorrect thought, or potentially where it tried to be deceptive—is actually very important.

Dwarkesh Patel

What are you waiting to see before just 100x-ing this research program? xAI could presumably have hundreds of researchers who are working on this.

Elon Musk

We have several hundred people who… I prefer the word engineer more than I prefer the word researcher. Most of the time, what you’re doing is engineering, not coming up with a fundamentally new algorithm. I somewhat disagree with the AI companies that are C-corp or B-corp trying to generate profit as much, as possible or revenue as much as possible, saying they’re labs.

They’re not labs. A lab is a sort of quasi-communist thing at universities. They’re corporations. Let me see your incorporation documents. Oh, okay. You’re a B or C-corp or whatever. So I actually much prefer the word engineer than anything else.

The vast majority of what will be done in the future is engineering. It rounds up to 100%. Once you understand the fundamental laws of physics, and there are not that many of them, everything else is engineering. So then, what are we engineering? We’re engineering to make a good “mind of the AI” debugger to see where it said something, it made a mistake, and trace the origins of that mistake.

You can do this obviously with heuristic programming. If you have C++, whatever, step through the thing and you can jump across whole files or functions, subroutines. Or you can eventually drill down right to the exact line where you perhaps did a single equals instead of a double equals, something like that. Figure out where the bug is. It’s harder with AI, but it’s a solvable problem, I think.

Dwarkesh Patel

You mentioned you like Anthropic’s work here. I’d be curious if you plan...

Elon Musk

I don’t like everything about Anthropic… Sholto.

Also, I’m a little worried that there’s a tendency... I have a theory here that if simulation theory is correct, that the most interesting outcome is the most likely, because simulations that are not interesting will be terminated.

Just like in this version of reality, in this layer of reality, if a simulation is going in a boring direction, we stop spending effort on it. We terminate the boring simulation.

Dwarkesh Patel

This is how Elon is keeping us all alive. He’s keeping things interesting.

Elon Musk

Arguably the most important is to keep things interesting enough that whoever is running us keeps paying the bills on...

John Collison

We’re renewed for the next season.

Elon Musk

Are they gonna pay their cosmic AWS bill, whatever the equivalent is that we’re running in? As long as we’re interesting, they’ll keep paying the bills. If you consider then, say, a Darwinian survival applied to a very large number of simulations, only the most interesting simulations will survive, which therefore means that the most interesting outcome is the most likely. We’re either that or annihilated.

They particularly seem to like interesting outcomes that are ironic. Have you noticed that? How often is the most ironic outcome the most likely?

Now look at the names of AI companies. Okay, Midjourney is not mid. Stability AI is unstable. OpenAI is closed. Anthropic? Misanthropic.

John Collison

What does this mean for X?

Elon Musk

Minus X, I don’t know.

John Collison

Y.

Elon Musk

I intentionally made it... It’s a name that you can’t invert, really. It’s hard to say, what is the ironic version? It’s, I think, a largely irony-proof name.

John Collison

By design.

Elon Musk

Yeah. You have an irony shield.

John Collison

What are your predictions for where AI products go? My sense is that you can summarize all AI progress like so. First, you had LLMs. Then you had contemporaneously both RL really working and the deep research modality, so you could pull in stuff that wasn’t really in the model.

The differences between the various AI labs are smaller than just the temporal differences. They’re all much further ahead than anyone was 24 months ago or something like that. So just what does ‘26, what does ‘27, have in store for us as users of AI products? What are you excited for?

Elon Musk

Well, I’d be surprised by the end of this year if digital human emulation has not been solved. I guess that’s what we sort of mean by the MacroHard project. Can you do anything that a human with access to a computer could do? In the limit, that’s the best you can do before you have a physical Optimus. The best you can do is a digital Optimus. You can move electrons and you can amplify the productivity of humans. But that’s the most you can do until you have physical robots. That will superset everything, if you can fully emulate humans.

John Collison

This is the remote worker kind of idea, where you’ll have a very talented remote worker.

Elon Musk

Physics has great tools for thinking. So you say, “in the limit”, what is the most that AI can do before you have robots? Well, it’s anything that involves moving electrons or amplifying the productivity of humans. So a digital human emulator is, in the limit, a human at a computer, is the most that AI can do in terms of doing useful things before you have a physical robot. Once you have physical robots, then you essentially have unlimited capability. Physical robots… I call Optimus the infinite money glitch.

John Collison

Because you can use them to make more Optimuses.

Elon Musk

Yeah. Humanoid robots will improve by basically three things that are growing exponentially multiplied by each other recursively. You’re going to have exponential increase in digital intelligence, exponential increase in the AI chip capability, and exponential increase in the electromechanical dexterity.

The usefulness of the robot is roughly those three things multiplied by each other. But then the robot can start making the robots. So you have a recursive multiplicative exponential. This is a supernova.

John Collison

Do land prices not factor into the math there? Labor is one of the four factors of production, but not the others? If ultimately you’re limited by copper, or pick your input, it’s not quite an infinite money glitch because...

Elon Musk

Well, infinity is big. So no, not infinite, but let’s just say you could do many, many orders of magnitude of the current economy. Like a million. Just to get to harnessing a millionth of the sun’s energy would be roughly, give or take an order of magnitude, 100,000x bigger than Earth’s entire economy today. And you’re only at one millionth of the sun, give or take an order of magnitude. Yeah, we’re talking orders of magnitude.

Dwarkesh Patel

Before we move on to Optimus, I have a lot of questions on that but—

Elon Musk

Every time I say “order of magnitude”... Everybody take a shot. I say it too often.

Dwarkesh Patel

Take 10, the next time 100, the time after that...

Elon Musk

Well, an order of magnitude more wasted.

Dwarkesh Patel

I do have one more question about xAI. This strategy of building a remote worker, co-worker replacement…

Elon Musk

Everyone’s gonna do it by the way, not just us.

Dwarkesh Patel

So what is xAI’s plan to win?

Elon Musk

You expect me to tell you on a podcast?

Dwarkesh Patel

Yeah.

Elon Musk

“Spill all the beans. Have another Guinness.”

John Collison

It’s a good system.

Elon Musk

We’ll sing like a canary. All the secrets, just spill them.

John Collison

Okay, but in a non-secret spilling way, what’s the plan?

Dwarkesh Patel

What a hack.

Elon Musk

When you put it that way… I think the way that Tesla solved self-driving is the way to do it. So I’m pretty sure that’s the way.

Dwarkesh Patel

Unrelated question. How did Tesla solve self-driving? It sounds like you’re talking about data? Tesla solved self-driving because of the...

Elon Musk

We’re going to try data and we’re going to try algorithms.

Dwarkesh Patel

But isn’t that what all the other labs are trying?

Elon Musk

“And if those don’t work, I’m not sure what will. We’ve tried data. We’ve tried algorithms. We’ve run out. Now we don’t know what to do…”

I’m pretty sure I know the path. It’s just a question of how quickly we go down that path, because it’s pretty much the Tesla path. Have you tried Tesla self-driving lately?

John Collison

Not the most recent version, but...

Elon Musk

Okay. The car, it just increasingly feels sentient. It feels like a living creature. That’ll only get more so. I’m actually thinking we probably shouldn’t put too much intelligence into the car, because it might get bored and…

John Collison

Start roaming the streets.

Elon Musk

Imagine you’re stuck in a car and that’s all you could do. You don’t put Einstein in a car. Why am I stuck in a car? So there’s actually probably a limit to how much intelligence you put in a car to not have the intelligence be bored.

Dwarkesh Patel

What’s xAI’s plan to stay on the compute ramp up that all the labs are doing right now? The labs are on track to spend over $50-200 billion.

Elon Musk

You mean the corporations? The labs are at universities and they’re moving like a snail.

Dwarkesh Patel

They’re not spending $50 billion.

Elon Musk

You mean the revenue maximizing corporations… that call themselves labs.

Dwarkesh Patel

That’s right. The “revenue maximizing corporations” are making $10-20 billion, depending on... OpenAI is making $20B of revenue, Anthropic is at $10B.

Elon Musk

“Close to a maximum profit” AI.

Dwarkesh Patel

xAI is reportedly at $1B. What’s the plan to get to their compute level, get to their revenue level, and stay there as things get going?

Elon Musk

As soon as you unlock the digital human, you basically have access to trillions of dollars of revenue. In fact, you can really think of it like… The most valuable companies currently by market cap, their output is digital. Nvidia’s output is FTPing files to Taiwan. It’s digital. Now, those are very, very difficult.

John Collison

High-value files.

Elon Musk

They’re the only ones that can make files that good, but that is literally their output. They FTP files to Taiwan.

John Collison

Do they FTP them?

Elon Musk

I believe so. I believe that File Transfer Protocol is the... But I could be wrong. But either way, it’s a bitstream going to Taiwan.

Apple doesn’t make phones. They send files to China. Microsoft doesn’t manufacture anything. Even for Xbox, that’s outsourced. Their output is digital. Meta’s output is digital. Google’s output is digital.

So if you have a human emulator, you can basically create one of the most valuable companies in the world overnight, and you would have access to trillions of dollars of revenue. It’s not a small amount.

Dwarkesh Patel

I see. You’re saying revenue figures today are all rounding errors compared to the actual TAM. So just focus on the TAM and how to get there.

Elon Musk

Take something as simple as, say, customer service. If you have to integrate with the APIs of existing corporations—many of which don’t even have an API, so you’ve got to make one, and you’ve got to wade through legacy software—that’s extremely slow.

However, if AI can simply take whatever is given to the outsourced customer service company that they already use and do customer service using the apps that they already use, then you can make tremendous headway in customer service, which is, I think, 1% of the world economy or something like that. It’s close to a trillion dollars all in, for customer service. And there’s no barriers to entry. You can immediately say, “We’ll outsource it for a fraction of the cost,” and there’s no integration needed.

John Collison

You can imagine some kind of categorization of intelligence tasks where there is breadth, where customer service is done by very many people, but many people can do it. Then there’s difficulty where there’s a best-in-class turbine engine. Presumably there’s a 10% more fuel-efficient turbine engine that could be imagined by an intelligence, but we just haven’t found it yet. Or GLP-1s are a few bytes of data…

Where do you think you want to play in this? Is it a lot of reasonably intelligent intelligence, or is it at the very pinnacle of cognitive tasks?

Elon Musk

I was just using customer service as something that’s a very significant revenue stream, but one that is probably not difficult to solve for. If you can emulate a human at a desktop, that’s what customer service is. It’s people of average intelligence. You don’t need somebody who’s spent many years. You don’t need several-sigma good engineers for that. But as you make that work, once you have effectively digital Optimus working, you can then run any application.

Let’s say you’re trying to design chips. You could then run conventional apps, stuff from Cadence and Synopsys and whatnot. You can run 1,000 or 10,000 simultaneously and say, “given this input, I get this output for the chip.” At some point, you’re going to know what the chip should look like without using any of the tools.

Basically, you should be able to do a digital chip design. You can do chip design. You march up the difficulty curve. You’d be able to do CAD. You could use NX or any of the CAD software to design things.

John Collison

So you think you start at the simplest tasks and walk your way up the difficulty curve?

Dwarkesh Patel

As a broader objective of having this full digital coworker emulator, you’re saying, “all the revenue maximizing corporations want to do this, xAI being one of them, but we will win because of a secret plan we have.” But everybody’s trying different things with data, different things with algorithms.

Elon Musk

“We tried data, we tried algorithms. What else can we do?”

Dwarkesh Patel

It seems like a competitive field. How are you guys going to win? That’s my big question.

Elon Musk

I think we see a path to doing it. I think I know the path to do this because it’s kind of the same path that Tesla used to create self-driving. Instead of driving a car, it’s driving a computer screen. It’s a self-driving computer, essentially.

John Collison

Is the path following human behavior and training on vast quantities of human behavior?

Dwarkesh Patel

Isn’t that... training?

Elon Musk

Obviously I’m not going to spell out the most sensitive secrets on a podcast. I need to have at least three more Guinnesses for that.

John Collison

What will xAI’s business be? Is it going to be consumer, enterprise? What’s the mix of those things going to be? Is it going to be similar to other labs—

Elon Musk

You’re saying “labs”. Corporations.

Dwarkesh Patel

The psyop goes deep, Elon.

Elon Musk

“Revenue maximizing corporations”, to be clear. Those GPUs don’t pay for themselves.

John Collison

Exactly. What’s the business model? What are the revenue streams in a few years’ time?

Elon Musk

Things are going to change very rapidly. I’m stating the obvious here. I call AI the supersonic tsunami. I love alliteration. What’s going to happen—especially when you have humanoid robots at scale—is that they will make products and provide services far more efficiently than human corporations. Amplifying the productivity of human corporations is simply a short-term thing.

Dwarkesh Patel

So you’re expecting fully digital corporations rather than SpaceX becoming part AI?

Elon Musk

I think there will be digital corporations but… Some of this is going to sound kind of doomerish, okay? But I’m just saying what I think will happen. It’s not meant to be doomerish or anything else. This is just what I think will happen.

Corporations that are purely AI and robotics will vastly outperform any corporations that have people in the loop. Computer used to be a job that humans had. You would go and get a job as a computer where you would do calculations. They’d have entire skyscrapers full of humans, 20-30 floors of humans, just doing calculations. Now, that entire skyscraper of humans doing calculations can be replaced by a laptop with a spreadsheet.

That spreadsheet can do vastly more calculations than an entire building full of human computers. You can think, “okay, what if only some of the cells in your spreadsheet were calculated by humans?” Actually, that would be much worse than if all of the cells in your spreadsheet were calculated by the computer. Really what will happen is that the pure AI, pure robotics corporations or collectives will far outperform any corporations that have humans in the loop. And this will happen very quickly.

Dwarkesh Patel

Speaking of closing the loop… Optimus. As far as manufacturing targets go, your companies have been carrying American manufacturing of hard tech on their back. But in the fields that Tesla has been dominant in—and now you want to go into humanoids—in China there are dozens and dozens of companies that are doing this kind of manufacturing cheaply and at scale that are incredibly competitive. So give us advice or a plan of how America can build the humanoid armies or the EVs, et cetera, at scale and as cheaply as China is on track to.

Elon Musk

There are really only three hard things for humanoid robots. The real-world intelligence, the hand, and scale manufacturing. I haven’t seen any, even demo robots, that have a great hand, with all the degrees of freedom of a human hand. Optimus will have that. Optimus does have that.

Dwarkesh Patel

How do you achieve that? Is it just the right torque density in the motor? What is the hardware bottleneck to that?

Elon Musk

We had to design custom actuators, basically custom design motors, gears, power electronics, controls, sensors. Everything had to be designed from physics first principles. There is no supply chain for this.

Dwarkesh Patel

Will you be able to manufacture those at scale?

Elon Musk

Yes.

John Collison

Is anything hard, except the hand, from a manipulation point of view? Or once you’ve solved the hand, are you good?

Elon Musk

From an electromechanical standpoint, the hand is more difficult than everything else combined. The human hand turns out to be quite something. But you also need the real-world intelligence. The intelligence that Tesla developed for the car applies very well to the robot, which is primarily vision in. The car takes in vision, but it actually also is listening for sirens. It’s taking in the inertial measurements, GPS signals, other data, combining that with video, primarily video, and then outputting the control commands.

Your Tesla is taking in one and a half gigabytes a second of video and outputting two kilobytes a second of control outputs with the video at 36 hertz and the control frequency at 18.

John Collison

One intuition you could have for when we get this robotic stuff is that it takes quite a few years to go from the compelling demo to actually being able to use it in the real world. 10 years ago, you had really compelling demos of self-driving, but only now we have Robotaxis and Waymo and all these services scaling up. Shouldn’t this make one pessimistic on household robots? Because we don’t even quite have the compelling demos yet of, say, the really advanced hand.

Elon Musk

Well, we’ve been working on humanoid robots now for a while. I guess it’s been five or six years or something. A bunch of the things that were done for the car are applicable to the robot. We’ll use the same Tesla AI chips in the robot as in the car. We’ll use the same basic principles. It’s very much the same AI.

You’ve got many more degrees of freedom for a robot than you do for a car. If you just think of it as a bitstream, AI is mostly compression and correlation of two bitstreams. For video, you’ve got to do a tremendous amount of compression and you’ve got to do the compression just right. You’ve got to ignore the things that don’t matter. You don’t care about the details of the leaves on the tree on the side of the road, but you care a lot about the road signs and the traffic lights, the pedestrians, and even whether someone in another car is looking at you or not looking at you. Some of these details matter a lot.

The car is going to turn that one and a half gigabytes a second ultimately into two kilobytes a second of control outputs. So you’ve got many stages of compression. You’ve got to get all those stages right and then correlate those to the correct control outputs. The robot has to do essentially the same thing.

This is what happens with humans. We really are photons in, controls out. That is the vast majority of your life: vision, photons in, and then motor controls out.

Dwarkesh Patel

Naively, it seems that between humanoid robots and cars… The fundamental actuators in a car are how you turn, how you accelerate. In a robot, especially with maneuverable arms, there’s dozens and dozens of these degrees of freedom. Then especially with Tesla, you had this advantage of millions and millions of hours of human demo data collected from the car being out there. You can’t equivalently deploy Optimuses that don’t work and then get the data that way. So between the increased degrees of freedom and the far sparser data...

Elon Musk

Yes. That’s a good point.

Dwarkesh Patel

How will you use the Tesla engine of intelligence to train the Optimus mind?

Elon Musk

You’re actually highlighting an important limitation and difference from cars. We’ll soon have 10 million cars on the road. It’s hard to duplicate that massive training flywheel. For the robot, what we’re going to need to do is build a lot of robots and put them in kind of an Optimus Academy so they can do self-play in reality. We’re actually building that out. We can have at least 10,000 Optimus robots, maybe 20-30,000, that are doing self-play and testing different tasks.

Tesla has quite a good reality generator, a physics-accurate reality generator, that we made for the cars. We’ll do the same thing for the robots. We actually have done that for the robots. So you have a few tens of thousands of humanoid robots doing different tasks. You can do millions of simulated robots in the simulated world. You use the tens of thousands of robots in the real world to close the simulation to reality gap. Close the sim-to-real gap.

Dwarkesh Patel

How do you think about the synergies between xAI and Optimus, given you’re highlighting that you need this world model, you want to use some really smart intelligence as a control plane, and Grok is doing the slower planning, and then the motor policy is a little lower level. What will the synergy between these things be?

Elon Musk

Grok would orchestrate the behavior of the Optimus robots. Let’s say you wanted to build a factory. Grok could organize the Optimus robots, assign them tasks to build the factory to produce whatever you want.

John Collison

Don’t you need to merge xAI and Tesla then? Because these things end up so...

Elon Musk

What were we saying earlier about public company discussions?

Dwarkesh Patel

We’re one more Guinness in, Elon. What are you waiting to see before you say, we want to manufacture 100,000 Optimuses?

Elon Musk

“Optimi”. Since we’re defining the proper noun, we’re going to define the plural of the proper noun too. We’re going to proper noun the plural and so it’s Optimi.

Dwarkesh Patel

Is there something on the hardware side you want to see? Do you want to see better actuators? Is it just that you want the software to be better? What are we waiting for before we get mass manufacturing of Gen 3?

Elon Musk

No, we’re moving towards that. We’re moving forward with the mass manufacturing.

Dwarkesh Patel

But you think current hardware is good enough that you just want to deploy as many as possible now?

Elon Musk

It’s very hard to scale up production. But I think Optimus 3 is the right version of the robot to produce something on the order of a million units a year. I think you’d want to go to Optimus 4 before you went to 10 million units a year.

John Collison

Okay, but you can do a million units at Optimus 3?

Elon Musk

It’s very hard to spool up manufacturing. The output per unit time always follows an S-curve. It starts off agonizingly slow, then it has this exponential increase, then a linear, then a logarithmic outcome until you eventually asymptote at some number. Optimus’ initial production will be a stretched out S-curve because so much of what goes into Optimus is brand new. There is not an existing supply chain.

The actuators, electronics, everything in the Optimus robot is designed from physics first principles. It’s not taken from a catalog. These are custom-designed everything. I don’t think there’s a single thing—

John Collison

How far down does that go?

Elon Musk

I guess we’re not making custom capacitors yet, maybe. There’s nothing you can pick out of a catalog, at any price. It just means that the Optimus S-Curve, the output per unit time, how many Optimus robots you make per day, is going to initially ramp slower than a product where you have an existing supply chain. But it will get to a million.

Dwarkesh Patel

When you see these Chinese humanoids, like Unitree or whatever, sell humanoids for like $6K or $13K, are you hoping to get your Optimus bill of materials below that price so you can do the same thing? Or do you just think qualitatively they’re not the same thing? What allows them to sell for so low? Can we match that?

Elon Musk

Our Optimus is designed to have a lot of intelligence and to have the same electromechanical dexterity, if not higher, as a human. Unitree does not have that. It’s also quite a big robot. It has to carry heavy objects for long periods of time and not overheat or exceed the power of its actuators. It’s 5’11”, so it’s pretty tall. It’s got a lot of intelligence. So it’s going to be more expensive than a small robot that is not intelligent.

John Collison

But more capable.

Elon Musk

But not a lot more. The thing is, over time as Optimus robots build Optimus robots, the cost will drop very quickly.

John Collison

What will these first billion Optimuses, Optimi, do? What will their highest and best use be?

Elon Musk

I think you would start off with simple tasks that you can count on them doing well.

John Collison

But in the home or in factories?

Elon Musk

The best use for robots in the beginning will be any continuous operation, any 24/7 operation, because they can work continuously.

Dwarkesh Patel

What fraction of the work at a Gigafactory that is currently done by humans could a Gen 3 do?

Elon Musk

I’m not sure. Maybe it’s 10-20%, maybe more, I don’t know. We would not reduce our headcount. We would increase our headcount, to be clear. But we would increase our output. The units produced per human... The total number of humans at Tesla will increase, but the output of robots and cars will increase disproportionately. The number of cars and robots produced per human will increase dramatically, but the number of humans will increase as well.

John Collison

We’re talking about Chinese manufacturing a bunch here. We’ve also talked about some of the policies that are relevant, like you mentioned, the solar tariffs. You think they’re a bad idea because we can’t scale up solar in the US.

Elon Musk

Electricity output in the US needs to scale up.

John Collison

It can’t without good power sources.

Elon Musk

You just need to get it somehow.

John Collison

Where I was going with this is, if you were in charge, if you were setting all the policies, what else would you change? You’d change the solar tariffs, that’s one.

Elon Musk

I would say anything that is a limiting factor for electricity needs to be addressed, provided it’s not very bad for the environment.

John Collison

So presumably some permitting reforms and stuff as well would be in there?

Elon Musk

There’s a fair bit of permitting reforms that are happening. A lot of the permitting is state-based, but anything federal... This administration is good at removing permitting roadblocks.

I’m not saying all tariffs are bad.

John Collison

Solar tariffs.

Elon Musk

Sometimes if another country is subsidizing the output of something, then you have to have countervailing tariffs to protect domestic industry against subsidies by another country.

John Collison

What else would you change?

Elon Musk

I don’t know if there’s that much that the government can actually do.

John Collison

One thing I was wondering... For the policy goal of creating a lead for the US versus China, it seems like the export bans have actually been quite impactful, where China is not producing leading-edge chips and the export bans really bite there. China is not producing leading-edge turbine engines. Similarly, there’s a bunch of export bans that are relevant there on some of the metallurgy. Should there be more export bans? As you think about things like the drone industry and things like that, is that something that should be considered?

Elon Musk

It’s important to appreciate that in most areas, China is very advanced in manufacturing. There’s only a few areas where it is not. China is a manufacturing powerhouse, next-level.

John Collison

It’s very impressive.

Elon Musk

If you take refining of ore, China does roughly twice as much ore refining on average as the rest of the world combined. There are some areas, like refining gallium which goes into solar cells. I think they are 98% of gallium refining. So China is actually very advanced in manufacturing in most areas.

John Collison

It seems like there is discomfort with this supply chain dependence, and yet nothing’s really happening on it.

Elon Musk

Supply chain dependence?

John Collison

Say, like the gallium refining that you’re saying. All the rare-earth stuff.

Elon Musk

Rare earths for sure, as you know, they’re not rare. We actually do rare earth ore mining in the US, send the rock, put it on a train, and then put it on a boat to China that goes to another train, and goes to the rare earth refiners in China who then refine it, put it into a magnet, put it into a motor sub-assembly, and then send it back to America. So the thing we’re really missing is a lot of ore refining in America.

John Collison

Isn’t this worth a policy intervention?

Elon Musk

Yes. I think there are some things being done on that front. But we kind of need Optimus, frankly, to build ore refineries.

Dwarkesh Patel

So, you think the main advantage China has is the abundance of skilled labor? That’s the thing Optimus fixes?

Elon Musk

Yes. China’s got like four times our population.

Dwarkesh Patel

I mean, there’s this concern. If you think human resources are the future, right now if it’s the skilled labor for manufacturing that’s determining who can build more humanoids, China has more of those. It manufactures more humanoids, therefore it gets the Optimi future first.

Elon Musk

Well, we’ll see. Maybe.

Dwarkesh Patel

It just keeps that exponential going. It seems like you’re sort of pointing out that getting to a million Optimi requires the manufacturing that the Optimi is supposed to help us get to. Right?

Elon Musk

You can close that recursive loop pretty quickly.

John Collison

With a small number of Optimi?

Elon Musk

Yeah. So you close the recursive loop to help the robots build the robots. Then we can try to get to tens of millions of units a year. Maybe. If you start getting to hundreds of millions of units a year, you’re going to be the most competitive country by far.

We definitely can’t win with just humans, because China has four times our population. Frankly, America has been winning for so long that… A pro sports team that’s been winning for a very long time tends to get complacent and entitled. That’s why they stop winning, because they don’t work as hard anymore. So frankly my observation is just that the average work ethic in China is higher than in the US. It’s not just that there’s four times the population, but the amount of work that people put in is higher.

So you can try to rearrange the humans, but you’re still one quarter of the—assuming that productivity is the same, which I think actually it might not be, I think China might have an advantage on productivity per person—we will do one quarter of the amount of things as China. So we can’t win on the human front.

Our birth rate has been low for a long time. The US birth rate’s been below replacement since roughly 1971. We’ve got a lot of people retiring, we’re close to more people domestically dying than being born. So we definitely can’t win on the human front, but we might have a shot at the robot front.

John Collison

Are there other things that you have wanted to manufacture in the past, but they’ve been too labor intensive or too expensive that now you can come back to and say, “oh, we can finally do the whatever, because we have Optimus?”

Elon Musk

Yeah, we’d like to build more ore refineries at Tesla. We just completed construction and have begun lithium refining with our lithium refinery in Corpus Christi, Texas. We have a nickel refinery, which is for the cathode, that’s here in Austin. This is the largest cathode refinery, largest nickel and lithium refinery, outside of China.

The cathode team would say, “we have the largest and the only, actually, cathode refinery in America.” Not just the largest, but it’s also the only.

John Collison

Many superlatives.

Elon Musk

So it was pretty big, even though it’s the only one. But there are other things. You could do a lot more refineries and help America be more competitive on refining capacity. There’s basically a lot of work for the Optimus to do that most Americans, very few Americans, frankly want to do.

John Collison

Is the refining work too dirty or what’s the—

Elon Musk

It’s not actually, no. We don’t have toxic emissions from the refinery or anything. The cathode nickel refinery is in Travis County.

John Collison

Why can’t you do it with humans?

Elon Musk

You can, you just run out of humans.

John Collison

Ah, I see. Okay.

Elon Musk

No matter what you do, you have one quarter of the number of humans in America than China. So if you have them do this thing, they can’t do the other thing. So then how do you build this refining capacity? Well, you could do it with Optimi.

Not very many Americans are pining to do refining. I mean, how many have you run into? Very few. Very few pining to refine.

Dwarkesh Patel

BYD is reaching Tesla production or sales in quantity. What do you think happens in global markets as Chinese production in EVs scales up?

Elon Musk

China is extremely competitive in manufacturing. So I think there’s going to be a massive flood of Chinese vehicles and basically most manufactured things. As it is, as I said, China is probably doing twice as much refining as the rest of the world combined. So if you go down to fourth and fifth-tier supply chain stuff…

At the base level, you’ve got energy, then you’ve got mining and refining. Those foundation layers are, like I said, as a rough guess, China’s doing twice as much refining as the rest of the world combined. So any given thing is going to have Chinese content because China’s doing twice as much refining work as the rest of the world. But they’ll go all the way to the finished product with the cars.

I mean China is a powerhouse. I think this year China will exceed three times US electricity output. Electricity output is a reasonable proxy for the economy. In order to run the factories and run everything, you need electricity. It’s a good proxy for the real economy. If China passes three times the US electricity output, it means that its industrial capacity—as rough approximation—will be three times that of the US.

Dwarkesh Patel

Reading between the lines, it sounds like what you’re saying is absent some sort of humanoid recursive miracle in the next few years, on the whole manufacturing/energy/raw materials chain, China will just dominate whether it comes to AI or manufacturing EVs or manufacturing humanoids.

Elon Musk

In the absence of breakthrough innovations in the US, China will utterly dominate.

Dwarkesh Patel

Interesting.

Elon Musk

Yes.

John Collison

Robotics being the main breakthrough innovation.

Elon Musk

Well, to scale AI in space, basically you need humanoid robots, you need real-world AI, you need a million tons a year to orbit. Let’s just say if we get the mass driver on the moon going, my favorite thing, then I think—

John Collison

We’ll have solved all our problems.

Elon Musk

I call that winning. I call it winning, big time.

John Collison

You can finally be satisfied. You’ve done something.

Elon Musk

Yes.

John Collison

You have the mass driver on the moon.

Elon Musk

I just want to see that thing in operation.

John Collison

Was that out of some sci-fi or where did you…?

Elon Musk

Well, actually, there is a Heinlein book. The Moon is a Harsh Mistress.

John Collison

Okay, yeah, but that’s slightly different. That’s a gravity slingshot or...

Elon Musk

No, they have a mass driver on the Moon.

John Collison

Okay, yeah, but they use that to attack Earth. So maybe it’s not the greatest...

Elon Musk

Well they use that to… assert their independence.

John Collison

Exactly. What are your plans for the mass driver on the Moon?

Elon Musk

They asserted their independence. Earth government disagreed and they lobbed things until Earth government agreed.

John Collison

That book is a hoot. I found that book much better than his other one that everyone reads, Stranger in a Strange Land.

Elon Musk

“Grok” comes from Stranger in a Strange Land. The first two-thirds of Stranger in a Strange Land are good, and then it gets very weird in the third portion. But there are still some good concepts in there.

John Collison

One thing we were discussing a lot is your system for managing people. You interviewed the first few thousand of SpaceX employees and lots of other companies.

Elon Musk

It obviously doesn’t scale.

John Collison

Well, yes, but what doesn’t scale?

Elon Musk

Me.

John Collison

Sure, sure. I know that. But what are you looking for?

Elon Musk

There literally are not enough hours in the day. It’s impossible.

John Collison

But what are you looking for that someone else who’s good at interviewing and hiring people… What’s the je ne sais quoi?

Elon Musk

At this point, I might have more training data on evaluating technical talent especially—talent of all kinds I suppose, but technical talent especially—given that I’ve done so many technical interviews and then seen the results. So my training set is enormous and has a very wide range.

Generally, the things I ask for are bullet points for evidence of exceptional ability. These things can be pretty off the wall. It doesn’t need to be in the specific domain, but evidence of exceptional ability. So if somebody can cite even one thing, but let’s say three things, where you go, “Wow, wow, wow,” then that’s a good sign.

Dwarkesh Patel

Why do you have to be the one to determine that?

Elon Musk

No, I don’t. I can’t be. It’s impossible. The total headcount across all companies is 200,000 people.

John Collison

But in the early days, what was it that you were looking for that couldn’t be delegated in those interviews?

Elon Musk

I guess I need to build my training set. It’s not like I batted a thousand here. I would make mistakes, but then I’d be able to see where I thought somebody would work out well, but they didn’t. Then why did they not work out well? What can I do, I guess RL myself, to in the future have a better batting average when interviewing people? My batting average is still not perfect, but it’s very high.

Dwarkesh Patel

What are some surprising reasons people don’t work out?

Elon Musk

Surprising reasons…

Dwarkesh Patel

Like, they don’t understand technical domain, et cetera, et cetera. But you’ve got the long tail now of like, “I was really excited about this person. It didn’t work out.” Curious why that happens.

Elon Musk

Generally what I tell people—I tell myself, I guess, aspirationally—is, don’t look at the resume. Just believe your interaction. The resume may seem very impressive and it’s like, “Wow, the resume looks good.” But if the conversation after 20 minutes is not “wow,” you should believe the conversation, not the paper.

John Collison

I feel like part of your method is that… There was this meme in the media a few years back about Tesla being a revolving door of executive talent. Whereas actually, I think when you look at it, Tesla’s had a very consistent and internally promoted executive bench over the past few years.

Then at SpaceX, you have all these folks like Mark Juncosa and Steve Davis—

Elon Musk

Steve Davis runs The Boring Company these days.

John Collison

Bill Riley, and folks like that. It feels like part of what has worked well is having very capable technical deputies. What do all of those people have in common?

Elon Musk

Well, the Tesla senior team, at this point has probably got an average tenure of 10-12 years. It’s quite long. But there were times when Tesla went through an extremely rapid growth phase, so things were just somewhat sped up. As you know, a company goes through different orders of magnitude of size. People that could help manage, say, a 50-person company versus a 500-person company versus a 5,000-person company versus a 50,000-person company.

John Collison

You outgrew people.

Elon Musk

It’s just not the same team. It’s not always the same team. So if a company is growing very rapidly, the rate at which executive positions will change will also be proportionate to the rapidity of the growth generally.

Tesla had a further challenge where when Tesla had very successful periods, we would be relentlessly recruited from. Like, relentlessly. When Apple had their electric car program, they were carpet bombing Tesla with recruiting calls. Engineers just unplugged their phones.

John Collison

“I’m trying to get work done here.”

Elon Musk

Yeah. “If I get one more call from an Apple recruiter…” But their opening offer without any interview would be like double the compensation at Tesla. So we had a bit of the “Tesla pixie dust” thing where it’s like, “Oh, if you hire a Tesla executive, suddenly everything’s going to be successful.”

I’ve fallen prey to the pixie dust thing as well, where it’s like, “Oh, we’ll hire someone from Google or Apple and they’ll be immediately successful,” but that’s not how it works. People are people. There’s no magical pixie dust. So when we had the pixie dust problem, we would get relentlessly recruited from.

Also, Tesla being engineering, especially being primarily in Silicon Valley, it’s easier for people to just... They don’t have to change their life very much. Their commute’s going to be the same.

John Collison

So how do you prevent that? How do you prevent the pixie dust effect where everyone’s trying to poach all your people?

Elon Musk

I don’t think there’s much we can do to stop it. That’s one of the reasons why Tesla… Really, being in Silicon Valley and having the pixie dust thing at the same time meant that there was just a very, very aggressive recruitment.

John Collison

Presumably being in Austin helps then?

Elon Musk

Austin, it helps. Tesla still has a majority of its engineering in California. Getting engineers to move… I call it the “significant other” problem.

John Collison

Yes, “significant others” have jobs.

Elon Musk

Exactly. So for Starbase that was particularly difficult, since the odds of finding a non-SpaceX job…

John Collison

In Brownsville, Texas…

Elon Musk

…are pretty low. It’s quite difficult. It’s like a technology monastery thing, remote and mostly dudes.

Dwarkesh Patel

Not much of an improvement over SF.

John Collison

If you go back to these people who’ve really been very effective in a technical capacity at Tesla, at SpaceX, and those sorts of places, what do you think they have in common other than... Is it just that they’re very sharp on the rocketry or the technical foundations, or do you think it’s something organizational?

Is it something about their ability to work with you? Is it their ability to be flexible but not too flexible? What makes a good sparring partner for you?

Elon Musk

I don’t think of it as a sparring partner. If somebody gets things done, I love them, and if they don’t, I hate them. So it’s pretty straightforward. It’s not like some idiosyncratic thing. If somebody executes well, I’m a huge fan, and if they don’t, I’m not. But it’s not about mapping to my idiosyncratic preferences. I certainly try not to have it be mapping to my idiosyncratic preferences.

Generally, I think it’s a good idea to hire for talent and drive and trustworthiness. And I think goodness of heart is important. I underweighted that at one point. So, are they a good person? Trustworthy? Smart and talented and hard working? If so, you can add domain knowledge.

But those fundamental traits, those fundamental properties, you cannot change. So most of the people who are at Tesla and SpaceX did not come from the aerospace industry or the auto industry.

Dwarkesh Patel

What has had to change most about your management style as your companies have scaled from 100 to 1,000 to 10,000 people? You’re known for this very micro management, just getting into the details of things.

Elon Musk

Nano management, please. Pico management. Femto management.

John Collison

Keep going.

Elon Musk

We’re going to go all the way down to Planck’s constant. All the way down to Heisenberg uncertainty principle.

Dwarkesh Patel

Are you still able to get into details as much as you want? Would your companies be more successful if they were smaller? How do you think about that?

Elon Musk

Because I have a fixed amount of time in the day, my time is necessarily diluted as things grow and as the span of activity increases. It’s impossible for me to actually be a micromanager because that would imply I have some thousands of hours per day. It is a logical impossibility for me to micromanage things.

Now, there are times when I will drill down into a specific issue because that specific issue is the limiting factor on the progress of the company. The reason for drilling into some very detailed item is because it is the limiting factor. It’s not arbitrarily drilling into tiny things.

From a time standpoint, it is physically impossible for me to arbitrarily go into tiny things that don’t matter. That would result in failure. But sometimes the tiny things are decisive in victory.

John Collison

Famously, you switched the Starship design from composites to steel.

Elon Musk

Yes.

John Collison

You made that decision. That wasn’t people going around saying, “Oh, we found something better, boss.” That was you encouraging people against some resistance. Can you tell us how you came to that whole concept of the steel switch?

Elon Musk

Desperation, I’d say. Originally, we were going to make Starship out of carbon fiber. Carbon fiber is pretty expensive. When you do volume production, you can get any given thing to start to approach its material cost.

The problem with carbon fiber is that material cost is still very high. Particularly if you go for a high-strength specialized carbon fiber that can handle cryogenic oxygen, it’s roughly 50 times the cost of steel. At least in theory, it would be lighter. People generally think of steel as being heavy and carbon fiber as being light.

For room temperature applications, like a Formula 1 car, static aero structure, or any kind of aero structure really, you’re probably going to be better off with carbon fiber. The problem is that we were trying to make this enormous rocket out of carbon fiber and our progress was extremely slow.

John Collison

It had been picked in the first place just because it’s light?

Elon Musk

Yes. At first glance, most people would think that the choice for making something light would be carbon fiber. The thing is that when you make something very enormous out of carbon fiber and then you try to have the carbon fiber be efficiently cured, meaning not room temperature cured, because sometimes you got 50 plies of carbon fiber… Carbon fiber is really carbon string and glue. In order to have high strength, you need an autoclave. Something that’s essentially a high pressure oven. If you have something that’s gigantic, that one’s got to be bigger than the rocket.

We were trying to make an autoclave that’s bigger than any autoclave that’s ever existed. Or you can do room temperature cure, which takes a long time and has issues. The final issue is that we were just making very slow progress with carbon fiber.

Dwarkesh Patel

The meta question is why it had to be you who made that decision. There’s many engineers on your team.

John Collison

How did the team not arrive at steel?

Dwarkesh Patel

Yeah exactly. This is part of a broader question, understanding your comparative advantage at your companies.

Elon Musk

Because we were making very slow progress with carbon fiber, I was like, “Okay, we’ve got to try something else.” For the Falcon 9, the primary airframe is made of aluminum lithium, which has a very good strength-to-weight. Actually, it has about the same, maybe better, strength to weight for its application than carbon fiber. But aluminum lithium is very difficult to work with.

In order to weld it, you have to do something called friction stir welding, where you join the metal without entering the liquid phase. It’s kind of wild that you can do that. But with this particular type of welding, you can do that. It’s very difficult. Let’s say you want to make a modification or attach something to aluminum lithium, you now have to use a mechanical attachment with seals. You can’t weld it on. So I wanted to avoid using aluminum lithium for the primary structure for Starship.

There was this very special grade of carbon fiber that had very good mass properties. With a rocket, you’re really trying to maximize the percentage of the rocket that is propellant, minimize the mass obviously. But like I said, we were making very slow progress. I said, “at this rate, we’re never going to get to Mars. So we’ve got to think of something else.”

I didn’t want to use aluminum lithium because of the difficulty of friction stir welding, especially doing that at scale. It was hard enough at 3.6 meters in diameter, let alone at 9 meters or above. Then I said, “what about steel?”

I had a clue here because some of the early US rockets had used very thin steel. The Atlas rockets had used a steel balloon tank. It’s not like steel had never been used before. It actually had been used. When you look at the material properties of stainless steel, full-hard, strain hardened stainless steel, at cryogenic temperature the strength to weight is actually similar to carbon fiber.

If you look at material properties at room temperature, it looks like the steel is going to be twice as heavy. But if you look at the material properties at cryogenic temperature of full-hard steel, stainless of particular grades, then you actually get to a similar strength to weight as carbon fiber.

In the case of Starship, both the fuel and the oxidizer are cryogenic. For Falcon 9, the fuel is rocket propellant-grade kerosene, basically a very pure form of jet fuel. That is roughly room temperature. Although we do actually chill it slightly below, we chill it like a beer.

John Collison

Delicious.

Elon Musk

We do chill it, but it’s not cryogenic. In fact, if we made it cryogenic, it would just turn to wax. But for Starship, it’s liquid methane and liquid oxygen. They are liquid at similar temperatures. Basically, almost the entire primary structure is at cryogenic temperature. So then you’ve got a 300-series stainless that’s strain hardened. Because almost all things are cryogenic temperature, it actually has similar strength to weight as carbon fiber.

But it costs 50x less in raw material and is very easy to work with. You can weld stainless steel outdoors. You could smoke a cigar while welding stainless steel. It’s very resilient. You can modify it easily. If you want to attach something, you just weld it right on. Very easy to work with, very low cost.

Like I said, at cryogenic temperature, it’s similar strength-to-weight to carbon fiber. Then when you factor in that we have a much reduced heat shield mass, because the melting point of steel, is much greater than the melting point of aluminum… It’s about twice the melting point of aluminum.

John Collison

So you can just run the rocket much hotter?

Elon Musk

Yes, especially for the ship which is coming in like a blazing meteor. You can greatly reduce the mass of the heat shield. You can cut the mass of the windward part of the heat shield, maybe in half, and you don’t need any heat shielding on the leeward side.

The net result is that actually the steel rocket weighs less than the carbon fiber rocket, because the resin in the carbon fiber rocket starts to melt. Basically, carbon fiber and aluminum have about the same operating temperature capabilities, whereas steel can operate at twice the temperature. These are very rough approximations.

John Collison

I won’t build the rocket.

Elon Musk

What I mean is people will say, “Oh, he said this twice. It’s actually 0.8.” I’m like, shut up, assholes.

Dwarkesh Patel

That’s what the main comment’s going to be about.

Elon Musk

God damn it. The point is, in retrospect, we should have started with steel in the beginning. It was dumb not to do steel.

John Collison

Okay, but to play this back to you, what I’m hearing is that steel was a riskier, less proven path, other than the early US rockets. Versus carbon fiber was a worse but more proven out path. So you need to be the one to push for, “Hey, we’re going to do this riskier path and just figure it out.” So you’re fighting a sort of conservatism in a sense.

Elon Musk

That’s why I initially said that the issue is that we weren’t making fast enough progress. We were having trouble making even a small barrel section of the carbon fiber that didn’t have wrinkles in it. Because at that large scale, you have to have many plies, many layers of the carbon fiber. You’ve got to cure it and you’ve got to cure it in such a way that it doesn’t have any wrinkles or defects.

Carbon fiber is much less resilient than steel. It has much less toughness. Stainless steel will stretch and bend, the carbon fiber will tend to shatter. Toughness being the area under the stress strain curve. You’re generally going to have to do better with steel, but stainless steel to be precise.

John Collison

One other Starship question. So I visited Starbase, I think it was two years ago, with Sam Teller, and that was awesome. It was very cool to see, in a whole bunch of ways.

One thing I noticed was that people really took pride in the simplicity of things, where everyone wants to tell you how Starship is just a big soda can, and we’re hiring welders, and if you can weld in any industrial project, you can weld here. But there’s a lot of pride in the simplicity.

Elon Musk

Well, factually Starship is a very complicated rocket.

John Collison

So that’s what I’m getting at. Are things simple or are they complex?

Elon Musk

I think maybe just what they’re trying to say is that you don’t have to have prior experience in the rocket industry to work on Starship. Somebody just needs to be smart and work hard and be trustworthy and they can work on a rocket. They don’t need prior rocket experience. Starship is the most complicated machine ever made by humans, by a long shot.

John Collison

In what regards?

Elon Musk

Anything, really. I’d say there isn’t a more complex machine. I’d say that pretty much any project I can think of would be easier than this. That’s why nobody has ever made a fully reusable orbital rocket. It’s a very hard problem. Many smart people have tried before, very smart people with immense resources, and they failed.

And we haven’t succeeded yet. Falcon is partially reusable, but the upper stage is not. Starship Version 3, I think this design can be fully reusable. That full reusability is what will enable us to become a multi-planet civilization. Any technical problem, even like a Hadron Collider or something like that, is an easier problem than this.

John Collison

We spent a lot of time on bottlenecks. Can you say what the current Starship bottlenecks are, even at a high level?

Elon Musk

Trying to make it not explode, generally. It really wants to explode.

John Collison

That old chestnut. All those combustible materials.

Elon Musk

We’ve had two boosters explode on the test stand. One obliterated the entire test facility. So it only takes that one mistake. The amount of energy contained in a Starship is insane.

John Collison

Is that why it’s harder than Falcon? It’s because it’s just more energy?

Elon Musk

It’s a lot of new technology. It’s pushing the performance envelope. The Raptor 3 engine is a very, very advanced engine. It’s by far the best rocket engine ever made. But it desperately wants to blow up. Just to put things into perspective here, on liftoff the rocket is generating over 100 gigawatts of power. That’s 20% of US electricity.

Dwarkesh Patel

It’s actually insane.

John Collison

It’s a great comparison.

Elon Musk

While not exploding.

John Collison

Sometimes.

Elon Musk

Sometimes, yes. So I was like, how does it not explode? There’s thousands of ways that it could explode and only one way that it doesn’t. So we want it not only to really not explode, but fly reliably on a daily basis, like once per hour. Obviously, if it blows up a lot, it’s very difficult to maintain that launch cadence.

John Collison

Yes.

Elon Musk

What’s the single biggest remaining problem for Starship? It’s having the heat shield be reusable. No one’s ever made a reusable orbital heat shield. So the heat shield’s gotta make it through the ascent phase without shucking a bunch of tiles, and then it’s gotta come back in and also not lose a bunch of tiles or overheat the main airframe.

John Collison

Isn’t that hard because it’s fundamentally a consumable?

Elon Musk

Well, yes, but your brake pads in your car are also consumable, but they last a very long time.

John Collison

Fair.

Elon Musk

So it just needs to last a very long time. We have brought the ship back and had it do a soft landing in the ocean. We’ve done that a few times. But it lost a lot of tiles. It was not reusable without a lot of work. Even though it did come to a soft landing, it would not have been reusable without a lot of work.

So it’s not really reusable in that sense. That’s the biggest problem that remains, a fully reusable heat shield. You want to be able to land it, refill propellant and fly again. You can’t do this laborious inspection of 40,000 tiles type of thing.

Dwarkesh Patel

When I read biographies of yours, it seems like you’re just able to drive the sense of urgency and drive the sense of “this is the thing that can scale.” I’m curious why you think other organizations of your…

SpaceX and Tesla are really big companies now. You’re still able to keep that culture. What goes wrong with other companies such that they’re not able to do that?

Elon Musk

I don’t know.

Dwarkesh Patel

Like today, you said you had a bunch of SpaceX meetings. What is it that you’re doing there that’s keeping that?

John Collison

It’s adding urgency?

Elon Musk

Well, I don’t know. I guess the urgency is going to come from whoever is leading the company. I have a maniacal sense of urgency. So that maniacal sense of urgency projects through the rest of the company.

Dwarkesh Patel

Is it because of consequences? They’re like, “Elon set a crazy deadline, but if I don’t get it, I know what happens to me.” Is it just that you’re able to identify bottlenecks and get rid of them so people can move fast? How do you think about why your companies are able to move fast?

Elon Musk

I’m constantly addressing the limiting factor. On the deadlines front, I generally actually try to aim for a deadline that I at least think is at the 50th percentile. So it’s not like an impossible deadline, but it’s the most aggressive deadline I can think of that could be achieved with 50% probability. Which means that it’ll be late half the time.

There is a law of gas expansion that applies to schedules. If you said we’re going to do something in five years, which to me is like infinity time, it will expand to fill the available schedule and it’ll take five years.

Physics will limit how fast you can do certain things. So scaling up manufacturing, there’s a rate at which you can move the atoms and scale manufacturing. That’s why you can’t instantly make a million units a year of something. You’ve got to design the manufacturing line. You’ve got to bring it up. You’ve got to ride the S-curve of production.

What can I say that’s actually helpful to people? Generally, a maniacal sense of urgency is a very big deal. You want to have an aggressive schedule and you want to figure out what the limiting factor is at any point in time and help the team address that limiting factor.

John Collison

So Starlink was slowly in the works for many years.

Elon Musk

We talked about it all the way in the beginning of the company.

John Collison

So then there was a team you had built in Redmond, and then at one point you decided this team is just not cutting it. It went for a few years slowly, and so why didn’t you act earlier, and why did you act when you did? Why was that the right moment at which to act?

Elon Musk

I have these very detailed engineering reviews weekly. That’s maybe a very unusual level of granularity. I don’t know anyone who runs a company, or at least a manufacturing company, that goes with the level of detail that I go into. It’s not as though... I have a pretty good understanding of what’s actually going on because we go through things in detail.

I’m a big believer in skip-level meetings where instead of having the person that reports to me say things, it’s everyone that reports to them saying something in the technical review. And there can’t be advanced preparation. Otherwise you’re going to get “glazed”, as I say these days.

John Collison

Exactly. Very Gen Z of you.

Dwarkesh Patel

How do you prevent advanced preparation? Do you call on them randomly?

Elon Musk

No, I just go around the room. Everyone provides an update. It’s a lot of information to keep in your head. If you have meetings weekly or twice weekly, you’ve got a snapshot of what that person said. You can then plot the progress points. You can sort of mentally plot the points on a curve and say, “are we converging to a solution or not?”

I’ll take drastic action only when I conclude that success is not in a set of possible outcomes. So when I finally reach the conclusion that unless drastic action is done, we have no chance of success, then I must take drastic action. I came to that conclusion in 2018, took drastic action and fixed the problem.

Dwarkesh Patel

You’ve got many, many companies. In each of them it sounds like you do this kind of deep engineering understanding of what the relevant bottlenecks are so you can do these reviews with people.

You’ve been able to scale it up to five, six, seven companies. Within one of these companies, you have many different mini companies within them. What determines the max amount here? Because you have like 80 companies…?

Elon Musk

80? No.

Dwarkesh Patel

But you have so many already. That’s already remarkable.

John Collison

By this current number.

Dwarkesh Patel

Exactly.

John Collison

We can barely keep one company together.

Elon Musk

It depends on the situation. I actually don’t have regular meetings with The Boring Company, so The Boring Company is sort of cruising along. Basically, if something is working well and making good progress, then there’s no point in me spending time on it.

I actually allocate time according to where the limiting factor. Where are things problematic? Where are we pushing against? What is holding us back? I focus, at the risk of saying the words too many times, on the limiting factor.

Elon Musk

The irony is if something’s going really well, they don’t see much of me. But if something is going badly, they’ll see a lot of me. Or not even badly…

John Collison

If something is the limiting factor.

Elon Musk

The limiting factor, exactly. It’s not exactly going badly but it’s the thing that we need to make go faster.

John Collison

When something’s a limiting factor at SpaceX or Tesla, are you talking weekly and daily with the engineer that’s working on it? How does that actually work?

Elon Musk

Most things that are the limiting factor are weekly and some things are twice weekly. The AI5 chip review is twice weekly. Every Tuesday and Saturday is the chip review.

John Collison

Is it open ended in how long it goes?

Elon Musk

Technically, yes, but usually it’s two or three hours. Sometimes less. It depends on how much information we’ve got to go through.

John Collison

That’s another thing. I’m just trying to tease out the differences here because the outcomes seem quite different. I think it’s interesting to know what inputs are different. It feels like in the corporate world, one, like you were saying, the CEO doing engineering reviews does not always happen despite the fact that that is what the company is doing.

But then time is often pretty finely sliced into half hour meetings or even 15 minute meetings. It seems like you hold more open-ended, “We’re talking about it until we figure it out” type things.

Elon Musk

Sometimes. But most of them seem to more or less stay on time. Today’s Starship engineering review went a bit longer because there were more topics to discuss. They’re trying to figure out how to scale to a million plus tons to orbit per year. It’s quite challenging.

Dwarkesh Patel

Can I ask a question? You said about Optimus and AI that they’re going to result in double digit growth rates within a matter of years.

Elon Musk

Oh, like the economy? Yes. I think that’s right.

Dwarkesh Patel

What was the point of the DOGE cuts if the economy is going to grow so much?

Elon Musk

Well, I think waste and fraud are not good things to have. I was actually pretty worried about... In the absence of AI and robotics, we’re actually totally screwed because the national debt is piling up like crazy. The interest payments to national debt exceed the military budget, which is a trillion dollars. So we have over a trillion dollars just in interest payments. I was pretty concerned about that. Maybe if I spend some time, we can slow down the bankruptcy of the United States and give us enough time for the AI and robots to help solve the national debt.

Or not help solve, it’s the only thing that could solve the national debt. We are 1000% going to go bankrupt as a country, and fail as a country, without AI and robots. Nothing else will solve the national debt. We just need enough time to build the AI and robots to not go bankrupt before then.

Dwarkesh Patel

I guess the thing I’m curious about is, when DOGE starts you have this enormous ability to enact reform.

Elon Musk

Not that enormous.

Dwarkesh Patel

Sure. I totally buy your point that it’s important that AI and robotics drive productivity improvements, drive GDP growth. But why not just directly go after the things you were pointing out, like the tariffs on certain components, or permitting?

Elon Musk

I’m not the president. And it is very hard to cut things that are obvious waste and fraud, like ridiculous waste and fraud. What I discovered is that it’s extremely difficult even to cut very obvious waste and fraud from the government because the government has to operate on who’s complaining.

If you cut off payments to fraudsters, they immediately come up with the most sympathetic sounding reasons to continue the payment. They don’t say, “Please keep the fraud going.” They’re like, “You’re killing baby pandas.” Meanwhile, no baby pandas are dying. They’re just making it up. The fraudsters are capable of coming up with extremely compelling, heart-wrenching stories that are false, but nonetheless sound sympathetic. That’s what happened.

Perhaps I should have known better. But I thought, wait, let’s try to cut some amount of waste and pork from the government. Maybe there shouldn’t be 20 million people marked as alive in Social Security who are definitely dead, and over the age of 115.

The oldest American is 114. So it’s safe to say if somebody is 115 and marked as alive in the Social Security database, there’s either a typo… Somebody should call them and say, “We seem to have your birthday wrong, or we need to mark you as dead.” One of the two things.

John Collison

Very intimidating call to get.

Elon Musk

Well, it seems like a reasonable thing. Say if their birthday is in the future and they have a Small Business Administration loan, and their birthday is 2165, we either have a typo or we have fraud. So we say, “we appear to have gotten the century of your birth incorrect.”

John Collison

Or a great plot for a movie.

Elon Musk

Yes. That’s what I mean by, ludicrous fraud.

Dwarkesh Patel

Were those people getting payments?

Elon Musk

Some were getting payments from Social Security. But the main fraud vector was to mark somebody as alive in Social Security and then use every other government payment system to basically do fraud. Because what those other government payment systems do, they would simply do an “are you alive” check to the Social Security database. It’s a bank shot.

Dwarkesh Patel

What would you estimate is the total amount of fraud from this mechanism?

Elon Musk

By the way, the Government Accountability Office has done these estimates before. I’m not the only one. In fact, I think the GAO did an analysis, a rough estimate of fraud during the Biden administration, and calculated it at roughly half a trillion dollars. So don’t take my word for it. Take a report issued during the Biden administration. How about that?

Dwarkesh Patel

From this Social Security mechanism?

Elon Musk

It’s one of many. It’s important to appreciate that the government is very ineffective at stopping fraud. It’s not like a company where, with stopping fraud, you’ve got a motivation because it’s affecting the earnings of your company. The government just prints more money. You need caring and competence. These are in short supply at the federal level.

When you go to the DMV, do you think, “Wow, this is a bastion of competence”? Well, now imagine it’s worse than the DMV because it’s the DMV that can print money.

At least the state level DMVs need to... The states more or less need to stay within their budget or they go bankrupt. But the federal government just prints more money.

Dwarkesh Patel

If there’s actually half a trillion of fraud, why was it not possible to cut all that?

Elon Musk

You really have to stand back and recalibrate your expectations for competence. Because you’re operating in a world where you’ve got to make ends meet. You’ve got to pay your bills...

Dwarkesh Patel

Find the microphones.

Elon Musk

Exactly. It’s not like there’s a giant, largely uncaring monster bureaucracy. It’s a bunch of anachronistic computers that are just sending payments. One of the things that the DOGE team did sounds so simple and probably will save $100-200 billion a year. It was simply requiring payments from the main Treasury computer—which is called PAM, Payment Accounts Master or something like that, there’s $5 trillion payments a year—that go out have a payment appropriation code. Make it mandatory, not optional, that you have anything at all in the comment field.

You have to recalibrate how dumb things are. Payments were being sent out with no appropriation code, not checking back to any congressional appropriation, and with no explanation. This is why the Department of War, formerly the Department of Defense, cannot pass an audit, because the information is literally not there. Recalibrate your expectations.

Dwarkesh Patel

I want to better understand this half a trillion number, because there’s an IG report in 2024.

Elon Musk

Why is it so low?

Dwarkesh Patel

Maybe, but we found that over seven years, the Social Security fraud they estimated was like $70 billion over seven years, so like $10 billion a year. So I’d be curious to see what the other $490 billion is.

Elon Musk

Federal government expenditures are $7.5 trillion a year. How competent do you think the government is?

Dwarkesh Patel

The discretionary spending there is like… 15%?

Elon Musk

But it doesn’t matter. Most of the fraud is non-discretionary. It’s basically fraudulent Medicare, Medicaid, Social Security, disability. There’s a zillion government payments. A bunch of these payments are in fact block transfers to the states. So the federal government doesn’t even have the information in a lot of cases to even know if there’s fraud.

Let’s consider reductio ad absurdum. The government is perfect and has no fraud. What is your probability estimate of that? Zero. Okay, so then would you say, fraud and waste at the government is 90% efficient? That also would be quite generous.

But if it’s only 90%, that means that there’s $750 billion a year of waste and fraud. And it’s not 90%. It’s not 90% effective.

Dwarkesh Patel

This seems like a strange way to first principles the amount of fraud in the government. Just like, how much do you think there is?

Anyways, we don’t have to do it live, but I’d be curious—

Elon Musk

You know a lot about fraud at Stripe? People are constantly trying to do fraud.

John Collison

Yeah, but as you say, it’s a little bit of a... We’ve really ground it down, but it’s a little bit of a different problem space because you’re dealing with a much more heterogeneous set of fraud vectors here than we are.

Elon Musk

But at Stripe, you have high competence and you try hard. You have high competence and high caring, but still fraud is non-zero. Now imagine it’s at a much bigger scale, there’s much less competence, and much less caring.

At PayPal back in the day, we tried to manage fraud down to about 1% of the payment volume. That was very difficult. It took a tremendous amount of competence and caring to get fraud merely to 1%. Now imagine that you’re an organization where there’s much less caring and much less competence. It’s going to be much more than 1%.

John Collison

How do you feel now looking back on politics and doing stuff there? Looking from the outside in, two things have been quite impactful: one, the America PAC, and two, the acquisition of Twitter at the time. But also it seems like there was a bunch of heartache. What’s your grading of the whole experience?

Elon Musk

I think those things needed to be done to maximize the probability that the future is good. Politics generally is very tribal. People lose their objectivity usually with politics. They generally have trouble seeing the good on the other side or the bad on their own side. That’s generally how it goes. That, I guess, was one of the things that surprised me the most.

You often simply cannot reason with people. If they’re in one tribe or the other. They simply believe that everything their tribe does is good and anything the other political tribe does is bad. Persuading them otherwise is almost impossible.

But I think overall those actions—acquiring Twitter, getting Trump elected, even though it makes a lot of people angry—I think those actions were good for civilization.

Dwarkesh Patel

How does it feed into the future you’re excited about?

Elon Musk

Well, America needs to be strong enough to last long enough to extend life to other planets and to get AI and robotics to the point where we can ensure that the future is good.

On the other hand, if we were to descend into, say, communism or some situation where the state was extremely oppressive, that would mean that we might not be able to become multi-planetary. The state might stamp out our progress in AI and robotics.

Dwarkesh Patel

Optimus, Grok, et cetera. Not just yours, but any revenue-maximizing company’s products will be leveraged by the government over time. How does this concern manifest in what private companies should be willing to give governments? What kinds of guardrails?

Should AI models be made to do whatever the government that has contracted them out to do and asks them to do? Should Grok get to say, “Actually, even if the military wants to do X, no, Grok will not do that”?

Elon Musk

I think maybe the biggest danger of AI and robotics going wrong is government. People who are opposed to corporations or worried about corporations should really worry the most about government. Because government is just a corporation in the limit. Government is just the biggest corporation with a monopoly on violence.

I always find it a strange dichotomy where people would think corporations are bad, but the government is good, when the government is simply the biggest and worst corporation. But people have that dichotomy. They somehow think at the same time that government can be good, but corporations bad, and this is not true. Corporations have better morality than the government.

I actually think it’s a thing to be worried about. The government could potentially use AI and robotics to suppress the population. That is a serious concern.

Dwarkesh Patel

As the guy building AI and robotics, how do you prevent that?

Elon Musk

If you limit the powers of government, which is really what the US Constitution is intended to do, to limit the powers of government, then you’re probably going to have a better outcome than if you have more government.

John Collison

Robotics will be available to all governments, right?

Elon Musk

I don’t know about all governments. It’s difficult to predict. I can say what’s the endpoint, or what is many years in the future, but it’s difficult to predict the path along that way. If civilization progresses, AI will vastly exceed the sum of all human intelligence. There will be far more robots than humans. Along the way what happens is very difficult to predict.

Dwarkesh Patel

It seems one thing you could do is just say, “whatever government X, you’re not allowed to use Optimus to do X, Y, Z.” Just write out a policy. I think you tweeted recently that Grok should have a moral constitution. One of those things could be that we limit what governments are allowed to do with this advanced technology.

Elon Musk

Technically if politicians pass a law and they can enforce that law, then it’s hard to not do that law. The best thing we can have is limited government where you have the appropriate crosschecks between the executive, judicial, and legislative branches.

Dwarkesh Patel

The reason I’m curious about it is that at some point it seems the limits will come from you. You’ve got the Optimus, you’ve got the space GPUs…

Elon Musk

You think I’ll be the boss of the government?

Dwarkesh Patel

Already it’s the case with SpaceX that for things that are crucial—the government really cares about getting certain satellites up in space or whatever—it needs SpaceX. It is the necessary contractor.

You are in the process of building more and more of the technological components of the future that will have an analogous role in different industries. You could have this ability to set some policy that suppressing classical liberalism in any way… “My companies will not help in any way with that”, or some policy like that.

Elon Musk

I will do my best to ensure that anything that’s within my control maximizes the good outcome for humanity. I think anything else would be shortsighted, because obviously I’m part of humanity, so I like humans. Pro human.

Dwarkesh Patel

You mentioned that Dojo 3 will be used for space-based compute.

Elon Musk

You really read what I say.

Dwarkesh Patel

I don’t know if you know, Elon, but you have a lot of followers.

Elon Musk

Dead giveaway. How did you discern my secrets? Oh I posted them on X.

Dwarkesh Patel

How do you design a chip for space? What changes?

Elon Musk

You want to design it to be more radiation tolerant and run at a higher temperature. Roughly, if you increase the operating temperature by 20% in degrees Kelvin, you can cut your radiator mass in half. So running at a higher temperature is helpful in space.

There are various things you can do for shielding the memory. But neural nets are going to be very resilient to bit flips. Most of what happens for radiation is random bit flips. But if you’ve got a multi-trillion parameter model and you get a few bit flips, it doesn’t matter. Heuristic programs are going to be much more sensitive to bit flips than some giant parameter file.

I just design it to run hot. I think you pretty much do it the same way that you do things on Earth, apart from making it run hotter.

Dwarkesh Patel

The solar array is most of the weight on the satellite. Is there a way to make the GPUs even more powerful than what Nvidia and TPUs and et cetera are planning on doing that would be especially privileged in the space-based world?

Elon Musk

The basic math is, if you can do about a kilowatt per reticle, then you’d need 100 million full reticle chips to do 100 gigawatts. Depending on what your yield assumptions are, that tells you how many chips you need to make. If you’re going to have 100 gigawatts of power, you need 100 million chips that are running at a kilowatt sustained, per reticle. Basic math.

Dwarkesh Patel

100 million chips depends on… If you look at the die size of something like Blackwell GPUs or something, and how many you can get out of a wafer, you can get on the order of dozens or less per wafer. So basically, this is a world where if we’re putting that out every single year, you’re producing millions of wafers a month. That’s the plan with TeraFab? Millions of wafers a month of advanced process nodes?

Elon Musk

Yeah it could be north of a million or something. You’ve got to do the memory too.

Dwarkesh Patel

Are you going to make a memory fab?

Elon Musk

I think the TeraFab’s got to do memory. It’s got to do logic, memory, and packaging.

Dwarkesh Patel

I’m very curious how somebody gets started. This is the most complicated thing man has ever made. Obviously, if anybody’s up to the task, you’re up to the task. So you realize it’s a bottleneck, and you go to your engineers. What do you tell them to do? “I want a million wafers a month in 2030.”

Elon Musk

That’s right. That’s exactly what I want.

Dwarkesh Patel

Do you call ASML? What is the next step?

John Collison

No so much to ask.

Elon Musk

We make a little fab and see what happens. Make our mistakes at a small scale and then make a big one.

Dwarkesh Patel

Is a little fab done?

Elon Musk

No, it’s not done. We’re not going to keep that cat in the bag. That cat’s going to come out of the bag. There’ll be drones hovering over the bloody thing. You’ll be able to see its construction progress on X in real time.

Look, I don’t know, we could just flounder in failure, to be fair. Success is not guaranteed. Since we want to try to make something like 100 million… We want 100 gigawatts of power and chips that can take 100 gigawatts by 2030. We’ll take as many chips as our suppliers will give us. I’ve actually said this to TSMC and Samsung and Micron: “please build more fabs faster”. We will guarantee to buy the output of those fabs. So they’re already moving as fast as they can. It’s us plus them.

John Collison

There’s a narrative that the people doing AI want a very large number of chips as quickly as possible. Then many of the input suppliers, the fabs, but also the turbine manufacturers, are not ramping up production very quickly.

Elon Musk

No, they’re not.

John Collison

The explanation you hear is that they’re dispositionally conservative. They’re Taiwanese or German, as the story may be. They just don’t believe... Is that really the explanation or is there something else?

Elon Musk

Well, it’s reasonable to... If somebody’s been in the computer memory business for 30 or 40 years…

John Collison

They’ve seen cycles.

Elon Musk

They’ve seen boom and bust 10 times. That’s a lot of layers of scar tissue. During the boom times, it looks like everything is going to be great forever. Then the crash happens and they’re desperately trying to avoid bankruptcy. Then there’s another boom and another crash.

John Collison

Are there other ideas you think others should go pursue that you’re not for whatever reasons right now?

Elon Musk

There are a few companies that are pursuing new ways of doing chips, but they’re just not scaling fast.

John Collison

I don’t even mean within AI, I mean just generally.

Elon Musk

People should do the thing where they find that they’re highly motivated to do that thing, as opposed to some idea that I suggest. They should do the thing that they find personally interesting and motivating to do.

But going back to the limiting factor… I used that phrase about 100 times. The current limiting factor that I see in the three to four year timeframe, it’s chips. In the one year timeframe, it’s energy, power production, electricity. It’s not clear to me that there’s enough usable electricity to turn on all the AI chips that are being made.

Towards the end of this year, I think people are going to have real trouble turning on... The chip output will exceed the ability to turn chips on.

Dwarkesh Patel

What’s your plan to deal with that world?

Elon Musk

We’re trying to accelerate electricity production. I guess that’s maybe one of the reasons that xAI will be maybe the leader, hopefully the leader. We’ll be able to turn on more chips than other people can turn on, faster, because we’re good at hardware.

Generally, the innovations from the corporations that call themselves labs, the ideas tend to flow… It’s rare to see that there’s more than about a six-month difference. The ideas travel back and forth with the people.

So I think you sort of hit the hardware wall and then whichever company can scale hardware the fastest will be the leader. So I think xAI will be able to scale hardware the fastest and therefore most likely will be the leader.

John Collison

You joked or were self-conscious about using the “limiting factor” phrase again. But I actually think there’s something deep here. If you look at a lot of things we’ve touched on over the course of it, it’s maybe a good note to end on. If you think of a senescent, low-agency company, it would have some bottleneck and not really be doing anything about it.

Marc Andreessen had the line of, “most people are willing to endure any amount of chronic pain to avoid acute pain”. It feels like a lot of the cases we’re talking about are just leaning into the acute pain, whatever it is. “Okay, we got to figure out how to work with steel, or we got to figure out how to run the chips in space.” We’ll take some near-term acute pain to actually solve the bottleneck. So that’s kind of a unifying theme.

Elon Musk

I have a high pain threshold. That’s helpful.

John Collison

To solve the bottleneck.

Elon Musk

Yes. One thing I can say is, I think the future is going to be very interesting. As I said at Davos—I think I was on the ground for like three hours or something—it’s better to err on the side of optimism and be wrong than err on the side of pessimism and be right, for quality of life. You’ll be happier if you err on the side of optimism rather than erring on the side of pessimism. So I recommend erring on the side of optimism.

John Collison

Here’s to that.

Dwarkesh Patel

Cool. Elon, thanks for doing this.

John Collison

Thank you.

Elon Musk

All right, thanks guys. All right.

John Collison

Great stamina.

Dwarkesh Patel

Hopefully this didn’t count as a pain in the pain tolerance.

My main bottleneck is finding excellent guests. So, I’m hiring a couple part time scouts to help me find the next David Reich/Sarah Paine/Adam Brown.

$100/hour, fully remote, work hours are flexible - I expect it’ll be 5-10 hours a week.

Ideal candidate is maybe a grad student, or a post doc, or working in one of the fields I wanna find guests in. I’m looking for people who are really plugged into some discipline and have high taste.

Beyond just scouting guests, I’ll want your help assembling curriculums that help me prep for interviews and rapidly get up to speed.

The application form is here, and it’s extremely simple - just pitch me on a guest and tell me a bit about yourself. Please submit by 11:59 PM Pacific, Friday, Jan 23.

I’m looking to hire ~one scout for each of the following fields: bio, history, econ, math/physics, AI/hardware.

However, it’s very possible I end up hiring more (or fewer), or break apart the domains of knowledge in a different way, based on the range of expertise of the best people who apply.

I’m looking for people who are deep experts in at least one field, and who are polymathic enough to think through all kinds of tangential questions in a really interesting way.

So I’m selecting for this synthetic ability to connect one’s expertise to all kinds of important questions about the world - an ability which is often deliberately masked in public academic work. Which means that it can only really come out in conversation.

That’s why I want to hire scouts. I need their network and context - they know who the polymathic geniuses are, who gave a fascinating lecture at the last big conference they attended, who can just connect all kinds of interesting ideas in the field together over conversation, etc.

We get tons of inbound from people who are working on impressive companies or doing interesting research projects. But almost always it’s a no; while I think their work is important, it’s self-contained in a way that I worry won’t lead to interesting broad discussion.

To get a little more concrete, here’s what worked well about some of my recent favorite guests:

Let me talk through why I think some interviews worked especially well, so you can think about what people in fields you’re familiar with fill a similar mold.

• Jacob Kimmel: A lot of people who pitch themselves as guests are capable of only talking about their own research. But the amazing thing about Jacob is that he is an insane polymath. For example, he could explain why evolution didn’t select for longevity by drawing deep analogies to how gradients flow in ML models. He had all these other random interesting takes, from why humans never evolved their own antibiotics to how there’s this gene that used to protect us from HIV-like viruses but got repurposed, which hints at some ghost scourge. And then he could zoom out and give a great diagnosis of what’s bottlenecking pharma progress. I really want to emphasize how that’s different from other brilliant people I get pitched – these people are also doing incredible research, but they don’t have this range of really deep, interesting takes. That part is super crucial.

• David Reich: It’s actually quite surprising that my second most popular guest of all time is a geneticist of ancient DNA. How did that happen? Here’s why I think this episode blew up. In high school, you get some vague explanation of human evolution. And you feel like you understand it and can move on with your life. And here comes David, showing you how this very fundamental topic, which you assume was settled and haven’t bothered thinking about in years, is actually way more murky and surprising than you realized, and how new discoveries are totally overturning our basic understanding of the field (in this case, the how, when, where of human evolution)1.

• Andrej Karpathy: It’s extremely rare to get someone who is expert-level in a technical, fast-moving, and frothy field, but who has no vested interest in a particular company or approach, and who is in a position to just give an unbiased lay of the land. I have a couple questions below about biotech or formal math or robotics in the wake of AI progress - if there’s a Karpathy-type person in those fields, I’d be very keen to get a technical lay of the land and vibe check of what claims are credible versus crazy.

In case it’s helpful for brainstorming a guest, I’ve listed out a few big questions that have been on my mind recently. But please feel free to ignore them - there’s way more interesting questions in the world than the ones I am aware of - feel free to say, “You might not yet be curious about antibody development/the history of language/the dark ages/battery tech, but the guest I have in mind for that topic is so good that it’s going to be your next big banger episode.”

• Dario’s Machines of Loving Grace argues we’ll compress a century of bio progress into a few years - that big breakthroughs like CAR-T therapy, mRNA vaccines, cheap genome sequencing, etc show how in the long run things which seem like data or physical bottlenecks can be solved by better tools to measure/predict/perturb/understand biological system, and these tools are downstream of intelligence. But here’s what I don’t fully understand: over the last 3 decades, we’ve seen a million-fold reduction in genome sequencing costs, 1000-fold decrease in DNA synthesis costs, the development of precise gene editing tools like CRISPR, and the ability to conduct massively parallel experiments through multiplexing techniques. But it doesn’t seem like we’re curing diseases or coming up with new treatments at a faster rate now than we were 30 years ago. If anything, drug development is slowing down. I want to find a biology researcher who can think through how plausible a 10x or 100x speedup in new drug discovery actually is. They should obviously know a lot about and have hot takes on what’s actually bottlenecking progress today, and they should be flexible enough to imagine what might change with much more intelligence.

• What exactly is the special sauce of the brain that we’re still missing? Adam Marblestone thinks it’s the curriculum of reward functions and the learning/steering subsystems. Others argue that gradient descent is fundamentally worse than how the brain learns within a lifetime (which is closer to in-context learning in its flexibility and sample efficiency).

• I’ve been really enjoying Strogatz’s Nonlinear Dynamics and Chaos textbook, and I want to make something podcast-shaped out of it. Strogatz himself has deferred until after he finishes his next book, so I’m looking for another mathematician on a related topic. I think the right format here isn’t a normal meandering interview - it’s something more like a lecture. A mathematician comes in with a specific topic or example we can deep dive on. He posts up at a blackboard, starts explaining a topic, and I interrupt to clarify confusions and ask follow-up questions. The model is something like Terence Tao and Grant Sanderson’s cosmic distance ladder video. Who can replicate something similar with me with some independently explainable topic in chaos/nonlinear dynamics or adjacent topics? I’d be especially keen if someone can present something on how the topics in this textbook tie into ML (see for example Neural network training makes beautiful fractals).

• What real world impact should we expect from the current batch of AI for math projects? What are the fields of technology where people are going, “Ah we could totally solve quantum computing (or fusion or AGI) only if we had more theorems!” But maybe problems in biology and physics and materials and so on reduce down to math in a way I’m not foreseeing, and automating formal math alone is enough to unlock a bunch of progress. See footnotes for some more questions I wanna ask the right guest on this topic.

  I started reading Proofs and Refutations, which is this famous 1976 book by the Hungarian mathematician Imre Lakatos about the philosophy of mathematics. He says math involves a lot of changing definitions and swapping lemmas in order to deal with different counterexamples. This seems fine for a good faith mathematical community, but super reward hackable for these AI-for-math models. Also it involves a lot of realizing how a problem in one domain is really a problem in another, and noticing the meta level pattern - AIs so far have been especially bad at this kind of thing. If math is just proof search within a fixed formal system, then AI can help a lot. But if its dialectical construction and refinement of concepts (based on what tasteful parsimonious definition can withstand counterexamples) , then I feel self play and ‘automated cleverness’ alone won’t do the trick. But maybe automated counterexamples are super useful. I’m sure for practicing mathematicians there’s a bunch of stuff that’s naive or wrong about the above. Would love to chat out what the actual research math process is like, and what good it would do to automate it.

• RL progress has been very fast, but it’s partly the result of going from almost nothing to 1e26 FLOPs training compute in a year (aka like going from GPT-1 to GPT-4.5). It’s still possible that it has terrible scaling exponents and further progress will be very slow. And also it’s not clear how much of the progress over the last year comes from inference scaling, which has worse variable economics. But on the other hand, maybe there’s a ton of low hanging fruit in improving RL - with pretraining, there’s been 5 years of developing the theory and empirics of optimal batch sizes, learning rates, architectures, etc. As that low hanging fruit is picked, maybe RL progress continues to be fast? The other big question about RL training is how much transfer learning are we seeing - is there all this crazy meta learning that’s not directly induced by any env and which will enable flexible human-like labor soon? I have no idea. My friends at labs who are actually doing this training obviously wouldn’t tell me. But I want to actually concretely understand what’s going on here.

• There’s the famous Needham question, which asks why China didn’t industrialize first despite leading the world in population, inventions, and bureaucratic sophistication. I find the standard explanation of how this centralized Ming/Qing regime damped invention and exploration unsatisfying. Or at least I don’t understand it concretely. It’s such a big country - how can you retard progress across the whole thing, especially given that state capacity was presumably weaker in the past? Or at least I assume it was - what did a provincial bureaucrat actually do day-to-day? Was there a price system? Private property? How did the state actually interfere with merchants and artisans?

• There’s something unsatisfying about the arguments that we’ll see 20%+ explosive economic growth from AI. Even if true, what does that mean? What is actually happening? I thought Machines of Loving Grace was a great account of what plausibly is happening on the human facing side of the singularity - aka the FLOPs that are going towards curing disease. But presumably most of what is happening is investment towards more robots, more compute, etc. My sense of what that side of things looks like is so murky and handwavy. There is a version of Machines of Loving Grace you can do that is somewhat concrete about all the sci fi shit - not just gesturing at the galaxies, but getting specific about the space GPUs and factorio like solar tiling and all the other things I’m not thinking of which are relevant to understanding 2040. Presumably the right guest is someone who is really strong in engineering/physics and economics and has a penchant for sci-fi and has a lot of concrete ideas here.

• What should India or Nigeria or for that matter any country not directly in the semiconductor/foundation model supply chain do right now? If the main mechanism of catchup growth goes away (namely, that the underutilized labor of developing countries can rapidly be made more productive with capital and know-how from the developed world), what happens to all these countries that are not China or the US?

Just to give you a sample of some of the surprising findings that he talked through:

• 70,000 years ago, half a dozen different species of humans (Neanderthals, Denisovans, ‘Hobbits’, etc) lived across Eurasia. And then some small group of modern humans (only 1,000 to 10,000 people) drove all of them to extinction. Everyone native to Eurasia and America is descended from this one tribe.

• Neanderthals may have gotten 30-70% of their DNA from modern humans. Which implies that maybe non-Africans today are actually “Neanderthals who became modernized by waves and waves of admixture” rather than modern humans with a bit of Neanderthal mixed in.

• Yersinia pestis (bubonic plague bacteria) may have killed a quarter to half of all people in Western Eurasia for thousands of years, starting around 5,000 years ago. And may be central to explaining everything from the Yamnaya expansion to the fall of Rome to the Industrial Revolution.

• It’s not clear modern humans were even primarily in Africa during the key period (2 million to 500,000 years ago) when human brains diverged from those of other species. Our lineage may have resided in Eurasia for significant stretches.

Okay I’ll stop, but you see my point. What are the other fields like human evolution, and the other presenters like David Reich, who will make you go, “What the fuck, I had no idea.”

David being David is actually a huge piece of the puzzle here which I want to replicate. He’s just incredibly deep and polymathic on what may from the outside look like one field but is in fact very many, from population genetics to archeology to linguistics. And while he’s intellectually humble enough to make qualifiers, he will (and this is very important) go ahead and give hot takes and start speculating about connections between fields and how different hypotheses relate to each other and so on. He won’t just stay at, “Our results show a genetic cline between North and South Indians.” He’ll say, “And we could be wrong here, but this suggests that the caste system which enforced this never otherwise seen levels of endogamy has been incredibly strong for millennia.”

I was recently chatting with a friend who has a similar job to mine. We were talking about how even though our jobs are fundamentally about learning about stuff, our time so easily gets sucked up by other things. So to hold myself accountable, I’m gonna try to publish a blog post every two weeks or so where I explain what I’ve been reading.

I’m totally gonna butcher this - please excuse. If you wanna get the real deal, go check out his summary blog post and his full talk on this topic.

Max is focused on two big sub-questions which together form “the binding problem”:

• Mode binding: how do color, shape, texture, and motion get combined into a unified visual percept of “a red cup”?

• Moment binding: why do we experience all the neurons firing across our entire brain over the course of 10s of milliseconds as a single quanta of experience?

Max thinks each of these binding sub-problems is related to a brain wave:

• Gamma waves - 40 Hz - Fast, local coordination of nearby neurons to get on the same page about what they’re representing.

• Alpha waves - 10 Hz - Slower waves that run through the whole brain and unify experience - think of these like the forward pass of the brain.

  • Two cool things about alpha waves I hadn’t realized. 1. that neurons ride the peak of this oscillation 2. when alpha waves slow down or speed up (fight or flight reactions, etc), people experience time dilation.

Anyways, Max points out that the brain is storing a bunch of structured representations about the world physically, and some feedback controller has to go in and make sure that these representations are correct. This is part of what the alpha waves are doing. And this feedback control and binding is consciousness. I’m glossing over a bunch of logical connections that I definitely don’t understand. But I’ll leave it here.

I know Max could provide a really good answer, but just talking to myself, I’m confused on what the reason is to think that feedback control = consciousness? By this logic, does memory refresh = consciousness too?

Max thinks that figuring out what’s up with consciousness will mean discovering new physics. And specifically, physics at the level of the 4 fundamental forces - some property as basic as mass or charge. His logic is that either consciousness has no real impact on the world (it’s just a byproduct of other stuff the brain does), which would be odd, or it actually has an effect, which would mean it’s new physics.

I’m not sure I buy this. 1. Can’t it be an effect that’s best understood at an implication of existing laws of physics - the fact that wood floats on water has an impact on the world, but you don’t need new physics to explain it 2. Doesn’t it seem implausible that evolution blindly stumbled upon and is now making good use of a whole undiscovered physical field which we have never managed to actually interact with using our technology, nor seen summoned anywhere else in the universe?

I’m only 3 chapters in, so I’ve only got the building blocks so far. The fundamental idea is this. It’s often hard to anticipate how a system will evolve just by observing a bunch of different trajectories over time. But it’s much easier to see what will happen if you plot how the system will evolve from different starting points. The examples get more and more interesting, and because Strogatz focuses on the graphical and geometric interpretations, the motivating problems are super satisfying; the book is really a bunch of 3Blue1Brown videos on a certain topic stapled together.

Side note: I could not have understood anything here if I didn’t have LLMs and couldn’t watch the lectures async. I paused every minute or so (to clarify some confusion with a chatbot or to try and anticipate the next step), and I had the same section of textbook open at the same time.

I’m now wondering to myself, “How the hell did I learn anything in college at all?” I would be so lost if I was actually taking this course in college and just attending the lectures live.

In college, I actually did bounce out of a difficult course I feel like I could totally learn today with LLMs and async lectures + my adult executive function.

As I was working through these examples (some inspired by actual papers), I kept thinking about what parts the “automated cleverness” (Terry Tao’s term) of today’s AIs could actually help with.

It’s crazy how much understanding you can get about a physical system through mathematics. But that understanding is so dependent on insight and interpretation.

To give one example, Section 3.7 has a really clever model of an insect outbreak, showing how budworms, birds, and trees play out against each other given different growth rates and other dynamics.

But first you have to figure out the right dimensionless forms. And that requires judgment about which dimensions actually matter. In the insect model, the choice was to think in terms of R and K and treat the bird population as basically an artifact of those parameters. But you could have done it the other way around—from the basis of birds.

Then there’s how you make the visualization. Once you’ve got the dynamics in dimensionless form, you could just graph the equation and find the fixed points. But the result would be almost impossible to interpret. Graph it a different way, though, and suddenly the intercepts align with your intuition. You can actually see the three regimes: where carrying capacity is so low the population never gets going, where birds keep things in check, and where the outbreak has outgrown the birds’ ability to control it.

This kind of insight is inseparable from understanding what you’re even trying to learn about the system. And I’m skeptical today’s AI helps much here. When these methods were first developed, the right forms and interpretations weren’t obvious. The mathematician who wrote the original paper had to come up with new insights about how to think about the problem.

Maybe models are now good enough to apply these methods to new systems that fit the same template. But that just means the few mathematicians who invent genuinely new frameworks are the only ones who stay relevant.

Starting with the biology section: Dario argues that we’ll get a century of bio progress in a few years. His argument:

• Most bio progress is driven by breakthrough discoveries which give you whole new primitives for what you can measure, change, or predict (CAR-T therapy, mRNA vaccines, CRISPR, genome sequencing costs declining so much, etc).

• These discoveries seem to have been made in scrappy haphazard ways, often years after they were initially possible, and often by people responsible for other breakthroughs as well. All 3 of these observations hint that they are bottlenecked by intelligence.

• Dario acknowledges that data is a huge bottleneck for bio. But the tools we have for collecting data can also be expanded by intelligence. Human researchers came up with multiplexing and AlphaFold and Perturb-Seq - the AI researchers will come up with even more.

Here’s the counterargument. The kinds of human researcher breakthroughs he uses as examples of what AI could do more of haven’t had a huge impact on health. Over the last 3 decades, we’ve seen a million-fold reduction in genome sequencing costs, 1000-fold decrease in DNA synthesis costs, the development of precise gene editing tools like CRISPR, and the ability to conduct massively parallel experiments through multiplexing techniques. But it doesn’t seem like we’re curing diseases or coming up with new treatments at a faster rate now than we were 30 years ago. If anything, drug development is slowing down. Why think that AI will be able to fundamentally change this dynamic?

Relatedly, Jacob Trefethan has an excellent blog post makes the the argument that AI won’t speed up medical progress that much (he also steelmans the opposite point in this other post). Jacob points out that making a drug to cure something like Alzheimer’s is really hard. Raw understanding of some of the disease life cycle (which more intelligence could give you more of) is not enough. We understand that Alzheimer’s is clearly linked to Amyloid beta, and there are now many different drugs trying to remove amyloid plaques which have all not worked. Even if we get more insights like the Amyloid beta thing from AI scientists, that alone will not be enough to identify the correct targets. You just have to do a bunch of experiments on live humans.

This is why Dario’s point about clinical trials falls flat. He argues that clinical trials are currently slow because we just don’t know whether a given drug will actually work. But if we had much greater confidence, like we did with the mRNA vaccines for COVID, then we could test and approve drugs much faster. However, I don’t see why we should think that modulo the full hyperrealistic simulation of the human body, we could tell ex ante which drugs are gonna work. I don’t yet buy the argument that a million George Church clones in a datacenter could derisk all the drug trials

Quick notes on other parts of the essay:

• Overall I find it pretty impressive that a tech CEO is this generally thoughtful.

• The poverty and econ section doesn’t address that the main mechanism of catchup growth goes away post AGI; namely developing countries have lots of underutilized labor which is bottlenecking production, and because the marginal product of labor is high in the world today, those countries can get rich fast. So how exactly are these other countries catching up?

• The key point that underlies his framework that intelligence can drive a century of progress in 5-10 years : “Things that are hard constraints in the short run may become more malleable to intelligence in the long run. For example, intelligence might be used to develop a new experimental paradigm that allows us to learn in vitro what used to require live animal experiments, or to build the tools needed to collect new data (e.g. the bigger particle accelerator), or to (within ethical limits) find ways around human-based constraints (e.g. helping to improve the clinical trial system, helping to create new jurisdictions where clinical trials have less bureaucracy, or improving the science itself to make human clinical trials less necessary or cheaper).”

  • it’s interesting to consider why this isn’t true for factors of production today. We live in a (relatively) capital-abundant and labor-scarce world. That is reflected in the labor share of income being 2x as high as the capital share of income. But this has been true for centuries upon centuries. Contra Piketty in “Capital in the 21st Century”, all these capital holders have not been able to get some runaway capital accumulation process going by figuring out a way around labor constraints. Why think that intelligence will be any different than capital in its ability to get around other factors of production? maybe the argument is that intelligence can actually help generate the other factors of production in a way that capital can’t.

Absolutely fascinating blog post.

You want to train your model at the highest possible learning rate under which it still converges. But the boundary of convergence versus divergence is fractal, which makes these hyperparameters really hard to optimize for via gradient descent.

Now you can ask the question: evolution somehow found the right hyperparameters to train our brains. How did evolution solve this wicked problem? Presumably because gradient free optimization fares better against these kinds of fractal landscapes - if you optimize for the part of the region where the average speed of convergence is high (rather than just take the gradient from a specific point that’s bounded in an unpredictable way by fractals), seems like you could do much better.

Backing up, why is the meta-loss landscape fractal in the first place? Jascha’s explanation is that fractals often emerge when iteratively applying a function. Gradient descent on the parameters is one such function that you iterate across training steps. But then the follow up question is this. There’s lots of other iterative functions you could think of, even within the context of neural networks. Do they all lead to fractals? For example:

• In chain of thought, you apply a model to a string, which makes a new string, to which you apply the model, etc.

• RNNs keep applying the same parameters to the hidden state.

Over conversation, an AI researcher friend revealed that CoT and RNNs both have variance problems that could well be explained by these fractal like dynamics. Though I only understand this claim at a hand-wavy level.

I recently released a blog post with Phil Trammell on Capital in the 22nd Century. l (I am coauthor by a generous definition of co-authorship - basically I got to have fun chatting with and learning from Phil as he came up with the ideas).

I was chatting with Charlie Songhurst about it, and he made an interesting argument against our main conclusion, namely that labor share of GDP will go down post AGI.

Here’s my recap of Charlie’s argument: If you look at how wealthy people spend their money—whether aristocrats through history or retired billionaires today—it’s often things like buying sports teams, writing novels, going to galas, and doing crazy fitness routines. When all our other needs are met, we have a deep yearning to play status games, earn the esteem of our peers, and feel like we’re improving ourselves.

Of course, in a world with ASI, the robots might be able to offer us all kinds of crazy new products that we’ll want to pay for: Starship v20 sending TWs of GPUs to space, Pfizer churning out anti-aging medicines, and so on. But Charlie’s story says that demand for these new products won’t dilute labor share of income. Over time, the price to manufacture anti-aging drugs or send a kilogram to space will crater to almost free. But the price to hire a Michelin chef for your dinner party, or get people to show up to your modern equivalent of a Roman triumph, will only increase.

For decades, chess AIs have been better than the best human players. Yet chess has never been more popular, and Magnus is a rich global celebrity1. Why not think there’ll be a million other competitions and LARPs that keep humans wealthy and engaged?

I do think future top chess and tennis players will be even wealthier than Magnus Carlsen and Roger Federer, and our descendants will come up with games we can’t even imagine. But I don’t think that keeps the labor share of value high.

The key mistake is assuming there’s some exhaustible lump of things we’ll want from AIs and robots. In this picture, you get your anti-aging serum and you’re set—the second dose is pointless, so all your remaining wealth goes to watching Magnus stream.

But many of the goods AIs will produce in the future won’t be like anti-aging serum. Each robot factory you buy lets you build more robot factories. You might think, “Look, AI is going to make producing spaceships super cheap, so the share of GDP that goes to making spaceships decreases.” But over time, you figure out new uses for the things you produce. Once we have a colony on Mars, a fledgling Dyson Swarm, the value of a spaceship increases.

In the far future, each additional spaceship you buy lets you colonize another star system. We have lots of examples in human history of rapid growth in demand for an input to production as we figure out more uses for it. Oil was just lamp fuel for the 60 years after its first commercial drilling in 1859. Then we got cars and planes and plastics. And US production is up a thousand-fold since 1869. Why think intelligence will be any different2?

Sure, evolution wired us to deeply desire status, not colonizing the galaxy. But evolution hasn’t stopped. Our current drives descend from the 0.001% of primates whose desires were most conducive to expansion. If different drives would have served us better in the ancestral environment, we’d have those instead.

Similarly, if 99% of humans are happy to spend their AGI bucks naming soccer stadiums, but 1% are willing to let all their wealth compound faster by buying shares in robot factories that build other robot factories, that 1% of the population will control most future expenditure—which they’ll keep reinvesting in sci-fi shit instead of aristocratic money sinks.

By the way, this doesn’t require any genetic evolution. Humans as they exist already have a wide variety of drives, and it’s no surprise that the wealthiest ones are intentionally trying to grow their capital by doing crazy sci-fi shit, rather than painting vistas or becoming competitive athletes. This is not to mention future AIs, which can be specifically trained to have whatever values are most competitive.

Basically, there is this overhang of explosive growth that will be available to any set of people and AI that defect from playing humanity’s post-retirement status games. Will Elon’s ASI advisors run out of ways to help him compound his wealth after he builds enough starships and TW datacenters? I don’t think so, for reasons related to the lump of labor fallacy. The billionaire that instead decides to spend his fortune buying human soccer leagues doesn’t grow his wealth nearly as fast. So he’ll control a shrinking share of future expenditure. And remember, constant labor share of GDP was counting on people like him staying relatively just as wealthy

To be clear, the labor share of GDP going down is totally compatible with absolute wages (at least for top human performers like Magnus) going up a ton. In fact I expect that to happen. I just expect that people who monomaniacally focus on compounding their wealth will grow their wealth faster than the rest of us who waste it away sponsoring the Olympics.

The only way to avoid this is some global totalitarian ban on pursuing the most productive investments, so the rest of us can spend our wealth on parties without getting totally outcompeted. To be clear, I don’t think we should do this. In the world where capital earns most of income, absolute wages may still go way up. Humans might become richer than ever before. But our labor income will constitute a smaller slice of a much larger pie.

By the way, chess is a weird example—it was for shits and giggles even when humans were better than AI. Being great at chess doesn’t let you produce some good or service that’s valuable independent of getting to watch you play. For things where we care about the actual end product, the preference for humans isn’t nearly as strong. Waymo vs. Uber pricing suggests we might actually prefer not having humans when there’s a job we just need done.

In fact, intelligence is different from other inputs to production, in that it can just directly find more uses for itself.

Adam Marblestone is CEO of Convergent Research. He’s had a very interesting past life: he was a research scientist at Google Deepmind on their neuroscience team and has worked on everything from brain-computer interfaces to quantum computing to nanotech and even formal mathematics.

In this episode, we discuss how the brain learns so much from so little, what the AI field can learn from neuroscience, and the answer to Ilya’s question: how does the genome encode abstract reward functions? Turns out, they’re all the same question.

Watch on YouTube; listen on Apple Podcasts or Spotify.

• Gemini 3 Pro recently helped me run an experiment to test multi-agent scaling: basically, if you have a fixed budget of compute, what is the optimal way to split it up across agents? Gemini was my colleague throughout the process — honestly, I couldn’t have investigated this question without it. Try Gemini 3 Pro today gemini.google.com

• Labelbox helps you train agents to do economically-valuable, real-world tasks. Labelbox’s network of subject-matter experts ensures you get hyper-realistic RL environments, and their custom tooling lets you generate the highest-quality training data possible from those environments. Learn more at labelbox.com/dwarkesh

To sponsor a future episode, visit dwarkesh.com/advertise.

• Intro to Brain-Like-AGI Safety - Steven Byrnes’s theory of the learning vs steering subsystem; referenced throughout the episode.

• A Brief History of Intelligence - Great book by Max Bennett on connections between neuroscience and AI

• Adam’s blog, and Convergent Research’s blog on essential technologies.

• A Tutorial on Energy-Based Learning by Yann LeCun

• What Does It Mean to Understand a Neural Network? - Kording & Lillicrap

• E11 Bio and their brain connectomics approach

• Sam Gershman on what dopamine is doing in the brain

• Gwern’s proposal on training models on the brain’s hidden states

Relevant episodes: Ilya Sutskever, Richard Sutton, Andrej Karpathy

(00:00:00) – The brain’s secret sauce is the reward functions, not the architecture

(00:22:20) – Amortized inference and what the genome actually stores

(00:42:42) – Model-based vs model-free RL in the brain

(00:50:31) – Is biological hardware a limitation or an advantage?

(01:03:59) – Why a map of the human brain is important

(01:23:28) – What value will automating math have?

(01:38:18) – Architecture of the brain

Dwarkesh Patel

The big million-dollar question that I have, that I’ve been trying to get the answer to through all these interviews with AI researchers: How does the brain do it? We’re throwing way more data at these LLMs and they still have a small fraction of the total capabilities that a human does. So what’s going on?

Adam Marblestone

This might be the quadrillion-dollar question or something like that. You can make an argument that this is the most important question in science. I don’t claim to know the answer. I also don’t think that the answer will necessarily come even from a lot of smart people thinking about it as much as they are. My overall meta-level take is that we have to empower the field of neuroscience to just make neuroscience a more powerful field technologically and otherwise, to actually be able to crack a question like this.

Maybe the way that we would think about this now with modern AI, neural nets, deep learning, is that there are certain key components of that. There’s the architecture. There’s maybe hyperparameters of how many layers you have or properties of that architecture. There is the learning algorithm itself. How do you train it? Backprop, gradient descent, is it something else? How is it initialized? If we take the learning part of the system, it still may have some initialization of the weights. And then there are also cost functions. What is it being trained to do? What’s the reward signal? What are the loss functions, supervision signals?

My personal hunch within that framework is that the field has neglected the role of these very specific loss functions, very specific cost functions. Machine learning tends to like mathematically simple loss functions. Predict the next token, cross-entropy, these simple computer scientist loss functions. I think evolution may have built a lot of complexity into the loss functions actually, many different loss functions for different areas turned on at different stages of development. A lot of Python code, basically, generating a specific curriculum for what different parts of the brain need to learn.

Because evolution has seen many times what was successful and unsuccessful, and evolution could encode the knowledge of the learning curriculum. In the machine learning framework, maybe we can come back and we can talk about where do the loss functions of the brain come from? Can different loss functions lead to different efficiency of learning?

Dwarkesh Patel

People say the cortex has got the universal human learning algorithm, the special sauce that humans have. What’s up with that?

Adam Marblestone

This is a huge question and we don’t know. I’ve seen models where the cortex… The cortex typically has this six-layered structure, layers in a slightly different sense than layers of a neural net. Any one location in the cortex has six physical layers of tissue as you go in layers of the sheet. And those areas then connect to each other and that’s more like the layers of a network.

I’ve seen versions of that where what you’re trying to explain is just, “How does it approximate backprop?” And what is the cost function for that? What is the network being asked to do, if you are trying to say it’s something like backprop? Is it doing backprop on next token prediction or is it doing backprop on classifying images or what is it doing? And no one knows. But one thought about it, one possibility about it, is that it’s just this incredibly general prediction engine. So any one area of the cortex is just trying to predict… Basically can it learn to predict any subset of all the variables it sees from any other subset? Omnidirectional inference, or omnidirectional prediction.

Whereas an LLM is just seeing everything in the context window and then it computes a very particular conditional probability which is, “Given all the last thousands of things, what are the probabilities for the next token.” But it would be weird for a large language model to say “the quick brown fox blank blank the lazy dog” and fill in the middle versus doing the next token, if it’s doing just forward. It can learn how to do that stuff at this emergent level of the context window and everything, but natively it’s just predicting the next token.

What if the cortex is natively made so that any area of cortex can predict any pattern in any subset of its inputs given any other missing subset? That is a little bit more like “probabilistic AI”. A lot of the things I’m saying, by the way, are extremely similar to what Yann LeCun would say. He’s really interested in these energy-based models and something like that is like, the joint distribution of all the variables. What is the likelihood or unlikelihood of just any combination of variables?

If I clamp some of them and I say that definitely these variables are in these states, then I can compute, with probabilistic sampling for example—conditioned on these being set in this state, and these could be any arbitrary subset of variables in the model—can I predict what any other subset is going to do and sample from any other subset given clamping this subset? And I could choose a totally different subset and sample from that subset. So it’s omnidirectional inference.

And so there could be some parts of the cortex, there might be association areas of cortex that predict vision from audition. There might be areas that predict things that the more innate part of the brain is going to do. Because remember, this whole thing is riding on top of a lizard brain and lizard body, if you will. And that thing is a thing that’s worth predicting too. You’re not just predicting do I see this or do I see that. Is this muscle about to tense? Am I about to have a reflex where I laugh? Is my heart rate about to go up? Am I about to activate this instinctive behavior?

Dwarkesh Patel

Based on my higher-level understanding… Like I can match somebody has told me there’s a spider on my back to this lizard part that would activate if I was literally seeing a spider in front of me. You learn to associate the two so that even just from somebody hearing you say “There’s a spider on your back”

Adam Marblestone

Well, let’s come back to this. This is partly having to do with Steve Byrnes’ theories, which I’m recently obsessed about. But on your podcast with Ilya, he said, “Look, I’m not aware of any good theory of how evolution encodes high-level desires or intentions.” I think this is very connected to all of these questions about the loss functions and the cost functions that the brain would use. And it’s a really profound question, right?

Let’s say that I am embarrassed for saying the wrong thing on your podcast because I’m imagining that Yann LeCun is listening and he says, “That’s not my theory. You described energy-based models really badly.” That’s going to activate in me innate embarrassment and shame, and I’m going to want to go hide and whatever. That’s going to activate these innate reflexes. That’s important because I might otherwise get killed by Yann LeCun’s marauding army of other…

Dwarkesh Patel

The French AI researchers are coming for you, Adam.

Adam Marblestone

So it’s important that I have that instinctual response. But of course, evolution has never seen Yann LeCun or known about energy-based models or known what an important scientist or a podcast is. Somehow the brain has to encode this desire to not piss off really important people in the tribe or something like this in a very robust way, without knowing in advance all the things that the Learning Subsystem of the brain, the part that is learning cortex and other parts… The cortex is going to learn this world model. It’s going to include things like Yann LeCun and podcasts. And evolution has to make sure that those neurons, whatever the Yann-LeCun-being-upset-with-me neurons, get properly wired up to the shame response or this part of the reward function. And this is important, right?

Because if we’re going to be able to seek status in the tribe or learn from knowledgeable people, as you said, or things like that, exchange knowledge and skills with friends but not with enemies… We have to learn all this stuff. It has to be able to robustly wire these learned features of the world, learned parts of the world model, up to these innate reward functions, and then actually use that to then learn more. Because next time I’m not going to try to piss off Yann LeCun if he emails me that I got this wrong. We’re going to do further learning based on that.

In constructing the reward function, it has to use learned information. But how can evolution, which didn’t know about Yann LeCun, do that? The basic idea that Steve Byrnes is proposing is that part of the cortex, or other areas like the amygdala that learn, what they’re doing is they’re modeling the Steering Subsystem. The Steering Subsystem is the part with these more innately programmed responses and the innate programming of these series of reward functions, cost functions, bootstrapping functions that exist.

There are parts of the amygdala, for example, that are able to monitor what those parts do and predict what those parts do. How do you find the neurons that are important for social status? Well, you have some innate heuristics of social status, for example, or you have some innate heuristics of friendliness that the Steering Subsystem can use. And the Steering Subsystem actually has its own sensory system, which is crazy. We think of vision as being something that the cortex does. But there’s also a Steering Subsystem, subcortical visual system called the superior colliculus with innate ability to detect faces, for example, or threats.

So there’s a visual system that has innate heuristics and the Steering Subsystem has its own responses. There’ll be part of the amygdala or part of the cortex that is learning to predict those responses. What are the neurons that matter in the cortex for social status or for friendship? They’re the ones that predict those innate heuristics for friendship. You train a predictor in the cortex and you say, “Which neurons are part of the predictor?” Those are the ones that, now you’ve actually managed to wire it up.

Dwarkesh Patel

This is fascinating. I feel like I still don’t understand… I understand how the cortex could learn how this primitive part of the brain would respond to… Obviously it has these labels on, “here’s literally a picture of a spider, and this is bad, be scared of this.” The cortex learns that this is bad because the innate part tells it that. But then it has to generalize to, “Okay, the spider’s on my back. And somebody’s telling me the spider’s on your back. That’s also bad.”

Adam Marblestone

Yes.

Dwarkesh Patel

But it never got supervision on that. So how does it…?

Adam Marblestone

Well, it’s because the Learning Subsystem is a powerful learning algorithm that does have generalization, that is capable of generalization. The Steering Subsystem, these are the innate responses. You’re going to have some built into your Steering Subsystem, these lower brain areas: hypothalamus, brainstem, et cetera. Again, they have their own primitive sensory systems.

So there may be an innate response. If I see something that’s moving fast toward my body that I didn’t previously see was there and is small and dark and high contrast, that might be an insect skittering onto my body. I am going to flinch. There are these innate responses. There’s going to be some group of neurons, let’s say, in the hypothalamus, that is the I-am-flinching or I-just-flinched neurons in the hypothalamus.

When you flinch, first of all, it’s a negative contribution to the reward function. You didn’t want that to happen, perhaps. But that’s a reward function that doesn’t have any generalization in it. I’m going to avoid that exact situation of the thing skittering toward me. Maybe I’m going to avoid some actions that lead to the thing skittering. That’s a generalization you can get, what Steve calls downstream of the reward function. I’m going to avoid the situation where the spider was skittering toward me, but you’re also going to do something else.

There’s going to be a part of your amygdala, say, that is saying, “Okay, a few milliseconds, hundreds of milliseconds or seconds earlier, could I have predicted that flinching response?” It’s going to be a group of neurons that is essentially a classifier of, “Am I about to flinch?” And I’m going to have classifiers for that for every important Steering Subsystem variable that evolution needs to take care of. Am I about to flinch? Am I talking to a friend? Should I laugh now? Is the friend high status? Whatever variables the hypothalamus, brainstem, contains… Am I about to taste salt?

It’s going to have all these variables and for each one it’s going to have a predictor. It’s going to train that predictor. Now the predictor that it trains, that can have some generalization. The reason it can have some generalization is because it just has a totally different input. Its input data might be things like the word “spider”, but the word “spider” can activate in all sorts of situations that lead to the word “spider” activating in your world model. If you have a complex world model with really complex features that inherently gives you some generalization. It’s not just the thing skittering toward me, it’s even the word “spider” or the concept of “spider” is going to cause that to trigger. This predictor can learn that. Whatever spider neurons are in my world model, which could even be a book about spiders or somewhere, a room where there are spiders or whatever that is…

Dwarkesh Patel

The amount of heebie-jeebies that this conversation is eliciting in the audience…

Adam Marblestone

Now I’m activating your Steering Subsystem, your Steering Subsystem spider hypothalamus subgroup of neurons of skittering insects are activating based on these very abstract concepts in the conversation.

Dwarkesh Patel

If you keep going, I’m going to put in a trigger warning.

Adam Marblestone

That’s because you learned this. The cortex inherently has the ability to generalize because it’s just predicting based on these very abstract variables and all these integrated information that it has. Whereas the Steering Subsystem only can use whatever the superior colliculus and a few other sensors can spit out.

Dwarkesh Patel

By the way, it’s remarkable that the person who’s made this connection between different pieces of neuroscience, Steve Byrnes, is a former physicist. For the last few years, he’s been trying to synthesize—

Adam Marblestone

He’s an AI safety researcher. He’s just synthesizing. This comes back to the academic incentives thing. I think that this is a little bit hard to say. What is the exact next experiment? How am I going to publish a paper on this? How am I going to train my grad student to do this? It’s very speculative. But there’s a lot in the neuroscience literature and Steve has been able to pull this together. And I think that Steve has an answer to Ilya’s question essentially, which is, how does the brain ultimately code for these higher-level desires and link them up to the more primitive rewards?

Dwarkesh Patel

Very naive question, but why can’t we achieve this omnidirectional inference by just training the model to not just map from a token to next token, but remove the masks in the training so it maps every token to every token, or come up with more labels between video and audio and text so that it’s forced to map one to each one?

Adam Marblestone

I mean, that may be the way. It’s not clear to me. Some people think that there’s a different way that it does probabilistic inference or a different learning algorithm that isn’t backprop. There might be other ways of learning, energy-based models or other things like that, that you can imagine that is involved in being able to do this and that the brain has that.

But I think there’s a version of it where what the brain does is crappy versions of backprop to learn to predict through a few layers and that it’s kind of like a multimodal foundation model. LLMs are maybe just predicting the next token. But vision models maybe are trained in learning to fill in the blanks or reconstruct different pieces or combinations. But I think that it does it in an extremely flexible way.

If you train a model to just fill in this blank at the center, okay, that’s great. But what if you didn’t train it to fill in this other blank over to the left? Then it doesn’t know how to do that. It’s not part of its repertoire of predictions that are amortized into the network. Whereas with a really powerful inference system, you could choose at test time, what is the subset of variables it needs to infer and which ones are clamped?

Dwarkesh Patel

Okay, two sub-questions. One, it makes you wonder whether the thing that is lacking in artificial neural networks is less about the reward function and more about the encoder or the embedding… Maybe the issue is that you’re not representing video and audio and text in the right latent abstraction such that they could intermingle and conflict.

Maybe this is also related to why LLMs seem bad at drawing connections between different ideas. Are the ideas represented at a level of generality at which you could notice different connections?

Adam Marblestone

Well, the problem is these questions are all commingled. If we don’t know if it’s doing a backprop-like learning, and we don’t know if it’s doing energy-based models, and we don’t know how these areas are even connected in the first place, it’s very hard to really get to the ground truth of this. But yeah, it’s possible.

I think that people have done some work. My friend Joel Dapello actually did something some years ago where he put a model—I think it was a model of V1, specifically how the early visual cortex represents images—as an input into a convnet and that improves some things. It could be differences. The retina is also doing motion detection and certain things are getting filtered out. There may be some preprocessing of the sensory data. There may be some clever combinations of which modalities are predicting which or so on, that lead to better representation. There may be much more clever things than that.

Some people certainly do think that there’s inductive biases built in the architecture that will shape the representations differently or that there are clever things that you can do. Astera, which is the same organization that employs Steve Byrnes, just launched this neuroscience project based on Doris Tsao’s work. She has some ideas about how you can build vision systems that basically require less training. They build into the assumptions of the design of the architecture things like objects are bounded by surfaces and surfaces have certain types of shapes and relationships of how they occlude each other and stuff like that. It may be possible to build more assumptions into the network. Evolution may have also put some changes of architecture. It’s just I think that also the cost functions and so on may be a key thing that it does.

Dwarkesh Patel

I want to talk about this idea that you just glanced off of which was amortized inference. Maybe I should try to explain what I think it means, because I think it’s probably wrong and this will help you correct me.

Adam Marblestone

It’s been a few years for me too.

Dwarkesh Patel

Right now, the way the models work is that you have an input, it maps it to an output, and this is amortizing a process, the real process, which we think is what intelligence is. It’s that you have some prior over how the world could be, what are the causes that make the world the way that it is. And then when you see some observation, you should be like, “Okay, here’s all the ways the world could be. This cause explains what’s happening best.”

Now, doing this calculation over every possible cause is computationally intractable. So then you just have to sample like, “Oh, here’s a potential cause. Does this explain this observation? No, forget it. Let’s keep sampling.” And then eventually you get the cause, then the cause explains the observation, and then this becomes your posterior.

Adam Marblestone

That’s actually pretty good. Bayesian inference in general is of this very intractable thing. The algorithms that we have for doing that tend to require taking a lot of samples, Monte Carlo methods, taking a lot of samples. And taking samples takes time. This is like the original Boltzmann machines and stuff. They’re using techniques like this, and still it’s used with probabilistic programming, other types of methods often. The Bayesian inference problem, which is basically the problem of perception, given some model of the world and given some data, how should I update my… What are the missing variables in my internal model?

Dwarkesh Patel

And I guess the idea is that neural networks are hopefully… Obviously, mechanistically, the neural network is not starting with, “Here is my model of the world, and I’m going to try to explain this data.” But the hope is that instead of starting with, “Hey, does this cause explain this observation? No. Did this cause explain this observation? Yes.” What you do is just like observation…

Adam Marblestone

What’s the cause that the neural net thinks is the best one?

Dwarkesh Patel

Observation to cause. So the feedforward goes observation to cause to then the output that…

Adam Marblestone

You don’t have to evaluate all these energy values or whatever and sample around to make them higher and lower. You just say, approximately that process would result in this being the top one or something like that.

Dwarkesh Patel

Exactly. One way to think about it might be that test-time compute, inference-time compute is actually doing this sampling again. You literally read its chain of thought. It’s actually doing this toy example we’re talking about where it’s like, “Oh, can I solve this problem by doing X? Nah, I need a different approach.” This raises the question. I mean, over time it is the case that the capabilities which required inference-time compute to elicit, get distilled into the model. So you’re amortizing the thing which previously you needed to do these rollouts, these Monte Carlo rollouts, to figure out.

In general, maybe there’s this principle that digital minds which can be copied, have different tradeoffs which are relevant, from biological minds which cannot. So in general, it should make sense to amortize more things because you can literally copy the amortization, or copy the things that you have sort of built in.

This is a tangential question where it might be interesting to speculate about. In the future, as these things become more intelligent and the way we train them becomes more economically rational, what will make sense to amortize into these minds, which evolution did not think was worth amortizing into biological minds? You have to retrain every time.

Adam Marblestone

First of all, I think the probabilistic AI people would be like, of course you need test-time compute, because this inference problem is really hard and the only ways we know how to do it involve lots of test-time compute. Otherwise it’s just this crappy approximation that’s never going to… You have to do infinite data or something to make this. I think some of the probabilistic people will be like, “No, it’s inherently probabilistic and amortizing it in this way just doesn’t make sense.” They might then also point to the brain and say, “Okay, well the brain, the neurons are stochastic and they’re sampling and they’re doing things. So maybe the brain actually is doing more like the non-amortized inference, the real inference.”

But it’s also strange how perception can work in just milliseconds or whatever. It doesn’t seem like it uses that much sampling. So it’s also clearly doing some baking things into approximate forward passes or something like that to do this. In the future, I don’t know. Is it already a trend to some degree that things that people were having to use test-time compute for, are getting used to train back the base model? Now it can do it in one pass.

Maybe evolution did or didn’t do that. I think evolution still has to pass everything through the genome to build the network and the environment in which humans are living is very dynamic. So maybe, if we believe this is true, there’s a Learning Subsystem per Steve Byrnes, and a Steering Subsystem, that the Learning Subsystem doesn’t have a lot of pre-initialization or pretraining. It has a certain architecture, but then within lifetime it learns. Then evolution didn’t actually amortize that much into that network. It amortized it instead into a set of innate behaviors in a set of these bootstrapping cost functions, or ways of building up very particular reward signals.

Dwarkesh Patel

This framework helps explain this mystery that people have pointed out and I’ve asked a few guests about, which is that if you want to analogize evolution to pretraining, well how do you explain the fact that so little information is conveyed through the genome? So 3 gigabytes is the size of the total human genome. Obviously a small fraction of that is actually relevant to coding the brain.

Previously people made this analogy, that actually evolution has found the hyperparameters of the model, the numbers which tell you how many layers there should be, the architecture, basically, how things should be wired together. But if a big part of the story is that increased sample efficiency aids learning, generally makes systems more performant, is the reward function, is the loss function—and if evolution found those loss functions that aid learning—then it actually makes sense how you can build an intelligence with so little information. Because the reward function, in Python the reward function is literally a line. So you just have a thousand lines like this, and that doesn’t take up that much space.

Adam Marblestone

Yes. It also gets to do this generalization thing with the thing I was describing where we were talking about the spider, where it learns just the word “spider” which triggers the spider reflex or whatever. It gets to exploit that too. It gets to build a reward function that actually has a bunch of generalization in it just by specifying these innate spider stuff and the Thought Assessors, as Steve calls them, that do the learning.

That’s potentially a really compact solution to building up these more complex reward functions too, that you need. It doesn’t have to anticipate everything about the future of the reward function. It just has to anticipate what variables are relevant and what are heuristics for finding what those variables are. And then it has to have a very compact specification for the learning algorithm and basic architecture of the Learning Subsystem. And then it has to specify all this Python code of all the stuff about the spiders and all the stuff about friends, and all the stuff about your mother, and all the stuff about mating and social groups and joint eye contact. It has to specify all that stuff.

So is this really true? I think that there is some evidence for it. Fei Chen and Evan Macosko and various other researchers have been doing these single-cell atlases. One of the things that scaling up neuroscience technology—again, this is one of my obsessions—has done through the BRAIN Initiative, a big neuroscience funding program, is they’ve basically gone through different areas, especially of the mouse brain, and mapped where the different cell types are? How many different types of cells are there in different areas of cortex? Are they the same across different areas? Then you look at these subcortical regions, which are more like the Steering Subsystem or reward-function-generating regions. How many different types of cells do they have? And which neuron types do they have?

We don’t know how they’re all connected and exactly what they do or what the circuits are or what they mean, but you can just quantify how many different kinds of cells there are with sequencing the RNA. And there are a lot more weird and diverse and bespoke cell types in the Steering Subsystem, basically, than there are in the Learning Subsystem. Like the cortical cell types, it seems like there’s enough to build a learning algorithm up there and specify some hyperparameters. And in this Steering Subsystem, there’s like a gazillion, thousands of really weird cells, which might be like the one for the spider flinch reflex and the one for I’m-about-to-taste-salt.

Dwarkesh Patel

Why would each reward function need a different cell type?

Adam Marblestone

Well, this is where you get innately wired circuits. In the learning algorithm part, in the Learning Subsystem, you specify the initial architecture, you specify a learning algorithm. All the juice is happening through plasticity of the synapses, changes of the synapses within that big network. But it’s a relatively repeating architecture, how it’s initialized. It’s just like how the amount of Python code needed to make an eight-layer transformer is not that different from one that makes a three-layer transformer. You’re just replicating.

Whereas all this Python code for the reward function, if superior colliculus sees something that’s skittering and you’re feeling goosebumps on your skin or whatever, then trigger spider reflex, that’s just a bunch of bespoke, species-specific, situation-specific crap. The cortex doesn’t know about spiders, it just knows about layers.

Dwarkesh Patel

But you’re saying that the only way to write this reward function is to have a special cell type?

Adam Marblestone

Yeah, well, I think so. I think you either have to have special cell types or you have to somehow otherwise get special wiring rules that evolution can say this neuron needs to wire to this neuron, without any learning. And the way that that is most likely to happen, I think, is that those cells express different receptors and proteins that say, “Okay, when this one comes in contact with this one, let’s form a synapse.” So it’s genetic wiring, and those need cell types to do it.

Dwarkesh Patel

I’m sure this would make a lot more sense if I knew 101 neuroscience, but it seems like there’s still a lot of complexity, or generality rather, in the Steering Subsystem. So if the Steering Subsystem has its own visual system that’s separate from the visual cortex, different features still need to plug into that vision system. So the spider thing needs to plug into it and also the love thing needs to plug into it, et cetera, et cetera. So it seems complicated.

Adam Marblestone

It’s still complicated. That’s all the more reason why a lot of the genomic real estate on the genome, and in terms of these different cell types and so on, would go into wiring up the Steering Subsystem, pre-wiring it.

Dwarkesh Patel

Can we tell how much of the genome is clearly working? So I guess you could tell how many are relevant to producing the RNA that manifest or the epigenetics that manifest in different cell types in the brain. Right?

Adam Marblestone

Yeah. This is what the cell types help you get at. I don’t think it’s exactly like, “Oh, this percent of the genome is doing this”, but you could say, “Okay, in all these Steering Subsystem subtypes, how many different genes are involved in specifying which is which and how they wire? And how much genomic real estate do those genes take up versus the ones that specify visual cortex versus auditory cortex? You’re just reusing the same genes to do the same thing twice. Whereas the spider reflex hooking up… Yes, you’re right. They have to build a vision system and they have to build some auditory systems and touch systems and navigation-type systems.

Even feeding into the hippocampus and stuff like that, there’s head direction cells. Even the fly brain has innate circuits that figure out its orientation and help it navigate in the world. It uses vision, figures out its optical flow of how it’s flying and how its flight is related to the wind direction. It has all these innate stuff that I think in the mammal brain we would all lump that into the Steering Subsystem. There’s a lot of work. So all the genes that basically go into specifying all the things a fly has to do, we’re going to have stuff like that too, just all in the Steering Subsystem.

Dwarkesh Patel

But do we have some estimate of like, “Here’s how many nucleotides, here are many megabases it takes to—”

Adam Marblestone

I don’t know. I mean, I think you might be able to talk to biologists about this. I mean, we have a lot in common with yeast from a genes perspective. Yeast is still used as a model for some amount of drug development and stuff like that in biology. And so much of the genome is just going towards you having a cell at all, it can recycle waste, it can get energy, it can replicate.

And then what do we have in common with a mouse? So we do know at some level that the differences between us and a chimpanzee or something—and that includes the social instincts and the more advanced differences in cortex and so on—it’s a tiny number of genes that go into this additional amount of making the eight-layer transformer instead of the six-layer transformer or tweaking that reward function.

Dwarkesh Patel

This would help explain why the hominid brain exploded in size so fast. Presumably, tell me if this is correct, but under this story, social learning or some other thing increased the ability to learn from the environment. It increased our sample efficiency. Instead of having to go and kill the boar yourself and figure out how to do that, you can just be like, “The elder told me this is how you make a spear.” Now it increases the incentive to have a bigger cortex, which can learn these things.

Adam Marblestone

Yes and that can be done with a relatively few genes, because it’s really replicating what the mouse already has, making more of it. It’s maybe not exactly the same and there may be tweaks, from a genome perspective, you don’t have to reinvent all this stuff.

Dwarkesh Patel

So then how far back in the history of the evolution of the brain does the cortex go back? Is the idea that the cortex has always figured out this omnidirectional inference thing, that’s been a solved problem for a long time? Then the big unlock with primates is that we got the reward function, which increased the returns to having omnidirectional inference?

Adam Marblestone

It’s a good question.

Dwarkesh Patel

Or is the omnidirectional inference also something that took a while to unlock?

Adam Marblestone

I’m not sure that there’s agreement about that. I think there might be specific questions about language. Are there tweaks, whether that’s through auditory and memory, some combination auditory memory regions? There may also be macro-wiring where you need to wire auditory regions into memory regions or something like that, and into some of these social instincts to get language, for example, to happen. But that might also be a small number of gene changes to be able to say, “Oh, I just need from my temporal lobe over here, going over to the auditory cortex, something.”

There is some evidence for the Broca’s area, Wernicke’s area. They’re connected with the hippocampus and so on and prefrontal cortex. So there’s like some small number of genes maybe for enabling humans to really properly do language. That could be a big one. But is it that something changed about the cortex and it became possible to do these things? Or is that that potential was already there, but there wasn’t the incentive to expand that capability and then use it, wire it to these social instincts and use it more? I would lean somewhat toward the latter. I think a mouse has a lot of similarity in terms of cortex as a human.

Dwarkesh Patel

Although there’s Suzana Herculano-Houzel‘s work on how the number of neurons scales better with weight with primate brains than it does with rodent brains. So does that suggest that there actually was some improvement in the scalability of the cortex?

Adam Marblestone

Maybe, maybe. I’m not super deep on this. There may have been changes in architecture, changes in the folding, changes in neuron properties and stuff that somehow slightly tweak this. But there’s still a scaling. either way.

Dwarkesh Patel

That’s right.

Adam Marblestone

So I’m not saying there isn’t something special about humans in the architecture of the Learning Subsystem at all. But yeah I think it’s pretty widely thought that this is expanded. But then the question is, “Okay, well, how does that fit in also with the Steering Subsystem changes and the instincts that make use of this and allow you to bootstrap using this effectively?”

But just to say a few other things, even the fly brain has some amount, even very far back… I mean, I think you’ve read this great book, A Brief History of Intelligence, right? I think this is a really good book. Lots of AI researchers think this is a really good book it seems.

You have some amount of learning going back all the way to anything that has a brain. Basically you have something like primitive reinforcement learning, going back at least to vertebrates. Imagine a zebrafish. Then you have these other branches. Birds may have reinvented something cortex-like. It doesn’t have the six layers, but they have something a little bit cortex-like. So some of those things after reptiles, in some sense birds and mammals both made a somewhat cortex-like, but differently organized thing.

But even a fly brain has associative learning centers that actually do things that maybe look a little bit like this Thought Assessor concept from Byrnes, where there’s a specific dopamine signal to train specific subgroups of neurons in the fly mushroom body to associate different sensory information with, “Am I going to get food now?” or “Am I going to get hurt now?”

Dwarkesh Patel

Brief tangent. I remember reading in one blog post that Beren Millidge wrote that the parts of the cortex which are associated with audio and vision have scaled disproportionately between other primates and humans, whereas the parts associated, say, with odor have not. And I remember him saying something like that this is explained by that kind of data having worse scaling law properties. Maybe he meant this, but I think another interpretation of actually what’s happening there is that these social reward functions that are built into the Steering Subsystem needed to make use more of being able to see your elders and see what the visual cues are and hear what they’re saying. And in order to make sense of these cues which guide learning, you needed to activate the vision and audio more than odor.

Adam Marblestone

I mean, there’s all this stuff. I feel like it’s come up in your shows before, actually. But like even the design of the human eye where you have the pupil and the white and everything, we are designed to be able to establish relationships based on joint eye contact. Maybe this came up in the Sutton episode. I can’t remember. But yeah, we have to bootstrap to the point where we can detect eye contact and where we can communicate by language. That’s like what the first couple years of life are trying to do.

Dwarkesh Patel

Okay, I want to ask you about RL. So currently, the way these LLMs are trained, if they solve the unit test or solve a math problem, that whole trajectory, every token in that trajectory is upweighted. What’s going on with humans? Are there different types of model-based versus model-free that are happening in different parts of the brain?

Adam Marblestone

Yeah, I mean, this is another one of these things. Again, all my answers to these questions, any specific thing I say, it’s all just saying that directionally we can explore around this. I find this interesting, maybe I feel like the literature points in these directions in some very broad way. What I actually want to do is go and map the entire mouse brain and figure this out comprehensively and make neuroscience a ground-truth science. So I don’t know, basically.

But first of all, I think with Ilya on the podcast, he was like, “It’s weird that you don’t use value functions, right?” You use the dumbest form of RL basically. Of course these people are incredibly smart and they’re optimizing for how to do it on GPUs and it’s really incredible what they’re achieving. But conceptually it’s a really dumb form of RL, even compared to what was being done 10 years ago. Even the Atari game-playing stuff was using Q-learning, which is basically a kind of temporal difference learning. The temporal difference learning basically means you have some kind of a value function of what action I choose now doesn’t just tell me literally what happens immediately after this. It tells me what is the long-run consequence of that for my expected total reward or something like that.

So you would have value functions like… The fact that we don’t have value functions at all in the LLMs is crazy. I think because Ilya said it, I can say it. I know one one-hundredth of what he does about AI, but it’s kind of crazy that this is working.

But in terms of the brain, I think there are some parts of the brain that are thought to do something that’s very much like model-free RL, that’s parts of the striatum and basal ganglia. It is thought that they have a certain finite relatively small action space. The types of actions they could take, first of all, might be like, “Tell the brainstem and spinal cord to do this motor action? Yes or no.” Or it might be more complicated cognitive-type actions like, “Tell the thalamus to allow this part of the cortex to talk to this other part,” or “Release the memory that’s in the hippocampus and start a new one or something.” But there’s some finite set of actions that come out of the basal ganglia, and that it’s just a very simple RL.

So there are probably parts of other brains and our brain that are just doing very simple naive-type RL algorithms. Layering one thing on top of that is that some of the major work in neuroscience, like Peter Dayan’s work, and a bunch of work that is part of why I think DeepMind did the temporal difference learning stuff in the first place. They were very interested in neuroscience. There’s a lot of neuroscience evidence that the dopamine is giving this reward prediction error signal, rather than just reward, “yes or no, a gazillion time steps in the future.” It’s a prediction error and that’s consistent with learning these value functions.

So there’s that and then there’s maybe higher-order stuff. We have the cortex making this world model. Well, one of the things the cortex world model can contain is a model of when you do and don’t get rewards. Again, it’s predicting what the Steering Subsystem will do. It could be predicting what the basal ganglia will do. You have a model in your cortex that has more generalization and more concepts and all this stuff that says, “Okay, these types of plans, these types of actions will lead in these types of circumstances to reward.” So I have a model of my reward.

Some people also think that you can go the other way. So this is part of the inference picture. There’s this idea of RL as inference. You could say, “Well, conditional on my having a high reward, sample a plan that I would have had to get there.” That’s inference of the plan part from the reward part. I’m clamping the reward as high and inferring the plan, sampling from plans that could lead to that. So if you have this very general cortical thing, it can just do. If you have this very general model-based system and the model, among other things, includes plans and rewards, then you just get it for free, basically.

Dwarkesh Patel

So in neural network parlance, there’s a value head associated to the omnidirectional inference that’s happening in the—

Adam Marblestone

Yes, or there’s a value input.

Dwarkesh Patel

Oh, okay. Interesting.

Adam Marblestone

Yeah and it can predict. One of the almost sensory variables it can predict is what rewards it’s going to get.

Dwarkesh Patel

By the way, speaking about amortizing things, obviously value is like amortized rollouts of looking up reward.

Adam Marblestone

Yeah, something like that. It’s like a statistical average or prediction of it.

Dwarkesh Patel

Tangential thought. Joe Henrich and others have this idea for the way human societies have learned to do things like, how do you figure out that this kind of bean, which actually just almost always poisons you, is edible if you do this ten-step incredibly complicated process, any one of which if you fail, at the bean will be poisonous? How do you figure out how to hunt this seal in this particular way, with this particular weapon, at this particular time of the year, et cetera? There’s no way but just like trying shit over generations. And it strikes me this is actually very much like model-free RL happening at a civilizational level. No, not exactly.

Adam Marblestone

Evolution is the simplest algorithm in some sense. If we believe that all of this can come from evolution, the outer loop can be extremely not foresighted.

Dwarkesh Patel

Right, that’s interesting. Just hierarchies of… Evolution: model-free…

Adam Marblestone

So what does that tell you? Maybe the simple algorithms can just get you anything if you do it enough.

Dwarkesh Patel

Right.

Adam Marblestone

Yeah, I don’t know.

Dwarkesh Patel

So, evolution: model-free. Basal ganglia: model-free. Cortex: model-based. Culture: model-free potentially. I mean you pay attention to your elders or whatever.

Adam Marblestone

Maybe there’s like group selection or whatever of these things is like more model-free. But now I think culture, well, it stores some of the model.

Dwarkesh Patel

Stepping back, is it a disadvantage or an advantage for humans that we get to use biological hardware, in comparison to computers as they exist now? What I mean by this question is, if there’s “the algorithm”, would the algorithm just qualitatively perform much worse or much better if inscribed in the hardware of today? The reason to think it might…. Here’s what I mean. Obviously the brain has had to make a bunch of tradeoffs which are not relevant to computing hardware. It has to be much more energetically efficient. Maybe as a result it has to run on slower speeds so that there can be a smaller voltage gap. So the brain runs at 200 hertz, it has to run on 20 watts. On the other hand, with robotics we’ve clearly experienced that fingers are way more nimble than we can make motors so far. So maybe there’s something in the brain that is the equivalent of cognitive dexterity, which is maybe due to the fact that we can do unstructured sparsity. We can co-locate the memory and the compute.

Adam Marblestone

Yes.

Dwarkesh Patel

Where does this all net out? Are you like, “Fuck, we would be so much smarter if we didn’t have to deal with these brains.” Or are you like—

Adam Marblestone

I think in the end we will get the best of both worlds somehow. I think an obvious downside of the brain is it cannot be copied. You don’t have external read-write access to every neuron and synapse, whereas you do. I can just edit something in the weight matrix in Python or whatever and load that up and copy that. In principle. So the fact that it can’t be copied and random-accessed is very annoying. But otherwise maybe it has a lot of advantages. It also tells you that you want to somehow do the co-design of the algorithm. It maybe even doesn’t change it that much from all of what we discussed, but you want to somehow do this co-design.

So yeah, how do you do it with really slow low-voltage switches? That’s going to be really important for energy consumption. Co-locating memory and compute. I think that hardware companies will probably just try to co-locate memory and compute. They will try to use lower voltages, allow some stochastic stuff.

There are some people that think that all this probabilistic stuff that we were talking about—“Oh, it’s actually energy-based models, so on”—it is doing lots of sampling. It’s not just amortizing everything. The neurons are also very natural for that because they’re naturally stochastic. So you don’t have to do a random number generator in a bunch of Python code basically to generate a sample. The neuron just generates samples and it can tune what the different probabilities are and learn those tunings. So it could be that it’s very co-designed with some kind of inference method or something.

Dwarkesh Patel

It’d be hilarious…. I mean the message I’m taking from this interview is that like all these people that folks make fun of on Twitter, Yann LeCun and Beff Jezos and whatever, I don’t know maybe they got it right.

Adam Marblestone

That is actually one read of it. Granted, I haven’t really worked on AI at all since LLMs took off, so I’m just out of the loop. But I’m surprised and I think it’s amazing how the scaling is working and everything. But yeah, I think Yann LeCun and Beff Jezos are kind of onto something about the probabilistic models or at least possibly. In fact that’s what all the neuroscientists and all the AI people thought until 2021 or something.

Dwarkesh Patel

Right. So there’s a bunch of cellular stuff happening in the brain that is not just about neuron-to-neuron synaptic connections. How much of that is functionally doing more work than the synapses themselves are doing versus it’s just a bunch of kludge that you have to do in order to make the synaptic thing work. So with a digital mind, you can nudge the synapse, sorry the parameter, extremely easily. But with a cell to modulate a synapse according to the gradient signal, it just takes all this crazy machinery. So is it actually doing more than it takes extremely little code to do?

Adam Marblestone

I don’t know, but I’m not a believer in the radical, “Oh, actually memory is not synapses mostly, or learning is mostly genetic changes” or something like that. I think it would just make a lot of sense, I think you put it really well for it to be more like the second thing you said. Let’s say you want to do weight normalization across all the weights coming out of your neuron or into your neuron. Well, you probably have to somehow tell the nucleus of the cell about this and then have that send everything back out to the synapses or something. So there’s going to be a lot of cellular changes. Or let’s say that you just had a lot of plasticity and you’re part of this memory. Now that’s got consolidated into the cortex or whatever. Now we want to reuse you as a new one that can learn again.

There’s going to be a ton of cellular changes, so there’s going to be tons of stuff happening in the cell. But algorithmically, it’s not really adding something beyond these algorithms. It’s just implementing something that in a digital computer is very easy for us to go and just find the weights and change them. In a cell, it just literally has to do all this with molecular machines itself without any central controller. It’s kind of incredible.

There are some things that cells do, I think, that seem more convincing. One of the things the cerebellum has to do is predict over time. What is the time delay? Let’s say that I see a flash and then some number of milliseconds later, I’m going to get a puff of air in my eyelid or something. The cerebellum can be very good at predicting what’s the timing between the flash and the air puff, so that now your eye will just close automatically. The cerebellum is involved in that type of reflex, learned reflex.

There are some cells in the cerebellum where it seems like the cell body is playing a role in storing that time constant, changing that time constant of delay, versus that all being somehow done with like, “I’m going to make a longer ring of synapses to make that delay longer.” No, the cell body will just store that time delay for you. So there are some examples, but I’m not a believer out of the box in essentially this theory that what’s happening is changes in connections between neurons and that that’s the main algorithmic thing that’s going on. I think there’s very good reason to still believe that it’s that rather than some crazy cellular stuff.

Dwarkesh Patel

Going back to this whole perspective of how our intelligence is not just this omnidirectional inference thing that builds a world model, but really this system that teaches us what to pay attention to what the important salient factors are to learn from, et cetera. I want to see if there’s some intuition we can drive from this about what different kinds of intelligences might be like. So it seems like AGI or superhuman intelligence should still have this ability to learn a world model that’s quite general, but then it might be incentivized to pay attention to different things that are relevant for the modern post-singularity environment. How different should we expect different intelligences to be?

Adam Marblestone

I think one way to think about this question is, is it actually possible to make the paperclip maximizer or whatever? If you try to make the paperclip maximizer, does it end up just not being smart or something like that because the only reward function it had was to make paperclips? I’d say, can you do that? I don’t know. If I channel Steve Byrnes more, I think he’s very concerned that the minimum viable things in the Steering Subsystem that you need to get something smart is way less than the minimum viable set of things you need for it to have human-like social instincts and ethics and stuff like that.

So a lot of what you want to know about the Steering Subsystem is actually the specifics of how you do alignment essentially, or what human behavior and social instincts is versus just what you need for capabilities. We talked about it in a slightly different way because we were sort of saying, “Well, in order for humans to learn socially, they need to make eye contact and learn from others.” But we already know from LLMs that depending on your starting point, you can learn language without that stuff. So I think that it probably is possible to make super powerful model-based RL optimizing systems and stuff like that that don’t have most of what we have in the human brain reward functions and as a consequence might want to maximize paperclips. And that’s a concern.

Dwarkesh Patel

But you’re pointing out that in order to make a competent paperclip maximizer, the kind of thing that can build spaceships and learn physics and whatever, it needs to have some drives which elicit learning, including say curiosity and exploration.

Adam Marblestone

Yeah, curiosity, interest in others, interest in social interactions. But that’s pretty minimal I think. And that’s true for humans, but it might be less true for something that’s already pretrained as an LLM or something. So most of why we want to know the Steering Subsystem, I think if I’m channeling Steve, is alignment reasons.

Dwarkesh Patel

How confident are we that we even have the right algorithmic conceptual vocabulary to think about what the brain is doing? What I mean by this is that there was one big contribution to AI from neuroscience which was this idea of the neuron in the 1950s, just this original contribution. But then it seems like a lot of what we’ve learned afterwards about what the high-level algorithm the brain is implementing, from the backprop to if there’s something analogous to backprop happening in the brain to “Oh is V1 doing something like CNNs” to TD learning and Bellman equations, actor-critic, whatever… It seems inspired by this dynamic where we come up with some idea, maybe we can make AI neural networks work this way, and then we notice that something in the brain also works that way. So why not think there’s more things like this.

Adam Marblestone

There may be. I think the reason that I think that we might be onto something is that the AIs we’re making based on these ideas are working surprisingly well. There’s also a bunch of just empirical stuff. Convolutional neural nets and variants of convolutional neural nets. I’m not sure what the absolute latest is, but compared to other models in computational neuroscience of what the visual system is doing, they are just more predictive. You can just score, even pretrained on cat pictures and stuff, CNNs, what is the representational similarity that they have on some arbitrary other image compared to the brain activations measured in different ways? Jim DiCarlo’s lab has this brain score and the AI model is actually… There seems to be some relevance there. Neuroscience doesn’t necessarily have something better than that.

So yes, that’s just recapitulating what you’re saying, that the best computational neuroscience theories we have seem to have been invented largely as a result of AI models and finding things that work. So find backprop works and then saying, “Can we approximate backprop with cortical circuits?” or something. There’s been things like that.

Now, some people totally disagree with this. György Buzsáki is a neuroscientist who has a book called The Brain from the Inside Out where he basically says all our psychology concepts, AI concepts, all this stuff is just made-up stuff. What we actually have to do is figure out what is the actual set of primitives that the brain actually uses. And our vocabulary is not going to be adequate to that. We have to start with the brain and make new vocabulary rather than saying backprop and then try to apply that to the brain or something like that. He studies a lot of oscillations and stuff in the brain as opposed to individual neurons and what they do.

I don’t know. I think that there’s a case to be made for that. And from a research program design perspective, one thing we should be trying to do is just simulate a tiny worm or a tiny zebrafish, almost as biophysical or as bottom-up as possible. Like get connectome, molecules, activity and just study it as a physical dynamical system and look at what it does.

But I don’t know, it just feels like AI is really good fodder for computational neuroscience. Those might actually be pretty good models. We should look at that. I both think that there should be a part of the research portfolio that is totally bottom-up and not trying to apply our vocabulary that we learn from AI onto these systems, and that there should be another big part of this that’s trying to reverse engineer it using that vocabulary or variant of that vocabulary. We should just be pursuing both. My guess is that the reverse engineering one is actually going to work-ish or something. Like we do see things like TD learning, which Sutton also invented separately.

Dwarkesh Patel

That must be a crazy feeling to just like—

Adam Marblestone

Yeah, that’s crazy.

Dwarkesh Patel

This equation I wrote down is like in the brain.

Adam Marblestone

It seems like the dopamine is doing some of that, yeah.

Dwarkesh Patel

So let me ask you about this. You guys are funding different groups that are trying to figure out what’s up in the brain. If we had a perfect representation, however you define it, of the brain, why think it would actually let us figure out the answer to these questions? We have neural networks which are way more interpretable, not just because we understand what’s in the weight matrices, but because there are weight matrices. There are these boxes with numbers in them. Even then we can tell very basic things. We can kind of see circuits for very basic pattern matching of following one token with another. I feel like we don’t really have an explanation of why LLMs are intelligent just because they’re interpretable.

Adam Marblestone

I think I would somewhat dispute it. We have some description of what the LLM is fundamentally doing. What that’s doing is that I have an architecture and I have a learning rule and I have hyperparameters and I have initialization and I have training data.

Dwarkesh Patel

But those are things we learned because we built them, not because we interpreted them from seeing the weights. The analogous thing to connectome is like seeing the weights.

Adam Marblestone

What I think we should do is we should describe the brain more in that language of things like architectures, learning rules, initializations, rather than trying to find the Golden Gate Bridge circuit and saying exactly how this neuron actually… That’s going to be some incredibly complicated learned pattern. Konrad Kording and Tim Lillicrap have this paper from a while ago, maybe five years ago, called “What does it mean to understand a neural network?” What they say is basically that you could imagine you train a neural network to compute the digits of pi or something. It’s like some crazy pattern. You also train that thing to predict the most complicated thing you find, predict stock prices, basically predict really complex systems, computationally complete systems. I could train a neural network to do cellular automata or whatever crazy thing. It’s like, we’re never going to be able to fully capture that with interpretability, I think. It’s just going to just be doing really complicated computations internally.

But we can still say that the way it got that way is that it had an architecture and we gave it this training data and it had this loss function. So I want to describe the brain in the same way. And I think that this framework that I’ve been kind of laying out is that we need to understand the cortex and how it embodies a learning algorithm. I don’t need to understand how it computes “Golden Gate Bridge.”

Dwarkesh Patel

But if you can see all the neurons, if you have the connectome, why does that teach you what the learning algorithm is?

Adam Marblestone

Well, I guess there are a couple different views of it. So it depends on these different parts of this portfolio. On the totally bottom-up, we-have-to-simulate-everything portfolio, it kind of just doesn’t. You have to make a simulation of the zebrafish brain or something and then you see what are the emergent dynamics in this and you come up with new names and new concepts and all that. That’s the most extreme bottom-up neuroscience view. But even there the connectome is really important for doing that biophysical or bottom-up simulation.

But on the other hand you can say, “Well, what if we can actually apply some ideas from AI?” We basically need to figure out, is it an energy-based model or is it an amortized VAE-type model? Is it doing backprop or is it doing something else? Are the learning rules local or global? If we have some repertoire of possible ideas about this, just think of the connectome as a huge number of additional constraints that will help to refine, to ultimately have a consistent picture of that.

I think about this for the Steering Subsystem stuff too, just very basic things about it. How many different types of dopamine signal or of Steering Subsystem signal or thought assessor or so on… How many different types of what broad categories are there? Like even this very basic information that there’s more cell types in the hypothalamus than there are in the cortex, that’s new information about how much structure is built there versus somewhere else. How many different dopamine neurons are there? Is the wiring between prefrontal and auditory the same as the wiring between prefrontal and visual? The most basic things, we don’t know. The problem is learning even the most basic things by a series of bespoke experiments takes an incredibly long time. Whereas just learning all that at once by getting a connectome is just way more efficient.

Dwarkesh Patel

What is the timeline on this? Presumably the idea of this is, first, to inform the development of AI. You want to be able to figure out how we get AIs to want to care about what other people think of its internal thought pattern. But interp researchers are making progress on this question just by inspecting normal neural networks. There must be some feature…

Adam Marblestone

You can do interp on LLMs that exist. You can’t do interp on a hypothetical model-based reinforcement algorithm like the brain that we will eventually converge to when we do AGI.

Dwarkesh Patel

Fair. But what timelines on AI do you need for this research to be practical and relevant?

Adam Marblestone

I think it’s fair to say it’s not super practical and relevant if you’re in an AI 2027 scenario. And so what science I’m doing now is not going to affect the science of ten years from now. Because what’s going to affect the science of 10 years from now is the outcome of this AI 2027 scenario. It kind of doesn’t matter that much probably if I have the connectome, maybe it slightly tweaks certain things.

But I think there’s a lot of reason to think maybe that we will get a lot out of this paradigm. But then the real thing, the thing that is the single event that is transformative for the entire future or something type event is still more than five years away or something.

Dwarkesh Patel

Is that because we haven’t captured omnidirectional inference, we haven’t figured out the right ways to get a mind to pay attention to things in a way that makes sense?

Adam Marblestone

I mean, I would take the entirety of your collective podcast with everyone as showing the distribution of these things. I don’t know. What was Karpathy’s timeline, right? What’s Demis’s timeline? So not everybody has a three-year timeline.

Dwarkesh Patel

But there are different reasons and I’m curious which ones are yours.

Adam Marblestone

What are mine? I don’t know, I’m just watching your podcast. I’m trying to understand the distribution. I don’t have a super strong claim that LLMs can’t do it.

Dwarkesh Patel

But is the crux the data efficiency or…?

Adam Marblestone

I think part of it is just that it is weirdly different from all this brain stuff. So intuitively it’s just weirdly different than all this brain stuff and I’m kind of waiting for the thing that starts to look more like brain stuff. I think if AlphaZero, and model-based RL and all these other things that were being worked on 10 years ago, had been giving us the GPT-5 type capabilities, then I would be like, “Oh wow, we’re both in the right paradigm and seeing the results a priori. So my prior and my data are agreeing.” Now it’s like, “I don’t know what exactly my data is. Looks pretty good, but my prior is sort of weird so I don’t have a super strong opinion on it.”

So I think there’s a possibility that essentially all other scientific research that is being done is somehow obviated. But I don’t put a huge amount of probability on that. I think my timelines might be more in the 10-year-ish range. If that’s the case, I think there is probably a difference between a world where we have connectomes on hard drives and we have an understanding of Steering Subsystem architecture. We’ve compared even the most basic properties of what are the reward functions, cost function, architecture, et cetera, of a mouse versus a shrew versus a small primate, et cetera.

Dwarkesh Patel

Is this practical in 10 years?

Adam Marblestone

I think it has to be a really big push.

Dwarkesh Patel

How much funding, how does it compare to where we are now?

Adam Marblestone

It’s like low billions-dollar scale funding in a very concerted way I would say.

Dwarkesh Patel

And how much is on it now?

Adam Marblestone

So if I just talk about some of the specific things we have going on with connectomics… E11 Bio is our main thing on connectomics. They are trying to make the technology of connectomic brain mapping several orders of magnitude cheaper. The Wellcome Trust put out a report a year or two ago that said to get one mouse brain, the first mouse brain connectome would be a several billion dollars project. E11 technology, and the suite of efforts in the field, is trying to get a single mouse connectome down to low tens of millions of dollars.

That’s a mammal brain. A human brain is about 1,000 times bigger. If with a mouse brain you can get to $10 million or $20 million, $30 million, with technology, if you just naively scale that, a human brain is now still billions of dollars, to just do one human brain. Can you go beyond that? Can you get a human brain for less than a billion? But I’m not sure you need every neuron in the human brain.

We want to, for example, do an entire mouse brain and a human Steering Subsystem and the entire brains of several different mammals with different social instincts. So with a bunch of technology push and a bunch of concerted effort, real significant progress if it’s focused effort can be done in the hundreds of millions to low billions scale.

Dwarkesh Patel

What is the definition of a connectome? Presumably it’s not a bottom-up biophysics model. So is it just that it can estimate the input-output of a brain? What is the level of abstraction?

Adam Marblestone

You can give different definitions and one of the things that’s cool… So the standard approach to connectomics uses the electron microscope and very, very thin slices of brain tissue. It’s basically labeling. The cell membranes are going to show up, scatter electrons a lot and everything else is going to scatter electrons less. But you don’t see a lot of details of the molecules, which types of synapses, different synapses of different molecular combinations and properties.

E11 and some other research in the field has switched to an optical microscope paradigm. With optical, the photons don’t damage the tissue, so you can wash it and look at fragile gentle molecules. So with E11’s approach, you can get a “molecularly annotated connectome.” That’s not just who is connected to who by some synapse, but what are the molecules that are present at the synapse? What type of cell is that?

A molecularly annotated connectome, that’s not exactly the same as having the synaptic weights. That’s not exactly the same as being able to simulate the neurons and say what’s the functional consequence of having these molecules and connections. But you can also do some amount of activity mapping and try to correlate structure to function. Train an ML model basically to predict the activity from the connectome.

Dwarkesh Patel

What are the lessons to be taken away from the Human Genome Project? One way you could look at it is that it was a mistake and you shouldn’t have spent billions of dollars getting one genome mapped. Rather you should have just invested in technologies which have now allowed us to map genomes for hundreds of dollars.

Adam Marblestone

Well, George Church was my PhD advisor and he’s pointed out that it was $3 billion or something, roughly $1 per base pair for the first genome. Then the National Human Genome Research Institute basically structured the funding process right. They got a bunch of companies competing to lower the cost. And then the cost dropped like a million-fold in 10 years because they changed the paradigm from macroscopic chemical techniques to these individual DNA molecules which would make a little cluster of DNA molecules on the microscope and you would see just a few DNA molecules at a time on each pixel of the camera. It would give you a different, in parallel, look at different fragments of DNA. So you parallelize the thing by millions-fold. That’s what reduced the cost by millions-fold.

With switching from electron microscopy to optical connectomics, potentially even future types of connectomics technology, we think there should be similar patterns. That’s why E11, the Focus Research Organization, started with technology development rather than starting with saying we’re going to do a human brain or something and let’s just brute force it. We said let’s get the cost down with new technology. But then it’s still a big thing. Even with new next-generation technology, you still need to spend hundreds of millions on data collection.

Dwarkesh Patel

Is this going to be funded with philanthropy, by governments, by investors?

Adam Marblestone

This is very TBD and very much evolving in some sense as we speak. I’m hearing some rumors going around of connectomics-related companies potentially forming. So far E11 has been philanthropy. The National Science Foundation just put out this call for Tech Labs, which is somewhat FRO-inspired or related. You could have a tech lab for actually going and mapping the mouse brain with us and that would be philanthropy plus government still in a nonprofit, open-source framework. But can companies accelerate that? Can you credibly link connectomics to AI in the context of a company and get investment for that? It’s possible.

Dwarkesh Patel

I mean the cost of training these AIs is increasing so much. If you could tell some story of not only are we going to figure out some safety thing, but in fact once we do that, we’ll also be able to tell you how AI works… You should go to these AI labs and just be like, “Give me one one-hundredth of your projected budget in 2030.”

Adam Marblestone

I sort of tried a little bit seven or eight years ago and there was not a lot of interest. Maybe now there would be. But all the things that we’ve been talking about, it’s really fun to talk about, but it’s ultimately speculation. What is the actual reason for the energy efficiency of the brain, for example? Is it doing real inference or amortized inference or something else? This is all answerable by neuroscience. It’s going to be hard, but it’s actually answerable. So if you can only do that for low billions of dollars or something to really comprehensively solve that, it seems to me, in the grand scheme of trillions of dollars of GPUs and stuff, it actually makes sense to do that investment.

Dwarkesh Patel

Also, there’s been many labs that have been launched in the last year where they’re raising on the valuation of billions for things which are quite credible but are not like, “Our ARR next quarter is going to be whatever.” It’s like we’re going to discover materials and—

Adam Marblestone

Yes, moonshot startups or billionaire-backed startups. Moonshot startups I see as on a continuum with FROs. FROs are a way of channeling philanthropic support and ensuring that it’s open source public benefit, various other things that may be properties of a given FRO. But yes, billionaire-backed startups, if they can target the right science, the exact right science.

I think there’s a lot of ways to do moonshot neuroscience companies that would never get you the connectome. It’s like, “Oh, we’re going to upload the brain” or something, but never actually get the mouse connectome or something. These fundamental things that you need to get to ground truth the science. There are lots of ways to have a moonshot company go wrong and not do the actual science. But there also may be ways to have companies or big corporate labs get involved and actually do it correctly.

Dwarkesh Patel

This brings to mind an idea that you had in a lecture you gave five years ago about. Do you want to explain behavior cloning?

Adam Marblestone

Actually this is funny because the first time I saw this idea, I think it might have been in a blog post by Gwern. There’s always a Gwern blog post. There are now academic research efforts and some amount of emerging company-type efforts to try to do this.

Normally, let’s say I’m training an image classifier or something. I show it pictures of cats and dogs or whatever and they have the label “cat” or “dog”. And I have a neural network that’s supposed to predict the label “cat” or “dog” or something. That is a limited amount of information per label that you’re putting in. It’s just “cat” or “dog”.

What if I also had, “Predict what is my neural activity pattern when I see a cat or when I see a dog and all the other things?” If you add that as an auxiliary loss function or an auxiliary prediction task, does that sculpt the network to know the information that humans know about cats and dogs and to represent it in a way that’s consistent with how the brain represents it and the kind of representational dimensions or geometry of how the brain represents things, as opposed to just having these labels? Does that let it generalize better? Does that let it have richer labeling?

Of course that sounds really challenging. It’s very easy to generate lots and lots of labeled cat pictures. Scale AI or whatever can do this. It is harder to generate lots and lots of brain activity patterns that correspond to things that you want to train the AI to do. But again, this is just a technological limitation of neuroscience. If every iPhone was also a brain scanner, you would not have this problem and we would be training AI with the brain signals. It’s just the order in which technology has developed is that we got GPUs before we got portable brain scanners.

Dwarkesh Patel

What is the ML analog, what you’d be doing here? Because when you distill models, you’re still looking at the final layer of the log probs across all—

Adam Marblestone

If you distill one model into another, that is a certain thing. You are just trying to copy one model into another. I think that we don’t really have a perfect proposal to distill the brain. To distill the brain you need a much more complex brain interface. Maybe you could also do that. You could make surrogate models. Andreas Tolias and people like that are doing some amount of neural network surrogate models of brain activity data. Instead of having your visual cortex do the computation, just have the surrogate model. So you’re distilling your visual cortex into a neural network to some degree. That’s a kind of distillation.

This is doing something a little different. This is basically just saying I’m adding an auxiliary… I think of it as regularization or I think of it as adding an auxiliary loss function that’s smoothing out the prediction task to also always be consistent with how the brain represents it. It might help you with things like adversarial examples, for example.

Dwarkesh Patel

But what exactly are you predicting? You’re predicting the internal state of the brain?

Adam Marblestone

Yes. So in addition to predicting the label, a vector of labels like yes cat, not dog, yes, not boat, one-hot vector or whatever of yes, it’s cat, instead of these gazillion other categories, let’s say in this simple example. You’re also predicting a vector which is all these brain signal measurements.

So Gwern, anyway, had this long-ago blog post of like, “Oh, this is an intermediate thing. We talk about whole brain emulation, we talk about AGI, we talk about brain-computer interface. We should also be talking about this brain-data-augmented thing that’s trained on all your behavior, but is also trained on predicting some of your neural patterns.”

Dwarkesh Patel

And you’re saying the Learning System is already doing this through the Steering System?

Adam Marblestone

Yeah, and our brain, our learning system also has to predict the Steering Subsystem as an auxiliary task. That helps the Steering Subsystem. Now, the Steering Subsystem can access that predictor and build a cool reward function using it.

Dwarkesh Patel

Separately, you’re on the board of Lean, which is this formal math language that mathematicians use to prove theorems and so forth. Obviously there’s a bunch of conversation right now about AI automating math. What’s your take?

Adam Marblestone

Well, I think that there are parts of math that it seems like it’s pretty well on track to automate. First of all, Lean was developed for a number of years at Microsoft and other places. It has become one of the Convergent Focused Research Organizations to kind of drive more engineering and focus onto it.

So Lean is this programming language where instead of expressing your math proof on pen and paper, you express it in this programming language Lean. And then at the end, if you do that that way, it is a verifiable language so that you can click “verify” and Lean will tell you whether the conclusions of your proof actually follow perfectly from your assumptions of your proof. So it checks whether the proof is correct automatically.

By itself, this is useful for mathematicians collaborating and stuff like that. If I’m some amateur mathematician and I want to add to a proof, Terry Tao is not going to just believe my result. But if Lean says it’s correct, it’s just correct. So it makes it easy for collaboration to happen, but it also makes it easy for correctness of proofs to be an RL signal in very much RLVR. Formalized math proofing—so formal means it’s expressed in something like Lean and verifiable—is now mechanically verifiable. That becomes a perfect RLVR task.

I think that that is going to just keep working, it seems like there is at least one billion-dollar valuation company, Harmonic, based on this. AlphaProof is based on this. A couple other emerging really interesting companies. I think that this problem of RLVRing the crap out of math proving is going to work and we will be able to have things that search for proofs and find them in the same way that we have AlphaGo or what have you that can search for ways of playing the game of Go. With that verifiable signal, it works.

So does this solve math? There is still the part that has to do with conjecturing new interesting ideas. There’s still the conceptual organization of math of what is interesting. How do you come up with new theorem statements in the first place? Or even the very high-level breakdown of what strategies you use to do proofs. I think this will shift the burden of that so that humans don’t have to do a lot of the mechanical parts of math. Validating lemmas and proofs and checking if the statement of this in this paper is exactly the same as that paper and stuff like that. That will just work.

If you really think we’re going to get all these things we’ve been talking about, real AGI would also be able to make conjectures. Bengio has a paper, more like a theoretical paper. There are probably a bunch of other papers emerging about this. Is there a loss function for good explanations or good conjectures? That’s a pretty profound question.

A really interesting math proof or statement might be one that compresses lots of information and has lots of implications for lots of other theorems. Otherwise you would have to prove those theorems using long complex passive inference. Here, if you have this theorem, this theorem is correct, and you have short passive inference to all the other ones. And it’s a short compact statement. So it’s like a powerful explanation that explains all the rest of math. And part of what math is doing is making these compact things that explain the other things.

Dwarkesh Patel

It’s like the Kolmogorov complexity of this statement or something.

Adam Marblestone

Yeah, of generating all the other statements, given that you know this one or stuff like that. Or if you add this, how does it affect the complexity of the rest of the network of proofs? So can you make a loss function that adds, “Oh, I want this proof to be a really highly powerful proof”? I think some people are trying to work on that. So maybe you can automate the creativity part.

If you had true AGI, it would do everything a human can do. So it would also do the things that the creative mathematicians do. But barring that, I think just RLVRing the crap out of proofs, I think that’s going to be just a really useful tool for mathematicians. It’s going to accelerate math a lot and change it a lot, but not necessarily immediately change everything about it.

Will we get mechanical proof of the Riemann hypothesis or something like that, or things like that? Maybe, I don’t know. I don’t know enough details of how hard these things are to search for, and I’m not sure anyone can fully predict that, just as we couldn’t exactly predict when Go would be solved or something like that.

I think it’s going to have lots of really cool applied applications. So one of the things you want to do is you want to have provably stable, secure, unhackable software. So you can write math proofs about software and say, “This code, not only does it pass these unit tests, but I can mathematically prove that there’s no way to hack it in these ways, or no way to mess with the memory”, or these types of things that hackers use, or it has these properties. You can use the same Lean and same proof to do formally verified software.

I think that’s going to be a really powerful piece of cybersecurity that’s relevant for all sorts of other AI hacking the world stuff. And if you can prove the Riemann hypothesis, you’re also going to be able to prove insanely complex things about very complex software. And then you’ll be able to ask the LLM, “Synthesize me a software that I can prove is correct.”

Dwarkesh Patel

Why hasn’t provable programming language taken off as a result of LLMs?

Adam Marblestone

I think it’s starting to. One challenge—we are actually incubating a potential Focused Research Organization on this—is the specification problem. So mathematicians know what interesting theorems they want to formalize. Let’s say I have some code that is involved in running the power grid or something and it has some security properties, well what is the formal spec of those properties? The power grid engineers just made this thing, but they don’t necessarily know how to lift the formal spec from that. And it’s not necessarily easy to come up with the spec that is the spec that you want for your code. People aren’t used to coming up with formal specs and there are not a lot of tools for it.

So you also have this user interface plus AI problem of what security specs should I be specifying? Is this the spec that I wanted? So there’s a spec problem and it’s just been really complex and hard. But it’s only just in the last very short time that the LLMs are able to generate verifiable proofs of things that are useful to mathematicians, starting to be able to do some amount of that for software verification, hardware verification.

But I think if you project the trends over the next couple years, it’s possible that it just flips the tide. Formal methods, this whole field of formal methods or formal verification, provable software. It’s this weird almost backwater of the more theoretical part of programming languages and stuff, very academically flavored often. Although there was this DARPA program that made a provably secure quadcopter helicopter and stuff like that.

Dwarkesh Patel

Secure against… What is the property that is exactly proved? Not for that particular project, but just in general. Because obviously things malfunction for all kinds of reasons.

Adam Marblestone

You could say that what’s going on in this part of the memory over here, which is supposed to be the part the user can access, can’t in any way affect what’s going on in the memory over here or something like that. Things like that.

Dwarkesh Patel

So there’s two questions. One is how useful is this? Two is, how satisfying, as a mathematician, would it be? The fact that there’s this application towards proving that software has certain properties or hardware has certain properties, if that works, that would obviously be very useful. But from a pure… Are we going to figure out mathematics? Is your sense that there’s something about finding that one construction cross-maps to another construction in a different domain, or finding that, “Oh, this lemma, if you redefine this term, it still satisfies what I meant by this term. But a counterexample that previously knocked it down no longer applies.” That kind of dialectical thing that happens in mathematics.

Adam Marblestone

Will the software replace that?

Dwarkesh Patel

Yeah. How much of the value of this sort of pure mathematics just comes from actually just coming up with entirely new ways of thinking about a problem, mapping it to a totally different representation? Do we have examples?

Adam Marblestone

I don’t know. I think of it maybe a little bit like when everybody had to write assembly code or something like that. The amount of fun cool startups that got created was just a lot less or something. Fewer people could do it, progress was more grinding and slow and lonely and so on. You had more false failures because you didn’t get something about the assembly code, rather than the essential thing of was your concept right. Harder to collaborate and stuff like that. And so I think it will be really good.

There is some worry that by not learning to do the mechanical parts of the proofs that you fail to generate the intuitions that inform the more conceptual parts, the creative part.

Dwarkesh Patel

It’s the same with assembly.

Adam Marblestone

Right. So at what point is that applying? With vibe coding, are people not learning computer science or actually are they vibe coding and they’re also simultaneously looking at the LLM that’s explaining these abstract computer science concepts to them and it’s all just all happening faster? Their feedback loop is faster and they’re learning way more abstract computer science and algorithm stuff because they’re vibe coding. I don’t know, it’s not obvious. That might be something about the user interface and the human infrastructure around it.

But I guess there’s some worry that people don’t learn the mechanics and therefore don’t build the grounded intuitions or something. But my hunch is it’s super positive. Exactly, on net, how useful that will be or how much overall math breakthroughs, or math breakthroughs even that we care about, will happen? I don’t know.

One other thing that I think is cool is the accessibility question. Okay, that sounds a little bit corny. Okay, yeah, more people can do math, but who cares? But I think there’s lots of people that could have interesting ideas. Like maybe the quantum theory of gravity or something. Yeah, one of us will come up with the quantum theory of gravity instead of a card-carrying physicist. In the same way that Steve Byrnes is reading the neuroscience literature and he hasn’t been in the neuroscience lab that much. But he’s able to synthesize across the neuroscience literature and be like, “Oh, Learning Subsystem, Steering Subsystem. Does this all make sense?” He’s an outsider neuroscientist in some ways. Can you have outsider string theorists or something, because the math is just done for them by the computer? And does that lead to more innovation in string theory? Maybe yes.

Dwarkesh Patel

Interesting. Okay, so if this approach works and you’re right that LLMs are not the final paradigm, and suppose it takes at least 10 years to get the final paradigm in that world. There’s this fun sci-fi premise where you have… Terence Tao today had a tweet where he’s like, “These models are like automated cleverness but not automated intelligence.” And you can quibble with the definitions there. But if you have automated cleverness and you have some way of filtering—which if you can formalize and prove things that the LLMs are saying you could do—then you could have this situation where quantity has a quality all of its own.

So what are the domains of the world which could be put in this provable symbolic representation? So in the world where AGI is super far away, maybe it makes sense to literally turn everything the LLMs ever do, or almost everything they do, into super provable statements. So LLMs can actually build on top of each other because everything they do is super provable.

Maybe this is just necessary because you have billions of intelligences running around. Even if they are super intelligent, the only way the future AGI civilization can collaborate with each other is if they can prove each step. They’re just brute force churning out… This is what the Jupiter brains are doing.

Adam Marblestone

It’s a universal language, it’s provable. It’s also provable from the perspective of, “Are you trying to exploit me or are you sending me some message that’s trying to hack into my brain effectively?” Are you trying to socially influence me? Are you actually just sending me just the information that I need and no more for this?

So davidad, who’s this program director at ARIA now in the UK, he has this whole design of an ARPA-style program, a sort of safeguarded AI that very heavily leverages provable safety properties. Can you apply proofs to… Can you have a world model? But that world model is actually not specified just in neuron activations, but it’s specified in equations. Those might be very complex equations, but if you can just get insanely good at just auto-proving these things with cleverness, auto-cleverness… Can you have explicitly interpretable world models as opposed to neural net world models and move back basically to symbolic methods just because you can just have insane amount of ability to prove things? Yeah, I mean that’s an interesting vision. I don’t know in the next 10 years whether that will be the vision that plays out, but I think it’s really interesting to think about.

Even for math, I mean, Terence Tao is doing some amount of stuff where it’s not about whether you can prove the individual theorems. It’s like let’s prove all the theorems en masse and then let’s study the properties of the aggregate set of proved theorems. Which are the ones that got proved and which are the ones that didn’t? Okay, well that’s the landscape of all the theorems instead of one theorem at a time.

Dwarkesh Patel

Speaking of symbolic representations, one question I was meaning to ask you is, how does the brain represent the world model? Obviously nets out in neurons, but I don’t mean extremely functionally. I mean conceptually, is it in something that’s analogous to the hidden state of a neural network or is it something that’s closer to a symbolic language?

Adam Marblestone

We don’t know. There’s some amount of study of this. There’s these things like face patch neurons that represent certain parts of the face that geometrically combine in interesting ways. That’s with geometry and vision. Is that true for other more abstract things? There’s this idea of cognitive maps. A lot of the stuff that a rodent hippocampus has to learn is place cells and, where is the rodent going to go next and is it going to get a reward there? It’s very geometric. And do we organize concepts with an abstract version of a spatial map?

There’s some questions of can we do true symbolic operations? Can I have a register in my brain that copies a variable to another register regardless of what the content of that variable is? That’s this variable binding problem. Basically I don’t know if we have that machinery or is it more like cost functions and architectures that make some of that approximately emerge, but maybe it would also emerge in a neural net? There’s a bunch of interesting neuroscience research trying to study this, what the representations look like.

Dwarkesh Patel

But what’s your hunch?

Adam Marblestone

Yeah, my hunch is that it’s going to be a huge mess and we should look at the architecture, the loss functions, and the learning rules. I don’t expect it to be pretty in there.

Dwarkesh Patel

Which is that it is not a symbolic language type thing?

Adam Marblestone

Yeah, probably it’s not that symbolic. But other people think very differently.

Dwarkesh Patel

Another random question speaking of binding, what is up with feeling like there’s an experience? All the parts of your brain which are modeling very different things, have different drives, and at least presumably feel like there’s an experience happening right now. Also that across time you feel like…

Adam Marblestone

Yeah, I’m pretty much at a loss on this one. I don’t know. Max Hodak has been giving talks about this recently. He’s another really hardcore neuroscience person, neurotechnology person. The thing I mentioned with Doris maybe also sounds like it might have some touching on this question. But yeah, I don’t think anybody has any idea. It might even involve new physics.

Dwarkesh Patel

Here’s another question which might not have an answer yet. Continual learning, is that the product of something extremely fundamental at the level of even the learning algorithm? You could say, “Look, at least the way we do backprop in neural networks is that you freeze the weight, there’s a training period and you freeze the weights. So you just need this active inference or some other learning rule in order to do continual learning.” Or do you think it’s more a matter of architecture and how memory is exactly stored and what kind of associative memory you have basically?

Adam Marblestone

So continual learning… I don’t know. At the architectural level, there’s probably some interesting stuff that the hippocampus is doing. People have long thought this. What kinds of sequences is it storing? How is it organizing, representing that? How is it replaying it back? What is it replaying back? How exactly does that memory consolidation work? Is it training the cortex using replays or memories from the hippocampus or something like that? There’s probably some of that stuff.

There might be multiple timescales of plasticity or clever learning rules that can simultaneously be storing short-term information and also doing backprop with it. Neurons may be doing a couple things: some fast weight plasticity and some slower plasticity at the same time, or synapses that have many states. I mean, I don’t know. From a neuroscience perspective, I’m not sure that I’ve seen something that’s super clear on what causes continual learning except maybe to say that this systems consolidation idea of hippocampus consolidating cortex. Some people think it is a big piece of this and we still don’t fully understand the details.

Dwarkesh Patel

Speaking of fast weights, is there something in the brain which is the equivalent of this distinction between parameters and activations that we see in neural networks? Specifically in transformers we have this idea that some of the activations are the key and value vectors of previous tokens that you build up over time.

There’s the so-called fast weights that whenever you have a new token, you query them against these activations, but you also obviously can’t query them against all the other parameters in the network which are part of the actual built-in weights. Is there some such distinction that’s analogous?

Adam Marblestone

I don’t know. I mean we definitely have weights and activations. Whether you can use the activations in these clever ways, different forms of actual attention, like attention in the brain… Is that based on, “I’m trying to pay attention”... I think there’s probably several different kinds of actual attention in the brain. I want to pay attention to this area of visual cortex. I want to pay attention to the content in other areas that is triggered by the content in this area. Attention that’s just based on reflexes and stuff like that.

So I don’t know. There’s not just the cortex, there’s also the thalamus. The thalamus is also involved in somehow relaying or gating information. There’s cortico-cortical connections. There’s also some amount of connection between cortical areas that goes through the thalamus. Is it possible that this is doing some sort of matching or constraint satisfaction or matching across keys over here and values over there? Is it possible that it can do stuff like that? Maybe. I don’t know. This is all part of the architecture of this corticothalamic system. I don’t know how transformer-like it is or if there’s anything analogous to that attention. It’d be interesting to find out.

Dwarkesh Patel

We’ve got to give you a billion dollars so you can come on the podcast again and tell me how exactly the brain works.

Adam Marblestone

Mostly I just do data collection. It’s really unbiased data collection so all the other people can figure out these questions.

Dwarkesh Patel

Maybe the final question to go off on is, what was the most interesting thing you learned from the Gap Map? Maybe you want to explain what the Gap Map is.

Adam Marblestone

In the process of incubating and coming up with these Focused Research Organizations, these nonprofit startup-like moonshots that we’ve been getting philanthropists and now government agencies to fund, we talked to a lot of scientists. Some of the scientists were just like, “Here’s the next thing my graduate student will do. Here’s what I find interesting. Exploring these really interesting hypothesis spaces, all the types of things we’ve been talking about.”

Some of them were like, “Here’s this gap. I need this piece of infrastructure. There’s no combination of grad students in my lab or me loosely collaborating with other labs with traditional grants that could ever get me that. I need to have an organized engineering team that builds the miniature equivalent of the Hubble Space Telescope. If I can build that Hubble Space Telescope, then I will unblock all the other researchers in my field or some path of technological progress in the way that the Hubble Space Telescope lifted the boats and improved the life of every astronomer.” But it wasn’t really an astronomy discovery in itself. It was just that you had to put this giant mirror in space with a CCD camera and organize all the people and engineering and stuff to do that. So some of the things we talked to scientists about looked like that.

The Gap Map is just a list of a lot of those things and we call it a Gap Map. I think it’s actually more like a fundamental capabilities map. What are all these things, like mini Hubble space telescopes? And then we organized that into gaps for helping people understand that or search that.

Dwarkesh Patel

What was the most surprising thing you found?

Adam Marblestone

I think I’ve talked about this before, but one thing is just the overall size or shape of it or something like that. It’s a few hundred fundamental capabilities. So if each of these were a deep tech startup-size project, that’s only a few billion dollars or something. If each one of those were a Series A, that’s only… It’s not like a trillion dollars to solve these gaps. It’s lower than that. So that’s one thing. Maybe we assumed that, and that’s what we got. It’s not really comprehensive. It’s really just a way of summarizing a lot of conversations we’ve had with scientists.

I do think that in the aggregate process, things like Lean are actually surprising because I did start from neuroscience and biology and it was very obvious that there’s these -omics. We need genomics, but we also need connectomics. We can engineer E. coli, but we also need to engineer the other cells. There’s somewhat obvious parts of biological infrastructure. I did not realize that math proving infrastructure was a thing and that was emergent from trying to do this.

So I’m looking forward to seeing other things where it’s not actually this hard intellectual problem to solve it. It’s maybe slightly the equivalent of AI researchers just needing GPUs or something like that and focus and really good PyTorch code to start doing this. Which are the fields that do or don’t need that? So fields that have had gazillions of dollars of investment, do they still need some of those? Do they still have some of those gaps or is it only more neglected fields? We’re even finding some interesting ones in actual astronomy, actual telescopes that have not been explored. Maybe because if you’re getting above a critical mass-size project, then you have to have a really big project and that’s a more bureaucratic process with the federal agencies.

Dwarkesh Patel

I guess you just need scale in every single domain of science these days.

Adam Marblestone

Yeah, I think you need scale in many of the domains of science. That does not mean that the low-scale work is not important. It does not mean that creativity, serendipity, etc., and each student pursuing a totally different direction or thesis that you see in universities is not also really key. But I think some amount of scalable infrastructure is missing in essentially every area of science, even math, which is crazy. Because mathematicians I thought just needed whiteboards, but they actually need Lean. They actually need verifiable programming languages and stuff. I didn’t know that.

Dwarkesh Patel

Cool. Adam, this is super fun. Thanks for coming on.

Adam Marblestone

Thank you so much. My pleasure.

Dwarkesh Patel

Where can people find your stuff?

Adam Marblestone

Pleasure. The easiest way now… My adammarblestone.org website is currently down, I guess. But convergentresearch.org can link to a lot of the stuff we’ve been doing.

Dwarkesh Patel

And then you have a great blog, Longitudinal Science.

Adam Marblestone

Longitudinal Science, yes, on WordPress.

Dwarkesh Patel

Cool.

Adam Marblestone

Thank you so much. Pleasure.

I’m confused why some people have short timelines and at the same time are bullish on the current scale up of reinforcement learning atop LLMs. If we’re actually close to a human-like learner, this whole approach of training on verifiable outcomes is doomed.

Currently the labs are trying to bake in a bunch of skills into these models through “mid-training” - there’s an entire supply chain of companies building RL environments which teach the model how to navigate a web browser or use Excel to write financial models.

Either these models will soon learn on the job in a self directed way - making all this pre-baking pointless - or they won’t - which means AGI is not imminent. Humans don’t have to go through a special training phase where they need to rehearse every single piece of software they might ever need to use.

Beren Millidge made interesting points about this in a recent blog post:

When we see frontier models improving at various benchmarks we should think not just of increased scale and clever ML research ideas but billions of dollars spent paying PhDs, MDs, and other experts to write questions and provide example answers and reasoning targeting these precise capabilities.

You can see this tension most vividly in robotics. In some fundamental sense, robotics is an algorithms problem, not a hardware or data problem — with very little training, humans can learn how to teleoperate current hardware to do useful work. So if we had a human like learner, robotics would (in large part) be solved. But the fact that we don’t have such a learner makes it necessary to go out into a thousand different homes to learn how to pick up dishes or fold laundry.

One counterargument I’ve heard from the takeoff-within-5-years crew is that we have to do this cludgy RL in service of building a superhuman AI researcher, and then the million copies of automated Ilya can go figure out how to solve robust and efficient learning from experience.

This gives the vibes of that old joke, “We’re losing money on every sale, but we’ll make it up in volume.” Somehow this automated researcher is going to figure out the algorithm for AGI - a problem humans have been banging their head against for the better part of a century - while not having the basic learning capabilities that children have? I find this super implausible.

Besides, even if you think the RLVR scaleup will soon help us automate AI research, the labs’ actions suggest otherwise. You don’t need to pre-bake the consultant’s skills at crafting Powerpoint slides in order to automate Ilya. So clearly the labs’ actions hint at a world view where these models will continue to fare poorly at generalizing and on-the-job learning, thus making it necessary to build in the skills that they hope will be economically valuable.

Another counterargument you could make is that even if the model could learn these skills on the job, it is just so much more efficient to build them up just once during training rather that again and again for each user or company. And look, it makes a lot of sense to just bake in fluency with common tools like browsers and terminals. Indeed one of the key advantages that AGIs will have is this greater capacity to share knowledge across copies. But people are underrating how much company and context specific skills are required to do most jobs. And there just isn’t currently a robust efficient way for AIs to pick up those skills.

I was at a dinner with an AI researcher and a biologist. The biologist said she had long timelines. We asked what she thought AI would struggle with. She said her work has recently involved looking at slides and decide if a dot is actually a macrophage or just looks like one. The AI researcher says, “Image classification is a textbook deep learning problem—we could easily train for that.”

I thought this was a very interesting exchange, because it revealed a key crux between me and the people who expect transformative economic impacts in the next few years. Human workers are valuable precisely because we don’t need to build schleppy training loops for every small part of their job. It’s not net-productive to build a custom training pipeline to identify what macrophages look like given the way this particular lab prepares slides, then another for the next lab-specific micro-task, and so on. What you actually need is an AI that can learn from semantic feedback or from self directed experience, and then generalize, the way a human does.

Every day, you have to do a hundred things that require judgment, situational awareness, and skills & context learned on the job. These tasks differ not just across different people, but from one day to the next even for the same person. It is not possible to automate even a single job by just baking in some predefined set of skills, let alone all the jobs.

In fact, I think people are really underestimating how big a deal actual AGI will be because they’re just imagining more of this current regime. They’re not thinking about billions of human-like intelligences on a server which can copy and merge all their learnings. And to be clear, I expect this (aka actual AGI) in the next decade or two. That’s fucking crazy!

Sometimes people will say that the reason that AIs aren’t more widely deployed across firms and already providing lots of value (outside of coding) is that technology takes a long time to diffuse. I think this is cope. People are using this cope to gloss over the fact that these models just lack the capabilities necessary for broad economic value.

Steven Byrnes has an excellent post on this and many other points:

New technologies take a long time to integrate into the economy? Well ask yourself: how do highly-skilled, experienced, and entrepreneurial immigrant humans manage to integrate into the economy immediately? Once you’ve answered that question, note that AGI will be able to do those things too.

If these models were actually like humans on a server, they’d diffuse incredibly quickly. In fact, they’d be so much easier to integrate and onboard than a normal human employee (they could read your entire Slack and Drive in minutes and immediately distill all the skills your other AI employees have). Plus, hiring is very much like a lemons market, where it’s hard to tell who the good people are, and hiring someone bad is quite costly. This is a dynamic you wouldn’t have to worry about when you just wanna spin up another instance of a vetted AGI model.

For these reasons, I expect it’s going to be much much easier to diffuse AI labor into firms than it is to hire a person. And companies hire lots of people all the time. If the capabilities were actually at AGI level, people would be willing to spend trillions of dollars a year buying tokens (knowledge workers cumulatively earn 10s of trillions of dollars of wages a year). The reason that lab revenue are 4 orders of magnitude off right now is that the models are nowhere near as capable as human knowledge workers.

AI bulls will often criticize AI bears for repeatedly moving the goal posts. This is often fair. AI has made a ton of progress in the last decade, and it’s easy to forget that.

But some amount of goal post shifting is justified. If you showed me Gemini 3 in 2020, I would have been certain that it could automate half of knowledge work. We keep solving what we thought were the sufficient bottlenecks to AGI (general understanding, few shot learning, reasoning), and yet we still don’t have AGI (defined as, say, being able to completely automate 95% of knowledge work jobs). What is the rational response?

It’s totally reasonable to look at this and say, “Oh actually there’s more to intelligence and labor than I previously realized. And while we’re really close to (and in many ways have surpassed) what I would have defined as AGI in the past, the fact that model companies are not making trillions is revenue clearly reveals that my previous definition of AGI was too narrow.”

I expect this to keep happening into the future. I expect that by 2030 that the labs will have made significant progress on my hobby horse of continual learning, and the models will start earning 100s of billions in revenue, but they won’t have automated all knowledge work, and I’ll be like, “We’ve made a lot of progress, but we’re not at AGI yet. We also need X, Y, and Z thing to get to trillions in revenue.”

Models keep getting more impressive at the rate the short timelines people predict, but more useful at the rate the long timelines people predict.

With pretraining, we had this extremely clean and general trend in improvement in loss across multiple orders of magnitude of compute (albeit on a power law, which is as weak as exponential growth is strong). People are trying to launder the presitge of pretraining scaling, which was almost as predictable as a physical law of the universe, to justify bullish projections about RLVR, for which we have no well fit publicly known trend. When intrepid researchers do try to piece together the implications from scarce public datapoints, they get quite bearish results. For example, Toby Ord has a great post where he cleverly connects the dots between different o-series benchmark charts, which suggested “we need something like a 1,000,000x scale-up of total RL compute to give a boost similar to a GPT level”.

People have spent a lot of time talking about a software only singularity (where AI models write the code for a smarter successor system), a software + hardware singularity (where AIs also improve their successor’s computing hardware), or variations therein.

All these scenarios neglect what I think will be the main driver of further improvements atop AGI: continual learning. Again, think about how humans become more capable at anything. It’s mostly from experience in the relevant domain.

Over conversation, Beren Millidge made the interesting suggestion that the future might look continual learning agents going out, doing jobs and generating value, and then bringing all their learnings back to the hive mind model, which does some kind of batch distillations on all these agents. The agents themselves could be quite specialized - containing what Karpathy called “the cognitive core” plus knowledge and skills relevant to the job they’re being deployed to do.

“Solving” continual learning won’t be a singular one-and-done achievement. Instead, it will feel like solving in context learning. GPT-3 demonstrated that in context learning could be very powerful (its ICL capabilities were so remarkable that the title of the GPT-3 paper is ‘Language Models are Few-Shot Learners’). But of course, we didn’t “solve” in-context learning when GPT-3 came out - and indeed there’s plenty of progress still to be made, from comprehension to context length. I expect a similar progression with continual learning. Labs will probably release something next year which they call continual learning, and which will in fact count as progress towards continual learning. But human level continual learning may take another 5 to 10 years of further progress.

This is why I don’t expect some kind of runaway gains to the first model that cracks continual learning, thus getting more and more widely deployed and capable. If you had fully solved continual learning drop out of nowhere, then sure, it’s “game set match”, as Satya put it. But that’s not what’s going to happen. Instead, some lab is going to figure out how to get some initial traction on the problem. Playing around with this feature will make it clear how it was implemented, and the other labs will soon replicate this breakthrough and improve it slightly.

Besides, I just have some prior that competition will stay fierce, informed by the observation that all these previous supposed flywheels (user engagement on chat, synthetic data, etc) have done very little to diminish the greater and greater competition between model companies. Every month (or less), the big three will rotate around the podium, with other competitors not that far behind. There is some force (potentially talent poaching, rumor mills, or reverse engineering) which has so far neutralized any runaway advantages a single lab might have had.

This is the final episode of the Sarah Paine lecture series, and it’s probably my favorite one.

Sarah gives a “tour of the arguments” on what ultimately led to the Soviet Union’s collapse, diving into the role of the US, the Sino-Soviet border conflict, the oil bust, ethnic rebellions and even the Roman Catholic Church. As she points out, this is all particularly interesting as we find ourselves potentially at the beginning of another Cold War.

As we wrap up this lecture series, I want to take a moment to thank Sarah for doing this with me. It has been such a pleasure.

If you want more of her scholarship, I highly recommend checking out the books she’s written. You can find them here.

Watch on YouTube; listen on Apple Podcasts or Spotify.

• Labelbox helped me create a tool to transcribe our episodes! I’ve struggled with transcription in the past because I don’t just want verbatim transcripts, I want transcripts reworded to read like essays. Labelbox helped me generate the exact data I needed for this. If you want to learn how Labelbox can help you (or if you want to try out the transcriber tool yourself), go to labelbox.com/dwarkesh

• Sardine doesn’t just assess customer risk for banking & retail. Their AI risk management platform is also extremely good at detecting fraudulent job applications, which I’ve found useful for my own hiring process. If you need help with hiring risk—or any other type of fraud prevention—go to sardine.ai/dwarkesh.

• Gemini’s Nano Banana Pro helped us make many of the visuals in this episode. For example, we used it to turn dense tables into clear charts so that’d it be easier to quickly understand the trends that Sarah discusses. You can try Nano Banana Pro now in the Gemini app. Go to gemini.google.com.

(00:00:00) – Did Reagan single-handedly win the Cold War?

(00:15:53) – Eastern Bloc uprisings & oil crisis

(00:30:37) – Gorbachev’s mistakes

(00:37:33) – German unification and NATO expansion

(00:48:31) – The Gulf War and the Cold War endgame

(00:56:10) – How central planning survived so long

(01:14:46) – Sarah’s life in the USSR in 1988

Sarah Paine

Thank you for coming. It’s a treat to be with you and sharing all this stuff. Since we seem to be in a second Cold War, maybe it’s a good time to revisit the last one to see why it turned out the way it did and why the participants in it thought it turned out the way it did.

I’m going to pose the question: why did Russia lose the Cold War? People have loads of different answers to that question. This is going to be a tour of the counter-arguments. I’m going to start with an answer that many Americans have. It’s a very simple one that’s like, “Ronald Reagan single-handedly defeated the Soviet Union.” That’s one possible answer. But then I’m going to give you all kinds of counter-arguments to that.

Some of them are going to be other external explanations of what others did to the Soviet Union. Others are internal ones of what the Soviet Union did, the cards that it didn’t play particularly well. And then I’ve got some umbrella explanations. So that’s my plan for this evening.

The story that Ronald Reagan did it… Well, here’s a picture at the Reagan Ranch after the Cold War is over. You see the Gorbachevs and you see the Reagans and they seem to be having a grand old time, which suggests there’s something maybe off with that explanation.

But anyway, the way the “Ronald Reagan did it” school goes is that Ronald Reagan did a massive military buildup and some would argue it bankrupted the Soviet Union. He was a man of words and deeds. He made really good speeches that were memorable.

Here’s one before Parliament where he says, “The regimes planted by totalitarianism have had more than 30 years to establish their legitimacy, but none—not one regime—has yet been able to risk free elections. Regimes planted by bayonets do not take root.”

And then here he is before the Brandenburg Gate, this is in Berlin, long a symbol of German greatness. But then it was a locked gate on the Berlin Wall. Here’s Ronald Reagan: “General Secretary Gorbachev, if you seek peace, if you seek prosperity for the Soviet Union and Eastern Europe, if you seek liberalization, come to this gate. Mr. Gorbachev, open this gate. Tear down this wall!“

And who can forget the “Evil Empire” speech, which he gave to the National Association of Evangelicals in Orlando, Florida, and they skipped Disneyland to hear it.

Reagan did a very significant military buildup that actually had started under Carter when the Soviets invaded Afghanistan, big mistake as we discovered. He also invested in and deployed missiles in Europe. He was busy funding anti-communist insurgencies and also others who didn’t like the Soviet Union all over the world. He started doing more aggressive military patrolling. By the time he’s out of office, he was like half a dozen ships short of this 600-ship navy or whatever it is he was planning to make. He was also trying to build a missile shield, his Strategic Defense Initiative.

The problem is the Soviets tried to match him on this. If you add up the GNPs of the United States, NATO allies, and Japan, well, that would be seven times larger than the Soviet GNP. You’ve got to be aware of asymmetric strategy. The CIA thought during the Cold War that perhaps Russia was spending up to 20% of its GNP on defense. After the Cold War ended, when you were getting more accurate statistics, it turns out it was at least 40 or 50%. Some people say it was up to a truly economy-busting 70%, if you take into account all the infrastructure investments that were associated with military things. If you look during the Cold War, the United States was spending less than 8%, Germany less than 6%, Japan less than 2%, and Nazi Germany, which is no piker, 55%. So you look at all this and it was difficult.

So I am going to be quoting lots of Russians today because they have thought deeply about the fate of their country, how life as they knew it disappeared, the Soviet Union gone, the empire gone. They thought a lot about it. Here is a former Soviet ambassador to West Germany, Valentin Falin. Here’s his take: “Following the American strategy of our exhaustion in the arms race, our crisis in public health and all the things that have to do with standard of living reached a new dimension.” Then if you add to the arms race of the United States the arms race that was going on with China on that border, the arms race plunged the Soviet economy into a permanent crisis.

Here you have Georgy Arbatov, who was the late Soviet Union’s finest expert on the United States, or at least the most famous one. He’s looking at the Soviet war in Afghanistan. He said, “It is quite clear that the Afghan war was most advantageous for the United States. And we got our Vietnam.” Because the United States is busy funding the other side, and it’s costly. Gorbachev is looking at this, as he’s telling the Politburo a year after he came into power. He said, “Look, the Americans are betting precisely on the fact that the Soviet Union is scared of this SDI, the Strategic Defense Initiative, a missile defense. That’s why they’re putting pressure on us, to exhaust us.” Correct.

So some would argue that the US victory in the arms race guaranteed victory in the Cold War. Go Ronnie. That’s one explanation. But I’m going to give you a tour of the counter-arguments and some other explanations, starting with Presidents Ford, Carter, and the Helsinki Declaration.

After World War II, the Soviets had wanted to convene a conference of European states to confirm its expanded World War II borders. And for a long time, nobody was interested. The Western Europeans are sick of all the drama. The United States still doesn’t want to show, but we go along with our allies, and our allies insist on including human rights provisions. We think this is crazy because we know the Soviets are never going to enforce those things. But you get the Helsinki Accords that have all sorts of human rights provisions.

Well, lo and behold, unbeknownst to anybody, dissidents across the Eastern Bloc and human rights activists across the West start holding the communists to account for the agreements that they have signed and start contrasting the liberation that communism promises versus the dictatorship actually delivered. This human rights movement within the Soviet bloc and abroad, took on a life of its own.

Here you have the former director of the CIA and former head of the Department of Defense, Robert Gates, saying, “The Soviets desperately wanted this big conference and it laid the foundations for the end of their empire. We resisted it for years only to discover years later that this conference had yielded benefits beyond our wildest imagination.” Go figure.

Here is Jimmy Carter with his human rights initiative. It was Gorbachev’s English language translator who said that Carter’s emphasis on precisely the human rights that were denied to Soviets really resonated and it made people think that they wanted a more democratic, open, liberal society. Here’s Carter giving a graduation address at Notre Dame. He said, “We have reaffirmed America’s commitment to human rights as a fundamental tenet of our foreign policy. What draws us Americans together is a belief in human freedom. We want the world to know that our nation stands for more than just financial prosperity. We’re bigger than that.”

And here is Eduard Shevardnadze, Gorbachev’s foreign minister, echoing some of these sentiments. He said, “Look, the belief that we are a great country is deeply ingrained in me, but great in what? Territory? Population, quantity of arms, people’s troubles, the individual’s lack of rights? And what do we, who have virtually the highest infant mortality rate in the world, take pride in? It’s not easy answering the questions. Who are you? Who do you wish to be? A country which is feared or a country which is respected? A country of power or a country of kindness.”

Others agreed that communism was essential to the survival of the Soviet Union, but it’s an undemocratic ideology. Fundamentally, it’s a foundation that can’t endure forever. That’s the take of Vitaly Ignatenko, who’s a Russian journalist. Oleg Grinevsky, who’s a Soviet career diplomat, is saying, “Look, communist ideology is associated above all with the Soviet Union. Its rejection created a vacuum and it determined its ultimate fate.” Boris Yeltsin, who is Gorbachev’s successor, said, “Look, no one wants a new Soviet Union.”

So some would argue, this counter-argument, that human rights clauses of the Helsinki Accords and Carter’s subsequent human rights campaign destroyed communist belief in communism. Okay. Another president, another counter-argument. Those who are fans of Richard Nixon would say, “No, no, no, no, no. It was Richard Nixon who played the China card so the United States and China could gang up on the Soviet Union and overextend it financially to wreck it militarily.”

I think the Chinese would beg to differ and say, “No, no, no, no. It was Mao who played the America card.” Because what’s going on in 1969? There’s a border war between China and the Soviet Union. China’s gotten its nuclear bomb in ‘64. It no longer has to defer to the Soviet Union and starts playing more tough on their border disagreements. So the Soviets are really upset. They come to the United States and ask us whether it would be okay to nuke these people, because they think Americans don’t like the Chinese . Well we didn’t, but we said, “No, it’s not okay to nuke those people.”

So the Chinese figure it out. The one that wants to nuke you is your primary adversary, right? Up until then… Think about it, China and Russia, for them the United States was the primary adversary. Now they’re primary adversaries with each other, freeing up the United States to decide which one it’s going to cozy up to. And the United States decides it’s going to cozy up to China.

Why? Well, Chinese belligerency forces the Soviets. Not only have they already got a big militarized border with Europe, now they’re going to do the same thing on a very long border with China. These are nuclear-armed mechanized forces, very expensive. Imagine if this country had to have such borders with Canada and Mexico. It would be bankrupting, and we are far richer than the Soviet Union was then, whenever. It was bankrupting. So some would argue that US cooperation with China fatally overextended the Soviet Union.

One could take all of these arguments, starting with President Nixon all the way through Reagan, to make an overarching argument that says, “Look, each president opened up opportunities for the others who then leveraged them.” So Nixon plays the China card, which others play with increasing dexterity. Ford comes in and begins dabbling in human rights. Carter then comes in and really goes for human rights and starts doing a military buildup, which then Ronald Reagan really does. So that by the time you get to Reagan, he is dealing in a position of both ideological and military strength vis-à-vis the Soviet Union.

For those who think that US foreign policy was not consistent during the Cold War, you’re not looking at it at the strategic level. There were certain different strategies going on and how best to achieve it, but both parties agreed the goals were free trade, democracy, containment of communism. Those were staples of US foreign policy, for both parties, for its duration.

So some would argue that Presidents Nixon through Reagan produced the cumulative presidential effects to defeat the Soviet Union.

Okay, others would say to forget this great man theory of history business, that’s really passé. What really accounted for the outcome of the Cold War was this military platform, that’s Pentagonese for large military systems. But anyway, it’s a nuclear-powered, nuclear-armed submarine. They say that this is the item.

The way deterrence theory worked during the Cold War, and I believe now as well, is that in order to deter the other side, you have to have a reliable second-strike capability. So if they thought of lobbing a nuke at you, they would be guaranteed that you would have the second strike to lob a nuke back. Therefore, they’re never going to lob the first nuke.

When Jimmy Carter became president, he was a graduate of Annapolis and also a submariner. The United States began a much more aggressive deployment of its fleet and that’s continued even more so under Reagan. We’re taking our submarines and we’re targeting Soviet submarines in their home water bastions. So the Soviets are thinking that we’re going to be able to destroy their second-strike capability on our first strike and they’re having a heart attack.

So here you have Valery Boldin, a longtime aide to Gorbachev, saying, “Look, the most powerful strength of the United States is the naval fleet and we aren’t going to get one, or our geography isn’t set up to use one the way the United States can.” And then you have Marshal Yazov saying, “For the Americans, the main means of atomic attack is the fleet.”

So then you get Marshal Akhromeyev, who’s visiting the United States in 1987. At the end of the Cold War he will kill himself, but he’s still around in ‘87. He’s telling his American hosts, “You know where our submarines are, but we don’t know where yours are. It’s destabilizing. You, you the United States Navy, are the problem.” Go Navy. And here’s his host, Admiral Trost, who’s going, “Yeah, the inability of the Soviet Union to maintain a strong defensive capability led to the demise of the Soviet Union and to the removal of the Soviets as a major threat to us.”

So you can make a perfectly good argument to say the Soviet Union could not counter technologically or financially the US submarine threat to its retaliatory nuclear forces, so war termination was the only thing it could do.

All of these preceding explanations are navel explanations, spelled with an ‘e’, as in staring at one’s own. They’re all about what the United States did or didn’t do. So let’s get beyond the half-court tennis of Team America. You need to look at the other side of the net. This is where the Western guru for things military, Carl von Clausewitz, emphasizes reciprocity in war and the interaction of both sides. You’re not going to do well unless you consider what the other side is doing.

So I have given you some external explanations and I’m going to do the internal ones. Here is Arnold Toynbee, he’s one of the finest historians of the 20th century. He wrote a big multi-volume history of the West, in which he argues that civilizations die from suicide, not by murder. So I discussed the murder, what the United States tried to do to the Soviet Union. Now I’m going to talk about the suicide, what the Soviets did to themselves. And here is counter-argument number one. The Soviet Union was an empire, and when that collapsed, that meant they lost the Cold War.

During the Cold War, the Korean War and the Vietnam War, there was much fear in the West of this domino theory. The idea is one country falls to communism, then the next and next and next and next would fall to communism. Turns out the domino theory did not apply to capitalism. It applied to communism because once the democratic contagion hit one Warsaw Pact country in Eastern Europe, it spread to the others until it was a seething mess and they fell like dominoes.

So in 1988-89, there were all kinds of demonstrations in the Eastern Bloc, the Soviet Union. In the Soviet Union, they’re for political freedoms. In the Eastern Bloc, they’re for freedom from the Soviet Union. Gorbachev may not have gotten that detail. They’re all about not only wanting political freedoms, but also they’re about crumbling economies and how to fix their miserable standards of living. Very uncharacteristically, the Russians didn’t send tanks. In fact, Gorbachev welcomed and encouraged reforms in the Eastern Bloc, both political and economic, just as he was doing in the Soviet Union. So his ideas of glasnost, openness, and perestroika, rebuilding, resonated at home and abroad.

These reforms began in Poland. Poland had been a scene of much worker unrest many times, in 1956, 1970, 1976, and 1980 and 1981. In 1981, this is when Solidarity, the workers movement, gets going and it gets a national and an international reputation. The next set of strikes are happening in 1988, because in the preceding several years, the Polish standard of living had shrunk by over 3%.

The government was out of cash and wanted to raise basic food prices, and Poles hit the streets. The government was in a panic, because it was worried the economy would go into free fall. So the government cut a deal with Solidarity. They said, “You call off the strikes and then we’ll let you into political talks,” and Solidarity agreed. There was a complicating factor on all of this. It’s called the Roman Catholic Church, which is an institution of enormous credibility and legitimacy in Poland, which had a partiality for Solidarity and it had a Polish pope.

So the roundtable discussions were these political talks. They occurred a year later in February 1989, and the Soviets encouraged them. In fact, here’s one Soviet person there advising the Poles: “Look, you’ve got to find some quick solutions out of your economic and political mess. You’re an itty-bitty country, so when you make mistakes, they’ll be itty-bitty mistakes. But if we make them, they’ll be big.” They got that one right.

The Polish Communist Party thought they had this one covered by the way they jiggered the election rules. Not quite. The day they held elections is exactly the same day that Deng Xiaoping turned the tanks on demonstrators in Beijing and you have the Tiananmen Massacre. Two solutions for the problem. So the way the elections worked out in Poland is that Solidarity won every single seat for which it could compete but one. And then only three people in the communist-designated seats actually won. So who won all the rest of them? The box on the ballot called “none of the above.” Yes, the Roman Catholic Church had helped instruct people that that’s the box you want. With that, the legitimacy of the Communist Party to rule had just been wrecked and we’re on to democracy in Poland.

This democratic contagion then spread into East Germany four months later. This is about the 40th anniversary of the founding of East Germany. 70,000 people demonstrated in Leipzig. Within the week around like 1.4 million Germans are demonstrating in over 200 demonstrations. Typically, the East Germans would have sent tanks. That was what they would have done in the past. But would-be tank man Erich Honecker was already out of a job. His ruinous policies of living off debt since he came to power in 1971 had just about wrecked East Germany. So he was out.

Then less than two weeks later, the Council of Ministers resigns. Then on November 8th, the Politburo resigns. Then on the 9th, whatever is left of that government is issuing new travel regulations. You might wonder what travel has got to do with it. I’ll get there.

So in response to a question at a news conference, this guy, Günter Schabowski, who was one of the remaining communists helping run the show, gets asked a question and he doesn’t know the answer. So he wings it. The question is, “When do these travel regulations go into effect?” And he goes, “Immediately.” Well, crowds immediately started gathering at the six gates to the Berlin Wall. At one of them, the border guards decided that discretion was the better part of valor, and they opened the gate and East Germans poured into West Berlin.

Within the first week alone, over half of East Germany’s population visited the West. Within the month, 1% of the population emigrated to the West. Like the Polish elections, this opening of the gate was a pivotal decision. A pivotal decision, whatever it is, means there’s no going back to the way it was. Here’s good old Günter going, “Gosh, we hadn’t a clue that opening the wall was the beginning of the end of East Germany.” Okay, better luck next time. And the Russians were shocked by how unpopular they were. They were thinking they were going to get credit, Gorbachev, for Eastern Europe’s liberation rather than blame for Eastern Europe’s enserfment.

Here you have Yuri Ryzhov, a scientist and parliamentarian going, “All of our former satellites by compulsion cast off from us as fast and as far as possible.” And Anatoly Kovalev, who is a deputy foreign minister, said, “Look, we had no confidence whatsoever concerning whom the East German army is going to shoot, the demonstrators or us. And the same thing for the Polish and Hungarian armies.” Great. With allies like this, who needs enemies? The allies kind of cover it. So this argument says unrest in the empire forced the Soviet Union to forfeit the Cold War.

Okay, I got another counterargument. It says, “Nonsense, the real problem was that the satellites were unhealthy. That’s why the whole thing fell apart.

So this map is 1960. You see all those tempting green places. They’re about to become independent, and they are really sick of their Western European colonizers. Enter the Soviet Union with a program to put the West out of business. There were many takers.

Fast forward to late 1980s. The Soviet Union is on a roll. Small hitch, in the late 1970s there was a big recession that continued into the 80s and tanked commodity prices. For some of the newfound pals like Angola, South Yemen, Ethiopia, Nicaragua, it wrecked their export earnings because they’re exporting commodities and these commodity prices are down. In many cases, it halved them. The Soviet Union was really dependent on oil exports, still is. Oil prices tanked and oil accounted for up to 55% of the Soviet budget. So here Brezhnev has got a deep bench of non-performing piles at a time when he doesn’t have the money to support all of them.

Worse yet from the Soviet point of view, it’s dumped all this money in these Third World friends but meanwhile, it’s got its own nationalities who are deeply unhappy and they want out of the empire. Most problematically, they all revolt at exactly the same time. One of the rules for continental empire is “no two-front wars”. Russia has so many fronts at this point, it can’t even keep count.

The unrest in the internal empire of nationalities started as soon as Gorbachev got in. There were student movements in Kazakhstan and Yakutia, opposite ends of things. By the time you get to 1990, there are like 76 seething ethnic rebellions in different parts of this. There was too much going on for the Soviet government to handle. So you could argue that the Soviet Union bankrupted itself with the Third World while ignoring its own internal Third World of nationalities, whose simultaneous revolts brought down the Soviet Union.

I got a completely different argument for you. If you don’t like all of those, I got another one for you. It’s the economy, stupid, right? That line. One could argue that communism failed as an economic system. If you look at growth statistics for the Soviet Union, they’re pretty good post-World War II when they’re rebuilding, but they really stagnate from the mid-70s onward. For the decade preceding Gorbachev’s coming to power, Soviet growth stats were one to two percent lower than those of the United States, and the compounding effects of that were enormous.

What’s going on? Everyone’s lying to each other. The data that Soviets are using is garbage. If you’re working for a subunit of an enterprise, you have to lie about the inventories you have, saying you have less than you do, and then you have to lie about what you need, saying you need more than you do because you’re worried about getting enough things. It’s not a market system where the price dictates it. This is all about the plan. You’ve got to enter the right numbers and then you get whatever inputs you get from the centralized plan.

So everyone’s lying. They’re aggregating all the lies. The higher up the food chain you aggregate these things, the worse the data is, so that the Soviet government has no idea what the actual value of capital or labor are. It has no idea what actual productivity is, and no one has any idea what consumer preferences are. You’re not using markets and prices. The misallocation of capital and labor goes unnoticed until it metastasizes into a catastrophe.

To give you a sense of these misallocations, the Soviet Union was rotting 20 to 40% of its crops. It’s using scarce hard currency for agricultural imports to make up for those crops, a total mess. You can look at what happens to the economy with oil prices down. We’re into a spiraling mess, so that from when Gorbachev comes in in ‘85 to when it hits a trough in Russia in 1998, you see this crashing share of world GDP by the Eastern bloc. If you look at Soviet statistics on deficits, trade balances, debt, they’re just soaring, and then GNP growth goes double-digit negative. That’s called shrinkage. It’s not the normal thing.

Marshall Yazov, here’s his take: “We simply lack the power of all these wealthy NATO nations. We had to find an alternative to the arms race.” And here’s a foreign service officer, Anatoly Adamishin. He said, “Look, our problems began with the departure from isolation. The main reasons for collapse were internal, not external reasons. The Soviet economy was literally exhausted from this monstrous arms race, militarism, enemies with half the world.” That’s his take. Gorbachev told the Central Committee, “Look, we’re encircled not by invincible armies, but by superior economies.” He often told people, “Living this way any longer is impossible.” So you can make a powerful argument that it’s the Soviet economy that lost the Cold War.

This gentleman, Alexis de Tocqueville, is very famous for writing a book about the last days of the French monarchy before the French Revolution overturned it. He also wrote something about Democracy in America, both excellent books. But this one’s from the one about France, where Tocqueville observes, “The most dangerous moment for a bad government is when it begins to reform.”

Russians of all political persuasions agree on at least one thing. That is that Gorbachev’s role in how the Cold War turned out was pivotal, that he played a very essential part. Gorbachev made his decision based on certain false assumptions. One of them was the irreversible direction of history. Gorbachev thought of history going always forward towards communism, never backwards to capitalism. Of course, Eastern Europe took a U-turn, went straight back to capitalism. Here is Leonid Shebarshin, who is a senior person in the KGB, their intelligence office. He said, “The thought never occurred to the government that it’s possible to withdraw from socialism.”

If you think about both communist theory and how imperialism works in practice, usually the mother country is more developed than whatever all the colonies are, right? Well, the Soviet Union was an inverted empire. People in Eastern Europe as a group were more well-educated and they were richer than Russians. It was like a donut empire. So when the empire went to Eastern Europe, Russians could no longer siphon off the wealth of these enserfed populations in Eastern Europe, which explains why they wanted to leave. It also suggests why Putin wants them back.

Another false assumption has to do with the sentiments of the neighbors. Gorbachev was convinced he was going to get credit for liberating Eastern Europe, rather than blame as a Russian for having enserfed them in the first place. For Gorbachev, the clock began on his watch. For other people, no, Stalin’s when it began, when he started shooting a lot of people.

Here you have Anatoly Chernyaev, foreign policy adviser to Gorbachev, saying that Gorbachev thought that bringing freedom to our Eastern European satellites would have them adopt socialism with a human face. “He made an enormous mistake because these countries brutally turned their back on us.” Really, if that’s brutal, then what pray tell was Stalin? And then it gets better: “The politics in connection with our former friends were totally unexpected to us.” Really? You occupy people, you never leave, you shoot a lot of people in their government, you put in a new government, you siphon off a lot of their wealth, and you impose a non-performing economic system, and you wonder why they don’t like you.

Think about the United States. It intervenes all around the world in other people’s troubles. It dumps billions in economic aid and even leaves and people don’t like us. I don’t know why the Russians think they’re so special.

Another false assumption: Gorbachev believed that if the Warsaw Pact, the military alliance of the Eastern Bloc, disappeared, then NATO would disappear. He also believed that if the Comecon, which is their trading organization, went away, then the European Community in those days—it becomes the European Union later—would disappear. Not quite. It turns out that organizations that are coercive versus those that are voluntary, they dissolve for different reasons.

And then Gorbachev also assumed that the United States would share a continental outlook of not wanting strong powers and that the United States therefore would not want a unified Germany, let alone a strong unified Germany. So when all the unrest is happening in Germany, Gorbachev is off taking a vacation. Poor life choice, because at that moment, President George Bush Sr. and Chancellor Kohl of Germany are working on fast-tracking German unification of a fully sovereign, unified Germany—both halves in NATO.

So many of Gorbachev’s closest supporters at the end of it all blamed him. They said, “Look, his foreign policy mistakes were a function of his domestic policy mistakes and it destroyed the Soviet Union.” Back to this America expert, Vladimir Lukin: “Gorbachev was no Deng Xiaoping.” And Arbatov, who’s their premier America expert: “The stupidity of our leaders caused the disintegration of the Soviet Union.” So the big bozo was playing with plastic bags, stuck one on his head, committed suicide. It was by mistake. Lukin continued: “In the West, they love Gorbachev because everything took place so easily and cheaply, basically like that, but only for you. For us, it was expensive.” But you could argue the time to reassess all the Stalinist stuff was long overdue.

Here’s a completely different way of looking at it. I’ve been giving you sins of commission, and now I’m going to do sins of omission. It’s a good framework. It’s useful for other things. The sins of commission are all the things Gorbachev did. Now what I’m going to do is what the army didn’t do. Some would argue that the Red Army should have done exactly what Deng Xiaoping ordered his army to do. You just send the tanks against civilian demonstrators and they truly crush them and it’ll be over. Communist Party is still in power in China 30 years later. So there are some people who believe that this was a terrible mistake.

So this argument would be that timely tank deployments—TTD, my contribution to military acronyms—would have changed the outcome of the Cold War. Others would be back to the great men of history and sins of commission, and they wouldn’t be picking on Gorbachev but his successor Boris Yeltsin. There are two big pieces of evidence when we look. He removed Article 6 from the Soviet Constitution, which guaranteed that the Communist Party would always monopolize power. And then in addition in the following year, Yeltsin’s the head of Russia, he gets together with the heads of Ukraine and Belarus, and they signed the Belavezha Accords, which then formally dissolved the Soviet Union. So according to this way of thinking, it’s his fault. It’s suicide on purpose. And what it does is it opens the door for multiple parties and for the nationalities within the Soviet Empire to become independent.

So I’ve given you internal explanations. I’ve given you external explanations. Now I’m going to give you some umbrella explanations. They’re based on all the preceding evidence, and they come to opposite conclusions. The first one was, well, any of the above, it’s inevitable. The opposite conclusion from the same evidence is that no, it took all of the above. The West barely won.

I’m going to start with “any of the above”. You could argue with this many serious problems, it was a matter of time before the Soviet Union collapsed. It was an objectionable system for precisely the reasons the West didn’t like it. It had a brutally inefficient economic system. Russians who invented the thing, at the end of the day, didn’t want it either. By this way of looking at it, you have people like Yuri Ryzhov, a genuine rocket scientist, who says, “Look, the main reason for the collapse of the Soviet Union is the rottenness of its system.” Then here’s a journalist, Teimuraz Stepanov, who said, “Look, I think from the beginning the genes of disintegration were contained in the genetics of this governmental political formation.” Don’t you love the products of the Soviet educational system? Don’t ever use wording like that.

So you could argue that the Soviet Union was destined to fail with this many problems. Others would come to the opposite conclusion. They would say, “No, it took every single one of them for the Cold War to end on Western terms.” Back to Anatoly Kovalev, the deputy foreign minister, he said, “Look, all these factors merge—internal, ideological, economic, military—it’s all of them. You remove any one of them and you get a different outcome. Maybe the Cold War ends, but it might end completely differently.” So by this line of reasoning, the West barely won and should feel very fortunate that it did.

One can take this last argument and say it was more than that. It also took the confluence in office of two very talented leaders: Helmut Kohl of Germany and George Bush Sr. of the United States, not the son who got into those forever wars, but the dad who didn’t. George Bush Sr. had one of the most amazing resumes of any person ever to become president of the United States. Just look at him. When he’s really young, he’s a war hero in World War II. He’s a Navy pilot, a dangerous thing to do. He did it. Then he comes back and he gets his BA at Yale and graduates with honors. Then he becomes a representative for this district in Texas after he’s already made himself a millionaire in the oil business that he started. Then he became ambassador to the UN, followed by US representative to the PRC, before we had formal diplomatic relations. So he’s the guy who’s setting that up. He becomes director of the CIA, and then he is Ronald Reagan’s understudy for eight years as vice president. He is incredibly fit for the job.

Helmut Kohl is equally fit for the job. He is the longest-serving chancellor in German history since his illustrious predecessor, Otto von Bismarck. He starts out getting a PhD in history and political science. He also starts out in business, but then he works for state government, initially as a representative, then as a governor. He becomes chairman of his political party, the Christian Democratic Union, for a quarter of a century.

Once he gets in, he decides he’s going to buy up East Germany one tourist at a time. How does that work? East Germans, it turns out, really like to travel. West Germans had always been able to travel to East Germany, or they long had been able to travel to East Germany, but East Germans definitely could not easily travel to West Germany. Why? Because they have a habit of staying. But all of a sudden, East Germany eases up on the travel regulations. You might ask why, and the answer would be money. Just like the Poles, the East Germans were deep in an economic mess of their own making.

Would-be tank man Erich Honecker, who got the boot at the very end, well, his staying-in-power paradigm that he implements in 1971 is that he’s going to live off debt. He needs to make certain social benefits available and consumer benefits available for labor stability, to not have labor unrest. The way he’s going to do that is he’s not going to do many domestic investments and he’s going to do a lot of borrowing, particularly from West Germany. Well, that’s unsustainable long-term. By the time you get to the end of the Cold War, if he’s going to fix that and even out the accounts, it would be a 30% decline in the East German standard of living. So he really needs the pocket change from the tourists.

So what Kohl does is a brisk business of tourists and things. What he does in return for the easing of travel restrictions, he pays East Germany several hundred million Deutschmarks extra to allow that to happen. And then he gets the Hungarians to go along. He gets the Hungarians to open up their Austrian border to let East Germans out that way, and he gives them a half a billion Deutschmarks for that little favor.

When Kohl introduces his 10-point unification program—because now he’s thinking he’s going to get both Germanys together—this is when he starts doling out big bucks to the Soviet Union, whose economy is unraveling. Gorbachev is going to be desperate for this cash as that’s happening. So West Germany provides 100 million in food, especially in meat, for the Soviet Union that doesn’t have these things.

Nevertheless, the unrest just keeps on going. The Berlin Wall, as I’ve told you, is breached, and then you wind up with a West German caretaker government, and the financial situation in Russia itself is unraveling. By the time you get to January 1990, Bush and Kohl get together and they decide they want to really fast-track German reunification. Why? Because they’ve got to get it done before this unraveling crisis causes Gorbachev to fall from power. So they have got a game going, the two of them. It’s complicated. Here’s why.

Gorbachev was dead-set against Germany, a united Germany, in NATO. He’s not really keen about a united Germany, let alone one in NATO. The US State Department experts, the guys who know everything, are saying, “No, you want to go slow on this unification business.” Kohl is also running a coalition government. There are people in that government he cannot fire because they’re from different political parties. One of them is his foreign minister, this guy Genscher, who is very skeptical about Germany being part of NATO. Then it turns out, although Britain had talked a good piece during the Cold War, it didn’t actually want a unified Germany, nor did France. Why? Because that unified Germany would eclipse them economically. They didn’t want that to happen.

So Kohl and Bush divide up the tasks. Kohl is going to reassure the Soviet Union that Germany is not going to be belligerent or do horrible things. And Kohl is going to work on financial unification because the Soviets are thinking in terms of military unification. You know, where you deploy your troops. That determines things. Wrong instrument of national power, precisely because the Soviets didn’t understand finance. That’s why they’re in such a mess. Whereas the Germans do. What they’re going to do is get East Germany on the West German Deutschmark, and at that point they will control all the money and they will control decisions. But the Russians aren’t going to see that coming.

Meanwhile, Bush is supposed to work the alliances particularly with Britain and France in the West. There are all sorts of meetings that are coming up. Bush’s job is to delay those meetings for as long as possible so German unification can proceed as far as possible. The two of them are doing a tag-team diplomacy with Gorbachev that he just can’t keep up with, given that his own home economy has got these double-digit shrinkage rates.

Here’s how they go. As the trades get bigger, the amount of money you pay Gorbachev gets bigger. First of all, it’s just to get a unified Germany. Then it’s to get a unified Germany with West Germany still in NATO. Then it’s to get a unified Germany with all of Germany in NATO. So here’s how the money goes. Gorbachev agrees to German unification. We are no longer paying hundreds of millions of Deutschmarks. We’re paying billions of Deutschmarks, five billion Deutschmarks for that one. Then Gorbachev agrees that states can choose their own alliances, i.e. whether or not to join NATO. The US offers nine assurances, but it’s also a trade agreement that Gorbachev really wants. Then the economic union goes into effect.

So we’ve now done the financial reunification of Germany. This is when there’s a London Declaration that’s inviting Eastern European countries to coordinate more closely with NATO. In return, Gorbachev gets a promise of a G7 summit meeting that’s going to fast-track aid to him, which it will do. And then Gorbachev agrees to German NATO membership.

At this point, even bigger things are happening. Germany’s going to agree to its border with Poland. I’ll get there and explain. Germany provides 15 billion in Deutschmarks, including building all kinds of new apartment buildings for repatriated Soviet soldiers who are going home. Why are you doing that? Because you want those soldiers focused on buying furniture, not running a military coup. That’s what they’re doing.

So the unification happens in mid-September 1990. Here’s the Polish borders. At the end of World War II, Stalin moved Poland 200 kilometers to the west, and it winds up taking a third of German territory by the time that’s all over. So the Germans don’t really want to sign all that away. In addition, as part of that, there were 12 million German refugees who were thrown out of wherever they were living to send them back to Germany, of whom 2 million died. So this is a big deal and it’s in living memory. Germany agrees to this, that the borders are done. German-Polish borders are set.

Complicating factor: a month and a half before this unification treaty is signed, Saddam Hussein decides he’s going to invade Kuwait because he’s broke. He’s had a long war with Iran, huge debts, many owed to Kuwait, which he doesn’t want to pay back. So if you invade them, that solves that problem. Also, he would take over Kuwait’s very rich oil fields, and together that would make Iraq probably the swing producer of oil. So he thinks that’s a great idea.

Except the Cold War’s over actually. The Russians are more than willing to cooperate with the United States. Gorbachev really needs more money, and he is willing to go along with getting Iraq out of Kuwait, but not with regime change in Iraq. Because think about it, Iraq is a very important debtor state to the Soviet Union. It owed them between $10–13 billion. That’s a lot of money for a broke creditor.

But Gorbachev is being extraordinarily cooperative with Bush Sr. He sends Yevgeny Primakov on multiple missions to Baghdad. The first one, Primakov gets all Russian hostages out of Iraq. Then on the second trip, he gets all Westerners out, Americans included. Third trip, not so lucky. He’s there for the coalition force bombing. I don’t think he liked that very much. But imagine that bombing going on if there were Western human shields going down with every target. Russia took that card right off the table.

Here’s some of the reasoning. Sergei Tarasenko was an aide to Foreign Minister Shevardnadze, and they understood that the United States was going to do something about this invasion of Kuwait. So the Russians thought, “It’ll be better if we force all of this to go through the UN, where Russia has a veto power.” He said, “Look, there was a division of roles.” It extends to China, the help that Russia provided. “When the Americans asked us to work with the Chinese, we told the Chinese, ‘Think about it. You’re one of the big five with veto power. Doesn’t it suit your interest to funnel everything through the UN where you can put your foot down?’ And the Chinese came around to that idea.”

However, the Russians had red lines. Here’s Anatoly Kovalev again, the deputy foreign minister. The red line is, American troops stay out of Iraq. No regime change in Iraq. You do that and you will tank the termination of the Cold War. And that would be the goal. Here’s Kovalev saying, “I advanced the basic principle that we must support the territorial integrity of Iraq. This was our sacred position. We must not permit a division of Iraq.”

So if you wonder why the ground war ended after 100 hours, this is it. The big thing out there is war termination of the Cold War. That’s the big thing. Saddam Hussein is a minor event over there. Sorry, but he was. If it had tanked Cold War termination or upset the reunification of Germany, France and Britain might have been very happy, because François Mitterrand, who is the president of France, and Margaret Thatcher, prime minister of Britain, were against German unification. They knew it would marginalize their own country. Germany’s going to be a bigger economy, which it is.

François Mitterrand eventually found solace in expanding the European Community to the European Union when you’re incorporating all these Eastern Bloc countries into it. He plays a really important role in concluding the Maastricht Treaty that forms the European Union. But Margaret Thatcher just plain lost. She was just upset about the whole thing. She said, “Germany will be the Japan of Europe and worse than Japan.” I guess she hadn’t been to Japan lately. She said, “The Germans will get in peace what Hitler couldn’t get in war.” She wanted to leave Red Army troops in Germany for the duration. Imagine if that had been the case and now dealing with Putin… If he had troops in Germany, we would be in trouble.

But Bush and Kohl worked around all of them. Bush said to Kohl at the end of it, “Look, I’m not going to beat my chest and dance on the Berlin Wall.” Both of them were very careful never to humiliate Gorbachev about the Soviet loss of the Cold War. Why? Because they knew that if they did that, he might fall from power sooner rather than later. Also, they were afraid that if they did that, the hardliners would come to power much more rapidly than they actually did. It was 20 years before Putin started consolidating his power.

The newly independent countries of Eastern Europe needed those 20 years to integrate militarily, politically, economically with the West so that the cement could set before you got the Russians trying to destabilize them. So they bought them 20 years to do this. But there’s a cost to all this. Bush never got credit for his essential role in ending the Cold War on Western terms. So he was not reelected for a second term.

Anyway, when it came time for Nobel Prizes and why the Cold War ended, Anatoly Adamishin, this Soviet Foreign Service officer, said, “Look, it’s difficult to deny the Soviet Union was the one that ended the Cold War.” And Edwin Meese, who was a counselor to Reagan and also his attorney general, said, “Look, the Cold War began because of Soviet policies and it ended in a sense because of Soviet policies.” The Nobel Prize Committee agreed. They awarded the prize to Gorbachev, not to Bush, for his role in liberating Eastern Europe.

So when you’re thinking about this question of why Russia lost the Cold War, I hope you will come up with a more complicated answer than, “Well, Ronnie did it.” There are probably other causes at work as well. Anyway, thank you for your attention. That’s what I have for you this evening.

Dwarkesh Patel

Sarah, thank you so much for doing these.

Sarah Paine

Thank you for having me. That would be the more important thing.

Dwarkesh Patel

There’s an interesting question of why the Soviet Union collapsed when it did. I think the even more interesting question is why a system that was so centrally planned, monstrously inefficient, brutal, a colonial land empire, how such a country could survive for so long into the 20th century. I feel like that’s the thing that actually needs explanation. How did this regime last for 74 years?

Sarah Paine

There are loads of dysfunctional places all over the planet that have been dysfunctional forever. You look at well, why are they dysfunctional? To me, the answer to that one in a way is the example of North Korea. Of all countries that should fall, a place that has ongoing famines in the 21st century, and it used to be the richest part of the Korean Peninsula.

These authoritarian regimes are really good at maintaining coercive powers. Think about it. In order to educate someone, it takes years as a parent to bring up a little person and then you get them educated and maybe they’re an A-list politician. It takes seconds to assassinate them. It’s the asymmetry between construction and destruction. Destruction is so easy. Dictatorships are all over the world. It’s a sad part of the human condition. They clearly know what they’re up to.

In the case of the Soviet Union, there were multiple intelligence organizations. That’s what Stalin was using to keep track of everyone. So you want to monopolize information so that you know more information than other people. And then they have a whole bunch of people who are the winners of the nomenklatura, the elites there. You make sure you pay all of them off. I mean think about it. Human societies, slaves, serfs… We humans have been doing these things to each other for a long time.

Dwarkesh Patel

So dictatorships can certainly sustain themselves for a long time. But the Soviet Union was special in that by the 60s and 70s, they had a GNP that’s 60% of America’s, this incredibly dynamic economy. In the 40s and 50s, they had much higher growth rates, so much so that prominent economists like Paul Samuelson are saying that by the 90s, based on what they’re seeing at the time, the Soviet Union will have a bigger economy than America.

This is just quite surprising that they would have such high growth rates. If you just think about how central planning works, people are going to tell you how much steel you can make and which company gets to use the cotton fabric and cement, etc. You have hundreds of millions of people living under this system. It’s actually quite shocking that they actually had notable growth rates after World War II for decades on end.

Sarah Paine

Well, first of all, it’s a war economy, essentially. You’re putting all your money into having a big military. Russians define greatness—this is part of it—as being a big power, and its a military power with territory. Most countries in wartime mobilize for the military. This country did it in World War II. All kinds of rationing, we’re not using market prices. You’re setting different prices, giving people ration cards and things. The thing about the Soviets is they kept it forever. They never got rid of it. So that’s one piece.

Another problem with the Soviet Union is all of the data. So I don’t know what data you’ve seen, and I know the data I’ve seen. It’s hard to know because the Ruble is a non-convertible currency and a lot of things they measured in weight and other things. Like they’re the greatest TV producer in the world, they said. Why? Because they made the heaviest TVs in the world. I’m serious, when I was there this was it. They would spontaneously combust, which is not the normal thing a TV should do for you, burn down the apartment building.

So they’re going to measure their heavy TVs as a positive, and the Ruble is non-convertible. So there was a guy named Murray Feshbach, and I can’t remember which part of the US government he was in, but he was really good at looking at their statistics and then adjusting them. But people didn’t know. I gave you the CIA ones. The CIA, they’re not stupid people. They’ve got the best data they could find and they’re coming up with 20% of the Soviet budget is probably devoted to the military. After the Cold War is over, they’re going, “Whoops, we missed.” It’s at least double that and maybe triple. So it’s really hard to know even with the statistics you’re getting. Certainly what Paul Samuelson had wouldn’t be accurate. It’s just a guess.

Dwarkesh Patel

My favorite example of this is that there were top-down commands that you had to produce a certain amount of steel. A steel factory would then be incentivized to make thicker bars of steel rather than thinner bars because that would count as greater production, except a lot of inputs actually do require the thinner sheets. So then the other factories have to thin down the steel, but that also counts towards GDP. So producing the inefficient steel and then cutting it down to size is both being double-counted towards GDP.

Sarah Paine

Oh, and just the whole waste of it. Like the heavy TVs, they probably have four times the inputs that they need to make them that would be good for other things. It’s this notion that you can actually plan an economy. Prices are a miracle. Good old Adam Smith, the invisible hand. Prices are the way to go and markets, it’s more efficient.

Dwarkesh Patel

I wonder if one thing that’s going on is that in the early and mid-20th century, you have economies which are much simpler, at least compared to today. So even then, obviously, command and control is less workable than capitalism. But if you just have heavy industry, you need a certain amount of cement, steel, concrete, fabrics, coal. That’s much more workable than, “We’ve got to centrally command what SaaS tools your enterprise is allowed to use.”

Sarah Paine

Oh, yeah. It’s interesting on the development thing. The communists have insisted on heavy industry. That’s the thing that they want. Forget about the consumer goods. If you look at the countries that really have made it, like Japan and the Meiji Restoration, they’re doing a lot of light industry and consumer goods. Then they move into heavy, but they’ve already got people on bicycles and they’ve got textiles and other things up and running.

That would also apply to Taiwan and Korea. They do, by all means, get heavy industry. But that’s not the starter program. The starter program is basic standard of living. Again I’m not an economist, but it turns out if you just look at who’s rich and who’s not, that seems to me a more workable thing.

Dwarkesh Patel

There’s also the fact that the centralized regime is building things according to the 30s plan. And even after post-war reconstruction, they’re still calling back on these plans from the 30s that call for heavy industry for a bygone era.

In the 70s, 80s, we had our rust belt collapse of manufacturing. People complain about this as, “Look, the US has this hollowed-out manufacturing base.” But it’s much better to have industries which are left behind so that the whole economy as a whole can be more dynamic and move on than the Soviet Union where the entire thing became a rust belt because they couldn’t move on.

Sarah Paine

It’s more exciting than that. Again, I’m not an economist, but apparently they missed the plastics revolution. I mean think about our own lives. Now we’re finding we have too many plastics, but plastics are an incredible material and they’re just missing that. I remember in Russia trying to figure out where to get sour cream and was being laughed at by Russians because I was so stupid in the store that I couldn’t find it. Well, we have little plastic tubs with the sour cream. Back in the late 80s, when I was there, you had to bring your glass jar with you so you could hand it over the counter so someone could take a filthy ladle and fill up your jar. I mean, this is part of not having plastics.

And then they totally missed the computer revolution. This plays into Ronald Reagan winning the military race. We’re putting these chips and things into our ballistic missiles and they can’t do that. And that’s a problem.

Dwarkesh Patel

Speaking of plastics, I didn’t realize before preparing for this lecture the overwhelming role that oil played in first explaining why the Soviet Union was able to sustain itself for so long and then why it collapsed. By the late 50s, Soviet growth rates were already starting to go down, especially compared to the postwar boom that America is experiencing. In ‘59, they discovered these massive oil fields in Siberia.

And then from 1973 to 1985, I think, 80% of the Soviet Union’s hard currency earnings were just from oil. They use this because central planning can’t produce even grain, let alone advanced technology. They use this to import a bunch of stuff to sustain the Red Army, to sustain the population, to subsidize Eastern Europe. And then of course, prices collapsed in 1985. Do you think that if the Siberian reserves weren’t found in the late 50s, that it’s possible that the Soviet Union would have collapsed 30 years prior?

Sarah Paine

I don’t know, but they wouldn’t have been able to do all the Africa program and things. It just would be too expensive. So certainly it would have been a reduced thing. It’s also the gas reserves they got up in like the north central Soviet Union. I can’t remember the places, but this is the gas that gets pumped to Europe because that’s the better place. They make those big investments and it takes a while for them to pay off. That was a big deal because they needed help from Western oil companies or whoever does the gas pipelines, compressors, whatever it is you need. There was a big to-do about that, about whether we should sell the stuff or whether we shouldn’t sell the stuff. The Europeans wanted to sell. We were trying not to. This was going on under Reagan as well.

But anyway, they had built a lot of it and it was essential to their pocket change. But then when they got all the pocket change, they never saved. Whatever the oil wealth was, they spent up to the max. Doesn’t it sound familiar? Governments, you have money, you spend it. Forget about rainy days.

Dwarkesh Patel

So after the Soviet Union collapses, there was a period when Putin was still winning somewhat free elections. So if you look at why Russia’s economy recovers and why Putin was so popular in the 2000s, from 2000 to 2008, oil goes from $10 a barrel to $140 a barrel. This goes to your point about how we give credit or blame to political leaders for often what are just long-run macro trends.

Sarah Paine

Well, what I didn’t cover is that when the Soviet Union collapses, Soviet living standards, Russian living standards, they implode and it’s a mess for 20 years. It is just unbelievably difficult.

Oh, and another piece of the brilliant Soviet management: in order to maintain control over the empire, instead of building things all in one place, you build some plane parts here, some plane parts there, some plane parts all over the empire. So when the empire goes, great, I’ve got a quarter of a plane, and then where do I get the other parts? So all of that fell apart.

When Putin suddenly has a lot of money he starts spending it on people, because initially there’s plenty of money. Russian standards of living do go up. So of course they like him, and they give him credit for all of that. But then that runs its course, right? And then it’s less good and then he’s more excited about… Well, it’s his mindset anyway. When you get more money, you want to get the empire back. And then Russians also like that, right?

Dwarkesh Patel

Speaking of the empire, Russia’s economy just had this terrible period after the collapse of the Soviet Union. A lot of the Eastern European satellites seem to recover in this gangbusters way. Obviously, East Germany. But even Poland today is such a big success story. What’s going wrong with the mainland itself that these other countries are able to recover from communism much better?

Sarah Paine

Well, they had always been much more connected to Western Europe. Czechoslovakia before the war was a full-up highly developed country absolutely tied to the West. Poland, I believe, Copernicus is from a place like Poland, right? It’s a center of the Enlightenment.

But when I was using the George Bush Sr. archives, it’s fascinating. So it’s ‘88, ‘89 when the Soviet Union’s imploding. There’s a lot of correspondence between Eastern European, particularly Polish, leaders coming to the Bush administration saying, “Hey, our banking system, we know it’s a mess. Our financial system’s a mess. We know we need expertise to help us figure out what our legal system is going to look like.” And Bush is all over that. I’m sure he farmed them out to the private sector who would also be all over that, like giving them free consulting. So as a result, you do have them really taking advantage of this 20 years.

At the same time when Bush would have loved to have given some of the same advice… There were people like Jeffrey Sachs and others who went to the Soviet Union, but it was not remotely the same thing. This is people throughout Polish society requesting this advice, not like one guy with an office in Moscow. Basically, the Russians thought they knew it all and they thought they understood. This is all the unknown unknowns, the things you don’t understand, your blind spots. Truly, economics is a blind spot for the Soviets.

Because think about it, when the tsars ran the show, it’s like a riff off the Mongol Empire. You take cuts from people’s businesses, from trade that comes through. Then it’s also about selling basic commodities. You’re not thinking, under the tsars, of Russia doing high-end manufacturing. I mean, I guess Fabergé and some jewelry if you want to do that. But really that’s not it. It doesn’t have this commercial tradition, being tied into this commercial tradition of Western Europe and all the sea routes for trade.

Then when you get the communists, they aren’t about that at all. So there’s really a dearth of knowledge. Think about this country with all the little kids selling lemonade, right? You see them on the streets. They’re already learning. The kids who are doing newspaper routes, they’re already learning about buying things, selling things at a very young age. We just take this knowledge for granted. It was just absent in the Soviet Union and not as much absent in Eastern Europe that had been more connected in.

Dwarkesh Patel

Before we get to the period of Russian collapse, let’s go back to the end of the Soviet period. Gorbachev starts instituting these economic reforms along with glasnost and perestroika. But what I find mysterious is that those economic reforms not only fail to prevent the stagnation that the Soviet Union is experiencing, but they in fact make things worse.

You would think that reform, even if it’s handled badly, would have some sort of positive impact. If you do it badly, then it’ll have a smaller positive impact. But here it just causes this huge hyperinflation, causes all these big problems. So why did reform have this backwards impact?

Sarah Paine

There’s so much that needs reforming there. But part of it, I think, is because he wanted to do political reforms. That’s what he understands. As a human being, that would be the thing that he’s very familiar with. Think about it. He’s an A-list member of the Communist Party to be the guy when they do generational change, he’s the one. So he’s obviously very astute at that level, but the problem is economics. He’s giving away political power before he’s fixed the economic problems. China’s conclusion is there is no way you’re going to touch political power. They’re going to hang onto that and then deal with as much of the economics as they’re going to deal with. That’s part of it.

But part of it is there’s no tradition for all of these things. Then you go, “Well, how did Russia get this way?” It’s a very difficult address. Prior to the Industrial Revolution, it’s flat, neighbors all invade, and so you needed a big army in order to defeat them. A big army is going to want a war economy. Historically, you’re going to want to support a big land force. I mean this is my take. Others who are actually experts on these various periods of Russian history can come up with something else. But I think you’re funneling, you’re channeling your economics into that.

Whereas you’re looking at Europe, particularly Britain, and it’s merchants. They have a big aristocracy who are not going to dirty themselves with buying and selling stuff, but there are a tremendous number of very rich merchants in Britain that are going to influence government laws and things, which is not going to take place in Russia. Then what’s nice about the Navy for Britain is you send them away. They’re not going to run a coup in the capital because they’re off on the ship somewhere. And there aren’t that many of them compared to a standing army. So I suspect, I can’t prove this, that this leads to different outcomes or contributes to them.

Dwarkesh Patel

One theory I heard that is complementary to your theory is that Gorbachev is instituting reforms because he thinks there should be decentralization and democratization, but he doesn’t fundamentally believe in the market system. So he’s delegating power to these quasi-firms. At the same time, he thinks the price system is immoral, private property is immoral. So they can’t intermediate between themselves using real prices.

So then how do these firms intermediate? Well, there’s corruption. If you can’t use actual prices and property to figure out who gets what allocation of scarce resources, you just backroom deal, which makes the problem worse.

Sarah Paine

Well there’s no legal system and you need a legal system. Legal systems take a long time to develop. So you’re telling the Soviet Union, “Okay, communism is down and now chop chop, we need a new legal system.” It’s not going to happen.

Dwarkesh Patel

You were mentioning the problem that Eastern European countries especially had, which is that they’re going more and more into debt because they’re not able to produce globally competitive exports. They have this last-ditch effort that “We’re going to solve our problems with some technological miracle. We need to get even more over-leveraged. We’ll get some Western machinery or technology, and then we’ll be able to finally produce something that the world wants.” I’m curious up to what point this was a plausible hope. Through the 80s and even till the end of the 80s, they still believed that Czechoslovakia or East Germany or something could catch up with West Europe?

Sarah Paine

They’re desperate. Think about it. If you’re a communist leader, how many other cards are there to play? You’re looking, “Okay, this is the only card I got.” And they’re doing other things because of the social unrest. They want to import food and consumer products because they’ve been so neglected.

Then there’s another piece, which is VCRs, the videos. All of a sudden, those things came around. I remember being in the Soviet Union, the academic year of 1988-89. One of my classmates had been an English language tutor of this person in Moscow and set me up because that was the only way to get a good meal once a week. For a meal, I would talk English for an hour.

What that family wanted more than anything else was a VCR player. You could have hard currency and buy it at the diplomatic store. So I basically got them a VCR by going to the diplomatic thing with my very limited foreign currency. I bought an overpriced VCR for them and got all kinds of meals for the rest of the year. But it meant that they could all of a sudden get Western movies.

There are things in movies where there’ll be a picture of a fugitive running by the fruit section of the Berkeley Bowl. The Russians would gasp like, “Oh.” It’s unbelievable. I think that Raisa Gorbachev, Gorbachev’s wife, when she came and visited, she must have realized that a welfare mother on food stamps had better buying power than she did by just being able to have access to Walmart.

I think the elites, as they’re traveling… I have no statistical data on this, but as you travel, it’s like I’m comparing me getting sour cream in a jar. That was the other thing, counting up all the things in a Soviet supermarket. The total was something like 77 items total in this supermarket. I don’t think that compares favorably to a candy rack as you leave a 7-Eleven. And when you went by the meat section, the smell just about knocked you out, rotten meat. It was really disgusting.

I got really good at making borscht. Go to the peasant market, pay hard currency for bones, because I couldn’t afford any meat, but I could afford the bones. Then I would buy… The Russians produce really good sugar beets so I got beets. Then you’re starting to get rotten apples over the winter, but they at least come from Hungary. Russians didn’t even produce apples in those days, but Hungarians did. The Romanians provided the canned tomatoes, and I could do a credible borscht.

But you’re talking about Moscow, the center of everything. I remember buying potatoes at the market and the rotten spots felt gelatinous. So you’d have to cut those out. And then you’re wondering how many nutrients are in the rest of that potato. It was a really gross year. I remember going to the candy store and I would buy caramel from Poland or somewhere. It was like a food item because it was actually edible.

Dwarkesh Patel

At this point, I bet you were wondering why you didn’t write a biography of Napoleon so you could just visit Paris instead.

Sarah Paine

My brother’s comment is, “You’re studying Russia and China, two countries in the breakdown lane.”

Dwarkesh Patel

By the way, the point about the grocery stores having 74 items is interesting in two ways. One, central control works much better if you have a much smaller amount of items to optimize over. So if things are standardized, it can work much better. And second, to your point about GDP being hard to compare between the Soviets and the United States, how do you compare a rotten tomato or a rotten potato to the Idaho ones that you can get?

Sarah Paine

They would have compared it by pound.

Dwarkesh Patel

Exactly.

Sarah Paine

Yeah.

Dwarkesh Patel

You said you were there in ‘88 and ‘89. So this is before the Berlin Wall has fallen.

Sarah Paine

I was watching the Tiananmen demonstrations on Soviet TV. The only reason you got that TV coverage is because Gorbachev was in Beijing. So all the press was there. That’s why you have the coverage. And they stayed on because the students were demonstrating and the Chinese closed society wasn’t aware of the power of television. Guys, they’re going to film you doing all of this stuff and they will get the film out.

Dwarkesh Patel

In ‘88, was the mood… Obviously things are going terribly, but did people realize that they’re only two years away, or three years away from the complete dissolution of the Soviet Union?

Sarah Paine

No. Maybe the end of the Soviet Union, but there was such optimism of thinking we’re finally going to be a full-up democratic country. It’s going to be wonderful, with no sense of the work schedules that go into a capitalist economy. To create the wealth in this country, a lot of people are working far more than 40 hours a week, particularly as they’re getting started, working enormous hours.

That was not something that was in most people’s minds. Sure, the kids who became the ballerinas in the Bolshoi are working long hours to do that. But as an economy as a whole, they didn’t understand the source of wealth and had no inkling of all the things that are missing, not least of which is that no one’s got the right education. Great, you got Marx memorized. That does you zero good.

Dwarkesh Patel

So around this time is when people are finally learning about what actually happened during the Stalinist period.

Sarah Paine

Oh, yeah.

Dwarkesh Patel

So people are optimistic that we can have a changing of the guard and maybe things will improve. But at the same time, they’re learning about how terrible their history actually was. Between these two things… Also at some point they must realize through the 90s that things actually aren’t improving. In fact, they’re getting worse. So what is the inflection point at which the mood is just...

Sarah Paine

I don’t know, because I wasn’t living there. I was thinking that there would be impending problems as a Chicken Little American. The sky is falling, the sky is falling. Americans always think disaster is coming. I sort of fit in that crowd. But I think there was a lot of optimism and exuberance thinking, “We have the freedom to really understand our history and what’s happening.”

This is for educated people, people with college degrees in Moscow and St. Petersburg. Now what’s going on in the rest of the country is undoubtedly a different story because as bad as living in Moscow was, living in the countryside was going back in time far further. So those people weren’t living well at all. And it’s going to get really bad for them.

Dwarkesh Patel

Okay, so people are learning about these things for the first time. Is the sense that they kind of suspected? I mean, people have family. They must have known, “My uncle was off in this little mining town that he was forced to go to for a decade right after World War II.” Were they totally shocked or was there some sense that things were pretty bad and now we’re just learning the extent of it?

Sarah Paine

I think there was an understanding it was terrible, but I think there’s this exuberance of thinking it’s going to get much better. Then the disappointment is equally extreme. And then there was this feeling that the West owes us because you’re all really rich and you now owe us to fix everything. The counterargument to that is, “No you are an enormous migraine. You set back all of these countries across the globe in time with this nonfunctioning communist model that you peddled around there. And now you want extra aid.”

The problem was that we wanted to do some of the aid, but they’re not going to be receptive to it. That was another conclusion with the Bush administration, that if we dumped a lot of money in it, it would just go straight into corruption. You need a legal structure in order to place money, and they just plain didn’t have it. That was another thing that was worrying the Bush administration. There’s nowhere to put the money.

Dwarkesh Patel

Speaking of these different countries that the Soviets and the United States were competing for during the Cold War, you had this presentation where you say Reagan alone didn’t do this. But I wonder if the broader lesson is that nothing any US president did in terms of foreign policy… That was all a sideshow, this tête-à-tête competition for different Third World countries: “We’re going to get Brazil, we’re going to get Vietnam, we’re going to get Algeria.” That just seems much less significant than the fact that liberal capitalism was more appealing and out-produced communism. So even if some country, even if Brazil goes communist, this is not going to change the fundamental playing board here.

Sarah Paine

If you do not protect the liberal economies of Europe, you’re not going to have anywhere to play the liberal economic game, and also Japan. One of the reasons you feel that liberal economies work is you’ve got economic miracles going on in Japan, Korea, Taiwan, Singapore, and Hong Kong back in the day. So if you abandon those places...

Also in the Cold War, there was a tremendous amount of economic growth across the world, particularly in the Third World. Why? Because in the past, if there’s a civil war, whoever’s losing either comes to us or comes to the Russians and says, “Help us.” So whoever it is helps, and then the other side feels obliged to help, and then you’re just destroying wealth ever more rapidly. The Cold War was anything but cold in the Third World. Tens of millions of people died in these conflicts. So when you end that, all of a sudden they can start compounding growth.

So there is a problem with not countering someone who’s going to impose communist systems all over the place. Communist systems are really good at putting dictators into power in a civil war situation. It’s very effective. That’s how Mao gets into power. The problem is, then they win the civil war, they’re in power, they annihilate the opposition, but then it produces compounding poverty thereafter.

Dwarkesh Patel

So there is this conundrum, and I genuinely don’t know the answer to this. In order to beat off these communist factions and guerrillas, we often through the Cold War had to support other dictators. Probably in many cases they were better than the communist alternative. It’s just very hard to beat Pol Pot and Mao in terms of how terrible you can be. But obviously this was in its own way problematic. Even if we didn’t have to support dictators, we had to alienate countries.

You had this previous lecture that you gave on the Indo-Pakistani chapter in history where we had to alienate India in order to fend off against the Soviet Union in this little episode. I don’t know what the solution to this is. If you think that this theater mattered less, then you could say we should have just kept our hands clean of these different Third World countries. But to your point, if you want to be able to show that these countries are going to experience growth under capitalism, then you want them to not be under the subjugation of communists. But then you have to support sometimes objectionable regimes.

Sarah Paine

I think you had a more optimistic generation, ironically, optimistic. The people who had survived World War II, there was a real generosity. American servicemen and women were welcomed all over Europe and they were adored in Europe. They came back and they were a very generous group of people. Others felt generous to them.

That’s when the GI Bill just passed saying, “You’ve saved everyone. Therefore we’re going to give you college educations, extend home loans to you.” Not to African Americans, they were excluded from this, which is a problem. But white Americans weren’t. It led to massive economic growth where people who’d never had a college education in their family, they did. All of a sudden, instead of having really hard manual labor, this real optimism. And then it extended to foreign countries. This is when this country was tremendously generous to others, and it worked very well for us.

Think about the Marshall Plan. It looks really generous putting all this money into Europe. We made a fortune off of it, as did Europe. If you’re smart, you’re looking for win-wins of things where you both benefit because that’ll incentivize the other side to join in. This is basic strategy. This is one of the reasons I’ve got problems with the United States’ turn to zero-sum approaches where “I’m going to get everything, you get nothing. Then I look so smart when we do the clickbait on this moment where I get everything and you get nothing.” It’s much smarter.

The other piece is that a lot of things don’t pay off immediately. George Bush is not reelected president. He absolutely deserved to be. Because what he did, the payoff was huge, ending the Cold War on Western terms. But it doesn’t pay off in time for the next election. I think this is where Americans miss it. You’re looking at what someone does on a given day when the real implications are what’s going to happen in a decade. Like on tax policy, if we keep racking up our debt, it may get us out of the corner today, but is it going to back us into a corner later on? This is where Americans need to think a little harder about long-term implications of things.

Dwarkesh Patel

I thought when you pointed out that it would cost 60 billion Deutschmarks for West Germany to pay Gorbachev to let East Germany join West Germany. That’s a lot of money. But if you think about decades and decades of future growth, it’s a huge bargain. It’s a mistake to think about how expensive things seem at the moment. It’s another huge country that you’ve turned.

Sarah Paine

There’s a statement that politicians think of the next election, statesmen think of the next generation. George Bush and Helmut Kohl are statesmen. They’re thinking of the next generation. The group that fought World War II, many of US and allied leaders, were statespeople. They’re thinking of the next generation. Or if you’re thinking of where I’ve got Mitterrand, who’s negotiating the Maastricht Treaty about the European Union, that is statesperson’s work of what’s the next generation. It’s important. We need more statesmen, statespeople, political leaders.

Dwarkesh Patel

To try out a different thesis on you, through this period the Soviet Union is also trying to buy off other countries, especially when it thinks its economy can grow. Especially when oil, after the 1973 oil crisis, oil prices just skyrocket. This is why some Soviet citizens remember the Brezhnev era favorably. Oil made it possible for the Soviets to not only import stuff, but through the Brezhnev period there’s actually a net export of resources to Eastern European satellites rather than the other way around.

Sarah Paine

That’s probably their data. I get it, their oil is really subsidized, but everything in the Soviet Union that was worth having came from somewhere else. The problem is how do you measure it? They’re just going to measure by weight or something else. It doesn’t really capture what they’re getting.

Dwarkesh Patel

The larger question being that, it’s not like the Soviet Union didn’t think of doing things like the Marshall Plan. Obviously nothing to that extent, but this idea that you can win people’s favors by providing them military aid, providing them foreign aid. They just didn’t have the resources to do it to the extent that the US could.

Sarah Paine

That’s true, but there’s a real coercive piece too. If you mess with them, it’ll be really ugly.

Dwarkesh Patel

Here’s what I don’t understand about the arms buildup during the Cold War. The Soviet Union is spending 2% of their GDP just on nuclear weapons alone at its peak. Arms control advocates will make this quip, which is that we’ve already got enough weapons to destroy the world many times over. Why do we need more? But that is sort of an interesting question. What was the point of spending so much of GDP on the marginal nuclear weapon or marginal weapon system?

Sarah Paine

I don’t know the answer, but you read the plans about these things and you wonder what people are thinking. We were trying to develop tactical nukes. There was only a little trick with that. Whoever deployed it would be within the blast range of the tactical nuke. You’re going, “Who develops a weapon like that?” Apparently we did. Luckily we didn’t deploy it.

I don’t know the answer of why we had such massive redundancy in these nuclear weapons, why the arsenals were so massive. I don’t know the story on how you maintain these things and how long they last. It doesn’t make much more sense to me than it does to you.

Dwarkesh Patel

Another question. Sino-Soviet split, this huge diplomatic coup. The Soviets had to put a million soldiers on the Siberian front against China. They had to spend 2% of GDP just stationing and garrisoning this area, which is obviously a lot. That’s often what many countries spend on defense as a whole, let alone just along one front.

At the same time, 2% GDP, well if they just had one or two more years of extra economic growth or faster growth, that could make up for this huge diplomatic coup. Again this goes back to the point of, if some domestic policy just caused slightly higher economic growth rates, that would make up for the biggest diplomatic coup of the entire Cold War. It goes back to economy first, diplomacy second.

Sarah Paine

Firstly, I have real problems with the statistics. I got a sample size of one, moi. I remember living in Moscow. It was so backwards. It’s just breathtakingly backwards in just about every way imaginable. They got a big fancy subway system that looks remarkably retro, and at least it works. But the consumer goods were so awful, the quality was so bad. You look at the buildings themselves.

I get it, they make nuclear weapons. Do they make anything else? Their cars were a joke, their Ladas or whatever they were. It’s just thing after thing. So you’re looking at all their stats because that’s what they are telling you, that we’re so great. It really is an Emperor Wears No Clothes moment that finally the little kid goes, “Oh, you’re actually naked.”

I can give you an example. These acquaintances in Moscow were talking about hospitals outside of Moscow that some of them didn’t have running water. How do you have a hospital without running water? I don’t even know how that’s even conceivable. Or when their kid had put her hand through a glass door or something. They wanted to get her stitched up because she’s bleeding. She’s not going to die, but she’s probably bleeding all over the place. They bring her to one place and, oh, they got no thread to do the stitches. So then they have to go to another place. Who runs a country like this?

Dwarkesh Patel

Alright, you convinced me. BART is acceptable. I’ll stick here. Subway’s not a big deal. I don’t want to move to Moscow.

Okay, while the Eastern European satellites are trying to leave the Soviet Union, this has happened many times through the 20th century. Hungary in 1956, Czechoslovakia in ‘68, Poland through Solidarity. Every previous time there’s a many-million-person-strong Red Army stationed in Eastern Europe left over from World War II, which rolls in the tanks and prevents these revolutions from taking place.

So what happens in the late 80s and early 90s? The Red Army is still there. There’s still millions of Red Army soldiers. They just don’t shoot.

Sarah Paine

Generational change. The leaders don’t have the stomach for it anymore. I don’t know how you’d feel about sending tanks and going, “Oh, we’re going to splatter all these people.” I think for many Americans, that would not be the choice that they would make. So this ruthless generation is gone.

Another piece is that Gorbachev had traveled and I think he had some Czech friends. I can’t remember all of his lists of friends. But they’d been horrified by Czechoslovakia in 1968 as young people watching, as Russian young people watching it and thinking, “It’s just wrong. We shouldn’t be doing this. If communism is what it should be, this is not what should be happening.”

This is of their youth. Gorbachev and his generation. It’s not just him, he reflects a whole generation of communists. They’re thinking, “There’s got to be another way. This is just not right.” So he thinks he’s got his other way. It’s this exuberance of the reforms and things that are happening in Russia. There’s a tremendous feeling of energy. He’s telling the Poles, “You get at it too. We’re all going to do this thing.” But it’s all the expertise and things that he’s missing, that he’s unaware that he’s missing, as are all these other people, because how could they have it? They’ve been living in a command economy.

Dwarkesh Patel

This is what I wanted to ask you about. You had the de Tocqueville quote about how revolutions happen when governments start to institute some kind of reform. Gorbachev is doing perestroika, glasnost. There’s the conservative reactionary parts of the Communist Party, which by the way is a phrase I wouldn’t expect to have said. But they’re trying to resist this. So Gorbachev goes about dismantling the party secretariat and instead devolving power down to the individual republics. We know what happens later. These republics are saying, “Look, we want our own country now.”

But this raises a question. If you do inherit a brutal regime, and now you say, “I want to do reforms.” You know this dynamic that de Tocqueville pointed out, which is that as soon as you start reforms, actually what tends to happen is that you lose power, not that people consolidate it under you. What actually should you do? Because you’re like, “I want to improve people’s lives.” But as soon as you try to do that, the whole thing’s going to fall apart.

Sarah Paine

This is so far above my pay grade. I’m a professor. I have trouble justifying a B+ on a paper. I’m a believer in gradual reforms. Do it incrementally. For the Soviet Union, it would be gradual legal reforms, work it through their Duma slowly, and do it that way. But seek out help from the European Union that has many, many experts that would be overjoyed if Putin and friends would cease doing their number on Ukraine. Now the problem is you’re going to get into reparations for the horrors they’ve inflicted. So that ship has sadly sailed for this generation. There’s no nice ending for Russians. It’s too late.

But you can look at Europe itself improving its institutions and Ukraine improving its institutions. If you think about what forces you to change, the existential threat on Ukraine, if they survive all this, this is forcing them really to clean up their institutions. So it’s happening rapidly there, but we don’t know the end of that story, how it ends.

Dwarkesh Patel

I do think these are interesting lessons here of whenever we look at a country from the outside, we have this thing of, “Well, just reform everything and just fix your economy.” Whenever we understand the system better… For example, in the United States, healthcare is 20% of GDP. This idea that Trump or Obama or Biden, whoever, could just come in and be like, “Well, I’ll just fix healthcare.” We recognize that this is a wildly implausible thing to happen. But then we have this expectation that in Russia, Gorbachev or Yeltsin could have just been like, “100% of my economy is messed up, and I’m just going to fix it.”

Sarah Paine

American hubris in action. Think about our country. We have one of the most crazy tax codes on the planet, and neither party can touch it. Because you touch any part of it, someone negotiated that wording exactly. Yet think of how much of our economy is taken up by the overhead of all the tax accountants, all the misdirected cash in order to take advantage of something that’s simply an invention of the tax system.

There was years ago when there was talk of doing a flat tax, “Wouldn’t that be much more efficient?” You can imagine what accountants thought about that one. That idea has totally died. Talk about inefficiency. Then we realize we have budgetary problems in this country. This would seem to be something that ought to be on people’s radar, clean up the tax code. But isn’t it precisely that many people don’t want the radar on the tax code? That’s why we’re wondering who can get in and out of girls’ or boys’ bathrooms, instead of looking at the tax code, which should be the real thing.

Dwarkesh Patel

I think there should be big deductions for podcasts. It should count for research and development.

Sarah Paine

Well, Dwarkesh, you’re almost at that stage. You need to add a lobbyist in DC.

Dwarkesh Patel

We’ll work on it.

There’s a very interesting book about North Korea, I forget the title, where the author is pointing out that North Korea could not even start doing reforms today because as soon as there was some sort of information from the outside world that North Koreans could see—which would be part of any reform—they would immediately realize that everything the government has told them is false. South Korea is enormously wealthier and they have this terrible standard of living.

Obviously, this is the same experience that Eastern Europeans had. Literally in many cases, you had a country that was bisected in half and the other half is living so much richer. In those situations, I guess this goes back to the question of, “Well, today in North Korea, how would it even kick off if Kim Jong-un just had a change of heart or if somebody else came into power?” They’re probably just trapped in this to the extent that they want to keep power.

Sarah Paine

Oh, he’s trapped because he’s a dead boy if he tries to take a go at retirement. In Asia—I don’t know exactly all of the parts of Asia where this applies to, it’s some parts—there’s a thought that things last for three generations and then it’s over. So he’s the third generation. Whether this is true or not doesn’t matter. If you believe it’s true, it will become a self-fulfilling prophecy. So I’ll be interested. I probably won’t live to see it, you in the room will, what happens to the Kim family, whether it makes it to generation four or not. But by their own belief system, in theory, they shouldn’t. So who knows?

Dwarkesh Patel

One more question about oil.

Sarah Paine

Based on my big expertise on oil, zero. Okay.

Dwarkesh Patel

During this period between ‘73 and ‘85, when they had these huge oil revenues, presumably there was some amount of exuberance. But did the government recognize and realize that they’re super fragile to the price of oil and if that collapses, they need some sort of contingency plan, some rainy day fund? You must notice that, “Oh, this is half my budget, and all of my foreign currency is coming from oil, and this is a very volatile commodity.” Nobody noticed that?

Sarah Paine

Yeah, well, it’s interesting. I was reading this long chronology that was put together sort of like early Putin. So before they really shut down all the information. It was just a chronology of the Cold War, big fat book. Just like someone like me to read a book like that. So I’m going date after date after date after date. It’s written by people who are really angry about how the Cold War turned out. One of the takeaways from the compilers of this thing is they kept criticizing. They showed how much for every year Russia was making in oil revenues. It was huge. But in their analysis it was, “And they saved none of it”, right? There was no sense of investing in something.

There’s something called consumption. There’s another thing called investment. Going around and buying a bunch of Western grain is consumption. There’s none of this being put in anything that’s going to yield anything. So that was a big criticism from the authors of this book. To the question you’re asking, “No, they just milked it while they were there.”

Dwarkesh Patel

Final question, this is not so much a question as an observation. I don’t know if you have a reaction to this. Just look at Russia’s history through the 20th century: tsarism, communism, collectivization, to more than 10% of your population dying from World War II, then back to Stalin, and then more communism, and then the economy collapses again, and then Putin. Especially if you look at the satellite states, they had all of this happen to them and worse because now they’re getting invaded.

Whereas you have other countries. Japan and Germany also had tragic histories, but then they recovered. Maybe it’s just the tragedy of Russia.

Sarah Paine

Yeah, you’re lucky you’re not Russian.

Dwarkesh Patel

Yeah, exactly.

Sarah Paine

No, it is tragic. It is tragic. It started out as a difficult address, pre-Industrial Revolution, that required certain things to survive. They were more ruthless than their neighbors. They did survive. I mean, in a previous lecture, I discussed how they wiped out entire princely states and Khanates and things, they just wiped them out. Then you’re using their elites because it’s a rough neighborhood. The problem is if you aren’t on the winning side, you’re going to be on the losing side, right?

But since the Industrial Revolution, where you can do compounded economic growth that comes from commerce and trade and industry and things, that’s the real way to get powerful because power becomes a function of your wealth. That involves having legal systems, institutions, and stability. Russia has found it very difficult getting with that program. It has to do with, I think, this very difficult historical legacy of who rises to power, and also all the missing things. They didn’t have the Renaissance, they didn’t have the Reformation, these fundamental movements that were very influential in the West.

So there’s a lot of negative space of things that didn’t happen. There’s all the awful stuff that you saw that did happen, but then they’re missing things. So it’s very difficult. Then people like Putin can set the clock way back because he’s killed so many Ukrainians. What he’s done will take a generation at minimum to get to anywhere where people are going to be thinking about… People will be talking about reparations from Russia for quite a while and they’re poor, they’re not going to want to do that.

Dwarkesh Patel

I should have thought to end on a more optimistic note, but...

Sarah Paine

Well, history’s ended, okay?

Dwarkesh Patel

Well, you’ve outlined the ways in which countries can chart a better course for themselves and that’s where the optimism can come from.

Sarah Paine

Actually, I’ve told a story about the last Cold War that stayed cold in the industrialized world, which was a good thing, because it could have been nuclear. It was tragic in many other parts of the world, but at least it stayed cold in the industrialized part. There was a strategy that a very thoughtful generation of people, not just in the United States but all over the West, put together to allow for a non-nuclear landing for the Soviet Union when it fell apart. From this, you can derive some of the strategies that worked for ending it that way. These are the kind of strategies that we’re going to have to use in order to navigate the second Cold War.

The other piece about the Cold War is the Soviet Union living miserable lives of their own making. But Americans were actually having a good time. They paid taxes, they had to pay for all the nuclear weapons. But as I recall, people are running around in Disneyland, they’re doing their European trips, they’re buying houses. So actually Americans, people in Western Europe, were living fulfilling lives while they’re waiting out for others to get with the program.

If we’re going to make it through this second one, we need to start cooperating with our allies, building institutions, and improving laws. Don’t just burn down the house. We will get through this one too, and we will live fulfilling lives while we’re waiting for Putin to come up with something different or Xi Jinping to come up with something different. But if we blow through our good hand of cards...

You interview all kinds of people at the cutting edge of technology. If we get rid of all of our university funding, we aren’t going to have the intellectual capital on which those businesses are based. If we’re going to dump all our allies for unknown reasons and just alienate them so they organize without us… If we’re going to just throw away entire institutions without thinking very carefully about what we’re doing… We become a cooperative adversary and we will be the bozo putting a plastic bag on our own head.

I look at the rhymes here. The Soviets had this ancient leadership who just couldn’t get their act together and they’re living off of debt instead of thinking creatively. The rhymes are awful, but we don’t have to do it that way. So it is more optimistic, but we need to get our house in order. That’s why I’m doing these lectures. They’re lectures in strategy to give you tools on how to come to your own decisions. That’s your business, not mine.

Dwarkesh Patel

That is an excellent note to close on. Sarah, I want to thank you so much for doing this lecture series with us. It has been a true education across these six lectures, everything from individual wars to the strategic and tactical decisions which explain them, to the broader lessons for today’s world. I do interview lots of different kinds of people, but from a sort of view-per-minute average-adjusted basis, I host a Sarah Paine podcast. If you just sort by popular, Sarah Paine comes up a lot.

Sarah Paine

But you’ve got it backwards. I was an unknown academic and then you cold-called me about doing an interview. I said, “Sure.” Dwarkesh, as a result of all this, I’m getting emails from all over the place. So let’s talk about who’s grateful to whom. Anyway, I’m devoted to your generation. Thank you for having me. Thank you for coming and being such a warm audience. Really appreciate it.

I’m confused why some people have short timelines and at the same time are bullish on the current scale up of reinforcement learning atop LLMs. If we’re actually close to a human-like learner, this whole approach of training on verifiable outcomes is doomed.

Currently the labs are trying to bake in a bunch of skills into these models through “mid-training” - there’s an entire supply chain of companies building RL environments which teach the model how to navigate a web browser or use Excel to write financial models.

Either these models will soon learn on the job in a self directed way - making all this pre-baking pointless - or they won’t - which means AGI is not imminent. Humans don’t have to go through a special training phase where they need to rehearse every single piece of software they might ever need to use.

Beren Millidge made interesting points about this in a recent blog post:

When we see frontier models improving at various benchmarks we should think not just of increased scale and clever ML research ideas but billions of dollars spent paying PhDs, MDs, and other experts to write questions and provide example answers and reasoning targeting these precise capabilities ... In a way, this is like a large-scale reprise of the expert systems era, where instead of paying experts to directly program their thinking as code, they provide numerous examples of their reasoning and process formalized and tracked, and then we distill this into models through behavioural cloning. This has updated me slightly towards longer AI timelines since given we need such effort to design extremely high quality human trajectories and environments for frontier systems implies that they still lack the critical core of learning that an actual AGI must possess.

You can see this tension most vividly in robotics. In some fundamental sense, robotics is an algorithms problem, not a hardware or data problem — with very little training, humans can learn how to teleoperate current hardware to do useful work. So if we had a human like learner, robotics would (in large part) be solved. But the fact that we don’t have such a learner makes it necessary to go out into a thousand different homes to learn how to pick up dishes or fold laundry.

One counterargument I’ve heard from the takeoff-within-5-years crew is that we have to do this cludgy RL in service of building a superhuman AI researcher, and then the million copies of automated Ilya can go figure out how to solve robust and efficient learning from experience.

This gives the vibes of that old joke, “We’re losing money on every sale, but we’ll make it up in volume.” Somehow this automated researcher is going to figure out the algorithm for AGI - a problem humans have been banging their head against for the better part of a century - while not having the basic learning capabilities that children have? I find this super implausible.

Besides, even if you think the RLVR scaleup will soon help us automate AI research, the labs’ actions suggest otherwise. You don’t need to pre-bake the consultant’s skills at crafting Powerpoint slides in order to automate Ilya. So clearly the labs’ actions hint at a world view where these models will continue to fare poorly at generalizing and on-the-job learning, thus making it necessary to build in the skills that they hope will be economically valuable.

Another counterargument you could make is that even if the model could learn these skills on the job, it is just so much more efficient to build them up just once during training rather that again and again for each user or company. And look, it makes a lot of sense to just bake in fluency with common tools like browsers and terminals. Indeed one of the key advantages that AGIs will have is this greater capacity to share knowledge across copies. But people are underrating how much company and context specific skills are required to do most jobs. And there just isn’t currently a robust efficient way for AIs to pick up those skills.

I was at a dinner with an AI researcher and a biologist. The biologist said she had long timelines. We asked what she thought AI would struggle with. She said her work has recently involved looking at slides and decide if a dot is actually a macrophage or just looks like one. The AI researcher says, “Image classification is a textbook deep learning problem—we could easily train for that.”

I thought this was a very interesting exchange, because it revealed a key crux between me and the people who expect transformative economic impacts in the next few years. Human workers are valuable precisely because we don’t need to build schleppy training loops for every small part of their job. It’s not net-productive to build a custom training pipeline to identify what macrophages look like given the way this particular lab prepares slides, then another for the next lab-specific micro-task, and so on. What you actually need is an AI that can learn from semantic feedback or from self directed experience, and then generalize, the way a human does.

Every day, you have to do a hundred things that require judgment, situational awareness, and skills & context learned on the job. These tasks differ not just across different people, but from one day to the next even for the same person. It is not possible to automate even a single job by just baking in some predefined set of skills, let alone all the jobs.

In fact, I think people are really underestimating how big a deal actual AGI will be because they’re just imagining more of this current regime. They’re not thinking about billions of human-like intelligences on a server which can copy and merge all their learnings. And to be clear, I expect this (aka actual AGI) in the next decade or two. That’s fucking crazy!

Sometimes people will say that the reason that AIs aren’t more widely deployed across firms and already providing lots of value (outside of coding) is that technology takes a long time to diffuse. I think this is cope. People are using this cope to gloss over the fact that these models just lack the capabilities necessary for broad economic value.

Steven Byrnes has an excellent post on this and many other points:

New technologies take a long time to integrate into the economy? Well ask yourself: how do highly-skilled, experienced, and entrepreneurial immigrant humans manage to integrate into the economy immediately? Once you’ve answered that question, note that AGI will be able to do those things too.

If these models were actually like humans on a server, they’d diffuse incredibly quickly. In fact, they’d be so much easier to integrate and onboard than a normal human employee (they could read your entire Slack and Drive in minutes and immediately distill all the skills your other AI employees have). Plus, hiring is very much like a lemons market, where it’s hard to tell who the good people are, and hiring someone bad is quite costly. This is a dynamic you wouldn’t have to worry about when you just wanna spin up another instance of a vetted AGI model.

For these reasons, I expect it’s going to be much much easier to diffuse AI labor into firms than it is to hire a person. And companies hire lots of people all the time. If the capabilities were actually at AGI level, people would be willing to spend trillions of dollars a year buying tokens (knowledge workers cumulatively earn 10s of trillions of dollars of wages a year). The reason that lab revenue are 4 orders of magnitude off right now is that the models are nowhere near as capable as human knowledge workers.

AI bulls will often criticize AI bears for repeatedly moving the goal posts. This is often fair. AI has made a ton of progress in the last decade, and it’s easy to forget that.

But some amount of goal post shifting is justified. If you showed me Gemini 3 in 2020, I would have been certain that it could automate half of knowledge work. We keep solving what we thought were the sufficient bottlenecks to AGI (general understanding, few shot learning, reasoning), and yet we still don’t have AGI (defined as, say, being able to completely automate 95% of knowledge work jobs). What is the rational response?

It’s totally reasonable to look at this and say, “Oh actually there’s more to intelligence and labor than I previously realized. And while we’re really close to (and in many ways have surpassed) what I would have defined as AGI in the past, the fact that model companies are not making trillions is revenue clearly reveals that my previous definition of AGI was too narrow.”

I expect this to keep happening into the future. I expect that by 2030 that the labs will have made significant progress on my hobby horse of continual learning, and the models will start earning 100s of billions in revenue, but they won’t have automated all knowledge work, and I’ll be like, “We’ve made a lot of progress, but we’re not at AGI yet. We also need X, Y, and Z thing to get to trillions in revenue.”

Models keep getting more impressive at the rate the short timelines people predict, but more useful at the rate the long timelines people predict.

With pretraining, we had this extremely clean and general trend in improvement in loss across multiple orders of magnitude of compute (albeit on a power law, which is as weak as exponential growth is strong). People are trying to launder the presitge of pretraining scaling, which was almost as predictable as a physical law of the universe, to justify bullish projections about RLVR, for which we have no well fit publicly known trend. When intrepid researchers do try to piece together the implications from scarce public datapoints, they get quite bearish results. For example, Toby Ord has a great post where he cleverly connects the dots between different o-series benchmark charts, which suggested “we need something like a 1,000,000x scale-up of total RL compute to give a boost similar to a GPT level”.

There is huge variance in the amount of value that different humans can add, especially in white collar with its O-ring dynamics. The village idiot adds ~0 value to knowledge work, while top AI researchers are worth billions of dollars to Mark Zuckerberg.

AI models at any given snapshot of time, however, are roughly equally capable. Humans have all this variance, whereas AI models don’t. Because a disproportionate share of value-add in knowledge work comes from the top percentile humans, if we try to compare the intelligence of these AI models to the median human, then we will systematically overestimate the value they can generate. But by the same token, when models finally do match top human performance, their impact might be quite explosive.

People have spent a lot of time talking about a software only singularity (where AI models write the code for a smarter successor system), a software + hardware singularity (where AIs also improve their successor’s computing hardware), or variations therein.

All these scenarios neglect what I think will be the main driver of further improvements atop AGI: continual learning. Again, think about how humans become more capable at anything. It’s mostly from experience in the relevant domain.

Over conversation, Beren Millidge made the interesting suggestion that the future might look continual learning agents going out, doing jobs and generating value, and then bringing all their learnings back to the hive mind model, which does some kind of batch distillations on all these agents. The agents themselves could be quite specialized - containing what Karpathy called “the cognitive core” plus knowledge and skills relevant to the job they’re being deployed to do.

“Solving” continual learning won’t be a singular one-and-done achievement. Instead, it will feel like solving in context learning. GPT-3 demonstrated that in context learning could be very powerful (its ICL capabilities were so remarkable that the title of the GPT-3 paper is ‘Language Models are Few-Shot Learners’). But of course, we didn’t “solve” in-context learning when GPT-3 came out - and indeed there’s plenty of progress still to be made, from comprehension to context length. I expect a similar progression with continual learning. Labs will probably release something next year which they call continual learning, and which will in fact count as progress towards continual learning. But human level continual learning may take another 5 to 10 years of further progress.

This is why I don’t expect some kind of runaway gains to the first model that cracks continual learning, thus getting more and more widely deployed and capable. If you had fully solved continual learning drop out of nowhere, then sure, it’s “game set match”, as Satya put it. But that’s not what’s going to happen. Instead, some lab is going to figure out how to get some initial traction on the problem. Playing around with this feature will make it clear how it was implemented, and the other labs will soon replicate this breakthrough and improve it slightly.

There’ll also probably be diminishing returns from learning-from-deployment. Each of the first 1000 consultant agents are each learning a ton from deployment. Less so the next 1000. And is there such a long tail to consultant work that the millionth deployed instance is likely to see something super important the other 999,999 instances missed? In fact, I wouldn’t be surprised if continual learning also ends up leading to a power law, but with respect to the number of instances deployed.

Besides, I just have some prior that competition will stay fierce, informed by the observation that all these previous supposed flywheels (user engagement on chat, synthetic data, etc) have done very little to diminish the greater and greater competition between model companies. Every month (or less), the big three will rotate around the podium, with other competitors not that far behind. There is some force (potentially talent poaching, rumor mills, or reverse engineering) which has so far neutralized any runaway advantages a single lab might have had.

I originally titled my podcast The Lunar Society. I changed it to Dwarkesh Podcast eventually because people kept thinking it was a crypto podcast (”to the moon!!!”). I named it after The Lunar Society of Birmingham, an informal club that met in the late 18th century. Members included James Watt, Matthew Boulton, Erasmus Darwin, Joseph Priestley, and Josiah Wedgwood. These were the scientists, inventors, and philosophers who had made first contact with the Industrial Revolution which was just starting to take shape around them. And they discussed everything from steam engines to abolition to chemistry to education reform.

Someday people will look back on this period the way we look back on the Enlightenment. Great thinkers having important debates right as the world was about to undergo these massive technological, economic, and political revolutions. And some of these thinkers actually managed to get a couple of the big things right.

Whatever happens next, I want the debates to have happened on this podcast, and to have happened well.

I wanna make essays a first class citizen of what I do. This is for a couple of reasons:

• Interviews are best when I have some take that I can bounce against my guest. You only get to see Federer’s skill when he’s rallying against a decent player, and certainly not if he’s just bouncing the ball against a wall.

• As AI becomes more and more closed off, the best people will not be in a position where they can explain their thinking clearly. This is why the Karpathy episode was so incredible. It’s rare to get an industry expert without any particular thing to pitch, and who can talk openly about the research. But I’m not aware of anyone else who is Karpathy-tier, and who is not obliged to keep his or her mouth shut about a couple of things.

• My essays have done much better than my expectations, in terms of reach, correctness and impact. I wrote the continual learning essay on a whim one afternoon, because I wanted to articulate why all these LLM scripts I’ve written for my business haven’t been helpful. And I’m still a little shocked to realize that I had stumbled upon (at least part of) what Ilya is working on at SSI. It’s not a crazy insight by any means, but it’s notable that you can just think about stuff, and there’s a good chance you’ll figure out what’s up. Btw, after I released the essay, both Sam Altman and Demis Hassabis have said that continual learning is a major bottleneck on the path to AGI. Of course, there’s no way to know whether they read my essay. But honestly, even if they hadn’t, I’d still be pretty stoked if I had independently pointed my finger at the exact same bottleneck as these guys, despite all their additional context.

• Which brings to my next point. I feel like there’s actually not that many secrets. The researchers and CEOs of the AI labs are a couple months ahead of you. This just doesn’t amount to any substantial secret knowledge that, if only you knew, you’d also have 2027 timelines. A ton of progress has been made in the last 3 years since ChatGPT, but none of it was super shocking based on the rumor mill and some connecting of the dots. And then there’s the big picture questions about AI’s impacts, where your thinking might very plausibly be much better than people at the labs, just because it takes time to think, and these people are busy running a damn company.

• Some of the questions I’m most interested in simply can’t be answered extemporaneously by any human being on the planet. They require knowledge across multiple different fields, and couple hours (to days) of crunching the numbers or thinking through shit.

• Because often enough my guests can’t just answer pretty complicated fractal questions in a satisfying way on the spot, I get frustrated with the whole enterprise. The main angst I’ve kept receding back to over and over is, “Okay what did I actually learn from this interview? And if I didn’t get that much concrete insight and understanding out of it, despite a week+ of research and hours of conversation, what hope is there for the audience? And if no one learned anything, what the fuck are we doing here?” I feel much essays survive this cynicism much better. For example, I’m often frustrated that social scientists won’t speculate with me about what their insights imply about AI civilization, or historians about how history might have turned out differently given different counterfactuals. But it’s ridiculous to count on a scholar who is thinking about AGI for first time in his life to start shooting off some galaxy brain implications from his theory. But I can go read their books, and use my understanding of the technology to come up with some hot takes.

• I can easily co-release my essays as narrations on my podcast and YouTube feed, so actually the essays are super complementary to this audio/video audience I’ve built up.

In the spirit of Thanksgiving: a lottery winner who then won another lottery is less lucky than I am.

Every once in a while, I’ll be grabbing dinner with a writer whose work I was obsessed with in college. And a part of me is just like, “What the fuck is happening right now?” Many of my greatest intellectual heroes are now my direct friends and teachers. My job is to spend a week learning about whatever I’m most interested in, and then talk to the world expert on that topic. A job I would pay to do has rewarded me - intellectually, financially, socially - beyond my wildest expectations. And there’s millions of people who are into this stuff! This audience contains some of the smartest people in the world, including many of the people I am a huge fan of. Then there’s my team. It’s unreal how talented, agentic, tasteful, and detail-oriented my colleagues are. I genuinely have no idea how I convinced people this good to come run a podcast.