Dwarkesh Podcast

Ilya & I discuss SSI’s strategy, the problems with pre-training, how to improve the generalization of AI models, and how to ensure AGI goes well.

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Gemini 3 is the first model I’ve used that can find connections I haven’t anticipated. I recently wrote a blog post on RL’s information efficiency, and Gemini 3 helped me think it all through. It also generated the relevant charts and ran toy ML experiments for me with zero bugs. Try Gemini 3 today at gemini.google

• Labelbox helped me create a tool to transcribe our episodes! I’ve struggled with transcription in the past because I don’t just want verbatim transcripts, I want transcripts reworded to read like essays. Labelbox helped me generate the exact data I needed for this. If you want to learn how Labelbox can help you (or if you want to try out the transcriber tool yourself), go to labelbox.com/dwarkesh

• Sardine is an AI risk management platform that brings together thousands of device, behavior, and identity signals to help you assess a user’s risk of fraud & abuse. Sardine also offers a suite of agents to automate investigations so that as fraudsters use AI to scale their attacks, you can use AI to scale your defenses. Learn more at sardine.ai/dwarkesh

To sponsor a future episode, visit dwarkesh.com/advertise.

Timestamps

(00:00:00) – Explaining model jaggedness

(00:09:39) - Emotions and value functions

(00:18:49) – What are we scaling?

(00:25:13) – Why humans generalize better than models

(00:35:45) – Straight-shotting superintelligence

(00:46:47) – SSI’s model will learn from deployment

(00:55:07) – Alignment

(01:18:13) – “We are squarely an age of research company”

(01:29:23) – Self-play and multi-agent

(01:32:42) – Research taste

Transcript00:00:00 – Explaining model jaggedness

Ilya Sutskever 00:00:00

You know what’s crazy? That all of this is real.

Dwarkesh Patel 00:00:04

Meaning what?

Ilya Sutskever 00:00:05

Don’t you think so? All this AI stuff and all this Bay Area… that it’s happening. Isn’t it straight out of science fiction?

Dwarkesh Patel 00:00:14

Another thing that’s crazy is how normal the slow takeoff feels. The idea that we’d be investing 1% of GDP in AI, I feel like it would have felt like a bigger deal, whereas right now it just feels...

Ilya Sutskever 00:00:26

We get used to things pretty fast, it turns out. But also it’s kind of abstract. What does it mean? It means that you see it in the news, that such and such company announced such and such dollar amount. That’s all you see. It’s not really felt in any other way so far.

Dwarkesh Patel 00:00:45

Should we actually begin here? I think this is an interesting discussion.

Ilya Sutskever 00:00:47

Sure.

Dwarkesh Patel 00:00:48

I think your point, about how from the average person’s point of view nothing is that different, will continue being true even into the singularity.

Ilya Sutskever 00:00:57

No, I don’t think so.

Dwarkesh Patel 00:00:58

Okay, interesting.

Ilya Sutskever 00:01:00

The thing which I was referring to not feeling different is, okay, such and such company announced some difficult-to-comprehend dollar amount of investment. I don’t think anyone knows what to do with that.

But I think the impact of AI is going to be felt. AI is going to be diffused through the economy. There’ll be very strong economic forces for this, and I think the impact is going to be felt very strongly.

Dwarkesh Patel 00:01:30

When do you expect that impact? I think the models seem smarter than their economic impact would imply.

Ilya Sutskever 00:01:38

Yeah. This is one of the very confusing things about the models right now. How to reconcile the fact that they are doing so well on evals? You look at the evals and you go, “Those are pretty hard evals.” They are doing so well. But the economic impact seems to be dramatically behind. It’s very difficult to make sense of, how can the model, on the one hand, do these amazing things, and then on the other hand, repeat itself twice in some situation?

An example would be, let’s say you use vibe coding to do something. You go to some place and then you get a bug. Then you tell the model, “Can you please fix the bug?” And the model says, “Oh my God, you’re so right. I have a bug. Let me go fix that.” And it introduces a second bug. Then you tell it, “You have this new second bug,” and it tells you, “Oh my God, how could I have done it? You’re so right again,” and brings back the first bug, and you can alternate between those. How is that possible? I’m not sure, but it does suggest that something strange is going on.

I have two possible explanations. The more whimsical explanation is that maybe RL training makes the models a little too single-minded and narrowly focused, a little bit too unaware, even though it also makes them aware in some other ways. Because of this, they can’t do basic things.

But there is another explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. When you do pre-training, you need all the data. So you don’t have to think if it’s going to be this data or that data.

But when people do RL training, they do need to think. They say, “Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.” From what I hear, all the companies have teams that just produce new RL environments and just add it to the training mix. The question is, well, what are those? There are so many degrees of freedom. There is such a huge variety of RL environments you could produce.

One thing you could do, and I think this is something that is done inadvertently, is that people take inspiration from the evals. You say, “Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?” I think that is something that happens, and it could explain a lot of what’s going on.

If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance, which is something that we don’t today even understand, what we mean by that.

Dwarkesh Patel 00:05:00

I like this idea that the real reward hacking is the human researchers who are too focused on the evals.

I think there are two ways to understand, or to try to think about, what you have just pointed out. One is that if it’s the case that simply by becoming superhuman at a coding competition, a model will not automatically become more tasteful and exercise better judgment about how to improve your codebase, well then you should expand the suite of environments such that you’re not just testing it on having the best performance in coding competition. It should also be able to make the best kind of application for X thing or Y thing or Z thing.

Another, maybe this is what you’re hinting at, is to say, “Why should it be the case in the first place that becoming superhuman at coding competitions doesn’t make you a more tasteful programmer more generally?” Maybe the thing to do is not to keep stacking up the amount and diversity of environments, but to figure out an approach which lets you learn from one environment and improve your performance on something else.

Ilya Sutskever 00:06:08

I have a human analogy which might be helpful. Let’s take the case of competitive programming, since you mentioned that. Suppose you have two students. One of them decided they want to be the best competitive programmer, so they will practice 10,000 hours for that domain. They will solve all the problems, memorize all the proof techniques, and be very skilled at quickly and correctly implementing all the algorithms. By doing so, they became one of the best.

Student number two thought, “Oh, competitive programming is cool.” Maybe they practiced for 100 hours, much less, and they also did really well. Which one do you think is going to do better in their career later on?

Dwarkesh Patel 00:06:56

The second.

Ilya Sutskever 00:06:57

Right. I think that’s basically what’s going on. The models are much more like the first student, but even more. Because then we say, the model should be good at competitive programming so let’s get every single competitive programming problem ever. And then let’s do some data augmentation so we have even more competitive programming problems, and we train on that. Now you’ve got this great competitive programmer.

With this analogy, I think it’s more intuitive. Yeah, okay, if it’s so well trained, all the different algorithms and all the different proof techniques are right at its fingertips. And it’s more intuitive that with this level of preparation, it would not necessarily generalize to other things.

Dwarkesh Patel 00:07:39

But then what is the analogy for what the second student is doing before they do the 100 hours of fine-tuning?

Ilya Sutskever 00:07:48

I think they have “it.” The “it” factor. When I was an undergrad, I remember there was a student like this that studied with me, so I know it exists.

Dwarkesh Patel 00:08:01

I think it’s interesting to distinguish “it” from whatever pre-training does. One way to understand what you just said about not having to choose the data in pre-training is to say it’s actually not dissimilar to the 10,000 hours of practice. It’s just that you get that 10,000 hours of practice for free because it’s already somewhere in the pre-training distribution. But maybe you’re suggesting there’s actually not that much generalization from pre-training. There’s just so much data in pre-training, but it’s not necessarily generalizing better than RL.

Ilya Sutskever 00:08:31

The main strength of pre-training is that: A, there is so much of it, and B, you don’t have to think hard about what data to put into pre-training. It’s very natural data, and it does include in it a lot of what people do: people’s thoughts and a lot of the features. It’s like the whole world as projected by people onto text, and pre-training tries to capture that using a huge amount of data.

Pre-training is very difficult to reason about because it’s so hard to understand the manner in which the model relies on pre-training data. Whenever the model makes a mistake, could it be because something by chance is not as supported by the pre-training data? “Support by pre-training” is maybe a loose term. I don’t know if I can add anything more useful on this. I don’t think there is a human analog to pre-training.

00:09:39 – Emotions and value functions

Dwarkesh Patel 00:09:39

Here are analogies that people have proposed for what the human analogy to pre-training is. I’m curious to get your thoughts on why they’re potentially wrong. One is to think about the first 18, or 15, or 13 years of a person’s life when they aren’t necessarily economically productive, but they are doing something that is making them understand the world better and so forth. The other is to think about evolution as doing some kind of search for 3 billion years, which then results in a human lifetime instance.

I’m curious if you think either of these are analogous to pre-training. How would you think about what lifetime human learning is like, if not pre-training?

Ilya Sutskever 00:10:22

I think there are some similarities between both of these and pre-training, and pre-training tries to play the role of both of these. But I think there are some big differences as well. The amount of pre-training data is very, very staggering.

Dwarkesh Patel 00:10:39

Yes.

Ilya Sutskever 00:10:40

Somehow a human being, after even 15 years with a tiny fraction of the pre-training data, they know much less. But whatever they do know, they know much more deeply somehow. Already at that age, you would not make mistakes that our AIs make.

There is another thing. You might say, could it be something like evolution? The answer is maybe. But in this case, I think evolution might actually have an edge. I remember reading about this case. One way in which neuroscientists can learn about the brain is by studying people with brain damage to different parts of the brain. Some people have the most strange symptoms you could imagine. It’s actually really, really interesting.

One case that comes to mind that’s relevant. I read about this person who had some kind of brain damage, a stroke or an accident, that took out his emotional processing. So he stopped feeling any emotion. He still remained very articulate and he could solve little puzzles, and on tests he seemed to be just fine. But he felt no emotion. He didn’t feel sad, he didn’t feel anger, he didn’t feel animated. He became somehow extremely bad at making any decisions at all. It would take him hours to decide on which socks to wear. He would make very bad financial decisions.

What does it say about the role of our built-in emotions in making us a viable agent, essentially? To connect to your question about pre-training, maybe if you are good enough at getting everything out of pre-training, you could get that as well. But that’s the kind of thing which seems... Well, it may or may not be possible to get that from pre-training.

Dwarkesh Patel 00:12:56

What is “that”? Clearly not just directly emotion. It seems like some almost value function-like thing which is telling you what the end reward for any decision should be. You think that doesn’t sort of implicitly come from pre-training?

Ilya Sutskever 00:13:15

I think it could. I’m just saying it’s not 100% obvious.

Dwarkesh Patel 00:13:19

But what is that? How do you think about emotions? What is the ML analogy for emotions?

Ilya Sutskever 00:13:26

It should be some kind of a value function thing. But I don’t think there is a great ML analogy because right now, value functions don’t play a very prominent role in the things people do.

Dwarkesh Patel 00:13:36

It might be worth defining for the audience what a value function is, if you want to do that.

Ilya Sutskever 00:13:39

Certainly, I’ll be very happy to do that. When people do reinforcement learning, the way reinforcement learning is done right now, how do people train those agents? You have your neural net and you give it a problem, and then you tell the model, “Go solve it.” The model takes maybe thousands, hundreds of thousands of actions or thoughts or something, and then it produces a solution. The solution is graded.

And then the score is used to provide a training signal for every single action in your trajectory. That means that if you are doing something that goes for a long time—if you’re training a task that takes a long time to solve—it will do no learning at all until you come up with the proposed solution. That’s how reinforcement learning is done naively. That’s how o1, R1 ostensibly are done.

The value function says something like, “Maybe I could sometimes, not always, tell you if you are doing well or badly.” The notion of a value function is more useful in some domains than others. For example, when you play chess and you lose a piece, I messed up. You don’t need to play the whole game to know that what I just did was bad, and therefore whatever preceded it was also bad.

The value function lets you short-circuit the wait until the very end. Let’s suppose that you are doing some kind of a math thing or a programming thing, and you’re trying to explore a particular solution or direction. After, let’s say, a thousand steps of thinking, you concluded that this direction is unpromising. As soon as you conclude this, you could already get a reward signal a thousand timesteps previously, when you decided to pursue down this path. You say, “Next time I shouldn’t pursue this path in a similar situation,” long before you actually came up with the proposed solution.

Dwarkesh Patel 00:15:52

This was in the DeepSeek R1 paper— that the space of trajectories is so wide that maybe it’s hard to learn a mapping from an intermediate trajectory and value. And also given that, in coding for example you’ll have the wrong idea, then you’ll go back, then you’ll change something.

Ilya Sutskever 00:16:12

This sounds like such lack of faith in deep learning. Sure it might be difficult, but nothing deep learning can’t do. My expectation is that a value function should be useful, and I fully expect that they will be used in the future, if not already.

What I was alluding to with the person whose emotional center got damaged, it’s more that maybe what it suggests is that the value function of humans is modulated by emotions in some important way that’s hardcoded by evolution. And maybe that is important for people to be effective in the world.

Dwarkesh Patel 00:17:00

That’s the thing I was planning on asking you. There’s something really interesting about emotions of the value function, which is that it’s impressive that they have this much utility while still being rather simple to understand.

Ilya Sutskever 00:17:15

I have two responses. I do agree that compared to the kind of things that we learn and the things we are talking about, the kind of AI we are talking about, emotions are relatively simple. They might even be so simple that maybe you could map them out in a human-understandable way. I think it would be cool to do.

In terms of utility though, I think there is a thing where there is this complexity-robustness tradeoff, where complex things can be very useful, but simple things are very useful in a very broad range of situations. One way to interpret what we are seeing is that we’ve got these emotions that evolved mostly from our mammal ancestors and then fine-tuned a little bit while we were hominids, just a bit. We do have a decent amount of social emotions though which mammals may lack. But they’re not very sophisticated. And because they’re not sophisticated, they serve us so well in this very different world compared to the one that we’ve been living in.

Actually, they also make mistakes. For example, our emotions… Well actually, I don’t know. Does hunger count as an emotion? It’s debatable. But I think, for example, our intuitive feeling of hunger is not succeeding in guiding us correctly in this world with an abundance of food.

00:18:49 – What are we scaling?

Dwarkesh Patel 00:18:49

People have been talking about scaling data, scaling parameters, scaling compute. Is there a more general way to think about scaling? What are the other scaling axes?

Ilya Sutskever 00:19:00

Here’s a perspective that I think might be true. The way ML used to work is that people would just tinker with stuff and try to get interesting results. That’s what’s been going on in the past.

Then the scaling insight arrived. Scaling laws, GPT-3, and suddenly everyone realized we should scale. This is an example of how language affects thought. “Scaling” is just one word, but it’s such a powerful word because it informs people what to do. They say, “Let’s try to scale things.” So you say, what are we scaling? Pre-training was the thing to scale. It was a particular scaling recipe.

The big breakthrough of pre-training is the realization that this recipe is good. You say, “Hey, if you mix some compute with some data into a neural net of a certain size, you will get results. You will know that you’ll be better if you just scale the recipe up.” This is also great. Companies love this because it gives you a very low-risk way of investing your resources.

It’s much harder to invest your resources in research. Compare that. If you research, you need to be like, “Go forth researchers and research and come up with something”, versus get more data, get more compute. You know you’ll get something from pre-training.

Indeed, it looks like, based on various things some people say on Twitter, maybe it appears that Gemini have found a way to get more out of pre-training. At some point though, pre-training will run out of data. The data is very clearly finite. What do you do next? Either you do some kind of souped-up pre-training, a different recipe from the one you’ve done before, or you’re doing RL, or maybe something else. But now that compute is big, compute is now very big, in some sense we are back to the age of research.

Maybe here’s another way to put it. Up until 2020, from 2012 to 2020, it was the age of research. Now, from 2020 to 2025, it was the age of scaling—maybe plus or minus, let’s add error bars to those years—because people say, “This is amazing. You’ve got to scale more. Keep scaling.” The one word: scaling.

But now the scale is so big. Is the belief really, “Oh, it’s so big, but if you had 100x more, everything would be so different?” It would be different, for sure. But is the belief that if you just 100x the scale, everything would be transformed? I don’t think that’s true. So it’s back to the age of research again, just with big computers.

Dwarkesh Patel 00:22:06

That’s a very interesting way to put it. But let me ask you the question you just posed then. What are we scaling, and what would it mean to have a recipe? I guess I’m not aware of a very clean relationship that almost looks like a law of physics which existed in pre-training. There was a power law between data or compute or parameters and loss. What is the kind of relationship we should be seeking, and how should we think about what this new recipe might look like?

Ilya Sutskever 00:22:38

We’ve already witnessed a transition from one type of scaling to a different type of scaling, from pre-training to RL. Now people are scaling RL. Now based on what people say on Twitter, they spend more compute on RL than on pre-training at this point, because RL can actually consume quite a bit of compute. You do very long rollouts, so it takes a lot of compute to produce those rollouts. Then you get a relatively small amount of learning per rollout, so you really can spend a lot of compute.

I wouldn’t even call it scaling. I would say, “Hey, what are you doing? Is the thing you are doing the most productive thing you could be doing? Can you find a more productive way of using your compute?” We’ve discussed the value function business earlier. Maybe once people get good at value functions, they will be using their resources more productively. If you find a whole other way of training models, you could say, “Is this scaling or is it just using your resources?” I think it becomes a little bit ambiguous.

In the sense that, when people were in the age of research back then, it was, “Let’s try this and this and this. Let’s try that and that and that. Oh, look, something interesting is happening.” I think there will be a return to that.

Dwarkesh Patel 00:24:10

If we’re back in the era of research, stepping back, what is the part of the recipe that we need to think most about? When you say value function, people are already trying the current recipe, but then having LLM-as-a-Judge and so forth. You could say that’s a value function, but it sounds like you have something much more fundamental in mind. Should we even rethink pre-training at all and not just add more steps to the end of that process?

Ilya Sutskever 00:24:35

The discussion about value function, I think it was interesting. I want to emphasize that I think the value function is something that’s going to make RL more efficient, and I think that makes a difference. But I think anything you can do with a value function, you can do without, just more slowly. The thing which I think is the most fundamental is that these models somehow just generalize dramatically worse than people. It’s super obvious. That seems like a very fundamental thing.

00:25:13 – Why humans generalize better than models

Dwarkesh Patel 00:25:13

So this is the crux: generalization. There are two sub-questions. There’s one which is about sample efficiency: why should it take so much more data for these models to learn than humans? There’s a second question. Even separate from the amount of data it takes, why is it so hard to teach the thing we want to a model than to a human? For a human, we don’t necessarily need a verifiable reward to be able to… You’re probably mentoring a bunch of researchers right now, and you’re talking with them, you’re showing them your code, and you’re showing them how you think. From that, they’re picking up your way of thinking and how they should do research.

You don’t have to set a verifiable reward for them that’s like, “Okay, this is the next part of the curriculum, and now this is the next part of your curriculum. Oh, this training was unstable.” There’s not this schleppy, bespoke process. Perhaps these two issues are actually related in some way, but I’d be curious to explore this second thing, which is more like continual learning, and this first thing, which feels just like sample efficiency.

Ilya Sutskever 00:26:19

You could actually wonder that one possible explanation for the human sample efficiency that needs to be considered is evolution. Evolution has given us a small amount of the most useful information possible. For things like vision, hearing, and locomotion, I think there’s a pretty strong case that evolution has given us a lot.

For example, human dexterity far exceeds… I mean robots can become dexterous too if you subject them to a huge amount of training in simulation. But to train a robot in the real world to quickly pick up a new skill like a person does seems very out of reach. Here you could say, “Oh yeah, locomotion. All our ancestors needed great locomotion, squirrels. So with locomotion, maybe we’ve got some unbelievable prior.”

You could make the same case for vision. I believe Yann LeCun made the point that children learn to drive after 10 hours of practice, which is true. But our vision is so good. At least for me, I remember myself being a five-year-old. I was very excited about cars back then. I’m pretty sure my car recognition was more than adequate for driving already as a five-year-old. You don’t get to see that much data as a five-year-old. You spend most of your time in your parents’ house, so you have very low data diversity.

But you could say maybe that’s evolution too. But in language and math and coding, probably not.

Dwarkesh Patel 00:28:00

It still seems better than models. Obviously, models are better than the average human at language, math, and coding. But are they better than the average human at learning?

Ilya Sutskever 00:28:09

Oh yeah. Oh yeah, absolutely. What I meant to say is that language, math, and coding—and especially math and coding—suggests that whatever it is that makes people good at learning is probably not so much a complicated prior, but something more, some fundamental thing.

Dwarkesh Patel 00:28:29

I’m not sure I understood. Why should that be the case?

Ilya Sutskever 00:28:32

So consider a skill in which people exhibit some kind of great reliability. If the skill is one that was very useful to our ancestors for many millions of years, hundreds of millions of years, you could argue that maybe humans are good at it because of evolution, because we have a prior, an evolutionary prior that’s encoded in some very non-obvious way that somehow makes us so good at it.

But if people exhibit great ability, reliability, robustness, and ability to learn in a domain that really did not exist until recently, then this is more an indication that people might have just better machine learning, period.

Dwarkesh Patel 00:29:29

How should we think about what that is? What is the ML analogy? There are a couple of interesting things about it. It takes fewer samples. It’s more unsupervised. A child learning to drive a car… Children are not learning to drive a car. A teenager learning how to drive a car is not exactly getting some prebuilt, verifiable reward. It comes from their interaction with the machine and with the environment. It takes much fewer samples. It seems more unsupervised. It seems more robust?

Ilya Sutskever 00:30:07

Much more robust. The robustness of people is really staggering.

Dwarkesh Patel 00:30:12

Do you have a unified way of thinking about why all these things are happening at once? What is the ML analogy that could realize something like this?

Ilya Sutskever 00:30:24

One of the things that you’ve been asking about is how can the teenage driver self-correct and learn from their experience without an external teacher? The answer is that they have their value function. They have a general sense which is also, by the way, extremely robust in people. Whatever the human value function is, with a few exceptions around addiction, it’s actually very, very robust.

So for something like a teenager that’s learning to drive, they start to drive, and they already have a sense of how they’re driving immediately, how badly they are, how unconfident. And then they see, “Okay.” And then, of course, the learning speed of any teenager is so fast. After 10 hours, you’re good to go.

Dwarkesh Patel 00:31:17

It seems like humans have some solution, but I’m curious about how they are doing it and why is it so hard? How do we need to reconceptualize the way we’re training models to make something like this possible?

Ilya Sutskever 00:31:27

That is a great question to ask, and it’s a question I have a lot of opinions about. But unfortunately, we live in a world where not all machine learning ideas are discussed freely, and this is one of them. There’s probably a way to do it. I think it can be done. The fact that people are like that, I think it’s a proof that it can be done.

There may be another blocker though, which is that there is a possibility that the human neurons do more compute than we think. If that is true, and if that plays an important role, then things might be more difficult. But regardless, I do think it points to the existence of some machine learning principle that I have opinions on. But unfortunately, circumstances make it hard to discuss in detail.

Dwarkesh Patel 00:32:28

Nobody listens to this podcast, Ilya.

00:35:45 – Straight-shotting superintelligence

Dwarkesh Patel 00:35:45

I’m curious. If you say we are back in an era of research, you were there from 2012 to 2020. What is the vibe now going to be if we go back to the era of research?

For example, even after AlexNet, the amount of compute that was used to run experiments kept increasing, and the size of frontier systems kept increasing. Do you think now that this era of research will still require tremendous amounts of compute? Do you think it will require going back into the archives and reading old papers?

You were at Google and OpenAI and Stanford, these places, when there was more of a vibe of research? What kind of things should we be expecting in the community?

Ilya Sutskever 00:36:38

One consequence of the age of scaling is that scaling sucked out all the air in the room. Because scaling sucked out all the air in the room, everyone started to do the same thing. We got to the point where we are in a world where there are more companies than ideas by quite a bit. Actually on that, there is this Silicon Valley saying that says that ideas are cheap, execution is everything. People say that a lot, and there is truth to that. But then I saw someone say on Twitter something like, “If ideas are so cheap, how come no one’s having any ideas?” And I think it’s true too.

If you think about research progress in terms of bottlenecks, there are several bottlenecks. One of them is ideas, and one of them is your ability to bring them to life, which might be compute but also engineering. If you go back to the ‘90s, let’s say, you had people who had pretty good ideas, and if they had much larger computers, maybe they could demonstrate that their ideas were viable. But they could not, so they could only have a very, very small demonstration that did not convince anyone. So the bottleneck was compute.

Then in the age of scaling, compute has increased a lot. Of course, there is a question of how much compute is needed, but compute is large. Compute is large enough such that it’s not obvious that you need that much more compute to prove some idea. I’ll give you an analogy. AlexNet was built on two GPUs. That was the total amount of compute used for it. The transformer was built on 8 to 64 GPUs. No single transformer paper experiment used more than 64 GPUs of 2017, which would be like, what, two GPUs of today? The ResNet, right? You could argue that the o1 reasoning was not the most compute-heavy thing in the world.

So for research, you definitely need some amount of compute, but it’s far from obvious that you need the absolutely largest amount of compute ever for research. You might argue, and I think it is true, that if you want to build the absolutely best system then it helps to have much more compute. Especially if everyone is within the same paradigm, then compute becomes one of the big differentiators.

Dwarkesh Patel 00:39:41

I’m asking you for the history, because you were actually there. I’m not sure what actually happened. It sounds like it was possible to develop these ideas using minimal amounts of compute. But the transformer didn’t immediately become famous. It became the thing everybody started doing and then started experimenting on top of and building on top of because it was validated at higher and higher levels of compute.

Ilya Sutskever 00:40:06

Correct.

Dwarkesh Patel 00:40:07

And if you at SSI have 50 different ideas, how will you know which one is the next transformer and which one is brittle, without having the kinds of compute that other frontier labs have?

Ilya Sutskever 00:40:22

I can comment on that. The short comment is that you mentioned SSI. Specifically for us, the amount of compute that SSI has for research is really not that small. I want to explain why. Simple math can explain why the amount of compute that we have is comparable for research than one might think. I’ll explain.

SSI has raised $3 billion, which is a lot by any absolute sense. But you could say, “Look at the other companies raising much more.” But a lot of their compute goes for inference. These big numbers, these big loans, it’s earmarked for inference. That’s number one. Number two, if you want to have a product on which you do inference, you need to have a big staff of engineers, salespeople. A lot of the research needs to be dedicated to producing all kinds of product-related features. So then when you look at what’s actually left for research, the difference becomes a lot smaller.

The other thing is, if you are doing something different, do you really need the absolute maximal scale to prove it? I don’t think that’s true at all. I think that in our case, we have sufficient compute to prove, to convince ourselves and anyone else, that what we are doing is correct.

Dwarkesh Patel 00:42:02

There have been public estimates that companies like OpenAI spend on the order of $5-6 billion a year just so far, on experiments. This is separate from the amount of money they’re spending on inference and so forth. So it seems like they’re spending more a year running research experiments than you guys have in total funding.

Ilya Sutskever 00:42:22

I think it’s a question of what you do with it. It’s a question of what you do with it. In their case, in the case of others, there is a lot more demand on the training compute. There’s a lot more different work streams, there are different modalities, there is just more stuff. So it becomes fragmented.

Dwarkesh Patel 00:42:44

How will SSI make money?

Ilya Sutskever 00:42:46

My answer to this question is something like this. Right now, we just focus on the research, and then the answer to that question will reveal itself. I think there will be lots of possible answers.

Dwarkesh Patel 00:43:01

Is SSI’s plan still to straight shot superintelligence?

Ilya Sutskever 00:43:04

Maybe. I think that there is merit to it. I think there’s a lot of merit because it’s very nice to not be affected by the day-to-day market competition. But I think there are two reasons that may cause us to change the plan. One is pragmatic, if timelines turned out to be long, which they might. Second, I think there is a lot of value in the best and most powerful AI being out there impacting the world. I think this is a meaningfully valuable thing.

Dwarkesh Patel 00:43:48

So then why is your default plan to straight shot superintelligence? Because it sounds like OpenAI, Anthropic, all these other companies, their explicit thinking is, “Look, we have weaker and weaker intelligences that the public can get used to and prepare for.” Why is it potentially better to build a superintelligence directly?

Ilya Sutskever 00:44:08

I’ll make the case for and against. The case for is that one of the challenges that people face when they’re in the market is that they have to participate in the rat race. The rat race is quite difficult in that it exposes you to difficult trade-offs which you need to make. It is nice to say, “We’ll insulate ourselves from all this and just focus on the research and come out only when we are ready, and not before.” But the counterpoint is valid too, and those are opposing forces. The counterpoint is, “Hey, it is useful for the world to see powerful AI. It is useful for the world to see powerful AI because that’s the only way you can communicate it.”

Dwarkesh Patel 00:44:57

Well, I guess not even just that you can communicate the idea—

Ilya Sutskever 00:45:00

Communicate the AI, not the idea. Communicate the AI.

Dwarkesh Patel 00:45:04

What do you mean, “communicate the AI”?

Ilya Sutskever 00:45:06

Let’s suppose you write an essay about AI, and the essay says, “AI is going to be this, and AI is going to be that, and it’s going to be this.” You read it and you say, “Okay, this is an interesting essay.” Now suppose you see an AI doing this, an AI doing that. It is incomparable. Basically I think that there is a big benefit from AI being in the public, and that would be a reason for us to not be quite straight shot.

Dwarkesh Patel 00:45:37

I guess it’s not even that, but I do think that is an important part of it. The other big thing is that I can’t think of another discipline in human engineering and research where the end artifact was made safer mostly through just thinking about how to make it safe, as opposed to, why airplane crashes per mile are so much lower today than they were decades ago. Why is it so much harder to find a bug in Linux than it would have been decades ago? I think it’s mostly because these systems were deployed to the world. You noticed failures, those failures were corrected and the systems became more robust.

I’m not sure why AGI and superhuman intelligence would be any different, especially given—and I hope we’re going to get to this—it seems like the harms of superintelligence are not just about having some malevolent paper clipper out there. But this is a really powerful thing and we don’t even know how to conceptualize how people interact with it, what people will do with it. Having gradual access to it seems like a better way to maybe spread out the impact of it and to help people prepare for it.

00:46:47 – SSI’s model will learn from deployment

Ilya Sutskever 00:46:47

Well I think on this point, even in the straight shot scenario, you would still do a gradual release of it, that’s how I would imagine it. Gradualism would be an inherent component of any plan. It’s just a question of what is the first thing that you get out of the door. That’s number one.

Number two, I believe you have advocated for continual learning more than other people, and I actually think that this is an important and correct thing. Here is why. I’ll give you another example of how language affects thinking. In this case, it will be two words that have shaped everyone’s thinking, I maintain. First word: AGI. Second word: pre-training. Let me explain.

The term AGI, why does this term exist? It’s a very particular term. Why does it exist? There’s a reason. The reason that the term AGI exists is, in my opinion, not so much because it’s a very important, essential descriptor of some end state of intelligence, but because it is a reaction to a different term that existed, and the term is narrow AI. If you go back to ancient history of gameplay and AI, of checkers AI, chess AI, computer games AI, everyone would say, look at this narrow intelligence. Sure, the chess AI can beat Kasparov, but it can’t do anything else. It is so narrow, artificial narrow intelligence. So in response, as a reaction to this, some people said, this is not good. It is so narrow. What we need is general AI, an AI that can just do all the things. That term just got a lot of traction.

The second thing that got a lot of traction is pre-training, specifically the recipe of pre-training. I think the way people do RL now is maybe undoing the conceptual imprint of pre-training. But pre-training had this property. You do more pre-training and the model gets better at everything, more or less uniformly. General AI. Pre-training gives AGI.

But the thing that happened with AGI and pre-training is that in some sense they overshot the target. If you think about the term “AGI”, especially in the context of pre-training, you will realize that a human being is not an AGI. Yes, there is definitely a foundation of skills, but a human being lacks a huge amount of knowledge. Instead, we rely on continual learning.

So when you think about, “Okay, so let’s suppose that we achieve success and we produce some kind of safe superintelligence.” The question is, how do you define it? Where on the curve of continual learning is it going to be?

I produce a superintelligent 15-year-old that’s very eager to go. They don’t know very much at all, a great student, very eager. You go and be a programmer, you go and be a doctor, go and learn. So you could imagine that the deployment itself will involve some kind of a learning trial-and-error period. It’s a process, as opposed to you dropping the finished thing.

Dwarkesh Patel 00:50:45

I see. You’re suggesting that the thing you’re pointing out with superintelligence is not some finished mind which knows how to do every single job in the economy. Because the way, say, the original OpenAI charter or whatever defines AGI is like, it can do every single job, every single thing a human can do. You’re proposing instead a mind which can learn to do every single job, and that is superintelligence.

Ilya Sutskever 00:51:15

Yes.

Dwarkesh Patel 00:51:16

But once you have the learning algorithm, it gets deployed into the world the same way a human laborer might join an organization.

Ilya Sutskever 00:51:25

Exactly.

Dwarkesh Patel 00:51:26

It seems like one of these two things might happen, maybe neither of these happens. One, this super-efficient learning algorithm becomes superhuman, becomes as good as you and potentially even better, at the task of ML research. As a result the algorithm itself becomes more and more superhuman.

The other is, even if that doesn’t happen, if you have a single model—this is explicitly your vision—where instances of a model which are deployed through the economy doing different jobs, learning how to do those jobs, continually learning on the job, picking up all the skills that any human could pick up, but picking them all up at the same time, and then amalgamating their learnings, you basically have a model which functionally becomes superintelligent even without any sort of recursive self-improvement in software. Because you now have one model that can do every single job in the economy and humans can’t merge our minds in the same way. So do you expect some sort of intelligence explosion from broad deployment?

Ilya Sutskever 00:52:30

I think that it is likely that we will have rapid economic growth. I think with broad deployment, there are two arguments you could make which are conflicting. One is that once indeed you get to a point where you have an AI that can learn to do things quickly and you have many of them, then there will be a strong force to deploy them in the economy unless there will be some kind of a regulation that stops it, which by the way there might be.

But the idea of very rapid economic growth for some time, I think it’s very possible from broad deployment. The question is how rapid it’s going to be. I think this is hard to know because on the one hand you have this very efficient worker. On the other hand, the world is just really big and there’s a lot of stuff, and that stuff moves at a different speed. But then on the other hand, now the AI could… So I think very rapid economic growth is possible. We will see all kinds of things like different countries with different rules and the ones which have the friendlier rules, the economic growth will be faster. Hard to predict.

00:55:07 – Alignment

Dwarkesh Patel 00:55:07

It seems to me that this is a very precarious situation to be in. In the limit, we know that this should be possible. If you have something that is as good as a human at learning, but which can merge its brains—merge different instances in a way that humans can’t merge—already, this seems like a thing that should physically be possible. Humans are possible, digital computers are possible. You just need both of those combined to produce this thing.

It also seems this kind of thing is extremely powerful. Economic growth is one way to put it. A Dyson sphere is a lot of economic growth. But another way to put it is that you will have, in potentially a very short period of time... You hire people at SSI, and in six months, they’re net productive, probably. A human learns really fast, and this thing is becoming smarter and smarter very fast. How do you think about making that go well? Why is SSI positioned to do that well? What is SSI’s plan there, is basically what I’m trying to ask.

Ilya Sutskever 00:56:10

One of the ways in which my thinking has been changing is that I now place more importance on AI being deployed incrementally and in advance. One very difficult thing about AI is that we are talking about systems that don’t yet exist and it’s hard to imagine them.

I think that one of the things that’s happening is that in practice, it’s very hard to feel the AGI. It’s very hard to feel the AGI. We can talk about it, but imagine having a conversation about how it is like to be old when you’re old and frail. You can have a conversation, you can try to imagine it, but it’s just hard, and you come back to reality where that’s not the case. I think that a lot of the issues around AGI and its future power stem from the fact that it’s very difficult to imagine. Future AI is going to be different. It’s going to be powerful. Indeed, the whole problem, what is the problem of AI and AGI? The whole problem is the power. The whole problem is the power.

When the power is really big, what’s going to happen? One of the ways in which I’ve changed my mind over the past year—and that change of mind, I’ll hedge a little bit, may back-propagate into the plans of our company—is that if it’s hard to imagine, what do you do? You’ve got to be showing the thing. You’ve got to be showing the thing. I maintain that most people who work on AI also can’t imagine it because it’s too different from what people see on a day-to-day basis.

I do maintain, here’s something which I predict will happen. This is a prediction. I maintain that as AI becomes more powerful, people will change their behaviors. We will see all kinds of unprecedented things which are not happening right now. I’ll give some examples. I think for better or worse, the frontier companies will play a very important role in what happens, as will the government. The kind of things that I think you’ll see, which you see the beginnings of, are companies that are fierce competitors starting to collaborate on AI safety. You may have seen OpenAI and Anthropic doing a first small step, but that did not exist. That’s something which I predicted in one of my talks about three years ago, that such a thing will happen. I also maintain that as AI continues to become more powerful, more visibly powerful, there will also be a desire from governments and the public to do something. I think this is a very important force, of showing the AI.

That’s number one. Number two, okay, so the AI is being built. What needs to be done? One thing that I maintain that will happen is that right now, people who are working on AI, I maintain that the AI doesn’t feel powerful because of its mistakes. I do think that at some point the AI will start to feel powerful actually. I think when that happens, we will see a big change in the way all AI companies approach safety. They’ll become much more paranoid. I say this as a prediction that we will see happen. We’ll see if I’m right. But I think this is something that will happen because they will see the AI becoming more powerful. Everything that’s happening right now, I maintain, is because people look at today’s AI and it’s hard to imagine the future AI.

There is a third thing which needs to happen. I’m talking about it in broader terms, not just from the perspective of SSI because you asked me about our company. The question is, what should the companies aspire to build? What should they aspire to build? There has been one big idea that everyone has been locked into, which is the self-improving AI. Why did it happen? Because there are fewer ideas than companies. But I maintain that there is something that’s better to build, and I think that everyone will want that.

It’s the AI that’s robustly aligned to care about sentient life specifically. I think in particular, there’s a case to be made that it will be easier to build an AI that cares about sentient life than an AI that cares about human life alone, because the AI itself will be sentient. And if you think about things like mirror neurons and human empathy for animals, which you might argue it’s not big enough, but it exists. I think it’s an emergent property from the fact that we model others with the same circuit that we use to model ourselves, because that’s the most efficient thing to do.

Dwarkesh Patel 01:02:06

So even if you got an AI to care about sentient beings—and it’s not actually clear to me that that’s what you should try to do if you solved alignment—it would still be the case that most sentient beings will be AIs. There will be trillions, eventually quadrillions, of AIs. Humans will be a very small fraction of sentient beings. So it’s not clear to me if the goal is some kind of human control over this future civilization, that this is the best criterion.

Ilya Sutskever 01:02:37

It’s true. It’s possible it’s not the best criterion. I’ll say two things. Number one, care for sentient life, I think there is merit to it. It should be considered. I think it would be helpful if there was some kind of short list of ideas that the companies, when they are in this situation, could use. That’s number two.

Number three, I think it would be really materially helpful if the power of the most powerful superintelligence was somehow capped because it would address a lot of these concerns. The question of how to do it, I’m not sure, but I think that would be materially helpful when you’re talking about really, really powerful systems.

Dwarkesh Patel 01:03:35

Before we continue the alignment discussion, I want to double-click on that. How much room is there at the top? How do you think about superintelligence? Do you think, using this learning efficiency idea, maybe it is just extremely fast at learning new skills or new knowledge? Does it just have a bigger pool of strategies? Is there a single cohesive “it” in the center that’s more powerful or bigger? If so, do you imagine that this will be sort of godlike in comparison to the rest of human civilization, or does it just feel like another agent, or another cluster of agents?

Ilya Sutskever 01:04:10

This is an area where different people have different intuitions. I think it will be very powerful, for sure. What I think is most likely to happen is that there will be multiple such AIs being created roughly at the same time. I think that if the cluster is big enough—like if the cluster is literally continent-sized—that thing could be really powerful, indeed. If you literally have a continent-sized cluster, those AIs can be very powerful. All I can tell you is that if you’re talking about extremely powerful AIs, truly dramatically powerful, it would be nice if they could be restrained in some ways or if there were some kind of agreement or something.

What is the concern of superintelligence? What is one way to explain the concern? If you imagine a system that is sufficiently powerful, really sufficiently powerful—and you could say you need to do something sensible like care for sentient life in a very single-minded way—we might not like the results. That’s really what it is.

Maybe, by the way, the answer is that you do not build an RL agent in the usual sense. I’ll point several things out. I think human beings are semi-RL agents. We pursue a reward, and then the emotions or whatever make us tire out of the reward and we pursue a different reward. The market is a very short-sighted kind of agent. Evolution is the same. Evolution is very intelligent in some ways, but very dumb in other ways. The government has been designed to be a never-ending fight between three parts, which has an effect. So I think things like this.

Another thing that makes this discussion difficult is that we are talking about systems that don’t exist, that we don’t know how to build. That’s the other thing and that’s actually my belief. I think what people are doing right now will go some distance and then peter out. It will continue to improve, but it will also not be “it”. The “It” we don’t know how to build, and a lot hinges on understanding reliable generalization.

I’ll say another thing. One of the things that you could say about what causes alignment to be difficult is that your ability to learn human values is fragile. Then your ability to optimize them is fragile. You actually learn to optimize them. And can’t you say, “Are these not all instances of unreliable generalization?” Why is it that human beings appear to generalize so much better? What if generalization was much better? What would happen in this case? What would be the effect? But those questions are right now still unanswerable.

Dwarkesh Patel 01:07:21

How does one think about what AI going well looks like? You’ve scoped out how AI might evolve. We’ll have these sort of continual learning agents. AI will be very powerful. Maybe there will be many different AIs. How do you think about lots of continent-sized compute intelligences going around? How dangerous is that? How do we make that less dangerous? And how do we do that in a way that protects an equilibrium where there might be misaligned AIs out there and bad actors out there?

Ilya Sutskever 01:07:58

Here’s one reason why I liked “AI that cares for sentient life”. We can debate on whether it’s good or bad. But if the first N of these dramatic systems do care for, love, humanity or something, care for sentient life, obviously this also needs to be achieved. This needs to be achieved. So if this is achieved by the first N of those systems, then I can see it go well, at least for quite some time.

Then there is the question of what happens in the long run. How do you achieve a long-run equilibrium? I think that there, there is an answer as well. I don’t like this answer, but it needs to be considered.

In the long run, you might say, “Okay, if you have a world where powerful AIs exist, in the short term, you could say you have universal high income. You have universal high income and we’re all doing well.” But what do the Buddhists say? “Change is the only constant.” Things change. There is some kind of government, political structure thing, and it changes because these things have a shelf life. Some new government thing comes up and it functions, and then after some time it stops functioning. That’s something that we see happening all the time.

So I think for the long-run equilibrium, one approach is that you could say maybe every person will have an AI that will do their bidding, and that’s good. If that could be maintained indefinitely, that’s true. But the downside with that is then the AI goes and earns money for the person and advocates for their needs in the political sphere, and maybe then writes a little report saying, “Okay, here’s what I’ve done, here’s the situation,” and the person says, “Great, keep it up.” But the person is no longer a participant. Then you can say that’s a precarious place to be in.

I’m going to preface by saying I don’t like this solution, but it is a solution. The solution is if people become part-AI with some kind of Neuralink++. Because what will happen as a result is that now the AI understands something, and we understand it too, because now the understanding is transmitted wholesale. So now if the AI is in some situation, you are involved in that situation yourself fully. I think this is the answer to the equilibrium.

Dwarkesh Patel 01:10:47

I wonder if the fact that emotions which were developed millions—or in many cases, billions—of years ago in a totally different environment are still guiding our actions so strongly is an example of alignment success.

To spell out what I mean—I don’t know whether it’s more accurate to call it a value function or reward function—but the brainstem has a directive where it’s saying, “Mate with somebody who’s more successful.” The cortex is the part that understands what success means in the modern context. But the brainstem is able to align the cortex and say, “However you recognize success to be—and I’m not smart enough to understand what that is— you’re still going to pursue this directive.”

Ilya Sutskever 01:11:36

I think there’s a more general point. I think it’s actually really mysterious how evolution encodes high-level desires. It’s pretty easy to understand how evolution would endow us with the desire for food that smells good because smell is a chemical, so just pursue that chemical. It’s very easy to imagine evolution doing that thing.

But evolution also has endowed us with all these social desires. We really care about being seen positively by society. We care about being in good standing. All these social intuitions that we have, I feel strongly that they’re baked in. I don’t know how evolution did it because it’s a high-level concept that’s represented in the brain.

Let’s say you care about some social thing, it’s not a low-level signal like smell. It’s not something for which there is a sensor. The brain needs to do a lot of processing to piece together lots of bits of information to understand what’s going on socially. Somehow evolution said, “That’s what you should care about.” How did it do it?

It did it quickly, too. All these sophisticated social things that we care about, I think they evolved pretty recently. Evolution had an easy time hard-coding this high-level desire. I’m unaware of a good hypothesis for how it’s done. I had some ideas I was kicking around, but none of them are satisfying.

Dwarkesh Patel 01:13:26

What’s especially impressive is it was desire that you learned in your lifetime, it makes sense because your brain is intelligent. It makes sense why you would be able to learn intelligent desires. Maybe this is not your point, but one way to understand it is that the desire is built into the genome, and the genome is not intelligent. But you’re somehow able to describe this feature. It’s not even clear how you define that feature, and you can build it into the genes.

Ilya Sutskever 01:13:55

Essentially, or maybe I’ll put it differently. If you think about the tools that are available to the genome, it says, “Okay, here’s a recipe for building a brain.” You could say, “Here is a recipe for connecting the dopamine neurons to the smell sensor.” And if the smell is a certain kind of good smell, you want to eat that.

I could imagine the genome doing that. I’m claiming that it is harder to imagine. It’s harder to imagine the genome saying you should care about some complicated computation that your entire brain, a big chunk of your brain, does. That’s all I’m claiming. I can tell you a speculation of how it could be done. Let me offer a speculation, and I’ll explain why the speculation is probably false.

So the brain has brain regions. We have our cortex. It has all those brain regions. The cortex is uniform, but the brain regions and the neurons in the cortex kind of speak to their neighbors mostly. That explains why you get brain regions. Because if you want to do some kind of speech processing, all the neurons that do speech need to talk to each other. And because neurons can only speak to their nearby neighbors, for the most part, it has to be a region.

All the regions are mostly located in the same place from person to person. So maybe evolution hard-coded literally a location on the brain. So it says, “Oh, when the GPS coordinates of the brain such and such, when that fires, that’s what you should care about.” Maybe that’s what evolution did because that would be within the toolkit of evolution.

Dwarkesh Patel 01:15:35

Yeah, although there are examples where, for example, people who are born blind have that area of their cortex adopted by another sense. I have no idea, but I’d be surprised if the desires or the reward functions which require a visual signal no longer worked for people who have their different areas of their cortex co-opted.

For example, if you no longer have vision, can you still feel the sense that I want people around me to like me and so forth, which usually there are also visual cues for.

Ilya Sutskever 01:16:12

I fully agree with that. I think there’s an even stronger counterargument to this theory. There are people who get half of their brains removed in childhood, and they still have all their brain regions. But they all somehow move to just one hemisphere, which suggests that the brain regions, their location is not fixed and so that theory is not true.

It would have been cool if it was true, but it’s not. So I think that’s a mystery. But it’s an interesting mystery. The fact is that somehow evolution was able to endow us to care about social stuff very, very reliably. Even people who have all kinds of strange mental conditions and deficiencies and emotional problems tend to care about this also.

01:18:13 – “We are squarely an age of research company”

Dwarkesh Patel 01:18:13

What is SSI planning on doing differently? Presumably your plan is to be one of the frontier companies when this time arrives. Presumably you started SSI because you’re like, “I think I have a way of approaching how to do this safely in a way that the other companies don’t.” What is that difference?

Ilya Sutskever 01:18:36

The way I would describe it is that there are some ideas that I think are promising and I want to investigate them and see if they are indeed promising or not. It’s really that simple. It’s an attempt. If the ideas turn out to be correct—these ideas that we discussed around understanding generalization—then I think we will have something worthy.

Will they turn out to be correct? We are doing research. We are squarely an “age of research” company. We are making progress. We’ve actually made quite good progress over the past year, but we need to keep making more progress, more research. That’s how I see it. I see it as an attempt to be a voice and a participant.

Dwarkesh Patel 01:19:29

Your cofounder and previous CEO left to go to Meta recently, and people have asked, “Well, if there were a lot of breakthroughs being made, that seems like a thing that should have been unlikely.” I wonder how you respond.

Ilya Sutskever 01:19:45

For this, I will simply remind a few facts that may have been forgotten. I think these facts which provide the context explain the situation. The context was that we were fundraising at a $32 billion valuation, and then Meta came in and offered to acquire us, and I said no. But my former cofounder in some sense said yes. As a result, he also was able to enjoy a lot of near-term liquidity, and he was the only person from SSI to join Meta.

Dwarkesh Patel 01:20:27

It sounds like SSI’s plan is to be a company that is at the frontier when you get to this very important period in human history where you have superhuman intelligence. You have these ideas about how to make superhuman intelligence go well. But other companies will be trying their own ideas. What distinguishes SSI’s approach to making superintelligence go well?

Ilya Sutskever 01:20:49

The main thing that distinguishes SSI is its technical approach. We have a different technical approach that I think is worthy and we are pursuing it.

I maintain that in the end there will be a convergence of strategies. I think there will be a convergence of strategies where at some point, as AI becomes more powerful, it’s going to become more or less clearer to everyone what the strategy should be. It should be something like, you need to find some way to talk to each other and you want your first actual real superintelligent AI to be aligned and somehow care for sentient life, care for people, democratic, one of those, some combination thereof.

I think this is the condition that everyone should strive for. That’s what SSI is striving for. I think that this time, if not already, all the other companies will realize that they’re striving towards the same thing. We’ll see. I think that the world will truly change as AI becomes more powerful. I think things will be really different and people will be acting really differently.

Dwarkesh Patel 01:22:14

Speaking of forecasts, what are your forecasts to this system you’re describing, which can learn as well as a human and subsequently, as a result, become superhuman?

Ilya Sutskever 01:22:26

I think like 5 to 20.

Dwarkesh Patel 01:22:28

5 to 20 years?

Ilya Sutskever 01:22:29

Mhm.

Dwarkesh Patel 01:22:30

I just want to unroll how you might see the world coming. It’s like, we have a couple more years where these other companies are continuing the current approach and it stalls out. “Stalls out” here meaning they earn no more than low hundreds of billions in revenue? How do you think about what stalling out means?

Ilya Sutskever 01:22:49

I think stalling out will look like…it will all look very similar among all the different companies. It could be something like this. I’m not sure because I think even with stalling out, I think these companies could make a stupendous revenue. Maybe not profits because they will need to work hard to differentiate each other from themselves, but revenue definitely.

Dwarkesh Patel 01:23:20

But something in your model implies that when the correct solution does emerge, there will be convergence between all the companies. I’m curious why you think that’s the case.

Ilya Sutskever 01:23:32

I was talking more about convergence on their alignment strategies. I think eventual convergence on the technical approach is probably going to happen as well, but I was alluding to convergence to the alignment strategies. What exactly is the thing that should be done?

Dwarkesh Patel 01:23:46

I just want to better understand how you see the future unrolling. Currently, we have these different companies, and you expect their approach to continue generating revenue but not get to this human-like learner. So now we have these different forks of companies. We have you, we have Thinking Machines, there’s a bunch of other labs. Maybe one of them figures out the correct approach. But then the release of their product makes it clear to other people how to do this thing.

Ilya Sutskever 01:24:09

I think it won’t be clear how to do it, but it will be clear that something different is possible, and that is information. People will then be trying to figure out how that works. I do think though that one of the things not addressed here, not discussed, is that with each increase in the AI’s capabilities, I think there will be some kind of changes, but I don’t know exactly which ones, in how things are being done. I think it’s going to be important, yet I can’t spell out what that is exactly.

Dwarkesh Patel 01:24:49

By default, you would expect the company that has that model to be getting all these gains because they have the model that has the skills and knowledge that it’s building up in the world. What is the reason to think that the benefits of that would be widely distributed and not just end up at whatever model company gets this continuous learning loop going first?

Ilya Sutskever 01:25:13

Here is what I think is going to happen. Number one, let’s look at how things have gone so far with the AIs of the past. One company produced an advance and the other company scrambled and produced some similar things after some amount of time and they started to compete in the market and push the prices down. So I think from the market perspective, something similar will happen there as well.

We are talking about the good world, by the way. What’s the good world? It’s where we have these powerful human-like learners that are also… By the way, maybe there’s another thing we haven’t discussed on the spec of the superintelligent AI that I think is worth considering. It’s that you make it narrow, it can be useful and narrow at the same time. You can have lots of narrow superintelligent AIs.

But suppose you have many of them and you have some company that’s producing a lot of profits from it. Then you have another company that comes in and starts to compete. The way the competition is going to work is through specialization. Competition loves specialization. You see it in the market, you see it in evolution as well. You’re going to have lots of different niches and you’re going to have lots of different companies who are occupying different niches. In this world we might say one AI company is really quite a bit better at some area of really complicated economic activity and a different company is better at another area. And the third company is really good at litigation.

Dwarkesh Patel 01:27:18

Isn’t this contradicted by what human-like learning implies? It’s that it can learn…

Ilya Sutskever 01:27:21

It can, but you have accumulated learning. You have a big investment. You spent a lot of compute to become really, really good, really phenomenal at this thing. Someone else spent a huge amount of compute and a huge amount of experience to get really good at some other thing. You apply a lot of human learning to get there, but now you are at this high point where someone else would say, “Look, I don’t want to start learning what you’ve learned.”

Dwarkesh Patel 01:27:48

I guess that would require many different companies to begin at the human-like continual learning agent at the same time so that they can start their different tree search in different branches. But if one company gets that agent first, or gets that learner first, it does then seem like… Well, if you just think about every single job in the economy, having an instance learning each one seems tractable for a company.

Ilya Sutskever 01:28:19

That’s a valid argument. My strong intuition is that it’s not how it’s going to go. The argument says it will go this way, but my strong intuition is that it will not go this way. In theory, there is no difference between theory and practice. In practice, there is. I think that’s going to be one of those.

Dwarkesh Patel 01:28:41

A lot of people’s models of recursive self-improvement literally, explicitly state we will have a million Ilyas in a server that are coming up with different ideas, and this will lead to a superintelligence emerging very fast.

Do you have some intuition about how parallelizable the thing you are doing is? What are the gains from making copies of Ilya?

Ilya Sutskever 01:29:02

I don’t know. I think there’ll definitely be diminishing returns because you want people who think differently rather than the same. If there were literal copies of me, I’m not sure how much more incremental value you’d get. People who think differently, that’s what you want.

01:29:23 – Self-play and multi-agent

Dwarkesh Patel 01:29:23

Why is it that if you look at different models, even released by totally different companies trained on potentially non-overlapping datasets, it’s actually crazy how similar LLMs are to each other?

Ilya Sutskever 01:29:38

Maybe the datasets are not as non-overlapping as it seems.

Dwarkesh Patel 01:29:41

But there’s some sense in which even if an individual human might be less productive than the future AI, maybe there’s something to the fact that human teams have more diversity than teams of AIs might have. How do we elicit meaningful diversity among AIs? I think just raising the temperature just results in gibberish. You want something more like different scientists have different prejudices or different ideas. How do you get that kind of diversity among AI agents?

Ilya Sutskever 01:30:06

So the reason there has been no diversity, I believe, is because of pre-training. All the pre-trained models are pretty much the same because they pre-train on the same data. Now RL and post-training is where some differentiation starts to emerge because different people come up with different RL training.

Dwarkesh Patel 01:30:26

I’ve heard you hint in the past about self-play as a way to either get data or match agents to other agents of equivalent intelligence to kick off learning. How should we think about why there are no public proposals of this kind of thing working with LLMs?

Ilya Sutskever 01:30:49

I would say there are two things to say. The reason why I thought self-play was interesting is because it offered a way to create models using compute only, without data. If you think that data is the ultimate bottleneck, then using compute only is very interesting. So that’s what makes it interesting.

The thing is that self-play, at least the way it was done in the past—when you have agents which somehow compete with each other—it’s only good for developing a certain set of skills. It is too narrow. It’s only good for negotiation, conflict, certain social skills, strategizing, that kind of stuff. If you care about those skills, then self-play will be useful.

Actually, I think that self-play did find a home, but just in a different form. So things like debate, prover-verifier, you have some kind of an LLM-as-a-Judge which is also incentivized to find mistakes in your work. You could say this is not exactly self-play, but this is a related adversarial setup that people are doing, I believe.

Really self-play is a special case of more general competition between agents. The natural response to competition is to try to be different. So if you were to put multiple agents together and you tell them, “You all need to work on some problem and you are an agent and you’re inspecting what everyone else is working,” they’re going to say, “Well, if they’re already taking this approach, it’s not clear I should pursue it. I should pursue something differentiated.” So I think something like this could also create an incentive for a diversity of approaches.

01:32:42 – Research taste

Dwarkesh Patel 01:32:42

Final question: What is research taste? You’re obviously the person in the world who is considered to have the best taste in doing research in AI. You were the co-author on the biggest things that have happened in the history of deep learning, from AlexNet to GPT-3 to so on. What is it, how do you characterize how you come up with these ideas?

Ilya Sutskever 01:33:14

I can comment on this for myself. I think different people do it differently. One thing that guides me personally is an aesthetic of how AI should be, by thinking about how people are, but thinking correctly. It’s very easy to think about how people are incorrectly, but what does it mean to think about people correctly?

I’ll give you some examples. The idea of the artificial neuron is directly inspired by the brain, and it’s a great idea. Why? Because you say the brain has all these different organs, it has the folds, but the folds probably don’t matter. Why do we think that the neurons matter? Because there are many of them. It kind of feels right, so you want the neuron. You want some local learning rule that will change the connections between the neurons. It feels plausible that the brain does it.

The idea of the distributed representation. The idea that the brain responds to experience therefore our neural net should learn from experience. The brain learns from experience, the neural net should learn from experience. You kind of ask yourself, is something fundamental or not fundamental? How things should be.

I think that’s been guiding me a fair bit, thinking from multiple angles and looking for almost beauty, beauty and simplicity. Ugliness, there’s no room for ugliness. It’s beauty, simplicity, elegance, correct inspiration from the brain. All of those things need to be present at the same time. The more they are present, the more confident you can be in a top-down belief.

The top-down belief is the thing that sustains you when the experiments contradict you. Because if you trust the data all the time, well sometimes you can be doing the correct thing but there’s a bug. But you don’t know that there is a bug. How can you tell that there is a bug? How do you know if you should keep debugging or you conclude it’s the wrong direction? It’s the top-down. You can say things have to be this way. Something like this has to work, therefore we’ve got to keep going. That’s the top-down, and it’s based on this multifaceted beauty and inspiration by the brain.

Dwarkesh Patel 01:35:31

Alright, we’ll leave it there.

Ilya Sutskever 01:35:33

Thank you so much.

Dwarkesh Patel 01:35:34

Ilya, thank you so much.

Ilya Sutskever 01:35:36

Alright. Appreciate it.

Dwarkesh Patel 01:35:37

That was great.

Ilya Sutskever 01:35:38

Yeah, I enjoyed it.

Dwarkesh Patel 01:35:39

Yes, me too.

Recently, people have been talking about how it takes way more FLOPs to get a single sample in RL than it does in supervised learning. In pretraining, you get a signal on every single token you train on. In RL, you have to unroll a whole thinking trajectory that’s 10s of 1000s of tokens long in order to get a single reward signal at the end (for example, did the unit test for my code pass/did I get the right answer to this math problem/etc).

But this is only half the problem. Here’s a simple way to compare the learning efficiency of reinforcement learning versus supervised learning:

Bits/FLOP = Samples/Flop * Bits/Sample.

What I haven’t heard people talk about is the other term in our equation: Bits/Sample. And for most of training, the information density per sample is way way lower for RL.

Subscribe now

Putting things in plain English

In supervised learning (aka pretraining), you’re just soaking up bits. Every token is a hint at the structure of language, and the mind crafting that language, and the world that mind is seeing. Early in training, when you have a totally random model, you’re just maximally uncertain over all of this content. So each token is just blowing your mind. And you’re getting this exact signal of how wrong you were about the right answer, and what parameters you need to update to be less wrong.

Suppose you start with a randomly initialized model, and you kickstart training. If you’re doing next-token-prediction using supervised learning on “The sky is”, the training loop goes, “It’s actually ‘blue’. You said the probability of ‘blue’ is .001%. Make the connections that were suggesting ‘blue’ way way stronger. Alright, next token.”

In RL with policy gradient, you upweight all the trajectories where you get the answer right, and downweight all the trajectories where you get the answer wrong. But a model that’s not already very smart is just astonishingly unlikely to get the answer right.

If you were doing next-token-prediction on “The sky is” with RL, the training loop would be something like, “Okay, ‘halcyon’ is wrong. Don’t do the thing that led to saying ‘halycon’ … Okay ‘serendipity’ is wrong …” Rinse and repeat this guesswork for somewhere around the number of tokens you have in your vocabulary (on the order of 100,000).

The details

Let’s think about how maximum bits/sample change as the pass rate (p) changes. Pass rate here means how likely you are to say the correct answer. To keep this simple, let’s say the answer is a token long. Then the pass rate when you have a totally untrained model is just 1/ (size of your vocabulary).

In supervised learning, you get told exactly what the right label is for each sample. The amount of new information you learn corresponds to how surprised you are to learn the correct answer - the lower your pass rate (aka prior probability of the correct answer), the more you learned from seeing the correct label. The basic formula for entropy tells us that you can learn -log(p) bits/sample from supervised learning.

In RL, you only get told whether you got the right answer or not. The amount of new information you can extract is bounded by how uncertain you are about this binary outcome. If you almost always pass (p ≈ 1) or almost always fail (p ≈ 0), each trial is very unlikely to surprise you. You’ll learn most when the probability of passing is like a coin toss (p ≈ 0.5). The basic formula for the information content of a binary random variable tells us that you can learn at most Entropy(p) = -p log(p) - (1-p) log(1-p)1 bits/sample from RL.

Okay let’s plot this.

Doesn’t look terrible. Yes, pretraining is much better for half of the pass rate range, but then RL is better for the other half. However, this graph is super misleading. Because what the power law (in scaling laws) implies is that you need an equivalent amount of compute to cross each order of magnitude improvement in the pass rate. If it took you X many FLOPs to go from 1/100,000 pass rate to 1/10,000, then it will take you X many FLOPs to go from 1/10,000 pass rate to 1/1,000. So, we should actually chart the pass rate on a log scale - again, to account for how each increment in the x-axis corresponds to the same number of FLOPs.

Oh boy, is that a sad picture. The regime where RL has comparable information density per sample to pre-training is this tiny slice at the very end of training, when you’ve got a pretty reasonable model anyways.

And again, I want to emphasize that this is totally separate from the point that getting a single sample from RL (aka unrolling a full trajectory before getting any signal) might take upwards of a million times more compute.

It’s even worse than this - variance

The situation for RL early in training is actually even worse than described above. When the pass rate is low, your gradient estimate is going to be incredibly noisy and unpredictable. Either you don’t sample the correct answer at all in your batch, in which you get almost no information. Or you do, and you get this giant spike. You’re getting jerked around, which is terrible for performant training.2

Interestingly, pretraining has the exact inverse problem. There, variance is super high at the END of training. As pretraining progresses, you exhaust more and more of the reducible loss (things your model can actually learn about the data). What remains is mostly the irreducible loss. The irreducible loss is the intrinsic unpredictability of internet text.

How should the prompt, “Bob’s favorite color is” end? Depends on Bob. There’s not some correct answer which your super smart model can actually get good at predicting. But your super smart model is still getting a gradient update on whatever random answer someone put on the internet. And this noise is drowning out the true signal that the couple of actually learnable tokens in the batch are giving you. I don’t know if this is accurate, but it seems like this explosion of variance at the end of pretraining is relevant to why batch sizes are increased as pretraining progresses.

Getting to the Goldilocks zone in RL

If RL works best in the regime where your pass rate is >>1%, then this raises the question, how can we construct the RL training to get (and keep) models in this learning flow state?

For example, we can think of pretraining AND inference scaling as increasing the pass rate during RL, allowing you to extract far more bits per sample.

It’s been noted that curriculum learning in not especially helpful for pretraining, but often essential for RL. This makes total sense when you think about how RL is only getting meaningful bits per sample in this Goldilocks zone of pass rate, so you really want to order the learning such that the difficulty of challenges increases in tandem with the model’s intelligence.

Our pass rate framework also gives us good intuitions for why self play has been so productive in the history of RL. If you’re competing against a player who is almost as good as you, you are balancing around a 50% pass rate, which peaks out the bits you get from a random binary variable.

But self play is not the only way we can imagine of keeping pass rate high through training. Perhaps we can come up with some proxy evaluation which is much more dense. Density here can mean one of two things:

1. Samples/FLOP density: You estimate the final reward using this proxy evaluation, but much earlier on in the episode, saving you the compute of unrolling the full trajectory. This is what a value function does.

2. Bits/Sample density: You come up with a proxy objective which is much easier to pass than the actual test under question. The simplest example I can think of is a process-reward model which says, “Hey, this rollout got the wrong answer, but I can see that its reasoning was on the right track at the start. So let’s up-weight those early tokens.”

Section 4.2 of the Deepseek R1 paper why so far, it’s been hard to develop useful proxy objectives like this for LLMs.

Fewer bits, sure, but very valuable bits

To be fair to RL, while you may be learning far fewer Bits/FLOP in RL, the bits you learn are very important. They are not apples-to-apples comparable to the bits in pretraining. This is for two key reasons:

1. Pre-training is teaching you what the data manifold of the internet looks like, which is only partially and indirectly related to, “How do I perform economically valuable tasks?” Whereas RL has the promise of giving you the good stuff directly.

2. Even if the pre-training corpus contains the instructions about how to accomplish a specific task, it does not have the thinking trace which teaches the model about how to correct its mistakes, or leverage its jagged and non-human repertoire of skills to accomplish the task.

The rebuttal is that those bits are only available for a small fraction of the pass rate range (again, weighted on a log scale to account for how pass rate is trash for most of training).

By the way, now we can understand all these claims about how RLVR is only eliciting the capabilities already latent in the pretrained model. Of course that’s the case. If the pretrained model didn’t have a high enough pass rate to begin with, then RL would have atrocious bits/sample, and thus not be able to learn at all. Move 37 is obviously one famous example where RL did teach a model a de-novo strategy. It’s worth noting that AlphaGo was trained on self play (see above re how self play increases pass rate), and that AlphaGo was surprisingly compute intensive for its time.

The jaggedness of RL

People have pointed out that RLVR is empirically just leading models to associate a thought pattern to a problem type rather than instilling a more general policy of stepping back and thinking through the best approach.

Think about it. How is it possible that we have models which are world-class at coding competitions but at the same time leave extremely foreseeable bugs and technical debt all throughout the codebase?

What explains this weird jaggedness? Perhaps RLVR can’t distinguish trajectories that were generated from a more generalizable procedure vs just greedily matching the problem shape to some associated thought process.

When you’re doing policy gradient rollouts, this more complex general policy is extremely unlikely to be ever be sampled, whereas the simple heuristic policy does get sampled and grows in frequency until it reaches fixation. Meanwhile, the general policy recedes further and further from sight.

Then the question is, how do we build a short bridge between simple heuristic solutions and the more complex general strategy? And will that bridge just spontaneously emerge as time horizons expand, thus potentially requiring generalization?

My concern is that this general policy of stepping back and making tasteful judgements based on your understanding of the world will continue to be hard to spot-light using verifiable rewards, even on longer time horizon tasks. And so the solution to this jaggedness will require a more robust training procedure, not just scaling RLVR.

Human learning

Here we’re only talking about the bits/sample learned from model free RL - aka from some binary outcome at the end of an episode. But of course humans are obviously learning way more efficiently than this. Think about a repeat entrepreneur. We say that she has a ton of hard-won wisdom and experience. Very little of that learning comes from the one bit of outcome from her previous episode (whether the startup succeeded or not).

It’s not clear what the ML analog is for human learning from experience. Clearly, our observations and reflections update our world model (independent of the outcome at the end). And this is playing a very important role in our learning.

Maybe we shouldn’t be asking how we model free RL to ≈50% pass rate, so that can squeeze out a full drop of information from the outcome. Maybe we should be asking, how do humans wring out the buckets of information from the environment?

Share

1

Basically, this equation is saying, Information learned from a binary outcome = p(sample is correct) * (information gained when sample is correct) + p(sample is incorrect) * (information gained when sample is incorrect).

2

Thank you to Lukas Berglund for spotting that my previous exposition on this point was incorrect.

As part of this interview, Satya Nadella gave Dylan Patel (founder of SemiAnalysis) and me an exclusive first-look at their brand-new Fairwater 2 datacenter.

Microsoft is building multiple Fairwaters, each of which has hundreds of thousands of GB200s & GB300s. Between all these interconnected buildings, they’ll have over 2 GW of total capacity. Just to give a frame of reference, even a single one of these Fairwater buildings is more powerful than any other AI datacenter that currently exists.

Satya then answered a bunch of questions about how Microsoft is preparing for AGI across all layers of the stack.

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Labelbox produces high-quality data at massive scale, powering any capability you want your model to have. Whether you’re building a voice agent, a coding assistant, or a robotics model, Labelbox gets you the exact data you need, fast. Reach out at labelbox.com/dwarkesh

• CodeRabbit automatically reviews and summarizes PRs so you can understand changes and catch bugs in half the time. This is helpful whether you’re coding solo, collaborating with agents, or leading a full team. To learn how CodeRabbit integrates directly into your workflow, go to coderabbit.ai/dwarkesh

To sponsor a future episode, visit dwarkesh.com/advertise.

Timestamps

(00:00:00) - Fairwater 2

(00:03:20) - Business models for AGI

(00:12:48) - Copilot

(00:20:02) - Whose margins will expand most?

(00:36:17) - MAI

(00:47:47) - The hyperscale business

(01:02:44) - In-house chip & OpenAI partnership

(01:09:35) - The CAPEX explosion

(01:15:07) - Will the world trust US companies to lead AI?

Transcript00:00:00 - Fairwater 2

Dwarkesh Patel 00:00:00

Today we are interviewing Satya Nadella. “We” being me and Dylan Patel, who is founder of SemiAnalysis. Satya, welcome.

Satya Nadella 00:00:07

Thank you. It’s great. Thanks for coming over to Atlanta.

Dwarkesh Patel 00:00:10

Thank you for giving us the tour of the new facility. It’s been really cool to see.

Satya Nadella 00:00:13

Absolutely.

Dwarkesh Patel 00:00:15

Satya and Scott Guthrie, Microsoft’s EVP of Cloud and AI, give us a tour of their brand new Fairwater 2 data center, the current most powerful in the world.

Scott Guthrie 00:00:25

We’ve tried to 10x the training capacity every 18 to 24 months. So this would effectively be a 10x increase from what GPT-5 was trained with. So to put it in perspective in the number of optics, the network optics in this building is almost as much as all of Azure across all our data centers two and a half years ago.

Satya Nadella 00:00:44

It’s got like five million network connections.

Dwarkesh Patel 00:00:47

You’ve got all this bandwidth between different sites in a region and between the two regions. So is this like a big bet on scaling in the future, that you anticipate in the future that there’s going to be some huge model that will require two whole different regions to train?

Satya Nadella 00:00:59

The goal is to be able to aggregate these flops for a large training job and then put these things together across sites. The reality is you’ll use it for training and then you’ll use it for data gen, you’ll use it for inference in all sorts of ways. It’s not like it’s going to be used only for one workload forever.

Scott Guthrie 00:01:20

Fairwater 4, which you’re going to see under construction nearby, will also be on that one petabit network so that we can actually link the two at a very high rate. Then we do the AI WAN connecting to Milwaukee where we have multiple other Fairwaters being built.

Satya Nadella 00:01:35

Literally you can see the model parallelism and the data parallelism. It’s kind of built for, essentially, the training jobs, the super pods across this campus. And then with the WAN, you can go to the Wisconsin data center. You literally run a training job with all of them getting aggregated.

Scott Guthrie 00:02:00

What we’re seeing right here is a cell with no servers in it yet, no racks.

Dylan Patel 00:02:04

How many racks are in a cell?

Scott Guthrie 00:02:06

We don’t necessarily share that per se, but…

Dylan Patel 00:02:10

That’s the reason I ask.

Scott Guthrie 00:02:12

You’ll see upstairs.

Dylan Patel 00:02:14

I’ll start counting.

Scott Guthrie 00:02:15

You can start counting. We’ll let you start counting.

Dylan Patel 00:02:16

How many cells are there in this building?

Scott Guthrie 00:02:17

That part also I can’t tell you.

Dwarkesh Patel 00:02:19

Well, division is easy, right?

Satya Nadella 00:02:24

My God, it’s kind of loud.

Dwarkesh Patel 00:02:27

Are you looking at this like, “Now I see where my money is going.”

Satya Nadella 00:02:30

It’s like, “I run a software company. Welcome to the software company.”

Dwarkesh Patel 00:02:35

How big is the design space once you’ve decided to use the GB200s and the NVLink? How many other decisions are there to be made?

Satya Nadella 00:02:41

There is coupling from the model architecture to what is the physical plan that’s optimized. And it’s also scary in that sense, which is, there’s going to be a new chip that’ll come out. Take Vera Rubin Ultra. That’s going to have power density that’s going to be so different, with cooling requirements that are going to be so different. So you kind of don’t want to just build all to one spec. That goes back a little bit to the dialogue we’ll have, which is that you want to be scaling in time as opposed to scale once and then be stuck with it.

00:03:20 - Business models for AGI

Dylan Patel 00:03:20

When you look at all the past technological transitions—whether it be railroads or the Internet or replaceable parts, industrialization, the cloud, all of these things—each revolution has gotten much faster in the time it goes from technology discovered to ramp and pervasiveness through the economy. Many folks who have been on Dwarkesh’s podcast believe this is the final technological revolution or transition, and that this time is very, very different.

At least so far in the markets, in three years we’ve already skyrocketed to hyperscalers doing $500 billion of capex next year, which is a scale that’s unmatched to prior revolutions in terms of speed. The end state seems to be quite different. Your framing of this seems quite different from what I would call the “AI bro” who’s like, “AGI is coming.” I’d like to understand that more.

Satya Nadella 00:04:17

I start with the excitement that I also feel for the idea that maybe after the Industrial Revolution this is the biggest thing. I start with that premise. But at the same time, I’m a little grounded in the fact that this is still early innings. We’ve built some very useful things, we’re seeing some great properties, these scaling laws seem to be working. I’m optimistic that they’ll continue to work. Some of it does require real science breakthroughs, but it’s also a lot of engineering and what have you.

That said, I also sort of take the view that even what has been happening in the last 70 years of computing has also been a march that has helped us move. I like one of the things that Raj Reddy has as a metaphor for what AI is. He’s a Turing Award winner at CMU. He had this, even pre-AGI. He had this metaphor for AI, it should either be a guardian angel or a cognitive amplifier. I love that. It’s a simple way to think about what this is. Ultimately, what is its human utility? It is going to be a cognitive amplifier and a guardian angel. If I view it that way, I view it as a tool.

But then you can also go very mystical about it and say this is more than a tool. It does all these things, which only humans did before so far. But that has been the case with many technologies in the past. Only humans did a lot of things, and then we had tools that did them.

Dwarkesh Patel 00:05:56

We don’t have to get wrapped up in the definition here, but one way to think about it is, maybe it takes five years, ten years, twenty years. At some point, eventually a machine is producing Satya tokens, and the Microsoft board thinks that Satya tokens are worth a lot.

Dylan Patel 00:06:10

How much are you wasting of this economic value by interviewing Satya?

Dwarkesh Patel 00:06:15

I could not afford the API costs of Satya tokens. Whatever you want to call it, are the Satya tokens a tool or an agent, whatever. Right now, if you have models that cost on the order of dollars or cents per million tokens, there’s just an enormous room for margin expansion there, where a million tokens of Satya are worth a lot. Where does that margin go and what level of that margin is Microsoft involved in is the question I have.

Satya Nadella 00:06:45

In some sense this goes back again to, essentially, what’s the economic growth picture going to really look like? What’s the firm going to look like? What’s productivity going to look like? That to me is where, again, if the Industrial Revolution created… After 70 years of diffusion is when you started seeing the economic growth. That’s the other thing to remember. Even if the tech is diffusing fast this time around, for true economic growth to appear it has to diffuse to a point where the work, the work artifact, and the workflow has to change.  So that’s one place where I think the change management required for a corporation to truly change is something we shouldn’t discount.

Going forward, do humans and the tokens they produce get higher leverage, whether it’s the Dwarkesh or the Dylan tokens of the future? Think about the amount of technology. Would you be able to run SemiAnalysis or this podcast without technology? No chance, at the scale that you have been able to achieve, there’s no chance.

So the question is, what’s that scale? Is it going to be 10x’ed with something that comes through? Absolutely. Therefore, whether you’re ramped to some revenue number or you’re ramped to some audience number or what have you, that I think is what’s going to happen. The point is, what took 70 years, maybe 150 years for the Industrial Revolution, may happen in 20 years, 25 years. I would love to compress what happened in 200 years of the Industrial Revolution into a 20-year period, if we’re lucky.

Dylan Patel 00:08:28

Microsoft historically has been perhaps the greatest software company, the largest software-as-a-service company. You’ve gone through a transition in the past where you used to sell Windows licenses and disks of Windows or Microsoft, and now you sell subscriptions to 365.

As we go from that transition to where your business is today, there’s also a transition going on after that. Software-as-a-service has incredibly low incremental cost per user. There’s a lot of R&D, there’s a lot of customer acquisition costs. This is sort of why, not Microsoft, but the SaaS companies have underperformed massively in the markets, because the COGS of AI is just so high, and that just completely breaks how these business models work.

How do you, as perhaps the greatest software-as-a-service company, transition Microsoft to this new age where COGS matters a lot and the incremental cost per user is different? Because right now you’re charging like, “Hey, it’s 20 bucks for Copilot.”

Satya Nadella 00:09:29

It’s a great question because in some sense with the business models themselves, the levers are going to remain similar. If you look at the menu of models starting from consumer all the way, there will be some ad unit, there will be some transaction, there will be some device gross margin for somebody who builds an AI device. There will be subscriptions, consumer and enterprise, and then there’ll be consumption. So I still think those are all the meters.

To your point, what is a subscription? Up to now, people like subscriptions because they can budget for them. They are essentially entitlements to some consumption rights that come encapsulated in a subscription. So I think that in some sense becomes a pricing decision. How much consumption you are entitled to is, if you look at all the coding subscriptions, kind of what they are, right? Then you have the pro tier, the standard tier, and what have you. So I think that’s how the pricing and the margin structures will get tiered.

The interesting thing is that at Microsoft, the good news for us is we are in that business across all those meters. At a portfolio level, we pretty much have consumption, subscriptions, to all of the other consumer levers as well. I think time will tell which of these models make sense in what categories.

One thing on the SaaS side, since you brought it up, which I think a lot about. Take Office 365 or Microsoft 365. Having a low ARPU is great, because here’s an interesting thing. During the transition from server to cloud, one of the questions we used to ask ourselves is, “Oh my God, if all we did was just basically move the same users who were using our Office licenses and our Office servers at the time to the cloud, and we had COGS, this is going to not only shrink our margins but we’ll be fundamentally a less profitable company.”

Except what happened was the move to the cloud expanded the market like crazy. We sold a few servers in India, we didn’t sell much. Whereas in the cloud suddenly everybody in India also could afford fractionally buying servers, the IT cost. In fact, the biggest thing I had not realized, for example, was the amount of money people were spending buying storage underneath SharePoint. In fact, EMC’s biggest segment may have been storage servers for SharePoint. All that sort of dropped in the cloud because nobody had to go buy. In fact, it was working capital, meaning basically, it was cash flow out. So it expanded the market massively.

So this AI thing will be that. If you take coding, what we built with GitHub and VS Code over decades, suddenly the coding assistant is that big in one year. That I think is what’s going to happen as well, which is the market expands massively.

00:12:48 - Copilot

Dwarkesh Patel 00:12:48

There’s a question of, the market will expand, but will the parts of the revenue that touch Microsoft expand? Copilot is an example. If you look earlier this year, according to Dylan’s numbers, GitHub Copilot revenue was like $500 million or something like that and there were no close competitors. Whereas now you have Claude Code, Cursor, and Copilot with around similar revenue, around a billion. Codex is catching up around $700–800 million. So the question is, across all the surfaces that Microsoft has access to, what is the advantage that Microsoft’s equivalents of Copilot have?

Satya Nadella 00:13:25

By the way, I love this chart.

I love this chart for so many reasons. One is we’re still on the top. Second is all these companies that are listed here are all companies that have been born in the last four or five years. That to me is the best sign. You have new competitors, new existential problems. When you say, who’s it now? Claude’s going to kill you, Cursor is going to kill you, it’s not Borland. Thank God. That means we are in the right direction.

This is it. The fact that we went from nothing to this scale is the market expansion. This is like the cloud-like stuff. Fundamentally, this category of coding and AI is probably going to be one of the biggest categories. It is the software factory category. In fact, it may be bigger than knowledge work. I want to keep myself open-minded about it.

We’re going to have tough competition. That’s your point, which is a great one. But I’m glad we have parlayed what we had into this and now we have to compete. On the competing side, even in the last quarter we just finished, we did our quarterly announcement and I think we grew from 20 to 26 million subs. I feel good about our sub growth and where the direction of travel on that is.

But the more interesting thing that has happened is, guess where all the repos of all these other guys who are generating lots and lots of code go? They go to GitHub. GitHub is at an all-time high in terms of repo creation, PRs, everything. In some sense we want to keep that open, by the way. That means we want to have that. We don’t want to conflate that with our own growth. Interestingly enough, we are getting one developer joining GitHub a second or something, that is the stat, I think. And 80% of them just fall into some GitHub Copilot workflow, just because there are. By the way, many of these things will even use some of our coding code review agents, which are by default on, just because you can use it. We’ll have many, many structural shots at this. The thing that we’re also going to do is what we did with Git. The primitives of GitHub, starting with Git, to issues, to actions, these are powerful, lovely things because they kind of are all built around your repo. We want to extend that.

Last week at GitHub Universe, that’s kind of what we did. We said Agent HQ was the conceptual thing that we said we’re going to build out. This is where, for example, you have a thing called Mission Control. You go to Mission Control, and now I can fire off. Sometimes I describe it as the cable TV of all these AI agents because I’ll have, essentially packaged into one subscription, Codex, Claude, Cognition stuff, anyone’s agents, Grok, all of them will be there. So I get one package and then I can literally go issue a task and steer them so they’ll all be working in their independent branches. I can monitor them. I think that’s going to be one of the biggest places of innovation, because right now I want to be able to use multiple agents. I want to be able to then digest the output of the multiple agents. I want to be able to then keep a handle on my repo.

If there’s some kind of a heads-up display that needs to be built and then for me to quickly steer and triage what the coding agents have generated, that to me, between VS Code, GitHub, and all of these new primitives we’ll build as Mission Control with a control plane. Observability… Just think about everyone who is going to deploy all this. It will require a whole host of observability of what agent did what at what time to what code base. I feel that’s the opportunity.

At the end of the day your point is well taken, which is we better be competitive and innovate. If we don’t, we will get toppled. But I like the chart, at least as long as we’re on the top, even with competition.

Dylan Patel 00:17:26

The key point here is sort of that GitHub will keep growing regardless of whose coding agent wins. But that market only grows at say 10, 15, 20%, which is way above GDP. It’s a great compounder. But these AI coding agents have grown from say $500 million run rate at the end of last year—which was just GitHub Copilot—to now where the current run rate across GitHub Copilot, Claude Code, Cursor, Cognition, Windsurf, Replit, OpenAI Codex… That’s run rating at $5–6 billion now for the Q4 of this year. That’s 10x.

When you look at the TAM of software agents, is it the $2 trillion of wages you pay people, or is it something beyond that? Because every company in the world will now be able to develop software more? No question Microsoft takes a slice of that. But you’ve gone from near 100%, or certainly way above 50%, to sub-25% market share in just one year. What is the confidence that people can get that Microsoft will keep winning?

Satya Nadella 00:18:30

It goes back a little bit, Dylan, to that there’s no birthright here, that we should have any confidence other than to say we should go innovate. Knowing the lucky break we have, in some sense, is that this category is going to be a lot bigger than anything we had high share in. Let me say it that way. You could say we had high share in VS Code, we had high share in the repos with GitHub, and that was a good market. But the point is that even having a decent share in what is a much more expansive market…

You could say we had a high share in client-server server computing. We have much lower share than that in hyperscale. But is it a much bigger business? By orders of magnitude. So at least it’s existence proof that Microsoft has been okay even if our share position has not been as strong as it was, as long as the markets we are competing in are creating more value. And there are multiple winners. That’s the stuff.

But I take your point that ultimately it all means you have to get competitive. I watch that every quarter. That’s why I’m very optimistic about what we’re going to do with Agent HQ, turning GitHub into a place where all these agents come. As I said, we’ll have multiple shots on goal on there. It need not be… Some of these guys can succeed along with us, so it doesn’t need to be just one winner and one subscription.

00:20:02 - Whose margins will expand most?

Dwarkesh Patel 00:20:02

I guess the reason to focus on this question is that it’s not just about GitHub, but fundamentally about Office and all the other software that Microsoft offers. One vision you could have about how AI proceeds is that the models are going to keep being hobbled and you’ll need this direct visible observability all the time.

Another vision is that over time these models which are now doing tasks that take two minutes, in the future, they’ll be doing tasks that take 10, 30 minutes. In the future, maybe they’re doing days worth of work autonomously. Then the model companies are charging thousands of dollars maybe for access to, really, a coworker which could use any UI to communicate with their human and migrate between platforms.

If we’re getting closer to that, why aren’t the model companies that are just getting more and more profitable, the ones that are taking all the margin? Why is the place where the scaffolding happens, which becomes less and less relevant as the AI becomes more capable, going to be that important? That goes to Office as it exists now versus coworkers that are just doing knowledge work.

Satya Nadella 00:21:07

That’s a great point. Does all the value migrate just to the model? Or does it get split between the scaffolding and the model? I think that time will tell. But my fundamental point also is that the incentive structure gets clear. Let’s take information work, or take even coding. Already in fact, one of my favorite settings in GitHub Copilot is called auto, which will just optimize. In fact I buy a subscription and the auto one will start picking and optimizing for what I am asking it to do. It could even be fully autonomous. It could arbitrage the tokens available across multiple models to go get a task done.

If you take that argument, the commodity there will be models. Especially with open source models, you can pick a checkpoint and you can take a bunch of your data and you’re seeing it. I think all of us will start, whether it’s from Cursor or from Microsoft, seeing some in-house models even. And then you’ll offload most of your tasks to it.

So one argument is if you win the scaffolding—which today is dealing with all the hobbling problems or the jaggedness of these intelligence problems, which you kind of have to—if you win that, then you will vertically integrate yourself into the model just because you will have the liquidity of the data and what have you. There are enough and more checkpoints that are going to be available. That’s the other thing.

Structurally, I think there will always be an open source model that will be fairly capable in the world that you could then use, as long as you have something that you can use that with, which is data and a scaffolding. I can make the argument that if you’re a model company, you may have a winner’s curse. You may have done all the hard work, done unbelievable innovation, except it’s one copy away from that being commoditized. Then the person who has the data for grounding and context engineering, and the liquidity of data can then go take that checkpoint and train it. So I think the argument can be made both ways.

Dylan Patel 00:23:36

Unpacking what you said, there’s two views of the world. One is that there are so many different models out there. Open source exists. There will be differences between the models that will drive some level of who wins and who doesn’t. But the scaffolding is what enables you to win.

The other view is that, actually, models are the key IP. And everyone’s in a tight race and there’s some, “Hey, I can use Anthropic or OpenAI.” You can see this in the revenue charts. OpenAI’s revenue started skyrocketing once they finally had a code model with similar capabilities to Anthropic, although in different ways.

There’s the view that the model companies are the ones that garner all the margin. Because if you look across this year, at least at Anthropic, their gross margins on inference went from well below 40% to north of 60% by the end of the year. The margins are expanding there despite more Chinese open source models than ever. OpenAI is competitive, Google is competitive, X/Grok is now competitive. All these companies are now competitive, and yet despite this, the margins have expanded at the model layer significantly. How do you think about that?

Satya Nadella 00:24:43

It’s a great question. Perhaps a few years ago people were saying, “Oh, I could just wrap a model and build a successful company.” That has probably gotten debunked just because of the model capabilities, and the tools used, in particular.

But the interesting thing is, when I look at Office 365, let’s take even this little thing we built called Excel Agent. It’s interesting. Excel Agent is not a UI-level wrapper. It’s actually a model that is in the middle tier. In this case, because we have all the IP from the GPT family, we are taking that and putting it into the core middle tier of the Office system to teach it what it means to natively understand Excel, everything in it. It’s not just, “Hey, I just have a pixel-level understanding.” I have a full understanding of all the native artifacts of Excel. Because if you think about it, if I’m going to give it some reasoning task, I need to even fix the reasoning mistakes I make. That means I need to not just see the pixels, I need to be able to see, “Oh, I got that formula wrong,” and I need to understand that.

To some degree, that’s all being done not at the UI wrapper level with some prompt, but it’s being done in the middle tier by teaching it all the tools of Excel. I’m giving it essentially a markdown to teach it the skills of what it means to be a sophisticated Excel user. It’s a weird thing that it goes back a little bit to the AI brain. You’re building not just Excel, business logic in its traditional sense. You’re taking the Excel business logic in the traditional sense and wrapping essentially a cognitive layer to it, using this model which knows how to use the tool. In some sense, Excel will come with an analyst bundled in and with all the tools used. That’s the type of stuff that will get built by everybody.

So even for the model companies, they’ll have to compete. If they price stuff high, guess what, if I’m a builder of a tool like this, I’ll substitute you. I may use you for a while. So as long as there’s competition… There’s always a winner-take-all thing. If there’s going to be one model that is better than everybody else with massive distance, yes, that’s a winner-take-all. But as long as there’s competition where there are multiple models, just like hyperscale competition, and there’s an open source check, there is enough room here to go build value on top of models.

At Microsoft, the way I look at it is that we are going to be in the hyperscale business, which will support multiple models. We will have access to OpenAI models for seven more years, which we will innovate on top of. Essentially, I think of ourselves as having a frontier-class model that we can use and innovate on with full flexibility. And we’ll build our own models with MAI. So we will always have a model level. And then we’ll build—whether it’s in security, whether it’s in knowledge work, whether it’s in coding, or in science—our own application scaffolding, which will be model-forward. It won’t be a wrapper on a model, but the model will be wrapped into the application.

Dwarkesh Patel 00:28:04

I have so many questions about the other things you mentioned. But before we move onto those topics, I still wonder whether this is not forward-looking on AI capabilities, where you’re imagining models like they exist today. It takes a screenshot of your screen, but it can’t look inside each cell and what the formula is. I think the better mental model here is just imagining that these models will be able to use a computer as well as a human. A human knowledge worker who is using Excel can look into the formulas, can use alternative software, can migrate data between Office 365 and another piece of software if the migration is necessary, et cetera.

Satya Nadella 00:28:43

That’s kind of what I’m saying.

Dwarkesh Patel 00:28:45

But if that’s the case, then the integration with Excel doesn’t matter that much.

Satya Nadella 00:28:48

No, no, don’t worry about the Excel integration. After all, Excel was built as a tool for analysts. Great. So whoever is this AI that is an analyst should have tools that they can use.

Dwarkesh Patel 00:29:03

They have the computer. Just the way a human can use a computer. That’s their tool.

Satya Nadella 00:29:05

The tool is the computer. So all I’m saying is that I’m building an analyst as essentially an AI agent, which happens to come with an a priori knowledge of how to use all of these analytical tools.

Dwarkesh Patel 00:29:20

Just to make sure we’re talking about the same thing, is it a thing that a human like me using Excel…

Satya Nadella 00:29:30

No, it’s completely autonomous. So we should now maybe lay out what I think the future of the company is. The future of the company would be the tools business in which I have a computer, I use Excel. In fact, in the future I’ll even have a Copilot, and that Copilot will also have agents. But it’s still me steering everything, and everything is coming back. That’s one world.

The second world is the company just literally provisions a computing resource for an AI agent, and that is working fully autonomously. That fully autonomous agent will have essentially an embodied set of those same tools that are available to it. So this AI tool that comes in also has not just a raw computer, because it’s going to be more token-efficient to use tools to get stuff done.

In fact, I kind of look at it and say that our business, which today is an end-user tools business, will become essentially an infrastructure business in support of agents doing work. It’s another way to think about it. In fact, all the stuff we built underneath M365 still is going to be very relevant. You need some place to store it, some place to do archival, some place to do discovery, some place to manage all of these activities, even if you’re an AI agent. It’s a new infrastructure.

Dwarkesh Patel 00:31:00

To make sure I understand, you’re saying theoretically a future AI that has actual computer use—which all these model companies are working on right now—could use, even if it’s not partnered with Microsoft or under our umbrella, Microsoft software. But you’re saying, if you’re working with our infrastructure, we’re going to give them lower-level access that makes it more efficient for you to do the same things you could have otherwise done anyways?

Satya Nadella 00:31:24

100%. What happened is we had servers, then there was virtualization, and then we had many more servers. That’s another way to think about this. Don’t think of the tool as the end thing. What is the entire substrate underneath that tool that humans use? That entire substrate is the bootstrap for the AI agent as well, because the AI agent needs a computer.

In fact, one of the fascinating things where we’re seeing a significant amount of growth is all these guys who are doing these Office artifacts and what have you, as autonomous agents and so on want to provision Windows 365. They really want to be able to provision a computer for these agents. Absolutely. That’s why we’re going to have essentially an end-user computing infrastructure business, which is going to just keep growing because it’s going to grow faster than the number of users.

That’s one of the other questions people ask me, “Hey, what happens to the per-user business?” At least the early signs maybe, the way to think about the per-user business is not just per user, it’s per agent. And if you say it’s per user and per agent, the key is what’s the stuff to provision for every agent? A computer, a set of security things around it, an identity around it. All those things, observability and so on, are the management layers. That’s all going to get baked into that.

Dylan Patel 00:32:54

The way to frame it—at least the way I currently think about it and I’d like to hear your view—is that these model companies are all building environments to train their models to use Excel or Amazon shopping or whatever it is, book flights. But at the same time, they’re also training these models to do migration. Because that is probably the most immediately valuable thing: converting mainframe-based systems to standard cloud systems, converting Excel databases into real databases with SQL, or converting what is done in Word and Excel to something that is more programmatic and more efficient in a classical sense that can be done by humans as well. It’s just not cost-effective for the software developer to do that.

That seems to be what everyone is going to do with AI, for the next few years at least, to massively drive value. How does Microsoft fit into that if the models can utilize the tools themselves to migrate to something? Yes, Microsoft has a leadership position in databases and in storage and in all these other categories, but the use of an Office ecosystem is going to be significantly less just like the use of a mainframe ecosystem could be potentially less. Now mainframes have grown for the last two decades actually, even though no one talks about them anymore. They’ve still grown.

Satya Nadella 00:34:16

100%, I agree with that.

Dylan Patel 00:34:17

How does that flow?

Satya Nadella 00:34:18

At the end of the day, there is going to be a significant amount of time where there’s going to be a hybrid world, because people are going to be using the tools that are going to be working with agents that have to use tools, and they have to communicate with each other. What’s the artifact I generate that then a human needs to see? All of these things will be real considerations in any place, the outputs, inputs. I don’t think it’ll just be about, “Oh, I migrated off.” The bottom line is that I have to live in this hybrid world.

But that doesn’t fully answer your question because there can be a real new efficient frontier where it’s just agents working with agents and completely optimized. Even when agents are working with agents, what are the primitives that are needed? Do you need a storage system? Does that storage system need to have e-discovery? Do you need to have observability? Do you need to have an identity system that is going to use multiple models with all having one identity system? These are all the core underlying rails we have today for what are the Office systems or what have you. And that’s what we will have in the future as well.

You’ve talked about databases. I mean man, I would love all of Excel to have a database backend. I would love for all that to happen immediately. And that database is a good database. Databases in fact will be a big thing that will grow. If I think about all of the Office artifacts being structured better, the ability to do the joins between structured and unstructured better because of the agentic world, that will grow the underlying infrastructure business. It happens that the consumption of that is all being driven by agents.

You could say all that is just-in-time generated software by a model company. That could also be true. We will be one such model company too. We will build in... The competition could be that we will build a model plus all the infrastructure and provision it, and then there will be competition between a bunch of those folks who can do that.

00:36:17 - MAI

Dwarkesh Patel 00:36:17

Speaking of model companies, you say not only will you have the infrastructure, you’ll have the model itself. Right now, Microsoft AI’s most recent model that was released two months ago is 36 in Chatbot Arena. You obviously have the IP rights to OpenAI. To the extent you agree with that, it seems to be behind. Why is that the case, especially given the fact that you theoretically have the right to fork OpenAI’s monorepo or distill their models, especially if it’s a big part of your strategy that you need to have a leading model company?

Satya Nadella 00:36:51

First of all, we are absolutely going to use the OpenAI models to the maximum across all of our products. That’s the core thing that we’re going to continue to do all the way for the next seven years, and not just use it but then add value to it. That’s where the analyst and this Excel agent, these are all things that we will do where we’ll do RL fine-tuning. We’ll do some mid-training runs on top of a GPT family where we have unique data assets and build capability.

With the MAI model, the way that I think we’re going to think about it is that the good news here with the new agreement is we can be very, very clear that we’re going to build a world-class superintelligence team and go after it with a high ambition. But at the same time, we’re also going to use this time to be smart about how to use both these things. That means we will, on one end, be very product-focused, and on the other end, be very research-focused. Because we have access to the GPT family, the last thing I want to do is use my flops in a way that is just duplicative and doesn’t add much value.

I want to be able to take the flops that we use to generate a GPT family and maximize its value, while my MAI flops are being used for… Let’s take the image model that we launched, which I think is at number nine in the image arena. We’re using it both for cost optimization, it’s on Copilot, it’s in Bing, and we’re going to use that. We have an audio model in Copilot. It’s got personality and what have you. We optimized it for our product. So we will do those.

Even on the LMArena, we started on the text one and it debuted at like 13. By the way, it was done only on around 15,000 H100s. It was a very small model. So it was, again, to prove out the core capability, the instruction following, and everything else. We wanted to make sure we could match what was state of the art. That shows us, given scaling laws, what we are capable of doing if we gave more flops to it. The next thing we will do is an omni-model where we will take the work we have done in audio, what we have done in image, and what we have done in text. That will be the next pit stop on the MAI side.

So when I think about the MAI roadmap, we are going to build a first-class superintelligence team. We are going to continue to drop, and do it in the open, some of these models. They will either be used in our products, because they’re going to be latency-friendly, cost-friendly, or what have you, or they’ll have some special capability. And we will do real research in order to be ready for the next five, six, seven, eight breakthroughs that are all needed on this march towards superintelligence—while exploiting the advantage we have of having the GPT family that we can work on top of as well.

Dylan Patel 00:39:45

Say we roll forward seven years, you no longer have access to OpenAI models. What does Microsoft do to make sure they are leading, or have a leading AI lab? Today, OpenAI has developed many of the breakthroughs, whether it be scaling or reasoning. Or Google’s developed all the breakthroughs like transformers.

But it is also a big talent game. You’ve seen Meta spend north of $20 billion on talent. You’ve seen Anthropic poach the entire Blueshift reasoning team from Google last year. You’ve seen Meta poach a large reasoning and post-training team from Google more recently. These sorts of talent wars are very capital intensive. Arguably, if you’re spending $100 billion on infrastructure, you should also spend X amount of money on the people using the infrastructure so that they’re more efficiently making these new breakthroughs.

What confidence can one get that Microsoft will have a team that’s world-class that can make these breakthroughs? Once you decide to turn on the money faucet—you’re being a bit capital efficient right now, which is smart it seems, to not waste money doing duplicative work—but once you decide you need to, how can one say, “Oh yeah, now you can shoot up to the top five model?”

Satya Nadella 00:41:00

At the end of the day, we’re going to build a world-class team and we already have a world-class team that’s beginning to be assembled. We have Mustafa coming in, we have Karen. We have Amar Subramanya who did a lot of the post-training at Gemini 2.5 who’s at Microsoft. Nando, who did a lot of the multimedia work at DeepMind, is there. We’re going to build a world-class team. In fact, later this week even, Mustafa will publish something with a little more clarity on what our lab is going to go do.

The thing that I want the world to know, perhaps, is that we are going to build the infrastructure that will support multiple models. Because from a hyperscale perspective, we want to build the most scaled infrastructure fleet that’s capable of supporting all the models the world needs, whether it’s from open source or obviously from OpenAI and others. That’s one job.

Secondly, in our own model capability, we will absolutely use the OpenAI model in our products and we’ll start building our own model. And we may—like in GitHub Copilot where Anthropic is used—even have other frontier models that are going to be wrapped into our products, as well. I think that’s how each time… At the end of the day, the eval of the product as it meets a particular task or a job is what matters. We’ll start back from there into the vertical integration needed, knowing that as long as you’re serving the market well with the product, you can always cost-optimize.

Dwarkesh Patel 00:42:31

There’s a question going forward. Right now, we have models that have this distinction between training and inference. One could argue that there’s a smaller and smaller difference between the different models. Going forward, if you’re really expecting something like human-level intelligence, humans learn on the job. If you think about your last 30 years, what makes Satya tokens so valuable? It’s the last 30 years of wisdom and experience you’ve gained in Microsoft.

We will eventually have models, if they get to human level, which will have this ability to continuously learn on the job. That will drive so much value to the model company that is ahead, at least in my view, because you have copies of one model broadly deployed through the economy learning how to do every single job. And unlike humans, they can amalgamate their learnings to that model. So there’s this sort of continuous learning exponential feedback loop, which almost looks like a sort of intelligence explosion.

If that happens and Microsoft isn’t the leading model company by that time… You’re saying that well, we substitute one model for another, et cetera. Doesn’t that then matter less? Because it’s like this one model knows how to do every single job in the economy, the others in the long tail don’t.

Satya Nadella 00:43:39

Your point, if there’s one model that is the only model that’s most broadly deployed in the world and it sees all the data and it does continuous learning, that’s game set match and you shut shop. The reality that at least I see is that in the world today, for all the dominance of any one model, that is not the case. Take coding, there are multiple models. In fact, everyday it’s less the case. There is not one model that is getting deployed broadly. There are multiple models that are getting deployed. It’s like databases. It’s always the thing, “Can one database be the one that is just used everywhere?” Except it’s not. There are multiple types of databases that are getting deployed for different use cases.

I think that there are going to be some network effects of continual learning—I call it data liquidity—that any one model has. Is it going to happen in all domains? I don’t think so. Is it going to happen in all geos? I don’t think so. Is it going to happen in all segments? I don’t think so. It’ll happen in all categories at the same time? I don’t think so. So therefore I feel like the design space is so large that there’s plenty of opportunity.

But your fundamental point is having a capability which is at the infrastructure layer, model layer, and at the scaffolding layer, and then being able to compose these things not just as a vertical stack, but to be able to compose each thing for what its purpose is. You can’t build an infrastructure that’s optimized for one model. If you do that, what if you fall behind? In fact, all the infrastructure you built will be a waste. You kind of need to build an infrastructure that’s capable of supporting multiple families and lineages of models. Otherwise the capital you put in, which is optimized for one model architecture, means you’re one tweak away, some MoE-like breakthrough that happens, and your entire network topology goes out of the window. That’s a scary thing.

Therefore you kind of want the infrastructure to support whatever may come in your own model family and other model families. You’ve got to be open. If you’re serious about the hyperscale business, you’ve got to be serious about that. If you’re serious about being a model company, you have to basically say, “What are the ways people can do things on top of the model so that I can have an ISV ecosystem?” Unless I’m thinking I’ll own every category, that just can’t be that. Then you won’t have an API business and that, by definition, will mean you’ll never be a platform company that’s successfully deployed everywhere. Therefore the industry structure is such that it will really force people to specialize. In that specialization, a company like Microsoft should compete in each layer by its merits, but not think that this is all about the road to game set match, where I just compose vertically all these layers. That just doesn’t happen.

00:47:47 - The hyperscale business

Dylan Patel 00:47:47

So last year Microsoft was on path to be the largest infrastructure provider by far. You were the earliest in 2023, so you went out there, you acquired all the resources in terms of leasing data centers, starting construction, securing power, everything. You guys were on pace to beat Amazon in 2026 or 2027. Certainly by 2028 you were going to beat them.

Since then, let’s call it, in the second half of last year, Microsoft did this big pause, where they let go of a bunch of leasing sites that they were going to take, which then Google, Meta, Amazon in some cases, Oracle, took these sites.

We’re sitting in one of the largest data centers in the world, so obviously it’s not everything, you guys are expanding like crazy. But there are sites that you just stopped working on. Why did you do this?

Satya Nadella 00:48:32

This goes back a little bit to, what is the hyperscale business all about? One of the key decisions we made was that if we’re going to build out Azure to be fantastic for all stages of AI—from training to mid-training to data gen to inference—we just need fungibility of the fleet. So that entire thing caused us basically not to go build a whole lot of capacity with a particular set of generations.

Because the other thing you have to realize is that having up to now 10x’ed every 18 months enough training capacity for the various OpenAI models, we realized that the key is to stay on that path. But the more important thing is to have a balance, to not just train, but to be able to serve these models all around the world. Because at the end of the day, the rate of monetization is what will then allow us to keep funding. And then the infrastructure was going to need us to support multiple models.

So once we said that that’s the case, we just course-corrected to the path we’re on. If I look at the path we’re on, we are doing a lot more starts now. We are also buying up as much managed capacity as we can, whether it’s to build, whether it’s to lease, or even GPUs as a service. But we’re building it for where we see the demand and the serving needs and our training needs. We didn’t want to just be a hoster for one company and have just a massive book of business with one customer. That’s not a business, you should be vertically integrated with that company.

Given that OpenAI was going to be a successful independent company, which is fantastic. It makes sense. And even Meta may use third-party capacity, but ultimately they’re all going to be first-party. For anyone who has large scale, they’ll be a hyperscaler on their own. To me, it was to build out a hyperscale fleet and our own research compute. That’s what the adjustment was. So I feel very, very good.

By the way, the other thing is that I didn’t want to get stuck with massive scale of one generation. We just saw the GB200s, the GB300s are coming. By the time I get to Vera Rubin, Vera Rubin Ultra, the data center is going to look very different because the power per rack, power per row, is going to be so different. The cooling requirements are going to be so different. That means I don’t want to just go build out a whole number of gigawatts that are only for a one-generation, one family. So I think the pacing matters, the fungibility and the location matters, the workload diversity matters, customer diversity matters and that’s what we’re building towards.

The other thing that we’ve learned a lot is that every AI workload does require not only the AI accelerator, but it requires a whole lot of other things. In fact, a lot of the margin structure for us will be in those other things. Therefore, we want to build out Azure as being fantastic for the long tail of the workloads, because that’s the hyperscale business, while knowing that we’ve got to be super competitive starting with the bare-metal for the highest end training.

But that can’t crowd out the rest of the business, because we’re not in the business of just doing five contracts with five customers being their bare-metal service. That’s not a Microsoft business. That may be a business for someone else, and that’s a good thing. What we have said is that we’re in the hyperscale business, which is at the end of the day a long tail business for AI workloads. And in order to do that, we will have some leading bare-metal-as-a-service capabilities for a set of models, including our own. And that, I think, is the balance you see.

Dylan Patel 00:52:41

Another question that comes around this whole fungibility topic. Okay, it’s not where you want it, you would rather have it in a good population center, like Atlanta. We’re here. There’s also the question of, how much does that matter as the horizon of AI tasks grows? 30 seconds for a reasoning prompt, or 30 minutes for a deep research, or it’s going to be hours for software agents at some point and days and so on and so forth, the time to human interaction. Why does it matter if it’s location A, B, or C?

Satya Nadella 00:53:14

It’s a great question. That’s exactly it. In fact, that’s one of the other reasons why we want to think about what an Azure region looks like and what is the networking between Azure regions. This is where I think as the model capabilities evolve and the usage of these tokens evolves, whether it’s synchronously or asynchronously, you don’t want to be out of position.

Then on top of that, by the way, what are the data residency laws? There’s the entire EU thing, where we literally had to create an EU Data Boundary. That basically meant that you can’t just roundtrip a call to wherever, even if it’s asynchronous. Therefore you need to have maybe regional things that are high density, and then the power costs and so on.

But you’re 100% right in bringing up that the topology as we build out will have to evolve. One, for tokens per dollar per watt. What are the economics? Overlay that with, what is the usage pattern? Usage pattern in terms of synchronous, asynchronous. But also what is the compute storage? Because the latencies may matter for certain things. The storage better be there. If I have a Cosmos DB close to this for session data or even for an autonomous thing, then that also has to be somewhere close to it, and so on. All of those considerations are what will shape the hyperscale business.

Dylan Patel 00:54:38

Prior to the pause, what we had forecasted for you, by 2028 you were going to be 12–13 gigawatts. Now we’re at nine and a half or so.

But something that’s even more relevant—and I just want you to more concretely state that this is the business you don’t want to be in—is that Oracle’s going from 1/5th your size to bigger than you by the end of 2027.

While it’s not a Microsoft-level quality of return on invested capital, they’re still making 35% gross margins. So the question is, maybe it’s not Microsoft’s business to do this, but you’ve created a hyperscaler now by refusing this business, by giving away the right of first refusal, et cetera.

Satya Nadella 00:55:23

First of all, I don’t want to take away anything from the success Oracle has had in building their business and I wish them well. The thing that I think I’ve answered for you is that it didn’t make sense for us to go be a hoster for one model company with limited time horizon RPO. Let’s just put it that way.

The thing that you have to think through is not what you do in the next five years, but what you do for the next 50. We made our set of decisions. I feel very good about our OpenAI partnership and what we’re doing. We have a decent book of business. We wish them a lot of success. In fact, we are buyers of Oracle capacity. We wish them success.

But at this point, I think the industrial logic for what we are trying to do is pretty clear, which is that it’s not about chasing… First of all, I track, by the way, your things whether it’s AWS or Google and ours, which I think is super useful. But it doesn’t mean I have to chase those. I have to chase them for not just the gross margin that they may represent in a period of time. What is this book of business that Microsoft uniquely can go clear, which makes sense for us to clear? That’s what we’ll do.

Dwarkesh Patel 00:56:40

I have a question even stepping back from this, I take your point that it’s a better business to be in, all else equal, to have a long tail of customers you can have higher margin from rather than serving bare metal to a few labs. But then there’s a question of, which way is the industry evolving? If we believe we’re on the path to smarter and smarter AIs, then why isn’t the shape of the industry that the OpenAIs and Anthropics and DeepMinds are the platform on which the long tail of enterprises are actually doing business? They need bare metal, but they are the platform. What is the long tail that is directly using Azure? Because you want to use the general cognitive core.

Satya Nadella 00:57:22

But those models are all going to be available on Azure, so any workload that says, “Hey, I want to use some open source model and an OpenAI model,” if you go to Azure Foundry today, you have all these models that you can provision, buy PTUs, get a Cosmos DB, get a SQL DB, get some storage, get some compute. That’s what a real workload looks like. A real workload is not just an API call to a model. A real workload needs all of these things to go build an app or instantiate an application.

In fact, the model companies need that to build anything. It’s not just like, “I have a token factory.” I have to have all of these things. That’s the hyperscale business. And it’s not on any one model, but all of these models. So if you want Grok plus, say, OpenAI plus an open source model, come to Azure Foundry, provision them, build your application. Here is a database. That’s kind of what the business is.

There is a separate business called just selling raw bare-metal services to model companies. And that’s the argument about how much of that business you want to be in and not be in and what that is. It’s a very different segment of the business, which we are in, and we also have limits to how much of it is going to crowd out the rest of it. But that’s kind of at least the way I look at it.

Dylan Patel 00:58:40

There are sort of two questions here. One is, why couldn’t you just do both? The other one is, given our estimates on what your capacity is in 2028, it’s three and a half gigawatts lower. Sure, you could have dedicated that to OpenAI training and inference capacity, but you could have also dedicated that to actually just running Azure, running Microsoft 365, running GitHub Copilot. I could have just built it and not given it to OpenAI.

Satya Nadella 00:59:08

Or I may want to build it in a different location. I may want to build it in the UAE, I may want to build it in India, I may want to build it in Europe. One of the things is, as I said, where we have real capacity constraints right now, given the regulatory needs and the data sovereignty needs, we’ve got to build all over the world. First of all, stateside capacity is super important, and we want to build everything.

But when I look out to 2030, I have a global view of what is Microsoft’s shape of business by first-party and third-party. Third-party segmented by the frontier labs and how much they want versus the inference capacity we want to build for multiple models, and our own research compute needs. That’s all going into my calculus. You’re rightfully pointing out the pause, but the pause was not done because we said, “Oh my God, we don’t want to build that.” We realized that we want to build what we want to build slightly differently by both workload type as well as geo-type and timing as well.

We’ll keep ramping up our gigawatts, and the question is at what pace and in what location. And how do I ride Moore’s law on it, which is, do I really want to overbuild three and a half in 2027 or do I want to spread that in 2027-28 knowing even… One of the biggest learnings we had even with Nvidia is that their pace increased in terms of their migrations.

That was a big factor. I didn’t want to go get stuck for four or five years of depreciation on one generation. In fact, Jensen’s advice to me was two things. One is, get on the speed-of-light execution. That’s why the execution in this Atlanta data center.... I mean, it’s like 90 days between when we get it and to hand off to a real workload. That’s real speed-of-light execution on that front. I wanted to get good on that.

And then that way I’m building each generation in scaling. And then every five years, you have something much more balanced. So it becomes literally like a flow for a large-scale industrial operation like this where you’re suddenly not lopsided, where you’ve built up a lot in one time and then you take a massive hiatus because you’re stuck with all this, to your point, in one location which may be great for training, or it may not be great for inference because I can’t serve, even if it’s all asynchronous, because Europe won’t let me round-trip to Texas. So that’s all of the things.

Dylan Patel 01:01:38

How do I rationalize this statement with what you’ve done over the last few weeks? You’ve announced deals with Iris Energy, with Nebius, and Lambda Labs, and there’s a few more coming as well. You’re going out there and securing capacity that you’re renting from the neoclouds rather than having built it yourself.

Satya Nadella 01:01:58

It’s fine for us because now when you have line of sight to demand, which can be served where people are building, it’s great. In fact we will take leases, we will take build-to-suit, we’ll even take GPUs-as-a-service where we don’t have capacity but we need capacity and someone else has that.

And by the way, I would even sort of welcome every neocloud to just be part of our marketplace. Because guess what? If they go bring their capacity into our marketplace, that customer who comes through Azure will use the neocloud, which is a great win for them, and will use compute, storage, databases, all the rest from Azure. So I’m not at all thinking of this as, “Hey, I should just go gobble up all of that myself.”

01:02:44 - In-house chip & OpenAI partnership

Dwarkesh Patel 01:02:44

You mentioned how this depreciating asset, in five or six years, is 75% of the TCO of a data center. And Jensen is taking a 75% margin on that. So what all the hyperscalers are trying to do is develop their own accelerator so that they can reduce this overwhelming cost for equipment, to increase their margins.

Dylan Patel 01:03:09

And when you look at where they are, Google’s way ahead of everyone else. They’ve been doing it for the longest.

They’re going to make something like five to seven million chips of their own TPUs. You look at Amazon and they’re trying to make three to five million [Lifetime shipment units]. But when we look at what Microsoft is ordering of their own chips, it’s way below that number. You’ve had a program for just as long. What’s going on with your internal chips?

Satya Nadella 01:03:32

It’s a good question. A couple of things. One is that the thing that is the biggest competitor for any new accelerator is kind of even the previous generation of Nvidia. In a fleet, what I’m going to look at is the overall TCO. The bar I have, even for our own… By the way, I was just looking at the data for Maia 200 which looks great, except that one of the things that we learned even on the compute side… We had a lot of Intel, then we introduced AMD, and then we introduced Cobalt. That’s how we scaled it. We have good existence proof of, at least in core compute, how to build your own silicon and then manage a fleet where all three are at play in some balance.

Because by the way, even Google’s buying Nvidia, and so is Amazon. It makes sense because Nvidia is innovating and it’s the general-purpose thing. All models run on it and customer demand is there. Because if you build your own vertical thing, you better have your own model, which is either going to use it for training or inference, and you have to generate your own demand for it or subsidize the demand for it. So therefore you want to make sure you scale it appropriately.

The way we are going to do it is to have a close loop between our own MAI models and our silicon, because I feel like that’s what gives you the birthright to do your own silicon, where you literally have designed the microarchitecture with what you’re doing, and then you keep pace with your own models. In our case, the good news here is that OpenAI has a program which we have access to. So therefore to think that Microsoft is not going to have something that’s—

Dylan Patel 01:05:15

What level of access do you have to that?

Satya Nadella 01:05:16

All of it.

Dylan Patel 01:05:17

You just get the IP for all of that? So the only IP you don’t have is consumer hardware?

Satya Nadella 01:05:20

That’s it.

Dylan Patel 01:05:21

Oh, okay. Interesting.

Satya Nadella 01:05:26

By the way, we gave them a bunch of IP as well to bootstrap them. This is one of the reasons why they… Because we built all these supercomputers together. We built it for them and they benefited from it, rightfully so. And now as they innovate, even at the system level, we get access to all of it. And we first want to instantiate what they build, for them, but then we’ll extend it.

So if anything, the way I think about your question is, Microsoft wants to be a fantastic, I’ll call it, speed-of-light execution partner for Nvidia. Because quite frankly that fleet is life itself. Obviously Jensen’s doing super well with his margins, but the TCO has many dimensions to it and I want to be great at that TCO. On top of that, I want to be able to really work with the OpenAI lineage and the MAI lineage and the system design, knowing that we have the IP rights on both ends.

Dwarkesh Patel 01:06:32

Speaking of rights, you had an interview a couple days ago where you said that in the new agreement you made with OpenAI you have rights, the exclusivity, to the stateless API calls that OpenAI makes. We were sort of confused about if there’s any state whatsoever. You were just mentioning a second ago that all these complicated workloads that are coming up are going to require memory and databases and storage and so forth. Is that now not stateless if ChatGPT is storing stuff on sessions?

Satya Nadella 01:07:03

That’s the reason why. The strategic decision we made, and also accommodating for the flexibility OpenAI needed in order to be able to procure compute for… Essentially think of OpenAI having a PaaS business and a SaaS business. The SaaS business is ChatGPT. Their PaaS business is their API. That API is Azure-exclusive. The SaaS business, they can run it anywhere.

Dylan Patel 01:07:31

And they can partner with anyone they want to to build SaaS products?

Satya Nadella 01:07:34

If they want a partner and that partner wants to use a stateless API, then Azure is the place where they can get the stateless API.

Dylan Patel 01:07:43

It seems like there’s a way for them to build the product together and it’s a stateful thing…

Satya Nadella 01:07:47

No, for even that they’ll have to come to Azure. Again, this is done in the spirit of “what is it that we value as part of our partnership.” And we made sure that, at the same time, we were good partners to OpenAI given all the flexibility they needed.

Dylan Patel 01:08:05

So for example, Salesforce wants to integrate OpenAI. It’s not through an API. They actually work together, train a model together and deploy it on, let’s say, Amazon now. Is that allowed or do they have to use your…

Satya Nadella 01:08:16

For any custom agreement like that, they will have to come run it… There are some few exceptions, the US government and so on, that we made, but other than that, they’d have to come to Azure.

01:09:35 - The CAPEX explosion

Dwarkesh Patel 01:09:35

Stepping back, when we were walking back and forth through the factory, one of the things you were talking about is that Microsoft, you can think of it as a software business, but now it’s really becoming an industrial business. There’s all this capex, there’s all this construction. If you just look over the last two years, your capex has sort of tripled. Maybe you extrapolate that forward, it actually just becomes this huge industrial explosion.

Dylan Patel 01:10:01

Other hyperscalers are taking loans. Meta has done a $20 billion loan at Louisiana. They’ve done a corporate loan. It seems clear everyone’s free cash flow is going to zero, which I’m sure Amy is going to beat you up if you even try to do that, but what’s happening?

Satya Nadella 01:10:19

I think the structural change is what you’re referencing, which is massive. I describe it as we are now a capital-intensive business and a knowledge-intensive business. In fact, we have to use our knowledge to increase the ROIC on the capital spend.

The hardware guys have done a great job of marketing Moore’s Law, which I think is unbelievable and it’s great. But if you even look at some of the stats I even did in my earnings call, for a given GPT family, the software improvements of really throughput in terms of tokens-per-dollar-per-watt that we’re able to get quarter-over-quarter, year-over-year, it’s massive. It’s 5x, 10x, maybe 40x in some of these cases, just because of how you can optimize. That’s knowledge intensity coming to bring out capital efficiency. That, at some level, is what we have to master.

Some people ask me, what is the difference between a classic old-time hoster and a hyperscaler? Software. Yes, it is capital intensive, but as long as you have systems know-how, software capability to optimize by workload, by fleet... That’s why when we say fungibility, there’s so much software in it. It’s not just about the fleet.

It’s the ability to evict a workload and then schedule another workload. Can I manage that algorithm of scheduling around? That is the type of stuff that we have to be world-class at. So yes, I think we’ll still remain a software company, but yes, this is a different business and we’re going to manage. At the end of the day, the cash flow that Microsoft has allows us to have both these arms firing well.

Dwarkesh Patel 01:12:18

It seems like in the short term you have more credence on things taking a while, being more jagged. But maybe in the long term you think the people who talk about AGI and ASI are correct. Sam will be right, eventually.

I have a broader question about what makes sense for a hyperscaler to do, given that you have to invest massively in this thing which depreciates over five years. So if you have 2040 timelines to the kind of thing that somebody like Sam anticipates in three years, what is a reasonable thing for you to do in that world?

Satya Nadella 01:12:52

There needs to be an allocation to, I’ll call it, research compute. That needs to be done like you did R&D. That’s the best way to even account for it, quite frankly. We should think of it as just R&D expense and you should say, “What’s the research compute and how do you want to scale it?” Let’s even say it’s an order of magnitude scale in some period. Pick your thing, is it two years? Is it 16 months? What have you. That’s sort of one piece, which is table stakes, that’s R&D expenses.

The rest is all demand driven. Ultimately, you’re allowed to build ahead of demand, but you better have a demand plan that doesn’t go completely off kilter.

Dwarkesh Patel 01:13:39

Do you buy… These labs are now projecting revenues of $100 billion in 2027–28 and they’re projecting revenue to keep growing at this rate of 3x, 2x a year…

Satya Nadella 01:13:50

In the marketplace there’s all kinds of incentives right now, and rightfully so. What do you expect an independent lab that is sort of trying to raise money to do? They have to put some numbers out there such that they can actually go raise money so that they can pay their bills for compute and what have you.

And it’s a good thing. Someone’s going to take some risk and put it in there, and they’ve shown traction. It’s not like it’s all risk without seeing the fact that they’ve been performing, whether it’s OpenAI, or whether it’s Anthropic. So I feel great about what they’ve done, and we have a massive book of business with these chaps. So therefore that’s all good.

But overall ultimately, there’s two simple things. One is you have to allocate for R&D. You brought up talent. The talent for AI is at a premium. You have to spend there. You’ve got to spend on compute. So in some sense researcher-to-GPU ratios have to be high. That is sort of what it takes to be a leading R&D company in this world. And that’s something that needs to scale, and you have to have a balance sheet that allows you to scale that long before it’s conventional wisdom and so on. That’s kind of one thing. But the other is all about knowing how to forecast.

01:15:07 - Will the world trust US companies to lead AI?

Dylan Patel 01:15:07

As we look across the world, America has dominated many tech stacks. The US owns Windows through Microsoft, which is deployed even in China, that’s the main operating system. Of course, there’s Linux, which is open source, but Windows is deployed everywhere in China on personal computers. You look at Word, it’s deployed everywhere. You look at all these various technologies, it’s deployed everywhere. And Microsoft and other companies have grown elsewhere. They’re building data centers in Europe and in India and in all these other places, in Southeast Asia and LatAm and Africa. In all of these different places, you’re building capacity.

But this seems quite different. Today, the political aspect of technology, of compute… The US administration didn’t care about the dot-com bubble. It seems like the US administration, as well as every other administration around the world, cares a lot about AI. The question is, we’re sort of in a bipolar world, at least with the US and China, but Europe and India and all these other countries are saying, “No, we’re going to have sovereign AI as well.”

How does Microsoft navigate the difference to the 90s—where there’s one country in the world that matters, it’s America, and our companies sell everywhere and therefore Microsoft benefits massively—to a world where it is bipolar? Where Microsoft can’t just necessarily have the right to win all of Europe or India or Singapore. There are actually sovereign AI efforts. What is your thought process here and how do you think about this?

Satya Nadella 01:16:36

It’s a super critical piece. I think that the key, key priority for the US tech sector and the US government is to ensure that we not only do leading innovative work, but that we also collectively build trust around the world on our tech stack. Because I always say the United States is just an unbelievable place. It’s just unique in history. It’s 4% of the world’s population, 25% of the GDP, and 50% of the market cap. I think you should think about those ratios and reflect on it.

That 50% happens because quite frankly the trust the world has in the United States, whether it’s its capital markets or whether it’s its technology and its stewardship of what matters at any given time in terms of leading sector. If that is broken, then that’s not a good day for the United States. We start with that, which I think President Trump gets, the White House, David Sacks, everyone really, I think, gets it.

So therefore I applaud anything that the United States government and the tech sector jointly does to, for example, put our own capital at risk, collectively as an industry, in every part of the world. I would like the USG to take credit for foreign direct investment by American companies all over the world. It’s the least talked about, but the best marketing that the United States should be doing is that it’s not just about all the foreign direct investment coming into the United States, but the most leading sector, which is these AI factories, are all being created all over the world. By whom? By America and American companies.

And so you start there, and then you even build other agreements around it, which are around their continuity, their legitimate sovereignty concerns, around whether it’s data residency, for them to have real agency and guarantees on privacy, and so on. In fact, our European commitments are worth reading. We made a series of commitments to Europe on how we will govern our hyperscale investment there such that the European Union and the European countries have sovereignty.

We’re also building sovereign clouds in France and in Germany. We have something called Sovereign Services on Azure, which literally gives people key management services along with confidential computing, including confidential computing in GPUs, which we’ve done great innovative work with Nvidia. So I feel very, very good about being able to build, both technically and through policy, this trust in the American tech stack.

Dwarkesh Patel 01:19:42

How do you see this shaking out as you have this network effect with continual learning and things on the model level? Maybe you have equivalent things at the hyperscaler level as well. Do you expect that the countries will say, “Look, it’s clear one model or a couple models are the best, and so we’re going to use them, but we’re going to have some laws around the weights having to be hosted in our country”? Or do you expect that there will be this push so that it has to be a model trained in our country?

Maybe an analogy here is that semiconductors are very important to the economy, and people would like to have their sovereign semiconductors, but TSMC is just better. And semiconductors are so important to the economy that you will just go to Taiwan and buy the semiconductors. You have to. Will it be like that with AI?

Satya Nadella 01:20:26

Ultimately, what matters is the use of AI in their economy to create economic value. That’s the diffusion theory, which ultimately, it’s not the leading sector, but it’s the ability to use the leading technology to create your own comparative advantage. So I think that will fundamentally be the core driver.

But that said, they will want continuity of that. So in some sense, that’s one of the reasons why, I believe, there’s always going to be a check to “Hey, can this one model have all the runaway deployment?” That’s why open source is always going to be there. There will be, by definition, multiple models. That’ll be one way. That’s one way for people to sort of demand continuity and not have concentration risk, that’s another way to say it.

And so you say, “Hey, I want multiple models, and then I want an open source.” I feel that as long as that’s there, every country will feel like, “Okay, I don’t have to worry about deploying the best model and broadly diffusing because I can always take what is my data and my liquidity and move it to another model, whether it’s open source or from another country or what have you.” Concentration risk and sovereignty, which is really agency, those are the two things that will drive the market structure.

Dylan Patel 01:21:49

The thing about this is that this doesn’t exist for semiconductors. All refrigerators, cars have chips made in Taiwan.

Satya Nadella 01:21:56

It didn’t exist until now.

Dylan Patel 01:22:00

Even then, if Taiwan is cut off, there are no more cars or no more refrigerators. TSMC Arizona is not replacing any real fraction of the production. The sovereignty is a bit of a scam, if you will. It’s worthwhile having it, it’s important to have it, but it’s not real sovereignty. We’re a global economy.

Satya Nadella 01:22:21

I think it’s kind of like saying, “Hey, at this point, we’ve not learned anything about what resilience means and what one needs to do.” Any nation state, including the United States, at this point will do what it takes to be more self-sufficient on some of these critical supply chains.

So I, as a multinational company, have to think about that as a first-class requirement. If I don’t, then I’m not respecting what is in the policy interests of that country long-term. I’m not saying they won’t make practical decisions in the short term. Absolutely, globalization can’t just be rewound. All these capital investments cannot be made in a way, at the pace at which… But at the same time, think about it, if somebody showed up in Washington and said, “Hey, we’re not going to build any semiconductor plants,” they’re going to be kicked out of the United States. The same thing is going to be true in every other country, too.

So therefore we have to, as companies, respect what the lessons learned are, whether it’s that the pandemic woke us up or whatever. But nevertheless people are saying, “Look, globalization was fantastic. It helped supply chains be globalized and be super efficient. But there’s such a thing called resilience, and we want resilience.” So therefore that feature will get built.

At what pace, I think, is the point you are making. You can’t snap your fingers and say all the TSMC plants now are all in Arizona with all their capability. They’re not going to be. But is there a plan? There will be a plan. And should we respect that? Absolutely. So I feel that that’s the world. I want to meet the world where it is and on what it wants to do going forward, as opposed to saying, “Hey, we have a point of view that doesn’t respect your view.”

Dwarkesh Patel 01:24:24

Just to make sure I understand, the idea here is that each country will want some kind of data residency, privacy, et cetera. And Microsoft is especially privileged here because you have relationships with these countries, you have expertise in setting up these kinds of sovereign data centers. Therefore Microsoft is uniquely fit for a world with more sovereignty requirements.

Satya Nadella 01:24:48

I don’t want to sort of describe it as somehow we’re uniquely privileged. I would just say I think of that as a business requirement that we have been doing all the hard work all these decades, and we plan to.

So my answer to Dylan’s previous question was that I take—whether it’s in the United States, or when the White House and the USG says, “We want you to allocate more of your wafer starts to fabs in the US”—we take that seriously. Or whether it is data centers and the EU boundary, we take that seriously. So to me, respecting what are legitimate reasons why countries care about sovereignty, building for it as a software and a physical plant, is what we’ll do.

Dylan Patel 01:25:40

As we go to the bipolar world—US, China—it’s not just you versus Amazon, or you versus Anthropic, or you versus Google. There is a whole host of competition. How does America rebuild the trust? What do you do to rebuild the trust? To say, “Actually, no, American companies will be the main provider for you.” And how do you think about competition with up and coming Chinese companies, whether it be ByteDance and Alibaba or Deepseek and Moonshot?

Dwarkesh Patel 01:26:13

To add to that question, one concern is how we’re talking about how AI is becoming this industrial capex race where you’re rapidly having to build quickly across all loads of supply chain. When you hear that, at least up until now, you just think about China. This is their comparative advantage. And especially if we’re not going to moonshot to ASI next year, but it’s going to be decades of buildouts and infrastructure, how do you deal with Chinese competition? Are they privileged in that world?

Satya Nadella 01:26:43

It’s a great question. In fact, you just made the point of why trust in American tech is probably the most important feature. It’s not even the model capability, maybe. It is, “can I trust you, the company, can I trust you, your country, and its institutions to be a long-term supplier?” That may be the thing that wins the world.

Dwarkesh Patel 01:27:10

That’s a good note to end on. Satya, thank you for doing this.

Satya Nadella 01:27:14

Thank you so much.

Dylan Patel 01:27:16

Thank you.

Satya Nadella 01:27:18

It’s awesome. You two guys are quite the team.

In this lecture, military historian Sarah Paine explains how Russia—and specifically Stalin—completely derailed China’s rise, slowing them down for over a century.

This lecture was particularly interesting to me because, in my opinion, the Chinese Civil War is 1 of the top 3 most important events of the 20th century. And to understand why it transpired as it did, you need to understand Stalin’s role in the whole thing.

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Mercury helps you run your business better. It’s the banking platform we use for the podcast — we love that we can see our cash balance, AR, and AP all in one place. Join us (and over 200,000 other entrepreneurs) at mercury.com

• Labelbox scrutinizes public benchmarks at the single data-row level to probe what’s really being evaluated. Using this knowledge, they can generate custom training data for hill climbing existing benchmarks, or design new benchmarks from scratch. Learn more at labelbox.com/dwarkesh

To sponsor a future episode, visit dwarkesh.com/advertise.

Timestamps

(00:00:00) – How Russia took advantage of China’s weakness

(00:22:58) – After Stalin, China’s rise

(00:33:52) – Russian imperialism

(00:45:23) – China and Russia’s existential problems

(01:04:55) – Q&A: Sino-Soviet Split

(01:22:44) – Stalin’s lessons from WW2

Transcript00:00:00 – How Russia took advantage of China’s weakness

Sarah Paine 00:00:00

People are worried about whether there’s going to be an enduring relationship with China and Russia. If you look at this picture, the relations look more glacial than cordial and the little one’s hauling on the arm of the big one. One wonders about that.

It turns out my expertise is on Russo-Chinese relations. That’s what I studied in graduate school. My dissertation was a history of their border from the Opium Wars in the mid-19th century until Outer Mongolia was snatched from the Chinese sphere of influence and parked in the Russian sphere in the 1920s. So it’s fun to talk about this particular topic.

Before I get going, I’m going to do some terminology. I’m going to use the word Russia to refer to the czarist, Soviet, and modern periods, the same way that you use France to describe its many monarchies and many republics. The Bolsheviks thought they were special, so they came up with special words for special people: Soviet, Soviet Union. But it turns out they were temporary and Russia is the enduring thing. So that’s it on terminology.

Before I speculate on what the future is going to look like, our only database that we have is whatever happened from this second backwards, what people call history. It’s just whatever is in the past. That’s it, that’s our database. So I’m going to look at when Russia was strong and China was weak, from the mid-19th century to the mid-20th century, then the reversal of the power balance, and then in the recent period when China has been strong and Russia weak.

China and Russia discovered each other late in their histories. It was the early part of their last dynasties when the Russians were after fur, very lucrative in those days. That brings them out to the Pacific. But their relations are only episodic until we get to the mid-19th century, which is where my story’s going to begin.

So in the 18th century, China was strong, Russia was weak. But that doesn’t last very long. Both empires followed the rules for continental empire. If you want to survive in a continental world— that’s what both of them historically have been—you don’t want to have two-front wars because you have multiple neighbors. Any one of them can come in at any time. If they gang up on you, that’s trouble. So you take on one at a time. Also you don’t want any great powers on the borders.

This is the fundamental problem with their relationship. Today’s friend can be tomorrow’s foe. That is truly problematic. So what do you do to solve that problem? Well, you take on your neighbors sequentially. You set them up to fail. You destabilize the rising, ingest the failing, and you set up buffer zones in between. You wait for the opportune moment to pounce and absorb. That is Vladimir Putin’s game.

But if you play this game, you’re surrounding yourself with failing states, because you’re either busy destabilizing them or ingesting them. So the curious might ask, are Russia and China unlucky with all the very dysfunctional places that surround them, or are they complicit?

Also, there are no enduring alliances in this world because the neighbors figure it out that the hegemonic power offers nothing but trouble in the long term. There’s also no counsel on when to stop expanding. So both Russia and China are known for overextending, overdoing it. That may help explain some of their periodic implosions over their long and bloody histories.

Very high mortality rates. Before you dismiss this paradigm, you’ve seen it operating in real time in Syria and Ukraine. There are people who do this. It also explains why all those ancient ruins are ruins. This sort of warfare is ruinous. But anyway, it lies at the basis of many of the great civilizations of Eurasia. This is how they did things.

I’m going to start my story in the mid-19th century, when the Chinese were beset by a whole series of rebellions that just about wrecked them. The Russians take advantage of all of this. Remember the second rule of continental empires? No great power neighbors. The Russians repeatedly derail the rise of China by scripting the Chinese to do things that are remarkably detrimental to Chinese interests, but pretty good for Russian interests. It takes the Chinese a long time to figure it out. They have governments coming and going in this period. It’s a difficult period, but they eventually figure it out by the time Mao reunifies China in 1949.

So I’m going to go through each of those examples in turn, starting with a really big one, which is the Opium Wars. This is when Britain and France are coming at China in order to force China to trade on their terms. This corresponds with the two biggest rebellions of China’s period of rebellions, the Taiping and the Nian Rebellion.

Here’s a big chart. That’s a simplified chart. The rebellions start in the late 18th century. Rebellion is a misnomer. These are civil wars.

Either people are minority peoples who want out of the empire, they want to secede, or other people who want to overthrow the government in Beijing. The peak period is in red. The really big ones are in white. So China has got the two-front war problem. It’s got Europeans coming at them, plus all of this. In fact, the Chinese have so many fronts, they don’t know how to deal with it.

The Russians come to the Chinese and say, “Hey, we can deal with the British and French for you and solve that problem. You can deal with all the internal stuff. However, we need to have you sign a couple pieces of paper for us, the Treaty of Aigun of 1858 and the Treaty of Peking of 1860.” What do they do? They cede to Russia large swaths of territory in Central Asia and the Pacific coastline.

The Qing Dynasty was vague on geography. They’re beset by these other things. They don’t understand that Europeans think these pieces of paper are permanent things. They figure that once they put their house back in order, they’re going to come back and get the territory later.

Okay, the second example of ruinous Russian mediation that is going to keep China in turmoil. So in the First Sino-Japanese War, Japan trounces China, boots them from their tributary in Korea. Then the Japanese also want some territory on the Liaodong Peninsula. The Chinese go to the Russians to help them counterbalance Japan so that Japan doesn’t take this Chinese territory in the Asian mainland.

Russia gets its buddies, France and Germany, the so-called Triple Intervention, to gang up on Japan. Japan looks at three great powers and says, “I don’t think so.” So they bail. From the Chinese point of view, so far so good. Except what the Russians promptly do is take for themselves the very territory that had just been denied to Japan.

The story gets worse. All the European powers, or many of them plus Japan, come in and they carve out big concession areas throughout China. So China’s not going to have full sovereignty over its territory for several generations. So instead of one relatively small Japanese concession area, they get foreigners everywhere. Think about the second rule of continental empire, no great power neighbors. That’s not happening while this is going on.

The Bolsheviks come to power and then they’re going to apply these rules as well. When they do come to power, they’re very weak because Russia’s been devastated by World War I. The Bolsheviks don’t win their own bitter Civil War until 1922. So then as now, they relied on a really cheap but incredibly effective strategy of strategic communication.

The Russians really understand other people’s emotional life and what sets them at odds with each other. They know just how to serve out the propaganda that sets people at each other’s throats. Their propaganda is going to help the Chinese really despise the Japanese and the Europeans, while Russia’s even greater predations, the ones you’ve already seen, go unnoticed.

Here’s Lev Karakhan. He was a deputy foreign minister. In 1919, he sends a missive, his Karakhan Manifesto, to the Chinese Foreign Ministry. He says, “Hey, we’re not imperialists, we’re Bolsheviks. We’re gonna return all the lands from those unequal treaties and be your friend forevermore, unlike all the other evil imperialist powers. We’re not like that anymore.”

The Chinese are looking at this and thinking, “Wow, here are the Bolsheviks who’ve gotten rid of their imperialistic government. They’re putting together their shattered land.” So this offers hope to the Chinese that they can do likewise. It’s a model potentially to follow and a mentor who might help them.

Except here’s the detail. When the Bolsheviks started doing better in their civil war, they really dialed back what their offer was. The original offer was, “tear up treaties, China gets all territory back, no payments necessary.” Under the new version of the Karakhan Manifesto, which the Russian Foreign Ministry goes and telegraphs to the Chinese Foreign Ministry. I’ve seen the document, or at least a copy of it in the archives in Taiwan. They send it back and say, “We’re willing to talk about these things. We’re going to hold some negotiations.”

The facts are they didn’t return these concessionaries until the mid-20th century, 1950s, after the Westerners had returned almost all of their concession areas. This is not trivial. When we think of concession areas in the age of imperialism in China, you think of British ones: Hong Kong. Well, Hong Kong isn’t actually very big. The reason you know about Hong Kong is it makes lots of money, or at least it used to.

The Russian concessionaries were huge. They didn’t make money but what else is new? But the Russians had by far the largest concession area of any other country. But from the Karakhan Manifesto is the origin of the myth of Sino-Soviet friendship, that the Russians somehow treated the Chinese nicely. And the Chinese Foreign Ministry officials, who would’ve known better, their government is overthrown within the decade. And presumably these documents just gather dust in the archives.

My fourth example of derailing China’s rise concerns the First United Front between the Chinese Nationalists and Chinese Communist Party. Here’s the leader of the Nationalist Party and also leader of its armies, Generalissimo Chiang Kai-shek. He led the Northern Expedition, reunifying China at least nominally by either defeating or co-opting all those warlords. There you can see different colors of where the major warlords were.

Previous South China attempts to do this or to secede from China, one or the other, had failed for lack of a proficient military. But Russia changes that. It provides aid, arms, and expertise and structures and things to fund the Whampoa Military Academy, which is in Canton/Guangzhou. That institution is going to educate the officers, both Communist and Nationalists, who would lead not only this, but some of their civil war-era officers that make this reunification of China possible.

But the Russians had a price. Give the Nationalists the aid, but the Nationalists then have to let the Communists into the Nationalist Party. That’s what the United Front is. This all coincides with a bitter succession struggle in Russia. This is the problem with dictatorships. They really don’t do succession well. It’s why elections are so convenient. Instead you have Stalin and Trotsky going at it over which one was going to be the big cheese.

Stalin is all about socialism in one country. He thinks that Russia ought to focus on its own internal development. Whereas Trotsky says, “Nonsense, we need to focus on world revolution because only if there are sister revolutions abroad can ours survive.” While this is all going on, the Chinese Communists really want to get out of that United Front. Why? Because it puts them in close proximity to the army, which is controlled by the Nationalists. They’re getting worried whether they’re about to get killed. The Russians say “No, no, no, no, it’s good. You stay in that United Front.” So they do.

Chiang Kai-shek goes roaring up China. I’ve shown you the map. He reaches his home base in Shanghai, pauses, and he turns his guns on the Communists and just massacres them in droves. This is when Mao has to think of a rural strategy to power because the urban strategy is no longer feasible. Once this happens, Stalin can use it to just trounce Trotsky in the internal power struggle because he can say, “Look, see, revolution in one country. It doesn’t work abroad.” A lot of Chinese died proving Stalin’s point.

Another example where Russia literally derails the rise of China. There’s a railway system in Manchuria. We’re going to talk rails. The Russo-Japanese War of 1904 and 1905, Japan wins it and gets from Russia, which built all these things in Chinese territory, the southern half of that railway net in lieu of an indemnity.

Japan invests massively in railways, infrastructure, and apparently local politicians. But the ruling warlord apparently wasn’t sufficiently attentive to Japanese needs, so they assassinated him in 1928. His son, Zhang Xueliang, the following year in 1929 decides he wants his railways back from Russia. What does Russia do? It’s not either version of the Karakhan Manifesto. The Russians deploy over 100,000 troops, tanks, airplanes, the works, and just pound this man. The Russians keep their railways. So if you want to delay the rise of China, that sort of thing delays the rise of China.

But now for the first rule of continental empire: no two-front wars. You move fast-forward to the 1930s and Stalin thinks he may very well face a two-front war with Germans in the west and Japanese in the east. Why would he think such thoughts? The Anti-Comintern Pact. Comintern is short for Communist International. It is the Soviet outreach program. It’s signed in 1936 between the Japanese and the Germans.

Stalin goes, “Uh-oh, they’re after me.” He plays every one of his China cards and he holds lots of them. If you want to disintegrate the neighbors in order to delay their rise, well then you fund all sides of their civil wars and any side in between because you just want them to go at it.

So he plays every card he’s got. What he wants are the Nationalists to stop fighting the Communists, vice versa, and unite to fight the Japanese. They’re willing to do this provided Stalin provides conventional aid, which he does, but they think he’s also going to provide soldiers. They don’t get it. Once they’re in, Russia is out of this thing.

Stalin’s plan, his script for the Chinese and Japanese, works beautifully. Because when the Nationalists unite in the Second United Front with the Communists, going back to the dark side, the Japanese are apoplectic. This is when they do the massive escalation in 1937 and they are off to overextension into parts due south of Russia.

So this two-front Japanese-German war never materialized. Stalin was very successful, the Chinese less so because the Chinese are fighting the Japanese so the Russians don’t have to. That comes at the price of millions of deaths, millions of refugees, that does indeed derail China’s rise yet again.

Next example. Per the Yalta agreement, Russia finally gets into the war in Asia. About time. In the very final weeks, in this August Storm the Russians deploy 1.5 million soldiers. It’s one of the largest military operations of World War II. They rapidly take Manchuria. That would be the normal thing, but here’s the abnormal thing. They also take away Manchuria’s industrial base. That would not normally be what you do to someone.

They take 83% of the electrical power equipment, take it home to Russia, not turning lights on in Manchuria. 86% of mining, 82% of cement making, 80% of metalworking equipment. Plus they take 640,000 Japanese POWs to be slave labor for decades, if they ever get home at all. And they also take the northern islands, which are still under dispute today.

But if you think about it, if you’re going to do indemnities or reparations or whatever this is, China had been fighting Japan in one form or another for 15 long years. Russia comes in for the cameo performance at the very end. So if there are indemnities to be paid for whatever Japan did in this war, surely China, not Russia, should have been the recipient for all this stuff.

In addition, there’s another example. Not only does Stalin walk away with the industrial base, but he walks away with Mongolia as well. How does that work? The Yalta agreement also stipulates that the status quo shall be maintained in Mongolia. So then you have to look at, well, what was the status quo? It was the Russian sphere of influence in the north, Chinese continuing control in the south. Mongolia, which had always been both those places, had been part of the Qing Empire, never been part of the Russian Empire. Moreover, Stalin had already taken Tannu Tuva in 1944. It looks small on this map, but it’s bigger than England. It had lots of gold, which the Soviets had monetized long ago.

So if you add up all the territory that the Russians took from the Chinese sphere of influence from the 1858 Treaty of Aigun and 1860 Treaty of Peking, fast-forward to detaching Outer Mongolia from the Chinese sphere of influence. Here’s what it really is. It’s greater than all US territory east of the Mississippi. This is not your normal land grab. So talk about derailing somebody, that would do it.

To be fair to the Russians, they did, albeit slowly, turn over all this Japanese-stockpiled military equipment in Manchuria to the Communists. They also, albeit belatedly, trained them on how to use the equipment and also how to run the Manchurian railway system. The Chinese Communists, as a rural peasant movement, how would they know how to do these things? They wouldn’t.

It allows this conventional aid and logistics of being able to move people around. It transforms the Communists from a lightly armed guerrilla movement to a very heavily armed conventional force capable for the showdown phase of the Chinese Civil War. So like the Whampoa Military Academy, this is essential aid for the Communist victory.

So neither the Communists or the Nationalists complain about the deindustrialization of Manchuria. The Communists probably traded that industry for all the conventional aid that they got. The Nationalists traded it and also Outer Mongolian independence for a promise from Stalin not to aid the Communists, a promise that he promptly breaks.

So Mao starts to figure out that something is up here. So when he’s on a roll in his offensives in the civil war, there’s really bitter fighting. The real movement in the last phase, the post-1940 phase of the Chinese Civil War, is in 1948. That year Mao just moves and he is roaring down south. He’s about to get to the Yangtze River and Stalin’s like, “Hey buddy, take a break at the Yangtze, don’t exhaust yourself.” Mao ignores it. Whereas Stalin might have wanted to keep a Nationalist rump state south of the Yangtze River, yielding a divided China in keeping with weakening your neighbor, Mao is not remotely interested in that.

Here’s my 10th example, which is the Korean War. If you look at the Korean War, the first year is a war of movement. It’s up and down, up and down the peninsula. It’s unbelievable how much movement there is. But then it stalemates for the last two years and you think, “Well, what’s going on? Why don’t they settle the war sooner?” Both sides are taking incredible losses.

Well, here’s how it goes. Once China intervenes in the Korean War and once they halt various offensives to start peace talks, the Chinese do incredible tunnel work, probably the North Koreans as well, and build an incredible tunnel system. It means the South Koreans and the UN forces are never going to get anywhere near the Soviet border ever again.

From that moment on, Stalin thinks he’s got a low-risk, high-reward strategy where he’s going to weaken the United States and delay the rise of China. So what’s not to love about fighting to the last Chinese in Korea? Stalin thinks this is great and it’s going to retard Chinese development. Also because China is so isolated by this war, it has no international friends but Russia, it’s going to tie China to Russia ever more firmly and give Russia breathing space to rebuild after World War II while its Western enemies are wasting time in Korea.

If you put it all together—Chinese Civil War, Korean War, Russia’s on-and-off again aid to different sides in the civil war, his double-dealing with both of them and what happened with Outer Mongolia and the Manchurian industrial base, Stalin’s advice to Mao to halt at the Yangtze, and then he’s fighting to the last Chinese in Korea—this is all consistent with the second rule of continental empire: no great power neighbors.

00:22:58 – After Stalin, China’s rise

Sarah Paine 00:22:58

Once Stalin dies, finest day of his life, there’s never as strong a leader in Russia again. By this time, Mao has figured out that the Russians don’t want a strong China. He has to bide his time for a while, but he understands what is going on. Mao has a growing list of gripes. It’s not only that he didn’t like Stalin’s tributary treatment, but also Mao thinks, with his resume, that he should become the leader of international communism.

Stalin’s successor, Nikita Khrushchev, is like, “No way.” Khrushchev does not remotely have Mao’s resume. Mao has just put together a continent by reunifying China. That’s not remotely what Khrushchev’s ever been able to do. Mao also can’t stand either Khrushchev’s domestic or foreign policy.

Domestically, Khrushchev is all about de-Stalinization. Mao doesn’t like that, he’s got a cult of personality. He doesn’t want to do things like that. Khrushchev is also interested in peaceful coexistence with the West, or at least nominally, whereas Mao is in the midst of the Cultural Revolution, which is based on a virulently anti-Western foreign policy. They’re also forever squabbling about who’s aiding North Vietnam in the Vietnam War and who’s going to get credit for it. So all that’s going on.

Now, Khrushchev has his own gripes about the Chinese. He looks around at the United States. The United States has got basing all over the world. Its allies allow the United States to have bases. China has hardly any Russian bases, these leftover concession areas, and China wants them back. Khrushchev can’t understand this.

What he really can’t understand are the two Taiwan Strait crises of 1954 and 1958, where Mao started lobbing ordnance on Nationalist islands that are very close to mainland shores. Khrushchev is apoplectic because Mao hasn’t given him any advance warning. By the way, this sort of thing could trigger some of the security clauses of the Sino-Soviet Friendship Treaty with nuclear follow-on effects.

Then while this is going on, Khrushchev wants to have a combined sub fleet base. If we’re going to take all these risks, we need to have subs in different places. Mao says, “No way.” Then Khrushchev has had enough and says, “Well, you’re not going to get the plans for the atomic bomb.”

The whole thing becomes public in 1960 with anger all around. The two, Russia and China, squabbled incessantly over the Vietnam War. The North Vietnamese were interested in getting maximum aid from both, which it did. They begged the Chinese to allow the Russians to ship things overland through China. The Chinese felt obliged to do it, but the terms of that overland trade were just food for all kinds of squabbling. It just didn’t end.

So the story of the reversal in the balance of power between Russia and China arose from multiple factors. It doesn’t happen all at once. It is both a story of China’s rise and of Russia’s decline. Step one for China is getting its own atomic weapon, which it does in 1964, so that it can get itself free of Soviet bondage.

In 1964, after they detonated that bomb, Mao goes, “Okay, there are too many places occupied by the Soviet Union. The Russians took everything they could. We have not yet presented an account for this list of stolen territory.” All the territory I’ve shown you, “We want it back.” Russians’ jaws drop, panic.

The Chinese then are much more aggressive about what they consider to be their territorial rights. There is a border war over territories, and in particular this one island, Zhenbao, or Damansky Island in the Amur River. Here’s how riverine borders work, international ones. Under international law, the border is the thalweg, which is the center of the main channel of whatever the river is. Russia had followed that with its European borders, not with China. It claimed both banks of the Amur.

The Chinese are fed up with that and they take Zhenbao Island and the Russians are furious and they come to the United States. They say, “Would it be okay if we nuke these people?” We’re like, “No.” The Russians scratch their heads and they come back and they go, “Okay, would it be okay if we use conventional weapons to blow up their nuclear stuff?” We go, “No, still not okay.”

Mao gets it. The one that wants to nuke you is your primary adversary. So there’s a reshuffling of the primary adversaries. Formerly the United States was the primary adversary of both China and Russia. That would be a reason for them to cooperate. Now they’re primary adversaries for each other. The United States can play the swing thing with all of this.

For Russia it’s really devastating having China as an enemy because it’s going to have to deploy mechanized nuclear-armed troops all along its really long Chinese border, Central Asia, the works. It’s already doing this with its European borders and occupying Eastern Europe where the garrisoning costs are significant. Imagine if this country had to put those kinds of forces on our long Canadian and Mexican borders. It’d be bankrupting. Their economy was and remains a fraction of ours.

But this breaking up of the earlier version of the bromance allows the United States to play the swing role. Then we cooperate. Both Nixon and Mao think ganging up on Russia would be a good thing, overextending Russia financially by overextending it militarily with all these armaments and things.

In addition, what’s going on as part of China’s rise are internal reforms under Deng Xiaoping when he abandons certain communist principles of economic management and gets a much more productive agricultural sector and industry and commerce. So China’s running double-digit growth rates for about 20 years with significant compounding effects.

That’s the story of China’s rise. Now for this dystopian alternate universe of Russian decline. Here on the left is Leonid Brezhnev, who apparently had a stroke in 1976 that permanently impaired his thinking and whose death in 1982 finished him off. He was replaced by Yuri Andropov, whose own health was pretty parlous and he dies within two years. Then Konstantin Chernenko barely makes it a year before he’s dead. It sort of sounds like us, but anyway it doesn’t work well.

If you look at Soviet growth statistics, they’re really good right after the end of World War II. They’re busy rebuilding. But when you get to the mid-’70s, they’re going into terminal decline. When Gorbachev comes to power in 1985, Soviet growth rates have been 1–2% less than U.S. growth rates for the preceding decade. The compounding effects of that are pretty horrendous.

In addition, there are other problems. Leonid Brezhnev was in power for 18 long years and not only did he collect cars, that was apparently the go-to gift for him, but in addition he was collecting a bunch of non-performing piles across the Third World because this is manifesting Russian power. But it’s expensive and he doesn’t have much of an economy to pay for it.

If you look at oil prices, they’re very much associated with the decline of the Soviet Union, rise of Vladimir Putin. It’s because in the Soviet era, government budgets relied as much as 55% on oil or energy revenues. So piggy banks are going to shrink if oil prices aren’t doing very well. Gorbachev repeatedly said, “You know, we can’t live this way anymore.”

He wanted, and he did initiate political and economic reforms to try to save communism, except he really ruined the sclerotic patient doing what he did. There are massive territorial implications for what he did. It’s the loss of empire in Eastern Europe and also the loss of the ethnically-based constituent republics of the Soviet Union.

I’m going to show you Russia’s territorial odyssey in maps. So this is 1938, before the Molotov-Ribbentrop Pact, the Nazi-Soviet Pact where they’re dividing up Poland and parts of Eastern Europe – Nazis’ and Soviets – of what they want. This is before then, but by the time you get to the end of the war, Russia’s got the Baltic states, it’s got Kaliningrad, it’s got all of Belarus, all of Ukraine. Then of course it gets Eastern Europe.

Gorbachev works his magic and you’re down to Kaliningrad. That’s where Russians stage a lot of weapons nowadays, apparently dump a lot of toxic waste. That’s what Kaliningrad’s all about. So it was significant. At the end of 1991, when Russia’s lost everything, they’re down to a much diminished rump state. That’s followed by years of instability in Russia.

00:33:52 – Russian imperialism

Sarah Paine 00:33:52

Russians agree that their country always has been, always should be, always will be a great power, but they don’t measure it in wealth. Their wealth has always been much less than their Western neighbors, although they often confiscate the wealth of other people. What they measure their strength, their greatness, is in vast territorial extent and the ability to run roughshod over others and make them do whatever it is Russia wants.

Also when they look at their security, they also look at it this way. “We need this vast territorial extent to be secure.” But they never turn it around. They’re always worried about other people invading them. “Do you suppose we pose a threat to anybody else?” They never turn it around that way. Except Russia has posed existential threat to its neighbors forever. There are so many neighbors you have never heard of because they’ve disappeared from the pages of history, courtesy of the Russians.

Let’s go to the medieval period where Russia starts out as the princely state of Muscovy, Moscow. Well, it wipes out the other princely states. There was Novgorod the Great. It was a more progressive place. They wiped that place out, Rostov, there are a lot of other places. Later they’re eliminating the Khanates of Central Asia. These are states. It’s a different way of organizing yourself. The Khanate of Crimea, Kazan, Astrakhan, Kokand, Khiva, Bukhara, they get rid of all of it.

Then there’s been this repeated vivisecting of European neighbors—Ukraine, Poland, Lithuania, Sweden and Finland—taking their territory one bite at a time. You can see it going on today. The Russians just don’t see that if you do this to other people… This is why at the end of the Cold War, everyone is stampeding into NATO. It’s not some conspiracy, it’s just what the Russians have done to them.

Now I’m going to illustrate Poland’s fate to show what happens when you’re surrounded by rapacious continental empires. Three partitions of Poland in the 18th century. This is when Prussia, Austria and Russia are taking things. The Russians say, “But we didn’t take Polish territory.” They’re taking the Polish Empire. It’s a technicality. If you wonder why Poles cannot stand Russians, it would be this. It would also be the genocide-laced occupation of Poland that went on for several generations after World War II.

Part of that, it’s Yalta again. The Russians insisted on moving Poland 200 kilometers to the west. That is not the normal thing to do to a country. It’s deep into German territory, so Russia can go eat a big part of Poland. Then Russia decides it’s going to cleanse the whole place, ethnically cleanse. So Germans are going to live in Germany, Poles in Poland, Ukrainians going to live in Ukraine. Millions of people—this is a war-devastated Europe where there’s no way to take care of refugees—are being sent hither and yon, and lots of them are dying.

For the first time in their histories, Czechoslovakia, Germany, Hungary, Poland, Ukraine, they become ethnically homogeneous states. Poland, Hungary, and Czechoslovakia require Russian troops to hang onto some of their current gains. So a lot of things go on here.

If you compare it to what the West is doing in the United States, the United States has a Marshall Plan, you’ve heard of it. The United States is pouring aid into our Western allies to restore them after the war. Well, this is what Russia’s doing. They’ve won the war, so their troops are there, but they never take their troops home. Then they’re busy running coups. Then they’re busy shooting anyone who disagrees with them or they send them to labor camps, whatever they’re going to do. Then they redistribute every form of wealth there is, either through nationalization or collectivization. They’re imposing this really non-functioning economic model on these people. It’s called communism. It doesn’t work. You wonder why Eastern Europeans aren’t thrilled about a repeat of this. Well, come on.

Moreover, the story gets better. The people who made all this happen include this charmer, some of the most toxic people from the Soviet Union. Andrey Vyshinsky goes to Romania, lucky Romanians. He was running the show trials that were sending the original Bolshevik leaders to summary execution. What does he do when he gets to Romania? He appoints the Ministers of Justice, War and Interior. If you do that, you control the courts, army and police. You can start expropriating everything and nationalizing things. You can start eliminating other parties. You make Romania trade only with the Soviet Union.

Another toxic personality, Marshal Kliment Voroshilov. He’s another part of the A-list purge team. What had he done? He signed loads of execution warrants to his military colleagues, officers, during the Great Purges. What a great guy. He’s in Hungary, lucky people. He’s not quite as competent as the other one. In the elections of 1945 in Hungary, the Smallholders Party wins. Oops, they aren’t Communists. He doesn’t ever make that mistake again.

Then he’s busily appointing Communist ministers to different ministries, the land reform, taking over the civil service, getting the propaganda up and running, taking over youth and women’s groups to infiltrate that way, and arresting all sorts of non-Communist politicians before he holds the 1947 elections, which of course the Communists win. Afterwards there’s a really big purge where 10% of the population faces tribunals. Think of that. Of this room, 10% of you are going to tribunals because Russians are in town.

Russia had a template for bloc building. I’m giving the details because you get a sense of how a continental empire runs business. The Russians looked at how when Italy surrendered to the Western Allies, they basically just ran the show in Italy. So the Russians are going, “Well, we’re going to, when we accept surrenders, run the show in Eastern Europe.”

But the Russians want to take advantage of the absolute turmoil that wars cause. Wars also often lead to war economies where you’ve got a lot of centralized government control to run the war effort. So the Russians want to leverage that one. Then they’ve got this big Red Army presence. They want to use all of those things, which they do.

Here’s their template. They’re controlling the power ministries. What are they? Defense and Interior. If you do that, you’re going to monopolize coercion and eliminate your opposition. If you control the Justice and Information ministries, you can arrest or kill at will. Then you control all the stories that are never told about this. If you control Agriculture, particularly in those days, you’re going to buy a lot of allegiance by redistributing other people’s lands. That’s going to wipe out class enemies, collaborators, all sorts of people you don’t like, and the original elite that have been running rural areas.

They also had a story that they told about this. It’s the big lie. Remember communism? What are the really bad people in the communist story of how we all live? It’s the imperialists, right? Imperialism is really bad. Well, I’ve just shown you. Russians were the greatest practitioners of imperialism of the 20th century. They’re doing this right as Western imperialists are giving up their colonies.

Yet there’s this big lie, the big fiction that the Russians are somehow nice guys. They’re using democratic forms. They don’t eliminate Poland as a state, as the czars had done. It’s still called Poland, but it’s a fiction that it’s independent. It’s not. There’s a fiction that they hold elections, but they’re not democratic. We already know the outcome. It’s a dictatorship in reality. They’re just purging the place. They’re killing lots of Eastern Europeans. If you think Eastern Europeans don’t remember this, they most certainly do.

Then there is a more regional template for how you do this. I’ve already talked about ethnic cleansing. They did that. Then they tried moving countries, Poland to the west, Mongolia, I haven’t talked about it, it was to the east in Chinese territory. Then they want to take some stepping stones for later expansion. That’s what Kaliningrad is all about. Another one is Moldova. That is territory between Romania and Russia on very important river systems. Moldova’s got problems with a place called Transnistria, which the Russians have poached. That story’s still ongoing.

Also, places like Azerbaijan. It’s split between Iran and…well now its independent Azerbaijan, but it was split with the Soviet Union. Then there are the divided states of Germany, Korea, Mongolia, and China. You want all your neighbors quarreling over their borders so that Russia can then set the terms and that it can nibble them away a bite at a time, or if it can take a whole thing, good for them.

So Russian national identity is not only about empire, territorial extent. It also has some big ideas attached to it, an ideology that somehow this territorial expansion is either progressive, beneficial or positive. It’s a myth. Under the czars, the ideology was this Third Rome, of the Russian Orthodox Church. The Third Rome was in Moscow. I know this is news to you. I’ll explain how it works.

There’s the Rome we all know about, the “Rome” Rome. That was Rome number one. The one where the Pope is. Rome number two, according to the Russians, has to do with the Byzantine Empire. That was Constantinople, or Istanbul. When that falls, the Russians go, “Ta-da, it’s Moscow.” Not really, but they used this Russian Orthodoxy and spread the Russian Orthodox Church back in the day deep into Eastern Europe.

When the Communists get in, they’re not spreading Russian Orthodoxy, they’re spreading communism. The problem for Putin is that today neither communism nor Russian Orthodoxy are marketable ideologies. He’s just stuck with being a really big place. That’s where his little focus is.

Russia has a nightmare scenario. They don’t always win these wars. They lose quite often. The Mongol yoke or the Yellow Peril, their terminology, not mine, is the 13th century when the Mongols just swept in and the Russian elites became tax collectors for the Mongols, an extractive role that has endured. You think about Russians, they extract resources, their own, other peoples, but they’re not known for producing wealth. It’s just resource extraction. This is some of the legacy of all of that.

The Russians have also suffered devastating defeats, to Napoleon, in World War I and World War II, that just devastated Western Russia. So they’ve had bad times. In our own time, Putin is dumping all of his ordnance on Ukraine, leaving Siberia wide open to China’s ambitions. So if he keeps the game up, he may well wind up with a Chinese yoke, and his nightmare will be there for him.

00:45:23 – China and Russia’s existential problems

Sarah Paine 00:45:23

The Chinese also have their own big ideas, not religious ideas or empire, but about civilization itself. Traditionally, the Chinese believe there’s only one civilization, that would be theirs. It’s based on Confucianism. That would be a world order unto itself. That worked for them for a couple thousand years.

There are certain pillars of legitimacy that have endured, probably originating from this, but enduring ones. One is ethical rule. Well, that’s gone for the Communist Party. Let’s try the next one. Economic prosperity. Whoops, going fast. That leaves the Communists today with territorial expansion. That’s ongoing with territory incursions into India, South China Sea island building, and then all these gathering threats to Taiwan.

In part, what’s going on is that the Communist Party wants to play the nationalism card the same way Putin is. If you haven’t got anything else for you, play jingoistic nationalism, because we human beings seem to be particularly attentive to that one. So if you focus on that, maybe the Chinese people won’t look at the ethical lapses of the Communist Party or the fact that their paychecks aren’t going anywhere anymore.

Good old Xi, he doesn’t have a marketable ideology anymore. What’s really scary is that Confucianism had just been this enduring feature of China—dynasties came and went, but Confucianism stayed—until China was unable to fend off Japan in the First Sino-Japanese War that I told you about and the Europeans in the Opium Wars, when all of a sudden Confucianism just seemed totally inadequate.

When enough Chinese cease believing in Confucianism, that’s when you get the 1911 Revolution and it just vanishes as an organizing principle for running your government. This is the total nightmare. One of the frightening things for the Communist Party is when people cease believing in communism. China’s nightmare scenario is these horrible periods of chaos. They’re afraid that if the Communists go, that China’s going to devolve into these civil wars, these periods of Luàn (乱).

A second one is that maybe the Soviet Union when it’s shattered, that may well be their fate, that maybe these communist regimes can’t last forever. So Russia’s nightmare scenario is other people invading Russia. China’s nightmare scenario is the collapse of China. It’s two different ways, two different things to worry about.

The Chinese face a conundrum. If you no longer believe the economic theories on which communism is based, how do you justify one-party rule? The Communist Party has tried to soldier on without solving that problem. The Chinese have learned a great deal from watching Russians as they play around with big ideas. They learned a great deal from Mikhail Gorbachev when he tries to fix communism, but he winds up killing the patient. The Chinese, I think, their takeaways from what Gorbachev did is don’t hesitate to deploy the tanks. If you’ve got unrest in the streets, you just send tanks, tanks against civilians. It’s a really quick fix. You want to focus on economic reforms to the extent you can, certainly not political reforms.

You also really want to sinify your minorities. Why? In the Soviet Union, the Russians had had this fiction that all these occupied minority people wanted to be there and had equal rights. So they would sponsor all the nice folk dances and the language classes, and they’d have a bunch of token minorities in traditional dress and various political institutions. But basically they had no power. It meant when the Soviet Union shattered, there were plausible divisions already set up on an ethnic basis. That is how you get places like Uzbekistan, Tajikistan, et cetera. China’s like, “No way.” This helps explain the genocide of the Uyghurs that’s ongoing now. This is their rationale for doing it.

Then what you want to do, you can see Xi Jinping doing it, is prioritize maintaining the monopoly of the Chinese Communist Party over economic efficiency when those two things run at cross purposes. That helps explain some of Xi Jinping’s growth-depressing choices that he’s making.

Gorbachev discovered when he tried to reform communism that democracy and communism just don’t mix. If you give people elections, they’re going to support multiple parties. When the communist track record becomes known—their incompetent economic management, summary executions, famines, just all the things that they’ve been up to—it’s very hard for them to remain in power. Also, as Gorbachev discovered, democracy and empire don’t mix. When you give different minorities a choice, or occupied places a choice to leave or stay, they bolt.

There’s a consensus among the Han, the preponderant ethnic group in China, that they want to maintain the empire. So they don’t care how many Tibetans immolate themselves. That’s no longer going on now because the mortar has descended into Tibet. They don’t care how many Mongols complain about lost lands. The Han want to keep it. Also for the generation that survived the Cultural Revolution, which would be Xi Jinping, he’s precisely that age group, they do not ever want to go back to the chaos of the Cultural Revolution. It was horrendous. So expect them to prioritize stability over liberty.

This brings a third problem. Democracy and communism don’t mix and democracy and empire don’t mix. In an Internet cell phone age, where people are much more interconnected, how does a government maintain legitimacy to rule? Because elections are an incredibly powerful way to give authority to a government. This is where nationalism comes in. The Chinese Communist Party and the Russians likewise are trying to use jingoistic nationalism to stoke up popular loyalties, to deflect people from domestic problems and go, “Look at the hated outsiders.” They aren’t the only country to do this sort of thing, blaming outsiders. It’s all the immigrants’ fault or whatever.

The problem with all of this, when the Chinese did it in 1980, right after the Tiananmen massacre, the Chinese Communist Party decided to rewrite all the textbooks. So they’re no longer going to focus on class enemies, but they’re going to be focused on using jingoistic nationalism. They’re going to focus on the evil Japanese, evil Americans, and focus things that way. Well, the problem with nationalism is it’s a very heady drink. If you imbibe too much, it clouds the judgment. Moreover, it repels minority people within China’s empire who are not interested in Han nationalism. It frightens neighbors to coordinate with each other and find big powers to counterbalance this mess. It also impedes the de-escalation of unforeseen international dust-ups or crises.

With Putin’s latest adventures, one would assume that the Chinese have some real thoughts about what’s going on now. In Putin’s case, he has thrown away the Soviet addendum to the rules for continental empire. What was that one? No hot wars. If you think about Soviet rulers, they were all veterans of World War II through Leonid Brezhnev. They understood that war is easy to get into, hard to get out of, very unpredictable. For them it was proxy wars. They loved it when the United States got into hot wars. What is not to like about the Korean War and the Vietnam War with the United States just tied down and Americans tearing each other’s eyes out about these things?

Great, great, great. Russians stayed out of it until after Brezhnev’s stroke. That’s when they make the big boo-boo in Afghanistan. They go into Afghanistan and then all of a sudden we have loads of fun giving Stinger missiles to the other side, inflicting costs on the Russians. The person who wanted to get into Afghanistan is Yuri Andropov, who was not a veteran. The uniformed military of Russia said, “Don’t go into Afghanistan.” Andropov and friends ignored it and did their thing.

If you look at Putin, he has risen to power on a diet of hot wars. He comes to power in the early phase of the Second Chechen War, where he levels the Chechen capital of Grozny and leaves most of the rest of Chechnya an environmental waste zone. But he sorts that one out. Then he gets quite popular for his war with Georgia in 2008, where he detaches South Ossetia and Abkhazia from Georgia. Now he wants to build a naval base in Abkhazia because he’s having trouble with his base in Crimea.

Then when he eliminates term limits for himself in 2012, which isn’t very popular in Russia, he solves that by going into Ukraine in 2014 and walks off with about 7% of Ukrainian territory at very little cost. And Russians think that’s great! If you look at these little stepping stones, this is the way continentalists look at it. You’ve got all these little places lined up. You want to color it in. That’s where Ukraine comes in, in 2022.

Putin wants to reverse all these territorial losses.

He wants the green things and the purple things. I don’t know if he wants to do the Napoleonic Wars and go all the way to Paris. Who knows? Here’s how he looks at NATO or doesn’t look at it. You look at when people join. There are two great big periods of accession to NATO. One is in the early Cold War with all the smaller countries of Europe, which is everybody. We’re all smaller than Russia.

The Europeans join to protect themselves from Soviet imperialism. Then at the end of the Cold War, when these Soviet satellites can finally slip the leash, I’ve given you all the reasons why, they stampede into NATO. It is not NATO tricking them to join. They’re stampeding in for excellent reasons. They’ve been proven right. Putin, when he works his magic the second time around in Ukraine in 2022, this is when the Finns and the Swedes decide neutrality’s not going to cut it and they’ve got to join NATO.

But when Putin looks at this, he goes, “Oh, they’re coming at us in arcs.” Well, this is ludicrous. Russia has posed an existential threat to these places forever. No one today wants to invade Russia. Who’d want it? It’s full of Russians. We want them to stay home. There’s no one there who wants Russia. The Russians like to gaslight everybody else, like, “You’re the problem.” It’s been very effective. Lots of Americans will talk about NATO expansion and say, “Oh, this is why Putin went to Ukraine. You did NATO expansion.” He’s gaslighting everybody.

Russians are the problem. Also it is their totally dysfunctional domestic system that offers everybody else nothing but problems. Instead of focusing on that and fixing it so they can be a productive member of the rest of the world, they’re not doing that. They’re just invading other people.

Back to the real problem, what are the prospects of a bromance with these two? Barring World War III, which would be the great powers going at it… If the great powers go at it, you might well superglue these two. If that happens, whatever side is losing might well use a nuclear weapon, and then we’re going to have toxic plumes everywhere.

So we had better get our diplomacy right because there are really serious consequences for getting it wrong in this particular era. We’re at the beginning of a second Cold War with a leadership that has not been chastened by surviving World War I, the Great Depression, and World War II. Which were the grownups who did a very effective job dealing with the early first Cold War. We haven’t got those people anymore.

So if you look at primary adversaries, primary theaters, you don’t want them to align for your enemies, because if they do, they’ll coordinate. Currently, Vlad the Bad has a Ukraine fixation. Xi Jinping has got a Taiwan fixation, opposite ends of Eurasia. Xi Jinping is also causing India problems and doing South China Sea island building. But they’re on opposite ends of their two empires. So far, nothing aligns with these two. Even in a world war, their primary theaters probably would not align in the same way that the Axis primary theaters did not align.

They might well disperse their resources in different directions. But the problem is we have a lot of people with nuclear weapons now. You really don’t want to do the hot war. If the West and others manage things correctly, I’ve given you a huge historical legacy of fraught relations. Play your cards appropriately and these two will take care of each other. The rest of us can, if we do it right, not launch trade wars on each other, but rather try to maximize our prosperity so that we get stronger and stronger while they’re busy in these wealth-destroying wars. Putin is just blowing through all of his assets in Ukraine to no purpose. That’s how we fought and won the last Cold War.

Putin, he’s trying to do empire in the age of nationalism. It’s a non-starter. He’s dumping all of his ordnance on Ukraine, leaving Siberia wide open. Xi Jinping has moved right in even before the Ukraine war. The Belt and Road Initiative is peeling away the former Russian sphere of influence in Central Asia.

Putin is doing this hot war gambit when Russia’s weak and China is strong, making it particularly damaging. Why would it matter? Siberia has precisely the resources that China now needs and covets and wants to have them contiguous so other people can’t mess with them. In particular they want water because they’ve blown through their water table in North China. Lake Baikal has over 20% of the world’s surface fresh water. So if you want a quick fix, it’s the best one. China is known for big water projects. They’ve been damming the Mekong and the Yangtze. They’ve done massive water projects. So that may be in Russia’s future. Who knows?

Xi Jinping now holds all of the cards. China has nine times the population, nine times the GNP of Russia, and their per capita GNPs are converging. Not good news for Putin. The question isn’t whether this bromance is going to last forever, but rather when it’s going to end. When is Xi Jinping going to decide he’s got the right amount of leverage to get whatever it is he wants?

What’s going on now illustrates the perils of getting your primary adversary wrong. Nicholas II fought a recreational war against Japan, the Russo-Japanese War of 1904-1905. He is pouring resources into really an irrelevant theater in those days, instead of taking those resources and putting it into a railway system that European Russia desperately needed to face off Germany, which it does within the decade and loses.

He is overthrown in part because of bread riots in St. Petersburg, because no foreign aid can get in, because he has inadequate railway system. They’ll be built later and it’ll be good for Lend-Lease. So he and his family wind up shot in a Siberian basement. It takes a decade to play out. I don’t know how it’s going to go for Vladimir Putin on all of this. If we play our cards right, maybe he’ll go down as lonesome. We may have to wait a long time.

On the other hand, if we get into a hot war, nuclear war, and in the meantime, if the United States kills off its alliance system which is one of its greatest strengths, by doing a gratuitous trade war on our allies, then we have a dust-up with China… Well, let’s think about this. Japan tried this go-it-alone, my-way-or-the-highway strategy with China back in World War II, when China was a failed state, and it wrecked Japan.

China now has a population that’s a multiple of ours, an industrial base that’s a multiple of ours. Why on earth would you ever want to take them on alone? So I’m going to quote Pericles, who was the leader of Athens at its height when it was building all these ancient ruins that weren’t ruins when they were being built. They’re gone. This warning came on the eve of the succession of Athenian blunders and ancient Athens never recovered: “I am more afraid of our own blunders than of the enemy’s devices.” Our leaders truly need to ponder this before they wreck all of us, themselves included. So thank you for your attention. That’s what I have to say about Sino-Soviet things.

01:04:55 – Q&A: Sino-Soviet split

Dwarkesh Patel 01:04:55

First question, I want to understand the role of ideology in the Sino-Soviet split. These are the two major communist countries in the world. They have this big ideological, world-changing mission. We’re going to spread communism around the world. That isn’t enough to prevent war over, in the case of 1969, literally just some islands in the middle of a river. There’s nothing super consequential at stake here. Why didn’t the fact that they were both these communist countries do more to cement the relationship?

Sarah Paine 01:05:24

I gave you all these bad things that Russia had done to China, which are not small things. Then isn’t the problem with communism that the economics of it don’t actually work? Fundamentally these two countries are continental powers, they’re continental empires. Sure they change out ideologies. The ideology of the tsars changes out to communism and China is changing out Confucianism for communism. But the principle that each one should be king of the roost hasn’t changed. That would be a mutually exclusive proposition of who’s going to run Asia and then Eurasia, because they have a shared address, this long border.

Also with communism, it’s supposed to take over the whole world. We’re supposed to have a classless society. That’s why we don’t even worry about governments and things because they’re eventually going to melt away. Well, that’s all nonsense. As things go on further and further, you have dictators more and more entrenched and they’re using communism. There’s another piece where it’s just amazing, about the big lie. I don’t know whether it was Stalin who said it, that if the lie is big enough, people will believe you. It’s just so out there, it’s so preposterous, they go with it. Which is this notion that it’s the West through these imperial powers, right? And we buy this, that they’re not the ones. But if you look at it, I’ve given you the data. Then this other one in our own time, this lie about, “Oh, it’s the West, it’s NATO expansion, this explains Putin.” I’ve given you the data, it’s nonsense.

Now why the Russians feel obliged to take other people’s territory, part of it is feeling good while other people squirm. There’s that aspect. They’re not the only people who do those sorts of things. But it’s also from being on the plains of Eurasia where, throughout history, people have invaded you. So you have an established paradigm of how you deal with other people that is deep-seated over thousands of years. I can’t totally answer the question because there are a lot of Russians and a lot of Chinese whose decisions together aggregate to what they do.

Dwarkesh Patel 01:07:53

I want to understand Stalin’s decisions in particular in the case of telling the Communists during the Chinese Civil War that they can’t go below the Yangtze River, so that he can split up the Chinese, the Communists and the Nationalists in China, and leave China weak. We know from all the actions in Stalin’s life that he was a devoted communist. He does collectivization and almost destroys his regime because he’s a devoted communist.

At the same time, communism says that there has to be this worldwide revolution at some point. If there is a worldwide revolution, there will be other powerful communist countries. That’s implied in the nature of what a worldwide communist revolution is. But at the same time he doesn’t seem communist enough to want China to become fully communist. He cares more about real geopolitik. Tell me about how this works.

Sarah Paine 01:08:45

His version of communism, the Russians invented it. Actually they didn’t, Marx invented it. But they are the guys who think, “Well, we operationalized it so we should run it forever.” So of course Russia is going to be the big communist country. A bunch of Chinese upstarts, from their point of view, are claiming, “No, no, no, we’re going to run the show.”

Dwarkesh Patel 01:09:06

But did Stalin as a communist think, “Okay at some point the whole world is going to be communist.” He thought that he personally would be managing the whole world?

Sarah Paine 01:09:18

Unclear. It’s a long horizon. In his lifetime he’s initially about communism in one country, then communism on your borders, which is what he’s working on. As for the rest of the world, it takes Brezhnev to start going all deep into Africa. Stalin wasn’t interested in, say, India because he thought they’re a bunch of lackeys of the British. How do they let themselves be colonized by the British? There’s something wrong with them.

You have to also think about where history is at a given moment. He dies in ‘53, not that long after World War II. How is it all going to turn out? His relationship with China, as far as he’s concerned, is going swimmingly when he’s in power. Mao goes there and kowtows in Moscow and does what Stalin wants him to do. That’s Stalin’s experience until he, I don’t know, chokes on his Cheerios or whatever happened to him that morning.

Dwarkesh Patel 01:10:19

If Stalin hadn’t died, do you think a Sino-Soviet split was still inevitable? Yes, there wouldn’t have been Khrushchev’s secret speech. Obviously Stalin wouldn’t have condemned the cult of personality. But the grievances you mentioned would still exist.

Sarah Paine 01:10:35

I think the grievances would still exist. Dictators’ deaths profoundly weaken their systems because they’ve got no good succession. It means you’re guaranteed a cat-fight royale going on. That’s what one waits for with Putin. Eventually the man will die one way or another, and there will be a cat-fight royale for what happens after him.

Stalin was pretty old when he died and Russians in those days had a lousy diet. I can’t remember if he smoked. His generation mostly did. So I don’t know how the future would have run with good old Uncle Joe there for longer. The Korean War would have still gone on longer. It’s really his death, why that war ends then. With the Korean War going on and on, I don’t know what the implications are for us or for Mao.

Dwarkesh Patel 01:11:32

This is a really interesting thing to consider from this lecture. In retrospect, maybe one of the most important things, if not the most important thing that happened in the 20th century, is the Chinese Civil War. At the time, you wouldn’t think of this as a major event, especially since World War II had just happened. If you were living in 1948 or 1949, you wouldn’t think of this as the main thing happening in the world.

But given that we know in retrospect this is a really big country with a huge population, we should be paying a lot of attention to its internal political development. I wonder if we should apply the same logic today where there are other countries with huge populations where we don’t think a lot about what’s happening. India or Nigeria. I really couldn’t tell you the first thing about Nigeria even though I know that it’s a big country with a huge population.

Sarah Paine 01:12:16

Let’s not try because I don’t know anything about it either.

On China and the Chinese Civil War, there was an understanding that it was a really big deal. That’s why Roosevelt—and the British are just thinking he’s laughable—keeps trying to treat China like a great power. The British are saying, “They’re not a great power, Franklin, not remotely.” But he wants to bring them into the Cairo Conference. He also wants them to be a veto-wielding member of the United Nations, which by China’s military status there’s just no way.

But Roosevelt is looking, there’s no Japan out there, we want to have something in Asia to counterbalance. But then when you get to the Chinese Civil War, Americans who’ve fought some big wars, they’re looking at it saying, “It is not feasible for us to alter this outcome.” The thinking was — I think it was the Eisenhower archives — that if they lived together long enough, they would go at each other eventually.

Dwarkesh Patel 01:13:31

I want to zoom in on one particular episode during this period that I think is fascinating. In 1936, the Communists in China kidnapped Chiang Kai-shek, who’s the leader of the Nationalists, and they’re about to kill him. Stalin radios in over the Comintern. He says, “No, you can’t kill Chiang Kai-shek even though he’s your enemy, he’s been massacring you. You gotta let him go.” In exchange, Chiang Kai-shek has to promise to create a united front against the Japanese.

There are 30,000 communist guerrillas when they’re fighting the Japanese at this period and there are 1.5–2 million nationalist troops. The communists get to claim the prestige of fighting the Japanese with a fraction of the effort. This just seems like a brilliant move on Stalin’s part in retrospect, because if they had killed Chiang Kai-shek, the Nationalist forces might have dissolved. There would be a puppet government installed by Japan. It would be a terrible thing for Stalin.

It just seems like Stalin makes a lot of right calls in the theater in Asia at this time. What is the understanding, big picture, that lets him?

Sarah Paine 01:14:31

There are dictators all over the world who stay in power for a long time. Do not dismiss their ability to stay in power. They’re surfing quite a wave wherever they live, of people who butcher each other with great regularity. I would suspect—suspect, because how would I have the evidence—that many of these long-term dictators have EQs that are off the charts. When you come in the room, they just gut-feeling know whether they’re going to cap you or not. If they get a few extra people, so what? Just so they get rid of all the people who are going to cause them problems.

Incredible abilities. Also what Stalin does—I think Saddam Hussein imitated him, but I could be wrong about that one—is have multiple security agencies with overlapping jurisdictions. No one quite knows who’s running the show and who’s in power, but they’re all funneling information to Stalin. So he has better information than anybody else. You can use one one day, another another day, which explains how all the heads of the equivalent of the KGB come and go. Most of the early heads of that wind up getting shot by their successors. You’d think they’d notice a pattern.

They’re very good at staying in power. They’re really good at managing that problem. It does not bring prosperity. That comes from the world of maritime trade, of following international law instead of capping your trade partner. You’re not going to get a second trade deal if you do that. It’s just so much more wealth-creating, this alternate universe, but that’s not where these continentalists are.

Dwarkesh Patel 01:16:20

There’s still something a little under-explained about…

Sarah Paine 01:16:23

Probably lots of things are under-explained.

Dwarkesh Patel 01:16:26

Hitler’s another dictator. Forget about having these moves like deciding, “Don’t kill this kidnapped person because it’ll be a bad idea in the long run.” On the opposite end, he’s declaring war against America even when he doesn’t have to. He’s just making these crazy moves. Stalin is a special thing to explain. It’s not just communism. It’s something else that’s making him a deft geopolitical actor.

Sarah Paine 01:16:51

You can do this in retrospect and go, “crazy move,” because no one realized that the US productive base could do what it did. Americans didn’t even know. You tell me right now how the future’s going to turn out. This is where if you flip it…

When you look at history, you think it had to turn out that way because you can reverse-engineer something, at least a plausible story of why you wind up where you did. It may not even be the right story, but it yields the correct answer. So you think you got it. But if you look at the future, how’s this thing going to end with Xi Jinping and Putin? We don’t know. Afterwards, you’ll go, “Oh, something is profoundly stupid based on subsequent events.” But we don’t know now.

There’s a Russian proverb which is: Чужа́я душа́ – потёмки (chuzhaya uma potyomki). Someone else’s mind, potyomki, darkness. You ultimately don’t know what other people are thinking. You make your guesses, but then they do things.

Dwarkesh Patel 01:17:54

When you showed the maps of China in 1850, Russia also in 1850, especially 1900, Russia and China are still big countries today. But if you just look at Qing dynasty China, it’s overwhelmingly big. It’s the same with Tsarist Russia before the Brest-Litovsk Treaty.

It makes you think that at various points through history, at most points through history, China was technologically dominant over the West, except during this one crucial period in the 18th and 19th century with the Industrial Revolution in high gear.

Sarah Paine 01:18:28

Which changes the whole world.

Dwarkesh Patel 01:18:30

Right. It reinforces the idea of how important it is to be at the technological frontier, especially at historically crux-y times. Of course we’re in San Francisco. Many of us here believe that this is especially a crux-y time in the development of technology.

Sarah Paine 01:18:45

Maybe we should continue investing in it instead of canceling all the science research projects. What will be the long term effects of doing this? You’re thinking of America’s great strengths, its alliance systems. Our great strength also used to be being at the head of innovation. We’re going to eliminate those things and think it’s going to go well for us?

Dwarkesh Patel 01:19:09

I want to ask about your, I don’t know if you call it a prediction but, your hypothesis that at some point China and Russia will finish each other off or have some sort of conflict which will be bad for both of them.

Sarah Paine 01:19:22

They won’t finish each other off. They’re too big to be finished off. Unless we do a nuclear war, then we might finish all of ourselves off.

Dwarkesh Patel 01:19:28

But the idea that there will be some sort of conflict between these two countries… I agree with the idea that China has a lot of people and a lot of wealth. Russia has a lot of water, and it has a lot of resources. These are complementary resources. They both have things that the other party has.

The question is, will they attain these things through war or through trade? It seems very improbable to me that… China’s got a lot of money. It can just pay for Russian oil, it can just pay for water. Why would it invade a nuclear-armed nation?

Sarah Paine 01:20:00

I never said it would do that. It’s a question of if Putin feels that his back is up against the wall. What price? We don’t really know the terms of their trade. What are the real energy prices that China’s paying for all of this? I don’t know. Maybe some people do know. I suspect that whatever the terms of trade are going to be, the Chinese are going to get a really good deal.

If you look at what happened to Gorbachev at the very end of the Cold War. He’s not going to war with anyone, but his internal situation is devolving so rapidly because their economy collapses. He’s already got lousy oil prices. In addition, he has so upset the centralized planning system such that revenues are just collapsing around him, and he’s about to be thrown out of power. He is desperate for the kind of loans that the Germans in particular are willing to give him in order to get, not only Germany reunited, but West Germany staying in NATO. The next thing is East Germany is going to be in NATO too, and then NATO can have whomever. Those things are trades he never would have made ever, but he’s so desperate.

In that case, the West likes to pay people, so instead of paying him a low price, the West is paying him a high price to back off. In China, I suspect it’ll be the opposite. They pay him less and less for whatever it is he wants if he really continues to corner himself in Ukraine. These terms of trade will get more and more in China’s favor.

Dwarkesh Patel 01:21:52

Does that look like the bromance ending? North Korea has for decades been in a very desperate position.

Sarah Paine 01:22:00

That’s quite a bromance.

Dwarkesh Patel 01:22:02

But there’s no risk of North Korea and China going to war. North Korea relies heavily on China. It’s not draining China in any way.

Sarah Paine 01:22:14

It’s the other way around. That’s what Russia’s future may be. Apparently certain Russians, when Putin had invaded the big way in Ukraine, said, “Oh my God, that’s our future. It’s North Korea. That’s where we’re headed.” That could well be their future. It’ll just keep going as they pour out wealth and the Chinese will try to get very good terms of trade for resources and be niggardly, meaning giving very few things back.

01:22:44 – Stalin’s lessons from WW2

Dwarkesh Patel 01:22:44

In 1931, when Japan attacks Manchuria, Japan commandeers the Chinese Eastern Railway, which the Treaty of Portsmouth in 1905 said was supposed to be under Russia’s purview. People in the Politburo are telling Stalin, “Look, we’ve got to be aggressive against this.” Stalin says, “Look, I don’t want to raise tensions against another great power. Let’s just let this slide, let’s keep tensions low.”

This is quite similar to what happened with splitting up Poland in 1939 and then Barbarossa, where in both theaters he makes this calculation, “I’m going to let certain things slide so that I don’t have to face off against these great powers on my border. I don’t want to go to war with them.” In the case of Japan, it works because Japan decides to attack China and not Russia. In the case of Germany, it doesn’t.

Sarah Paine 01:23:32

Yeah, in Germany, he’s trying to run the same script. He thinks it’s going to work for him. That’s what the Molotov-Ribbentrop Pact is about. Not remotely. He needed to read Mein Kampf to understand, “No, you’re a menu item for Hitler. He’s eventually going to come around.”

In our own day we don’t really want to absorb the bad news. I could be wrong, but I believe that Putin wants not only all of Ukraine, which he says he wants, but also he wants the Baltic states and he wants to keep on going. This is profoundly bad news. If that’s the case, then it means for Europeans at least, they’re going to have to divert all kinds of resources to this problem, which is something they don’t want to do. Because like us, they’re indebted too, and there are other things they would prefer to spend their money on.

We don’t like the bad news that this is where we’re at. You have to make nasty trade-offs. Stalin tries to work the same magic and thinks it’s going to work for him and he doesn’t get it. He becomes the menu item for Hitler and it just about wrecks him. But it works beautifully against Japan.

Dwarkesh Patel 01:24:57

It’s just interesting to see. It makes sense why he thought it would work. When he’s getting information that the Germans are lining up millions of men along this huge border, he says, “This is British disinformation.” He uses the exact same logic when Japan is invading Manchuria, that the idea that he had to fight Japan was British disinformation. Even though it was a mistake, it helps us understand why he thought this way.

Sarah Paine 01:25:25

There’s another concept that’s useful: cooperative adversary. I’ve introduced this before. It doesn’t mean the adversary wants to cooperate with you, but they just don’t play their cards particularly well.

We’re talking about China that has had all of those rebellions I told you about and civil wars and regional wars. It doesn’t have these strong government institutions. It is much easier to script-write a place like that, to have them do things that are antithetical to their interests. You’ve got a cooperative adversary. The Chinese aren’t trying to cooperate, but from a Russian point of view they might be a cooperative adversary. Whereas with Hitler, Germany has all kinds of institutions and all kinds of well-educated people. Hoodwinking Hitler is going to be much more difficult than whoever he’s dealing with in China in this period.

Dwarkesh Patel 01:26:20

An interesting connection to make between World War II and this Russo-Chinese relationship is that the way that we used Russia—as this reservoir of military men that we can just ship armaments to and they can do the dying for us—that’s similar to how Russia uses China in the war against Japan. Well, they’re not actually during that war shipping that much to China, but at least during the Korean War.

Sarah Paine 01:26:51

Korean War, yes.

Dwarkesh Patel 01:26:54

Interestingly, Russia is doing to them the kind of alliance that we had with Russia during World War II.

Sarah Paine 01:26:59

Think about today. It’s like the Korean War in reverse because China’s dribbling out aid to Russia. Russia’s getting wrecked by Ukraine. Russia’s got three times the population or something of Ukraine, but the death rate that it’s suffering is really high. These are only children. This is no longer the 10 children, six children, peasant families in Russia.

Look at it from Xi Jinping’s perspective, the longer this goes on, the better. Just let Putin keep on working his magic and I get to sell him stuff. I’ll get him to lower the prices of resources because the man’s desperate. Putin will find himself under a Chinese yoke that he does not like. Then at some point the Russians will reassess and I don’t think they’ll like that. The Russians fancy themselves being Europeans and have a whole, I believe, racist package that goes with the Chinese so that they won’t like that whole situation. I don’t know how long it takes them to do whatever they’re going to do.

A lot of things in foreign policy, you’re not going to solve it. They’re huge countries, there’s no way you’re going to solve it. But you need to figure out how to manage it, to have a blast shield so that whatever fallout comes from their toxic, whatever they’re doing, it minimizes how it hits your friends and partners. This is why you should be focusing on maximizing the economic growth of your friends and partners, because that is the only effective way to deal with them.

If you go it alone, who deals with a bully alone? Always gang up on them. Why would you ever want to go alone against a bully? You’d want to go in with lots of friends. This is what the World War II generation did. The Marshall Plan, you could never get something through like Congress now. Yet it’s passed overwhelmingly in Congress because people get it that you have to spend real money so the European economies recover. How are they gonna do it? Buying our stuff. It’s a tremendous win-win. In strategy, you wanna figure out win-win things instead of these zero-sum things where, “Oh, I invite you over, I humiliate you, I feel good,” and then you’re mad forever. It’s pointless.

Dwarkesh Patel 01:29:32

Great note to close on.

Sarah Paine 01:29:34

Don’t humiliate people. Be kind to each other, yes.

Dwarkesh Patel 01:29:38

It’s a good callback because I’ve been asking you all these naive questions and you’ve been very kind to not humiliate me.

Thank you all very much for coming. It is so fun to be able to do these things with a live audience and not trapped inside a dark studio room. I hope we see you guys again on Friday when we’re doing the lecture on the proper way to do war termination. And next Tuesday is about why Russia lost the Cold War. See you on Friday. Thank you, Sarah.

Sam Altman says he wants to “create a factory that can produce a gigawatt of new AI infrastructure every week.”

What would it take to make this vision happen? Is it even physically feasible in the first place? What would it mean for different energy sources, upstream CAPEX in everything from fabs to gas turbine factories, and for US vs China competition?

These are not simple questions to answer. We wrote this blog post to teach ourselves more about them. We were surprised by some of the things we learned.

Subscribe now

The fab CapEx overhang

With a single year of earnings in 2025, Nvidia could cover the last 3 years of TSMC’s ENTIRE CapEx.

TSMC has done a total of $150B of CapEx over the last 5 years. This has gone towards many things, including building the entire 5nm and 3nm nodes (launched in 2020 and 2022 respectively) and the advanced packaging that Nvidia now uses to make datacenter chips. With only 20% of TSMC capacity1, Nvidia has generated $100B in earnings.

Suppose TSMC nodes depreciate over 5 years - this is enormously conservative (newly built leading edge fabs are profitable for more than 5 years). That would mean that in 2025, NVIDIA will turn around $6B in depreciated TSMC Capex value into $200B in revenue.

Further up the supply chain, a single year of NVIDIA’s revenue almost matched the past 25 years of total R&D and capex from the five largest semiconductor equipment companies combined, including ASML, Applied Materials, Tokyo Electron...

We think this situation is best described as a ‘fab capex’ overhang.

The reason we’re emphasizing this point is that if you were to naively speculate about what would be the first upstream component to constrain long term AI CapEx growth, you wouldn’t talk about copper wires or transformers - you’d start with the most complicated things that humans have ever made - which are the fabs that make semiconductors. We were stunned to learn that the cost to build these fabs pales in comparison to how much people are already willing to pay for AI hardware!

Nvidia could literally subsidize entire new fab nodes if they wanted to. We don’t think they will actually directly do this (or will they, wink wink, Intel deal) but this shows how much of a ‘fab capex’ overhang there is.

Upstream suppliers to datacenters will have to expand production

For the last two decades, datacenter construction basically co-opted the power infrastructure left over from US deindustrialization. One person we talked to in the industry said that until recently, every single data center had a story. Google’s first operated data center was across a former aluminum plant. The hyperscalers are used to repurposing the power equipment from old steel mills and automotive factories.

This is honestly a compelling ode to capitalism. As soon as one sector became more relevant, America was quickly and efficiently able to co-opt the previous one’s carcass. But now we are in a different regime. Not only are hyperscalers building new data centers at a much bigger scale than before, they are building them from scratch, and competing for the same inputs with each other - not least of which is skilled labor.

Even McKinsey thinks these CapEx numbers are not crazy ($6.7T CapEx cumulative through 2030 implies around $2T in 2030).

For you to deploy $2 trillion of AI CapEx a year, someone else needs to be producing $2 trillion worth of all the other datacenter components, not just the chips. The people upstream to the datacenters - the companies producing everything from copper wire to turbines to transformers and switchgear - would need to build more factories.

The problem is that those factories have to be amortized over a many decade lifespan. At usual margins, those factories are only worth building if this AI demand lasts for 10-30 years.

The companies building those factories are not AGI pilled - they’re decades old industrials running low margin businesses which have been burned by many previous swings in demand. In the early 2000s, electricity demand seemed set to explode, so gas turbine manufactures like GE, Siemens, and others massively expanded manufacturing capacity. Then demand collapsed, leaving them with huge (almost bankrupting) overcapacity.

If there’s a financial overhang not just for fabs, but also for other datacenter components, could hyperscalers simply pay higher margins to accelerate capacity expansion? Especially given that chips are currently 60-70% of datacenter CapEx, the hyperscalers might just tell the companies which are building the other 30% not to worry about long run demand: “We’ll make you whole with just a couple of years of outrageous margins.”

The largest gas turbine manufacturers (GE Vernova, Siemens Energy, and Mitsubishi Electric) are expecting to make $100B from gas turbines over the next 5 years. That corresponds to around 100GW of generation capacity. These companies are doing a combined CapEx of around $5B/yr across all their divisions.

If the hyperscalers were willing to pay $200B instead of $100B for this 100GW of generation (for example, to incentivize faster delivery), they’d effectively be covering 20 years of these turbine manufacturers’ entire CapEx (at current rates).

To build 100 GW of datacenters, the hyperscalers are going to have to invest many trillions of dollars anyways. Power generation is only about 7% of a datacenter’s cost. If natural gas ends up being the fastest way to scale generation, then doubling the cost of generation to 14% in order to make sure the power comes online fast enough might be easily worth it.

We think a similar dynamic is probably true for most of the components in a data center. Do not underrate the elasticity of supply.

The coming labor bottleneck?

Labor might actually end up being the most acute shortage - we can’t simply stamp out more workers (at least, not yet).

The 1.2 GW Stargate facility in Abilene has a workforce of over 5,000 people. Of course, there will be greater efficiencies as we scale this up, but naively that looks like 417,000 people to build 100 GW. And that’s on the low end of 2030 AI power consumption estimates. We’re gonna need stadiums full of electricians, heavy equipment operators, ironworkers, HVAC technicians,... you name it.

For reference, there’s 800K electricians and 8 million construction workers in the US. We hear that this labor pool is aging fast, but at least over the next few years, it seems like reallocation and big salary offers should be able to ameliorate the labor bottleneck.

$400B+ ARR by end of decade is plausible

Anthropic and OpenAI’s combined AI CapEx per year (being done indirectly, mostly by Amazon and Microsoft in 2025) seems to be around $100B2.

Revenues for OpenAI and Anthropic have been 3xing a year for the past 2 years. Together, they are on track to earn $20B in 2025.

This means they’re spending 5 times as much on CapEx as they’re earning in revenue. This will probably change over time - more mature industries usually have CapEx less than sales. But AI is really fast growing, so it makes sense to keep investing more than you’re making right now.

Currently, America’s AI CapEx is $400B/year. For AI to not be a bubble in the short term, the datacenters currently being built right now need to generate $400B in revenue over their lifetime. Will they?

Google, Facebook, etc. have already shown us that if you can make a product which is modestly useful to billions of people, you can generate $100s of billions in revenue a year (Google+Meta make $400B/yr from ads alone).

OpenAI is approaching 1 billion, unmonetized free users, and we think a $12B to $100B revenue scale up is plausible just from their current products (e.g., see this vision: GPT-5 Set the Stage for Ad Monetization and the SuperApp). The question lies more in whether they can make a GPT-6 (or other products) in 3-5 years time that looks promising and economically useful enough to bring them into the $400B+ revenue range.

Of course, questions of revenue ultimately come down to your timelines. If AI truly lives up to its promise, then it’s in the reference class (at the very least) of white-collar wages, which are $10s of trillions of dollars a year.

Do you think that AI models will be able to do much of what a software engineer does by the end of a decade? If the 27M Software engineers worldwide are all on super charged $1000/month AI agent plans that double their productivity (for 10-20% of their salary), that would be $324B revenue already.

Lead times

It takes about two years to build a new GW+ datacenter—and that’s before factoring in time to debug new chip generations. This means if you want to deploy 2 trillion in capex in 2030, you need to plan that out in 2028. At current trends in perf/watt and perf/$, $2T corresponds to roughly 66 GW of AI datacenter capacity.

For the last few years, hyperscalers and labs have consistently wanted more compute than they had previously made plans to develop. If the hyperscalers are planning for a 30% CAGR over the next 5 years and instead end up wanting to average 40% (hitting $2T capex in 2030) that’s a 20GW gap they’ll have to close in 2030 relative to the long term plans.

Elon (who didn’t even have an AI company during the time in which a traditional hyperscaler would have had to precommit to building capacity) solved his constraints by doing insane things. It’s unclear whether mere mortals can assemble 60 GW of greenfield capacity when they’d only planned for 30 or 40 GW.

If AI demand continues outpacing advance planning, there will be enormous pressure to compress datacenter construction timelines. The question is: what does this mean for energy sources and datacenter design? Some energy sources have much longer lead times than others, and some datacenter designs are more amenable to rapid deployment.

Chips overwhelm everything else in the total cost of ownership of a datacenter. This is because 1. Chips are really expensive, and 2. The data center shell can be depreciated over 12-20 years, whereas the chips are fully depreciated (and have to be replaced) every 3 years.

So in terms of choosing an energy source, you can see very clearly what the order of priority should be.

1. Lead times - Every month that the shell is not set up is a month that the chips (which are the overwhelming majority of your cost) aren’t being used.

2. Non chip CapEx - Much more expensive than electricity OpEx over a 3 year period.

3. Electricity OpEx

So you can see why natural gas, for example, is much preferred over current nuclear reactors. Nuclear has extremely low op-ex, but has extremely long lead times and high CapEx. Natural gas may not be renewable, but you can just set up a couple dozen gas turbines next to the datacenter, and get your chips whirring fast.

Solar panels themselves are very cheap, but can be expensive to levelize across night + seasons. If you’re going pure solar, you have to build a bunch of overcapacity (4-7 GW solar capacity for 1 GW data center due to typical 15-25% capacity factor) and add a ton of batteries. Otherwise, you’re risking your expensive chips sitting idle during winter or at night.

Solar farms also have a massive land footprint, and require massive labor for installation – good luck hiring 30,000 people to lay out 20,000 acres of solar panels and batteries across the desert in order to power a single 1 GW datacenter3. The largest solar park in the world is currently the Gonghe Talatan Solar Park. It generates enough electricity for around 3 GW of smoothed continuous power - but this requires 15 GW peak capacity - meaning 7.2 million solar panels - that’s the area of seven Manhattans.

Aside from lead times and costs, you could ask the question: which energy source is physically plentiful enough to supply this demand? The answer, it turns out, is literally all of them. The theoretical limits of any of the energy sources mentioned are orders of magnitude higher than what is needed for even the most explosive end of decade AI scenarios.

Off grid?

A key question is whether datacenters will go “off-grid”—generating power on-site rather than connecting to the utility grid. Some of the largest datacenters are already doing this, e.g., Meta’s Orion or XAI’s Colossus.

Why would datacenters want to make power themselves rather than relying on the grid? They’re trying to get around interconnection delays. Connecting large new electricity sources to the grid now takes over 5 years.

For 20+ years, US electricity consumption has been either flat or growing slowly. Grid operators are now expecting huge increases in demand from AI datacenters, manufacturing reshoring, and electrification all happening simultaneously.

ERCOT’s Annual Energy Forecast. Who knew they were so AGI pilled?

One potential workaround: a Duke study found that if datacenters agreed to curtail load just 0.25% of the time (roughly 22 hours per year), 76 GW of spare transmission capacity could be made available. Most transmission lines run well below capacity on average—the bottleneck only hits during peak demand.

But even if hyperscalers are able to perfectly capture this 76 GW, that only gets you from 2026-2028 in bullish AI scenarios. After that, either the grid expands or datacenters go off-grid.

Distribution of datacenter sizes

What will the distribution of individual datacenter sizes be? Here’s the argument for why we might end up seeing what looks like a thick sprinkle of 100 MW datacenters everywhere:

• If you can plop down a medium sized datacenter here and there, you can soak up any excess capacity in the grid. You can do this kind of arb with a 100 MW datacenter, but there’s no local excess capacity in the grid at the scale of 1 or 10 GW - that much power is on the scale of a whole grid itself!

• For pretraining like learning, you want to have large contiguous blobs of compute. But already we’re moving to a regime of RL and midtraining, where learning involves a lot of inference. And the ultimate vision here is some kind of continual learning, where models are widely deployed through the economy and learning on the job/from experience. This seems compatible with medium sized datacenters housing 10s of thousands of instances of AIs working, generating revenue, and learning from deployment.

Here’s the other vision. 1-10 GW datacenters, and then inference on device. Basically nothing in between.

• If we move to a world with vertically integrated industrial scale production of off-grid datacenters, maybe what you want to do is just buy a really big plot of land, build a big factory on site to stamp out as many individual compute halls and power/cooling/network blocks as possible. You can’t be bothered to build bespoke infrastructure for 100 MW here and there, when your company needs 50 GW total. A good analogy might be how a VC with billions to deploy won’t look at any deal smaller than deca millions.

A machine that spits out a GW a week

Today’s datacenter construction resembles building a car in your driveway: the engine ships from Germany, transmission from Japan, harness from Detroit, and a mechanic spends months assembling it all on-site. Each datacenter is custom-built over 1-2 years, with networking, MEP systems, and racks assembled piece by piece.

You’re not getting to a gigawatt per week this way.

Could you have pre-fabricated compute halls? Fully wired racks, cooling systems, power equipment, batteries—assembled in factories and shipped as complete modules. Instead of 18 months of on-site construction, you’re sliding skids into place.

The design space is surprisingly flexible. Liquid cooling enables 500kW-1MW racks but requires totally different plumbing and construction. If solar dominates, maybe you go full DC-to-DC (panels generate DC, chips need DC) and skip all the AC conversion steps. Each design choice cascades into others—power source affects cooling approach affects rack density affects building design.

By the way, if AI turns out to be a bubble and we’re much further from AGI than Silicon Valley thinks, what lasting value gets built? You could tell a story about how the dot-com bubble paved the way for all the value the internet has generated. What’s the equivalent for this AI buildout?

The GPUs—70% of capex—are worthless after 3 years. The buildings and power infrastructure last decades but are overbuilt for non-AI workloads. Perhaps the enduring value is this new industrial capability: the ability to rapidly manufacture and deploy massive compute infrastructure on demand. Like how the dot-com bubble left us with fiber in the ground, maybe an AI bubble leaves us with an industrialized datacenter supply chain and an expanded electric grid.

Crypto actually did this for the current wave of AI. For example, Crusoe - which is helping OpenAI build out Stargate in Abilene - was founded to build Bitcoin mining datacenters on stranded natural gas.

Does China win by default on long timelines?

Why doesn’t China just win by default? For every component other than chips which is required for this industrial scale ramp up (solar panels, HV transformers, switchgear, new grid capacity), China is the dominant global manufacturer. China produces 1 TW of solar PV a year, whereas the US produces 20 GW (and even for those, the cells and wafers themselves are manufactured in China, and only the final module is assembled in the US).

Not only does China generate more than twice the electricity than the US, but that generation has been growing more than 10 times faster than in the US. The reason this is significant is that the power build out can be directed to new datacenter sites. China State Grid could collaborate with Alibaba, Tencent, and Baidu to build capacity where it is most helpful to the AI buildout, and avoid the zero-sum race in the US between different hyperscalers to take over capacity that already exists.

Is China privileged in the long timelines world? SMIC will probably *eventually* catch up to TSMC (and maybe SMEE or SiCarrier to ASML, CXMT and YMTC to SK Hynix and Micron, NAURA and AMEC to Applied Materials, LAM, Tokyo Electron and KLA) - export controls won’t preserve the lead forever. If there’s not a software only intelligence explosion before 2030, and AI just becomes a massive industrial race across the entire supply chain from robotics to solar panels and batteries to steel, then why doesn’t China end up leading? Isn’t China’s differential advantage precisely these kinds of rapid and massive infrastructure build-outs?

Semianalysis projects that China will actually be able to ship fewer chips next year than it did this year, mainly because of domestic HBM production constraints. But we wonder how much that matters in the long to medium term.

GPUs are fully depreciated over 3 years (because new designs and better underlying process nodes make previous generations irrelevant). And lead times on building new datacenters are only a year or two long - datacenter design as a whole might be redone to accommodate industrial scale vertical integration.

All this makes us wonder whether every 3 years, the AI race starts totally fresh. While we might be able to constrain China’s production up till 2028 (or maybe even into the end of the decade with the lithography chokepoint), what’s the story for why this matters 2030 onwards?

Two scenarios - AI winter, and AI explosion

In order to get a handle on questions like these, we think it’s really helpful to just put numbers into a spreadsheet. Even if the scenarios are pulled out of your ass, they give you some sense of what would have to be true about the world in which they came to pass. For example, it’s interesting to ask: given trends in hardware price and performance, how much power would $2T CapEx correspond to in 2030? How about $500B?

We decided to chart two different potential trajectories:

1. Explosive growth, where AI investment grows smoothly and then starts to accelerate on the back of booming economic growth from AI automation (30% GDP growth in 2035).

2. AI winter, where Investment growth crashes around 2029 and then grows at a smooth 5% annual growth rate from 2032 onwards.

Here’s AI CapEx and AI power up to 2040 in both scenarios4:

In the explosive growth scenario, Sam Altman’s vision of 1 GW a week for the leading company comes true in 2036. But in that world, global AI power draw would be twice US’s current electricity generation.

We had a lot of fun building and playing around the spreadsheet that led to these scenario projections. You might too.

Concluding thoughts

Romeo and I decided to write this blog post because we had lots of questions about the AI buildout. We’re left with many more questions.

We don’t have any hard conclusions. What we’ve managed to do is assemble some considerations relevant to answering our questions.

There is no one person in the world who is the expert in all of the relevant fields. And even if there was such a person, they wouldn’t have a crystal ball. So we’ve spent a lot of time hoping on calls, reading PDFs, and talking to LLMs.

If you do have more information or thoughts, please reach out to us. We’re eager to learn more.

Share

About the authors

• Romeo is a researcher at AI Futures Project working on writing AI scenarios and recently graduated from Harvard. For their first scenario he focused on compute forecasts – now he’s also thinking about broader economic impacts for a new scenario with longer timelines and a positive vision for governance. If you have expertise in economics and an interest in future AI scenarios, please reach out at [email protected].

• Dwarkesh is a podcaster. He’s keen to interview someone really good on all these topics. Reach out at [email protected].

  Subscribe now

1

In 2023 and 2024 Nvidia had grown to 11-12% of TSMC’s revenue (source 1, source 2). This year TSMC expects $110B total revenue, 26% higher than last year’s $88B (breaking from the 8% average increase the previous 2 years). Assuming this increase (26% vs. 8%) is from AI revenue, and in 2024 AI was already 15% of their revenue, that gives us an estimate of around $30B TSMC 2025 AI Revenue.

2

OpenAI and Anthropic look on track to get access to around 3 GW of combined AI capacity this year (the exact number is obscure, you’d have to buy Semianalysis’s model to know exactly). At around $30/W of chip-only CapEx ($3.9M per 132kW NVL72 GB200 rack) that’s about $100B.

3

These figures assume solar tracking systems that rotate panels to follow the sun throughout the day. Austin Vernon points out that fixed-tilt systems (panels just mounted on the ground) can be packed more densely and installed much faster—potentially requiring only ~6,000 acres and a few thousand workers versus 20,000 acres and 30,000 workers. The tradeoff is lower capacity factor, but you can compensate by installing more panels. Panels are cheap—installation time is expensive.

4

For these scenarios, we’re assuming Moore’s Law continues, and that there’s continued improvements in power efficiency and cost efficiency of chips (albeit all three trends proceed at rates slightly slower than historically). Check out the spreadsheet for a full accounting of assumptions.

The Andrej Karpathy episode.

Andrej explains why reinforcement learning is terrible (but everything else is much worse), why model collapse prevents LLMs from learning the way humans do, why AGI will just blend into the previous ~2.5 centuries of 2% GDP growth, why self driving took so long to crack, and what he sees as the future of education.

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Labelbox helps you get data that is more detailed, more accurate, and higher signal than you could get by default, no matter your domain or training paradigm. Reach out today at labelbox.com/dwarkesh

• Mercury helps you run your business better. It’s the banking platform we use for the podcast — we love that we can see our accounts, cash flows, AR, and AP all in one place. Apply online in minutes at mercury.com

• Google’s Veo 3.1 update is a notable improvement to an already great model. Veo 3.1’s generations are more coherent and the audio is even higher-quality. If you have a Google AI Pro or Ultra plan, you can try it in Gemini today by visiting https://gemini.google

Timestamps

(00:00:00) – AGI is still a decade away

(00:29:45) – LLM cognitive deficits

(00:40:05) – RL is terrible

(00:49:38) – How do humans learn?

(01:06:25) – AGI will blend into 2% GDP growth

(01:17:36) – ASI

(01:32:50) – Evolution of intelligence & culture

(01:42:55) - Why self driving took so long

(01:56:20) - Future of education

Transcript00:00:00 – AGI is still a decade away

Dwarkesh Patel 00:00:00

Today I’m speaking with Andrej Karpathy. Andrej, why do you say that this will be the decade of agents and not the year of agents?

Andrej Karpathy 00:00:07

First of all, thank you for having me here. I’m excited to be here.

The quote you’ve just mentioned, “It’s the decade of agents,” is actually a reaction to a pre-existing quote. I’m not actually sure who said this but they were alluding to this being the year of agents with respect to LLMs and how they were going to evolve. I was triggered by that because there’s some over-prediction going on in the industry. In my mind, this is more accurately described as the decade of agents.

We have some very early agents that are extremely impressive and that I use daily—Claude and Codex and so on—but I still feel there’s so much work to be done. My reaction is we’ll be working with these things for a decade. They’re going to get better, and it’s going to be wonderful. I was just reacting to the timelines of the implication.

Dwarkesh Patel 00:00:58

What do you think will take a decade to accomplish? What are the bottlenecks?

Andrej Karpathy 00:01:02

Actually making it work. When you’re talking about an agent, or what the labs have in mind and maybe what I have in mind as well, you should think of it almost like an employee or an intern that you would hire to work with you. For example, you work with some employees here. When would you prefer to have an agent like Claude or Codex do that work?

Currently, of course they can’t. What would it take for them to be able to do that? Why don’t you do it today? The reason you don’t do it today is because they just don’t work. They don’t have enough intelligence, they’re not multimodal enough, they can’t do computer use and all this stuff.

They don’t do a lot of the things you’ve alluded to earlier. They don’t have continual learning. You can’t just tell them something and they’ll remember it. They’re cognitively lacking and it’s just not working. It will take about a decade to work through all of those issues.

Dwarkesh Patel 00:01:44

Interesting. As a professional podcaster and a viewer of AI from afar, it’s easy for me to identify what’s lacking: continual learning is lacking, or multimodality is lacking. But I don’t really have a good way of trying to put a timeline on it. If somebody asks how long continual learning will take, I have no prior about whether this is a project that should take 5 years, 10 years, or 50 years. Why a decade? Why not one year? Why not 50 years?

Andrej Karpathy 00:02:16

This is where you get into a bit of my own intuition, and doing a bit of an extrapolation with respect to my own experience in the field. I’ve been in AI for almost two decades. It’s going to be 15 years or so, not that long. You had Richard Sutton here, who was around for much longer. I do have about 15 years of experience of people making predictions, of seeing how they turned out. Also I was in the industry for a while, I was in research, and I’ve worked in the industry for a while. I have a general intuition that I have left from that.

I feel like the problems are tractable, they’re surmountable, but they’re still difficult. If I just average it out, it just feels like a decade to me.

Dwarkesh Patel 00:02:57

This is quite interesting. I want to hear not only the history, but what people in the room felt was about to happen at various different breakthrough moments. What were the ways in which their feelings were either overly pessimistic or overly optimistic? Should we just go through each of them one by one?

Andrej Karpathy 00:03:16

That’s a giant question because you’re talking about 15 years of stuff that happened. AI is so wonderful because there have been a number of seismic shifts where the entire field has suddenly looked a different way. I’ve maybe lived through two or three of those. I still think there will continue to be some because they come with almost surprising regularity.

When my career began, when I started to work on deep learning, when I became interested in deep learning, this was by chance of being right next to Geoff Hinton at the University of Toronto. Geoff Hinton, of course, is the godfather figure of AI. He was training all these neural networks. I thought it was incredible and interesting. This was not the main thing that everyone in AI was doing by far. This was a niche little subject on the side. That’s maybe the first dramatic seismic shift that came with the AlexNet and so on.

AlexNet reoriented everyone, and everyone started to train neural networks, but it was still very per-task, per specific task. Maybe I have an image classifier or I have a neural machine translator or something like that. People became very slowly interested in agents. People started to think, “Okay, maybe we have a check mark next to the visual cortex or something like that, but what about the other parts of the brain, and how can we get a full agent or a full entity that can interact in the world?”

The Atari deep reinforcement learning shift in 2013 or so was part of that early effort of agents, in my mind, because it was an attempt to try to get agents that not just perceive the world, but also take actions and interact and get rewards from environments. At the time, this was Atari games.

I feel that was a misstep. It was a misstep that even the early OpenAI that I was a part of adopted because at that time, the zeitgeist was reinforcement learning environments, games, game playing, beat games, get lots of different types of games, and OpenAI was doing a lot of that. That was another prominent part of AI where maybe for two or three or four years, everyone was doing reinforcement learning on games. That was all a bit of a misstep.

What I was trying to do at OpenAI is I was always a bit suspicious of games as being this thing that would lead to AGI. Because in my mind, you want something like an accountant or something that’s interacting with the real world. I just didn’t see how games add up to it. My project at OpenAI, for example, was within the scope of the Universe project, on an agent that was using keyboard and mouse to operate web pages. I really wanted to have something that interacts with the actual digital world that can do knowledge work.

It just so turns out that this was extremely early, way too early, so early that we shouldn’t have been working on that. Because if you’re just stumbling your way around and keyboard mashing and mouse clicking and trying to get rewards in these environments, your reward is too sparse and you just won’t learn. You’re going to burn a forest computing, and you’re never going to get something off the ground. What you’re missing is this power of representation in the neural network.

For example, today people are training those computer-using agents, but they’re doing it on top of a large language model. You have to get the language model first, you have to get the representations first, and you have to do that by all the pre-training and all the LLM stuff.

I feel maybe loosely speaking, people kept trying to get the full thing too early a few times, where people really try to go after agents too early, I would say. That was Atari and Universe and even my own experience. You actually have to do some things first before you get to those agents. Now the agents are a lot more competent, but maybe we’re still missing some parts of that stack.

I would say those are the three major buckets of what people were doing: training neural nets per-tasks, trying the first round of agents, and then maybe the LLMs and seeking the representation power of the neural networks before you tack on everything else on top.

Dwarkesh Patel 00:07:02

Interesting. If I were to steelman the Sutton perspective, it would be that humans can just take on everything at once, or even animals can take on everything at once. Animals are maybe a better example because they don’t even have the scaffold of language. They just get thrown out into the world, and they just have to make sense of everything without any labels.

The vision for AGI then should just be something which looks at sensory data, looks at the computer screen, and it just figures out what’s going on from scratch. If a human were put in a similar situation and had to be trained from scratch… This is like a human growing up or an animal growing up. Why shouldn’t that be the vision for AI, rather than this thing where we’re doing millions of years of training?

Andrej Karpathy 00:07:41

That’s a really good question. Sutton was on your podcast and I saw the podcast and I had a write-up about that podcast that gets into a bit of how I see things. I’m very careful to make analogies to animals because they came about by a very different optimization process. Animals are evolved, and they come with a huge amount of hardware that’s built in.

For example, my example in the post was the zebra. A zebra gets born, and a few minutes later it’s running around and following its mother. That’s an extremely complicated thing to do. That’s not reinforcement learning. That’s something that’s baked in. Evolution obviously has some way of encoding the weights of our neural nets in ATCGs, and I have no idea how that works, but it apparently works.

Brains just came from a very different process, and I’m very hesitant to take inspiration from it because we’re not actually running that process. In my post, I said we’re not building animals. We’re building ghosts or spirits or whatever people want to call it, because we’re not doing training by evolution. We’re doing training by imitation of humans and the data that they’ve put on the Internet.

You end up with these ethereal spirit entities because they’re fully digital and they’re mimicking humans. It’s a different kind of intelligence. If you imagine a space of intelligences, we’re starting off at a different point almost. We’re not really building animals. But it’s also possible to make them a bit more animal-like over time, and I think we should be doing that.

One more point. I do feel Sutton has a very... His framework is, “We want to build animals.” I think that would be wonderful if we can get that to work. That would be amazing. If there were a single algorithm that you can just run on the Internet and it learns everything, that would be incredible. I’m not sure that it exists and that’s certainly not what animals do, because animals have this outer loop of evolution.

A lot of what looks like learning is more like maturation of the brain. I think there’s very little reinforcement learning for animals. A lot of the reinforcement learning is more like motor tasks; it’s not intelligence tasks. So I actually kind of think humans don’t really use RL, roughly speaking.

Dwarkesh Patel 00:09:52

Can you repeat the last sentence? A lot of that intelligence is not motor task…it’s what, sorry?

Andrej Karpathy 00:09:54

A lot of the reinforcement learning, in my perspective, would be things that are a lot more motor-like, simple tasks like throwing a hoop. But I don’t think that humans use reinforcement learning for a lot of intelligence tasks like problem-solving and so on. That doesn’t mean we shouldn’t do that for research, but I just feel like that’s what animals do or don’t.

Dwarkesh Patel 00:10:17

I’m going to take a second to digest that because there are a lot of different ideas. Here’s one clarifying question I can ask to understand the perspective. You suggest that evolution is doing the kind of thing that pre-training does in the sense of building something which can then understand the world.

The difference is that evolution has to be titrated in the case of humans through three gigabytes of DNA. That’s very unlike the weights of a model. Literally, the weights of the model are a brain, which obviously does not exist in the sperm and the egg. So it has to be grown. Also, the information for every single synapse in the brain simply cannot exist in the three gigabytes that exist in the DNA.

Evolution seems closer to finding the algorithm which then does the lifetime learning. Now, maybe the lifetime learning is not analogous to RL, to your point. Is that compatible with the thing you were saying, or would you disagree with that?

Andrej Karpathy 00:11:17

I think so. I would agree with you that there’s some miraculous compression going on because obviously, the weights of the neural net are not stored in ATCGs. There’s some dramatic compression. There are some learning algorithms encoded that take over and do some of the learning online. I definitely agree with you on that. I would say I’m a lot more practically minded. I don’t come at it from the perspective of, let’s build animals. I come from it from the perspective of, let’s build useful things. I have a hard hat on, and I’m just observing that we’re not going to do evolution, because I don’t know how to do that.

But it does turn out we can build these ghosts, spirit-like entities, by imitating internet documents. This works. It’s a way to bring you up to something that has a lot of built-in knowledge and intelligence in some way, similar to maybe what evolution has done. That’s why I call pre-training this crappy evolution. It’s the practically possible version with our technology and what we have available to us to get to a starting point where we can do things like reinforcement learning and so on.

Dwarkesh Patel 00:12:15

Just to steelman the other perspective, after doing this Sutton interview and thinking about it a bit, he has an important point here. Evolution does not give us the knowledge, really. It gives us the algorithm to find the knowledge, and that seems different from pre-training.

Perhaps the perspective is that pre-training helps build the kind of entity which can learn better. It teaches meta-learning, and therefore it is similar to finding an algorithm. But if it’s “Evolution gives us knowledge, pre-training gives us knowledge,” that analogy seems to break down.

Andrej Karpathy 00:12:42

It’s subtle and I think you’re right to push back on it, but basically the thing that pre-training is doing, you’re getting the next-token predictor over the internet, and you’re training that into a neural net. It’s doing two things that are unrelated. Number one, it’s picking up all this knowledge, as I call it. Number two, it’s actually becoming intelligent.

By observing the algorithmic patterns in the internet, it boots up all these little circuits and algorithms inside the neural net to do things like in-context learning and all this stuff. You don’t need or want the knowledge. I think that’s probably holding back the neural networks overall because it’s getting them to rely on the knowledge a little too much sometimes.

For example, I feel agents, one thing they’re not very good at, is going off the data manifold of what exists on the internet. If they had less knowledge or less memory, maybe they would be better. What I think we have to do going forward—and this would be part of the research paradigms—is figure out ways to remove some of the knowledge and to keep what I call this cognitive core. It’s this intelligent entity that is stripped from knowledge but contains the algorithms and contains the magic of intelligence and problem-solving and the strategies of it and all this stuff.

Dwarkesh Patel 00:13:50

There’s so much interesting stuff there. Let’s start with in-context learning. This is an obvious point, but I think it’s worth just saying it explicitly and meditating on it. The situation in which these models seem the most intelligent—in which I talk to them and I’m like, “Wow, there’s really something on the other end that’s responding to me thinking about things—is if it makes a mistake it’s like, “Oh wait, that’s the wrong way to think about it. I’m backing up.” All that is happening in context. That’s where I feel like the real intelligence is that you can visibly see.

That in-context learning process is developed by gradient descent on pre-training. It spontaneously meta-learns in-context learning, but the in-context learning itself is not gradient descent, in the same way that our lifetime intelligence as humans to be able to do things is conditioned by evolution but our learning during our lifetime is happening through some other process.

Andrej Karpathy 00:14:42

I don’t fully agree with that, but you should continue your thought.

Dwarkesh Patel 00:14:44

Well, I’m very curious to understand how that analogy breaks down.

Andrej Karpathy 00:14:48

I’m hesitant to say that in-context learning is not doing gradient descent. It’s not doing explicit gradient descent. In-context learning is pattern completion within a token window. It just turns out that there’s a huge amount of patterns on the internet. You’re right, the model learns to complete the pattern, and that’s inside the weights. The weights of the neural network are trying to discover patterns and complete the pattern. There’s some adaptation that happens inside the neural network, which is magical and just falls out from the internet just because there’s a lot of patterns.

I will say that there have been some papers that I thought were interesting that look at the mechanisms behind in-context learning. I do think it’s possible that in-context learning runs a small gradient descent loop internally in the layers of the neural network. I recall one paper in particular where they were doing linear regression using in-context learning. Your inputs into the neural network are XY pairs, XY, XY, XY that happen to be on the line. Then you do X and you expect Y. The neural network, when you train it in this way, does linear regression.

Normally when you would run linear regression, you have a small gradient descent optimizer that looks at XY, looks at an error, calculates the gradient of the weights and does the update a few times. It just turns out that when they looked at the weights of that in-context learning algorithm, they found some analogies to gradient descent mechanics. In fact, I think the paper was even stronger because they hardcoded the weights of a neural network to do gradient descent through attention and all the internals of the neural network.

That’s just my only pushback. Who knows how in-context learning works, but I think that it’s probably doing a bit of some funky gradient descent internally. I think that that’s possible. I was only pushing back on your saying that it’s not doing in-context learning. Who knows what it’s doing, but it’s probably maybe doing something similar to it, but we don’t know.

Dwarkesh Patel 00:16:39

So then it’s worth thinking okay, if in-context learning and pre-training are both implementing something like gradient descent, why does it feel like with in-context learning we’re getting to this continual learning, real intelligence-like thing? Whereas you don’t get the analogous feeling just from pre-training. You could argue that.

If it’s the same algorithm, what could be different? One way you could think about it is, how much information does the model store per information it receives from training? If you look at pre-training, if you look at Llama 3 for example, I think it’s trained on 15 trillion tokens. If you look at the 70B model, that would be the equivalent of 0.07 bits per token that it sees in pre-training, in terms of the information in the weights of the model compared to the tokens it reads. Whereas if you look at the KV cache and how it grows per additional token in in-context learning, it’s like 320 kilobytes. So that’s a 35 million-fold difference in how much information per token is assimilated by the model. I wonder if that’s relevant at all.

Andrej Karpathy 00:17:46

I kind of agree. The way I usually put this is that anything that happens during the training of the neural network, the knowledge is only a hazy recollection of what happened in training time. That’s because the compression is dramatic. You’re taking 15 trillion tokens and you’re compressing it to just your final neural network of a few billion parameters. Obviously it’s a massive amount of compression going on. So I refer to it as a hazy recollection of the internet documents.

Whereas anything that happens in the context window of the neural network—you’re plugging in all the tokens and building up all those KV cache representations—is very directly accessible to the neural net. So I compare the KV cache and the stuff that happens at test time to more like a working memory. All the stuff that’s in the context window is very directly accessible to the neural net.

There’s always these almost surprising analogies between LLMs and humans. I find them surprising because we’re not trying to build a human brain directly. We’re just finding that this works and we’re doing it. But I do think that anything that’s in the weights, it’s a hazy recollection of what you read a year ago. Anything that you give it as a context at test time is directly in the working memory. That’s a very powerful analogy to think through things.

When you, for example, go to an LLM and you ask it about some book and what happened in it, like Nick Lane’s book or something like that, the LLM will often give you some stuff which is roughly correct. But if you give it the full chapter and ask it questions, you’re going to get much better results because it’s now loaded in the working memory of the model. So a very long way of saying I agree and that’s why.

Dwarkesh Patel 00:19:11

Stepping back, what is the part about human intelligence that we have most failed to replicate with these models?

Andrej Karpathy 00:19:20

Just a lot of it. So maybe one way to think about it, I don’t know if this is the best way, but I almost feel like — again, making these analogies imperfect as they are — we’ve stumbled by with the transformer neural network, which is extremely powerful, very general. You can train transformers on audio, or video, or text, or whatever you want, and it just learns patterns and they’re very powerful, and it works really well. That to me almost indicates that this is some piece of cortical tissue. It’s something like that, because the cortex is famously very plastic as well. You can rewire parts of brains. There were the slightly gruesome experiments with rewiring the visual cortex to the auditory cortex, and this animal learned fine, et cetera.

So I think that this is cortical tissue. I think when we’re doing reasoning and planning inside the neural networks, doing reasoning traces for thinking models, that’s kind of like the prefrontal cortex. Maybe those are like little checkmarks, but I still think there are many brain parts and nuclei that are not explored. For example, there’s a basal ganglia doing a bit of reinforcement learning when we fine-tune the models on reinforcement learning. But where’s the hippocampus? Not obvious what that would be. Some parts are probably not important. Maybe the cerebellum is not important to cognition, its thoughts, so maybe we can skip some of it. But I still think there’s, for example, the amygdala, all the emotions and instincts. There’s probably a bunch of other nuclei in the brain that are very ancient that I don’t think we’ve really replicated.

I don’t know that we should be pursuing the building of an analog of a human brain. I’m an engineer mostly at heart. Maybe another way to answer the question is that you’re not going to hire this thing as an intern. It’s missing a lot of it because it comes with a lot of these cognitive deficits that we all intuitively feel when we talk to the models. So it’s not fully there yet. You can look at it as not all the brain parts are checked off yet.

Dwarkesh Patel 00:21:16

This is maybe relevant to the question of thinking about how fast these issues will be solved. Sometimes people will say about continual learning, “Look, you could easily replicate this capability. Just as in-context learning emerged spontaneously as a result of pre-training, continual learning over longer horizons will emerge spontaneously if the model is incentivized to recollect information over longer horizons, or horizons longer than one session.” So if there’s some outer loop RL which has many sessions within that outer loop, then this continual learning where it fine-tunes itself, or it writes to an external memory or something, will just emerge spontaneously. Do you think things like that are plausible? I just don’t have a prior over how plausible that is. How likely is that to happen?

Andrej Karpathy 00:22:07

I don’t know that I fully resonate with that. These models, when you boot them up and they have zero tokens in the window, they’re always restarting from scratch where they were. So I don’t know in that worldview what it looks like. Maybe making some analogies to humans—just because I think it’s roughly concrete and interesting to think through—I feel like when I’m awake, I’m building up a context window of stuff that’s happening during the day. But when I go to sleep, something magical happens where I don’t think that context window stays around. There’s some process of distillation into the weights of my brain. This happens during sleep and all this stuff.

We don’t have an equivalent of that in large language models. That’s to me more adjacent to when you talk about continual learning and so on as absent. These models don’t really have a distillation phase of taking what happened, analyzing it obsessively, thinking through it, doing some synthetic data generation process and distilling it back into the weights, and maybe having a specific neural net per person. Maybe it’s a LoRA. It’s not a full-weight neural network. It’s just some small sparse subset of the weights that are changed.

But we do want to create ways of creating these individuals that have very long context. It’s not only remaining in the context window because the context windows grow very, very long. Maybe we have some very elaborate, sparse attention over it. But I still think that humans obviously have some process for distilling some of that knowledge into the weights. We’re missing it. I do also think that humans have some very elaborate, sparse attention scheme, which I think we’re starting to see some early hints of. DeepSeek v3.2 just came out and I saw that they have sparse attention as an example, and this is one way to have very, very long context windows. So I feel like we are redoing a lot of the cognitive tricks that evolution came up with through a very different process. But we’re going to converge on a similar architecture cognitively.

Dwarkesh Patel 00:24:02

In 10 years, do you think it’ll still be something like a transformer, but with much more modified attention and more sparse MLPs and so forth?

Andrej Karpathy 00:24:10

The way I like to think about it is translation invariance in time. So 10 years ago, where were we? 2015. In 2015, we had convolutional neural networks primarily, residual networks just came out. So remarkably similar, I guess, but quite a bit different still. The transformer was not around. All these more modern tweaks on the transformer were not around. Maybe some of the things that we can bet on, I think in 10 years by translational equivariance, is that we’re still training giant neural networks with a forward backward pass and update through gradient descent, but maybe it looks a bit different, and it’s just that everything is much bigger.

Recently I went back all the way to 1989 which was a fun exercise for me, a few years ago, because I was reproducing Yann LeCun’s 1989 convolutional network, which was the first neural network I’m aware of trained via gradient descent, like modern neural network trained gradient descent on digit recognition. I was just interested in how I could modernize this. How much of this is algorithms? How much of this is data? How much of this progress is compute and systems? I was able to very quickly halve the learning just by time traveling by 33 years.

So if I time travel by algorithms 33 years, I could adjust what Yann LeCun did in 1989, and I could halve the error. But to get further gains, I had to add a lot more data, I had to 10x the training set, and then I had to add more computational optimizations. I had to train for much longer with dropout and other regularization techniques.

So all these things have to improve simultaneously. We’re probably going to have a lot more data, we’re probably going to have a lot better hardware, probably going to have a lot better kernels and software, we’re probably going to have better algorithms. All of those, it’s almost like no one of them is winning too much. All of them are surprisingly equal. This has been the trend for a while.

So to answer your question, I expect differences algorithmically to what’s happening today. But I do also expect that some of the things that have stuck around for a very long time will probably still be there. It’s probably still a giant neural network trained with gradient descent. That would be my guess.

Dwarkesh Patel 00:26:16

It’s surprising that all of those things together only halved the error, 30 years of progress…. Maybe half is a lot. Because if you halve the error, that actually means that…

Andrej Karpathy 00:26:30

Half is a lot. But I guess what was shocking to me is everything needs to improve across the board: architecture, optimizer, loss function. It also has improved across the board forever. So I expect all those changes to be alive and well.

Dwarkesh Patel 00:26:43

Yeah. I was about to ask you a very similar question about nanochat. Since you just coded it up recently, every single step in the process of building a chatbot is fresh in your RAM. I’m curious if you had similar thoughts about, “Oh, there was no one thing that was relevant to going from GPT-2 to nanochat.” What are some surprising takeaways from the experience?

Andrej Karpathy 00:27:08

Of building nanochat? So nanochat is a repository I released. Was it yesterday or the day before? I can’t remember.

Dwarkesh Patel 00:27:15

We can see the sleep deprivation that went into the…

Andrej Karpathy 00:27:18

It’s trying to be the simplest complete repository that covers the whole pipeline end-to-end of building a ChatGPT clone. So you have all of the steps, not just any individual step, which is a bunch. I worked on all the individual steps in the past and released small pieces of code that show you how that’s done in an algorithmic sense, in simple code. But this handles the entire pipeline. In terms of learning, I don’t know that I necessarily found something that I learned from it. I already had in my mind how you build it. This is just the process of mechanically building it and making it clean enough so that people can learn from it and that they find it useful.

Dwarkesh Patel 00:28:04

What is the best way for somebody to learn from it? Is it to just delete all the code and try to reimplement from scratch, try to add modifications to it?

Andrej Karpathy 00:28:10

That’s a great question. Basically it’s about 8,000 lines of code that takes you through the entire pipeline. I would probably put it on the right monitor. If you have two monitors, you put it on the right. You want to build it from scratch, you build it from the start. You’re not allowed to copy-paste, you’re allowed to reference, you’re not allowed to copy-paste. Maybe that’s how I would do it.

But I also think the repository by itself is a pretty large beast. When you write this code, you don’t go from top to bottom, you go from chunks and you grow the chunks, and that information is absent. You wouldn’t know where to start. So it’s not just a final repository that’s needed, it’s the building of the repository, which is a complicated chunk-growing process. So that part is not there yet. I would love to add that probably later this week. It’s probably a video or something like that. Roughly speaking, that’s what I would try to do. Build the stuff yourself, but don’t allow yourself copy-paste.

I do think that there’s two types of knowledge, almost. There’s the high-level surface knowledge, but when you build something from scratch, you’re forced to come to terms with what you don’t understand and you don’t know that you don’t understand it.

It always leads to a deeper understanding. It’s the only way to build. If I can’t build it, I don’t understand it. That’s a Feynman quote, I believe. I 100% have always believed this very strongly, because there are all these micro things that are just not properly arranged and you don’t really have the knowledge. You just think you have the knowledge. So don’t write blog posts, don’t do slides, don’t do any of that. Build the code, arrange it, get it to work. It’s the only way to go. Otherwise, you’re missing knowledge.

00:29:45 – LLM cognitive deficits

Dwarkesh Patel 00:29:45

You tweeted out that coding models were of very little help to you in assembling this repository. I’m curious why that was.

Andrej Karpathy 00:29:53

I guess I built the repository over a period of a bit more than a month. I would say there are three major classes of how people interact with code right now. Some people completely reject all of LLMs and they are just writing by scratch. This is probably not the right thing to do anymore.

The intermediate part, which is where I am, is you still write a lot of things from scratch, but you use the autocomplete that’s available now from these models. So when you start writing out a little piece of it, it will autocomplete for you and you can just tap through. Most of the time it’s correct, sometimes it’s not, and you edit it. But you’re still very much the architect of what you’re writing. Then there’s the vibe coding: “Hi, please implement this or that,” enter, and then let the model do it. That’s the agents.

I do feel like the agents work in very specific settings, and I would use them in specific settings. But these are all tools available to you and you have to learn what they’re good at, what they’re not good at, and when to use them. So the agents are pretty good, for example, if you’re doing boilerplate stuff. Boilerplate code that’s just copy-paste stuff, they’re very good at that. They’re very good at stuff that occurs very often on the Internet because there are lots of examples of it in the training sets of these models. There are features of things where the models will do very well.

I would say nanochat is not an example of those because it’s a fairly unique repository. There’s not that much code in the way that I’ve structured it. It’s not boilerplate code. It’s intellectually intense code almost, and everything has to be very precisely arranged. The models have so many cognitive deficits. One example, they kept misunderstanding the code because they have too much memory from all the typical ways of doing things on the Internet that I just wasn’t adopting. The models, for example—I don’t know if I want to get into the full details—but they kept thinking I’m writing normal code, and I’m not.

Dwarkesh Patel 00:31:49

Maybe one example?

Andrej Karpathy 00:31:51

You have eight GPUs that are all doing forward, backwards. The way to synchronize gradients between them is to use a Distributed Data Parallel container of PyTorch, which automatically as you’re doing the backward, it will start communicating and synchronizing gradients. I didn’t use DDP because I didn’t want to use it, because it’s not necessary. I threw it out and wrote my own synchronization routine that’s inside the step of the optimizer. The models were trying to get me to use the DDP container. They were very concerned. This gets way too technical, but I wasn’t using that container because I don’t need it and I have a custom implementation of something like it.

Dwarkesh Patel 00:32:26

They just couldn’t internalize that you had your own.

Andrej Karpathy 00:32:28

They couldn’t get past that. They kept trying to mess up the style. They’re way too over-defensive. They make all these try-catch statements. They keep trying to make a production code base, and I have a bunch of assumptions in my code, and it’s okay. I don’t need all this extra stuff in there. So I feel like they’re bloating the code base, bloating the complexity, they keep misunderstanding, they’re using deprecated APIs a bunch of times. It’s a total mess. It’s just not net useful. I can go in, I can clean it up, but it’s not net useful.

I also feel like it’s annoying to have to type out what I want in English because it’s too much typing. If I just navigate to the part of the code that I want, and I go where I know the code has to appear and I start typing out the first few letters, autocomplete gets it and just gives you the code. This is a very high information bandwidth to specify what you want. You point to the code where you want it, you type out the first few pieces, and the model will complete it.

So what I mean is, these models are good in certain parts of the stack. There are two examples where I use the models that I think are illustrative. One was when I generated the report. That’s more boilerplate-y, so I partially vibe-coded some of that stuff. That was fine because it’s not mission-critical stuff, and it works fine.

The other part is when I was rewriting the tokenizer in Rust. I’m not as good at Rust because I’m fairly new to Rust. So there’s a bit of vibe coding going on when I was writing some of the Rust code. But I had a Python implementation that I fully understand, and I’m just making sure I’m making a more efficient version of it, and I have tests so I feel safer doing that stuff. They increase accessibility to languages or paradigms that you might not be as familiar with. I think they’re very helpful there as well. There’s a ton of Rust code out there, the models are pretty good at it. I happen to not know that much about it, so the models are very useful there.

Dwarkesh Patel 00:34:23

The reason this question is so interesting is because the main story people have about AI exploding and getting to superintelligence pretty rapidly is AI automating AI engineering and AI research. They’ll look at the fact that you can have Claude Code and make entire applications, CRUD applications, from scratch and think, “If you had this same capability inside of OpenAI and DeepMind and everything, just imagine a thousand of you or a million of you in parallel, finding little architectural tweaks.”

It’s quite interesting to hear you say that this is the thing they’re asymmetrically worse at. It’s quite relevant to forecasting whether the AI 2027-type explosion is likely to happen anytime soon.

Andrej Karpathy 00:35:05

That’s a good way of putting it, and you’re getting at why my timelines are a bit longer. You’re right. They’re not very good at code that has never been written before, maybe it’s one way to put it, which is what we’re trying to achieve when we’re building these models.

Dwarkesh Patel 00:35:19

Very naive question, but the architectural tweaks that you’re adding to nanochat, they’re in a paper somewhere, right? They might even be in a repo somewhere. Is it surprising that they aren’t able to integrate that into whenever you’re like, “Add RoPE embeddings” or something, they do that in the wrong way?

Andrej Karpathy 00:35:42

It’s tough. They know, but they don’t fully know. They don’t know how to fully integrate it into the repo and your style and your code and your place, and some of the custom things that you’re doing and how it fits with all the assumptions of the repository. They do have some knowledge, but they haven’t gotten to the place where they can integrate it and make sense of it.

A lot of the stuff continues to improve. Currently, the state-of-the-art model that I go to is the GPT-5 Pro, and that’s a very powerful model. If I have 20 minutes, I will copy-paste my entire repo and I go to GPT-5 Pro, the oracle, for some questions. Often it’s not too bad and surprisingly good compared to what existed a year ago.

Overall, the models are not there. I feel like the industry is making too big of a jump and is trying to pretend like this is amazing, and it’s not. It’s slop. They’re not coming to terms with it, and maybe they’re trying to fundraise or something like that. I’m not sure what’s going on, but we’re at this intermediate stage. The models are amazing. They still need a lot of work. For now, autocomplete is my sweet spot. But sometimes, for some types of code, I will go to an LLM agent.

Dwarkesh Patel 00:36:53

Here’s another reason this is really interesting. Through the history of programming, there have been many productivity improvements—compilers, linting, better programming languages—which have increased programmer productivity but have not led to an explosion. That sounds very much like the autocomplete tab, and this other category is just automation of the programmer. It’s interesting you’re seeing more in the category of the historical analogies of better compilers or something.

Andrej Karpathy 00:37:26

Maybe this gets to one other thought. I have a hard time differentiating where AI begins and stops because I see AI as fundamentally an extension of computing in a pretty fundamental way. I see a continuum of this recursive self-improvement or speeding up programmers all the way from the beginning: code editors, syntax highlighting, or checking even of the types, like data type checking—all these tools that we’ve built for each other.

Even search engines. Why aren’t search engines part of AI? Ranking is AI. At some point, Google, even early on, was thinking of themselves as an AI company doing Google Search engine, which is totally fair.

I see it as a lot more of a continuum than other people do, and it’s hard for me to draw the line. I feel like we’re now getting a much better autocomplete, and now we’re also getting some agents which are these loopy things, but they go off-rails sometimes. What’s going on is that the human is progressively doing a bit less and less of the low-level stuff. We’re not writing the assembly code because we have compilers. Compilers will take my high-level language in C and write the assembly code.

We’re abstracting ourselves very, very slowly. There’s this what I call “autonomy slider,” where more and more stuff is automated—of the stuff that can be automated at any point in time—and we’re doing a bit less and less and raising ourselves in the layer of abstraction over the automation.

00:40:05 – RL is terrible

Dwarkesh Patel 00:40:05

Let’s talk about RL a bit. You tweeted some very interesting things about this. Conceptually, how should we think about the way that humans are able to build a rich world model just from interacting with our environment, and in ways that seem almost irrespective of the final reward at the end of the episode?

If somebody is starting a business, and at the end of 10 years, she finds out whether the business succeeded or failed, we say that she’s earned a bunch of wisdom and experience. But it’s not because the log probs of every single thing that happened over the last 10 years are up-weighted or down-weighted. Something much more deliberate and rich is happening. What is the ML analogy, and how does that compare to what we’re doing with LLMs right now?

Andrej Karpathy 00:40:47

Maybe the way I would put it is that humans don’t use reinforcement learning, as I said. I think they do something different. Reinforcement learning is a lot worse than I think the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before it is much worse because previously we were just imitating people, so it has all these issues.

In reinforcement learning, say you’re solving a math problem, because it’s very simple. You’re given a math problem and you’re trying to find the solution. In reinforcement learning, you will try lots of things in parallel first. You’re given a problem, you try hundreds of different attempts. These attempts can be complex. They can be like, “Oh, let me try this, let me try that, this didn’t work, that didn’t work,” etc. Then maybe you get an answer. Now you check the back of the book and you see, “Okay, the correct answer is this.” You can see that this one, this one, and that one got the correct answer, but these other 97 of them didn’t. Literally what reinforcement learning does is it goes to the ones that worked really well and every single thing you did along the way, every single token gets upweighted like, “Do more of this.”

The problem with that is people will say that your estimator has high variance, but it’s just noisy. It’s noisy. It almost assumes that every single little piece of the solution that you made that arrived at the right answer was the correct thing to do, which is not true. You may have gone down the wrong alleys until you arrived at the right solution. Every single one of those incorrect things you did, as long as you got to the correct solution, will be upweighted as, “Do more of this.” It’s terrible. It’s noise.

You’ve done all this work only to find, at the end, you get a single number of like, “Oh, you did correct.” Based on that, you weigh that entire trajectory as like, upweight or downweight. The way I like to put it is you’re sucking supervision through a straw. You’ve done all this work that could be a minute of rollout, and you’re sucking the bits of supervision of the final reward signal through a straw and you’re broadcasting that across the entire trajectory and using that to upweight or downweight that trajectory. It’s just stupid and crazy.

A human would never do this. Number one, a human would never do hundreds of rollouts. Number two, when a person finds a solution, they will have a pretty complicated process of review of, “Okay, I think these parts I did well, these parts I did not do that well. I should probably do this or that.” They think through things. There’s nothing in current LLMs that does this. There’s no equivalent of it. But I do see papers popping out that are trying to do this because it’s obvious to everyone in the field.

The first imitation learning, by the way, was extremely surprising and miraculous and amazing, that we can fine-tune by imitation on humans. That was incredible. Because in the beginning, all we had was base models. Base models are autocomplete. It wasn’t obvious to me at the time, and I had to learn this. The paper that blew my mind was InstructGPT, because it pointed out that you can take the pretrained model, which is autocomplete, and if you just fine-tune it on text that looks like conversations, the model will very rapidly adapt to become very conversational, and it keeps all the knowledge from pre-training. This blew my mind because I didn’t understand that stylistically, it can adjust so quickly and become an assistant to a user through just a few loops of fine-tuning on that kind of data. It was very miraculous to me that that worked. So incredible. That was two to three years of work.

Now came RL. And RL allows you to do a bit better than just imitation learning because you can have these reward functions and you can hill-climb on the reward functions. Some problems have just correct answers, you can hill-climb on that without getting expert trajectories to imitate. So that’s amazing. The model can also discover solutions that a human might never come up with. This is incredible. Yet, it’s still stupid.

We need more. I saw a paper from Google yesterday that tried to have this reflect & review idea in mind. Was it the memory bank paper or something? I don’t know. I’ve seen a few papers along these lines. So I expect there to be some major update to how we do algorithms for LLMs coming in that realm. I think we need three or four or five more, something like that.

Dwarkesh Patel 00:44:54

You’re so good at coming up with evocative phrases. “Sucking supervision through a straw.” It’s so good.

You’re saying the problem with outcome-based reward is that you have this huge trajectory, and then at the end, you’re trying to learn every single possible thing about what you should do and what you should learn about the world from that one final bit. Given the fact that this is obvious, why hasn’t process-based supervision as an alternative been a successful way to make models more capable? What has been preventing us from using this alternative paradigm?

Andrej Karpathy 00:45:29

Process-based supervision just refers to the fact that we’re not going to have a reward function only at the very end. After you’ve done 10 minutes of work, I’m not going to tell you you did well or not well. I’m going to tell you at every single step of the way how well you’re doing. The reason we don’t have that is it’s tricky how you do that properly. You have partial solutions and you don’t know how to assign credit.  So when you get the right answer, it’s just an equality match to the answer. It’s very simple to implement. If you’re doing process supervision, how do you assign in an automatable way, a partial credit assignment? It’s not obvious how you do it.

Lots of labs are trying to do it with these LLM judges. You get LLMs to try to do it. You prompt an LLM, “Hey, look at a partial solution of a student. How well do you think they’re doing if the answer is this?” and they try to tune the prompt.

The reason that this is tricky is quite subtle. It’s the fact that anytime you use an LLM to assign a reward, those LLMs are giant things with billions of parameters, and they’re gameable. If you’re reinforcement learning with respect to them, you will find adversarial examples for your LLM judges, almost guaranteed. So you can’t do this for too long. You do maybe 10 steps or 20 steps, and maybe it will work, but you can’t do 100 or 1,000. I understand it’s not obvious, but basically the model will find little cracks. It will find all these spurious things in the nooks and crannies of the giant model and find a way to cheat it.

One example that’s prominently in my mind, this was probably public, if you’re using an LLM judge for a reward, you just give it a solution from a student and ask it if the student did well or not. We were training with reinforcement learning against that reward function, and it worked really well. Then, suddenly, the reward became extremely large. It was a massive jump, and it did perfect. You’re looking at it like, “Wow, this means the student is perfect in all these problems. It’s fully solved math.”

But when you look at the completions that you’re getting from the model, they are complete nonsense. They start out okay, and then they change to “dhdhdhdh.” It’s just like, “Oh, okay, let’s take two plus three and we do this and this, and then dhdhdhdh.” You’re looking at it, and it’s like, this is crazy. How is it getting a reward of one or 100%? You look at the LLM judge, and it turns out that “dhdhdhdh” is an adversarial example for the model, and it assigns 100% probability to it.

It’s just because this is an out-of-sample example to the LLM. It’s never seen it during training, and you’re in pure generalization land. It’s never seen it during training, and in the pure generalization land, you can find these examples that break it.

Dwarkesh Patel 00:47:52

You’re basically training the LLM to be a prompt injection model.

Andrej Karpathy 00:47:56

Not even that. Prompt injection is way too fancy. You’re finding adversarial examples, as they’re called. These are nonsensical solutions that are obviously wrong, but the model thinks they are amazing.

Dwarkesh Patel 00:48:07

To the extent you think this is the bottleneck to making RL more functional, then that will require making LLMs better judges, if you want to do this in an automated way. Is it just going to be some sort of GAN-like approach where you have to train models to be more robust?

Andrej Karpathy 00:48:22

The labs are probably doing all that. The obvious thing is, “dhdhdhdh” should not get 100% reward. Okay, well, take “dhdhdhdh,” put it in the training set of the LLM judge, and say this is not 100%, this is 0%. You can do this, but every time you do this, you get a new LLM, and it still has adversarial examples. There’s an infinity of adversarial examples.

Probably if you iterate this a few times, it’ll probably be harder and harder to find adversarial examples, but I’m not 100% sure because this thing has a trillion parameters or whatnot. I bet you the labs are trying. I still think we need other ideas.

Dwarkesh Patel 00:48:57

Interesting. Do you have some shape of what the other idea could be?

Andrej Karpathy 00:49:02

This idea of a review solution encompassing synthetic examples such that when you train on them, you get better, and meta-learn it in some way. I think there are some papers that I’m starting to see pop out. I am only at a stage of reading abstracts because a lot of these papers are just ideas. Someone has to make it work on a frontier LLM lab scale in full generality because when you see these papers, they pop up, and it’s just a bit noisy. They’re cool ideas, but I haven’t seen anyone convincingly show that this is possible. That said, the LLM labs are fairly closed, so who knows what they’re doing now.

00:49:38 – How do humans learn?

Dwarkesh Patel 00:49:38

I can conceptualize how you would be able to train on synthetic examples or synthetic problems that you have made for yourself. But there seems to be another thing humans do—maybe sleep is this, maybe daydreaming is this—which is not necessarily to come up with fake problems, but just to reflect.

I’m not sure what the ML analogy is for daydreaming or sleeping, or just reflecting. I haven’t come up with a new problem. Obviously, the very basic analogy would just be fine-tuning on reflection bits, but I feel like in practice that probably wouldn’t work that well. Do you have some take on what the analogy of this thing is?

Andrej Karpathy 00:50:17

I do think that we’re missing some aspects there. As an example, let’s take reading a book. Currently when LLMs are reading a book, what that means is we stretch out the sequence of text, and the model is predicting the next token, and it’s getting some knowledge from that. That’s not really what humans do. When you’re reading a book, I don’t even feel like the book is exposition I’m supposed to be attending to and training on. The book is a set of prompts for me to do synthetic data generation, or for you to get to a book club and talk about it with your friends. It’s by manipulating that information that you actually gain that knowledge. We have no equivalent of that with LLMs. They don’t really do that. I’d love to see during pre-training some stage that thinks through the material and tries to reconcile it with what it already knows, and thinks through it for some amount of time and gets that to work. There’s no equivalence of any of this. This is all research.

There are some subtle—very subtle that I think are very hard to understand—reasons why it’s not trivial. If I can just describe one: why can’t we just synthetically generate and train on it? Because every synthetic example, if I just give synthetic generation of the model thinking about a book, you look at it and you’re like, “This looks great. Why can’t I train on it?” You could try, but the model will get much worse if you continue trying. That’s because all of the samples you get from models are silently collapsed. Silently—it is not obvious if you look at any individual example of it—they occupy a very tiny manifold of the possible space of thoughts about content. The LLMs, when they come off, they’re what we call “collapsed.” They have a collapsed data distribution. One easy way to see it is to go to ChatGPT and ask it, “Tell me a joke.” It only has like three jokes. It’s not giving you the whole breadth of possible jokes. It knows like three jokes. They’re silently collapsed.

You’re not getting the richness and the diversity and the entropy from these models as you would get from humans. Humans are a lot noisier, but at least they’re not biased, in a statistical sense. They’re not silently collapsed. They maintain a huge amount of entropy. So how do you get synthetic data generation to work despite the collapse and while maintaining the entropy? That’s a research problem.

Dwarkesh Patel 00:52:20

Just to make sure I understood, the reason that the collapse is relevant to synthetic data generation is because you want to be able to come up with synthetic problems or reflections which are not already in your data distribution?

Andrej Karpathy 00:52:32

I guess what I’m saying is, say we have a chapter of a book and I ask an LLM to think about it, it will give you something that looks very reasonable. But if I ask it 10 times, you’ll notice that all of them are the same.

Dwarkesh Patel 00:52:44

You can’t just keep scaling “reflection” on the same amount of prompt information and then get returns from that.

Andrej Karpathy 00:52:54

Any individual sample will look okay, but the distribution of it is quite terrible. It’s quite terrible in such a way that if you continue training on too much of your own stuff, you actually collapse.

I think that there’s possibly no fundamental solution to this. I also think humans collapse over time. These analogies are surprisingly good. Humans collapse during the course of their lives. This is why children, they haven’t overfit yet. They will say stuff that will shock you because you can see where they’re coming from, but it’s just not the thing people say, because they’re not yet collapsed. But we’re collapsed. We end up revisiting the same thoughts. We end up saying more and more of the same stuff, and the learning rates go down, and the collapse continues to get worse, and then everything deteriorates.

Dwarkesh Patel 00:53:39

Have you seen this super interesting paper that dreaming is a way of preventing this kind of overfitting and collapse? The reason dreaming is evolutionary adaptive is to put you in weird situations that are very unlike your day-to-day reality, so as to prevent this kind of overfitting.

Andrej Karpathy 00:53:55

It’s an interesting idea. I do think that when you’re generating things in your head and then you’re attending to it, you’re training on your own samples, you’re training on your synthetic data. If you do it for too long, you go off-rails and you collapse way too much. You always have to seek entropy in your life. Talking to other people is a great source of entropy, and things like that. So maybe the brain has also built some internal mechanisms for increasing the amount of entropy in that process. That’s an interesting idea.

Dwarkesh Patel 00:54:25

This is a very ill-formed thought so I’ll just put it out and let you react to it. The best learners that we are aware of, which are children, are extremely bad at recollecting information. In fact, at the very earliest stages of childhood, you will forget everything. You’re just an amnesiac about everything that happens before a certain year date. But you’re extremely good at picking up new languages and learning from the world. Maybe there’s some element of being able to see the forest for the trees.

Whereas if you compare it to the opposite end of the spectrum, you have LLM pre-training, where these models will literally be able to regurgitate word-for-word what is the next thing in a Wikipedia page. But their ability to learn abstract concepts really quickly, the way a child can, is much more limited. Then adults are somewhere in between, where they don’t have the flexibility of childhood learning, but they can memorize facts and information in a way that is harder for kids. I don’t know if there’s something interesting about that spectrum.

Andrej Karpathy 00:55:19

I think there’s something very interesting about that, 100%. I do think that humans have a lot more of an element, compared to LLMs, of seeing the forest for the trees. We’re not actually that good at memorization, which is actually a feature. Because we’re not that good at memorization, we’re forced to find patterns in a more general sense.

LLMs in comparison are extremely good at memorization. They will recite passages from all these training sources. You can give them completely nonsensical data. You can hash some amount of text or something like that, you get a completely random sequence. If you train on it, even just for a single iteration or two, it can suddenly regurgitate the entire thing. It will memorize it. There’s no way a person can read a single sequence of random numbers and recite it to you.

That’s a feature, not a bug, because it forces you to only learn the generalizable components. Whereas LLMs are distracted by all the memory that they have of the pre-training documents, and it’s probably very distracting to them in a certain sense. So that’s why when I talk about the cognitive core, I want to remove the memory, which is what we talked about. I’d love to have them have less memory so that they have to look things up, and they only maintain the algorithms for thought, and the idea of an experiment, and all this cognitive glue of acting.

Dwarkesh Patel 00:56:36

And this is also relevant to preventing model collapse?

Andrej Karpathy 00:56:41

Let me think. I’m not sure. It’s almost like a separate axis. The models are way too good at memorization, and somehow we should remove that. People are much worse, but it’s a good thing.

Dwarkesh Patel 00:56:57

What is a solution to model collapse? There are very naive things you could attempt. The distribution over logits should be wider or something. There are many naive things you could try. What ends up being the problem with the naive approaches?

Andrej Karpathy 00:57:11

That’s a great question. You can imagine having a regularization for entropy and things like that. I guess they just don’t work as well empirically because right now the models are collapsed. But I will say most of the tasks that we want from them don’t actually demand diversity. That’s probably the answer to what’s going on.

The frontier labs are trying to make the models useful. I feel like the diversity of the outputs is not so much... Number one, it’s much harder to work with and evaluate and all this stuff, but maybe it’s not what’s capturing most of the value.

Dwarkesh Patel 00:57:42

In fact, it’s actively penalized. If you’re super creative in RL, it’s not good.

Andrej Karpathy 00:57:48

Yeah. Or maybe if you’re doing a lot of writing, help from LLMs and stuff like that, it’s probably bad because the models will silently give you all the same stuff. They won’t explore lots of different ways of answering a question.

Maybe this diversity, not as many applications need it so the models don’t have it. But then it’s a problem at synthetic data generation time, et cetera. So we’re shooting ourselves in the foot by not allowing this entropy to maintain in the model. Possibly the labs should try harder.

Dwarkesh Patel 00:58:17

I think you hinted that it’s a very fundamental problem, it won’t be easy to solve. What’s your intuition for that?

Andrej Karpathy 00:58:24

I don’t know if it’s super fundamental. I don’t know if I intended to say that. I do think that I haven’t done these experiments, but I do think that you could probably regularize the entropy to be higher. So you’re encouraging the model to give you more and more solutions, but you don’t want it to start deviating too much from the training data. It’s going to start making up its own language. It’s going to start using words that are extremely rare, so it’s going to drift too much from the distribution.

So I think controlling the distribution is just tricky. It’s probably not trivial in that sense.

Dwarkesh Patel 00:58:58

How many bits should the optimal core of intelligence end up being if you just had to make a guess? The thing we put on the von Neumann probes, how big does it have to be?

Andrej Karpathy 00:59:10

It’s really interesting in the history of the field because at one point everything was very scaling-pilled in terms of like, “Oh, we’re gonna make much bigger models, trillions of parameter models.” What the models have done in size is they’ve gone up and now they’ve come down. State-of-the-art models are smaller. Even then, I think they memorized way too much. So I had a prediction a while back that I almost feel like we can get cognitive cores that are very good at even a billion parameters.

If you talk to a billion parameter model, I think in 20 years, you can have a very productive conversation. It thinks and it’s a lot more like a human. But if you ask it some factual question, it might have to look it up, but it knows that it doesn’t know and it might have to look it up and it will just do all the reasonable things.

Dwarkesh Patel 00:59:54

That’s surprising that you think it’ll take a billion parameters. Because already we have billion parameter models or a couple billion parameter models that are very intelligent.

Andrej Karpathy 01:00:02

Well, state-of-the-art models are like a trillion parameters. But they remember so much stuff.

Dwarkesh Patel 01:00:06

Yeah, but I’m surprised that in 10 years, given the pace… We have gpt-oss-20b. That’s way better than GPT-4 original, which was a trillion plus parameters. Given that trend, I’m surprised you think in 10 years the cognitive core is still a billion parameters. I’m surprised you’re not like, “Oh it’s gonna be like tens of millions or millions.”

Andrej Karpathy 01:00:30

Here’s the issue, the training data is the internet, which is really terrible. There’s a huge amount of gains to be made because the internet is terrible. Even the internet, when you and I think of the internet, you’re thinking of like The Wall Street Journal. That’s not what this is. When you’re looking at a pre-training dataset in the frontier lab and you look at a random internet document, it’s total garbage. I don’t even know how this works at all. It’s some like stock tickers, symbols, it’s a huge amount of slop and garbage from like all the corners of the internet. It’s not like your Wall Street Journal article, that’s extremely rare. So because the internet is so terrible, we have to build really big models to compress all that. Most of that compression is memory work instead of cognitive work.

But what we really want is the cognitive part, delete the memory. I guess what I’m saying is that we need intelligent models to help us refine even the pre-training set to just narrow it down to the cognitive components. Then I think you get away with a much smaller model because it’s a much better dataset and you could train it on it. But probably it’s not trained directly on it, it’s probably distilled from a much better model still.

Dwarkesh Patel 01:01:35

But why is the distilled version still a billion?

Andrej Karpathy 01:01:39

I just feel like distillation works extremely well. So almost every small model, if you have a small model, it’s almost certainly distilled.

Dwarkesh Patel 01:01:46

Right, but why is the distillation in 10 years not getting below 1 billion?

Andrej Karpathy 01:01:50

Oh, you think it should be smaller than a billion? I mean, come on, right? I don’t know. At some point it should take at least a billion knobs to do something interesting. You’re thinking it should be even smaller?

Dwarkesh Patel 01:02:01

Yeah. If you look at the trend over the last few years of just finding low-hanging fruit and going from trillion plus models to models that are literally two orders of magnitude smaller in a matter of two years and having better performance, it makes me think the sort of core of intelligence might be even way, way smaller. Plenty of room at the bottom, to paraphrase Feynman.

Andrej Karpathy 01:02:22

I feel like I’m already contrarian by talking about a billion parameter cognitive core and you’re outdoing me. Maybe we could get a little bit smaller. I do think that practically speaking, you want the model to have some knowledge. You don’t want it to be looking up everything because then you can’t think in your head. You’re looking up way too much stuff all the time. Some basic curriculum needs to be there for knowledge, but it doesn’t have esoteric knowledge.

Dwarkesh Patel 01:02:48

We’re discussing what plausibly could be the cognitive core. There’s a separate question which is what will be the size of frontier models over time? I’m curious if you have predictions. We had increasing scale up to maybe GPT 4.5 and now we’re seeing decreasing or plateauing scale. There are many reasons this could be going on. Do you have a prediction going forward? Will the biggest models be bigger, will they be smaller, will they be the same?

Andrej Karpathy 01:03:14

I don’t have a super strong prediction. The labs are just being practical. They have a flops budget and a cost budget. It just turns out that pre-training is not where you want to put most of your flops or your cost. That’s why the models have gotten smaller. They are a bit smaller, the pre-training stage is smaller, but they make it up in reinforcement learning, mid-training, and all this stuff that follows. They’re just being practical in terms of all the stages and how you get the most bang for the buck.

Forecasting that trend is quite hard. I do still expect that there’s so much low-hanging fruit. That’s my basic expectation. I have a very wide distribution here.

Dwarkesh Patel 01:03:51

Do you expect the low-hanging fruit to be similar in kind to the kinds of things that have been happening over the last two to five years? If I look at nanochat versus nanoGPT and the architectural tweaks you made, is that the flavor of things you expect to continue to keep happening? You’re not expecting any giant paradigm shifts.

Andrej Karpathy 01:04:11

For the most part, yeah. I expect the datasets to get much, much better. When you look at the average datasets, they’re extremely terrible. They’re so bad that I don’t even know how anything works. Look at the average example in the training set: factual mistakes, errors, nonsensical things. Somehow when you do it at scale, the noise washes away and you’re left with some of the signal. Datasets will improve a ton.

Everything gets better. Our hardware, all the kernels for running the hardware and maximizing what you get with the hardware. Nvidia is slowly tuning the hardware itself, Tensor Cores, all that needs to happen and will continue to happen. All the kernels will get better and utilize the chip to the max extent. All the algorithms will probably improve over optimization, architecture, and all the modeling components of how everything is done and what the algorithms are that we’re even training with. I do expect that nothing dominates. Everything plus 20%. This is roughly what I’ve seen.

01:06:25 – AGI will blend into 2% GDP growth

Dwarkesh Patel 01:06:25

People have proposed different ways of charting how much progress we’ve made towards full AGI. If you can come up with some line, then you can see where that line intersects with AGI and where that would happen on the x-axis. People have proposed it’s the education level. We had a high schooler, and then they went to college with RL, and they’re going to get a Ph.D.

Andrej Karpathy 01:06:44

I don’t like that one.

Dwarkesh Patel 01:06:45

Or they’ll propose horizon length. Maybe they can do tasks that take a minute, they can do those autonomously. Then they can autonomously do tasks that take an hour, a human an hour, a human a week. How do you think about the relevant y-axis here? How should we think about how AI is making progress?

Andrej Karpathy 01:07:05

I have two answers to that. Number one, I’m almost tempted to reject the question entirely because I see this as an extension of computing. Have we talked about how to chart progress in computing, or how do you chart progress in computing since the 1970s or whatever? What is the y-axis? The whole question is funny from that perspective a little bit.

When people talk about AI and the original AGI and how we spoke about it when OpenAI started, AGI was a system you could go to that can do any economically valuable task at human performance or better. That was the definition. I was pretty happy with that at the time. I’ve stuck to that definition forever, and then people have made up all kinds of other definitions. But I like that definition.

The first concession that people make all the time is they just take out all the physical stuff because we’re just talking about digital knowledge work. That’s a pretty major concession compared to the original definition, which was any task a human can do. I can lift things, etc. AI can’t do that, obviously, but we’ll take it. What fraction of the economy are we taking away by saying, “Oh, only knowledge work?” I don’t know the numbers. I feel about 10% to 20%, if I had to guess, is only knowledge work, someone could work from home and perform tasks, something like that. It’s still a really large market. What is the size of the economy, and what is 10% or 20%? We’re still talking about a few trillion dollars, even in the US, of market share or work. So it’s still a very massive bucket.

Going back to the definition, what I would be looking for is to what extent is that definition true? Are there jobs or lots of tasks? If we think of tasks as not jobs but tasks. It’s difficult because the problem is society will refactor based on the tasks that make up jobs, based on what’s automatable or not. Today, what jobs are replaceable by AI? A good example recently was Geoff Hinton’s prediction that radiologists would not be a job anymore, and this turned out to be very wrong in a bunch of ways. Radiologists are alive and well and growing, even though computer vision is really, really good at recognizing all the different things that they have to recognize in images. It’s just a messy, complicated job with a lot of surfaces and dealing with patients and all this stuff in the context of it.

I don’t know that by that definition AI has made a huge dent yet. Some of the jobs that I would be looking for have some features that make it very amenable to automation earlier than later. As an example, call center employees often come up, and I think rightly so. Call center employees have a number of simplifying properties with respect to what’s automatable today. Their jobs are pretty simple. It’s a sequence of tasks, and every task looks similar. You take a phone call with a person, it’s 10 minutes of interaction or whatever it is, probably a bit longer. In my experience, a lot longer. You complete some task in some scheme, and you change some database entries around or something like that. So you keep repeating something over and over again, and that’s your job.

You do want to bring in the task horizon—how long it takes to perform a task—and then you want to also remove context. You’re not dealing with different parts of services of companies or other customers. It’s just the database, you, and a person you’re serving. It’s more closed, it’s more understandable, it’s purely digital. So I would be looking for those things.

But even there, I’m not looking at full automation yet. I’m looking for an autonomy slider. I expect that we are not going to instantly replace people. We’re going to be swapping in AIs that do 80% of the volume. They delegate 20% of the volume to humans, and humans are supervising teams of five AIs doing the call center work that’s more rote. I would be looking for new interfaces or new companies that provide some layer that allows you to manage some of these AIs that are not yet perfect. Then I would expect that across the economy. A lot of jobs are a lot harder than a call center employee.

Dwarkesh Patel 01:11:02

With radiologists, I’m totally speculating and I have no idea what the actual workflow of a radiologist involves. But one analogy that might be applicable is when Waymos were first being rolled out, there’d be a person sitting in the front seat, and you just had to have them there to make sure that if something went really wrong, they’re there to monitor. Even today, people are still watching to make sure things are going well. Robotaxi, which was just deployed, still has a person inside it.

Now we could be in a similar situation where if you automate 99% of a job, that last 1% the human has to do is incredibly valuable because it’s bottlenecking everything else. If it were the case with radiologists, where the person sitting in the front of Waymo has to be specially trained for years in order to provide the last 1%, their wages should go up tremendously because they’re the one thing bottlenecking wide deployment. Radiologists, I think their wages have gone up for similar reasons, if you’re the last bottleneck and you’re not fungible. A Waymo driver might be fungible with others. So you might see this thing where your wages go up until you get to 99% and then fall just like that when the last 1% is gone. And I wonder if we’re seeing similar things with radiology or salaries of call center workers or anything like that.

Andrej Karpathy 01:12:17

That’s an interesting question. I don’t think we’re currently seeing that with radiology. I think radiology is not a good example. I don’t know why Geoff Hinton picked on radiology because I think it’s an extremely messy, complicated profession.

I would be a lot more interested in what’s happening with call center employees today, for example, because I would expect a lot of the rote stuff to be automatable today. I don’t have first-level access to it but I would be looking for trends of what’s happening with the call center employees. Some of the things I would also expect is that maybe they are swapping in AI, but then I would still wait for a year or two because I would potentially expect them to pull back and rehire some of the people.

Dwarkesh Patel 01:13:00

There’s been evidence that that’s already been happening generally in companies that have been adopting AI, which I think is quite surprising.

I also found what was really surprising. AGI, right? A thing which would do everything. We’ll take out physical work, but it should be able to do all knowledge work. What you would have naively anticipated is that the way this progression would happen is that you take a little task that a consultant is doing, you take that out of the bucket. You take a little task that an accountant is doing, you take that out of the bucket. Then you’re just doing this across all knowledge work.

But instead, if we do believe we’re on the path of AGI with the current paradigm, the progression is very much not like that. It does not seem like consultants and accountants are getting huge productivity improvements. It’s very much like programmers are getting more and more chiseled away at their work. If you look at the revenues of these companies, discounting normal chat revenue—which is similar to Google or something—just looking at API revenues, it’s dominated by coding. So this thing which is “general”, which should be able to do any knowledge work, is just overwhelmingly doing only coding. It’s a surprising way that you would expect the AGI to be deployed.

Andrej Karpathy 01:14:13

There’s an interesting point here. I do believe coding is the perfect first thing for these LLMs and agents. That’s because coding has always fundamentally worked around text. It’s computer terminals and text, and everything is based around text. LLMs, the way they’re trained on the Internet, love text. They’re perfect text processors, and there’s all this data out there. It’s a perfect fit.

We also have a lot of infrastructure pre-built for handling code and text. For example, we have Visual Studio Code or your favorite IDE showing you code, and an agent can plug into that. If an agent has a diff where it made some change, we suddenly have all this code already that shows all the differences to a code base using a diff. It’s almost like we’ve pre-built a lot of the infrastructure for code.

Contrast that with some of the things that don’t enjoy that at all. As an example, there are people trying to build automation not for coding, but for slides. I saw a company doing slides. That’s much, much harder. The reason it’s much harder is because slides are not text. Slides are little graphics, they’re arranged spatially, and there’s a visual component to it. Slides don’t have this pre-built infrastructure. For example, if an agent is to make a change to your slides, how does a thing show you the diff? How do you see the diff? There’s nothing that shows diffs for slides. Someone has to build it. Some of these things are not amenable to AIs as they are, which are text processors, and code surprisingly is.

Dwarkesh Patel 01:15:48

I’m not sure that alone explains it. I personally have tried to get LLMs to be useful in domains which are just pure language-in, language-out, like rewriting transcripts, coming up with clips based on transcripts. It’s very plausible that I didn’t do every single possible thing I could do. I put a bunch of good examples in context, but maybe I should have done some kind of fine-tuning.

Our mutual friend, Andy Matuschak, told me that he tried 50 billion things to try to get models to be good at writing spaced repetition prompts. Again, very much language-in, language-out tasks, the kind of thing that should be dead center in the repertoire of these LLMs. He tried in-context learning with a few-shot examples. He tried supervised fine-tuning and retrieval. He could not get them to make cards to his satisfaction.

So I find it striking that even in language-out domains, it’s very hard to get a lot of economic value out of these models separate from coding. I don’t know what explains it.

Andrej Karpathy 01:16:57

That makes sense. I’m not saying that anything text is trivial. I do think that code is pretty structured. Text is maybe a lot more flowery, and there’s a lot more entropy in text, I would say. I don’t know how else to put it. Also code is hard, and so people feel quite empowered by LLMs, even from simple knowledge. I don’t know that I have a very good answer. Obviously, text makes it much, much easier, but it doesn’t mean that all text is trivial.

01:17:36 – ASI

Dwarkesh Patel 01:17:36

How do you think about superintelligence? Do you expect it to feel qualitatively different from normal humans or human companies?

Andrej Karpathy 01:17:45

I see it as a progression of automation in society. Extrapolating the trend of computing, there will be a gradual automation of a lot of things, and superintelligence will an extrapolation of that. We expect more and more autonomous entities over time that are doing a lot of the digital work and then eventually even the physical work some amount of time later. Basically I see it as just automation, roughly speaking.

Dwarkesh Patel 01:18:10

But automation includes the things humans can already do, and superintelligence implies things humans can’t do.

Andrej Karpathy 01:18:16

But one of the things that people do is invent new things, which I would just put into the automation if that makes sense.

Dwarkesh Patel 01:18:20

But I guess, less abstractly and more qualitatively, do you expect something to feel like… Because this thing can either think so fast, or has so many copies, or the copies can merge back into themselves, or is much smarter, any number of advantages an AI might have, will the civilization in which these AIs exist just feel qualitatively different from humans?

Andrej Karpathy 01:18:51

I think it will. It is fundamentally automation, but it will be extremely foreign. It will look really strange. Like you mentioned, we can run all of this on a computer cluster and much faster.

Some of the scenarios that I start to get nervous about when the world looks like that is this gradual loss of control and understanding of what’s happening. I think that’s the most likely outcome, that there will be a gradual loss of understanding. We’ll gradually layer all this stuff everywhere, and there will be fewer and fewer people who understand it. Then there will be a gradual loss of control and understanding of what’s happening. That to me seems the most likely outcome of how all this stuff will go down.

Dwarkesh Patel 01:19:31

Let me probe on that a bit. It’s not clear to me that loss of control and loss of understanding are the same things. A board of directors at TSMC, Intel—name a random company—they’re just prestigious 80-year-olds. They have very little understanding, and maybe they don’t practically actually have control.

A better example is the President of the United States. The President has a lot of fucking power. I’m not trying to make a good statement about the current operant, or maybe I am, but the actual level of understanding is very different from the level of control.

Andrej Karpathy 01:20:06

I think that’s fair. That’s a good pushback. I think I expect loss of both.

Dwarkesh Patel 01:20:15

How come? Loss of understanding is obvious, but why loss of control?

Andrej Karpathy 01:20:20

We’re really far into a territory where I don’t know what this looks like, but if I were to write sci-fi novels, they would look along the lines of not even a single entity that takes over everything, but multiple competing entities that gradually become more and more autonomous. Some of them go rogue and the others fight them off. It’s this hot pot of completely autonomous activity that we’ve delegated to. I feel it would have that flavor.

Dwarkesh Patel 01:20:52

It is not the fact that they are smarter than us that is resulting in the loss of control. It’s the fact that they are competing with each other, and whatever arises out of that competition leads to the loss of control.

Andrej Karpathy 01:21:06

A lot of these things, they will be tools to people, they’re acting on behalf of people or something like that. So maybe those people are in control, but maybe it’s a loss of control overall for society in the sense of outcomes we want. You have entities acting on behalf of individuals that are still roughly seen as out of control.

Dwarkesh Patel 01:21:30

This is a question I should have asked earlier. We were talking about how currently it feels like when you’re doing AI engineering or AI research, these models are more in the category of compiler rather than in the category of a replacement.

At some point, if you have AGI, it should be able to do what you do. Do you feel like having a million copies of you in parallel results in some huge speed-up of AI progress? If that does happen, do you expect to see an intelligence explosion once we have a true AGI? I’m not talking about LLMs today.

Andrej Karpathy 01:22:01

I do, but it’s business as usual because we’re in an intelligence explosion already and have been for decades. It’s basically the GDP curve that is an exponential weighted sum over so many aspects of the industry. Everything is gradually being automated and has been for hundreds of years. The Industrial Revolution is automation and some of the physical components and tool building and all this stuff. Compilers are early software automation, et cetera. We’ve been recursively self-improving and exploding for a long time.

Another way to see it is that Earth was a pretty boring place if you don’t look at the biomechanics and so on, and looked very similar. If you look from space, we’re in the middle of this firecracker event, but we’re seeing it in slow motion. I definitely feel like this has already happened for a very long time. Again, I don’t see AI as a distinct technology with respect to what has already been happening for a long time.

Dwarkesh Patel 01:23:00

You think it’s continuous with this hyper-exponential trend?

Andrej Karpathy 01:23:03

Yes. That’s why this was very interesting to me, because I was trying to find AI in the GDP for a while. I thought that GDP should go up. But then I looked at some of the other technologies that I thought were very transformative, like computers or mobile phones or et cetera. You can’t find them in GDP. GDP is the same exponential.

Even the early iPhone didn’t have the App Store, and it didn’t have a lot of the bells and whistles that the modern iPhone has. So even though we think of 2008, when the iPhone came out, as this major seismic change, it’s actually not. Everything is so spread out and it so slowly diffuses that everything ends up being averaged up into the same exponential. It’s the exact same thing with computers. You can’t find them in the GDP like, “Oh, we have computers now.” That’s not what happened, because it’s such slow progression.

With AI we’re going to see the exact same thing. It’s just more automation. It allows us to write different kinds of programs that we couldn’t write before, but AI is still fundamentally a program. It’s a new kind of computer and a new kind of computing system. But it has all these problems, it’s going to diffuse over time, and it’s still going to add up to the same exponential. We’re still going to have an exponential that’s going to get extremely vertical. It’s going to be very foreign to live in that kind of an environment.

Dwarkesh Patel 01:24:10

Are you saying that, if you look at the trend before the Industrial Revolution to now, you have a hyper-exponential where you go from 0% growth to then 10,000 years ago, 0.02% growth, and to now when we’re at 2% growth. That’s a hyper-exponential. Are you saying if you’re charting AI on there, then AI takes you to 20% growth or 200% growth?

Or are you saying that if you look at the last 300 years, what you’ve been seeing is that you have technology after technology—computers, electrification, steam engines, railways, et cetera—but the rate of growth is the exact same, it’s 2%. Are you saying the rate of growth will go up?

Andrej Karpathy 01:24:46

The rate of growth has also stayed roughly constant, right?

Dwarkesh Patel 01:24:49

Only over the last 200, 300 years. But over the course of human history it’s exploded. It’s gone from 0% to faster, faster, faster. Industrial explosion, 2%.

Andrej Karpathy 01:25:01

For a while I tried to find AI or look for AI in the GDP curve, and I’ve convinced myself that this is false. Even when people talk about recursive self-improvement and labs and stuff like that, this is business as usual. Of course it’s going to recursively self-improve, and it’s been recursively self-improving.

LLMs allow the engineers to work much more efficiently to build the next round of LLM, and a lot more of the components are being automated and tuned and et cetera. All the engineers having access to Google Search is part of it. All the engineers having an IDE, all of them having autocomplete or having Claude code, et cetera, it’s all just part of the same speed-up of the whole thing. It’s just so smooth.

Dwarkesh Patel 01:25:41

Just to clarify, you’re saying that the rate of growth will not change. The intelligence explosion will show up as it just enabled us to continue staying on the 2% growth trajectory, just as the Internet helped us stay on the 2% growth trajectory.

Andrej Karpathy 01:25:53

Yes, my expectation is that it stays in the same pattern.

Dwarkesh Patel 01:25:58

Just to throw the opposite argument against you, my expectation is that it blows up because I think true AGI—and I’m not talking about LLM coding bots, I’m talking about actual replacement of a human in a server—is qualitatively different from these other productivity-improving technologies because it’s labor itself.

I think we live in a very labor-constrained world. If you talk to any startup founder or any person, you can be like, what do you need more of? You need really talented people. And if you have billions of extra people who are inventing stuff, integrating themselves, making companies bottom start to finish, that feels qualitatively different from a single technology. It’s as if you get 10 billion extra people on the planet.

Andrej Karpathy 01:26:44

Maybe a counterpoint. I’m pretty willing to be convinced one way or another on this point. But I will say, for example, computing is labor. Computing was labor. Computers, a lot of jobs disappeared because computers are automating a bunch of digital information processing that you now don’t need a human for. So computers are labor, and that has played out.

Self-driving as an example is also computers doing labor. That’s already been playing out. It’s still business as usual.

Dwarkesh Patel 01:27:13

You have a machine which is spitting out more things like that at potentially faster pace. Historically, we have examples of the growth regime changing where you went from 0.2% growth to 2% growth. It seems very plausible to me that a machine which is then spitting out the next self-driving car and the next Internet and whatever…

Andrej Karpathy 01:27:33

I see where it’s coming from. At the same time, I do feel like people make this assumption of, “We have God in a box, and now it can do everything,” and it just won’t look like that. It’s going to be able to do some of the things. It’s going to fail at some other things. It’s going to be gradually put into society, and we’ll end up with the same pattern. That is my prediction.

This assumption of suddenly having a completely intelligent, fully flexible, fully general human in a box, and we can dispense it at arbitrary problems in society, I don’t think that we will have this discrete change. I think we’ll arrive at the same kind of gradual diffusion of this across the industry.

Dwarkesh Patel 01:28:14

It often ends up being misleading in these conversations. I don’t like to use the word intelligence in this context because intelligence implies you think there’ll be a single superintelligence sitting in a server and it’ll divine how to come up with new technologies and inventions that cause this explosion. That’s not what I’m imagining when I’m imagining 20% growth. I’m imagining that there are billions of very smart human-like minds, potentially, or that’s all that’s required.

But the fact that there’s hundreds of millions of them, billions of them, each individually making new products, figuring out how to integrate themselves into the economy. If a highly experienced smart immigrant came to the country, you wouldn’t need to figure out how we integrate them in the economy. They figure it out. They could start a company, they could make inventions, or increase productivity in the world.

We have examples, even in the current regime, of places that have had 10-20% economic growth. If you just have a lot of people and less capital in comparison to the people, you can have Hong Kong or Shenzhen or whatever with decades of 10% plus growth. There’s a lot of really smart people who are ready to make use of the resources and do this period of catch-up because we’ve had this discontinuity, and I think AI might be similar.

Andrej Karpathy 01:29:33

I understand, but I still think that you’re presupposing some discrete jump. There’s some unlock that we’re waiting to claim. And suddenly we’re going to have geniuses in data centers. I still think you’re presupposing some discrete jump that has no historical precedent that I can’t find in any of the statistics and that I think probably won’t happen.

Dwarkesh Patel 01:29:52

I mean, the Industrial Revolution is such a jump. You went from 0.2% growth to 2% growth. I’m just saying you’ll see another jump like that.

Andrej Karpathy 01:30:00

I’m a little bit suspicious, I would have to take a look. For example, some of the logs are not very good from before the Industrial Revolution. I’m a bit suspicious of it but I don’t have strong opinions. You’re saying that this was a singular event that was extremely magical. You’re saying that maybe there’s going to be another event that’s going to be just like that, extremely magical. It will break the paradigm, and so on.

Dwarkesh Patel 01:30:23

I actually don’t think… The crucial thing with the Industrial Revolution was that it was not magical. If you just zoomed in, what you would see in 1770 or 1870 is not that there was some key invention. But at the same time, you did move the economy to a regime where the progress was much faster and the exponential 10x’d. I expect a similar thing from AI where it’s not like there’s going to be a single moment where we’ve made the crucial invention.

Andrej Karpathy 01:30:51

It’s an overhang that’s being unlocked. Like maybe there’s a new energy source. There’s some unlock—in this case, some kind of a cognitive capacity—and there’s an overhang of cognitive work to do.

Dwarkesh Patel 01:31:02

That’s right.

Andrej Karpathy 01:31:03

You’re expecting that overhang to be filled by this new technology when it crosses the threshold.

Dwarkesh Patel 01:31:06

Maybe one way to think about it is throughout history, a lot of growth comes because people come up with ideas, and then people are out there doing stuff to execute those ideas and make valuable output. Through most of this time, the population has been exploding. That has been driving growth.

For the last 50 years, people have argued that growth has stagnated. The population in frontier countries has also stagnated. I think we go back to the exponential growth in population that causes hyper-exponential growth in output.

Andrej Karpathy 01:31:37

It’s really hard to tell. I understand that viewpoint. I don’t intuitively feel that viewpoint.

01:32:50 – Evolution of intelligence & culture

Dwarkesh Patel 01:32:50

You recommended Nick Lane’s book to me. On that basis, I also found it super interesting and I interviewed him. I have some questions about thinking about intelligence and evolutionary history.

Now that you, over the last 20 years of doing AI research, you maybe have a more tangible sense of what intelligence is, what it takes to develop it. Are you more or less surprised as a result that evolution just spontaneously stumbled upon it?

Andrej Karpathy 01:33:19

I love Nick Lane’s books. I was just listening to his podcast on the way up here. With respect to intelligence and its evolution, it’s very, very recent. I am surprised that it evolved.

I find it fascinating to think about all the worlds out there. Say there’s a thousand planets like Earth and what they look like. I think Nick Lane was here talking about some of the earliest parts. He expects very similar life forms, roughly speaking, and bacteria-like things in most of them. There are a few breaks in there. The evolution of intelligence intuitively feels to me like it should be a fairly rare event.

Maybe you should base it on how long something has existed. If bacteria were around for 2 billion years and nothing happened, then going to eukaryote is probably pretty hard because bacteria came up quite early in Earth’s evolution or history. How long have we had animals? Maybe a couple hundred million years, multicellular animals that run around, crawl, et cetera. That’s maybe 10% of Earth’s lifespan. Maybe on that timescale it’s not too tricky. It’s still surprising to me, intuitively, that it developed.  I would maybe expect just a lot of animal-like life forms doing animal-like things. The fact that you can get something that creates culture and knowledge and accumulates it is surprising to me.

Dwarkesh Patel 01:34:42

There’s a couple of interesting follow-ups. If you buy the Sutton perspective that the crux of intelligence is animal intelligence… The quote he said is “If you got to the squirrel, you’d be most of the way to AGI.”

We got to squirrel intelligence right after the Cambrian explosion 600 million years ago. It seems like what instigated that was the oxygenation event 600 million years ago. But immediately the intelligence algorithm was there to make the squirrel intelligence. It’s suggestive that animal intelligence was like that. As soon as you had the oxygen in the environment, you had the eukaryote, you could just get the algorithm. Maybe it was an accident that evolution stumbled upon it so fast, but I don’t know if that suggests that at the end it’s going to be quite simple.

Andrej Karpathy 01:35:31

It’s so hard to tell with any of this stuff. You can base it a bit on how long something has existed or how long it feels like something has been bottlenecked. Nick Lane is very good about describing this very apparent bottleneck in bacteria and archaea. For two billion years, nothing happened. There’s extreme diversity of biochemistry, and yet nothing grows to become animals. Two billion years.

I don’t know that we’ve seen exactly that kind of an equivalent with animals and intelligence, to your point. We could also look at it with respect to how many times we think certain intelligence has individually sprung up.

Dwarkesh Patel 01:36:07

That’s a really good thing to investigate.

Andrej Karpathy 01:36:09

One thought on that. There’s hominid intelligence, and then there’s bird intelligence. Ravens, etc., are extremely clever, but their brain parts are quite distinct, and we don’t have that much in common. That’s a slight indication of maybe intelligence springing up a few times. In that case, you’d expect it more frequently.

Dwarkesh Patel 01:36:32

A former guest, Gwern, and Carl Shulman, they’ve made a really interesting point about that. Their perspective is that the scalable algorithm which humans have and primates have, arose in birds as well, and maybe other times as well. But humans found an evolutionary niche which rewarded marginal increases in intelligence and also had a scalable brain algorithm that could achieve those increases in intelligence.

For example, if a bird had a bigger brain, it would just collapse out of the air. It’s very smart for the size of its brain, but it’s not in a niche which rewards the brain getting bigger. It’s maybe similar to some really smart…

Andrej Karpathy

Like dolphins?

Dwarkesh Patel

Exaclty, humans, we have hands that reward being able to learn how to do tool use. We can externalize digestion, more energy to the brain, and that kicks off the flywheel.

Andrej Karpathy 01:37:28

Also stuff to work with. I’m guessing it would be harder if I were a dolphin. How do you have fire? The universe of things you can do in water, inside water, is probably lower than what you can do on land, just chemically.

I do agree with this viewpoint of these niches and what’s being incentivized. I still find it miraculous. I would have expected things to get stuck on animals with bigger muscles. Going through intelligence is a really fascinating breaking point.

Dwarkesh Patel 01:38:02

The way Gwern put it is the reason it was so hard is that it’s a very tight line between being in a situation where something is so important to learn that it’s not worth distilling the exact right circuits directly back into your DNA, versus it’s not important enough to learn at all. It has to be something that incentivizes building the algorithm to learn in a lifetime.

Andrej Karpathy 01:38:28

You have to incentivize some kind of adaptability. You want environments that are unpredictable so evolution can’t bake your algorithms into your weights. A lot of animals are pre-baked in this sense. Humans have to figure it out at test time when they get born. You want these environments that change really rapidly, where you can’t foresee what will work well. You create intelligence to figure it out at test time.

Dwarkesh Patel 01:38:55

Quintin Pope had this interesting blog post where he’s saying the reason he doesn’t expect a sharp takeoff is that humans had the sharp takeoff where 60,000 years ago we seem to have had the cognitive architectures that we have today. 10,000 years ago, agricultural revolution, modernity. What was happening in that 50,000 years? You had to build this cultural scaffold where you can accumulate knowledge over generations.

This is an ability that exists for free in the way we do AI training. In many cases they are literally distilled. If you retrain a model, they can be trained on each other, they can be trained on the same pre-training corpus, they don’t literally have to start from scratch. There’s a sense in which it took humans a long time to get this cultural loop going, but it just comes for free with the way we do LLM training.

Andrej Karpathy 01:39:45

Yes and no. Because LLMs don’t really have the equivalent of culture. Maybe we’re giving them way too much and incentivizing not to create it or something like that. But the invention of culture and of written record and of passing down notes between each other, I don’t think there’s an equivalent of that with LLMs right now. LLMs don’t really have culture right now and it’s one of the impediments I would say.

Dwarkesh Patel 01:40:05

Can you give me some sense of what LLM culture might look like?

Andrej Karpathy 01:40:09

In the simplest case it would be a giant scratchpad that the LLM can edit and as it’s reading stuff or as it’s helping out with work, it’s editing the scratchpad for itself. Why can’t an LLM write a book for the other LLMs? That would be cool. Why can’t other LLMs read this LLM’s book and be inspired by it or shocked by it or something like that? There’s no equivalence for any of this stuff.

Dwarkesh Patel 01:40:29

Interesting. When would you expect that kind of thing to start happening? Also, multi-agent systems and a sort of independent AI civilization and culture?

Andrej Karpathy 01:40:40

There are two powerful ideas in the realm of multi-agent that have both not been really claimed or so on. The first one I would say is culture and LLMs having a growing repertoire of knowledge for their own purposes.

The second one looks a lot more like the powerful idea of self-play. In my mind it’s extremely powerful. Evolution has a lot of competition driving intelligence and evolution. In AlphaGo more algorithmically, AlphaGo is playing against itself and that’s how it learns to get really good at Go. There’s no equivalent of self-playing LLMs, but I would expect that to also exist. No one has done it yet. Why can’t an LLM for example, create a bunch of problems that another LLM is learning to solve? Then the LLM is always trying to serve more and more difficult problems, stuff like that.

There’s a bunch of ways to organize it. It’s a realm of research, but I haven’t seen anything that convincingly claims both of those multi-agent improvements. We’re mostly in the realm of a single individual agent, but that will change. In the realm of culture also, I would also bucket organizations. We haven’t seen anything like that convincingly either. That’s why we’re still early.

Dwarkesh Patel 01:41:53

Can you identify the key bottleneck that’s preventing this kind of collaboration between LLMs?

Andrej Karpathy 01:41:59

Maybe the way I would put it is, some of these analogies work and they shouldn’t, but somehow, remarkably, they do. A lot of the smaller models, or the dumber models, remarkably resemble a kindergarten student, or an elementary school student or high school student. Somehow, we still haven’t graduated enough where this stuff can take over. My Claude Code or Codex, they still feel like this elementary-grade student. I know that they can take PhD quizzes, but they still cognitively feel like a kindergarten or an elementary school student.

I don’t think they can create culture because they’re still kids. They’re savant kids. They have perfect memory of all this stuff. They can convincingly create all kinds of slop that looks really good. But I still think they don’t really know what they’re doing and they don’t really have the cognition across all these little checkboxes that we still have to collect.

01:42:55 - Why self driving took so long

Dwarkesh Patel 01:42:55

You’ve talked about how you were at Tesla leading self-driving from 2017 to 2022. And you firsthand saw this progress from cool demos to now thousands of cars out there actually autonomously doing drives. Why did that take a decade? What was happening through that time?

Andrej Karpathy 01:43:11

One thing I will almost instantly push back on is that this is not even near done, in a bunch of ways that I’m going to get to. Self-driving is very interesting because it’s definitely where I get a lot of my intuitions because I spent five years on it. It has this entire history where the first demos of self-driving go all the way to the 1980s. You can see a demo from CMU in 1986. There’s a truck that’s driving itself on roads.

Fast forward. When I was joining Tesla, I had a very early demo of Waymo. It basically gave me a perfect drive in 2014 or something like that, so a perfect Waymo drive a decade ago. It took us around Palo Alto and so on because I had a friend who worked there. I thought it was very close and then it still took a long time.

For some kinds of tasks and jobs and so on, there’s a very large demo-to-product gap where the demo is very easy, but the product is very hard. It’s especially the case in cases like self-driving where the cost of failure is too high. Many industries, tasks, and jobs maybe don’t have that property, but when you do have that property, that definitely increases the timelines.

For example, in software engineering, I do think that property does exist. For a lot of vibe coding, it doesn’t. But if you’re writing actual production-grade code, that property should exist, because any kind of mistake leads to a security vulnerability or something like that. Millions and hundreds of millions of people’s personal Social Security numbers get leaked or something like that. So in software, people should be careful, kind of like in self-driving. In self-driving, if things go wrong, you might get injured. There are worse outcomes. But in software, it’s almost unbounded how terrible something could be.

I do think that they share that property. What takes the long amount of time and the way to think about it is that it’s a march of nines. Every single nine is a constant amount of work. Every single nine is the same amount of work. When you get a demo and something works 90% of the time, that’s just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine. While I was at Tesla for five years or so, we went through maybe three nines or two nines. I don’t know what it is, but multiple nines of iteration. There are still more nines to go.

That’s why these things take so long. It’s definitely formative for me, seeing something that was a demo. I’m very unimpressed by demos. Whenever I see demos of anything, I’m extremely unimpressed by that. If it’s a demo that someone cooked up just to show you, it’s worse. If you can interact with it, it’s a bit better. But even then, you’re not done. You need the actual product. It’s going to face all these challenges when it comes in contact with reality and all these different pockets of behavior that need patching.

We’re going to see all this stuff play out. It’s a march of nines. Each nine is constant. Demos are encouraging. It’s still a huge amount of work to do. It is a critical safety domain, unless you’re doing vibe coding, which is all nice and fun and so on. That’s why this also enforced my timelines from that perspective.

Dwarkesh Patel 01:46:25

It’s very interesting to hear you say that, that the safety guarantees you need from software are not dissimilar to self-driving. What people will often say is that self-driving took so long because the cost of failure is so high. A human makes a mistake on average every 400,000 miles or every seven years. If you had to release a coding agent that couldn’t make a mistake for at least seven years, it would be much harder to deploy.

But your point is that if you made a catastrophic coding mistake, like breaking some important system every seven years...

Andrej Karpathy 01:46:56

Very easy to do.

Dwarkesh Patel 01:46:57

In fact, in terms of wall clock time, it would be much less than seven years because you’re constantly outputting code like that. In terms of tokens, it would be seven years. But in terms of wall clock time...

Andrej Karpathy 01:47:09

In some ways, it’s a much harder problem. Self-driving is just one of thousands of things that people do. It’s almost like a single vertical, I suppose. Whereas when we’re talking about general software engineering, it’s even more... There’s more surface area.

Dwarkesh Patel 01:47:20

There’s another objection people make to that analogy, which is that with self-driving, what took a big fraction of that time was solving the problem of having basic perception that’s robust, building representations, and having a model that has some common sense so it can generalize to when it sees something that’s slightly out of distribution. If somebody’s waving down the road this way, you don’t need to train for it. The thing will have some understanding of how to respond to something like that.

These are things we’re getting for free with LLMs or VLMs today, so we don’t have to solve these very basic representation problems. So now deploying AIs across different domains will sort of be like deploying a self-driving car with current models to a different city, which is hard but not like a 10-year-long task.

Andrej Karpathy 01:48:07

I’m not 100% sure if I fully agree with that. I don’t know how much we’re getting for free. There’s still a lot of gaps in understanding what we are getting. We’re definitely getting more generalizable intelligence in a single entity, whereas self-driving is a very special-purpose task that requires. In some sense building a special-purpose task is maybe even harder in a certain sense because it doesn’t fall out from a more general thing that you’re doing at scale, if that makes sense.

But the analogy still doesn’t fully resonate because the LLMs are still pretty fallible and they have a lot of gaps that still need to be filled in. I don’t think that we’re getting magical generalization completely out of the box, in a certain sense.

The other aspect that I wanted to return to is that self-driving cars are nowhere near done still. The deployments are pretty minimal. Even Waymo and so on has very few cars. They’re doing that roughly speaking because they’re not economical. They’ve built something that lives in the future. They’ve had to pull back the future, but they had to make it uneconomical. There are all these costs, not just marginal costs for those cars and their operation and maintenance, but also the capex of the entire thing. Making it economical is still going to be a slog for them.

Also, when you look at these cars and there’s no one driving, I actually think it’s a little bit deceiving because there are very elaborate teleoperation centers of people kind of in a loop with these cars. I don’t have the full extent of it, but there’s more human-in-the-loop than you might expect. There are people somewhere out there beaming in from the sky. I don’t know if they’re fully in the loop with the driving. Some of the time they are, but they’re certainly involved and there are people. In some sense, we haven’t actually removed the person, we’ve moved them to somewhere where you can’t see them.

I still think there will be some work, as you mentioned, going from environment to environment. There are still challenges to make self-driving real. But I do agree that it’s definitely crossed a threshold where it kind of feels real, unless it’s really teleoperated. For example, Waymo can’t go to all the different parts of the city. My suspicion is that it’s parts of the city where you don’t get good signal. Anyway, I don’t know anything about the stack. I’m just making stuff up.

Dwarkesh Patel 01:50:23

You led self-driving for five years at Tesla.

Andrej Karpathy 01:50:27

Sorry, I don’t know anything about the specifics of Waymo. By the way, I love Waymo and I take it all the time. I just think that people are sometimes a little bit too naive about some of the progress and there’s still a huge amount of work. Tesla took in my mind a much more scalable approach and the team is doing extremely well. I’m kind of on the record for predicting how this thing will go. Waymo had an early start because you can package up so many sensors. But I do think Tesla is taking the more scalable strategy and it’s going to look a lot more like that. So this will still have to play out and hasn’t. But I don’t want to talk about self-driving as something that took a decade because it didn’t take it yet, if that makes sense.

Dwarkesh Patel 01:51:08

Because one, the start is at 1980 and not 10 years ago, and then two, the end is not here yet.

Andrej Karpathy 01:51:14

The end is not near yet because when we’re talking about self-driving, usually in my mind it’s self-driving at scale. People don’t have to get a driver’s license, etc.

Dwarkesh Patel 01:51:22

I’m curious to bounce two other ways in which the analogy might be different. The reason I’m especially curious about this is because the question of how fast AI is deployed, how valuable it is when it’s early on is potentially the most important question in the world right now. If you’re trying to model what the year 2030 looks like, this is the question you ought to have some understanding of.

Another thing you might think is one, you have this latency requirement with self-driving. I have no idea what the actual models are, but I assume it’s like tens of millions of parameters or something, which is not the necessary constraint for knowledge work with LLMs. Maybe it might be with computer use and stuff.

But the other big one is, maybe more importantly, on this capex question. Yes, there is additional cost to serving up an additional copy of a model, but the opex of a session is quite low and you can amortize the cost of AI into the training run itself, depending on how inference scaling goes and stuff. But it’s certainly not as much as building a whole new car to serve another instance of a model. So the economics of deploying more widely are much more favorable.

Andrej Karpathy 01:52:37

I think that’s right. If you’re sticking to the realm of bits, bits are a million times easier than anything that touches the physical world. I definitely grant that. Bits are completely changeable, arbitrarily reshuffleable at a very rapid speed. You would expect a much faster adaptation also in the industry and so on. What was the first one?

Dwarkesh Patel 01:52:59

The latency requirements and its implications for model size?

Andrej Karpathy 01:53:02

I think that’s roughly right. I also think that if we are talking about knowledge work at scale, there will be some latency requirements, practically speaking, because we’re going to have to create a huge amount of compute and serve that.

The last aspect that I very briefly want to also talk about is all the rest of it. What does society think about it? What are the legal ramifications? How is it working legally? How is it working insurance-wise? What are those layers of it and aspects of it? What is the equivalent of people putting a cone on a Waymo? There are going to be equivalents of all that. So I feel like self-driving is a very nice analogy that you can borrow things from. What is the equivalent of a cone in the car? What is the equivalent of a teleoperating worker who’s hidden away and all the aspects of it.

Dwarkesh Patel 01:53:53

Do you have any opinions on what this implies about the current AI buildout, which would 10x the amount of available compute in the world in a year or two and maybe more than 100x it by the end of the decade. If the use of AI will be lower than some people naively predict, does that mean that we’re overbuilding compute or is that a separate question?

Andrej Karpathy 01:54:15

Kind of like what happened with railroads.

Dwarkesh Patel 01:54:18

With what, sorry?

Andrej Karpathy 01:54:19

Was it railroads or?

Dwarkesh Patel 01:54:20

Yeah, it was.

Andrej Karpathy 01:54:21

Yeah. There’s historical precedent. Or was it with the telecommunication industry? Pre-paving the internet that only came a decade later and creating a whole bubble in the telecommunications industry in the late ‘90s.

I understand I’m sounding very pessimistic here. I’m actually optimistic. I think this will work. I think it’s tractable. I’m only sounding pessimistic because when I go on my Twitter timeline, I see all this stuff that makes no sense to me. There’s a lot of reasons for why that exists. A lot of it is honestly just fundraising. It’s just incentive structures. A lot of it may be fundraising. A lot of it is just attention, converting attention to money on the internet, stuff like that. There’s a lot of that going on, and I’m only reacting to that.

But I’m still overall very bullish on technology. We’re going to work through all this stuff. There’s been a rapid amount of progress. I don’t know that there’s overbuilding. I think we’re going to be able to gobble up what, in my understanding, is being built. For example, Claude Code or OpenAI Codex and stuff like that didn’t even exist a year ago. Is that right? This is a miraculous technology that didn’t exist. There’s going to be a huge amount of demand, as we see the demand in ChatGPT already and so on.

So I don’t know that there’s overbuilding. I’m just reacting to some of the very fast timelines that people continue to say incorrectly. I’ve heard many, many times over the course of my 15 years in AI where very reputable people keep getting this wrong all the time. I want this to be properly calibrated, and some of this also has geopolitical ramifications and things like that with some of these questions. I don’t want people to make mistakes in that sphere of things. I do want us to be grounded in the reality of what technology is and isn’t.

01:56:20 - Future of education

Dwarkesh Patel 01:56:20

Let’s talk about education and Eureka. One thing you could do is start another AI lab and then try to solve those problems. I’m curious what you’re up to now, and why not AI research itself?

Andrej Karpathy 01:56:33

I guess the way I would put it is I feel some amount of determinism around the things that AI labs are doing. I feel like I could help out there, but I don’t know that I would uniquely improve it. My personal big fear is that a lot of this stuff happens on the side of humanity, and that humanity gets disempowered by it. I care not just about all the Dyson spheres that we’re going to build and that AI is going to build in a fully autonomous way, I care about what happens to humans. I want humans to be well off in the future.

I feel like that’s where I can a lot more uniquely add value than an incremental improvement in the frontier lab. I’m most afraid of something depicted in movies like WALL-E or Idiocracy or something like that, where humanity is on the side of this stuff. I want humans to be much, much better in this future. To me, this is through education that you can achieve this.

Dwarkesh Patel 01:57:35

So what are you working on there?

Andrej Karpathy 01:57:36

The easiest way I can describe it is we’re trying to build the Starfleet Academy. I don’t know if you’ve watched Star Trek.

Dwarkesh Patel 01:57:44

I haven’t.

Andrej Karpathy 01:57:44

Starfleet Academy is this elite institution for frontier technology, building spaceships, and graduating cadets to be the pilots of these spaceships and whatnot. So I just imagine an elite institution for technical knowledge and a kind of school that’s very up-to-date and a premier institution.

Dwarkesh Patel 01:58:05

A category of questions I have for you is explaining how one teaches technical or scientific content well, because you are one of the world masters at it. I’m curious both about how you think about it for content you’ve already put out there on YouTube, but also, to the extent it’s any different, how you think about it for Eureka.

Andrej Karpathy 01:58:25

With respect to Eureka, one thing that is very fascinating to me about education is that I do think education will pretty fundamentally change with AIs on the side. It has to be rewired and changed to some extent.

I still think that we’re pretty early. There’s going to be a lot of people who are going to try to do the obvious things. Have an LLM and ask it questions. Do all the basic things that you would do via prompting right now. It’s helpful, but it still feels to me a bit like slop. I’d like to do it properly, and I think the capability is not there for what I would want. What I’d want is an actual tutor experience.

A prominent example in my mind is I was recently learning Korean, so language learning. I went through a phase where I was learning Korean by myself on the internet. I went through a phase where I was part of a small class in Korea taking Korean with a bunch of other people, which was really funny. We had a teacher and 10 people or so taking Korean. Then I switched to a one-on-one tutor.

I guess what was fascinating to me was, I think I had a really good tutor, but just thinking through what this tutor was doing for me and how incredible that experience was and how high the bar is for what I want to build eventually. Instantly from a very short conversation, she understood where I am as a student, what I know and don’t know. She was able to probe exactly the kinds of questions or things to understand my world model. No LLM will do that for you 100% right now, not even close. But a tutor will do that if they’re good. Once she understands, she really served me all the things that I needed at my current sliver of capability. I need to be always appropriately challenged. I can’t be faced with something too hard or too trivial, and a tutor is really good at serving you just the right stuff.

I felt like I was the only constraint to learning. I was always given the perfect information. I’m the only constraint. I felt good because I’m the only impediment that exists. It’s not that I can’t find knowledge or that it’s not properly explained or etc. It’s just my ability to memorize and so on. This is what I want for people.

Dwarkesh Patel 02:00:27

How do you automate that?

Andrej Karpathy 02:00:29

Very good question. At the current capability, you don’t. That’s why I think it’s not actually the right time to build this kind of an AI tutor. I still think it’s a useful product, and lots of people will build it, but the bar is so high and the capability is not there. Even today, I would say ChatGPT is an extremely valuable educational product. But for me, it was so fascinating to see how high the bar is. When I was with her, I almost felt like there’s no way I can build this.

Dwarkesh Patel 02:01:02

But you are building it, right?

Andrej Karpathy 02:01:03

Anyone who’s had a really good tutor is like, “How are you going to build this?” I’m waiting for that capability.

I did some AI consulting for computer vision. A lot of times, the value that I brought to the company was telling them not to use AI. I was the AI expert, and they described the problem, and I said, “Don’t use AI.” This is my value add. I feel like it’s the same in education right now, where I feel like for what I have in mind, it’s not yet the time, but the time will come. For now, I’m building something that looks maybe a bit more conventional that has a physical and digital component and so on. But it’s obvious how this should look in the future.

Dwarkesh Patel 02:01:43

To the extent you’re willing to say, what is the thing you hope will be released this year or next year?

Andrej Karpathy 02:01:49

I’m building the first course. I want to have a really, really good course, the obvious state-of-the-art destination you go to to learn, AI in this case. That’s just what I’m familiar with, so it’s a really good first product to get to be really good at it. So that’s what I’m building. Nanochat, which you briefly mentioned, is a capstone project of LLM101N, which is a class that I’m building. That’s a really big piece of it. But now I have to build out a lot of the intermediates, and then I have to hire a small team of TAs and so on and build the entire course.

One more thing that I would say is that many times, when people think about education, they think more about what I would say is a softer component of diffusing knowledge. I have something very hard and technical in mind. In my mind, education is the very difficult technical process of building ramps to knowledge. In my mind, nanochat is a ramp to knowledge because it’s very simple. It’s the super simplified full-stack thing. If you give this artifact to someone and they look through it, they’re learning a ton of stuff. It’s giving you a lot of what I call eurekas per second, which is understanding per second. That’s what I want, lots of eurekas per second. So to me, this is a technical problem of how do we build these ramps to knowledge.

So I almost think of Eureka as maybe not that different from some of the frontier labs or some of the work that’s going on there. I want to figure out how to build these ramps very efficiently so that people are never stuck and everything is always not too hard or not too trivial, and you have just the right material to progress.

Dwarkesh Patel 02:03:25

You’re imagining in the short term that instead of a tutor being able to probe your understanding, if you have enough self-awareness to be able to probe yourself, you’re never going to be stuck. You can find the right answer between talking to the TA or talking to an LLM and looking at the reference implementation. It sounds like automation or AI is not a significant part. So far, the big alpha here is your ability to explain AI codified in the source material of the class. That’s fundamentally what the course is.

Andrej Karpathy 02:04:00

You always have to be calibrated to what capability exists in the industry. A lot of people are going to pursue just asking ChatGPT, etc. But I think right now, for example, if you go to ChatGPT and you say, teach me AI, there’s no way. It’s going to give you some slop. AI is never going to write nanochat right now. But nanochat is a really useful intermediate point. I’m collaborating with AI to create all this material, so AI is still fundamentally very helpful.

Earlier on, I built CS231n at Stanford, which I think was the first deep learning class at Stanford, which became very popular. The difference in building out 231n then and LLM101N now is quite stark. I feel really empowered by the LLMs as they exist right now, but I’m very much in the loop. They’re helping me build the materials, I go much faster. They’re doing a lot of the boring stuff, etc. I feel like I’m developing the course much faster, and it’s LLM-infused, but it’s not yet at a place where it can creatively create the content. I’m still there to do that. The trickiness is always calibrating yourself to what exists.

Dwarkesh Patel 02:05:04

When you imagine what is available through Eureka in a couple of years, it seems like the big bottleneck is going to be finding Karpathys in field after field who can convert their understanding into these ramps.

Andrej Karpathy 02:05:18

It would change over time. Right now, it would be hiring faculty to help work hand-in-hand with AI and a team of people probably to build state-of-the-art courses. Over time maybe some of the TAs can become AIs. You just take all the course materials and then I think you could serve a very good automated TA for the student when they have more basic questions or something like that. But I think you’ll need faculty for the overall architecture of a course and making sure that it fits. So I see a progression of how this will evolve. Maybe at some future point I’m not even that useful and AI is doing most of the design much better than I could. But I still think that’s going to take some time to play out.

Dwarkesh Patel 02:05:59

Are you imagining that people who have expertise in other fields are then contributing courses, or do you feel like it’s quite essential to the vision that you, given your understanding of how you want to teach, are the one designing the content? Sal Khan is narrating all the videos on Khan Academy. Are you imagining something like that?

Andrej Karpathy 02:06:20

No, I will hire faculty because there are domains in which I’m not an expert. That’s the only way to offer the state-of-the-art experience for the student ultimately. I do expect that I would hire faculty, but I will probably stick around in AI for some time. I do have something more conventional in mind for the current capability than what people would probably anticipate.

When I’m building Starfleet Academy, I do probably imagine a physical institution, and maybe a tier below that a digital offering that is not the state-of-the-art experience you would get when someone comes in physically full-time and we work through material from start to end and make sure you understand it. That’s the physical offering. The digital offering is a bunch of stuff on the internet and maybe some LLM assistant. It’s a bit more gimmicky in a tier below, but at least it’s accessible to 8 billion people.

Dwarkesh Patel 02:07:08

I think you’re basically inventing college from first principles for the tools that are available today and just selecting for people who have the motivation and the interest of really engaging with material.

Andrej Karpathy 02:07:26

There’s going to have to be a lot of not just education but also re-education. I would love to help out there because the jobs will probably change quite a bit. For example, today a lot of people are trying to upskill in AI specifically. I think it’s a really good course to teach in this respect. Motivation-wise, before AGI motivation is very simple to solve because people want to make money. This is how you make money in the industry today. Post-AGI is a lot more interesting possibly because if everything is automated and there’s nothing to do for anyone, why would anyone go to a school?

I often say that pre-AGI education is useful. Post-AGI education is fun. In a similar way, people go to the gym today. We don’t need their physical strength to manipulate heavy objects because we have machines that do that. They still go to the gym. Why do they go to the gym? Because it’s fun, it’s healthy, and you look hot when you have a six-pack. It’s attractive for people to do that in a very deep, psychological, evolutionary sense for humanity. Education will play out in the same way. You’ll go to school like you go to the gym.

Right now, not that many people learn because learning is hard. You bounce from material. Some people overcome that barrier, but for most people, it’s hard. It’s a technical problem to solve. It’s a technical problem to do what my tutor did for me when I was learning Korean. It’s tractable and buildable, and someone should build it. It’s going to make learning anything trivial and desirable, and people will do it for fun because it’s trivial. If I had a tutor like that for any arbitrary piece of knowledge, it’s going to be so much easier to learn anything, and people will do it. They’ll do it for the same reasons they go to the gym.

Dwarkesh Patel 02:09:17

That sounds different from using… So post-AGI, you’re using this as entertainment or as self-betterment. But it sounded like you had a vision also that this education is relevant to keeping humanity in control of AI. That sounds different. Is it entertaining for some people, but then empowerment for some others? How do you think about that?

Andrej Karpathy 02:09:41

I do think eventually it’s a bit of a losing game, if that makes sense. It is in the long term. In the long term, which is longer than maybe most people in the industry think about, it’s a losing game. I do think people can go so far and we’ve barely scratched the surface of how much a person can go. That’s just because people are bouncing off of material that’s too easy or too hard. People will be able to go much further. Anyone will speak five languages because why not? Because it’s so trivial. Anyone will know all the basic curriculum of undergrad, et cetera.

Dwarkesh Patel 02:10:18

Now that I’m understanding the vision, that’s very interesting. It has a perfect analog in gym culture. I don’t think 100 years ago anybody would be ripped. Nobody would have been able to just spontaneously bench two plates or three plates or something. It’s very common now because of this idea of systematically training and lifting weights in the gym, or systematically training to be able to run a marathon, which is a capability most humans would not spontaneously have. You’re imagining similar things for learning across many different domains, much more intensely, deeply, faster.

Andrej Karpathy 02:10:54

Exactly. I am betting a bit implicitly on some of the timelessness of human nature. It will be desirable to do all these things, and I think people will look up to it as they have for millennia. This will continue to be true. There’s some evidence of that historically. If you look at, for example, aristocrats, or you look at ancient Greece or something like that, whenever you had little pocket environments that were post-AGI in a certain sense, people have spent a lot of their time flourishing in a certain way, either physically or cognitively. I feel okay about the prospects of that.

If this is false and I’m wrong and we end up in a WALL-E or Idiocracy future, then I don’t even care if there are Dyson spheres. This is a terrible outcome. I really do care about humanity. Everyone has to just be superhuman in a certain sense.

Dwarkesh Patel 02:11:52

It’s still a world in which that is not enabling us to… It’s like the culture world, right? You’re not fundamentally going to be able to transform the trajectory of technology or influence decisions by your own labor or cognition alone. Maybe you can influence decisions because the AI is asking for your approval, but it’s not because I’ve invented something or I’ve come up with a new design that I’m really influencing the future.

Andrej Karpathy 02:12:21

Maybe. I think there will be a transitional period where we are going to be able to be in the loop and advance things if we understand a lot of stuff. In the long-term, that probably goes away. It might even become a sport. Right now you have powerlifters who go extreme in this direction. What is powerlifting in a cognitive era? Maybe it’s people who are really trying to make Olympics out of knowing stuff. If you have a perfect AI tutor, maybe you can get extremely far. I feel that the geniuses of today are barely scratching the surface of what a human mind can do, I think.

Dwarkesh Patel 02:12:59

I love this vision. I also feel like the person you have the most product-market fit with is me because my job involves having to learn different subjects every week, and I am very excited.

Andrej Karpathy 02:13:17

I’m similar, for that matter. A lot of people, for example, hate school and want to get out of it. I really liked school. I loved learning things, et cetera. I wanted to stay in school. I stayed all the way until Ph.D. and then they wouldn’t let me stay longer, so I went to the industry. Roughly speaking, I love learning, even for the sake of learning, but I also love learning because it’s a form of empowerment and being useful and productive.

Dwarkesh Patel 02:13:39

You also made a point that was subtle and I want to spell it out. With what’s happened so far with online courses, why haven’t they already enabled us to enable every single human to know everything? They’re just so motivation-laden because there are no obvious on-ramps and it’s so easy to get stuck. If you had this thing instead—like a really good human tutor—it would just be such an unlock from a motivation perspective.

Andrej Karpathy 02:14:10

I think so. It feels bad to bounce from material. It feels bad. You get negative reward from sinking an amount of time in something and it doesn’t pan out, or being completely bored because what you’re getting is too easy or too hard. When you do it properly, learning feels good. It’s a technical problem to get there. For a while, it’s going to be AI plus human collab, and at some point, maybe it’s just AI.

Dwarkesh Patel 02:14:36

Can I ask some questions about teaching well? If you had to give advice to another educator in another field that you’re curious about to make the kinds of YouTube tutorials you’ve made. Maybe it might be especially interesting to talk about domains where you can’t test someone’s technical understanding by having them code something up or something. What advice would you give them?

Andrej Karpathy 02:14:58

That’s a pretty broad topic. There are 10–20 tips and tricks that I semi-consciously do probably. But a lot of this comes from my physics background. I really, really did enjoy my physics background. I have a whole rant on how everyone should learn physics in early school education because early school education is not about accumulating knowledge or memory for tasks later in the industry. It’s about booting up a brain. Physics uniquely boots up the brain the best because some of the things that they get you to do in your brain during physics is extremely valuable later.

The idea of building models and abstractions and understanding that there’s a first-order approximation that describes most of the system, but then there’re second-order, third-order, fourth-order terms that may or may not be present. The idea that you’re observing a very noisy system, but there are these fundamental frequencies that you can abstract away. When a physicist walks into the class and they say, “Assume there’s a spherical cow,” everyone laughs at that, but this is brilliant. It’s brilliant thinking that’s very generalizable across the industry because a cow can be approximated as a sphere in a bunch of ways.

There’s a really good book, for example, Scale. It’s from a physicist talking about biology. Maybe this is also a book I would recommend reading. You can get a lot of really interesting approximations and chart scaling laws of animals. You can look at their heartbeats and things like that, and they line up with the size of the animal and things like that. You can talk about an animal as a volume. You can talk about the heat dissipation of that, because your heat dissipation grows as the surface area, which is growing as a square. But your heat creation or generation is growing as a cube. So I just feel like physicists have all the right cognitive tools to approach problem solving in the world.

So because of that training, I always try to find the first-order terms or the second-order terms of everything. When I’m observing a system or a thing, I have a tangle of a web of ideas or knowledge in my mind. I’m trying to find, what is the thing that matters? What is the first-order component? How can I simplify it? How can I have a simplest thing that shows that thing, shows it in action, and then I can tack on the other terms?

Maybe an example from one of my repos that I think illustrates it well is called micrograd. I don’t know if you’re familiar with this. So micrograd is 100 lines of code that shows backpropagation. You can create neural networks out of simple operations like plus and times, et cetera. Lego blocks of neural networks. You build up a computational graph and you do a forward pass and a backward pass to get the gradients. Now, this is at the heart of all neural network learning.

So micrograd is a 100 lines of pretty interpretable Python code, and it can do forward and backward arbitrary neural networks, but not efficiently. So micrograd, these 100 lines of Python, are everything you need to understand how neural networks train. Everything else is just efficiency. Everything else is efficiency. There’s a huge amount of work to get efficiency. You need your tensors, you lay them out, you stride them, you make sure your kernels, orchestrating memory movement correctly, et cetera. It’s all just efficiency, roughly speaking. But the core intellectual piece of neural network training is micrograd. It’s 100 lines. You can easily understand it. It’s a recursive application of chain rule to derive the gradient, which allows you to optimize any arbitrary differentiable function.

So I love finding these small-order terms and serving them on a platter and discovering them. I feel like education is the most intellectually interesting thing because you have a tangle of understanding and you’re trying to lay it out in a way that creates a ramp where everything only depends on the thing before it. I find that this untangling of knowledge is just so intellectually interesting as a cognitive task. I love doing it personally, but I just have a fascination with trying to lay things out in a certain way. Maybe that helps me.

Dwarkesh Patel 02:18:41

It also makes the learning experience so much more motivated. Your tutorial on the transformer begins with bigrams, literally a lookup table from, “Here’s the word right now, or here’s the previous word, here’s the next word.” It’s literally just a lookup table.

Andrej Karpathy 02:18:58

That’s the essence of it, yeah.

Dwarkesh Patel 02:18:59

It’s such a brilliant way, starting with a lookup table and then going to a transformer. Each piece is motivated. Why would you add that? Why would you add the next thing? You could memorize the attention formula, but having an understanding of why every single piece is relevant, what problem it solves.

Andrej Karpathy 02:19:13

You’re presenting the pain before you present a solution, and how clever is that? You want to take the student through that progression. There are a lot of other small things that make it nice and engaging and interesting. Always prompting the student.

 There’s a lot of small things like that are important and a lot of good educators will do this. How would you solve this? I’m not going to present the solution before you guess. That would be wasteful. That’s a little bit of a…I don’t want to swear but it’s a dick move towards you to present you with the solution before I give you a shot to try to come up with it yourself.

Dwarkesh Patel 02:19:51

Because if you try to come up with it yourself, you get a better understanding of what the action space is, what the objective is, and then why only this action fulfills that objective.

Andrej Karpathy 02:20:03

You have a chance to try it yourself, and you have an appreciation when I give you the solution. It maximizes the amount of knowledge per new fact added.

Dwarkesh Patel 02:20:11

Why do you think, by default, people who are genuine experts in their field are often bad at explaining it to somebody ramping up?

Andrej Karpathy 02:20:24

It’s the curse of knowledge and expertise. This is a real phenomenon, and I suffered from it myself as much as I try not to. But you take certain things for granted, and you can’t put yourself in the shoes of new people who are just starting out. This is pervasive and happens to me as well.

One thing that’s extremely helpful. As an example, someone was trying to show me a paper in biology recently, and I just instantly had so many terrible questions. What I did was I used ChatGPT to ask the questions with the paper in the context window. It worked through some of the simple things. Then I shared the thread to the person who wrote that paper or worked on that work. I felt like if they could see the dumb questions I had, it might help them explain better in the future.

For my material, I would love it if people shared their dumb conversations with ChatGPT about the stuff that I’ve created because it really helps me put myself again in the shoes of someone who’s starting out.

Dwarkesh Patel 02:21:19

Another trick that just works astoundingly well. If somebody writes a paper or a blog post or an announcement, it is in 100% of cases that just the narration or the transcription of how they would explain it to you over lunch is way more, not only understandable, but actually also more accurate and scientific, in the sense that people have a bias to explain things in the most abstract, jargon-filled way possible and to clear their throat for four paragraphs before they explain the central idea. But there’s something about communicating one-on-one with a person which compels you to just say the thing.

Andrej Karpathy 02:22:07

Just say the thing. I saw that tweet, I thought it was really good. I shared it with a bunch of people. I noticed this many, many times.

The most prominent example is that I remember back in my PhD days doing research. You read someone’s paper, and you work to understand what it’s doing. Then you catch them, you’re having beers at the conference later, and you ask them, “So this paper, what were you doing? What is the paper about?”

They will just tell you these three sentences that perfectly captured the essence of that paper and totally give you the idea. And you didn’t have to read the paper. It’s only when you’re sitting at the table with a beer or something, and they’re like, “Oh yeah, the paper is just, you take this idea, you take that idea and try this experiment and you try out this thing.” They have a way of just putting it conversationally just perfectly. Why isn’t that the abstract?

Dwarkesh Patel 02:22:51

Exactly. This is coming from the perspective of how somebody who’s trying to explain an idea should formulate it better. What is your advice as a student to other students, if you don’t have a Karpathy who is doing the exposition of an idea? If you’re reading a paper from somebody or reading a book, what strategies do you employ to learn material you’re interested in in fields you’re not an expert at?

Andrej Karpathy 02:23:20

I don’t know that I have unique tips and tricks, to be honest. It’s a painful process. One thing that has always helped me quite a bit is—I had a small tweet about this—learning things on demand is pretty nice. Learning depth-wise. I do feel you need a bit of alternation of learning depth-wise, on demand—you’re trying to achieve a certain project that you’re going to get a reward from—and learning breadth-wise, which is just, “Oh, let’s do whatever 101, and here’s all the things you might need.” Which is a lot of school—does breadth-wise learning, like, “Oh, trust me, you’ll need this later,” that kind of stuff. Okay, I trust you. I’ll learn it because I guess I need it. But I love the kind of learning where you’ll get a reward out of doing something, and you’re learning on demand.

The other thing that I’ve found extremely helpful. This is an aspect where education is a bit more selfless, but explaining things to people is a beautiful way to learn something more deeply. This happens to me all the time. It probably happens to other people too because I realize if I don’t really understand something, I can’t explain it. I’m trying and I’m like, “Oh, I don’t understand this.” It’s so annoying to come to terms with that. You can go back and make sure you understood it. It fills these gaps of your understanding. It forces you to come to terms with them and to reconcile them.

I love to re-explain things and people should be doing that more as well. That forces you to manipulate the knowledge and make sure that you know what you’re talking about when you’re explaining it.

Dwarkesh Patel 02:24:48

That’s an excellent note to close on. Andrej, that was great.

Andrej Karpathy 02:24:51

Thank you.

Nick Lane has some pretty wild ideas about the evolution of life.

He thinks early life was continuous with the spontaneous chemistry of undersea hydrothermal vents.

Nick’s story may be wrong, but I find it remarkable that with just that starting point, you can explain so much about why life is the way that it is — the things you’re supposed to just take as givens in biology class:

• Why are there two sexes? Why sex at all?

• Why are bacteria so simple despite being around for 4 billion years? Why is there so much shared structure between all eukaryotic cells despite the enormous morphological variety between animals, plants, fungi, and protists?

• Why did the endosymbiosis event that led to eukaryotes happen only once, and in the particular way that it did?

• Why is all life powered by proton gradients? Why does all life on Earth share not only the Krebs Cycle, but even the intermediate molecules like Acetyl-CoA?

His theory implies that early life is almost chemically inevitable (potentially blooming on hundreds of millions of planets in the Milky Way alone), and that the real bottleneck is the complex eukaryotic cell.

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Gemini in Sheets lets you turn messy text into structured data. We used it to classify all our episodes by type and topic, no manual tagging required. If you’re a Google Workspace user, you can get started today at docs.google.com/spreadsheets/

• Labelbox has a massive network of domain experts (called Alignerrs) who help train AI models in a way that ensures they understand the world deeply, not superficially. These Alignerrs are true experts — one even tutored me in chemistry as I prepped for this episode. Learn more at labelbox.com/dwarkesh

• Lighthouse helps frontier technology companies like Cursor and Physical Intelligence navigate the U.S. immigration system and hire top talent from around the world. Lighthouse handles everything, maximizing the probability of visa approval while minimizing the work you have to do. Learn more at lighthousehq.com/employers

To sponsor a future episode, visit dwarkesh.com/advertise.

Timestamps

(00:00:00) – The singularity that unlocked complex life

(00:08:26) – Early life continuous with Earth’s geochemistry

(00:23:36) – Eukaryotes are the great filter for intelligent life

(00:42:16) – Mitochondria are the reason we have sex

(01:08:12) – Are bioelectric fields linked to consciousness?

Transcript00:00:00 – The singularity that unlocked complex life

Dwarkesh Patel 00:00:00

Today I’m chatting with Nick Lane, who is an evolutionary biochemist at University College London. He has many books and papers which help us reconceptualize life’s 4 billion years in terms of energy flow and helps explain everything from how life came to be in the first place, to the origin of eukaryotes, to many contingencies we see today in how life works.

Nick, maybe a good place to start would be here. Why are eukaryotes so significant in your worldview of why life is the way it is?

Nick Lane 00:00:31

First, thanks for having me here. This is fun. I love talking about this kind of thing.

Eukaryotes. What’s a eukaryote? It’s basically the cells that make us up, but also make up plants and make up things like amoeba or fungi, algae. Everything that’s large and complex that you can see is composed of this one cell type called the eukaryotic cell. We have a nucleus where all the DNA is, where all the genes are, and then all this machinery, cell membranes and things. There’s a lot of kit in these cells.

The weirdness is, if you look inside a plant cell or a fungal cell, it looks exactly the same under an electron microscope to one of our cells. But they have a completely different lifestyle. Why would they have all the same kit, if they evolved to be a single-celled algae living in an ocean doing photosynthesis? It’s still got the same kit that our cells have.

We know that because they share all of these things, they arose once in the whole history of life on Earth. There could have been multiple origins, but there’s no evidence for that. If there was, it disappeared without trace. We’ve got this singularity which happened about 2 billion years ago, about 2 billion years into the history of life on Earth. This thing happens once that gives rise to all complex life on Earth.

The one thing you could conclude from that is bacteria and archaea, in terms of their genetic repertoire, they’ve actually got a lot more genes, a lot more versatility than eukaryotes do. It’s just that a single bacterial cell has much less in it. But there’s so many different types of bacterial cells that overall they’ve explored genetic sequence space. They had 4 billion years to have a go at that and they never came up with a trick which says it’s not in the genes, it’s not about information. There’s something else which is controlling it. That something is the acquisition of these power packs in our cells called mitochondria.

Dwarkesh Patel 00:02:18

Now let’s go to the origins of life. You have this really compelling story where you imagine that the first life forms were continuous with Earth’s geochemistry. Can you recapitulate the story a little bit?

Nick Lane 00:02:37

I’ll tell you how I got there first. I started out working on mitochondria. That took me into the evolution of eukaryotes. Eukaryotes acquire these endosymbionts that become mitochondria and they change the potential of evolution. It doesn’t change everything immediately, but it changes where the endpoints can be. It allows the evolution of these large, complex cells and eventually multicellular organisms and us.

What are mitochondria actually doing? What they’re actually doing is respiration. They’re generating energy for cells. They’re doing plenty of other things as well, but the main thing we can think about is they’re the energy producers. They’re derived from bacteria, and bacteria produce their energy in exactly the same way. They’re generating energy by generating an electrical charge on the membrane.

That charge is small, but the membrane’s really thin. The charge is about 150 to 200 millivolts, but the membrane is five nanometers in thickness, so that’s five millionths of a millimeter. If you shrank yourself down to the size of a molecule and stood next to that membrane, you would experience 30 million volts per meter, which is equivalent to a bolt of lightning. That’s the strength of the force of the voltage across the membrane, which is colossal.

It’s generated by really sophisticated proteins that pump protons across the membrane. Then it’s ATP synthase, which is pretty much universal, and it’s a rotating nanomotor that sits in the membrane. This is colossally complex, interesting machinery, and it’s universally conserved. It’s as conserved as, say, a ribosome, the protein-building factory. It’s pretty much everywhere across life. You wonder, how on earth did life come to be that way? If it’s conserved universally across life, it looks like it goes right back to the common ancestors of all cells. So there’s the question. How did it arise in the first place?

That was, for me, tremendously thrilling because it’s a way in, as a researcher, to the origin of life. How did these energy-generating systems arise in the first place? My way in was, the gates were opened by Bill Martin and Mike Russell who, around the early 2000s, were publishing some amazing papers together. They were saying that in this deep-sea hydrothermal vent, rather than it being like a black smoker with a chimney with smoke belching out of the top, it’s like a mineralized sponge with lots of pores that are cell-like in their structure.

You’ve got an acidic early ocean. You’ve got alkaline fluids coming out of these. You’ve got mixing going on in this whole system. You could at least imagine that you’ve got a pore in here which is a bit like a cell in terms of its size and its shape. On the outside, you’ve got acid ocean waters percolating in, and on the inside, you’ve got these hydrothermal fluids. So you’ve got a barrier, you’ve got an inside and an outside, and you’ve got more protons outside coming in, potentially driving work. It’s very much like a cell is structured.

The other thing is, what are these minerals? You’ve got these mineralized pores with minerals. The minerals, we think, on early Earth would have been a lot of metals in there, things like iron sulfide or nickel sulfides and things like that. The reason that’s important is that what plant cells do, but also what autotrophic bacteria do, is they take CO2 and they take hydrogen and they react them together to make all the building blocks of life. Plants get the hydrogen from water, H2O. They take the H2 out of water and throw away the oxygen, and that collects in the atmosphere. But what bacteria very often do is that they have got hydrogen bubbling out of a hydrothermal vent. They just take the hydrogen straight as gas, and they react it with CO2 and they make all the building blocks of life.

What are the enzymes that they use to do that? They’re very often using these same metals that you would have found in the early oceans—nickel and iron and so on. How are they powering the reaction between hydrogen and CO2? They’re using this membrane potential, the electrical potential—the difference in protons between the outside and the inside—to drive that work, effectively, to power the reaction between hydrogen and CO2 to make organics and drive growth.

This was all in place before I came along. This was coming from Mike Russell and Bill Martin. The details are very uncertain. Whether or not you can really drive any biochemistry that way is very uncertain. But it’s a thrilling idea because you’ve got a continuity between a geological environment and cells as we know them. If it did emerge that way, then it would say, “Here’s why bacteria have got this charge on their membrane,” because it was there in a hydrothermal vent from the beginning. It always powered work from the very beginning. That’s why, in the end, an endosymbiosis that gives rise to eukaryotes would free you from the constraints of generating a charge on the membrane. Now you internalize that in eukaryotes, and now you’re free to become larger and more complex.

You’ve gone from thinking about a puzzle about why eukaryotes are special to thinking about planetary systems and thinking about the origin of life. What are the forces that are going to give rise to life, how would that constrain life, and would we see the same things on other planets or something different? What are the fundamental reasons that it works this way? It becomes astrobiology, really. It’s a thrilling change of perspective to come from my own background, which was to do with mitochondrial biology, an organ transplantation once upon a time, and spinning on a pinhead, you end up working on the origin of life. It’s fantastic.

00:08:26 – Early life continuous with Earth’s geochemistry

Dwarkesh Patel 00:08:26

It’s so fascinating. Just to recapitulate, for my own understanding and the audience’s, let’s just break down what we have here. You have the analog of a cell in these pores. You have something which concentrates the buildup of these organics so that they don’t just all diffuse in some big primordial soup. This is why you think some primordial lake is not where this happened. It had to be concentrated in some entity. Then you’ve got a chemiosmotic gradient, a proton gradient, which drives work. Specifically, it favors the fixation of carbon dioxide to drive the reaction with hydrogen gas to make organics. Then you’ve got, along this membrane, catalysts, which are basically early enzymes. You’ve got enzymes, you’ve got the cell, you’ve got the proton gradient.

The story is that you make very simple organics with CO2 and H2, and then those simple organics are then recatalyzed to make more and more complex organics, and TL;DR, metabolism, fatty acids, and nucleotides, everything else.

Nick Lane 00:09:38

That’s basically it. What do you get if you react hydrogen and CO2? What you get are what are called Krebs cycle intermediates. So carboxylic acids, small molecules made only of carbon, hydrogen, and oxygen, with this organic acid group at the end, which can be 2, 3, 4, 5 carbon units in the chain. This is your basic building blocks. You add on ammonia to this and you get an amino acid. You add more hydrogen on, and you’re going to get a sugar. You react amino acids with sugars, and you’re going to get nucleotides. There are lots of steps along here, but this is the basic starting point for all of biosynthesis in biochemistry.

Dwarkesh Patel 00:10:16

Then if you make fatty acids, they will spontaneously, because of the hydrophilic nature of their different sides, they will spontaneously form a membrane.

Nick Lane 00:10:26

As I say, Krebs cycle intermediates are short-chain carboxylic acids. A fatty acid is a long chain, 10, 12, 15 carbons in the chain instead of four or five. They will spontaneously, not just alone usually but if you’ve got other long-chain hydrocarbons mixed up with them—then you will form a bilayer membrane spontaneously. We’ve done this in the lab, and it’s pretty robust. You can make these things at 70-90° centigrade, across a range of pH from around pH 7 up to about pH 12, and in the presence of ions like calcium and magnesium and other salts and so on.

You make a vesicle with a bilayer membrane around it, which is the same as a cell membrane. They’re amazingly dynamic things. They’re always fusing with each other and breaking apart, fissioning, separating into two or three. They’re very dynamic things under a microscope.

Dwarkesh Patel 00:11:20

You could have imagined that life is this Frankenstein-like moment where things zap alive, and now you’ve got life.

Nick Lane 00:11:28

I hate that as an idea, but go on.

Dwarkesh Patel 00:11:30

Yeah, but that’s the alternative, where the bolt of lightning makes these organics, et cetera. Here you have this story where every life form you see is continuous with something which is continuous with something which is eventually just continuous with entirely spontaneous chemical reactions. That’s just a very interesting way to think about the evolution of life.

Nick Lane 00:11:54

A cell is effectively reduced inside, which is to say it’s got electrons inside. Outside it’s relatively oxidized. You pump all these protons out, so it’s acidic outside, it’s alkaline inside, and it’s reduced inside. That’s like the Earth. All the electrons are in the iron in the core and the mantle of the Earth. It’s relatively alkaline inside. That’s why there’s alkaline fluids in these vents. The outside is relatively oxidized. You’ve got all the CO2 in the oceans.

The cells are a little battery with the same structure as the Earth. If you look in a hydrothermal system, the cell membranes around the Earth, the crust of the Earth is like the membrane. Where you have traffic going between the inside and the outside is the hydrothermal systems. The pores in these hydrothermal systems are little cell-like entities as well. You keep having on multiple scales this same kind of… The idea that the Earth is a giant battery that produces little living, cell-like, mini batteries, it’s a rather beautiful idea. You can’t allow yourself to get too hung up on a metaphor, but it’s a beautiful image.

Dwarkesh Patel 00:13:02

Yeah, 100%. Basically you’ve got Earth as this giant cell, and then from the hydrothermal vent, this little bubble pops off.

Nick Lane 00:13:10

Bubbling off many copies of the Earth.

Dwarkesh Patel 00:13:13

It’s such a fascinating theory. The thing I want to understand is what part of life the way it works now is contingent, and which would you expect to be shared even if you found life on another planet? It sounds like you’re saying that carbon, the chemical profile, is just the obvious candidate to build life on top of. Proton gradients? Is there another way you could build these chemiosmotic gradients that drive work? We have other chemistry.

Nick Lane 00:13:39

In principle, yes, you could use sodium ions instead of protons, but it’s very different. If you’re starting with carbon dioxide, the first thing to realize about that is that carbon is extremely good at the chemistry that it does. It’s forming very strong bonds with all kinds of molecules, so you can form complex, interesting molecules. I think of CO2 as a Lego brick that you pluck out of the air and you bind it onto something. You can build things one brick at a time that way. Then you can build really interesting complex molecules like DNA and RNA from doing that.

You can’t do that with silicon. With intelligent design, you can make really complex AI robots, whatever it may be, but the whole thing requires humans to do it. But if you’re thinking about how life would start on a planet where there isn’t an intelligent designer who’s putting it all together, you need molecules that can do that chemistry, and CO2 is the outstanding example.

Water is everywhere. Hydrogen, oxygen, these are all elements that are very common in the universe. So you’re going to keep on getting this same chemistry everywhere. We know that there are, from discoveries of exoplanets in recent years, if you extrapolate how many we’ve not seen yet, the number of wet rocky planets or moons in, say, the Milky Way is probably in the order of 20, 30, 40 billion of them.

Dwarkesh Patel 00:15:03

What fraction of them would you say have a non-eukaryotic life?

Nick Lane 00:15:07

I’ll take a punt here. I would expect that if you’ve got these same conditions on a wet, rocky planet, you’re going to be producing these same vents because it’s the same chemistry that’s going to happen.

Dwarkesh Patel 00:15:19

So even the vents are not contingent in your view?

Nick Lane 00:15:21

No. The vents are produced by a mineral called olivine, which is really common in interstellar dust. The mantle of the Earth is made of this mineral called olivine. It will react with water. When it reacts with water, it’s slow, if you were to put a lump of olivine in a bucket of water, you’ll not see very much. But if you’re dealing with the pressures down at the bottom of the ocean and warmer temperatures, you’re producing bucket loads of hydrogen gas in alkaline fluids.

That’s what these hydrothermal vents are. Any wet, rocky planet will produce these vents. There’s evidence for them on Mars from the early days of Mars when there were oceans on Mars. There’s evidence now on moons, the icy moons, Enceladus and Europa. This is going on in our own solar system right now.

Dwarkesh Patel 00:16:09

If there are 20–30 billion Earth-like planets and presumably some big fraction of them have these vents if they all have these rock formations, is your view that a notable fraction of them have life that also operates…

Nick Lane 00:16:24

My view would be yes. Any wet, rocky planet would have a decent…

Dwarkesh Patel 00:16:26

With the same metabolism?

Nick Lane 00:16:28

Yes. If you’re starting with CO2 and hydrogen, what I’m saying is the metabolism is thermodynamically favored chemistry. This same chemistry will just go on happening because if you react hydrogen with CO2, and with another CO2 molecule, the parts of the molecules that are going to react are quite predictable.

Dwarkesh Patel 00:16:44

This is a naive question, but what is the reason to think that there are no alternative chemistries which lead to alternative metabolisms?

Nick Lane 00:16:50

Perhaps under very different conditions, you could end up with... But if you’ve got essentially similar conditions… The other thing is that we know that even with very different chemistries, you end up with a similar subset of molecules. The kind of organics you see on meteorites, it’s utterly different chemistry going on. You’re dealing with helium radicals, but you’re still seeing amino acids and you’re still seeing nuclear bases and so on. These are molecules which are basically stable and tend to be formed under a wide range of conditions.

Dwarkesh Patel 00:17:23

So 20 billion Earth-like planets with water and these rocks.

Nick Lane 00:17:27

Not necessarily Earth-like, but wet and rocky.

Dwarkesh Patel 00:17:29

If you just had to pull a number out of nowhere and just say, “This fraction has nucleotides,” what fraction would you say?

Nick Lane 00:17:37

I would say a substantial fraction.

Dwarkesh Patel 00:17:39

Like over 1%?

Nick Lane 00:17:40

Yes. I would imagine 50% or something.

Dwarkesh Patel 00:17:45

Really?

Nick Lane 00:17:46

You say pull a number out of a hat. I’m doing exactly what you’re saying. I’m pulling a number out of a hat. I think this kind of chemistry is going to give you the same nucleotides repeatedly.

Dwarkesh Patel 00:17:55

Again, I know we’re just chatting here. But according to this story, pretty sophisticated organics are extremely abundant throughout the universe.

Nick Lane 00:18:06

That’s not to say they’re collecting in an ocean at a high concentration. What you have in a hydrothermal vent is a continuous throughflow. Within pockets within this vent, within the pores within this vent, bound to the walls pretty much within cells. So within a vent system you could have very high concentrations of things ultimately, but not necessarily in the oceans or in the atmosphere or anywhere else.

Dwarkesh Patel 00:18:28

I guess you could have prokaryotes then who just take over. We did have this, they proliferated through the oceans and changed the composition of the atmosphere.

Nick Lane 00:18:39

Not just the atmosphere, but also the whole of geology. Hundreds of minerals are basically the product of life.

Dwarkesh Patel 00:18:46

So your view is that if eukaryotes are the fundamental bottleneck, you can go from geochemistry to early life, that’s easy. Going from early life to changing the entire composition of the Earth through early prokaryotes is easy. If those two things are easy and then you’ve got 10 billion planets in the Milky Way that have gone to the middle step, does that imply that there’s on the order of 10 billion planets that…

Nick Lane 00:19:10

From nucleotides, you’ve then got to get to RNA and DNA and ribosomes and molecular machines. So there’s a long gap there as well. So just having nucleotides, that’s a requirement to get any further.

Dwarkesh Patel 00:19:27

I see. Again, if you had to pull a number out of the air, what fraction have done that?

Nick Lane 00:19:31

Well, a lower fraction, obviously.

Dwarkesh Patel 00:19:34

Over a billion?

Nick Lane 00:19:35

I would like to be optimistic. I would like to think that these processes are going to drive life into existence on a substantial proportion of these planets or moons. I would expect that there would be similarities in the genetic code. I would expect that a lot of metabolism would look similar. I would expect that they would have a membrane potential driving the work. Because if you’re dealing with CO2 and hydrogen, you’ve got this same fundamental problem. How do you make them react?

Dwarkesh Patel 00:20:04

So there are hundreds of millions of planets in the Milky Way which presumably have something like ribosomes and DNA and RNA?

Nick Lane 00:20:10

Yes, that’s my own thinking. We’re talking about serious planetary driving forces driving fairly deterministic chemistry that’s going to give you the same kind of intermediates which are going to have the same kind of chemistry, the same kind of feedbacks. They’re going to push things into similar directions. Now, the further from CO2 fixation towards genetics you get, the less similarity there’s going to be.

Dwarkesh Patel 00:21:50

This is not my inclination, but if I were a God-fearing person, I would hear this and I’d be like, “Wow, this is a vindication of intelligent design.” The laws of the universe just favor this chemistry which leads to life, at least according to the story, so strongly that it’s hard to resist this formation. I’m curious about your interpretation.

Nick Lane 00:22:12

I agree with you. I find it almost a little disturbing. I have to say that I’m not a religious person either, but I don’t object to religion. I’m not a militant atheist at all. I like the fact that religions have searched for meaning and searched for origins. I have some fellow feeling with that search and truth in some sense, with a small T, in my own case.

But insofar as this is consistent with the idea of a God, the God would be a deist God that effectively set the laws of the universe in motion and they’re left to play out. This is Einstein’s God. In terms of what most people understand by God, most people look for comfort in God and are looking for something which is meaningful to them and who’s been involved in humanity. This is a very cold kind of “God as thermodynamics” who sets the laws of the universe in motion, reproducibly gives rise to the same kinds of things. Yes, you could interpret it in a natural theistic way, but I don’t think many people would get that much comfort or meaning from that way of seeing the world.

00:23:36 – Eukaryotes are the great filter for intelligent life

Dwarkesh Patel 00:23:36

A very basic question, but if life is not only abundant but almost inevitable in all these rocky planets, then the bottleneck to not seeing aliens everywhere, presumably, is eukaryotes which leads to complexity.

Nick Lane 00:23:53

There’s more than one bottleneck, but eukaryotes is in my own mind the big one.

Dwarkesh Patel 00:23:59

It would have to be the case that out of billions of potential planets that could give rise to eukaryotes, only on Earth does this chance occurrence happen.

Nick Lane 00:24:10

I wouldn’t argue that. Only on Earth? No, I don’t think so. What I would dig my heels in about a little bit is there’s a Carl Sagan cosmological view. We’re talking about almost the inevitability of life arising according to these laws of chemistry and thermodynamics, and you get life. Then is it going to roll on and inevitably give rise to complex life and to humans and to intelligence?

It’s a beautiful thought. It would be lovely if that was how the universe worked. But what we know on Earth is that you have 2 billion years of stasis, and then this apparent singular event where eukaryotes arose, and then another long gap before you get to animals. If you roll back the clock 2 million years, there aren’t any humans around either. We’re just the icing.

Dwarkesh Patel 00:25:07

Why is it supposedly this hard to have this successful endosymbiotic event?

Nick Lane 00:25:14

There are multiple reasons. One of them is that prokaryotes, we should say archaea and bacteria, are pretty small things. Having another cell inside you is already a difficult thing to do. There are occasional phagocytes in bacteria that can engulf other cells, but it’s pretty uncommon. Once you’ve got these cells inside you, that may have happened on scores of occasions. There’s some tentative evidence that suggests it happened with archaea. There’s one nice example where the haloarchaea seem to have acquired more than a thousand bacterial genes from the same source, implying perhaps they had got an endosymbiont that they then lost later on.

The question is, how often would it go wrong and you lose your endosymbiont? I guess that would be the more likely outcome, that you pick up a bunch of genes and you lose your endosymbiont. It simply doesn’t work out.

It’s hard to know exactly what all the bottlenecks are here. But there has been some modeling work done to see if you get an endosymbiont, are you going to grow faster if you don’t have the endosymbiont or you do have the endosymbiont? And if you’re the endosymbiont, are you going to grow faster if you’re outside or if you’re inside? Under most conditions that these people have looked at there in Santa Fe, the answer is you do better if you’re not part of the symbiosis. Only under certain conditions will you do better. So predictably, the endpoint is that it doesn’t work.

Dwarkesh Patel 00:26:39

Given how many bacteria and archaea there are, throughout Earth’s history there are trillions, trillions, and trillions of these running around, there are many situations in which there was an endosymbiosis, and in only one case it succeeded. The odds would have to be remarkable. It would have to be extremely, extremely tough.

Nick Lane 00:27:01

It is a vivid way of seeing it. We know what bacteria and archaea look like and people have been studying these things and finding new examples. There’s a group discovered 10 years ago called the Asgard archaea, and they’re relatively eukaryotic-like, which is to say they’ve got proteins in there and genes that are pretty similar to eukaryotic ones. They’re interesting cells. They’ve got long processes and possibly they can move vesicles around inside them. So they’re doing a few eukaryotic things. But if you look at their internal structure, it’s not very complex. It’s nothing like a eukaryotic cell. And if you look at their genome size, it’s a standard prokaryotic genome size. You’re talking four or five thousand genes. So these are not eukaryotic by any stretch of the imagination.

Then you look at a eukaryotic cell. I said this at the beginning, you look at a plant cell or an animal cell or a fungal cell, or an alga or amoeba under a microscope, and they’ve all got the same stuff, and it’s weird. Why would a single-celled alga living in the ocean have all the same kit that one of my kidney cells has? The easiest way to understand that is to say it wasn’t adaptation to an external environment to a way of life. It was adaptation to an internal selection pressure

If you think about it in terms of a battle between the host cell and the endosymbiont for finding a way of living together, you can argue for the nucleus arising that there is all kinds of genetic parasites coming out of the mitochondria forcing you to do something to protect your own genome. So you can construct a lot of this history of eukaryogenesis, it’s called. You start with simple cells with a cell inside, and you end up with the same cell structure everywhere, all these endomembrane systems and everything else.

Dwarkesh Patel 00:28:44

The broader thing we’re trying to understand here is if this story is true, there’s life everywhere. But eukaryotes giving rise to intelligent life, which is about to go explore the cosmos, is as far as we can tell, happening only in one place in our light cone. So why is that? You could say, “Well, the bottleneck is the eukaryote and it is very hard to get a successful endosymbiosis which then continues over time. But what is the fundamental problem this is solving?

Nick Lane 00:29:18

Large genomes. To have a multicellular organism where effectively you’re deriving from a single cell, that restricts the chances of effectively all the cells having a fight. There are plenty of examples of multicellular slime molds, for example, where the cells come together. They can form structures like a stalk, for example, which loosens spores into the environment, but they fight because they’re genetically different to each other. So you start with a single cell and you develop, so there’s less genetic fighting going on between the cells than there would be if they come together.

But that means then if you want to have complex functions—if you want to have a liver doing one thing and kidneys doing something else, and the brain doing something else—all of the cells have to have the same genes. You express this lot in the liver and that lot in the brain. So you must have a large genome. The only way you can have a large genome is by having mitochondria and having a eukaryotic cell. There are no examples of this level of sophistication of a multicellular bacterium.

Dwarkesh Patel 00:30:18

That’s quite interesting. The reason you need a large genome is just to put all your eggs in one basket so that every cell in the body feels incentivized to make the germline continue.

Nick Lane 00:30:28

You’re restricting the amount of fighting.

Dwarkesh Patel 00:30:31

The thing I was getting at is, the eukaryote is solving for a large genome. It’s allowing the cell to get much bigger. Why are we so confident that this is the only way this problem could have been solved? It just seems like if there are billions of planets which have gotten to the precursor stage here, none of them can find an alternative solution to mitochondria for just letting themselves get bigger?

Nick Lane 00:30:56

Beggars belief. I know where you’re coming from.

Dwarkesh Patel 00:30:58

It kind of makes me wonder whether we’re, because we’ve only observed one way to solve this solution, we’re assuming that there must be only one way to solve the problem. The problem itself doesn’t seem… You just want a smaller copy of the genome sitting next to the site of respiration. That’s the basic problem. There’s no other way to solve that?

Nick Lane 00:31:13

Maybe there is, but I think we have to look at the probability of certain things happening. If you want to have a giant bacterium, there are a bunch of giant bacteria around on Earth. There’s at least six or seven different, quite unrelated species that have evolved giant size. The thing that they all have in common is they have what’s called extreme polyploidy, which is to say they have literally tens of thousands of copies of their complete genome. So it may be a small genome, we’re talking a three-megabase genome, so around 3,000 genes in it. And you’ve got tens of thousands of copies. Sometimes the very largest ones have 700,000 to 800,000 copies of their complete genome. The energy requirements for copying all of that and expressing all of those genomes are colossal.

What we have with endosymbiosis, we still have extreme polyploidy, but we’ve whittled away all the genes that you don’t need. A symbiosis is based on effectively complementarity. You’ve got a symbiont that’s doing something for the host cell and the host cell that’s taking something or giving something back to the endosymbiont. So it’s a relationship which is based on mutual needs. One of them becomes much smaller and that allows the other one to become much larger.

So a symbiosis will do it. Now there could be multiple ways of having a symbiosis, but there’s no examples of it. All of these examples of very large bacteria and they all have extreme polyploidy. None of them have come up with a complex trafficking network where you effectively take things in and you ship it over there. There’s just not enough genetic space.

Dwarkesh Patel 00:32:49

But just to make sure I understood the feature request correctly, it’s basically: you want a smaller copy of the genome that is only relevant to respiration sitting across the entire membrane, and many copies of it sitting across the entire membrane. I guess I’m just, it seems hard for me to…

Nick Lane 00:33:05

You’re incredulous that this same thing would be repeated. Yes.

Dwarkesh Patel 00:33:10

There’s no other way to solve this on the billions of planets? Because if there were another way to solve it, then what you would expect is that as soon as you get to the stage of prokaryotes that have other niches that they could colonize if only they could drive towards complexity, this would somehow be solved and then you’d have eukaryote, and then intelligence…

Nick Lane 00:33:28

A couple of things I’d say. Number one, there’s a thing called Orgel’s second rule, which is that evolution is cleverer than you are. Of course, I cannot say that there’s no other way that it could possibly happen. But it’s also hand-waving to say, “Oh, evolution’s so clever, the universe is so big, there’s got to be another way that it can happen.” You know, engage your brain and tell me how it’s going to work.

I cannot say it’s the only way it could possibly happen. But what I’ve said is that wet, rocky planets are common. They’re everywhere. You’re going to have these same serpentinizing things. You’re going to have CO2. You’re going to have a similar biochemistry. You’re going to give rise to bacterial cells that have got a charge on their membrane. That constrains them and every example that we know on Earth where they seem to have got bigger, there’s a constraint that probabilistically happens every time. They always end up with extreme polyploidy and they don’t end up with sophisticated transport networks.

So that’s not to say it’s got to happen that way every time. Maybe there’s a way around it, but it’s not an easy way around it because they haven’t done it regularly on Earth. They haven’t done it at all on Earth. The only occasion where it worked on Earth was where they came up with eukaryotes. That’s not to say it’s the only possible way of doing it. But if you try and dissect, what are the alternatives? I can’t think of any. Ok, I’m limited. But if you think there are some, then you tell me what they might be and you test them.

I get this a lot, and it’s fair enough. Because if I assert to you that life’s going to be this way somewhere else in the universe… I grew up watching Star Wars and Star Trek and reading Hitchhiker’s Guide to the Galaxy. I love the idea that the universe is full of all kinds of stuff as much as anybody. So I don’t like my position of saying, “Actually it’s quite limited and you’re going to see the same kinds of things elsewhere.”

It’s not a position that I dreamt of having. It’s just a position that I’ve been forced into by everything that I’ve learned about life on Earth. Now, maybe I’m just wrong. But if you simply say you’re limited by your imagination, you’re wrong because you can’t think of it. Well, that’s not science anymore. Now we’re talking about just imagination and hand-waving, but it’s not science.

So I’m giving reasons why probabilistically it’s going to be this way. What I would say is if you’ve got a thousand planets with life on them, maybe life is going to be the same way 999 out of a thousand times because it’s going to be carbon-based, it’s going to be water, it’s going to be cells, it’s going to be charges, it’s going to be hydrogen and CO2, and you’re going to face the same constraints. But maybe one other occasion, it’s something completely different that I never thought of and under very different conditions. But there’s a probabilistic thing that carbon is so common, water is so common, you are going to keep seeing the same constraints again and again.

Dwarkesh Patel 00:36:20

If it’s the case that a significant fraction of rocky planets should have at least organics and cells and so forth, it feels like we should be able to learn pretty soon whether this story is correct, right? If that part ends up being true, and we also don’t see eukaryotes elsewhere, then the whole picture is lent a lot more credence. But are we about to go to a couple of moons and see if we can find some organics there and so forth?

Nick Lane 00:36:50

That may take us a while but yeah, we already know that there are organics. On Enceladus, for example, one of the moons of Saturn, when Cassini flew by some years ago, there were plumes coming through cracks in the ice of water, but with organics dissolved in the water, and hydrogen and organic molecules. The pH is around eight or nine, so it implies that underneath that frozen surface, which people say is about 5km thick, underneath that there’s a liquid ocean. Underneath that there are hydrothermal systems producing alkaline fluids which have made the oceans alkaline. It’s the same chemistry going on.

So we know there’s organics in these plumes. We don’t know what’s under the ice. I do think that the incentives to go to these places and drill into the ice and have a look will get the better of us. There will always be people saying we shouldn’t introduce bacteria from our own system into there. Bacteria from the Earth would probably survive extremely well in a place like Enceladus. It would be lovely to know, and I’m all in favor of exploration.

Dwarkesh Patel 00:39:09

Help me understand how replicators arise in this world. If you’ve got these independent pores and they’re each individually accumulating their own organics through the spontaneous processes, initially at least there’s no shared inheritance. It’s not like if there’s a very successful pore, it then causes there to be more pores exactly like it.

Nick Lane 00:39:31

Think what I would call protocells inside these pores. The organics that you’re making are self-organizing. A fatty acid bilayer membrane will form. What you really need for positive feedbacks is to be making the organics inside this protocell and for that protocell to grow and to make a copy of itself. Now it will make a copy of itself, because the chemistry, if the chemistry is deterministic, it says, this is the chemistry you’re going to get. If you drive that chemistry through by the pressure of hydrogen in the system, you’re just going to make twice as many molecules and they’re going to divide in two. Now you’ve got two protocells. So there’s a form of heredity to that, which is they get the same molecules because that’s effectively all you’re allowed to do.

Dwarkesh Patel 00:40:09

So the thing buds off and then settles into another pore?

Nick Lane 00:40:14

Yes.

Dwarkesh Patel 00:40:15

I see. Okay got it. And this happens relatively early in this process?

Nick Lane 00:40:18

Yes.

Dwarkesh Patel 00:40:18

So the rise of replicators happens relatively early.

Nick Lane 00:40:21

I would hesitate to use the word replicator here. These are growing. I would say they are growing protocells that are effectively making more of themselves. You could call it a replicator, but I would prefer to use the word replicator for something more like RNA, which would be the conventional term for a replicator, where you are literally replicating the exact sequence of this RNA.

Dwarkesh Patel 00:40:42

At what point do we get to the gene’s-eye point of view, where the gene is the coherent unit of replication?

Nick Lane 00:40:50

The sooner the better. Which is to say, if you’ve got this deterministic chemistry, which is going to drive growth and make more cells, it’s also a dead end. You can’t do anything else. You’re entirely dependent on the environment. You can’t evolve into something more complex. To some extent you can, but you’re always going to get the same thing. The same environment will always give you the same thing.

As soon as you start introducing random bits of RNA into this, then you’ve got what you call evolvability, which is to say you can begin to resist the environment. You can begin to do things which are not just dictated by the environment. You can evolve and change and leave vents in the end and do other things. So as soon as you’ve got genes, you’ve got the potential to do almost anything.

If you’ve got naked bits of RNA, what tends to happen is they’re selected for their replication speed. They just go on making copies of themselves. They don’t become more complex, they don’t start encoding metabolism, they just go on copying themselves and it’s a dead end. If you’re trapping them inside growing protocells, then effectively they’re sharing the same fate. If some of them are capable of making that protocell grow faster, then they will get more copies of themselves because they’re inside this protocell. The protocell’s growing faster, it makes a copy of itself and it’s still associated. So you’ve got selection as we know it in cells today, where the replicators are the genes, but the system which is being reproduced is the cell.

00:42:16 – Mitochondria are the reason we have sex

Dwarkesh Patel 00:42:16

Your sort of mitochondria-first viewpoint that helps explain why there’s two sexes. Maybe you can recapitulate that argument. But I’m just curious if there were a world where prokaryotes had evolved sex, do you think they would have likely evolved just one sex?

Nick Lane 00:42:32

I’m going to unpack that a little bit because what have mitochondria got to do with sex? So what they have to do with sex is that effectively the female sex – and this goes even for single cells, things that don’t have any obvious differences between gametes, which is to say they don’t have oocytes and sperm or anything. They produce little motile gametes that look more like sperm than anything else. Both sexes would do that. But by definition, the female sex passes on the mitochondria and the male does not. That’s an approximation, it’s not always true. There are exceptions to that rule, but it’s a rule of thumb in biology that the females pass on the mitochondrial DNA.

Why would that happen? With sex, what you’re doing is you’re increasing the variance in the nuclear genome and you’re subjecting that to selection, and the winners are coming through that. Everything which is worse than it would have been gets eliminated by selection. So you’re increasing variance on nuclear genes, the genomes, and then selecting for what works.

With the mitochondria, they’re passing on asexually down the generations. There’s a very small genome, but there are multiple copies of it. The question is, how do you keep that clean? How do you prevent that from degrading and degenerating over time? Because let’s say you’ve got 100 copies of mitochondrial DNA and two of them acquire mutations, but you’ve still got 98 which are doing their job fine, what’s the penalty for those two mutations? It’s not very much. You’ll hardly notice them. Now you acquire another couple of mutations and you can degenerate over time, a process called Muller’s ratchet. It’s basically, these mutations are somewhat screened from selection by being compensated for by other clean copies you have.

So how do you get rid of those mutations that are building up over time? Well, the answer is what you need to do is increase variance of mitochondrial genes. What you need to do is effectively segregate into these cells all the mutants and into those ones, all the wild-type ones. You can do that by multiple rounds of cell division. But it helps if you’ve got two sexes where effectively only one sex passes on the mitochondria. You’re already sampling, you’re already increasing the variance and you’re increasing visibility to selection. It’s about the quality of mitochondrial genes.

Dwarkesh Patel 00:44:57

Can you help me understand why it’s the case that uniparental inheritance of mitochondria helps increase variance?

Nick Lane 00:45:02

We’re talking about variance between cells. If you imagine that you have 100 cells and they all come from the same parent, let’s say. If you give all the mitochondria that you have straight into a single cell without changing any of the ratios there, then it’s exactly the same as you are. It’s fully clonal.

But if you take a small subsection of those and say you take a random 10%—you give 10% to this one, a random 10% to that one, a random 10% to this one—randomly, this cell is going to happen to have got all the good copies and this cell is going to happen to have got all the bad copies. Now you subject these 100 cells to selection and say, “How are you doing?” The one that got all the good copies does well, that gets on.

So what you’re doing is increasing the variance between this next generation of cells. The ones that got all the mutants, they get hit. The ones that got all the clean copies, they do all right. The parent had got both the mutations and the clean copies. But how do you distinguish between them? It’s about sampling, basically. And uniparental inheritance is a form of sampling. You’re taking the mitochondria only from one of the two parents. You’re not mixing up mutations that both parents had. You’re taking a subset. So you’re always increasing variance between the daughter cells. Uniparental inheritance is giving you a subset.

Dwarkesh Patel 00:46:35

Then there’s the question of why there’s two sexes. We explained why there’s this evolutionary niche for only one parent to pass on the mitochondria. So there’s at least two niches. One is to pass on the mitochondria, one is don’t pass on the mitochondria. Once you’ve established those two, then you can ask the question, “Why aren’t there more than two sexes?” Then you could just say, “Well, there would just be a repetition of one of these two. These are the two fundamental ones.”

Nick Lane 00:46:57

I mean, it’s more complex, but the thing about two sexes is you could say it’s the worst of all possible worlds. Let’s take it away from humans so we can be dispassionate about it. You’ve got these single-celled critters swimming around and they’re all producing gametes. The gametes look the same as each other and they’ll fuse in the same way as sex and they’ll line up the chromosomes. They do exactly the same thing that we do on a single-cell scale.

But having two sexes means that you can only mate with 50% of the population. The other 50% is the same sex as you and it’s not going to accept your gametes. If you had three sexes or four sexes, then you would be able to mate with a larger proportion of the population. With some fungi, they still have two sexes, but they have mating types as well. You can have 27,000 mating types in some fungi, which is all about outbreeding. So you can mate with just about anything.

Dwarkesh Patel 00:47:52

If you’ve been to some college campuses today, they’re replicating some portion of that.

Nick Lane 00:47:57

Becoming fungal, yes. Two sexes then, in that sense, is the worst of all possible worlds. If you had only one sex, if everyone was a hermaphrodite, you could mate with everybody. If you had three sexes, you could mate with two-thirds of the population and so on.

So why two? Well, there’s the fundamental difference that one is passing on the mitochondria and the other is not. Beyond that, if you’ve got multiple mating types, you still have one that passes on the mitochondria and the other one doesn’t. So in these fungi that have all of these mating types, there’s kind of a pecking order that the dominant one will pass on the mitochondria and the less dominant one doesn’t pass on the mitochondria.

So you end up with really complex systems. You can imagine it’s pretty hard to enforce this. Stuff can go wrong. The more complex the system is, the more it will go wrong. So I guess in that sense, why do you end up with two sexes? It’s partly a minimization of error.

Dwarkesh Patel 00:48:48

You have this really interesting discussion about how this not only explains why there’s two sexes, but the particular differences in why eggs and sperm developed the way they do, why there’s different amounts of replications before they are mature, et cetera. I wonder if we can recapitulate that.

Nick Lane 00:49:04

So as soon as you’ve got this fundamental difference, even in single-celled critters, that one of the sexes passes on the mitochondria and the other one doesn’t… Males do not pass on their mitochondria. This is beginning to explain differences in multicellular organisms between the sexes, between the nature of the germline.

In some sense, men do not really have a germline in the sense that women have a germline. In the female germline, you make these oocytes and you put them on ice effectively. You look after them, you switch them off as much as you can, you try and protect them from mutations, you mollycoddle them effectively. Whereas men just mass-produce sperm full of mutations. There’s a lovely phrase from James Crow, who’s a geneticist: “there’s no greater genetic health hazard in the population than fertile old men.”

So why would you go on mass-producing sperm all the time? Part of it is you don’t have to pass on the mitochondria, so you’re freeing yourself up to mass-produce sperm. Some of them are full of mutations, but a lot of them aren’t. You mass-produce them and the chances are it’s going to work out okay because the ones that can swim best, for example, are the ones that are more likely to… That’s not strictly true, but you can imagine it along those lines.

But in the case of the oocytes, in the case of the egg cells, you’re passing on those mitochondria. You don’t want to be accumulating mutations in that mitochondrial DNA. You want to switch them off as much as possible, keep them on ice as much as possible. Very much the differences between how the sexes end up becoming different to each other, boils down to what are the constraints on your reproductive system.

Dwarkesh Patel 00:50:52

Let’s talk about the Y chromosome, which is also not recombined. Just the same way that female egg cells try to minimize the amount of duplications in order to preserve the quality of the mitochondrial DNA and prevent errors, why isn’t the same thing happening with the Y chromosome? Shouldn’t all this sperm duplication be resulting in all kinds of errors in the Y chromosome?

Nick Lane 00:51:18

Well, it does. The Y chromosome is degenerate.

Dwarkesh Patel 00:51:22

I’m going to make that the title.

Nick Lane00:51:25

But there are some things that have lost their Y chromosome altogether. And they still have sexes because it’s not strictly dependent on the Y chromosome. If you look at what determines sexes across the whole canvas of evolution, it’s weird. Because amphibians, for example, have temperature-dependent sex determination. So males would develop at a higher temperature than females, or sometimes it’s the other way around. Birds have different sex chromosomes to mammals, for example. So sex chromosomes have evolved on multiple different occasions.

What’s the Y chromosome doing? Well, the Y chromosome is encoding a growth factor, and that growth factor switches on other growth factors. The earliest difference that you could tell between the two sexes in embryonic development is not the activation of the Y chromosome, the SRY gene. It’s the growth rate. There was a woman at UCL, where I am, called Ursula Mittwoch, who spent her career… She had about 15 Nature papers in the 1960s. She worked on these kinds of questions. She saw the growth rate as the common denominator that the Y chromosome is saying, “Grow fast.”

Why would you grow fast? Well, in part, you can grow fast. You don’t have any constraints on trashing your own mitochondria because you’re not passing them on. So you can grow fast. This might be an advantage to growing fast. If you’re a male, you’re going to get the resources. You grow faster. If you’re a female, you don’t want to grow so fast because you need to effectively cordon off your germline to preserve the oocytes for the next generation. Until you’ve done that, you don’t want to trash your mitochondria. So you’ve got a delay phase before you can start growing fast.

Dwarkesh Patel 00:53:08

Interesting. Is this why women live longer?

Nick Lane 00:53:11

Ursula Mittwoch argued that that was exactly the case. We don’t know for a fact that that’s true. But it’s quite common that females live longer than males, not just in humans, but in Drosophila as well, they usually do.

Dwarkesh Patel 00:53:22

Suppose that evolution on humans just continued naturally for the next billion years. We didn’t have AGI and human gene editing, et cetera. Is the equilibrium that you’d anticipate that the Y chromosome would then just fade away altogether and there’d be some other way of determining sex and sex-dependent characteristics?

Nick Lane 00:53:44

There are, and it has disappeared altogether in some species. Usually what you retain is one gene which causes a different rate of growth. The Y chromosome is degenerate. It’s lost most of its genes. The thing about Muller’s ratchet—which is the degradation of things when you don’t have sex or you don’t have any recombination—is that there are two factors that influence it. One of them is the population size. In bacteria, if you’ve got a small population and they’re not sexual, then you accumulate mutations in that population. But if you’ve got a much larger population, the closer you get towards an infinitely large population, they’re not all going to accumulate the same mutations. So the population as a whole is going to be fine. This goes back decades in population genetics.

The other thing which is less explored in population genetics is the size of the genome. With bacteria, if you increase their genome size up to eukaryotic-sized genomes, you can’t maintain a larger genome. You’ll accumulate mutations in that genome, and it’ll shrink again. With the Y chromosome, yes it shrunk. It’s a tiny chromosome in comparison with all of the rest. It’s really how many genes can you maintain in a good state?

With the Y chromosome, you only need a couple of genes in there. It’s the SRY gene saying grow faster, and you only need that to remain functional. Selection at the level of fertile or infertile men will weed out the ones that have got a non-functional SRY gene. It’s not as if you’ve got a patchwork of mutations. You can afford to degenerate your Y chromosome down to almost nothing and you’ll still be functional.

Dwarkesh Patel 00:55:26

It’s quite interesting because you were saying that the same thing happened to the mitochondrial DNA, which is a tiny genome, and has shrunk over time, starting from the original bacteria that was engulfed.

Nick Lane 00:55:36

It’s gone down from, say, 3,000 or 4,000 genes to, in our own case, 37 genes. You cannot sustain a large genome if you’re inside. As I said, population size matters. If you were a free-living bacterium living out there in the wild with a population of a million, and now you shelter inside another cell and it’s a small cell, now you’ve got a population of five. You will accumulate mutations and you can’t resist them, so you’ll lose genes. So your genome shrinks. That’s what happened to the mitochondria. You just can’t maintain a bacterial-sized genome.

Dwarkesh Patel 00:56:07

It might be worth explaining why it is the case that sex is preferable to lateral gene transfer in the sense of systematic pooling and parallel search across gene space. If there is this advantage of sex and bacteria have some antecedent to it, why didn’t they just get the whole thing? Is it just that it’s not compatible with their size?

Nick Lane 00:56:29

I think they had no need for it. What they do is lateral gene transfer. Basically you pick up random bits of DNA from the environment. It can be a bit more sinister than that. You can kill the cell next to you and take its DNA and load that in. That does happen, but for the most part, you pick up bits of DNA from the environment. It’s usually small pieces, usually one gene’s worth or something. You’d only do that if you’re a bit stressed. If things aren’t going well for you, you will then pick up bits of DNA, bind it into your genome and hope for the best. For most critters, most of the time it’s not going to work, but for one of them it does, and they will take over. It speeds up adaptation to a changing environment.

Why are they only using one gene? There’s two ways of seeing this. You’ve got a bacterial-sized genome, it’s pretty small. You’re going to replicate faster if you keep that genome small. It’s kind of a disadvantage to have a big, unwieldy genome. Eukaryotes have that. It’s an interesting question. Why would you have such a big, unwieldy genome that takes longer to copy? Bacteria are really streamlined. They get rid of genes they don’t need and then they can grow faster. But now the conditions change and now you need this gene. So what do you do? You pick it up. You just pick up random genes and hope for the best, pick up the right one and off you go again.

Bacterial genome sizes are small. They’ve got what you’d say is a small genome, but then a large pan-genome, which is all of the genes they have access to. So an E. coli cell might have 3,000 to 4,000 genes in a single cell, but access to 30,000 to 40,000 genes.

Dwarkesh Patel 00:58:02

What is keeping the metagenome around? Why doesn’t everybody just converge to this streamlined thing that is needed for the current context?

Nick Lane 00:58:10

What keeps the metagenome around is the fact that different strains of E. coli, or whatever bacteria they may be, are living in different environments. You could have commensal bacteria living in your gut. You could have bacteria E. coli living on your skin, a very different environment. You can then have non-commensal pathogenic E. coli which are behaving differently again. They can differ in 50% of their genome.

You’ve got all of these things going on side by side and they can all borrow genes from each other. This is within the same species, whatever species exactly means with bacteria, it doesn’t quite have a meaning. This is the dynamic of bacterial evolution. They retain small genomes with access to large pan-genomes, and they’re forever borrowing, matching and so on. They effectively remain competitive by keeping their own genome pretty small.

Eukaryotes threw all of that out and got larger genomes. Then the question is, if you try to do that with a large genome, a eukaryotic-sized genome and then you go on picking up little bits of DNA from the environment—the chances of you replacing the right gene gets lower.

It just becomes less and less efficient the bigger your genome is. By the time you get to eukaryotes, they have a large genome. Why do they have a large genome? I would say it’s because you acquired this endosymbiont, they become the mitochondria. Now you have a lot more energy available. There’s all kinds of reasons why eukaryotes will tolerate a larger genome. But the bottom line is you’ve got the energy to do something with it which bacteria never really had.

Now lateral gene transfer is just not good enough to maintain this larger genome. You’re going to have to do something more systematic. So you pull on an entire genome, you line everything up, you cross over between them. Now it’s systematic, it’s reciprocal, and you can maintain the quality of genes in a much larger genome. Bacteria never had the need to do that.

Dwarkesh Patel 00:59:57

As I was reading your book, just to ease my own ignorance, I was trying to come up with an analogy. Please let me know in which ways it’s naive. Also thanks for tolerating all my other naive questions today. Here in Silicon Valley, maybe an analogy that will work for us is to think about a GitHub repository.

Nick Lane 01:00:15

I’m already out of my depth now.

Dwarkesh Patel 01:00:19

Basically you have this code base, and you have ways in which you do version control. The usual way this is done, and this may be analogous to sexual recombination, is that somebody makes what is called a new branch. In that branch, they might make changes which are organized next to the function that they’re trying to change. When the maintainer is looking at the code, they can see what the original code was at this point. Here’s the modification to that point of code, and you see the diff, and then you can merge it back if it seems sensible. The analogy here might be sexual recombination that’s organized along the relevant gene. You see this allele, you see that allele. Evolution here is a maintainer which is then driving one of them to fixation.

The analogy for asexual reproduction, cloning with mutation, would be one where you fork the repository, then you make a random change. You just change some random variable, you change a word, you change a bit. Almost every single time this will be deleterious. And even when it’s not deleterious, there’s no merge functionality. You’ve got millions of repositories that are then spawning millions of other repositories. Even if some improvement has been made on one of them, there’s no systematic way in which the improvements can be merged together.

Nick Lane 01:01:44

It sounds quite similar, yes.

Dwarkesh Patel 01:01:45

Finally lateral gene transfer. Here the analogy might be, you’ve got one repository for editing web pages and another repository for controlling airline software. What you just do is you take a random 500-line sequence in this web page editing software and you just put it in a random point in the airplane management software. There’s no systematic organization of, “Here’s where the relevant functionality is.”

Nick Lane 01:02:18

There is a bit, which is to say with lateral gene transfer, you would normally match the ends to something you’ve got already. I don’t know enough about coding to give a comparable example, but effectively you would be picking up a module which had some resemblance in terms of, “Okay, it fits into this part of the code.” So you’d only put that in. It may or may not be useful there, but it’s not just completely random. It’s plugged into a place where you know you have something like that that used to be there or could be there. So it’s not just random, but you don’t know what you put in.

Dwarkesh Patel 01:02:55

So then I don’t really have a good intuition for why lateral gene transfer does not produce similar benefits to recombination.

Nick Lane 01:03:05

It’s really just a scaling thing. If you pick up a random piece of DNA and you’ve got a genome which is 10 times larger, how fast can you pick up DNA from the environment? You’d have to pick up 10 times as much to do that. Do you have the capacity to pick up 10 times as much?

There’s also a penalty for doing it, which is to say, like a mutation, you’ve got no idea what you’re plugging in. It could be almost anything. You know where you’re plugging it, you’re plugging it in the right place, but what’s in that cassette, you don’t really know. So the more you do it, the more you will degenerate yourself as well. There are costs and benefits to doing it.

Dwarkesh Patel 01:04:53

Maybe to close this off,  what is the experiment or method of interrogation, which would give us the most amount of information about the this story?

Nick Lane 01:05:08

There are so many aspects of this story, so many possible answers I could give there. In terms of eukaryotes, giant bacteria, the likelihood of life, a lot depends on observation. We simply don’t know enough about what’s out there. So it’s not necessarily experimentation. If I assert that giant bacteria are always going to have extreme polyploidy with multiple copies of their genome, and you find an example that’s not like that, my ideas are already breaking up. So that’s useful to know.

For the origin of life, I really wish I could come up with a convincing reason why I should go down in a submersible to a deep-sea hydrothermal system like Lost City. I would love to go to Lost City. But the trouble is that the ocean chemistry is completely different now to what it was 4 billion years ago. It’s now full of oxygen. It’s full of bacteria and things as well. But the ocean chemistry is different because there’s oxygen. There’s no iron, there’s no nickel in the oceans. You can go to a vent like Lost City and the walls are not made of catalytic minerals anymore. They’re made of aragonite and brucite, so calcium carbonate and magnesium hydroxides and things like that. So the chemistry it can do is very different, and there’s lots of bacteria living there. I would gain, beyond just the sheer amazement of seeing it, there’s not a lot it would be able to tell me.

What we’re actually doing is experiments in a lab in an anaerobic glove box where you exclude the oxygen. So you can do these experiments reacting hydrogen and CO2. How many of the molecules in biochemistry can we produce that way? It’s slow and laborious, and you get small amounts and sometimes you get contaminations. Sometimes you have to start all over again. It’s slow work, but it’s moving forward.

It’s not just us, either. There are other groups around the world. Joseph Moran’s group, for example, has done a lot of really nice biochemistry along these lines. That’s moving forward, but we’re talking decades before we’re getting to the level where we can say, “Right, we can drive flux through all of metabolism, and here’s the set of conditions that will do it.” Certainly some years.

There are big crux points, like making purine nucleotides where there are 12 steps in this synthetic pathway, and all the intermediates are unstable and break down easily. It has been done in things like methanol, so not in water. In water, stuff breaks down. We’re trying to do it. It’s difficult. I believe we’ll get there, which is why we’re trying to do it, but maybe we won’t, in which case, again, the hypothesis is wrong. You’ve got to wake up every morning and think the hypothesis could be wrong. It’s beautiful, it makes sense, but there are so many beautiful ideas killed by ugly facts. There’s no good believing that you’re right. You’ve got to believe you’re probably wrong and keep going anyway.

01:08:12 – Are bioelectric fields linked to consciousness?

Nick Lane 01:08:12

The other thing which I’m excited about at the moment is work on anesthetics and mitochondria, it turns out - I heard this from a guy called Luca Turin a few years ago now—who pointed out to me that anesthetics affect mitochondria. I had no idea that anesthetics affect mitochondria. They do. We’ve been doing experiments on it, and it seems not fully established yet, but it does seem as if their main effect is mitochondria. Anesthetics work on all kinds of things, including things like amoeba. It doesn’t prove anything but it’s beginning to say, if you can make an amoeba unconscious, then was it conscious before?

Not as we understand consciousness. The way we would understand consciousness is really about neural nets, a nervous system, and all the complexity of human consciousness. That’s what we primarily think about. But there’s a deep problem which goes back. It’s the mind-body problem, but it was framed by David Chalmers as the hard problem of consciousness, which boils down, as my understanding of this is, to more or less that we don’t know what a feeling is in physical terms. You can understand the information processing of a neural network. But if you feel miserable or you feel pain or you feel love or whatever it may be, what actually is that in the chemistry of the system?

The problem is that you have all of these neural nets firing and some of them are conscious. We’re aware of what we’re thinking about. Others, which seem to have all the same properties in terms of the neurons—they have synapses, they have neurotransmitters, they depolarize, they pass on an action potential—but we’re not conscious of it. It’s non-conscious information processing. So there’s this question. If anesthetics affect things that don’t have neural nets, and feelings are something that we can’t define in terms of a neural net, could it be that feelings are somehow linked more broadly to life?

So why would they be? The way I think about this is as an evolutionary biologist. The first question is, would we think that the feelings are real? I would say yes. Do we think that they evolved? I would say yes. I think any evolutionary biologist would say yes to those questions. If it’s real and it evolved, then natural selection must be able to see it and act on it in some way. In other words, there’s something physical about it that can be selected for. I don’t think there’s anything controversial about that statement. But if it’s physical and real and has been selected on, the implication is we should be able to measure it. It has to offer an advantage for selection to act on, and if it’s a physical process, it should be measurable. But we don’t really know what we’re trying to measure here.

I then revert back to thinking, what would a bacterial cell need to do? This is just back-of-the-envelope thinking. I immediately think about metabolism. What’s the difference between the inside of a bacterial cell and the outside world? The inside is metabolically alive. It’s doing stuff with its chemistry all the time, and it’s at a colossal rate. A bacterial cell will have about a billion reactions every second in this metabolism. I’m immediately left wondering, how is it all controlled? How do you get this cell to have a coherent behavior so it decides, “I’m going to crawl over there”? How do you even know what state you’re in? How do you synchronize all of this biochemistry?

Probably most people’s answer to that would be metabolic regulation of one sort or another. But that’s not really the driver. The driver in the end is the thermodynamic drivers. How many electrons do you have? That’s in the form of food or NADH or whatever it may be. How much energy do you have in the form of ATP? These are the things that are going to synchronize reactions in the same phase. The problem there is when you’re dealing with molecules, you’re dealing with tens of thousands of them, so you’ve got a large statistical sampling which is time-consuming to figure out.

But there is a better way of doing it, which is to say, if you’re taking electrons from food in NADH and you’re passing them to oxygen, but you’re generating a membrane potential and that’s driving ATP synthesis, you can measure the rate of change and the membrane potential and the fields that would be generated, electrostatic and electromagnetic fields. That’s going to give you a handle on your state, on your metabolic state in relation to the outside world. Is there enough food there? Is there enough oxygen there? Is it too hot? Is there a virus? Do I have enough iron to be able to do all these reactions? You’ve got all these potentially conflicting feedback loops, and you’ve got to make a decision.

Just thinking loosely about how a bacterial cell is going to behave, you find that you’re already framing it in terms of, as an entity, as a cell, it’s got to make some decision about what to do. It’s got to integrate all this information and make a coherent decision as a self, as an entity. Is that free will? Probably not in any way that we recognize it, but it makes a decision in relation to its environment, and the outcome is survival or not.

What I think a feeling is then is effectively the electromagnetic fields generated by membrane potential, which is telling you what your physical metabolic state is in relation to the environment you’re in.

That leads me to a question. If consciousness is somehow about mitochondria, are the mitochondria in that sense just really simply an ATP-generating engine, and you interfere with the way they make ATP and so anesthetics work by effectively giving you an energy deficit so the brain closes down? That would be dull if it were true, but it would be useful to know if it were true.

Much more exciting would be, do mitochondria generate the kind of fields that I was talking about in bacteria that are giving some indication of your status in certain mitochondria, certain neurons, and the anesthetics interfere with that? That would be magical if that were true. That would be a whole new direction of research, which would be fantastic. It’s very difficult to measure fields. It’s very easy to measure artifacts that you don’t know what you’re really doing. We need more physicists working in this area to do the hard calculations, and we need more data. Is it really just in one of these respiratory complexes, complex I? So there’s lots of standard molecular biology that we can do. It’s beginning to point to this idea that yes, there’s something going on about the way that complex I works which may link to generating fields that may link to how anesthetics work.

That’s just fun. The thing that’s great about science is it’s really fun. It’s one thing I’m always trying to get across to the people in my lab. You can’t forget the fun. If it becomes drudgery, then you best go because you’ll make much more money somewhere else. You’ll have a better life somewhere else. But if what you really care about is the science and the experiments, it’s got to be fun. You’ve got to really enjoy wanting to go and do that. I have to say, one of the great things for me is it’s always been fun.

Dwarkesh Patel 01:15:49

It’s been great to vicariously get a sense of that feeling from reading your books.

Nick Lane 01:15:55

Thank you.

Dwarkesh Patel 01:15:56

For the audience, this conversation has been most coupled with Nick’s book, The Vital Question. I would recommend getting that if you want to better follow the argument here. There’s way more detail there that would be helpful.

One, this is a thing I was telling you earlier, it fills a niche of books which, unfortunately, there are just very few of. There are textbooks where you can spend 2,000 pages learning about molecular biology. But a layperson who’s curious is just practically not going to get a chance to do that. On the other end, there are what are basically just anecdotes about scientists or anecdotes about the history of science. This one discoverer was really mercurial, and here’s how he ran his lab, and here’s how his parents were like. But it never really talks about the actual relevant science. A book like this actually does fill the explanatory middle.

Nick Lane 01:16:54

Thank you. Physicists are very good at writing books about the big questions of the universe. There’s a large readership for having your mind blown by a book that you’re not going to understand everything because you know it’s difficult.

How do we know anything at all about the Big Bang or how black holes work or background radiation or whatever it may be? With life, the origin of life or the trajectory of life on a planet, and whether we get complex life inevitably or whether we’re going to get stuck with bacteria in most places, these are big universe-sized questions. There’s not many people writing about them and trying to take you to the edge of what we know in the way that the physicists very often do and just saying, “Well, here’s how I see it. Here are the questions through my eyes.” You’ve got to try and be honest and say, “Okay, I see it this way, other people see it differently.”

Dwarkesh Patel 01:17:51

By the way, the fact that LLMs exist has made the process of reading a book like this much more feasible and productive. I had a book club with a couple of my friends. We’re not biologists. We’re laypeople to this audience. I do encourage people for a book like this to see if you can form a book club or something and just talk to LLMs a bunch because there’s just a bunch of extremely basic remedial chemistry and biology that we were able to recapitulate with the help of the LLMs. This whole thing of, “Why is the CO2 and H2 reaction incentivized when one side is alkaline and one side is acidic in this early environment?” You just go through the remedial chemistry with the LLM.

Nick Lane 01:18:37

Yes, I did my best to explain it in the book, and it seems that I didn’t do a great job of it. There’s so much detail, and you can’t avoid that because it’s there in the questions. This is a problem with biology, it’s incredibly complex. Physicists look at biology and they think it’s just too hard to explain, and biologists have got all of this terminology and often get lost in the terminology.

I find myself, by nature, trying to find simple common denominators. That lends itself then to writing about them. I probably oversimplify all the time, or maybe I fail and don’t simplify it enough. But you wrestle with it and you try and make it work. It’s genuinely interesting for me to talk to you and the other guys in the book club to see where you were struggling with it. I will build this into the next time I’m writing a book and try to figure out how I do that better.

Dwarkesh Patel 01:19:32

Nick, this has been great. Thank you for the guide through both the remedial biology and chemistry, but also through many of the most interesting questions that you could ask about life.

Nick Lane 01:19:43

Been great fun. Thanks a lot.

Boy do you guys have a lot of thoughts about the Sutton interview.

I’ve been thinking about it myself. I have a much better understanding of Sutton’s perspective now than I did during the interview itself. So I want to reflect on it a bit.

Richard, apologies for any errors or misunderstandings. It’s been very productive to learn from your thoughts.

The steelman

What is the bitter lesson about? It is not saying that you just want to throw as much compute away as possible. The bitter lesson says that you want to come up with techniques which most effectively and scalably leverage compute.

Most of the compute spent on an LLM is used on running it in deployment. And yet it’s not learning anything during this time! It’s only learning during this special phase we call training. That is not an effective use of compute. And even the training period by itself is highly inefficient - these models are usually trained on the equivalent of 10s of 1000s of years of human experience.

What’s more, during this training phase, all their learning comes straight from human data. This is an obvious point in the case of pretraining data. But it’s even kind of true for the RLVR we do on LLMs: these RL environments are human furnished playgrounds to teach LLMs the specific skills we have prescribed for them.

The agent is in no substantial way learning from organic and self-directed engagement with the world. Having to learn only from human data (an inelastic hard-to-scale resource) is not a scalable use of compute.

What these LLMs learn from training is not a true world model (which tells you how the environment changes in response to different actions). Rather, they are building a model of what a human would say next. And this leads them to rely on human-derived concepts. If you trained an LLM on the data from 1900, it wouldn’t be able to come up with relativity from scratch.

LLMs aren’t capable of learning on-the-job, so we’ll need some new architecture to enable continual learning. And once we have it, we won’t need a special training phase — the agent will just learn on-the-fly, like all humans, and indeed, like all animals. This new paradigm will render our current approach with LLMs obsolete.

TLDR of my current thoughts

My main difference with Rich is that I think the concepts he’s using to distinguish LLMs from true intelligence are not actually mutually exclusive and dichotomous.

Imitation learning is continuous with and complementary to RL. Models of humans can give you a prior which facilitates learning “true” world models. I also wouldn’t be surprised if some future version of test-time fine-tuning could replicate continual learning, just the same way that in context learning does so now.

Imitation learning is continuous with and complementary to RL

I tried to ask Richard a couple of times whether pretrained LLMs can serve as a good prior on which to accumulate the experiential learning (aka do the RL) which will lead to AGI.

In a talk a few months ago, Ilya compared pretraining data to fossil fuels. This analogy has remarkable reach. Just because fossil fuels are not renewable does not mean that our civilization ended up on a dead-end track by using them. You simply couldn’t have transitioned from the water wheels in 1800 to solar panels and fusion power plants. We had to use this cheap, convenient, plentiful intermediary.

AlphaGo (which was conditioned on human games) and AlphaZero (which was bootstrapped from scratch) were both superhuman Go players. AlphaZero was better.

Will we (or the first AGIs) eventually come up with a general learning technique that requires no initialization of knowledge - that just bootstraps itself from the very start? And will it outperform the very best AIs that have been trained to that date? Probably yes.

But does this mean that imitation learning must not play any role whatsoever in developing the first AGI, or even the first ASI? No. AlphaGo was still superhuman, despite being initially shepherded by human player data. The human data isn’t necessarily actively detrimental - at enough scale it just isn’t significantly helpful.

The accumulation of knowledge over tens of thousands of years has clearly been essential to humanity’s success. In any field of knowledge, thousands (and likely millions) of previous people were involved in building up our understanding and passing it on to the next generation. We didn’t invent the language we speak, nor the legal system we use, nor even most of the knowledge relevant to the technologies in our phones. This process is more analogous to imitation learning than to RL from scratch.

Are we literally predicting the next token (like an LLM) in order to do this cultural learning? No, and so even imitation learning for humans is not like supervised learning for AI. But neither are we running around trying to collect some well defined scalar reward. No ML learning regime perfectly describes human learning. We do things which are analogous to both RL and supervised learning. As planes are to birds, supervised learning might be to human cultural learning.

I also don’t think these learning techniques are categorically different. Imitation learning is just short horizon RL. The episode is a token long. The LLM makes a conjecture about the next token based on its understanding of the world and the other information in the sequence. And it receives reward in proportion to how well it predicted the true token.

Now, I already hear people saying: “No no, that’s not ground truth! It’s just learning what a human was likely to say.”

Agreed. But there’s a different question which I think is more relevant to the scalability of these models: can we leverage imitation learning to help models learn better from ground truth?

And I think the answer is, obviously yes? We have RLed models to win Gold in IMO and code up entire working applications from scratch. These are “ground truth” examinations. But you couldn’t RL a model to accomplish these feats from scratch. You needed a reasonable prior over human data in order to kick start the RL process.

Whether you wanna call this prior a proper “world model”, or just a model of humans, seems like a semantic debate to be honest. Because what you really care about is whether this model of humans helps you start learning from ground truth (aka become a “true” world model).

It’s a bit like saying to someone pasteurizing milk, “Hey stop boiling that milk - we eventually want to serve it cold!” Yes, of course. But this is an intermediate step to facilitate the final output.

By the way, LLMs have clearly developed a deep representation of the world (because their training process incentivizes them to). I use LLMs to teach me about everything from biology to AI to history, and they do so with remarkable flexibility and coherence. Are LLMs specifically trained to model how their actions will affect the world? No.

But if we’re not allowed to call their representations a “world model”, then we’re defining the term “world model” by the process we think is necessary to build one, rather than by the obvious capabilities the concept implies.

Continual learning

Sorry to bring up my hobby horse again. I’m like a comedian who’s only come up with one good bit.

An LLM being RLed on outcome-based rewards learns O(1) bits per episode, and the episode may be tens of thousands of tokens long.

We animals clearly extract far more information from interacting with their environment than just the reward signal at the end of each episode. Conceptually, how should we think about what is happening with animals?

We’re learning to model the world through observations. The outer loop RL is incentivizing some other learning system to pick up maximum signal from the environment. In Richard’s OaK architecture, he calls this the transition model.

If we were trying to pigeonhole this feature spec into LLMs, what you’d do is to fine tune on all your observed tokens. From what I hear from my researcher friends, in practice the most naive way of doing this doesn’t work well.

Being able to continuously learn from the environment in a high throughput way is obviously necessary for true AGI. And it clearly doesn’t exist with LLMs trained on RLVR.

But there might be some relatively straightforward ways to shoehorn continual learning atop LLMs. For example, one could imagine making SFT a tool call for the model. So the outer loop RL is incentivizing the model to teach itself effectively using supervised learning, in order to solve problems that don’t fit in the context window.

I’m genuinely agnostic about how well techniques like this will work—I’m not an AI researcher. But I wouldn’t be surprised if they basically replicate continual learning. Models already demonstrate something resembling human continual learning within their context windows. The fact that in-context learning emerged spontaneously from the training incentive to process long sequences suggests that if information could flow across windows longer than the current context limit, models could meta-learn the same flexibility they already show in-context.

Concluding thoughts

Evolution does meta-RL to make an RL agent. That agent can selectively do imitation learning. With LLMs, we’re going the opposite way. We first made a base model that does pure imitation learning. Then we do RL on it to make it a coherent agent with goals and self-awareness.

Maybe this won’t work!

But I don’t think these super first-principle arguments (for example, about how an LLM doesn’t have a true world model) prove much. I also don’t think they’re strictly accurate for the models we have today, which undergo a lot of RL on “ground truth”.

Even if Sutton’s Platonic ideal doesn’t end up being the path to first AGI, he’s identifying genuine basic gaps which we don’t even notice because they are so pervasive in the current paradigm: lack of continual learning, abysmal sample efficiency, dependence on exhaustible human data. If the LLMs do get to AGI first, the successor systems they build will almost certainly be based on Richard’s vision.

I just interviewed Nick Lane yesterday. It turned out great. I’m planning on publishing the episode Friday.

In the meantime, I am sharing the notes I was taking as I try to comprehend his book, The Vital Question: Energy, Evolution, and the Origins of Complex Life.

The story the book tells of life’s evolution is both fascinating and super complicated. Hopefully there notes are helpful to others who are interested in this topic.

I wrote up these notes piecemeal, part by part, on Twitter, rather than all at once at the end. This blog compiles all of them in place.

Subscribe for future book notes like this, and for the episode when it comes out.

Part 1 - Why eukaryotes are so special

In the intro he lists out the motivating questions:

Why are bacteria so relatively simple despite being around for 4 billion years? Why is there so much shared structure between all eukaryotic cells despite the enormous morphological variety between animals, plants, fungi, and protists? Why did the endosymbiosis event that led to eukaryotes happen only once, and in the particular way that it did? And why is all life powered by proton gradients?

Nick says all these questions are connected.

Lane says there’s 2 different philosophies on what bottlenecks evolutionary exploration: the niches made available by the environment, OR the internal structure necessary to exploit those niches.

Textbook view is that the environment constrains exploration, whereas structure is flexible and can accommodate once the right environment is in place. Nick Lane thinks it’s the opposite.

There’s been 2 big oxidation events - the first one (2.4 billion years ago) paved the way for eukaryotic cells. The second one (600 million years ago) led to the Cambrian explosion, resulting in all the variety in animals and plants and other complex life we see. So it seems the environment is central. Once you get a bunch of oxygen up in the air and into the oceans, you can start making all kinds of cool shit.

But hold on. Here’s what you’d expect to see if the environment was the key constraint: With this key unlock of aerobic respiration, different brands of bacteria independently evolve towards greater complexity to fill the new niches opened up (one masters osmotrophy and branches off into fungi, another photosynthesis, another phagocytosis, etc). However, you don’t see this.

Instead you see that all complex life emerges from a single common eukaryotic ancestor (2.2 billion years ago). There is no independent convergent evolution towards this kind of complexity (bacteria have had 4 billion years to evolve this kind of complexity, and have stayed remarkably similar through the whole time).

In fact, once you do get this key structural unlock, eukaryotic organisms proliferate widely, filling niches ranging from 100 feet long blue whales to 0.8 meter long picoplankton.

What’s more:

• The amount of shared structure between all eukaryotic cells is remarkable. They have almost all the same organelles and components. Nick writes:

  “Most of us couldn’t distinguish between a plant cell, a kidney cell and a protist from the local pond down the electron microscope.”

• There’s no intermediate proto-eukaryotes, which have some, but not all, of the functionality available to eukaryotic cells. This is wild given how evolution works. We have an extensive record of the incremental upgrades between photoreceptive amoebas and mammalian eyes. Why don’t we have proto-eukaryotic cells which reproduce via meiosis but don’t have compartmentalized nucleuses, or have mitochondria but no cytoskeleton?

Nick argues that the fact that no such subset of eukaryotic traits exists suggests that it is not structurally possible to survive with only some fraction of eukaryotic equipment - you need the whole package all at once.

Obviously this raised the question of how the whole package was evolved at once. Which I think he will address in future chapters.

Some questions for Nick:

• If his view is that structure was the main bottleneck, and we’ve had eukaryotes for 2.2 billion years, then why didn’t we have all these animals and shit for 2 billion years? Why did they only arise 600 million years ago (aka the Cambrian explosion)?

• Nick argues that eukaryotic cells are a much more significant unlock than multi-cellularity. Multi-cellularity evolved independently dozens of times, but we only have evidence of one event like the emergence of the first eukaryotic cell. If multi-cellularity evolved independently so many times (between fungi, slime molds, algae, etc etc), do we see interesting differences based on the situations in which they evolved? Do they regulate the differentiation of cells, the organization of the body differently, and communication between tissues differently? TODO look it up later.

A tangential thought. This whole debate about whether structure or environment matters more seems analogous to the discussion in ML of whether architecture or data matters more. And there it seems like data is quite crucial, but for meta-learning and generality to kick off, the architecture has to make it possible for information to flow in the right way. For example, in context learning is a kind of meta-learning that arises only once the model has the capability to attend to hundreds of previous tokens, which became tractable with transformers.

Part 2 - How the first cells evolved

His main argument here is that life is continuous with the planet’s geochemistry.

Aka a lot of the main characteristics of cells - membranes, enzymes, energy via proton gradients - descend from spontaneous processes in the Earth.

But you can’t have these characteristics evolve piecemeal in different locations. You need one location that houses all the processes which could then give rise to the first cell.

Important context, by the way, is that all life descends from a single common ancestor - LUCA (last universal common ancestor).

Okay, so what candidate environment could give rise to LUCA? It needs two main characteristics:

1. There’s a continuous flux of carbon and energy (in some sense, all life is a flux of carbon and energy, but you need some geochemistry to maintain this disequilibrium before the first cells can co-opt it).

2. Something which concentrates and catalyzes the reactions which lead to organics (aka inorganic equivalents of cells and enzymes).

This rules out a lot of old theories: a warm pond with ammonia and salts and the odd lightning bolt doesn’t drive continuous flux, nor concentrate early organics in a cell-like volume to drive forward reactions.

Nick thinks alkaline sea vents are a unique fit to this challenge, and also help explain a lot of the contingent biochemistry that all life ended up using because of our shared inheritance.

Okay, let’s dig in: and for context, basically Nick here is trying to explain how you end up with an early version of the reverse Krebs cycle spontaneously. Reverse Krebs cycle takes in H2 and CO2 and makes organic molecules that are the precursors of fatty acids, proteins, and sugars.

Another important bit of context: All life runs on proton gradients. Burning food with oxygen (or other oxidants in anaerobic respiration) pumps H+ ions across a membrane, like filling a dam. These ions flow back through ATP synthase—a molecular turbine—which harnesses the flow to attach phosphate to ADP, creating ATP. Your body contains just 60 grams of ATP, but the ATP→ADP→ATP cycle is so rapid you process your body weight in ATP daily.

Sidenote: If a solution is acidic, it means there’s a lot of H+ ions in it. And if it’s basic (aka alkaline), it means there’s a lot of OH- ions in it.

Okay so what was happening in these alkaline hydrothermal vents? There’s 3 sides to this picture: the inside of the vent, the vent wall, and the ocean side of the vent.

On the inside of the vent, you’ve got iron rich rock basically rusting, which lets out H2 and OH- into the stream of water piping through (aka making the water basic/alkaline).

The wall is made up of catalytic minerals like FeS, and also has a ton of tiny pores which connect the inside to the outside.

And the ocean side has a bunch of dissolved CO2 - early Earth was basically a giant ocean, but also had a lot of volcanoes that let out lots of CO2. And the oceans are quite acidic too, because CO2 becomes carbonic acid when dissolved in water.

Within the tiny pores inside these vents, you have H2 reacting with CO2 to form simple organics like formaldehyde (CH2O) and methanol (CH3OH), instigated by the FeS in the walls, which acts as a catalyst for this reaction.

Remedial chemistry: feel free to skip this para - I’m just going to include it since it took me some effort to relearn the high school chemistry involved. And it was quite satisfying to understand. Why do you need the H2 side inside to be basic? And why do you need the CO2 side outside to be acidic? My understanding is that in an alkaline solution, H2 -> H+ is favored, since the OH- (which definitionally makes the solution alkaline) really wants to react with H+ to make H2O. But now you’ve got some intermediate H+ lying around to be involved in other reactions. On the ocean side, the more acidic the water, the less likely that the marginal CO2 added will be turned into carbonic acid (since there’s so much of it around already) and will instead be available to react with.

Now that you’ve got these early organics building up inside these tiny pores, you can kick off this positive feedback loop where these early organics act as precursors or enzymes to make more and more of the molecules life uses. You build amino acids (which become enzymes for other reactions), and fatty acids (which spontaneously form membranes because they have hydrophobic heads and hydrophilic tails), and sugars, and peptides, and eventually DNA and RNA. Claude illustrates:

The fact that this early proto cell doesn’t have to generate proton gradients itself, and can just take advantage of the geochemical disequilibrium, is a huge boon: “Methanogens spend practically 98% of their energy budget on generating proton gradients by methanogenesis, and little more than 2% producing new organic matter. With natural proton gradients and leaky membranes, none of that excessive energy spend is needed. The power available is exactly the same but the overheads are cut by at least 40-fold, a very substantial advantage.”

In addition to the H+ gradient, which exists spontaneously in these vents, some protocells also started to extrude Na+ ions. And since there’s no natural gradient for these, this creates an incentive for developing non-porous membranes (and for proteins on that membrane to pump protons out). Once you develop such a membrane, you can exit this wall cavity and float around like a real cell.

Is the implication that inheritance only got kicked off at this point? Because beforehand, I guess you have selection amongst the pores, but you have no way to pass down traits. This buildup of organics and metabolism is happening independently across all the pores.

Yet you already had DNA and RNA by this point. So what was this genetic information doing before inheritance? I guess just organizing information to facilitate buildup of more organics?

Does this imply that there were millions of protocells with no shared lineage between them, each developing their own unique versions of all the basic biochemistry of life? LUCA just happened to be one that had DNA, RNA, and ATP synthase, but all 3 of those could have been wildly different based on which proto cells made it out of the nook first?

Yet the fact that these three building blocks are considered across all life suggests that they are uniquely well-engineered? Or maybe it means that evolution can’t effectively improve upon its foundations. The same way that backprop can find the best network to map a function, but can’t rewire the GPU you’re training it on at the same time. Anyways, once you have this proto cell, it can ‘infect’ contiguous vent systems all across the ocean floor.

Contingent biochemistry explained by this theory:

• Why all life is powered by proton gradients

• Why all carbon fixation pathways, whether they’re in bacteria, archaea, or eukaryotes, use acetyl-CoA as the entry point. It forms spontaneously at these vents when catalyzed by the FeS in the walls. And basically all life still uses this molecule to store energy and build other molecules.

• Why a lot of the enzymes involved in energy metabolism (and the Krebs cycle specifically) still use FeS minerals as their backbone

• Why Archaea and Bacteria (the two different kingdoms of eukaryotes) split up - apparently it has something to do with how they create proton gradients, but honestly the relevant biochemistry went over my head. Though this bifurcation is supposed to explain why all life shares DNA, RNA, and ATP synthase, but nothing else: not the cell membrane, nor the DNA replication enzymes, nor the pumps for excretion. Apparently all of these things were implicated in the different choice that archaea and bacteria made during this bifurcating event.

Questions for Nick:

• I guess this theory is incompatible with panspermia, right?

• Does this Alkaline vents theory suggest that life might be very rare or very abundant in the universe? In some sense, it suggests it should be rare. It’s just a very specific type of hydrothermal vent with the right pH gradient and pore size and durability. But in another sense, it’s just a random fucking vent. There could theoretically be thousands of similar geological structures across the universe that could also drive the flux of carbon and energy across tiny membranes.

• Isn’t ATP synthase super complicated? How did the first protocells have ATP synthase but almost nothing else nearly as complex?

• How did all this complexity build up before evolution with heredity? All these pores are just independently building up their own microcosm of unique organics? I guess it’s possible that these early building blocks are floating from hole to hole without a fully formed membrane? DNA plus enzymes float from one pore to another, and kick off more reactions? Does Nick Lane think this is likely? If not, does it suggest that there were many other equally viable alternatives for the building blocks once LUCA was able to break out?

Part 3 - Why bacteria can’t become complex

Why are bacteria relatively simple, whereas eukaryotes gave rise to all the wonderful complexity we see around us?

Eukaryotes are typically 1000x bigger in volume and genome size. And of course gave rise to internal compartmentalization, multicellularity, sex, and much else

Here’s a subtly wrong theory: it’s all about surface area to volume ratios. Eukaryotes generate energy in mitochondria (whose quantity scales with cell volume). Prokaryotes generate energy along the cell membrane surface (since they don’t have an internal organelle like the mitochondria to generate and store the proton gradients which power life). Surface area (aka bacteria’s energy production) scales quadratically with radius, whereas volume (aka energy consumption) scales cubicly. Ergo, bacteria can’t become as big, and therefore, can’t spawn lots of complexity.

But we know it’s totally possible for membranes to be folded up in all sorts of weird ways to increase surface area/volume ratio. And we know that bacteria can create vacuoles inside (where they could presumably store a proton gradient). Why didn’t bacteria make use of these tricks to scale up the ladder of complexity?

Nick Lane explains that the key advantage eukaryotes have is that the mitochondrial genome is distinct from the bacterial genome (due of course to the endosymbiotic event which engulfed the bacterial ancestor of the mitochondria).

For some reason that I don’t fully understand, there needs to be super-local control of the redox reactions in the electron transport chain which drive respiration. You need the relevant genes on site. Mitochondria already have their own internal genomes and ribosomes to regulate their work.

If a bacterial cell were to become much bigger, it would need to store copies of the relevant genes close to the membrane. But bacteria don’t have a way to make specific piece-meal cuts to the genome. So they would need to copy their entire genome across the entire membrane many, many times over. And also store many copies of ribosomes and other infrastructure. This is simply impractical.

Nick also explains that over time, most of the original mitochondrial genes drifted to the nucleus because it’s more efficient to keep a single copy there. And only the ones that were absolutely necessary locally are kept in the mitochondria. The exact mechanism of this drift, and how it led to the evolution of the nuclear membrane and individual linear chromosomes, is best left to the book.

Questions for Nick Lane:

• Why are mitochondria the only organelle that needs to have its own genome right on site? Is it the case that other organelles would also benefit from local control but don’t have this unique endosymbiotic history which would plausibly have led to their own genomes? Or is it just that the Krebs cycle is so complex and fragile that you need to respond to perturbations right on site?

• Why haven’t there been more endosymbiotic events?

Part 4 - Sex

Why do eukaryotes have sex? And why 2 sexes in particular? Nick Lane thinks this again can be explained by (you guessed it) mitochondria.

First, why sex? Solves two problems:

1. Muller’s ratchet: since almost any random mutation will be deleterious, variation via mutation produces children with lower expected fitness. Whereas variation with recombination (which doesn’t just do random bit flips - rather it randomly samples alleles which are known to be plausible) produces children with the same expected fitness.

2. Clonal interference: even if a beneficial mutation is found, without systematic pooling of genes via recombination, the different lineages are just gonna have to battle it out. One lineage has beneficial mutation X, the other lineage is beneficial mutation Y, but there’s no way to fuse those improvements. You’re either going to have to lose one or the other as each lineage tries to win over the other.

Bacteria, of course, do have lateral gene transfer. But this is non-reciprocal and piecemeal. It doesn’t enable the same kind of genome-wide parallel search that recombination does.

Analogize this to a Github repo. Recombination is like a normal pull request - the diff is organized, made at the same site where the previous analogous functionality was, and then merged back into the main branch if maintainer evaluates it to be better (analogy is imperfect, but this is like evolution driving that allele into fixation after the systematic pooling that recombination enables).

Asexual reproduction is if you just forked the repo millions of times, making random char changes. And even if a couple of these forks end up accidentally producing an improvement, there’s no way for them to merge.

Horizontal gene transfer is if you just took a random 500 line snippet and shoved it in some totally different repo in a random place. There’s no organized diff at the site of the relevant functionality.

This kind of systematic parallel search across the genome became necessary once the genome size exploded after the endosymbiosis of the ancestral mitochondria (which kept shoving a bunch of its genes into the host cell’s DNA).

Okay fine. But why 2 sexes? Why not just 1, so that everyone could mate with everyone else? Or failing that, why not more than 2, so that you can mate with every sex but your own? 2 is the worst possible number in terms of the number of potential mates it makes available.

To explain why there can’t be one sex, we need to consider the mitochondrial DNA, which is separate from the nuclear DNA, and also doesn’t recombine. Because it doesn’t recombine, it suffers from Muller’s ratchet.

Because of how important it is that your mitochondria are not fucked up and compatible with each other, you need some pre-conception selection of mitochondria. How do you do that? The best way evolution has found to control mitochondrial quality is to have only one parent pass them down. This one parent should generate millions of oocyte candidates, and then filter them down to a couple hundred eggs based on mitochondrial quality. (How this selection happens is well beyond me).

So we need one sex that specializes in preserving mitochondrial quality, and another which is just there to provide the variance that sexual reproduction depends on.

And there’s no benefit to a third sex. There’s two useful niches: either you transmit mitochondria or you don’t. A third sex would just be redundant with one of the first two.

This then explains a ton of differences in males versus females. For example, human females start with ~6-7 million primordial germ cells during fetal development. But this drops down to a couple 100 viable eggs through their lifetime. I think Nick Lane’s theory is that partially what’s happened is that potential gametes with bad mitochondrial DNA have been purged.

Also, why are women born with all their eggs, whereas men produce sperm at will? Because women are tasked with protecting mitochondrial DNA, they want to minimize mutations. The way to minimize mutations is to keep cell duplications down. There’s only 20 mitotic divisions between a primordial germ cell and an egg. Whereas there would be hundreds of divisions between a spermatogenic stem cell and sperm.

Questions for Nick Lane:

• I feel like my explanation for why lateral gene transfer doesn’t produce the same benefits as sexual reproduction is kind of hand-wavy. I want to get a better understanding of what’s missing. For example, in his textbook on information theory, David McKay has an interesting tangential chapter on sexual vs. asexual reproduction. And there he proves that with recombination, you can acquire information from the environment √genome-size faster, and tolerate a mutation rate √genome-size higher. What would the analogous information-theoretic bound on lateral gene transfer be?

• If prokaryotes had evolved sex, is the implication of this logic that they would only have one sex? - Given the advantages of sexual reproduction (which are smaller, but still present, for the smaller genomes of bacteria) why didn’t bacteria evolve sex? Why do they just stick with horizontal gene transfer? Why did you need this endosymbiotic event to evolve sex?

• Do a lot of male abnormalities and diseases originate in the Y-chromosome? Because it also goes through Muller’s ratchet, right? It would explain why the Y chromosome has been shrinking over time. According to an LLM, it had 1400 genes 300 million years ago, now only 50-70 - presumably the ones that are absolutely essential to sexual bifurcation? Did evolution come up with some clever way to screen for Y chromosome quality the way it did with mitochondrial DNA? Did evolution try to get your important genes out of the Y-chromosome? Does this in any way connect to the greater male variability hypothesis, where supposedly men supply the world with more idiots and more geniuses? To the extent this is true, the cause would have to reside in the Y chromosome, right?

Subscribe for future book notes likes this, and for the episode when it comes out.

Richard Sutton is the father of reinforcement learning, winner of the 2024 Turing Award, and author of The Bitter Lesson. And he thinks LLMs are a dead end.

After interviewing him, my steel man of Richard’s position is this: LLMs aren’t capable of learning on-the-job, so no matter how much we scale, we’ll need some new architecture to enable continual learning.

And once we have it, we won’t need a special training phase — the agent will just learn on-the-fly, like all humans, and indeed, like all animals.

This new paradigm will render our current approach with LLMs obsolete.

In our interview, I did my best to represent the view that LLMs might function as the foundation on which experiential learning can happen… Some sparks flew.

A big thanks to the Alberta Machine Intelligence Institute for inviting me up to Edmonton and for letting me use their studio and equipment.

Enjoy!

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Labelbox makes it possible to train AI agents in hyperrealistic RL environments. With an experienced team of applied researchers and a massive network of subject-matter experts, Labelbox ensures your training reflects important, real-world nuance. Turn your demo projects into working systems at labelbox.com/dwarkesh

• Gemini Deep Research is designed for thorough exploration of hard topics. For this episode, it helped me trace reinforcement learning from early policy gradients up to current-day methods, combining clear explanations with curated examples. Try it out yourself at gemini.google.com

• Hudson River Trading doesn’t silo their teams. Instead, HRT researchers openly trade ideas and share strategy code in a mono-repo. This means you’re able to learn at incredible speed and your contributions have impact across the entire firm. Find open roles at hudsonrivertrading.com/dwarkesh

Timestamps

(00:00:00) – Are LLMs a dead end?

(00:13:04) – Do humans do imitation learning?

(00:23:10) – The Era of Experience

(00:33:39) – Current architectures generalize poorly out of distribution

(00:41:29) – Surprises in the AI field

(00:46:41) – Will The Bitter Lesson still apply after AGI?

(00:53:48) – Succession to AI

Transcript00:00:00 – Are LLMs a dead end?

Dwarkesh Patel 00:00:00

Today I’m chatting with Richard Sutton, who is one of the founding fathers of reinforcement learning and inventor of many of the main techniques used there, like TD learning and policy gradient methods. For that, he received this year’s Turing Award which, if you don’t know, is the Nobel Prize for computer science. Richard, congratulations.

Richard Sutton 00:00:17

Thank you, Dwarkesh.

Dwarkesh Patel 00:00:18

Thanks for coming on the podcast.

Richard Sutton 00:00:20

It’s my pleasure.

Dwarkesh Patel 00:00:21

First question. My audience and I are familiar with the LLM way of thinking about AI. Conceptually, what are we missing in terms of thinking about AI from the RL perspective?

Richard Sutton 00:00:33

It’s really quite a different point of view. It can easily get separated and lose the ability to talk to each other. Large language models have become such a big thing, generative AI in general a big thing. Our field is subject to bandwagons and fashions, so we lose track of the basic things. I consider reinforcement learning to be basic AI.

What is intelligence? The problem is to understand your world. Reinforcement learning is about understanding your world, whereas large language models are about mimicking people, doing what people say you should do. They’re not about figuring out what to do.

Dwarkesh Patel 00:01:19

You would think that to emulate the trillions of tokens in the corpus of Internet text, you would have to build a world model. In fact, these models do seem to have very robust world models. They’re the best world models we’ve made to date in AI, right? What do you think is missing?

Richard Sutton 00:01:38

I would disagree with most of the things you just said. To mimic what people say is not really to build a model of the world at all. You’re mimicking things that have a model of the world: people. I don’t want to approach the question in an adversarial way, but I would question the idea that they have a world model. A world model would enable you to predict what would happen. They have the ability to predict what a person would say. They don’t have the ability to predict what will happen.

What we want, to quote Alan Turing, is a machine that can learn from experience, where experience is the things that actually happen in your life. You do things, you see what happens, and that’s what you learn from. The large language models learn from something else. They learn from “here’s a situation, and here’s what a person did”. Implicitly, the suggestion is you should do what the person did.

Dwarkesh Patel 00:02:39

I guess maybe the crux, and I’m curious if you disagree with this, is that some people will say that imitation learning has given us a good prior, or given these models a good prior, of reasonable ways to approach problems. As we move towards the era of experience, as you call it, this prior is going to be the basis on which we teach these models from experience, because this gives them the opportunity to get answers right some of the time. Then on this, you can train them on experience. Do you agree with that perspective?

Richard Sutton 00:03:12

No. I agree that it’s the large language model perspective. I don’t think it’s a good perspective. To be a prior for something, there has to be a real thing. A prior bit of knowledge should be the basis for actual knowledge. What is actual knowledge? There’s no definition of actual knowledge in that large-language framework. What makes an action a good action to take?

You recognize the need for continual learning. If you need to learn continually, continually means learning during the normal interaction with the world. There must be some way during the normal interaction to tell what’s right. Is there any way to tell in the large language model setup what’s the right thing to say? You will say something and you will not get feedback about what the right thing to say is, because there’s no definition of what the right thing to say is. There’s no goal. If there’s no goal, then there’s one thing to say, another thing to say. There’s no right thing to say.

There’s no ground truth. You can’t have prior knowledge if you don’t have ground truth, because the prior knowledge is supposed to be a hint or an initial belief about what the truth is. There isn’t any truth. There’s no right thing to say. In reinforcement learning, there is a right thing to say, a right thing to do, because the right thing to do is the thing that gets you reward.

We have a definition of what’s the right thing to do, so we can have prior knowledge or knowledge provided by people about what the right thing to do is. Then we can check it to see, because we have a definition of what the actual right thing to do is.

An even simpler case is when you’re trying to make a model of the world. When you predict what will happen, you predict and then you see what happens. There’s ground truth. There’s no ground truth in large language models because you don’t have a prediction about what will happen next. If you say something in your conversation, the large language models have no prediction about what the person will say in response to that or what the response will be.

Dwarkesh Patel 00:05:29

I think they do. You can literally ask them, “What would you anticipate a user might say in response?” They’ll have a prediction.

Richard Sutton 00:05:37

No, they will respond to that question right. But they have no prediction in the substantive sense that they won’t be surprised by what happens. If something happens that isn’t what you might say they predicted, they will not change because an unexpected thing has happened. To learn that, they’d have to make an adjustment.

Dwarkesh Patel 00:05:56

I think a capability like this does exist in context. It’s interesting to watch a model do chain of thought. Suppose it’s trying to solve a math problem. It’ll say, “Okay, I’m going to approach this problem using this approach first.” It’ll write this out and be like, “Oh wait, I just realized this is the wrong conceptual way to approach the problem. I’m going to restart with another approach.”

That flexibility does exist in context, right? Do you have something else in mind or do you just think that you need to extend this capability across longer horizons?

Richard Sutton 00:06:28

I’m just saying they don’t have in any meaningful sense a prediction of what will happen next. They will not be surprised by what happens next. They’ll not make any changes if something happens, based on what happens.

Dwarkesh Patel 00:06:41

Isn’t that literally what next token prediction is? Prediction about what’s next and then updating on the surprise?

Richard Sutton 00:06:47

The next token is what they should say, what the actions should be. It’s not what the world will give them in response to what they do.

Let’s go back to their lack of a goal. For me, having a goal is the essence of intelligence. Something is intelligent if it can achieve goals. I like John McCarthy’s definition that intelligence is the computational part of the ability to achieve goals. You have to have goals or you’re just a behaving system. You’re not anything special, you’re not intelligent. You agree that large language models don’t have goals?

Dwarkesh Patel 00:07:25

No, they have a goal.

Richard Sutton 00:07:26

What’s the goal?

Dwarkesh Patel 00:07:27

Next token prediction.

Richard Sutton 00:07:29

That’s not a goal. It doesn’t change the world. Tokens come at you, and if you predict them, you don’t influence them.

Dwarkesh Patel 00:07:39

Oh yeah. It’s not a goal about the external world.

Richard Sutton 00:07:43

It’s not a goal. It’s not a substantive goal. You can’t look at a system and say it has a goal if it’s just sitting there predicting and being happy with itself that it’s predicting accurately.

Dwarkesh Patel 00:07:55

The bigger question I want to understand is why you don’t think doing RL on top of LLMs is a productive direction. We seem to be able to give these models the goal of solving difficult math problems. They are in many ways at the very peaks of human-level in the capacity to solve math Olympiad-type problems. They got gold at IMO.

So it seems like the model which got gold at the International Math Olympiad does have the goal of getting math problems right. Why can’t we extend this to different domains?

Richard Sutton 00:08:27

The math problems are different. Making a model of the physical world and carrying out the consequences of mathematical assumptions or operations, those are very different things. The empirical world has to be learned. You have to learn the consequences. Whereas the math is more computational, it’s more like standard planning. There they can have a goal to find the proof, and they are in some way given that goal to find the proof.

Dwarkesh Patel 00:09:10

It’s interesting because you wrote this essay in 2019 titled “The Bitter Lesson,” and this is the most influential essay, perhaps, in the history of AI. But people have used that as a justification for scaling up LLMs because, in their view, this is the one scalable way we have found to pour ungodly amounts of compute into learning about the world. It’s interesting that your perspective is that the LLMs are not “bitter lesson”-pilled.

Richard Sutton 00:09:41

It’s an interesting question whether large language models are a case of the bitter lesson. They are clearly a way of using massive computation, things that will scale with computation up to the limits of the Internet. But they’re also a way of putting in lots of human knowledge. This is an interesting question. It’s a sociological or industry question. Will they reach the limits of the data and be superseded by things that can get more data just from experience rather than from people?

In some ways it’s a classic case of the bitter lesson. The more human knowledge we put into the large language models, the better they can do. So it feels good. Yet, I expect there to be systems that can learn from experience. Which could perform much better and be much more scalable. In which case, it will be another instance of the bitter lesson, that the things that used human knowledge were eventually superseded by things that just trained from experience and computation.

Dwarkesh Patel 00:11:17

I guess that doesn’t seem like the crux to me. I think those people would also agree that the overwhelming amount of compute in the future will come from learning from experience. They just think that the scaffold or the basis of that, the thing you’ll start with in order to pour in the compute to do this future experiential learning or on-the-job learning, will be LLMs.

I still don’t understand why this is the wrong starting point altogether. Why do we need a whole new architecture to begin doing experiential, continual learning? Why can’t we start with LLMs to do that?

Richard Sutton 00:11:58

In every case of the bitter lesson you could start with human knowledge and then do the scalable things. That’s always the case. There’s never any reason why that has to be bad. But in fact, and in practice, it has always turned out to be bad. People get locked into the human knowledge approach, and they psychologically… Now I’m speculating why it is, but this is what has always happened. They get their lunch eaten by the methods that are truly scalable.

Dwarkesh Patel 00:12:34

Give me a sense of what the scalable method is.

Richard Sutton 00:12:37

The scalable method is you learn from experience. You try things, you see what works. No one has to tell you. First of all, you have a goal. Without a goal, there’s no sense of right or wrong or better or worse. Large language models are trying to get by without having a goal or a sense of better or worse. That’s just exactly starting in the wrong place.

00:13:04 – Do humans do imitation learning?

Dwarkesh Patel 00:13:04

Maybe it’s interesting to compare this to humans. In both the case of learning from imitation versus experience and on the question of goals, I think there’s some interesting analogies. Kids will initially learn from imitation. You don’t think so?

Richard Sutton 00:13:24

No, of course not.

Dwarkesh Patel 00:13:27

Really? I think kids just watch people. They try to say the same words…

Richard Sutton 00:13:32

How old are these kids? What about the first six months?

Dwarkesh Patel 00:13:37

I think they’re imitating things. They’re trying to make their mouth sound the way they see their mother’s mouth sound. Then they’ll say the same words without understanding what they mean. As they get older, the complexity of the imitation they do increases. You’re imitating maybe the skills that people in your band are using to hunt down the deer or something. Then you go into the learning from experience RL regime. But I think there’s a lot of imitation learning happening with humans.

Richard Sutton 00:14:04

It’s surprising you can have such a different point of view. When I see kids, I see kids just trying things and waving their hands around and moving their eyes around. There’s no imitation for how they move their eyes around or even the sounds they make. They may want to create the same sounds, but the actions, the thing that the infant actually does, there’s no targets for that. There are no examples for that.

Dwarkesh Patel 00:14:37

I agree. That doesn’t explain everything infants do, but I think it guides a learning process. Even an LLM, when it’s trying to predict the next token early in training, it will make a guess. It’ll be different from what it actually sees. In some sense, it’s very short-horizon RL, where it’s making this guess, “I think this token will be this.” It’s this other thing, similar to how a kid will try to say a word. It comes out wrong.

Richard Sutton 00:14:58

The large language models are learning from training data. It’s not learning from experience. It’s learning from something that will never be available during its normal life. There’s never any training data that says you should do this action in normal life.

Dwarkesh Patel 00:15:15

I think this is more of a semantic distinction. What do you call school? Is that not training data?

Richard Sutton 00:15:22

School is much later. Okay, I shouldn’t have said never. I don’t know, I think I would even say that about school. But formal schooling is the exception. You shouldn’t base your theories on that.

Dwarkesh Patel 00:15:35

But there are phases of learning where there’s the programming in your biology early on, you’re not that useful. Then why you exist is to understand the world and learn how to interact with it. It seems like a training phase. I agree that then there’s a more gradual… There’s not a sharp cutoff to training to deployment, but there seems to be this initial training phase right?

Richard Sutton 00:15:59

There’s nothing where you have training of what you should do. There’s nothing. You see things that happen. You’re not told what to do. Don’t be difficult. I mean this is obvious.

Dwarkesh Patel 00:16:14

You’re literally taught what to do. This is where the word training comes from, from humans.

Richard Sutton 00:16:20

I don’t think learning is really about training. I think learning is about learning, it’s about an active process. The child tries things and sees what happens. We don’t think about training when we think of an infant growing up.

These things are actually rather well understood. If you look at how psychologists think about learning, there’s nothing like imitation. Maybe there are some extreme cases where humans might do that or appear to do that, but there’s no basic animal learning process called imitation. There are basic animal learning processes for prediction and for trial-and-error control.

It’s really interesting how sometimes the hardest things to see are the obvious ones. It’s obvious—if you look at animals and how they learn, and you look at psychology and our theories of them—that supervised learning is not part of the way animals learn. We don’t have examples of desired behavior. What we have are examples of things that happen, one thing that followed another. We have examples of, “We did something and there were consequences.” But there are no examples of supervised learning.

Supervised learning is not something that happens in nature. Even if that were the case with school, we should forget about it because that’s some special thing that happens in people. It doesn’t happen broadly in nature. Squirrels don’t go to school. Squirrels can learn all about the world. It’s absolutely obvious, I would say, that supervised learning doesn’t happen in animals.

Dwarkesh Patel 00:18:11

I interviewed this psychologist and anthropologist, Joseph Henrich, who has done work about cultural evolution, basically what distinguishes humans and how humans pick up knowledge.

Richard Sutton 00:18:26

Why are you trying to distinguish humans? Humans are animals. What we have in common is more interesting. What distinguishes us, we should be paying less attention to.

Dwarkesh Patel 00:18:38

We’re trying to replicate intelligence. If you want to understand what it is that enables humans to go to the moon or to build semiconductors, I think the thing we want to understand is what makes that happen. No animal can go to the moon or make semiconductors. We want to understand what makes humans special.

Richard Sutton 00:18:54

I like the way you consider that obvious, because I consider the opposite obvious. We have to understand how we are animals. If we understood a squirrel, I think we’d be almost all the way there to understanding human intelligence. The language part is just a small veneer on the surface.

This is great. We’re finding out the very different ways that we’re thinking. We’re not arguing. We’re trying to share our different ways of thinking with each other.

Dwarkesh Patel 00:19:29

I think argument is useful. I do want to complete this thought. Joseph Henrich has this interesting theory about a lot of the skills that humans have had to master in order to be successful. We’re not talking about the last thousand years or the last 10,000 years, but hundreds of thousands of years. The world is really complicated.

It’s not possible to reason through how to, let’s say, hunt a seal if you’re living in the Arctic. There’s this many, many-step, long process of how to make the bait and how to find the seal, and then how to process the food in a way that makes sure you won’t get poisoned. It’s not possible to reason through all of that. Over time, there’s this larger process of whatever analogy you want to use—maybe RL, something else—where culture as a whole has figured out how to find and kill and eat seals.

In his view, what is happening when this knowledge is transmitted through generations, is that you have to imitate your elders in order to learn that skill. You can’t think your way through how to hunt and kill and process a seal. You have to watch other people, maybe make tweaks and adjustments, and that’s how knowledge accumulates. The initial step of the cultural gain has to be imitation. But maybe you think about it a different way?

Richard Sutton 00:21:00

No, I think about it the same way. Still, it’s a small thing on top of basic trial-and-error learning, prediction learning. It’s what distinguishes us, perhaps, from many animals. But we’re an animal first. We were an animal before we had language and all those other things.

Dwarkesh Patel 00:21:25

I do think you make a very interesting point that continual learning is a capability that most mammals have. I guess all mammals have it. It’s quite interesting that we have something that all mammals have, but our AI systems don’t have. Whereas the ability to understand math and solve difficult math problems—depends on how you define math—is a capability that our AIs have, but that almost no animal has. It’s quite interesting what ends up being difficult and what ends up being easy.

Richard Sutton 00:21:57

Moravec’s paradox.

Dwarkesh Patel 00:21:58

That’s right, that’s right.

00:23:10 – The Era of Experience

Dwarkesh Patel 00:23:10

This alternative paradigm that you’re imagining…

Richard Sutton 00:23:12

The experiential paradigm. Let’s lay it out a little bit. It says that experience, action, sensation—well, sensation, action, reward—this happens on and on and on for your life. It says that this is the foundation and the focus of intelligence. Intelligence is about taking that stream and altering the actions to increase the rewards in the stream.

Learning then is from the stream, and learning is about the stream. That second part is particularly telling. What you learn, your knowledge, is about the stream. Your knowledge is about if you do some action, what will happen. Or it’s about which events will follow other events. It’s about the stream. The content of the knowledge is statements about the stream. Because it’s a statement about the stream, you can test it by comparing it to the stream, and you can learn it continually.

Dwarkesh Patel 00:24:19

When you’re imagining this future continual learning agent…

Richard Sutton 00:24:22

They’re not “future”. Of course, they exist all the time. This is what the reinforcement learning paradigm is, learning from experience.

Dwarkesh Patel 00:24:29

Yeah, I guess what I meant to say is a general human-level, general continual learning agent. What is the reward function? Is it just predicting the world? Is it then having a specific effect on it? What would the general reward function be?

Richard Sutton 00:24:46

The reward function is arbitrary. If you’re playing chess, it’s to win the game of chess. If you’re a squirrel, maybe the reward has to do with getting nuts. In general, for an animal, you would say the reward is to avoid pain and to acquire pleasure. I think there also should be a component having to do with your increasing understanding of your environment. That would be sort of an intrinsic motivation.

Dwarkesh Patel 00:25:27

I see. With this AI, lots of people would want it to be doing lots of different kinds of things. It’s performing the task people want, but at the same time, it’s learning about the world from doing that task.

Let’s say we get rid of this paradigm where there’s training periods and then there’s deployment periods. Do we also get rid of this paradigm where there’s the model and then instances of the model or copies of the model that are doing certain things? How do you think about the fact that we’d want this thing to be doing different things? We’d want to aggregate the knowledge that it’s gaining from doing those different things.

Richard Sutton 00:26:11

I don’t like the word “model” when used the way you just did. I think a better word would be “the network” because I think you mean the network. Maybe there are many networks. Anyway, things would be learned. You’d have copies and many instances. Sure, you’d want to share knowledge across the instances. There would be lots of possibilities for doing that.

Today, you have one child grow up and learn about the world, and then every new child has to repeat that process. Whereas with AIs, with a digital intelligence, you could hope to do it once and then copy it into the next one as a starting place. This would be a huge savings. I think it’d be much more important than trying to learn from people.

Dwarkesh Patel 00:27:02

I agree that the kind of thing you’re talking about is necessary regardless of whether you start from LLMs or not. If you want human or animal-level intelligence, you’re going to need this capability.

Suppose a human is trying to make a startup. This is a thing which has a reward on the order of 10 years. Once in 10 years you might have an exit where you get paid out a billion dollars. But humans have this ability to make intermediate auxiliary rewards or have some way of…Even when they have extremely sparse rewards, they can still make intermediate steps having an understanding of what the next thing they’re doing leads to this grander goal we have. How do you imagine such a process might play out with AIs?

Richard Sutton 00:27:43

This is something we know very well. The basis of it is temporal difference learning where the same thing happens in a less grandiose scale. When you learn to play chess, you have the long-term goal of winning the game. Yet you want to be able to learn from shorter-term things like taking your opponent’s pieces.

You do that by having a value function which predicts the long-term outcome. Then if you take the guy’s pieces, your prediction about the long-term outcome is changed. It goes up, you think you’re going to win. Then that increase in your belief immediately reinforces the move that led to taking the piece.

We have this long-term 10-year goal of making a startup and making a lot of money. When we make progress, we say, “Oh, I’m more likely to achieve the long-term goal,” and that rewards the steps along the way.

Dwarkesh Patel 00:28:47

You also want some ability for information that you’re learning. One of the things that makes humans quite different from these LLMs is that if you’re onboarding on a job, you’re picking up so much context and information. That’s what makes you useful at the job. You’re learning everything from how your client has preferences to how the company works, everything.

Is the bandwidth of information that you get from a procedure like TD learning high enough to have this huge pipe of context and tacit knowledge that you need to be picking up in the way humans do when they’re just deployed?

Richard Sutton 00:29:27

I’m not sure but I think at the crux of this, the big world hypothesis seems very relevant. The reason why humans become useful on the job is because they are encountering their particular part of the world. It can’t have been anticipated and can’t all have been put in in advance. The world is so huge that you can’t.

The dream of large language models, as I see it, is you can teach the agent everything. It will know everything and won’t have to learn anything online, during its life. Your examples are all, “Well, really you have to” because you can teach it, but there’s all the little idiosyncrasies of the particular life they’re leading and the particular people they’re working with and what they like, as opposed to what average people like. That’s just saying the world is really big, and you’re going to have to learn it along the way.

Dwarkesh Patel 00:30:28

It seems to me you need two things. One is some way of converting this long-run goal reward into smaller auxiliary predictive rewards of the future reward, or the future reward that leads to the final reward. But initially, it seems to me, I need to hold on to all this context that I’m gaining as I’m working in the world. I’m learning about my clients, my company, and all this information.

Richard Sutton 00:31:04

I would say you’re just doing regular learning. Maybe you’re using “context” because in large language models all that information has to go into the context window. But in a continual learning setup, it just goes into the weights.

Dwarkesh Patel 00:31:17

Maybe context is the wrong word to use because I mean a more general thing.

Richard Sutton 00:31:20

You learn a policy that’s specific to the environment that you’re finding yourself in.

Dwarkesh Patel 00:31:25

The question I’m trying to ask is, you need some way of getting…How many bits per second is a human picking up when they’re out in the world? If you’re just interacting over Slack with your clients and everything.

Richard Sutton 00:31:41

Maybe you’re trying to ask the question of, it seems like the reward is too small of a thing to do all the learning that we need to do. But we have the sensations, we have all the other information we can learn from. We don’t just learn from the reward. We learn from all the data.

Dwarkesh Patel 00:31:59

What is the learning process which helps you capture that information?

Richard Sutton 00:32:06

Now I want to talk about the base common model of the agent with the four parts.

We need a policy. The policy says, “In the situation I’m in, what should I do?” We need a value function. The value function is the thing that is learned with TD learning, and the value function produces a number. The number says how well it’s going. Then you watch if that’s going up and down and use that to adjust your policy. So you have those two things. Then there’s also the perception component, which is construction of your state representation, your sense of where you are now.

The fourth one is what we’re really getting at, most transparently anyway. The fourth one is the transition model of the world. That’s why I am uncomfortable just calling everything “models,” because I want to talk about the model of the world, the transition model of the world. Your belief that if you do this, what will happen? What will be the consequences of what you do? Your physics of the world. But it’s not just physics, it’s also abstract models, like your model of how you traveled from California up to Edmonton for this podcast. That was a model, and that’s a transition model. That would be learned. It’s not learned from reward. It’s learned from, “You did things, you saw what happened, you made that model of the world.”

That will be learned very richly from all the sensation that you receive, not just from the reward. It has to include the reward as well, but that’s a small part of the whole model, a small, crucial part of the whole model.

00:33:39 – Current architectures generalize poorly out of distribution

Dwarkesh Patel 00:33:39

One of my friends, Toby Ord, pointed out that if you look at the MuZero models that Google DeepMind deployed to learn Atari games, these models were initially not a general intelligence itself, but a general framework for training specialized intelligences to play specific games. That is to say that you couldn’t, using that framework, train a policy to play both chess and Go and some other game. You had to train each one in a specialized way.

He was wondering whether that implies that with reinforcement learning generally, because of this information constraint, you can only learn one thing at a time? The density of information isn’t that high? Or whether it was just specific to the way that MuZero was done. If it’s specific to AlphaZero, what needed to be changed about that approach so that it could be a general learning agent?

Richard Sutton 00:34:35

The idea is totally general. I do use all the time, as my canonical example, the idea of an AI agent is like a person. People, in some sense, have just one world they live in. That world may involve chess and it may involve Atari games, but those are not a different task or a different world. Those are different states they encounter. So the general idea is not limited at all.

Dwarkesh Patel 00:35:06

Maybe it would be useful to explain what was missing in that architecture, or that approach, which this continual learning AGI would have.

Richard Sutton 00:35:19

They just set it up. It was not their ambition to have one agent across those games. If we want to talk about transfer, we should talk about transfer not across games or across tasks, but transfer between states.

Dwarkesh Patel 00:35:36

I guess I’m curious if historically, have we seen the level of transfer using RL techniques that would be needed to build this kind of…

Richard Sutton 00:35:49

Good. Good. We’re not seeing transfer anywhere. Critical to good performance is that you can generalize well from one state to another state. We don’t have any methods that are good at that. What we have are people trying different things and they settle on something, a representation that transfers well or generalizes well. But we have very few automated techniques to promote transfer, and none of them are used in modern deep learning.

Dwarkesh Patel 00:36:26

Let me paraphrase to make sure that I understood that correctly. It sounds like you’re saying that when we do have generalization in these models, that is a result of some sculpted…

Richard Sutton 00:36:42

Humans did it. The researchers did it. Because there’s no other explanation. Gradient descent will not make you generalize well. It will make you solve the problem. It will not make you, if you get new data, generalize in a good way.

Generalization means to train on one thing that’ll affect what you do on other things. We know deep learning is really bad at this. For example, we know that if you train on some new thing, it will often catastrophically interfere with all the old things that you knew. This is exactly bad generalization.

Generalization, as I said, is some kind of influence of training on one state on other states. The fact that you generalize is not necessarily good or bad. You can generalize poorly, you can generalize well. Generalization always will happen, but we need algorithms that will cause the generalization to be good rather than bad.

Dwarkesh Patel 00:37:41

I’m not trying to kickstart this initial crux again, but I’m just genuinely curious because I think I might be using the term differently. One way to think about these LLMs is that they’re increasing the scope of generalization from earlier systems, which could not really even do a basic math problem, to now where they can do anything in this class of Math Olympiad-type problems.

You initially start with them being able to generalize among addition problems. Then they can generalize among problems which require use of different kinds of mathematical techniques and theorems and conceptual categories, which is what the Math Olympiad requires. It sounds like you don’t think of being able to solve any problem within that category as an example of generalization. Let me know if I’m misunderstanding that.

Richard Sutton 00:38:33

Large language models are so complex. We don’t really know what information they have had prior. We have to guess because they’ve been fed so much. This is one reason why they’re not a good way to do science. It’s just so uncontrolled, so unknown.

Dwarkesh Patel 00:38:52

But if you come up with an entirely new…

Richard Sutton 00:38:54

They’re getting a bunch of things right, perhaps. The question is why. Well maybe that they don’t need to generalize to get them right, because the only way to get some of them right is to form something which gets all of them right. If there’s only one answer and you find it, that’s not called generalization. It’s just it’s the only way to solve it, and so they find the only way to solve it. But generalization is when it could be this way, it could be that way, and they do it the good way.

Dwarkesh Patel 00:39:24

My understanding is that this is working better and better, with coding agents. With engineers, obviously if you’re trying to program a library, there are many different ways you could achieve the end spec. An initial frustration with these models has been that they’ll do it in a way that’s sloppy. Over time they’re getting better and better at coming up with the design architecture and the abstractions that developers find more satisfying. It seems like an example of what you’re talking about.

Richard Sutton 00:39:56

There’s nothing in them which will cause it to generalize well. Gradient descent will cause them to find a solution to the problems they’ve seen. If there’s only one way to solve them, they’ll do that. But if there are many ways to solve it, some which generalize well, some which generalize poorly, there’s nothing in the algorithms that will cause them to generalize well. But people, of course, are evolved and if it’s not working out they fiddle with it until they find a way, perhaps until they find a way which generalizes well.

00:41:29 – Surprises in the AI field

Dwarkesh Patel 00:41:29

I want to zoom out and ask about being in the field of AI for longer than almost anybody who is commentating on it, or working in it now. I’m curious about what the biggest surprises have been. How much new stuff do you feel like is coming out? Or does it feel like people are just playing with old ideas? Zooming out, you got into this even before deep learning was popular. So how do you see the trajectory of this field over time and how new ideas have come about and everything? What’s been surprising?

Richard Sutton 00:42:06

I thought a little bit about this. There are a handful of things. First, the large language models are surprising. It’s surprising how effective artificial neural networks are at language tasks. That was a surprise, it wasn’t expected. Language seemed different. So that’s impressive.

There’s a long-standing controversy in AI about simple basic principle methods, the general-purpose methods like search and learning, compared to human-enabled systems like symbolic methods. In the old days, it was interesting because things like search and learning were called weak methods because they’re just using general principles, they’re not using the power that comes from imbuing a system with human knowledge. Those were called strong. I think the weak methods have just totally won. That’s the biggest question from the old days of AI, what would happen. Learning and search have just won the day.

There’s a sense in which that was not surprising to me because I was always hoping or rooting for the simple basic principles. Even with the large language models, it’s surprising how well it worked, but it was all good and gratifying. AlphaGo was surprising, how well that was able to work, AlphaZero in particular. But it’s all very gratifying because again, simple basic principles are winning the day.

Dwarkesh Patel 00:44:00

Whenever the public conception has been changed because some new application was developed— for example, when AlphaZero became this viral sensation—to you as somebody who has literally came up with many of the techniques that were used, did it feel to you like new breakthroughs were made? Or did it feel like, “Oh, we’ve had these techniques since the ‘90s and people are simply combining them and applying them now”?

Richard Sutton 00:44:28

The whole AlphaGo thing had a precursor, which is TD-Gammon. Gerry Tesauro did reinforcement learning, temporal difference learning methods, to play backgammon. It beat the world’s best players and it worked really well. In some sense, AlphaGo was merely a scaling up of that process. But it was quite a bit of scaling up and there was also an additional innovation in how the search was done. But it made sense. It wasn’t surprising in that sense.

AlphaGo actually didn’t use TD learning. It waited to see the final outcomes. But AlphaZero used TD. AlphaZero was applied to all the other games and it did extremely well. I’ve always been very impressed by the way AlphaZero plays chess because I’m a chess player and it just sacrifices material for positional advantages. It’s just content and patient to sacrifice that material for a long period of time. That was surprising that it worked so well, but also gratifying and it fit into my worldview.

This has led me where I am. I’m in some sense a contrarian or someone thinking differently than the field is. I’m personally just content being out of sync with my field for a long period of time, perhaps decades, because occasionally I have been proved right in the past. The other thing I do—to help me not feel I’m out of sync and thinking in a strange way—is to look not at my local environment or my local field, but to look back in time and into history and to see what people have thought classically about the mind in many different fields. I don’t feel I’m out of sync with the larger traditions. I really view myself as a classicist rather than as a contrarian. I go to what the larger community of thinkers about the mind have always thought.

00:46:41 – Will The Bitter Lesson still apply after AGI?

Dwarkesh Patel 00:46:41

Some sort of left-field questions for you if you’ll tolerate them. The way I read the bitter lesson is that it’s not necessarily saying that human artisanal researcher tuning doesn’t work, but that it obviously scales much worse than compute, which is growing exponentially. So you want techniques which leverage the latter.

Richard Sutton 00:47:06

Yep.

Dwarkesh Patel 00:47:07

Once we have AGI, we’ll have researchers which scale linearly with compute. We’ll have this avalanche of millions of AI researchers. Their stock will be growing as fast as compute. So maybe this will mean that it is rational or it will make sense to have them doing good old-fashioned AI and doing these artisanal solutions. As a vision of what happens after AGI in terms of how AI research will evolve, I wonder if that’s still compatible with a bitter lesson.

Richard Sutton 00:47:40

How did we get to this AGI? You want to presume that it’s been done.

Dwarkesh Patel 00:47:45

Suppose it started with general methods, but now we’ve got the AGI. And now we want to go…

Richard Sutton 00:47:52

Then we’re done.

Dwarkesh Patel 00:47:53

Interesting. You don’t think that there’s anything above AGI?

Richard Sutton 00:47:58

But you’re using it to get AGI again.

Dwarkesh Patel 00:48:01

Well, I’m using it to get superhuman levels of intelligence or competence at different tasks.

Richard Sutton 00:48:05

These AGIs, if they’re not superhuman already, then the knowledge that they might impart would be not superhuman.

Dwarkesh Patel 00:48:15

I guess there are different gradations.

Richard Sutton 00:48:16

I’m not sure your idea makes sense because it seems to presume the existence of AGI and that we’ve already worked that out.

Dwarkesh Patel 00:48:27

Maybe one way to motivate this is, AlphaGo was superhuman. It beat any Go player. AlphaZero would beat AlphaGo every single time. So there are ways to get more superhuman than even superhuman. It was also a different architecture. So it seems possible to me that the agent that’s able to generally learn across all domains, there would be ways to give it better architecture for learning, just the same way that AlphaZero was an improvement upon AlphaGo and MuZero was an improvement upon AlphaZero.

Richard Sutton 00:48:56

And the way AlphaZero was an improvement was that it did not use human knowledge but just went from experience.

Dwarkesh Patel 00:49:04

Right.

Richard Sutton 00:49:04

So why do you say, “Bring in other agents’ expertise to teach it”, when it’s worked so well from experience and not by help from another agent?

Dwarkesh Patel 00:49:19

I agree that in that particular case that it was moving to more general methods. I meant to use that particular example to illustrate that it’s possible to go superhuman to superhuman++, to superhuman+++. I’m curious if you think those gradations will continue to happen by just making the method simpler. Or, because we’ll have the capability of these millions of minds who can then add complexity as needed, will that continue to be a false path, even when you have billions of AI researchers or trillions of AI researchers?

Richard Sutton 00:49:51

It’s more interesting just to think about that case. When you have many AIs, will they help each other the way cultural evolution works in people? Maybe we should talk about that. The bitter lesson, who cares about that? That’s an empirical observation about a particular period in history. 70 years in history, it doesn’t necessarily have to apply to the next 70 years.

An interesting question is, you’re an AI, you get some more computer power. Should you use it to make yourself more computationally capable? Or should you use it to spawn off a copy of yourself to go learn something interesting on the other side of the planet or on some other topic and then report back to you?

I think that’s a really interesting question that will only arise in the age of digital intelligences. I’m not sure what the answer is. More questions, will it be possible to really spawn it off, send it out, learn something new, something perhaps very new, and then will it be able to be reincorporated into the original? Or will it have changed so much that it can’t really be done? Is that possible or is that not? You could carry this to its limit as I saw one of your videos the other night. It suggests that it could. You spawn off many, many copies, do different things, highly decentralized, but report back to the central master. This will be such a powerful thing.

This is my attempt to add something to this view. A big issue will become corruption. If you really could just get information from anywhere and bring it into your central mind, you could become more and more powerful. It’s all digital and they all speak some internal digital language. Maybe it’ll be easy and possible. But it will not be as easy as you’re imagining because you can lose your mind this way. If you pull in something from the outside and build it into your inner thinking, it could take over you, it could change you, it could be your destruction rather than your increment in knowledge.

I think this will become a big concern, particularly when you’re like, “Oh, he’s figured out all about how to play some new game or he’s studied Indonesia, and you want to incorporate that into your mind.” You could think, “Oh, just read it all in, and that’ll be fine.” But no, you’ve just read a whole bunch of bits into your mind, and they could have viruses in them, they could have hidden goals, they can warp you and change you. This will become a big thing. How do you have cybersecurity in the age of digital spawning and re-reforming again?

00:53:48 – Succession to AI

Dwarkesh Patel 00:53:48

I guess this brings us to the topic of AI succession. You have a perspective that’s quite different from a lot of people that I’ve interviewed and a lot of people generally. I also think it’s a very interesting perspective. I want to hear about it.

Richard Sutton 00:54:03

I do think succession to digital intelligence or augmented humans is inevitable. I have a four-part argument. Step one is, there’s no government or organization that gives humanity a unified point of view that dominates and that can arrange... There’s no consensus about how the world should be run. Number two, we will figure out how intelligence works. The researchers will figure it out eventually. Number three, we won’t stop just with human-level intelligence. We will reach superintelligence. Number four, it’s inevitable over time that the most intelligent things around would gain resources and power.

Put all that together and it’s sort of inevitable. You’re going to have succession to AI or to AI-enabled, augmented humans. Those four things seem clear and sure to happen. But within that set of possibilities, there could be good outcomes as well as less good outcomes, bad outcomes. I’m just trying to be realistic about where we are and ask how we should feel about it.

Dwarkesh Patel 00:55:35

I agree with all four of those arguments and the implication. I also agree that succession contains a wide variety of possible futures. Curious to get more thoughts on that.

Richard Sutton 00:55:50

I do encourage people to think positively about it. First of all, it’s something we humans have always tried to do for thousands of years, try to understand ourselves, trying to make ourselves think better, just understanding ourselves. This is a great success for science, humanities. We’re finding out what this essential part of humanness is, what it means to be intelligent.

Then what I usually say is that this is all human-centric. But if we step aside from being a human and just take the point of view of the universe, this is I think a major stage in the universe, a major transition, a transition from replicators. We humans and animals, plants, we’re all replicators. That gives us some strengths and some limitations.

We’re entering the age of design because our AIs are designed. Our physical objects are designed, our buildings are designed, our technology is designed. We’re designing AIs now, things that can be intelligent themselves and that are themselves capable of design. This is a key step in the world and in the universe. It’s the transition from the world in which most of the interesting things that are, are replicated. Replicated means you can make copies of them, but you don’t really understand them. Right now we can make more intelligent beings, more children, but we don’t really understand how intelligence works.

Whereas we’re reaching now to having designed intelligence, intelligence that we do understand how it works. Therefore we can change it in different ways and at different speeds than otherwise. In our future, they may not be replicated at all. We may just design AIs, and those AIs will design other AIs, and everything will be done by design and construction rather than by replication.

I mark this as one of the four great stages of the universe. First there’s dust, it ends with stars. Stars make planets. The planets can give rise to life. Now we’re giving rise to designed entities. I think we should be proud that we are giving rise to this great transition in the universe.

It’s an interesting thing. Should we consider them part of humanity or different from humanity? It’s our choice. It’s our choice whether we should say, “Oh, they are our offspring and we should be proud of them and we should celebrate their achievements.”Or we could say, “Oh no, they’re not us and we should be horrified.” It’s interesting that it feels to me like a choice. Yet it’s such a strongly held thing that, how could it be a choice? I like these sort of contradictory implications of thought.

Dwarkesh Patel 00:58:57

It is interesting to consider if we are just designing another generation of humans. Maybe design is the wrong word. But we know a future generation of humans is going to come up. Forget about AI. We just know in the long run, humanity will be more capable and more numerous, maybe more intelligent. How do we feel about that? I do think there are potential worlds with future humans that we would be quite concerned about.

Richard Sutton 00:59:22

Are you thinking like, maybe we are like the Neanderthals that give rise to Homo sapiens. Maybe Homo sapiens will give rise to a new group of people.

Dwarkesh Patel 00:59:32

Something like that. I’m basically taking the example you’re giving. Even if we consider them part of humanity, I don’t think that necessarily means that we should feel super comfortable.

Richard Sutton 00:59:42

Kinship.

Dwarkesh Patel 00:59:43

Like Nazis were humans, right? If we thought, “Oh, the future generation will be Nazis, I think we’d be quite concerned about just handing off power to them.” So I agree that this is not super dissimilar to worrying about more capable future humans, but I don’t think that addresses a lot of the concerns people might have about this level of power being attained this fast with entities we don’t fully understand.

Richard Sutton 01:00:09

I think it’s relevant to point out that for most of humanity, they don’t have much influence on what happens. Most of humanity doesn’t influence who can control the atom bombs or who controls the nation states. Even as a citizen, I often feel that we don’t control the nation states very much. They’re out of control.

A lot of it has to do with just how you feel about change. If you think the current situation is really good, then you’re more likely to be suspicious of change and averse to change than if you think it’s imperfect. I think it’s imperfect. In fact, I think it’s pretty bad. So I’m open to change. I think humanity has not had a super good track record. Maybe it’s the best thing that there has been, but it’s far from perfect.

Dwarkesh Patel 01:01:13

I guess there are different varieties of change. The Industrial Revolution was change, the Bolshevik Revolution was also change. If you were around in Russia in the 1900s and you were like, “Look, things aren’t going well, the tsar is kind of messing things up, we need change”, I’d want to know what kind of change you wanted before signing on the dotted line. Similarly with AI, where I’d want to understand and, to the extent that it’s possible, change the trajectory of AI such that the change is positive for humans.

Richard Sutton 01:01:49

We should be concerned about our future, the future. We should try to make it good. We should also though recognize the limit, our limits. I think we want to avoid the feeling of entitlement, avoid the feeling of, “Oh, we are here first, we should always have it in a good way.” How should we think about the future? How much control should a particular species on a particular planet have over it? How much control do we have? A counterbalance to our limited control over the long-term future of humanity should be how much control do we have over our own lives. We have our own goals. We have our families. Those things are much more controllable than trying to control the whole universe.

I think it’s appropriate for us to really work towards our own local goals. It’s kind of aggressive for us to say, “Oh, the future has to evolve this way that I want it to.” Because then we’ll have arguments where different people think the global future should evolve in different ways, and then they have conflict. We want to avoid that.

Dwarkesh Patel 01:03:13

Maybe a good analogy here would be this. Suppose you are raising your own children. It might not be appropriate to have extremely tight goals for their own life, or also have some sense of like, “I want my children to go out there in the world and have this specific impact. My son’s going to become president and my daughter is going to become CEO of Intel. Together they’re going to have this effect on the world.”

But people do have the sense—and I think this is appropriate—of saying, “I’m going to give them good robust values such that if and when they do end up in positions of power, they do reasonable, prosocial things.” Maybe a similar attitude towards AI makes sense, not in the sense of we can predict everything that they will do, or we have this plan about what the world should look like in a hundred years. But it’s quite important to give them robust and steerable and prosocial values.

Richard Sutton 01:04:13

Prosocial values?

Dwarkesh Patel 01:04:15

Maybe that’s the wrong word.

Richard Sutton 01:04:16

Are there universal values that we can all agree on?

Dwarkesh Patel 01:04:21

I don’t think so, but that doesn’t prevent us from giving our kids a good education, right? Like we have some sense of wanting our children to be a certain way.

Maybe prosocial is the wrong word. High integrity is maybe a better word. If there’s a request or if there’s a goal that seems harmful, they will refuse to engage in it. Or they’ll be honest, things like that. We have some sense that we can teach our children things like this, even if we don’t have some sense of what true morality is, where everybody doesn’t agree on that. Maybe that’s a reasonable target for AI as well.

Richard Sutton 01:04:57

So we’re trying to design the future and the principles by which it will evolve and come into being. The first thing you’re saying is, “Well, we try to teach our children general principles which will promote more likely evolutions.” Maybe we should also seek for things to be voluntary. If there is change, we want it to be voluntary rather than imposed on people. I think that’s a very important point. That’s all good.

I think this is the big or one of the really big human enterprises to design society that’s been ongoing for thousands of years again. The more things change, the more things they stay the same. We still have to figure out how to be. The children will still come up with different values that seem strange to their parents and their grandparents. Things will evolve.

Dwarkesh Patel 01:05:57

“The more things change, the more they stay the same” also seems like a good capsule into the AI discussion. The AI discussion we were having was about how techniques, which were invented even before their application to deep learning and backpropagation was evident, are central to the progression of AI today. Maybe that’s a good place to wrap up the conversation.

Richard Sutton 01:06:19

Okay. Thank you very much.

Dwarkesh Patel 01:06:21

Awesome. Thank you for coming on.

Richard Sutton 01:06:22

My pleasure.