Dwarkesh Podcast

One definition of intelligence is sample efficiency - that is to say, how much data do you need to see in a given domain in order to operate fluently and competently. It’s not clear that we’ve actually made much progress on training sample efficiency over the last few years - it seems like more so we’ve dramatically widened and improved the data distribution.

The main way that AIs have been getting better is from adding more and better data, and scaling the compute to develop that data in the first place. Obviously RL is the main way that has happened. You can think of RL as a kind of synthetic data generation - you dump a lot of compute against a verifier in order to find the “good” data. Then you train your model to predict these correct rollouts, much in the same way that you might train it to predict the next word in internet text.

For this process to work, the model must have at least prior some probability to anticipate the correct solution, which is why you also need mind-stretching amounts of human expert trajectories in every single field and skill you want the model to be competent at.

It’s hard to overstate how task specific and bespoke this human expert data is. If you want to get some intuition, go read some job descriptions at Mercor or Surge’s websites. There are listings for a word specialists who will convert legacy documents into polished Word files, and legal experts who will write realistic M&A diligences or securities filings, and management consultants who will write up template market research, and dozens more other particular categories.

And it is not only that the data have to be so domain specific, but there has to be so much of it! Each skill corresponds to at least hundreds of human experts who are generating example completions, writing rubrics, and explaining their chain of thought. There’s a reason that the data industry producing these expert labels (and the RL environments in which their meticulously catalogued skills can congeal) is earning billions a year in revenue, soon deca-billions.

Imagine if it took a couple decades worth of courses with hundreds of concurrent professors and millions of practice tasks for you to learn how to polish a word file. Even the task count difference understates the gap - the models have to grind their far more numerous tasks each far harder. Whereas a human student might practice a textbook problem once or twice, GRPO has the model generate hundreds to thousands of rollouts per task. We are building some Frankenstein’s monster, with a billion grafts of carefully constructed examples sewn together.

Epoch recently reported that open models only lag state-of-the-art closed models by 4 months. I think the reason it is relatively easy for open source and previous laggards to catch up to within months of the frontier is that data is the real driver of progress. And data can be easily distilled from public APIs, whereas hyper-parameters and training tricks and architectural micro-optimizations cannot - if the latter were driving most of progress, then catching up would be harder than we are observing it to be.

It is easy to forget how much data these models are trained on, and how much more it is than what we humans see in our lifetimes. We see these AIs as a galaxy glittering with capabilities, but at their center, invisible to the naked eye, holding all the constellations together, is an unimaginably massive black hole of data.

• If a person hears and sees on average ~2,000 words an hour, then from birth to adulthood, they’ll see ~200 millions tokens. By contrast, frontier models are trained on somewhere between 10s to 100s of trillions of tokens. That is close to a million fold difference.

• A person can learn to teleoperate any random humanoid or robot arm within hours. The reason robotics isn’t already a deca-trillion dollar industry, with a endless army of Unitree G1s doing all kinds of useful work in world, is that our AIs learn so much less efficiently than humans, and even the millions of hours of demonstrations we’ve collected is not enough to allow them to perform complex, open ended tasks.

• A teenager can learn to drive a car with about 20 hours of practice. Even if you include their ~16 years of accumulated physical intuition as relevant training data, that is at least 3-4 orders of magnitude less than the amount of data Waymo and Tesla have needed to train their self-driving car models.

I wanna deal with some common objections to this kind of comparison:

• Many billions of years of evolution is our pre-training, so it’s unfair to compare how little data we see simply within our lifetime to what these cold-started LLMs have to learn from.

  • Our genome is 3GB, about 1-2% protein coding. That is just not enough space to store the model parameters that are supposedly pretrained (frontier models are terabytes sized). The closer analogy is probably that evolution has found the right hyperparameters and loss functions (Sidenote: I had an interesting podcast with Adam Marblestone where he argued that the loss functions were the more significant find from evolution), but that the equivalent of parameter training is still happening within lifetime, and is encoded in the map of neural connections in the brain built up over a lifetime.

  • Even if it were the case that we can explain away the trillions of tokens required to pretrain a base model as catching up to evolution, it doesn’t explain why the marginal capabilities take so much data - once you have been educated, you don’t need 100 different professors to learn a new programming language, but the AIs (even once pretrained) do.

• These comparisons are not including the multimodal data we see in our lifetimes. If you include all this sensory information, we’re probably in the 10s to 100s of billions of tokens range from birth to adulthood

  • Blind/deaf people who are cut off from this kind of sensory information might lack faculty with the relevant sense, but still have the same general intelligence as everyone else. Which suggests that all these billions of sensory tokens are not really the thing making humans smart.

  • In fact, deaf people who can only communicate via sign language and reading (and not from hearing) are ingesting far less than the 200 million language tokens we calculated earlier, and even this is sufficient for them to be fully general intelligences.

• Scaling laws tell us that bigger models are more sample efficient. The human brain is 100T synapses - if each synapse is ~1 parameter, and frontier models are currently roughly ~5T parameters, then maybe we could achieve human-level sample efficiency with another order of magnitude or two of parameter scaling.

  • The way the scaling law equations work is that parameter and data terms are added to the loss independently. If you have a model that is trained compute optimally, and suppose you ask, well what if I just wanna maximize sample efficiency and use less data - and I’ll throw in as many parameters as it takes to make that happen. With the constants from the Chinchilla scaling laws paper (and the nature of the result wouldn’t change even with different constants), even if you increased the number of parameters by infinity, that would only decrease by a factor of ~10 the amount of data you need in order to keep the same loss. Humans are somewhere between thousands to millions of times more sample efficient than these models. Scaling of current models simply can’t make up for that discrepancy. This really does suggest that humans are on a different scaling curve altogether.

But you might ask, why does sample efficiency matter? The labs have two overarching objectives: automate white collar work, and automate AI research itself. Is human level sample efficiency necessary for either?

The bet with white collar work is the common tasks that a software engineer or analyst or accountant does are, well common. And we can bring common tasks into distribution quite easily through RL and SFT. The revenue curves of these AI labs suggest that there is enormous value from bringing tasks into distribution, even if we don’t replicate human sample efficiency.

Yes it is far more inefficient to train AIs to do these tasks than it is to train humans. But so what? Human lifespan does not allow for the quantity and breath of training these models experience. If you as a human had some weird learning disability where you needed to read through every public repository on Github before you could be a competent developer, it would not make sense to train you up. You’d be on Social Security by the early stages of your education, and even once you were trained, you could work on only one project at a time. But AIs can learn these skills by firehosing gigawatts of training at a time. And what they learn can be amortized across billions of sessions, so we can be ludicrously inefficient in training them and still be wildly in the green.

How much “out-of-distribution” thinking do white collar employees need to do that you simply can’t train for in advance? Well this is more a question about the nature of different jobs rather than a question about AI research. And also depends on the job - some jobs are mechanical and predictable enough that they were automated long before the modern era of AI, for example bank tellers or travel agents. And there are other jobs which require dealing on a daily basis with problems that are quite distant from the data distribution. Even software engineering (the jobs AIs are supposed to take first) is one such. I would be willing to bet that there’s overall more demand for human software engineers in 2028 than there is now, largely due to the complementary input of AI.

The labs’ plan for these later kinds of jobs is to first automate AI research, and then have the automated AI researchers solve this sample efficiency problem. So then the question is, can AIs, which do not have human-level sample efficiency, nonetheless solve the remaining research problems on the way to human-like intelligence and learning.

That question I’ll address in a future blog post - I think the way that people currently think about an intelligence explosion is pretty clumsy. Either people dismiss the possibility of AIs speeding up AI progress altogether, or they just assume that God pops out the other end. People are not reasoning about what extremely rapid progress, but starting with LLMs, looks like.

Thanks to Mercury for sponsoring this essay! Mercury is my banking platform, and they just released a new AI feature called Command. Since I already use Mercury to run basically my entire business, Command has access to all the info it needs to get real work done. I can ask it to send invoices, or categorize expenses, or even transfer money… and Command just handles it. Learn more at mercury.com/command

Had Ada Palmer back on – this time to talk about Machiavelli, perhaps the most misunderstood thinker of all time.

Machiavelli cut his teeth as a high-level diplomat for Florence, a position from which he got to closely observe the most important rulers in Europe at the time, including the ones who were on the path to destroying his dearly beloved Florence.

In 1513 the Medici retook control of Florence and, wrongly suspecting Machiavelli of participating in a coup attempt, fired, tortured, and exiled him.

Machiavelli could have fled his exile and worked for any number of different principalities that would have been eager to make use of his talents.

Instead, he decided to rot in the countryside and compile his career’s lessons about power, politics, and human nature into a book he dedicated to the very man whose new regime had tortured and exiled him, Lorenzo di Piero de’ Medici.

But at least the Medici were in a position to use his insights to defend Florence. Machiavelli the patriot did not want any other hands to touch this book, because those hands, armed with these lessons, might pose an existential danger to Florence.

The closest modern analogy, at least as Machiavelli would have seen it, would be Szilard’s letter warning FDR about the possibility of a nuclear fission bomb.

What were those insights? And how were they inspired by Machiavelli’s dangerous diplomatic missions all across Europe, and his extensive reading of antiquity? Watch this episode with Ada Palmer to find out!

By the way, Ada is launching a new podcast which I’m very excited about. The first season will be about Machiavelli – a perfect way to dive deeper into the topics we discussed in this episode. Subscribe at Beforecast’s website to be notified of the first episode, subscribe on YouTube, follow her on Patreon, and if you want even more Ada, check out her FixTheNews Podcast episode, and check out her books and more.

Watch on YouTube; listen on Apple Podcasts or Spotify.

• Cursor recently saved one of my podcast recordings. When a video file from a shoot came out corrupted, I pointed Cursor at it: it recovered the footage on its own, tracking down the right reference file from the file’s metadata and realigning the out-of-sync audio. My whole team now uses Cursor for everyday tasks, not just coding. Get started at cursor.com/dwarkesh

• Jane Street’s hiring process has been going viral on Twitter lately. The memes are pretty funny, but I wanted to see what their interviews were actually like. So I had Ricson, one of Jane Street’s ML researchers, walk me through a retired puzzle: he gave me an image dataset where 50% of the files had been corrupted – I had to figure out how to recover them. If you’re interested in these sorts of puzzles, you can find Jane Street’s open roles at janestreet.com/dwarkesh

• Crusoe is turning the AI datacenter buildout into an industrial process. At their massive Colorado factory, they assemble Spark units, modular datacenters with power, cooling, and fire suppression built in. They also manufacture specific components in-house to skip the longest lead times. Crusoe has experience running these Spark units on a range of energy sources, including solar and used EV batteries, ensuring they don’t get bottlenecked by grid availability. Learn more at crusoe.ai/dwarkesh

(00:00:00) – How Florence bargained with Cesare Borgia for survival

(00:15:08) – Machiavelli’s analytical innovations

(00:23:58) – Why popes became warlords

(00:36:13) – Why the common people demanded nepotism

(00:47:57) – Cesare Borgia brought terror to rulers and justice to the people

(00:57:55) – Art as a proxy for war

(01:06:41) – Florence, a city famous in hell

(01:15:57) – The Prince was a job application to Machiavelli’s torturers

(01:41:39) – During the Renaissance, original ideas had to be couched in antiquity

(01:50:44) – Why copyright began with the Inquisition

(02:02:12) – Machiavelli wasn’t Machiavellian

Dwarkesh Patel

Okay, I’m back with Ada Palmer, who is a science fiction author, composer, and historian at the University of Chicago. Today, I want to talk to you about Machiavelli. He writes The Prince. He dedicates it to Lorenzo di Piero de’ Medici and gives it to him in 1513. He says in the final chapter, “You’re the only person who can bring Italy from its current place of ruin and ravage.” Why were things so bad? What is the historical context in which he’s writing The Prince?

Ada Palmer

I’m going to give a two-part answer to that, although of course with any granular history there can be many parts. The papacy is part of it, and then the city-state structure of Italy is another part of it.

I’ll start with the city-state structure. There’s a principle in politics that when there’s long continuity of a government, and the government has been in power a long time, that government has a lot of legitimacy. People believe in its institutions. People are used to it. Even if you complain about it, it’s the government. When you break that—when you overthrow the ruler, when you dissolve the republic, when you put in a new thing—it doesn’t have that same staying power. So it’s very common when there’s one regime change for there to then be five regime changes, rapid fire, over and over.

We see this with how many iterations the French Republic goes through: the French Republic, and then restored monarchy, and then republic, and then monarchy. When a long thing cracks—boom, boom, boom, boom, boom—you get chaos. England’s Wars of the Roses are similar. There was one stable dynasty for a long time. The moment that a king is overthrown, then you have overthrow, overthrow, overthrow, overthrow for a long time, because the thread of continuity was cut.

In Machiavelli’s lifetime, that thread of continuity is cut for the majority of cities in Italy. And that guarantees, from his perspective, that there are going to be more, and more, and more overthrows in those governments. When Machiavelli was born, there were six or seven city-states in Italy that had had their governments uprooted recently. By the time he’s writing The Prince, it’s dozens, in fact, the majority of these places. So it’s volatile. Almost no government has staying power. Almost every government is ripe for yet another replacement, yet another replacement, yet another replacement. That’s half the answer of why he perceives there to be this urgency and this guarantee that there cannot be stability.

The other half is the papacy. The papacy, of course, is a long and evolving organism. The papacy is one of the oldest institutions in the world now. It was one of the oldest institutions in the world even then, even though this is 500 years ago. As we all know, when you have power centralized in an authority, especially an executive, there can be changes in how that executive uses that power. Each one sets norms for the next one.

Over the course of Machiavelli’s lifetime and just before, a bunch of consecutive popes expanded executive power, especially in the military side, and launched more wars, or did more arbitrary overthrow of governments. You have a number of city-states that are directly ruled by the papacy, and in theory, the pope can appoint anybody to be ruler of that city. Here is a pope. He has an illegitimate son. He wants his illegitimate son to be ruler of something, so he overthrows the government of a city and puts in his son. The next pope does it to three cities. The next pope does it to five. Soon we have a precedent that every new pope feels he has the authority to knock down every pawn upon the chessboard if he feels like it.

Once that is the norm, even a fairly nice pope still inherits the idea that the pope is going to overthrow and replace governments. This creates a unique instability within Italy that no other part of Europe is subject to, because there is no predictability to who’s going to be pope next. It isn’t hereditary. You can’t plan for it. The next pope is elected. As is often the case with elections, very frequently the next pope will be elected by a coalition of all the people who hate the current pope.

One of the things that electoral politics does is that it tends to swing, in which those outside of power work hard to get into power with the next regime. Let’s assume the average length of a papacy is ten years in this period. So every ten years, you suddenly have a completely unpredictable new monarch who’s almost guaranteed to be one of the enemies of the old monarch, and will therefore rip up and replace all of the things that that monarch tried to do with new things.

So Machiavelli, when he’s writing the last chapter of The Prince, is looking around and saying, “Okay, we have a perfect storm. Practically every polity in this region has just had the thread of legitimacy cut. Its institutions have no traditions. Its people have no investment in its current rulers. These are all pawns that have been knocked over before and barely stood up again. They’re ready to fall.” Meanwhile, nothing will stop the turnover of popes. The only thing that could stop the turnover of popes would be one person gaining enough power and ascendancy near this region, who has staying power, who has sons and heredity, that he can do what Cesare Borgia tried to do: have enough power near the papacy to strongly influence the next pope to create a kind of stability that’s otherwise impossible.

Dwarkesh Patel

So he wants the Medicis to not unify Italy, but stabilize Italy at the very least.

Ada Palmer

Exactly, by having conquered enough of a chunk that the papacy fears them and must negotiate with them, as opposed to the papacy being surrounded by small, weakened powers that will constantly be turned over and turned over and turned over.

Dwarkesh Patel

Right, and the pope now is a Medici, right?

Ada Palmer

At that moment, yes.

Dwarkesh Patel

So it makes it even more plausible. Let’s lay down a little more historical context. Before Machiavelli writes The Prince, he’s a bureaucratic diplomat. He meets through his career a lot of these famous figures. I want to know what he makes, for example, of King Louis of France, Maximilian of Germany, the Holy Roman Empire. I want to know what he made of Cesare Borgia.

Ada Palmer

He spends a lot of The Prince, in fact, trying to veil how much more he cares about Cesare Borgia than everyone else. It’s so interesting. He tries to be balanced. He tries to talk about this example, and this example, and this example, and Valentino, and this example. Sometimes he just can’t.

There’s that incredible, magical moment when he’s discussing Valentino’s fall. It’s the moment when he has amassed all this power, he’s successfully conquered almost everything within Italy. Suddenly both his father, the Pope, and him fall ill at once. When Machiavelli describes this, he’s saying, “Everything Cesare Borgia did, he did right. He conquered this kingdom. He would’ve kept it. The only reason he lost it was fortune.”

What Machiavelli should say is, “Valentino had planned for every contingency at his father’s death, except the possibility that he would also be on death’s door.” But that’s not what Machiavelli says. What Machiavelli says is, “He told me that he had planned.” The first person breaks in. Our historian cannot veil himself anymore. He cares too much. “He told me”, first person, that he had prepared for everything in the event of his father’s death, except the possibility that he himself would also be incapacitated at the moment.

It’s such a magical moment where the veil between the author and the reader breaks for just that moment. We realize that all of these others, he observed from a distance. But Machiavelli was in the room next to Valentino, at Valentino’s side through this. He had the most incredible, life-changing, first-person view of this man so unique, and charismatic, and terrifying, that when you read accounts of him, they range from “This was the most incredible, charismatic leader I’ve ever met,” to “This man was supernaturally charismatic to the degree that he must be literally the Antichrist or an incarnation of the angel of death on Earth, because I have no other explanation of how he could be so persuasive and charismatic.” Machiavelli was in the room. And every so often you just feel that he’s still in the spell of this incredible figure at whose side he had the scariest job in the world.

Machiavelli’s job dealing with Cesare Borgia… It’s very clear that the Borgia plan is to conquer the Papal States in the middle of Italy. Tuscany, Florence’s dominion, is this little notch, like a puzzle piece out of the side of the Papal States. Anybody with a map looking at it is like, “You’ve got to conquer that. You just have to conquer this. You can’t have a kingdom without it.” There is no way to stop it. So what do you do?

Machiavelli’s advice to his polity is: this time we’re not going to succeed in persuading this conqueror to pass us by. We can’t bribe him into doing something else permanently. But we can buy time. We can absolutely and abjectly swear to do anything he wants. We can give him our forces, and we can give him our money. We can pay him and help him conquer the rest of it, and betray our allies. Betray Bologna. Florence had had a 300-year alliance to defend Bologna. He said, “We have to break it. The whole world is broken right now. We have to break every promise and every hereditary alliance we had. We must be at the side of this man.”

The only possible survival mechanism is to win from him through loyalty, through support, and through Machiavelli being at his ear whispering forever, “Florence is loyal. Florence is loyal.” By that, we buy the boon of Polyphemus, the terrifying promise of the conqueror: “I like you, my guest. I’ll eat you last.” That’s the republic’s only hope. That’s Machiavelli’s job: to stand next to the scariest man who has lived in Europe since Frederick Barbarossa and whisper constantly in his ear, “The Florentine Republic will support you and will give your grace anything you ask. Just eat us last.”

Dwarkesh Patel

Doesn’t it contradict what he was saying in The Prince about how you should never rise with the help of great powers, for even in success you have empowered somebody who is stronger than you and at whose mercy you are?

Ada Palmer

This is not Florence aiming to rise. This is not Florence expecting that it will gain anything by this. This is Florence knowing it will lose. Machiavelli’s very open about the fact that if Alexander had lived another year, Valentino would have finished his conquests, and taken Florence at last, and it would’ve been over. But popes are mortal. Buying time is sometimes the survival mechanism.

So Machiavelli has this incredible firsthand experience of being with Valentino through all of these decisions, being with him at the massacre at Senigallia when rumor had reached Valentino that some of his people were terrified of him and plotting to overthrow him. They were so scared of him, they decided to abandon the plot, and he heard. He met with them and told them, “I forgive you. It’s okay. You’ve renewed your loyalty to me. You’ve passed the test. I trust you. All is well.” He invites them to the banquet, and then massacres them all. The forgiveness is false. The betrayals are punished.

There’s this amazing letter a couple of months afterward where Machiavelli’s loved ones are writing from Florence because they’ve received a letter from him after the massacre at Senigallia. They say, “Oh, thank God, you’re alive. We had no idea. All we heard was that he had massacred a large number of the people who were with him. We didn’t know if you were alive.” It took months in the chaos, the postal system had completely broken down. It took months for them to get word that Machiavelli was still alive. They didn’t know whether he had been caught up in the conspiracy. He easily could’ve been on a list of names of people the conspirators intended to recruit, and been gone. So his wife and his loved ones back at home, his children, had to wait months to find out whether he too had been slaughtered. It felt to them like a miracle that he hadn’t.

But it meant that he watched these incredible deeds: you encounter them, you forgive them, you renew vows of amity—sacred vows taken in the cathedral—and then you slaughter them at dinner, violating the laws of hospitality. Dante would say if you do that, you’ve committed such a grave sin that you’re not just regular damned. A devil comes up out of Hell and takes your soul out of your body and inhabits you. You’re actually already in Hell even though your body is still alive on Earth, because that’s how heinous a sin this is. And yet, it works, and all the rest of Valentino’s men are more loyal to him afterward than ever before, and won’t even whisper to each other about dissatisfaction, because even the faintest whiff of conspiracy might result in death.

So why does Valentino’s kingdom, for which he did everything right, ultimately fall apart? Because he happens to eat the same thing that gives him food poisoning as his father, and happens to be ill at the wrong moment. Also the puppet that he manages to get in power, Pius III, dies too fast, and then he’s outmaneuvered by Julius. If all those things hadn’t gone wrong in a row, the kingdom would’ve stood, and indeed, he would’ve conquered Florence.

Machiavelli is constantly reminding us that, yes, we have all of these things we can try to do. We can remember it’s better to be feared than loved. We can remember not to be hated. We have power over, at maximum, half of what causes outcomes. The other half is always going to be fortune. We look at Machiavelli. We know he’s the origin of utilitarian thought, and that he says we need to evaluate people’s deeds based on outcome. But he doesn’t just say we need to evaluate their deeds based on outcome. He says we need to evaluate their deeds based on what the most probable outcome was before fortune intervened.

So he says people look at Valentino Borgia and say, “But the Borgias fell. They were feared, and then they were hated, and then they fell, and then their enemies took power and chiseled their coats of arms off of every surface in Rome, so that to this day you’re walking through Rome and you sit down at a pizzeria and there’s a weird scar on the wall, and that scar is where the Borgia bull is no more.” People want to make the moral of that be, “Don’t do what the Borgias did. They fell.” Machiavelli’s like, “No, they did not fall because of their choices. They fell because half of what happens in the world is never in our control. You can do everything right, and it’s out of your control. But we have to evaluate what would have happened, and therefore we should imitate them, because everything they did was right.”

Dwarkesh Patel

I think one misconception of Machiavelli that I had, because I had not read these books before, is that he says the means don’t matter, the end matters. There’s a virtue ethic sense in which maybe he doesn’t think the means matter, but he is way more concerned about the means than I would’ve naively thought. He thinks the means are incredibly important, because the means by which you achieve power determine how stable and how fruitful that power will be.

In the context of military conflicts, he says if you achieve some power with the help of mercenaries or with the help of great powers—people who become stronger than you as a result of you achieving power—that is a very precarious spot to be in. But speaking of Julius, he makes another point that if you achieve your power by lying, by breaking oaths, by being unfaithful, that this is okay, because his view is that people will forget that you are not faithful. They will just take you at your word the next time they encounter you.

It’s actually a very interesting meditation on by what means you can achieve power that will make that power stable versus not. The fact that he thinks breaking your word is totally fine…

Ada Palmer

It’s even subtler than that. Because if you are someone who breaks your word and you break it this way, it’ll bite you in the ass. If you break it these ways, it’ll be okay.

He also does analysis of figures like Savonarola, who would make prophecies and promises, and then some of them would happen and some of them wouldn’t. He would then make new ones and correct what he said yesterday. He handled his manipulation and untruths badly, in Machiavelli’s analysis, in a way that did turn people against him and make him lose power. Partly because Savonarola, as a religious demagogue, the core of his power was people believing that he was divinely inspired and that he wouldn’t make mistakes and wouldn’t err. So his power base was fragile vis-à-vis untruth. For him, because of the specific shape of his power and then the specific way he handled his contradictions, that did hurt him.

Whereas if it’s somebody like Cesare Borgia, who will make an alliance and work with that ally for a while and then betray them—because meanwhile he was such an effective conqueror and he was so scary and everyone was so afraid of him—even when he would betray an ally, his other allies would say, “I’ve got to be more faithful to him so that the next person to be betrayed isn’t me, and try to work hard to be in the good graces of the prince so that I’m not next”, as opposed to turning on him, because he was so scary.

Savonarola was not scary. Savonarola was charismatic and persuasive and had one of these voices that made crowds thrill and women swoon. Decades later, when people asked Michelangelo what Savonarola had been like—when Savonarola had been dead for decades—Michelangelo’s answer was, “I still hear his voice.” He had one of those charismatic presences. That wasn’t enough when he started flip-flopping on policy and truth. Whereas Valentino was so scary that he could betray his top general and seize his lands and overthrow his city, and all of his other generals would say, “Better step even further into line.”

So it’s not just that lying is okay, it’s that lying is sometimes okay if you check these other boxes, and it’s not if you don’t. So this is even more reinforcement of how much he zooms in on the means. If you do A and B, you’re okay, but if you do A and C, you’re not. He’s looking at the minutiae of different ways you can wield power and different reasons people can have to follow you. If you’re a prince who’s decided to invest in being loved, you have to keep it up, or cultivate being feared alongside it. If you’ve invested heavily in being feared, there are things you can then do that you can’t do if you’re a prince whose power depends on being loved.

Dwarkesh Patel

This actually gets to the famous quote in The Prince, “It is better to be feared than loved.” What he’s getting at there is, I think, that he’s very cynical about people’s nature. If people make you a promise, they’ll just go back on it. If your power base depends on people’s promises and loyalties, as soon as your rule seems to be tattering, they’ll go back on it. Whereas if your rule depends on people having the expectation that if they break their oath to you, they’ll be punished, that’s much more stable.

He basically thinks people will act as badly as they are allowed to, whether they’re the tyrant or the people or the nobles. This goes back to the thing in The Discourses. His whole justification of checks and balances is not dissimilar to the founders of the US and their reason for wanting checks and balances and wanting to put different factions against each other. He’s just cynical and thinks people will act as badly as you allow them to.

Ada Palmer

On that topic, Machiavelli is the first person that we have ever in the European tradition to suggest that it could be viable for there to be more than one political party in a state at the same time, and that they would compete against each other and vent the society’s tension through competition and vie to try to dominate an election and then the next one. This is what we’re used to, but this is innovative in Machiavelli. He talks about how competition within a city, if the parties are kind of stable—he’s observing Siena as one of the examples of this—can vent local tensions and allow interior adjustments of who has power, and be stable. I’m going to come back to interior adjustments of who has power in a second.

The standard attitude toward political parties is that if there are two political parties in a polity, it will not be stable until one of those political parties is dead, and their heads have been cut off and put on spikes, and their houses have been burned down and paved over. That has been Florence’s solution to political parties before. Florence massacred its Ghibellines and killed all of them, and rubbed salt into the earth where the houses used to be so nothing can grow there. Nothing still grows there. Then when the Black Guelphs and the White Guelphs split into two sub-parties, they immediately started slaughtering each other as well. The standard was that one party must wipe out the other party for there to be stability. There are comparatively few examples, although Florence’s neighbor Siena is one, where political parties managed to not only coexist side by side but be politically helpful.

Dwarkesh Patel

One element of governance, or of being a good prince at the time, that I didn’t appreciate but Machiavelli makes a huge point of, is how formidable and reputable people consider you to be. That’s relevant both for preventing others from invading you and for extracting concessions from other people. In his diplomatic career, he is sent out to a bunch of different foreign polities to basically determine, “Hey, is this a serious person?” So Maximilian is trying to extract a bribe from Florence to not invade it on its way down to Italy. Florence says, “Go check out if this guy’s for real.” He has to make some judgment about this person.

Ada Palmer

It’s useful to remember, Florence has paid these bribes a lot. Florence’s tactic is: if someone is invading the area, can we bribe them? Because paying somebody to not attack you is a much more surefire thing than preparing to actually fight against them. Your family’s lands get trampled by soldiers. You suffer economically. So it’s an old Florentine tactic. It’s not a new thing that Maximilian is threatening to invade Italy and trying to extract a bribe.

Florence basically every year is like, “Okay, who do we need to bribe this year to not invade us? Here’s this year’s bribing-a-king budget. To whom does it go? Is Maximilian a serious threat, or are we saving this money in case there’s a threat from somebody more serious like the King of Naples or the King of France or Milan or the Venetians?”

Dwarkesh Patel

At this time, the pope is not just a spiritual leader but a temporal power.

Ada Palmer

Very much so.

Dwarkesh Patel

He and his son are literally fighting wars against other Catholics, but the other Catholics are fighting them back. What does it mean to be a Catholic who is fighting a war against the pope?

Ada Palmer

Here is where geographic proximity is everything. If you’re far from Rome—when you’re Denmark or Iceland, and the pope is all the way over there—the way you interact with him is that occasionally an incredibly impressive papal legate will visit. There’ll be vast pomp and circumstance, and the city will rename a street in honor of the fact that somebody sent by the pope has visited. He has this great power to say yes or no to petitions, and different countries have been trying to petition for specific things for ages, and the pope’s legate is here to interview the emperor, to judge whether the queen can be queen or not, and it feels like a big deal, and the pope is very abstract.

It’s easy to have a lot of respect for that pope, because what do you see that pope do? You see that pope in pomp and circumstance. You see that pope make judgments about fates of popes and kings. You see that pope put out papal bulls and edicts that give theological answers to questions. You see that pope exercise judgment of life or death over people at a distance. He’s very abstract, and the difference between one pope and the next pope is kind of small from your perspective. You don’t see their policy differences.

If you’re in Italy, the pope is that asshole who went to college with your brother and beat him up when they were at college, and then was drunken and irresponsible at middle age, and you’ve been negotiating with him in these other jobs, and you know this jerk. You know his family. You know the other jerks who are also competitors for this. You’re allied with him, you’re not allied with him. His ancestors are allies of yours or not allies of yours. He’s a specific dude.

You’re much more likely to judge a pope based on, “He’s that guy.” This is not Pope Julius II. This is Giuliano della Rovere, and I judge him based on his uncle who put him in power, and the actions of his friends, and the actions of the city he’s from. You know all of his dirty laundry. You are subject to the fact that when he moves into power, everyone who’s related to him is going to get promoted within Italy, and everybody who’s not is going to get removed from Italy. So it’s much easier for an Italian to see this pope, and it’s actually quite hard to see the papacy. That’s how you have these fascinating wars where even the cities that are hereditarily incredibly loyal to the papacy will sometimes be fighting a war against the papacy.

All Italy is divided into these two factions, the Guelphs and Ghibellines. Theoretically, what these two factions mean is that Guelph powers, Guelph families, Guelph cities believe the correct successor to the Roman emperors is the pope. The pope is the emperor. He has the right to be the ruler of Italy, and indeed of everything that was once Rome’s. He is the ultimate political and military power, and he is the rightful and only rightful overlord of Italy.

The Ghibellines believe that in 800 AD, when Charlemagne conquered a bunch of stuff and made the empire that we now refer to as the Holy Roman Empire, when the Pope crowned Charlemagne, he delegated the political and military side of his authority to that emperor, and made himself the spiritual authority, but the emperor the political and temporal authority. Therefore, the rightful ruler of Italy is the emperor, the successors of Charlemagne.

These are the two factions for which these parties fought originally, 300 years ago. These days, what these factions actually mean is, “Those jerks murdered Uncle Tybalt, and we will never forgive them.” They are the team that is our enemy, and we are this team. They are that team, and we hate them. We want to crush them because they want to crush us.

This means that sometimes a pope will be elected who’s from a hereditarily Ghibelline family, and the pope will start promoting people from the anti-papal faction, and the pro-papal faction will unite against the pope. It makes no rational sense until we remember that they are serving the pope abstractly. So you get multiple situations where there’s a war between Rome and Florence over the fact that Florence wants to defend papal authority in papal lands against the pope itself, because that individual pope was from the anti-papal faction.

Dwarkesh Patel

Do they not believe that he is the vicar of Christ on Earth? It makes sense in a normal political state for you to think, “I believe in America, but I don’t like the president,” or something. But isn’t the pope supposed to be…?

Ada Palmer

Yes and no. Again, when you’re far away, yes. When you’re close up, you know too much of the dirty laundry of these people. So let me use a fun example: the most passive-aggressive letter ever written in the entire history of time, in my opinion.

There’s a type of ceremony that happens when a new pope is elected, which is the giving of oaths of obedience. A major ambassador from every polity in Christendom comes to Rome. They wait in line for a long time, and then they give a long-winded speech about how great the monarch is that they’re there to represent, and how vast his power is, blah, blah, blah, and how pious he is, and how glad he is, Your Holiness, that you’re the pope now. Congratulations on behalf of my wonderful king.

And you’re supposed to send the highest-status possible person who can leave your polity without it falling down. You might send a younger son of the king. You might send a lord chancellor. In the case of Florence, you’re going to send the most prominent citizen you can.

So when Pope Sixtus was elected, it was Lorenzo de’ Medici himself—not the dedicatee of The Prince, the grandfather and namesake of the dedicatee of The Prince—who went to deliver this oration of obedience, which means literally prostrating yourself in front of the pope, literally kissing his feet, and giving this oath. Lorenzo did this for Pope Sixtus, with whom he was negotiating to try desperately to get a cardinalship for his brother. Pope Sixtus instead organized the Pazzi conspiracy to try to butcher the Medici family, killed Lorenzo’s brother, killed a number of his allies as well, and attempted to have a coup to take over Florence.

Then the next pope was elected after Sixtus, Pope Innocent, who was as everyone knew, a puppet of the same faction that Sixtus was from. So we go from this very dangerous pope who had tried to wipe out Lorenzo’s family to a puppet of the same faction. Lorenzo sent his son, instead of himself, to give this oath. He had his son deliver the message, apologizing to His Holiness that, “I could not come myself, but the last time this duty fell upon me, I had a brother upon whom I could leave the burden of the state in my absence. Since now I have no brother, I cannot come in person.” It’s a very respectful letter, but it’s also very overt about the fact that he does not trust and will not again trust this faction.

So they negotiate very carefully how to deal with the fact that the popes have this great spiritual power, but sometimes the popes are acting as horrifically selfish warlords. That’s also something which has worsened over time, and it’s important for us to remember that the papacy becomes gradually more corrupted over time. This is because with every generation, more people leave donations of wealth to the Church. A widow who has no son and has property decides to piously leave this to a monastery. The Church gets wealthier and wealthier. As the Church gets wealthier, with wealth comes power. More and more power is in the state. This makes a stronger and stronger incentive for every ambitious family to send their second son into the Church.

And this goes all the way down. We have personal letters of Machiavelli writing to and from relatives of his, where they’re debating the correct-sized bribe to offer to buy a priesthood for his little brother, Totto. They don’t want to offer too big a bribe, because it would impoverish the family. They don’t want to offer too small a bribe. They’ve heard that another family that’s after this priesthood offered an extra big bribe. That’s kind of not fair. How do they respond to being out-bribed? They just write about this as the most everyday, normal thing in the world. This is a wealthy merchant-prince-level family. They are in the top 5% of wealth and power in Florence, but not in the top 1%. But for them, too, it’s normal to talk about paying a bribe to get a priesthood. That’s just how it works.

Every generation sees the Church get wealthier and have more power. Therefore the incentives to corrupt it are even greater. It even becomes a kind of prisoner’s dilemma system. If you’re the duke and you don’t manipulate the papacy, if you don’t bribe the pope, if you don’t work hard to get your brother to be bishop, and your enemies do, you’re screwed. So you even see it as defensive: “I must manipulate the Church. It’s the only way my people will be safe. If I don’t manipulate the Church, my enemies may manipulate the Church, and then there’s danger.”

This happens all the way up to the scale of kings, where popes can make your enemy the most powerful bishop in your kingdom or can deny you the right to marry, because inevitably the person you want to marry is a cousin, and you’re going to need a special dispensation to marry them. The pope can prevent that and mess with your marriage alliances. You need the pope very desperately if you’re a king. You also need the pope all the way down. That means bribes and other kinds of incentives make the papacy more corrupt with each generation.

So the papacy is worse in everyone’s lived experience than it used to be even a few popes ago. You see every generation for 100 years say, “Popes are much worse now than popes used to be when I was young.” Everybody says that. Dante says that in 1300. Machiavelli’s grandparents’ generation is saying that in 1400. Machiavelli is saying that in 1500. In everybody’s lived experience, the popes are getting more secular, more military, and more corrupt over time. It’s a gradual accumulation, and it comes to a peak, as such things do, triggering the reformation, when it becomes so bad that there has to be a massive move against it.

Machiavelli, in an interesting way, anticipates this, because Machiavelli says, “All institutions are gradually corrupted and need to be reformed and returned to their foundations, or they will collapse under the weight of their corruption.” He thinks that the papacy has been undergoing this, and that Christianity has been undergoing this. And that, if not for the fact that St. Francis of Assisi—and also to some extent St. Dominic a couple of centuries before his time—reformed the Church and brought in a lot more popular support, Christianity would already have cracked under the weight of its own corruption 200 years before, and that it will need such a restoration again—as any institution needs, as city governments need, as republics need—as corruption accumulates over time.

Dwarkesh Patel

One big way in which our world is different from 500 years ago is the focus on patronage and it being the basis of political power. It was much more prominent, right? So that’s something that would be interesting to understand.

Ada Palmer

It’s not just that it was more prominent, but that it was the fundamental glue of the society, as opposed to one of several glues of the society. Patronage, which was also familial and therefore entangled with nepotism, was so fundamental. For example, when Alessandro Farnese was elected Pope Paul III in the middle of the 1500s, he didn’t corruptly make one of his kinsmen commander of the papal armies. He instead appointed a really competent, experienced general instead of his own not very competent, illegitimate son.

And there were riots in Rome. “Your Holiness, the people demand more nepotism. You must appoint your illegitimate son to command your armies, because your illegitimate son will never betray you, and we will know we can trust the papal armies not to turn on Rome if the Pope’s son is the commander. We don’t know that about this other commander. He might turn against Your Holiness. There might be a rift between the Pope and the papal armies if he’s not somebody who rises and falls with you the way your son does. Therefore, by popular demand, the people want more nepotism, because the system depends on it.”

That’s how you see how the system depends on it. There are levels of trust that the patronage system creates, because it involves multi-generational entanglement of families. If these families rise, they rise together. If they fall, they fall together. That creates levels of trust that can sustain things like this world where the oath of a soldier is to his commander, not to the polity that he serves.

In modernity, we realize another solution to that. The oath of the soldier is not to the commander, but to the Constitution, or to the country, or to the people. But in this period, the oath of the soldier is to the commander, mostly because communications are so slow that the commander has to be able to give speedy field commands. But it means you’re creating an army and handing it to a man. If you cannot trust that man, then the people will be terrified that there could be a rift between Rome and its own armies, or between Rome and its treasurer, or between Rome and its other allies. Patronage is the glue that makes things work all the way down.

All the way down to the level of, if you need a defense attorney, that’s done through patronage. The outcomes of trials are a really great way to see patronage. We’re all familiar with the fact that law codes in the Middle Ages are really cruel: death for everything. Death for theft, death for adultery, death for homosexuality, death for setting fire to the prince’s beehive. Whatever it is, that’s the sentence on the books. You look at the actual trial records and maybe one in 100 convictions for that crime actually ends in a capital sentence. Almost all of the other ones end in a fine or a public flogging, but not in the sentence that’s on the books. We say, “Why and how is that happening?” Patronage is the answer.

So say it’s the Middle Ages or the Renaissance, and you’re a carpenter, and your teenage son gets drunk and punches somebody in a brawl, breaks the guy’s nose in a way that makes him die. He accidentally kills a guy in a drunken brawl and your son is now on trial for murder. You’re a carpenter. You have worked for the rich family whose family carpenter you are. Let’s say it’s the Medici family. Whenever they need new pews for the family church or new furniture or repairs for the family gates, they go to you.

So you go to them and say, “My son is in danger. He’s on trial. Please put in a good word.” Your patron has the ability to influence the judges, and they will put in a good word for you, and you will get a lighter sentence. This is an ancestor of having a character witness to say, “So-and-so is such a good person, they should have the milder punishment, not the more severe one.”

The norm is: you’re accused of a severe crime, you’re put on trial for your life, your patron intervenes, and you get a lighter sentence. This is how justice is supposed to work. This is a very severe line that changes in the 18th century with the Enlightenment. Because we now think of proportional justice: the sentence for the crime should be this, and ideal justice is that everyone who is guilty of the crime gets that sentence. That is fair. It doesn’t matter who you know. It doesn’t matter whether you’re rich or poor. The sentence should be the same. This is the ideal of Enlightenment justice.

The ideal of this period’s justice, which is much more shaped by Christianity, is that the purpose of the trial is the spiritual interior correction of the soul of the sinner. Therefore the ideal outcome is for them to fear for their life. They’re before a terrifying judge who is the earthly representation of God. They know that they’re guilty, and they deserve to be thrown into the pits of hell. But miraculously, they are given grace, and they are pardoned. The process of being put on trial, fearing for your life, begging to the patron, and then receiving mercy is supposed to be an earthly preview of the process your soul will undergo when you are before divine judgment. Therefore it should make you come out the other end a good person.

The goal of the justice system is the spiritual improvement of the sinner and the hope that they will come out the other end better and more likely to go to heaven. Even when people are being sentenced to death, there are religious organizations who sit with them overnight, having a final prayer group, and walk with them to the gallows, holding their hand and holding a painting of the Virgin Mary in front of their face, so that to the very last moment, the person who’s about to be executed is thinking about heaven. The ideal outcome of the execution is that the soul goes to heaven.

So the whole structure of the justice system expects the intervention of a patron, who represents the intervention of a patron saint, persuading the judge, who is God, to give you mercy. So when we see 100 trials end with 99 where the person paid a small fine, and one where the person was executed, what that actually means is that in 99, their patron stepped in. Somebody persuaded somebody who put in a good word, and they got the light sentence. In one, that person had fallen out of the patronage network. That person had angered their boss, their protector. That’s why it went all the way to being a capital offense.

Probably a lot of people listening are familiar with Giordano Bruno, very famous as a martyr for science because he was burnt at the stake by the Inquisition. Fewer people know that that was not his first Inquisition trial. He was investigated a number of times by the Inquisition for doing various radical forms of thought. The earlier trials had the usual outcome for that kind of trial. He had a patron, there were rich people that he worked with or for, the university was hosting him. They put in a good word. He’s fine. The Inquisition tells him, “Be good,” and things continue as they are.

That time, he had angered the person he worked for. He pissed off his patron. It’s his patron who turns him into the Inquisition and says, “This guy is a charlatan. He promised he could teach me these things, and he can’t. I don’t trust him. He’s no good. Throw the book at him.” The reason that trial goes all the way to a capital sentence is that he doesn’t have a patron. He’s the 99th case that time.

If he had had a patron protecting him, despite how radical his stuff was, he would’ve been okay. We see that in the trial of Giovanni Pico della Mirandola, who was candidly substantially more radical than Giordano Bruno. But when Pico is on trial, Lorenzo de’ Medici and other powerful people really care about Pico, and they pull out all the stops. Lorenzo talks to his brother-in-law, who’s an Orsini. The Orsini have enormous influence in Rome. They get permission for Pico to be let go and sent home to Lorenzo to sort of live under house arrest, under Lorenzo’s promise that he’ll be good from then on.

Or you have Marsilio Ficino who is this radical Platonist who publishes a book on how to project your soul outside of time and summon angels, arguing for the existence of reincarnation, and is very clearly being extremely theologically weird. This is the man who wrote the best letter of recommendation ever written in the history of time when he was recommending a young scholar for a job with the King of Hungary. He writes in the recommendation letter, “This young man is the reincarnation of St. Thomas Aquinas, so you should give him a job.” Now, that is a letter of recommendation. But you think, “The reincarnation of St. Thomas Aquinas, huh?”

And the Inquisition comes knocking on Ficino’s door and is like, “Hmm, reincarnation?” Ficino’s like, “Oh, no. Help. Talk to Lorenzo.” Lorenzo talks to his brother-in-law, Cardinal Orsini. Cardinal Orsini shuts it down, and Ficino is told, “Maybe lay off talking quite so overtly about the reincarnation.” Ficino says, “Yes, of course, and I will only teach very pious people how to summon angels and project their souls out of their bodies. I promise I won’t teach it to anybody who will use these powers irresponsibly.” The Inquisition is like, “Okay,” and goes home. Because patronage kicked in.

Patronage is the glue that makes everything work. You can’t even stay in a hotel or buy an apple—I’m not kidding—without a patron. You arrive at a city. Nobody knows you. You’re a stranger. What you have is a letter of recommendation from your patron who’s friends with some important person there. You present that at the hotel. That’s why they let you stay.

Dwarkesh Patel

To tie together a couple of threads you were talking about, The Prince is painting a picture of regimes being incredibly unstable. You have to worry about foreign powers. You have to worry about rival factions within your own country. You have to worry about mercenaries. You have to worry about lots of different things. So any given regime is very unstable. So what had to happen for things to get more stable? We’re talking about a couple of the ways in which people owed their loyalties not to the regime, but to others within the regime, which created instability.

In Discourses on Livy, Machiavelli talks about how one of the reasons the Roman Empire’s fall was instigated is that these generals were months away on the frontier fighting these wars because the empire was so big, they had to amass for periods of years—or in some cases, for Caesar of course, decades—the command of so many men who have for decades just been listening to this guy tell them what to do, who to fight next. This is the person they’re loyal to. As opposed to, say, if the consuls could be giving dictates every single day, then the loyalty could be to the regime in Rome.

Same with patronage, if there’s not a system of deterministic justice like we have in the modern world today. A lot of The Prince and Discourses on Livy is dedicated to: how do you make sure that a family is not pissed off that their son got killed and it wasn’t avenged? If you just have a reliable criminal justice system, that problem goes away. It’s the same with the welfare state and getting rid of the patronage system. If you don’t have to rely on this family, then this disintermediates them, and the state can have your loyalty.

So it’s interesting to connect all these threads together—communication time, impartial justice system, impartial welfare state—as being what is required for the regime to have enough legitimacy and then, as a result, enough stability to have modern nation states.

Ada Palmer

Yeah. One thing that everyone is surprised by is that when Cesare Borgia—Valentino is much more what he’s called in the period—conquers these cities in Central Italy, he goes in, and he massacres the ruling family. He works hard to kill every member of them that he can so that there isn’t a potential rival claimant to come displace him. He implements neutral justice, because he and his cronies have no side in that city. They aren’t connected with one group of families against another. When they implement justice, they do so neutrally because they aren’t interested in the local backstory of factions.

As a result, to everyone’s surprise, he moves into a city, he massacres the rulers, he implements an authoritarian regime, and he’s incredibly popular and beloved by the people. Everyone says, “Why are they liking this man? He is a cruel, murdering tyrant.” The answer is, for the first time in generations, they have something close to fair justice.

Meaning, it used to be that there was one faction in power and another faction out of power. In our scenario where a carpenter’s son gets drunk and kills someone in a drunken brawl, if that carpenter’s son is the carpenter of the power that’s in power, then there will be no justice and no consequences for this murder. It’ll be maybe the smallest of fines. If that carpenter works for the families that are out of power, then throw the book at him. His son will be executed for that death. There will be no fair justice. The outcome of the sentence will be entirely who’s in power and out of power, not the fairness of the case.

But when both of those ruling families have been wiped out, and an outside power is here, and a homicide takes place, the neutral judge hears this neutrally and gives the same answer regardless of whose family’s carpenter that is. The people who have lived in generations of “justice for some and injustice for others,” suddenly having equitable justice, are delighted by this and find that wrongs are finally being punished. The people that they’ve resented and hated for so long who are in power are finally being punished for the crimes that they commit.

This makes Valentino’s conquering and violent regime incredibly popular with the everyday people of these cities, who are therefore willing to sign up for his armies and help defend his conquests and keep them in power and man his fortresses. So Machiavelli and others are startled by this. They had expected that if a conqueror moves in and massacres the rulers of a city, everyone in the city will hate and fear that conqueror. But if the conqueror is feared and not hated, because he wiped them out but then was fair toward the people, then it works.

Dwarkesh Patel

So why would it have been so bad if Valentino took over Florence and he had survived? He would’ve massacred maybe the ruling regime at the time, the republic, but I don’t know what Machiavelli is especially concerned about. Would the cultural treasures of Florence and everything have survived?

Ada Palmer

The cultural treasures of Florence would potentially have been okay. There’s two answers to that. One of them is that Machiavelli is very adamant that if you live such that there is somebody who can have you summarily executed—he can walk by you in the street and point at you and say, “Him, kill him,” and it happens—then you are not free. In his vocabulary in the text, if you live in a state where there is an arbitrary power who can have you put to death, you are a slave. If instead you live in a system where there must be a trial, and there must be a process, and this must be examined and public, if there is a system, then you have liberty. That system may be unfair. It may be biased. It may be, in Machiavelli’s case, the very system that tortured and exiled him. But there was a system. He considers that difference to be enormously important.

So if Valentino conquers Florence, it’s not going to be that system anymore. There will be a man who can walk down the street and point at a Florentine citizen and say, “Kill him,” and they will kill him. Will that tyrant be fair? Maybe. Will that tyrant exercise this power well? Perhaps. Will his successor be worse or better than him? We don’t know. We can’t predict. It’s a monarchy. It’s vulnerable to good successors and bad successors. But the people of Florence are not free if there exists a man who can say, “Execute him.”

That meant a lot to Machiavelli, and it meant a lot to the Florentine people. It’s kind of hard for us to see how few liberties and how little franchise they had and yet how much they cared. Florentines are constantly willing to go into the street and risk their lives flying the banner that says “Libertas” across it: liberty. The banner LIBERTAS is the coat of arms of the Signoria, the Senate, which is selected from the 1% super mega elite, tiny minority of the city that is eligible to be in government.

They aren’t rioting to defend their right to participate in the republic. They’re rioting to defend their boss’s boss’s right to be in the republic. Yet they care so deeply about it, and they consider it fundamentally different from the situation in which there is a man who can walk down the street and point at you and say, “Him, kill him.” That tradition of liberty means a lot and would be gone even if the most beneficent tyrant in the world took the city. So that’s half of the answer.

Dwarkesh Patel

Can I ask about that real quick? When  Lorenzo de’ Piero di Medici takes over, is he not that guy?

Ada Palmer

That’s the second half of the answer. There is a huge difference between when the conqueror is from your city, loves your city, and wants to take care of your city, and when the conqueror is from the outside. When the Medici take over Florence, they want Florence, and they want Florence to be Florence. They want all of its beauty and all of its treasures to still exist and be theirs. They would never consider razing important parts of it to the ground. They would never consider threatening the Florentines with, “We will destroy your city walls,” or, “We will destroy your cathedral if you rebel.” Any outsider would.

So Florence looks more like Florence under a Medici duke than Milan looks like Republican Milan under a Visconti or Sforza duke, or than Ferrara, which has no remnants of its republic, does under the dukes d’Este, who can do anything they want, including murderously gouging each other’s eyes out and the city will never take one step against them.

So Machiavelli is aware that if Florence has to fall, falling to the Medici is gentlest. It’s most volatile perhaps, because they aren’t going to be feared as much as an outside conqueror would be feared, but certainly gentlest. You preserve some important rights when you’re conquered from inside that you don’t when you’re conquered from outside.

Dwarkesh Patel

Obviously, we remember this period for producing all these great cultural artifacts, all these amazing buildings, all this art. Then we’re talking about the precariousness of the prince, the constant wars, how they’re literally fighting all the time. How is there this surplus that is available for all these different projects?

You write in your book about how the older Lorenzo de’ Medici spends what would be today, because of the expense of building libraries and buying books, $30 million to build a library to educate his grandsons. How is there all the surplus available for education and arts and so forth in a period where everybody’s fighting everybody, and if you lose a war, your city will get, if not razed, at least the ruling faction of it will get killed?

Ada Palmer

Half of that answer is finance is incredibly profitable. If you’re the banking center, the amount of money that is flowing in is staggering. Big Wool, the big industry for Florence, is also incredibly wealthy. In the same way that Henry Ford becomes incredibly rich, in a period when a suit of clothing is something you save up for like buying a car, and everybody needs one, you can get very rich that way.

So, there’s lots of money. But, do you remember how it’s often said that the biggest impact per dollar for US defense spending is the Fulbright Program? Because diplomacy is cheaper than war. Sending a bright-eyed, bushy-tailed young graduate student out to a country to enthuse about its culture and make connections and make everyone feel positive does a lot more to avoid conflict, and also get help in conflict, than the same amount of spending on the actual army does. Dollar for dollar, diplomacy is cheaper than war.

They’re using the art to do diplomacy. So in one sense, if you’re not doing the art, you would have to spend more on the war. It’s not that the art is being made from a surplus of the war. It’s, “Oh, no, we can’t afford enough armies to actually defend us against France. Even if we spent every penny we have on armies, it would not defend us against France. But we sure can spend it on painting fleur-de-lis all over our seat of government and creating beautiful, expensive gifts for the King of France, so that when the King of France comes, he will feel like we are friends and we are giving him all of this cultural output. If we fought him, we would lose. But if we play the culture victory game, that’s cheaper, and we can try to win.”

Dwarkesh Patel

We talked about this last time, the experience of what it must have been like for a French diplomat to arrive at Florence and look at these people he considered to be nothing, not even descended from the Caesars, and they’re producing all this stuff.

When one goes to visit Florence now, the interest is in part because these are historical artifacts, because somebody made them 500 years ago. But if you’re seeing them at the time, this would be something either you thought only the Romans could have done that we can’t do anymore, or something that even the Romans couldn’t have done.

Ada Palmer

Right. They’re high-tech then. They’re like when we look at an incredibly impressive skyscraper that’s taller and more precarious and amazing than any past skyscraper.

Dwarkesh Patel

I think that is an underrated aspect of what it must have been like to be a foreign power evaluating Florence at the time.

Ada Palmer

Yeah. We have to remind ourselves that these are high-tech achievements as well as historic achievements. Also that this is a period in which backwards is forwards. That is to say, this is not a period that, like us, thinks of the future as where potential is, and that humanity might get better and better over time. The potential of humanity is recapturing Rome. Backwards is forwards. If we can get more and more like that, that’ll be better. That’s what we aspire to.

They do debate: can we surpass the Romans? Can we make things even better than the Romans? But it’s an “if,” it’s a debate. It’s not a “definitely, of course”. For us, it’s “definitely, of course.” We’re moving forward. We’re trying to build bigger and more impressive things. Even people who are cynical about progress will say, “Yeah, we will be more powerful in the future. We’ll be able to do more. We may use it to stab ourselves in the foot, but we will be more powerful in the future.”

For them, it’s very much: will we ever be as powerful as the Romans were? We don’t know. We can debate it. We hope so. We aspire to it. Will there be another Pax Romana? Will there be another universal peace someday? Will we ever achieve that again?

So when we look at something like Florence’s cathedral or Florence’s neoclassical buildings, we look at it and we know they’re imitating the past. So we don’t think of it as cutting-edge technology. But for them, cutting-edge technology is imitating the past.

Dwarkesh Patel

We talked about last time how both Machiavelli and the other umanisti, in the different ways they understood virtue, were trying to emulate the virtues that made Rome originally great. How much are they going off of just these random myths that Livy or whoever would write down about something that supposedly happened, where Brutus killed his own sons, or who was that guy who put his hand in a fire to show that the Roman people will be loyal and you shouldn’t fight us?

You look at actual Roman history, and it’s incredibly fucked up. We were just talking before we started recording about the life of Claudius and the period of the emperors and so forth, and surely this must have been known to them that actually…

Ada Palmer

Part of it is they’re zooming in on different emperors. When we want to make an HBO drama, we don’t make it about the boring, competent emperors who just do a really good job. Our society might be better off if we did. But the dramatic emperors where there’s lots of stabbing and lots of orgies make for good television.

Everybody curates their history. Often when you’re writing the history of your own culture, you pick the heroes. You look at a middle school history textbook, it’s going to celebrate the heroes of that country. If it’s trying hard to be unbiased, it will also acknowledge the faults, but the heroes are going to be in there.

When they are trying to create a handbook of what was, what stands out for them is what’s different from their present. Their present has plenty of tyrants. Their present has plenty of orgies. Their present has plenty of massacres. Their present does not have 70 years of peace. So that’s what stands out as different.

I think for us, some of the orgies and massacres stand out more because we don’t have as many orgies and massacres now, or at least not publicly that we know about. When we do expose that our leaders have been involved in scandalous orgies, we get very upset about it. But to them, they read about all of this, and they read about the successes and the stability from Trajan, Hadrian, and Marcus Aurelius, and they say, “That is alien to us. That we haven’t had in so long. That is what we want to have again.”

Dwarkesh Patel

So much so that when Gibbon is writing The History of the Decline and Fall of the Roman Empire, in multiple volumes, in the late 18th century he says that there’s never been a better time for humanity than during the era of the five good emperors.

Ada Palmer

Yeah and to the degree that medieval Europe can’t cope with the idea that these good emperors were also pagans and are therefore in hell… It’s where you get this gorgeous legend that Pope Gregory the Great summoned the ghost of Trajan and baptized his ghost so that he could go to heaven, even though Trajan is a during-the-persecution-of-the-Christians emperor. But they just love him so much, they can’t handle the idea that he would be in hell despite being a great Caesar.

So in the medieval world, it is canon that Emperor Trajan was posthumously baptized so that he could go to heaven, because he’s such a good emperor. Dante centers him in Paradiso as the ideal Christian ruler. But he wasn’t Christian. He was persecuting the Christians. But medieval and Renaissance Europe are very good at having their cake and eating it too—in terms of getting to pick and choose the best parts of the pagan world and the best parts of the Christian world when constructing their imagined antiquity—to have both and celebrate both at once.

Dwarkesh Patel

Here’s something I’m confused about. Machiavelli makes a point of pointing out Cesare Borgia’s betrayals because of how remarkable they are. For example…

Ada Palmer

Ramiro d’Orco?

Dwarkesh Patel

Yes. Slaying the very deputy that he had tasked with being harsh—and as a result bringing peace to a region—for that harshness that he had delegated. Or inviting, as a gesture of goodwill, some people who are going to do a revolution against him, and then killing them all at the banquet.

But should we take the fact that he’s making a special point—”hey, take this kind of betrayal or this kind of deed as something you should consider doing”—as evidence that this was actually rare at the time? Maybe another way to ask the question is, to the extent that they are all Christians at the time, surely they really did believe they’re going to go to hell if they betray people or lie or break their oaths, right?

So just as you were saying a second ago about how capital punishment was actually less prominent at the time than we in retrospect think it to be, are these kinds of crazy political intrigues less common than these stories make them out to be?

Ada Palmer

Two halves to that answer, and I’ll do the second one first, the second one being about the religious one. They all believe in this religion that says, “If you do this, you’re going to go to hell,” and then they all do this. That’s something that this period really wrestles with.

Everybody is sinning and breaking their rules all the time: killing for honor, committing usury, lending money at interest. They’re all sinning all the time. They’re all doing these things that are against the rules all the time. People in the period do bring that up and say, “Hey, this is not okay.” This is one of the big focuses of Dante’s Commedia.

Dante in it says, “Look, when you do these things, you will go to hell for them”. He fills his hell with Florentines. There’s that wonderful line where he meets yet another group of Florentines, and he says, “Congratulations, Florence, a city famous in hell,” because he considers his Florentine peers to be particularly hypocritical.

As he goes through, we see Florentines especially in the sections for usury and for sodomy, but also heretics and unbelievers. All through he’s encountering his countrymen, including people he himself loves and respects. Because Dante is making this painful point of, “Guys, it says that if we do this, we go to hell. I’m going to make a book where that’s literally true.”

One of the chapters of Inferno that hits extra hard in his period and in ours is Canto V, where he’s encountering the lustful, and we see Paolo and Francesca. Paolo and Francesca is a story that was an incredibly popular love story at the time. There was a young, beautiful noblewoman who had an older, horrible husband. While he was away, there was also this wonderful, handsome, young nobleman who visited her, and they read the romantic stories about King Arthur and Guinevere and Lancelot. One thing led to another, and they committed adultery together. Then her husband came home and found them and murdered them both. Everyone loves this story. It is the ubiquitous love story. It’s their cultural equivalent to Romeo and Juliet, a touchstone story. People sing songs about it. Everyone knows this exciting love tragedy.

And he puts them in hell because they were guilty of adultery. It’s really shocking to everyone who has celebrated this love story. “No, if this is true, and this is our religion, then this is where they would be.” Dante is very stern and very strict and very unusual, and starts a lot of discussion of this question. “We’re breaking these rules all the time. Should we just take this more seriously than we have?” He says, “Repent, or you will all go to hell, my fellow citizens.” So they’re worried about that.

But another part of it is that Christianity as practiced then has much less of a focus on purity than the Christianity that especially America is used to, and also the Protestant-dominated parts of Europe. There was a big change in Christianity that comes in the course of the Reformation, primarily from Calvin, Calvinism, and then Puritanism, which has a greater focus on trying to live an unspotted and pure life. It’s the idea of, “we’re going to create a community of people who are all going to stick to the rules and live by them. And if you are a sinner and have broken these rules, you should be expelled from this community. You are impure, you are stained.”

That is not the way Christianity thinks in this period. The assumption is everybody sins all the time. There is no such thing as purity. Everybody sins every five minutes. Everybody is envious. Everybody is lustful. Everybody is slothful. Everybody will make these mistakes, and then you repent of them, and you feel sorry, and you do penance, and you make spiritual progress, and you are forgiven, and then you sin again. Everybody sins. St. Francis of Assisi sins. He had a big focus on himself as a sinner and was constantly self-flagellating despite being, in many ways, the most virtuous man in all of Europe, but stressing his own sin.

So one saint who’s super popular in the Renaissance who is not very popular today is St. Julian the Hospitaller, patron saint of murderers. He is the patron saint of murderers because his legend is an Oedipus-like legend. When he was born, he was cursed by a witch that when he grew up, he would slay his parents. He runs far away hoping that he will never encounter his parents and so not meet them. But eventually he feels homesick and comes home, and is tricked by the devil into slaughtering his parents, and he slaughters his parents. He spends the rest of his life trying to make up for it, going on pilgrimage, and then dedicating his life to running pilgrim hostels to help others be pilgrims. He is the patron saint for people who have committed murder and feel really sorry and need to live with it and repent of it.

That’s not the attitude we have toward murderers right now. Our cultural attitude toward murderers is, “That person is a murderer. They should be shunned. They should be locked in a box without the key, or they should be executed. They should be removed from society. There is no turning back from homicide.” But the Renaissance’s idea is sometimes you have to commit homicide, and then what’s important is that you feel sorry. You need to have a patron saint whose job it is to be a spiritual mentor for you, he too committed homicide. He committed a worse homicide than you did, because he killed his parents. If he went on a spiritual journey to recover from being a murderer, so can you.

There are dozens and dozens and dozens of icons of St. Julian all over Renaissance Florence. Everywhere you go and you see one, you’re like, “That was commissioned by somebody who committed a homicide and is trying to live with it.” This is a society that really thinks about sin as something you do, and then you pay for it afterward.

And people like Dante and Savonarola come to people and say, “No, this is not okay. You are perverting these things. No, you cannot put your family’s coat of arms all over the inside of a church, turning the church into an advertisement for your banking business when it should be a place of God. That’s inappropriate, and no, God will not forgive you for it.”

And society says, “Yeah, well, but God forgives maybe anything if we repent a lot.” So it’s a complicated, sophisticated hypocrisy that builds up a lot of apparatus to let the society’s actions be at odds with its religious precepts to that degree.

Dwarkesh Patel

I couldn’t get enough of Ada or of Machiavelli, and so there are a few more questions I wanted to ask you. Thanks for hopping on again.

Ada Palmer

Oh, my treat.

Dwarkesh Patel

We didn’t talk last time about the fact that Machiavelli was exiled. He’s writing these books in exile. We were talking about his diplomatic career, so maybe you can give a bit of context around how he ends up in exile, and what his plan is once he’s there.

Ada Palmer

Here we have to start with the fact that everybody who’s anybody in the intellectual tradition lives in exile for a while. Dante does. Voltaire does. Rousseau does. Thomas Hobbes does. Machiavelli does. More importantly, exile is a very common thing in Florence and doesn’t have the permanence that one expects. In Florence, exile means the people who are in charge of the regime distrust you right now. They want you out of the city, but they’re testing your loyalty. They’re testing whether you will stay true to them, and you’re told not, “Get out of the city,” like a Roman exile, but, “Go to a specific place. Go to London. Go to Bruges. Go here. Stay there, and we will send you instructions.”

You’re expected to act as a kind of unofficial official emissary for the government of Florence while in your exile. You’ll be asked to do diplomatic missions after a while. They’ll say, “Go talk to this person on our behalf,” or, “Go deliver this trusted letter.” If you’re good and you behave, then after some years of service to the republic, you’ll be recalled. So it’s a provisional exile. They pick a specific place to send you, and if you go and are good and do what they say, then after a while, they consider bringing you home. If you don’t–if you leave and you don’t stay where they said, if you run off to work for someone else—then you’re not allowed back in Florence anymore. You’re an exile at this point.

Machiavelli’s exile is unusual because they really don’t trust him. So they don’t send him to Bruges or London or Barcelona or the Germanies or any number of other places where he actually has political contacts. They send him to a middle-of-nowhere hamlet in the countryside outside of Florence in Tuscany, where there is nobody important and there is nothing to do. This isn’t a “go wait for instructions.” This is a “go rot and we’re testing whether you will faithfully stay and do basically nothing and be forbidden to talk to important people, be in isolation.”

When that exile is given, everybody expects that Machiavelli’s response will be, “Okay. They’re not giving me even a second chance. I’m going to run off and work for somebody else.” Because there are a jillion people in Europe who would love to employ a skillful classicist historian with military and diplomatic capacities who has political contacts in Rome and in France and has visited the court of the emperor. He could have worked for any number of cardinals. He could have gotten a very prestigious diplomatic job in any of a dozen courts. A Florentine historian especially is something that you absolutely want to hire to write a flattering history of your own family. For even a century before this, kings as far away as England had been trying to hire Florentine historians to come write about them.

So he could easily do this, and this is what is expected, and he doesn’t. Machiavelli says, “No. I’m going to stay, and I’m going to rot, and I’m going to write The Prince, which is my job application begging the new regime to bring me back and let me work for them and demonstrating my loyalty, and I’m going to send it to them and only them, them and my immediate friends. I’m not going to share it with anybody else.” Because Machiavelli is a patriot, and he will not serve any cause that is not his country.

No matter whether the pay at a royal court somewhere would be three times what he would ever get at home, that doesn’t matter to him. No matter whether this is the regime that just arrested, tortured, and exiled him despite him not having plotted against them, he wants to work for that. Because Machiavelli fundamentally is possibly one of the most patriotic patriots in Earth’s history. He will faithfully sit in the countryside and rot while begging to work for the people who ordered his torture, so long as they will recall him so that he can serve his country.

And this connects to the question we always ask about the target audience of The Prince, because his other work—his discourses, his histories, his comedic play—those were for public circulation. Those increased his fame. Those made important arguments. His history of Florence joined other important histories of Florence circulating, influencing the way people thought about politics. Not The Prince. The Prince is secret and proprietary, the secret sauce of how to maintain power.

He will not let any other power have that. It’s like a nuclear scientist with diplomatic secrets who is faithful to his country and will not sell out and let those secrets fall into other hands. Machiavelli knows that he has the beginnings of a new world of political science. He will only share that with the government of his country because he wants it to protect his country, and he will not serve any other cause.

This is why it’s so weirdly ironic to me that the reputation—the word “Machiavellian”—means “self-serving”, when Machiavelli himself is one of the most selfless men I’ve ever read about in the history of the Earth. He will give up and sacrifice career, diplomacy, fame, friends, the opportunity to even be in a city and have a nice day, to rot in the countryside to be faithful to his country. He would rather serve nothing and no one than give an hour of his time to advancing anything that is not Florence.

Dwarkesh Patel

You’re making the point that he is advocating a viciousness and a realism and a cynicism, but in service of protecting Florence, not in service of a generic prince of any generic principality.

Ada Palmer

Exactly, and he doesn’t let copies of it circulate to anybody but the rulers of Florence and his immediate scholarly, social, intimate circle of friends, people that he’s known for decades who are scholar peers who have discussed his ideas with him. That’s the audience of The Prince during his lifetime.

Dwarkesh Patel

Does he expect that at some point it will be more widely distributed? Is he writing in a way that suggests that? It is a literary masterpiece as well. I’ve only read, obviously, the translation, so I don’t know what it’s like in the original Italian. But somebody putting in that much literary effort into something that is just supposed to be a very pragmatic manual for a particular person seems a bit weird.

Ada Palmer

We have to remember this is a moment of transition from the manuscript to the print period, and also therefore an important moment of transition in what makes a written work important and how that written work is important to the career of someone who’s written it. It’s a normal thing in Machiavelli’s youth for a major important scholar like, say, Pontano, one of the greatest scholars of the previous generation, to be hired to write a handbook of princes that will exist in just one copy or three or four copies that are written for a specific prince.

For example, you have King Alfonso of Naples, the Spanish king who conquered Naples, Alfonso the Magnanimous, made famous for his vast patronage of arts and letters and for carefully cultivated personal anecdotes. There’s a moment when he was in the middle of fighting a war and a messenger rushed into his room, sweaty and covered with things, to interrupt the king’s morning time with his scholar friends discussing Plato. The king turned angrily on the messenger and said, “Get out. This is a place for men in togas, not for men in armor,” and refused to listen to the urgent message until he’d finished his hour of scholarly contemplation of the soul. As a result of which, he lost that battle but actually won the war. His reputation cultivated by anecdotes like that make him beloved.

He will pay a salary five times what the Republic of Florence will pay to hire somebody like Machiavelli. What does he hire them to do? He has a lot of children, princes and princesses, and he commissions a scholar to write a unique bespoke handbook of how to rule and use power for each of his children. These exist in manuscript only in one copy or three copies, and the addressee is the Duchess of Ferrara, who is a daughter of King Alfonso. That book is never intended to circulate. It’s intended to be private guidance for her and for her to perhaps pass on to her sons and daughters.

Meanwhile, the author’s fame is magnified by being told the special bespoke handbook of princes cultivated secretly for this important princess was written by so and so. That’s so cool, and letters circulate and let you know that it’s happening. In the same way that a scientist might become famous because we know he’s developing cool proprietary technology that only his government has, but we know that it’s happening, we have to think of these books as proprietary technology.

In that sense, it’s not an unusual thing to write a book with an audience of one, or an audience of one and her immediate circle. This is also one of the moments where the handbook of princes also means for women. The title of that book for the princess who becomes Duchess of Ferrara addresses her as a prince, because prince is a gender-neutral word at this point. It’s lexically masculine in terms of ending, the same way a table is feminine, but prince is used for men and women. Even Queen Elizabeth is Prince Elizabeth at this period of her life.

Dwarkesh Patel

That is so fascinating.

Ada Palmer

We have trouble wrapping our heads around the idea of writing a book for an audience of one. It’s just not what a book is to us.

Dwarkesh Patel

The funny thing is, I think we are entering a new era where that might be once again possible. It already is somewhat true, where at least half of the words I read on a given day are generated specifically for me and nobody else, because of AI. Obviously, AI is not capable of writing something which I think would be a literary masterpiece that everybody would want to read if they had access to it just yet. But eventually it will be. So it’s interesting to consider that as this knowledge progresses, it would bring us back to this era of bespoke scholars dedicated to a particular prince.

Ada Palmer

It’s important to remember that that never went away. Two halves of that. One, for ages it’s been true that half of the words we read every day are bespoke only for us, because they’re email. They’re letters. They’re the correspondence back and forth which has the audience of one, the addressee, and that’s the majority of what all of us read and write in our lives.

It’s also always been the case that in the halls of power, there are book-long things with an audience of one or an audience of five. There are historians and other scholars and scientists whose job is to provide that 100-page report on the history of Syria, to be given to a committee of Congress, where these nine senators or these nine congresspeople need the background on what’s happening so that they can understand a current events thing.

There are historian friends of mine who work for the Department of Defense Intelligence, who produce these book-length research projects with an audience of five or an audience of eight or an audience of a couple dozen, because it is the bespoke proprietary knowledge needed by the government at that moment. Sometimes it’s technological knowledge, but just as often it’s going to be historical knowledge of, “Here are the important rivers where military things are likely to happen,” says the historian who knows the history of this stuff.

Dwarkesh Patel

That actually brings up the question of how many such tracts through history, which are of the quality of The Prince—as original at their time as The Prince is and as wonderfully crafted and so on—have been lost to history? Maybe one way to answer this question or think about it is to talk about how The Prince itself went into mass publishing.

At some point in 1532, the Medici pope allows for its publishing, and then 27 years later, it is censored by that same papacy. So how does this book that Machiavelli himself did not want out in wide circulation end up in wide circulation and then stop ending up in wide circulation and then end up in wide circulation again?

Ada Palmer

It goes in and out and in and out, like a lot of important works. I’ll give the zoomed out answer and then the zoomed in answer to that question. It is often the case that a work which contains radically unusual ideas will drift along being not particularly zoomed in on by society and not widely read, until it hits a moment that the new questions being asked in that century or that decade are answered by something in that text. Then suddenly everyone will start reading it.

A different example of this, probably well known to the audience, because everyone here is a cool, smart, learned person, is Lucretius’ On the Nature of Things, De rerum natura, which is our best capsule of ancient atomism and the atoms and vacuum theory of matter. It’s written around the BC/AD turn, and drifts along being not very important for ages, until the 1600s when we’re getting the first ideas of germ theory of disease, very interested in new science. Suddenly it gets 30 print editions and is all over the place and influences science, and is even more influential in the 19th century when we’re interested in atoms and cells. So a book can exist for literally 2,000 years or close to it, and then suddenly answer the questions of that decade.

In that sense, The Prince will drift along and be not very important for a while. Why is it first published? It’s first published when Machiavelli’s still surviving relatives want fame for the family and fame for their beloved now dead kinsman. Here is a work of his that hasn’t been published yet. They ask for permission because this can spread his fame. It’s also dedicated to members of the Medici family, so the Medici are like, “Yeah, we get fame for publishing this thing too.” They don’t think as seriously about the power of its contents as its author did.

And so it’s one more book that can spread the fame both of the Machiavelli family and of the Medici family, and it goes around, and people are like, “Oh, that’s actually full of fairly scandalous ideas. Hmm.” That’s how it then ends up on the index as book censorship kicks up as a result of the printing press. Mini thesis: every time there’s a new information technology, there’s a subsequent wave of censorship to try to censor the new technology, and a bajillion books get banned all at once. Machiavelli’s is not a particularly prominent example among this. The index of banned books that contains his work carefully differentiates between the dangerous books by arch-heretics and the slightly dangerous books by meh people, and arch-heretics are in all caps.

I remember when I was first reading through one of these indexes, I was so excited to flip through and find Machiavelli, and there he was, not in all caps, and I was so angry. I was like, “What’s wrong with that? He should be in all caps.” But all the all caps people are Luther, Calvin, Zwingli, a bajillion Protestant theologians you’ve never heard of. All-caps arch-heretic status is reserved for Protestantism in this period. Machiavelli doesn’t catch on until later. So he’s censored in a wave of censoring everything, when there’s a big censorious wave, and then it diminishes and goes up and down.

The second zoomed-in half of that: if we say Lucretius becomes exciting when people want to know about the germ theory of disease, when does Machiavelli become exciting? Machiavelli becomes exciting first in the aftermath of the publication of Hobbes’s Leviathan, because Hobbes’s Leviathan hits European thought like a truck full of bricks. It has this incredibly persuasive, gorgeous reasoning that lands you on a terrifying vision of what humanity is and a terrifying vision of what God is that people find very scary, but also incredibly persuasive. It’s no exaggeration to say that in the aftermath of publishing Leviathan, there’s a 40-year period where the sole goal of Western European philosophy is coming up with a good way to refute Hobbes.

At that moment, they say, “Okay, Hobbes is using a lot of logics about politics and about history that sound like Machiavelli.” He’s doing these utilitarian consequentialist analyses of “if we do this, there’s that result”. He’s analyzing the origins of government as if there’s no divinity setting it up. He has this man in a state of nature inventing government instead of God from on high telling Adam, “Here is how you should organize the world.” So they say, “Okay, Hobbes is the monster. Hobbes is Leviathan the great, or the beast of Malmesbury,” as newspapers call him during his lifetime.

How do we refute the monster? Let’s look at the daddy monster that spawned the baby monster. If we can read Machiavelli and find holes in Machiavelli, maybe we can use those to refute Hobbes. So Machiavelli is suddenly useful not to people who sympathize with him, but to people who see him as an enemy and want to use him to try to defeat what to them is the greater enemy. So he surges in popularity at that point.

A different surge happens in the 19th century, and it’s not until the 19th century that Machiavelli’s Prince becomes a major global staple that you would put in a great book series. In the 19th century, in the aftermath of the Enlightenment and the Enlightenment’s revolutions—the American Republic, the French Republic, the transformations and democratic movements that are happening in lots of other governments—people want new ways to think about politics, and they want to think about politics in separation of church and state.

If you want to think about separation of church and state, which is a new Enlightenment-era value, what do you need? You need an apparatus for thinking about politics and ethics that doesn’t depend on God being part of it. The vast majority of political treatises available to humanity at that point have some sort of entanglement of religion with politics at their root, but Machiavelli doesn’t. Machiavelli is this early foundational “what if we think about government in a box without plugging into religion? What if we just think about government operating by itself and its earthly consequences?” It’s incredibly useful in the 19th century for developing a statecraft for separation of church and state.

It’s also useful for Italian nationalism to celebrate and claim, “Hey, we invented separation of church and state. Here’s Machiavelli, the first modern man. He’s our bid at ‘Italian culture invented modernity’ via Machiavelli.” At the same time, England is saying Francis Bacon is the first modern man because he invented the scientific method. At the same time, France is saying René Descartes was the first modern man because he invented logical reasoning and modern principles of logical deduction. There’s a competition in the 19th century, a nationalist one, of different countries that want to claim their cool thinker as the first modern man. Machiavelli becomes one of Italy’s big bids for the first modern man because he came up with separation of church and state. It’s a phrase that Machiavelli would not have recognized if you said it to him, but he would have thought about it for a long time, decided it was cool, and then written letters about it.

Dwarkesh Patel

Can I try out a counter-thesis just so you can dispel my confusion? One reason why Machiavelli might have gained a special significance in the 19th century is that now that you have these republics in the world, there’s a question of how you make sure that they are maintained. That is really the question that at least the first third of the Discourses is obsessed with. But one of the ways it says that you do this is by having a religion that people take into very strong consideration.

I think he says at some point early in the Discourses that more significant than Romulus in the founding of Rome was Numa, or whoever it was who was the prophet who gave the Roman gods and the Roman religion some legitimacy. Then it is this legitimacy and the fear of offending virtues, because you believe in some god that will punish you, that motivates people to act in a way that defends the republic.

He gives the example of Scipio after the battle in which Hannibal absolutely destroys the Roman armies. The people are about to flee Rome as a result because they think Hannibal’s coming. Scipio himself, with his sword, goes down and says, “Swear to our gods that you will stay and defend our homeland.” Just having them give the oath in that moment is enough to convince them, “Hannibal can’t be worse than the gods, so I have to stay here and defend our republic.”

It seems like he thinks that religion is super important to the legitimacy of the state.

Ada Palmer

Agreed. He’s thinking about it in a way parallel to the way late 18th-century, 19th-century figures are also thinking about it. We have to separate the institution of religion from the psychological effect of religion on the populace. The useful example here is Thomas Paine. We all know Thomas Paine’s “Common Sense”. Thomas Paine does a lot of thinking about the foundation of the institutions of the US. Thomas Paine is a deist and a radical. He has lots of treatises about how the most destructive force in the world is institutional religion. Whether it’s Catholicism or the Church of England, these institutions are giant, centuries-old or millennia-old conspiracies to control your mind and steal your money, and are incredibly pernicious to everything.

However, he says, religion is vital to citizenship because it is what makes people be good and is what makes people fear laws and want to obey the laws. So, says Thomas Paine, every country must have religion, and religious education must be mandatory in schools, but it doesn’t matter which religion. Thomas Paine advocated mandatory religion in total indifference to what religion it is, with the idea that fearing God and posthumous punishment is necessary to make a citizen, in a practical sense, willing to obey law.

Notice how that is Paine thinking in a utilitarian way about the psychological effects of religion being there. It’s very different from the older view that the state and a state religion are entangled with each other. The state promotes this state religion because it believes it to be true, and we’re going to have a Christian nationalist or Catholic nationalist or Roman paganism nationalist religion that advances X against others.

So Machiavelli is absolutely thinking about the psychological effects of a religion on the people. He has that wonderful analysis in the Discourses of the utility of Roman religion. He talks in one really striking and memorable passage about how Roman religion says that your ghost depends on being remembered. This is out of the Homeric tradition. Your ghost only retains its identity to the degree you are still remembered on Earth. If on Earth your name is forgotten, your ghost forgets its name.

This is not a “your ghost is okay forever,” like in Christianity. It is a “your ghost depends on being honored by your descendants on Earth.” When you are forgotten, your soul becomes an empty, mindless, wandering shade. Therefore, you have an incredibly strong incentive to be remembered by doing great deeds, especially sacrificing yourself for your country, because then your name will be honored for as long as your country lasts. Machiavelli says this is one of the big motivators that makes people sacrifice themselves for the state in ancient Rome, because then they’re guaranteeing their good afterlife.

While Christianity, he points out, says all that matters for a good afterlife is being pious and then ideally being martyred. You have no incentive to sacrifice yourself for your state. The safety of your afterlife is guaranteed by your interiority. This is going to encourage a citizen to sit in a box and be a monk, not to sign up for the military and defend his country. So, says Machiavelli, Roman religion was much better for patriotism and political stability than Christianity. But he says at the end of the chapter, “Christianity has the advantage of being true,” period, end of chapter. You’d say, “Hello, Machiavelli, we know that you had the mandatory subscript there.”

So think about Thomas Paine and Machiavelli in parallel. They’re thinking about the utility of religion for forming a citizen, but they’re not thinking about “this religion is true, we are doing God’s work, we need to craft our state to match the values of our religion,” which is what a theocrat would argue.

What you have is separation of church and state with the expectation that religiosity will be there. It will affect the people, it will affect the citizenry and their behavior. You need to think about it. You need to decide whether to cultivate it, but you need to think about it in the same neutral way you think about cultivating literacy skills or math skills in your citizenry. What skills do we want our citizenry to have for them to be well-informed citizens? What do we need? We need religion and we need good newspapers so that people are up on the news and can vote prudently. You are evaluating those things side by side from a utilitarian standpoint instead of “this religion is true, it is the obligation of our government to advance it, and our government expects to receive divine blessings if we advance the correct religion, and divine curses if we don’t.” It’s a radically different way of thinking about religion, while still recognizing it as a powerful factor affecting the psychology of the populace.

Dwarkesh Patel

That makes sense. Last episode we were talking about the psychological impact on scholarship of having books be so expensive and having to meditate on the same copies that are available in one library. Maybe Machiavelli’s the strongest example of this, where maybe through his life we’re seeing the impact of the printing press diffusing and making printing cheaper. But early on in his life, it’s still not been that long since Gutenberg came up with the first printing press. As a result—correct this story for me—his dad has to do months of drudge work indexing Livy in order to get a copy of Livy.

Ada Palmer

In the infancy of printing, books are scarce and few. For example, one of my favorite manuscripts ever that I’ve worked with is a copy of Lucretius in Machiavelli’s hand. He copied out the entire poem. This is in the Vatican library.

But what’s really neat is he copied the text from a printed copy. But as he copied it, he integrated into it corrections and improvements of errors in that one, taken from a manuscript copy, so that what he produced was better than either the printed version or the manuscript version. And then he made his marginal comments as he went.

But notice this is somebody who, even though print copies of this book exist, is so much in the manuscript world that he’s happy to spend months probably copying out and making his own custom improved version of this text that he can then work from, even though inevitably new print copies will come out in a few years that may have the very corrections that he’s working with. But he isn’t going to wait for that, and he’s not sure, so he makes his version. So he’s from this moment when print and manuscript are parallel technologies being used at the same time. The very people who are buying the first printed books are also producing manuscripts imitating those printed books and influenced by those printed books.

Dwarkesh Patel

I want to think about the impact that having this copy of Livy—which presumably is one of the very few books that young Machiavelli had access to—has on his intellectual development. We have this mode of scholarship at the time. Why does he spend two decades writing Discourses on Livy? Unlike us—where we can go through an audiobook a week or read our Kindle at night— he’s presumably just reading this book again and again and again, and is trying to connect it to the events he’s seeing in his own life on his 10th reread. So that I feel is very interesting psychologically in understanding how scholarship and intellectual thought must have been different at that time as compared to now.

Ada Palmer

Machiavelli can easily access other books by visiting friends, by asking to go to the library of his Medici patrons when he’s working for the Medici, of his Soderini patrons when he’s working for the Soderini. But that’s different from having it at home and being able to have it at your bedside and look at it at all hours and have this intimacy with it, and it’s your father’s copy and it’s your copy.

There’s another part of that though, and this is weird for modern people to understand. In the Renaissance, there is so much enthusiasm for antiquity. Antiquity is the cutting-edge thing. Antiquity is where it’s at. Antiquity is how we’re going to end the chaos of the previous world and have this new world where we’re basing everything on ancient Rome. There’s going to be peace. There’s going to be a golden age. It’s all coming from and imitating antiquity.

Therefore, if your book is a comment on an ancient, it is going to be way more popular and sell way better, and people will care more and think more of you than if your ideas are original. Nobody wants original ideas. Original ideas are out of vogue. Original ideas are dead. All ideas need to be from the ancients.

So a Renaissance scholar will bend over backwards to pretend that his beautiful original ideas are actually Livy or are actually Plato, or to couch them as a commentary on these things. That’s going to have a way bigger audience and be more popular and taken more seriously than if it’s original. So there are points where Giordano Bruno, in his commentaries on Aristotle, claims that Aristotle says things absolutely Aristotle does not, the opposite of what Aristotle says. But if he claims it’s Aristotle, people will take it more seriously.

The most extreme version of this is the brilliant and fascinating figure of Annius of Viterbo, who Tony Grafton has a great book about. Annius of Viterbo had this radical vision of how he wanted to rethink history and faked ancient texts. He made them up. He faked archaeological digs. He would secretly bury artifacts and then dig them up to great drama. And he forged antiquities to create this book that advanced his visionary original idea of ancient history, because if he pretended he got it from antiquity, people would take it more seriously than if it was an original book.

So Machiavelli’s Discourses on Livy are his big bid to have a popular, important, prestigious thing, because discourses on Livy are a bigger deal and more important and more interesting to everybody, and more likely to sell and get attention, than a Florentine history or a treatise of original thought on princes. Who wants that? That’s a very niche kind of thing. “Discourses on Livy, oh, exciting, we have to have this.”

This goes on for the next century. For example, huge amounts of radical political thought, including, believe it or not, commentaries on Machiavelli, happen in the footnotes in editions of Seneca and Livy. The text of Seneca will be a small square in the middle of the page, and then there’ll be these masses of footnotes and commentary. Huge original moments of political thought for the entirety of the 1600s are going on in wars, in footnotes, in editions of Seneca. But it’s not original thought. It’s all about Seneca, because that was what was in vogue then. The vogue of scholarly stuff shifts fast and is very interesting.

This is one of the weird reasons that Renaissance philosophy and Renaissance innovative thought—with the exception of a couple of oddball works like The Prince—gets pushed out of the history of philosophy, especially in the 19th century. Because when you get to the 19th century, the vogue is that everything has to be original. The philosopher’s ideas should be born like Athena, fully formed from the head of Zeus. The ideal philosopher lives in a cabin by the raging sea, contemplating in the wilderness. What they want is original treatises.

If you look at a 19th-century historian of philosophy, they’ll say, “in the Renaissance, there was almost no original thought.” There was Machiavelli’s Prince, and there was maybe a little bit of Giovanni Pico della Mirandola’s Oration on the Dignity of Man. (We have since proved it’s not an oration and it’s not about the dignity of man.) These things are the few lights in the darkness, and everything else in Renaissance philosophy is... Here’s a quote from a philosophy department person who actually said this to me: “The Renaissance is 200 years of people being wrong about Plato.”

A lot of people look at it, and you pick up Ficino, and he’s like, “Plato said these things.” You’re like, “No, Plato totally did not say those things at all. That’s absolute gibberish. No, Plato didn’t say that. What are you saying, Ficino?” If you think Ficino is what he says he is, a commentary on Plato, then indeed the Renaissance is 200 years of people being wrong about Plato, being wrong about Livy, being wrong about Aristotle. But if you realize that their style guide requires original thought to be presented in the form of a commentary on an ancient, what it is is 200 years of original thought using the ancients as the trellis up which the rose climbs in order to bloom.

When you restore that and recognize that in order to get at the real Renaissance, you need to not read the goofy outlier works like Machiavelli’s Prince, which present themselves as original—which is a weird thing to do—but read the commentaries on Livy. That’s where the original stuff is hidden, by pretending and claiming and sometimes sincerely convincing themselves that this is the secret coded true meaning of the ancient thing.

Like Ficino, the translator of Plato, definitely genuinely believes that all of the incredibly original cosmology and magic that he’s figured out is secretly coded in Plato, and he’s wrong. It’s not. It’s so adorable that he really, really believes it is. But what it is is an incredibly original vision of the universe that he got from reading Plato and thinking hard about it and combining it with other things. So he presents it as commentary on Plato, commentary on Dionysius the Areopagite.

That’s core to why this is a discourse on Livy, because a discourse on Livy is what a scholar is supposed to be doing. All the other things Machiavelli does are second-tier weird things for a scholar to be doing on the side of discourses on Livy.

Dwarkesh Patel

So adult Machiavelli is now seeing some of his work start to get mass-produced. What is his reaction to this?

Ada Palmer

At first excitement, but also horror, because Machiavelli is facing this fascinating moment in the history of being an author when printing has come into being, but there isn’t copyright yet. In the manuscript period, there’s no such thing as copyright. If you find out that someone has made a copy of your book, you say, “Oh, thank God. There’s another copy of my book.” That reduces the chances of it being completely destroyed in a fire. Making one copy of a book is six months of incredibly difficult labor. You’re just grateful every time a text is reproduced.

But when it comes to printing, then you have this experience, which Machiavelli is one of the first men ever to have, of finding out that a local printer is printing a work of his without ever having asked him, without ever having talked to him. He looks at it, and it’s full of typos and minor errors. He’s panicking in these letters and saying, “Oh, no, everyone’s going to think I’m a bad scholar. There are all these little mistakes in the text, and they aren’t me. They’re the compositor having made typos when setting it up, and no one will know that. They’ll blame me, and it’ll destroy my reputation. What do I do? There’s nothing I can do because there’s no legal process and no legal recourse. Printing has just come into being.”

It’s neat seeing him and friends writing to each other about, “What can I do about the fact that this printer has printed my book without asking me?” There is no law. There is no apparatus. There is no anything. His friends are like, “Well, write letters to everybody who matters and tell them that the typos aren’t you. That’s all I can suggest,” because they don’t have the idea of authorial copyright yet. It’s going to come in the next couple decades.

The weird thing is how this gets entangled with censorship. Copyright and censorship are born together in Machiavelli’s world, counterintuitively, from the Inquisition. When the Inquisition begins book censorship after 1515, which is during Machiavelli’s lifetime, the policy that the Catholic Church promulgates is: before you may print any text, you must take it to an authority licensed by the church to do this—meaning an inquisitor or a bishop—and they must read it and give permission for it to be printed. This is so that they can make sure there isn’t heresy in it. So all books are effectively born pre-banned until you get permission for them to be printed.

In return for this, you get a monopoly license, and only the printer that took the book through the process can print it. You may now use the actual Inquisition record of you having gone through censorship as the document to prove that you and only you have the right to print the book. Therefore you can sue people for plagiarizing it or printing an unauthorized edition. So the very first version of copyright is the Inquisition.

Places outside the Catholic world then, like England, look at this. There’s actually popular demand in England for censorship, when they say, “Hey, we need what the Inquisition does, because the Inquisition is so cool. They let printers have a monopoly on printing a book, and they let authors deny print permission. We need something like that.” The very first version of what is not yet copyright passed in England—which is of course the ancestor of what applies in all Commonwealth nations and in the US—was originally an imitation of the Inquisition. It was: you need a license before you can print your thing, and then in return you get a monopoly.

Later, when there was a freedom of the press push—and by later, I mean this is happening over the course of the first half of the 1600s, so about a century after Machiavelli’s death, it takes a century for all this to get ironed out—the first version of copyright law is them basically saying, “Okay, we’re going to keep the copyright half of censorship while getting rid of the censorship half of censorship, or changing the censorship half of censorship.”

But it’s all born out of the Inquisition having met this weird demand that you feel in Machiavelli, where he’s like, “They printed my book. They did a bad job. There’s nothing I can do. Help. Authorities, give me some way to do something about this.” So that’s where you can feel Machiavelli as one of the first generation that needs copyright, which will then be born in the aftermath.

Dwarkesh Patel

Fascinating. And what was the Inquisition’s incentive to enforce the author’s prerogative on the text?

Ada Palmer

Partly the Inquisition does it because that encourages authors to come to them. It makes people much more willing to collaborate with their process. But also, think of an individual Inquisitor as an individual person who lives in a place and needs to have relationships in that place, and needs to have an income, and who is not usually getting enough to live on from the Inquisition itself.

If you’re working for the Inquisition, you’re an officer of the Inquisition, you’re probably a Dominican monk. You get some support from the monastery, but you have reason to want money, and you have family, they want money. You’re as pragmatic and self-serving as any other average human. So the fact that people want to have this positive relationship with you, they might gift you some bottles of wine in return for you being extra generous in your reading of their text.

They also have to negotiate with authorities. The Inquisition wants us to think of it as very centralized and very monopolar—the Inquisition, the Vatican, it controls everything—which is completely untrue and is propagandistic. The Inquisition is overseen by a whole bunch of isolated guys who are in isolated towns, and it takes weeks or months to even communicate with the Vatican. They’re making their own decisions.

For the most part, they don’t have their own large amount of funding. They don’t have their own officers to jail people. They don’t have their own jails. They don’t have their own authority to arrest directly. They get all of those from the local government. They collaborate with the local government, which means if the local government likes them and is pleased by them, and is like, “Ooh, the Inquisition, I can use this to scapegoat my enemies,” then the local government will drown the Inquisition in funding and give them all the guards and all the incentives they could want.

So when we hear about the infamous Spanish Inquisition, which everyone was expecting me to mention, the Spanish Inquisition is infamous because Ferdinand and Isabella of Spain really want to scapegoat the Jewish and Muslim populations that they’re anxious about. So they throw money at their Inquisition and really cultivate and make it big. That’s coming from them. It’s not coming from Rome.

Meanwhile, if you’re in somewhere like Florence, where the duke—if it’s early Medician ducal Florence, right when this is happening—is a Medici, he’s in deep with the weird Ficinian Platonic soul projection magic people. He’s an intellectual radical descended from intellectual radicals. His court is full of intellectual radicals. Here you are, the Inquisitor, and you’re like, “Your Grace, can I arrest this guy?” He’s like, “No, that guy works for me. You can’t touch him.”

You can only arrest as many people as the duke will give you funding for, or the local republic will give you funding for. So you need to please the local government if you’re the Inquisitor. We have letters of Inquisitors complaining, “This is a really liberal duke. He’s protecting all of these heretics around him, and there’s nothing I can do about it because I depend on the local authority for my ability to do stuff.”

So this is a really bizarre comparison, but think of the Inquisition operating kind of like Doctors Without Borders. It’s not the government. It’s an international organization that’s set up to try to achieve a goal that it believes is beneficial in different places. But it’s only as strong as, or as weak as, the government’s willingness to collaborate with it. If the government collaborates with it, it can be enormously powerful in an area and do a lot.

If the government is hostile to it and starves it of resources and doesn’t let its people in and insists on pushing it out, then you can get bubbles where the Inquisition is nearly impotent. Every time they want to arrest someone, they have to go to the duke’s agents, and if the duke’s agents keep saying no, they can’t do anything. What this really creates is bubbles of privileged access, where if you’re in with the government, you can be as heretical as you like, and the Inquisition can’t touch you.

This is also a lot of how homosexuality operates at the time. If you are in the protection of a powerful person, they can prevent the Inquisition or other officers of the church from getting at you. They just won’t do it, and they’re more powerful than those agents are, so they can’t touch you.

Machiavelli was, I would say, very definitely solidly bisexual, in that this is a man who recreationally had boyfriends and girlfriends throughout his life that he writes to. We have homoerotic poetry. We have heterosexual poetry. He’s definitely very excited by both sexes. He has a lot of gay friends.

He and his gay friends are writing back and forth about how at this particular moment in Rome, one of the agents in charge of Rome’s enforcement is really cracking down on homosexuality. Therefore all of their gay scholar and artist friends are rushing to get jobs working for cardinals, because if you work for a cardinal, nobody can touch you. Almost all of their friends have succeeded in getting jobs working for cardinals, except for one. So he has resorted to hiring two female prostitutes to hang out with him all the time and make him seem straight, by having him hang out with sexy courtesans, to defend himself against charges of homosexuality.

Heresy and homosexuality operate very similarly in this period. They’re both forbidden by the same things and policed by the same structures. So if you work for the cardinal or you work for the duke, you can be doing very radical magic, radical philosophy, radical politics, radical sexuality, and nobody in authority can touch you because authority’s trumped by a higher authority that is protecting you. This is part of the patronage system.

Dwarkesh Patel

How does that come back to the copyright?

Ada Palmer

The way that comes back to copyright stuff is that the Inquisition needs to please local authorities in order to get to operate at all. So the Inquisition will therefore try to figure out things that will please local authorities. If a book is being presented for publication that has a recommendation letter at the beginning written by an important political figure, the Inquisition will push it through. When printing presses and authors say, “Hey, can we have this be a monopoly license?”, figures like Machiavelli realize we could ask for, “Hey, you’re giving us permission. Can you deny everyone else permission?” The Inquisition immediately realized, this is a great way to get publishers on our side, to get authors on our side, and to get their bosses on our side, because we are protecting the book that is important to the duke because it’s dedicated to the duke, or it’s dedicated to his grandfather.

The Medici give permission to print The Prince partly because it’s dedicated to a member of the family, and it celebrates their fame. They want to be able to control its quality and make sure that it’s published in good quality and that it always has that dedicatory letter at the front. They have an incentive to control what we would now think of as copyright. The Inquisition, wanting to please them, has an incentive to give them that control.

Dwarkesh Patel

To close off, do you have some sense of how to think about why Machiavelli’s remembered so differently from not only what he wrote, but why he was writing?

Ada Palmer

Sometimes in the history of thought, there are authors who become separated from their work. You have a parallel where there is the actual content of what the person did and said, and separately there is the idea of this person. In the case of Machiavelli, we have Machiavelli the patriot, Machiavelli who did all this work, and separately we have “Machiavellian” — “the murderous Machiavel”, as Shakespeare calls him. Old Nick, which is a nickname for the devil but became popular because of Niccolò Machiavelli. Old Nick, literally a synonym for the devil.

He splits, so that the idea of Machiavelli—the Machiavellian villainous figure that Shakespeare’s Richard III invokes as someone he’s modeling himself on—is useful to people as a character, as an idea. It’s the idea of the scheming politician who is probably atheistic, definitely self-serving, and who wants nothing but to advance himself in power. Of course that isn’t the real Machiavelli if you read the work. The real Machiavelli is not about advancing yourself. It’s not a manual for getting ahead. It shouldn’t be shelved next to How to Win Friends and Influence People, because it’s a manual not of how to gain power, but of how to keep power. If you have a government and want it to be stable and protect the people’s lives, do this.

But the idea of the murderous Machiavelli is very exciting, and this happens at other times to other intellectual figures. It happens to Thomas Hobbes in the phase that Thomas Hobbes is the Beast of Malmesbury, and the idea of Thomas Hobbes separates. It happens fascinatingly to Spinoza, an important radical Jewish thinker of the later 17th century. Spinoza is a neat one, because when you actually read Spinoza, he’s really warm and sweet. Like Machiavelli, he’s passionate and cares about people, and in his case is an incredibly pious theist. He’s a monist. He believes the entire universe is the body of God. You are a part of God. The table is part of God. The camera is part of God. Everything is God. Isn’t that great?

But a fact about Spinoza—and I know this feels tangential, but it’s not—was that he was the first person in ages and ages to be targeted with the Jewish equivalent of excommunication, the ceremonial, “Your radicalism is too radical. We are expelling you from the community of Jews.” It was such a rare ceremony that the Jews of his region actually had to send somebody traveling all around Europe to find a Jew who knew the ceremony, because it was so incredibly rarely done.

The fact of that spread around, and people had the idea that Spinoza must be even more weird and heretical than any heretic if even the Jews would expel him. The idea of Spinoza the arch-heretic becomes a character. Everyone talks about Spinoza the arch-heretic, and then you read him and it’s nothing like it.

But sometimes the character is useful. The thought experiment figure of Machiavelli the villain is useful for our philosophy. We like to talk about, “what is a Machiavellian self-serving politician?” What would they do? This has a separate life from Machiavelli’s real ideas, to the degree that all the way through the 16th century, there’s these amazing discussions of Machiavellianism in Spain. They’re talking about the Jews as Machiavellian and Machiavelli as the prince of the Jews. You’re like, “Machiavelli was in no way a Jew at all.”

But what they mean by Machiavellian and by Jewish is somehow the political thought that is undermining our good Catholic Spain. So Jewish and Machiavellian can become synonyms, mad as that is for us. Because for them, both of these are labels for the sinister underground of thought, and now we’re talking about the sinister underground of thought.

The idea of Machiavelli as the villain is itself enchanting and interesting. As we look at when Machiavelli is invoked in the modern day—when The Prince sits on the shelf and it feels like something exciting to buy and to read and to think of as a manual of getting ahead, when having it on your shelf makes it feel like you’re participating in the idea of strategic advancement and rationalism—that’s much more Machiavelli the character, Old Nick, than it is the Niccolò Machiavelli who faithfully sat in exile, willing to give up wealth, fame, society, the ability to visit his wife, anything, in order to serve his country.

To me, I think even more fascinating than looking at either Old Nick, the fictitious Machiavellian villain, or Machiavelli the patriot, is to look at how did we double-image this? What is the fascinating tendency of our society to take something real, powerful, exciting, intimate, and then say, “But we can also make the character,” and the character is itself interesting.

So if you take away a main message from this with Machiavelli, it’s that Machiavelli, the character of thought experiment, is an important backbone of our society. We use him as we think about politics. Machiavelli, the actual innovator, is a different backbone of our society and how we think about politics. If Machiavelli can be two such different things, Old Nick and Machiavelli the patriot, so many other things we encounter in life have actually been teased apart by our social utility and made into multiple things which are useful to us in different contexts.

If you have The Prince on your shelf, read it and remember it was written by somebody who was willing to give up anything to serve his country, and you’ll see a very different Machiavelli come through.

Dwarkesh Patel

I think that’s an excellent place to close. Ada, thanks so much for hopping on.

Ada Palmer

This was a pleasure, as always. I hope it won’t be the last time.

Dwarkesh Patel

I hope so, too.

Economics of AGI episode w Alex Imas and Phil Trammell.

There’s a bunch of important questions about how we deal with AI that only economics can answer.

What is the optimal way to tax and redistribute the wealth that will be generated? How should countries not in the AI supply chain index into the gains? Is there any world where inequality doesn’t explode?

It might seem like these questions have obvious answers, but the first thing economics teaches you is that your intuitions can often be entirely wrong.

It was very helpful to chat through these things with Alex and Phil.

Watch on YouTube; listen on Apple Podcasts or Spotify.

• Jane Street invests heavily in turning smart people into exceptional researchers and engineers. In addition to their apprenticeship model, Jane Street runs lectures and bootcamps in their in-office classrooms -- managers clear their teams’ schedules to encourage attendance. If you’d like to work at a place that takes learning this seriously, Jane Street is hiring. Check out their open roles at janestreet.com/dwarkesh

• Google’s Gemini Omni has incredible video editing capabilities -- you can upload a video and have Omni change the background, adjust lighting, or add specific elements. But Omni is also a preview of how future frontier models will be trained -- fully multimodal on both input and output. You can try it yourself in the Gemini app at gemini.google or in Flow at flow.google

• Cursor used targeted RL with textual feedback to help train their Composer 2.5 model. One of their researchers, Sasha Rush, gave me an impromptu blackboard lecture to explain how this form of on-policy self-distillation works -- I posted the full thing on X. If you want to try Composer 2.5, go to cursor.com/dwarkesh

(00:00:00) – Will capital share increase?

(00:19:36) – Messy Middle scenario

(00:25:57) – How to tax and redistribute AI wealth

(00:30:02) – Why demand collapse is unlikely

(00:39:26) – Human employees would be hard to integrate into the machine economy

(00:43:08) – What if some humans (or AIs) value wealth accumulation intrinsically?

(01:01:28) – What should developing countries do?

Dwarkesh Patel

Today I’m chatting with Alex Imas, who is Director of AGI Economics at Google DeepMind and Professor of Economics at the University of Chicago, and Phil Trammell, who is Head of Economics at Epoch and research scholar at Stanford.

In general what I want to understand in this interview is what economics tells us about what we can expect in a world with more and more automation and more advanced AI. I want to understand what that tells us about what will happen to wages and the labor share, what the best way to tax and redistribute the wealth generated by AGI will be, and what kinds of things will be scarce. What is scarce tells you where the value will accrue.

I want to start there. What are some plausible candidates of what will be scarce?

Alex Imas

Something like the relational sector, which is defined as services and goods where the fact that a human was in the loop is part of the value of that product. Because humans are naturally scarce, if we have automation where a lot of other things stop being scarce, we will still have scarcity in the things that humans are involved in and in the loop for.

Dwarkesh Patel

I’m curious to understand whether humans doing services for other humans can ever be a big part of the economy. Here’s maybe one intuition pump. In a world where AI can physically do anything humans can do, there’s this whole machine economy where they’re building factories and doing research and coming up with new ideas. Humans may or may not be involved in the physical production of those things, but probably not in the ultimate limit, if robotics is solved. If you don’t care about humans being involved in that process, why would they be?

But then there are these other things you point out where we actually do want the ballerina or the barista to be a human. That’s part of the value of going to a cafe or a performance. But only humans have that preference. So there’s this human economy where humans are doing services for each other, and part of their wealth is flowing to other humans.

But part of their wealth is also flowing out, because they will want some of the automated goods this machine-only economy is creating. This is not a closed loop. A lot of things in the machine-only economy are a closed loop because the machines don’t care about getting the human barista to make them a coffee.

Within that model, isn’t it intrinsic that the human-only economy will become a smaller and smaller share?

Alex Imas

I would like to pitch a rephrasing of that question. My view is that the individual forecasts economists like us would make, as individual forecasts, are not necessarily very useful.

There was a blog post by Andrey Fradkin, Brian Jabarian, and Andrew Koh that came out yesterday looking at economists’ forecasts about the labor market. What they found is that there’s a ton of disagreement in every single direction.

What they advocate for, and I’m in agreement here, is that rather than thinking about individual forecasts, we should be generating prediction markets where you get aggregate forecasts and wisdom-of-the-crowd effects. The reason I think this is because we have been famously terrible at forecasting.

Let’s go all the way back to 1820. This debate we’ve been having is actually 200 years old. David Ricardo is one of the classical economists, not neoclassical. When the Industrial Revolution started happening, he wrote a bunch of stuff saying, “This is going to be great for everybody. Prices are going to come down.” But then he turned around and said, “Wait, I can see all these jobs that are creating value are going to be automated by these machines. This is going to be really bad. Everybody’s going to become unemployed, and there’s going to be political unrest.”

And if you look at Ricardo’s predictions, they’re actually right. All those jobs that made money in Ricardo’s time got automated. If David Ricardo woke up and somebody told him all those jobs did get automated, and then asked him, “What do you think the prime-age employment rate is in 2026?”, I think he’d be surprised to be told it was the highest it’s ever been other than 2000. We have the highest number of employed people that could potentially be employed since 2000. That was the peak and now it’s the second peak basically.

What David Ricardo ended up missing is that you have these economics of structural change, where everything that got automated became cheap. People had more money to spend, and then they started spending it on services. This is the lump-of-labor fallacy. David Ricardo didn’t consider that new jobs would be created.

But it’s not obvious that money would go to services. Why wouldn’t it go to more automated goods and something like that? I’m not using this anecdote to say this is what’s going to happen now and that we’re going to have full employment. I’m using it to say it’s really hard to make predictions.

What may be a really useful tool that economists have is to instead start with a premise. Maybe we start today: labor share is zero. Labor share has gone down. What could possibly explain this? Let’s write down an economic model of what happened. Phil will talk about this later today. Or you can write down a model that asks, “What if labor share just stays the same? What can make that happen?”

If you don’t take anything else out of this conversation from me: We don’t have any data. I’ve been saying we need a Manhattan Project for data. We don’t have data on consumer demand elasticities. We don’t know what they are. We’re not really tracking what jobs are getting created or destroyed. The O*NET database, with all of the tasks and different jobs, has been rarely updated and is super low quality.

What is really useful is to think about the potential scenarios, map them out, and say what dimension of scarcity will generate each scenario. If there’s full employment, we can talk about the relational sector. If the labor share collapses, we can talk about other sorts of scenarios. That will tell us what data we should be collecting.

Dwarkesh Patel

It’s probably worth defining labor share and capital share real quick. The whole economy, the total sum of goods and services sold, is either paid out to people in wages or it’s paid out to capital, which is to say, there’s rent on buildings and shareholders of companies that get paid out.

For many hundreds of years, ~60% of the economy basically gets paid out to humans in wages, and the other 30-40% gets paid out to people who own machines and land and claims on companies. The question is, if 60% is going to wages right now, does that shrink as AIs get smarter and better?

Alex Imas

This is a Kaldor fact. We should stress this. It’s incredibly surprising that it’s over 60% after the Industrial Revolution and all of the automation we’ve ever seen. Some people are worried it’s an accounting error that it’s been so constant.

There’s even a controversy right now. Some might say labor share has been falling in the last 20 to 30 years. But there have been a lot of accounting changes in the last 30 to 40 years. For example, Atkinson has a paper showing that if you keep the accounting constant over the years, labor share hasn’t even fallen ever.

Dwarkesh Patel

But it’s not that surprising, right? Phil, you made this point that if labor and capital are complements, you need both to do anything. It would make sense that you’d need to pay both of them to get something done.

Alex Imas

You have had stuff be completely automated.

Phil Trammell

There’s a sense in which nothing has yet been completely automated. Look at the network-adjusted factor shares of a good. Look down the supply chain and not just the final step and how much of that is done by capital and labor, but what went into the machines that can automate that final step. You’ll find that labor is adding a lot of value down the supply chain.

Computer and electronic products in the US have a very stable network-adjusted capital share of around 50%. It’s not 100%. I do think there’s this qualitative shift that I think we agree is coming, which is that there will be at least some goods whose network-adjusted capital share goes to one. The whole supply chain can be automated, and there’s no part in it that we care intrinsically about having a human do. That will be a qualitative shift.

Interestingly, the implications of that shift for the overall capital share are ambiguous. Let’s say we’ve got two sectors: the human-intrinsic sector with the ballerinas, and everything else. Right now, everything else has been scarce because of the lack of labor in it. But if we fully automate the supply chains for everything else, and we satiate in everything else really fast, then the quantity of everything that’s not a ballerina goes to infinity, but the marginal utility in that stuff goes to zero faster than the quantity is rising.

Alex Imas

I also want to move away from the ballerina example. The point I was trying to make in my post—working backwards from a particular scenario—was that the ballerina and the performer are the wrong reference class. Right now we have a lot of jobs where you have different tasks. This is the task-based model of jobs. Take a doctor, what is their job? They’re filling out insurance documents. They’re going and calling different pharmaceutical companies. One of their tasks is to see the patient and talk to them, but that’s not the main part of the job.

You could have a job and a service or a good be a product of different types of tasks, and you can automate a ton of those tasks. If the consumer is willing to pay more for a product or service where every single task is automated except for that one part where the doctor is delivering the diagnosis and providing support, we would call that job part of the relational sector. People are willing to pay more for the human to stay in the loop in the job.

We don’t have data to say, “Here are relational jobs, here are not.” You literally need to collect data of the following sort. Do a conjoint analysis of your willingness to pay for this service or good. Here’s the counterfactual where everything is produced by machine. Here’s the counterfactual where this one task is not produced by a machine. What is your willingness to pay? What is your elasticity for the human to not be in the loop?

If I don’t have that data, what prediction am I going to make in this story?

Dwarkesh Patel

Isn’t there another point, which is that there are a lot of fully automated goods that don’t even exist yet? And you can’t collect any data right now about, say, how much people will want to keep buying more and more of some drug that makes you healthier that is fully produced by the AIs.

Alex Imas

Absolutely. That’s kind of Phil’s point. You could have an increase in variety in capital where you don’t get the satiation. You’re increasing variety, so you’re not hitting that diminishing marginal utility point where most of your income is going to the human sector. If that increasing variety is fast enough, and there is no such increasing variety in the human sector, then you can get all of the relational goods you want, but it doesn’t matter for labor share. It goes to zero.

Dwarkesh Patel

Phil, I liked your analogy to some Mongolian economist sitting around in 1400 thinking about what will be scarce and the limits of that kind of analysis. I think you should talk about that.

Phil Trammell

Just look at the goods available to a Mongolian of the distant past. I’m no expert on this society, but I know that they didn’t have nearly the variety that we have now. Look at the jobs that were intrinsically human, like being a singer. And then you look at the things that were not intrinsically human, like the transportation services provided by their horses or the different kinds of food they had.

If they just held the varieties fixed in both categories and asked, “What will happen once we have a lot more automation?”, they might have said, “We’ll just satiate in horse-like transportation and in yogurt and in yurts. Those shares will all go to zero, and we’ll be left spending all of our money on singers.”

But of course, that’s not what happened. As we’ve accumulated more wealth and more advanced machines, we’ve expanded the range of things other than singers to spend our money on, and the share spent on singers has stayed negligible. Likewise, that’s my central prediction about how the future unfolds, though it could go either way.

Dwarkesh Patel

I was going to make a point and I realize it’s a fallacy, but the reason it’s a fallacy is interesting. It’s just hard to imagine a world where there are trillions upon trillions of robots, but there’s only some billion-odd humans, and the cumulative amount we’re spending on robots is less than what we’re spending to pay Magnus Carlsen or—

Alex Imas

Financial advisors or doctors or tutors.

Dwarkesh Patel

Right, or podcasters or whatever. But then I realized why it’s a fallacy. The number of transistors in the world has literally trillion-X’d, maybe quadrillion-X’d. Your colleague Chad Jones has a very interesting result about how the share of the economy that is going towards paying for computing, paying for the transistors, has been decreasing.

The point you made is that one way to think about Moore’s law is… What sets price? Supply and demand. So not only are we producing more transistors more cheaply, but also the value of the marginal transistor is decreasing. As you were saying, another way of saying Moore’s law is…

Phil Trammell

I like the pessimistic framing of Moore’s law: every 18 months, the value of computation halves. We’re running out of uses for computation so fast that it’s sustaining Moore’s law.

Dwarkesh Patel

This is relevant to a conversation about AI where maybe for the first time, this is no longer true. The famous fact here is that an H100 costs more to rent now than it did three years ago, even though we have much superior technology and much more compute in the world. Because as models get smarter, the opportunity cost of compute gets higher.

Alex Imas

This is Phil’s point about increasing variety. What we have done is increased the types of things that people demand from capital. Now all of a sudden you have a new variety that you could be using capital for, and you jump back up.

Dwarkesh Patel

You could imagine we just never satiate demand for compute. As long as that stays the case, the share of the economy that is going towards compute would keep increasing.

Alex Imas

That’s the big question. That is the ultimate question that we need to be looking at. What number of new uses are we finding for that compute where you have the demand for these uses? What I want to emphasize is that a lot of models in economics, especially in the space that we’re talking about, take demand as almost exogenous. They don’t unpack the psychology of what people actually want.

What got me thinking about the idea of the relational sector is work that I was doing on the fact that there does seem to be this intrinsic value. It’s not just because it’s scarce; it’s because there’s some intrinsic preference that people have for empathy, connection, and interacting with another person.

One of the experiments that we ran involved an art print. We have an incentive-compatible way of asking, “How much are you willing to pay for this art print?” People are actually paying real money for it. Then we say, “Look, there’s only one of those art prints, and it’s either made by AI or by a person.” These are between-subject conditions. With one, you get the effect that the person-produced art print is valued much higher than the AI version.

Then, in a set of other conditions, we say there’s 500 of these being produced. For the human-made one, the price goes down a lot because it’s no longer seen as making a connection with this one artist. With AI there’s no difference. AI is already viewed as a commodity.

We need to do a lot more research on this, but it seems that’s the key difference between this and something like a horse. A horse was an input into an output, where you can replace the horse with something else. You only care about the output.

The only way this relational story works—and this is what we need more data on—is if a human is not a horse in the sense that they are providing value from the output, where if you replace the human, the value of the output decreases. If that’s not strong enough, and if it doesn’t hold for enough sectors or enough jobs, then this story doesn’t work anymore.

Dwarkesh Patel

There’s one possibility which Molly Kinder has written about, this “Messy Middle” scenario. That possibility made me think about whether it might be better to have—at least as far as wealth distribution and redistribution go—a much faster AI takeoff.

I want to ask you whether the following possibility is at all likely, or if there’s any set of assumptions that can make it so. AI makes it possible to automate jobs such that many people are losing their jobs, but it doesn’t create enough wealth, while the process of automation is happening, to basically pay off the people who are getting laid off and create a Pareto improvement, where everybody’s getting better as a result of AI automation.

Of course, there’s a trivial sense in which that must be true. Whatever money the company is saving by not paying the humans instead of just paying the AIs, those resources still exist in the economy and can just be paid out to people.

But there’s going to be some allocative inefficiency. The government doesn’t know exactly who got laid off because of AI. There’s a political problem. If the Meta worker gets laid off first and they were making $200,000 a year, is there a politically sustainable situation where you give them a $200,000 check a year when there are many working people making much less? Do you find this scenario plausible, where AI is automating a bunch of things, but there isn’t as much wealth creation as there is automation?

Phil Trammell

I think it’s possible. To me, it does seem like a pretty narrow window. My guess is that if we have the technology to automate so many jobs that it becomes a new kind of political problem, then the pie will also be growing really fast.

Well, unless in all of those professions it’s automating, it’s just a hair more productive. So the cost of all the capital to replace all the software engineers is just a hair less than the cost of what we’ve been paying the software engineers.

Dwarkesh Patel

Why is it implausible that a company can save money by laying off a bunch of software engineers? And in the long run, there’s a Jevons paradox, and we can’t anticipate in advance what we’d do with more software, and surely there will be more uses. But in the short run, the effect is just that a lot of people are laid off, and they still need to figure out how they can use a million times more JavaScript tokens.

Alex Imas

Phil and I have been writing about these things, and we have mathematical models in the back of these things. We don’t have any political economy in any of our models.

Andy Hall wrote a really nice blog post about the politics of AGI, and he made a really interesting observation. If there’s a 2% increase in unemployment, the political winds completely change. Unemployment has a huge effect on what happens politically.

Referring to Molly’s excellent essay, I think in some ways one of the worst scenarios is a drip scenario because of the political economy piece. What you might see is people not really being unemployed en masse, but moving into sectors that pay them less money. This is what happened with phone operators between 1920 and 1940. Phone operators were completely automated, but it took 20 years, even though the technology existed. There was this drip. It wasn’t like this giant sector just disappeared.

There’s a really nice QJE paper on this showing that they got reabsorbed into the economy, but at lower salaries, and they were mostly underemployed. That’s the scenario Molly was writing about, this messy middle where things aren’t a disaster. We saw with COVID that the fiscal response can move quickly if there’s an emergency. An emergency is a quick uptick in unemployment, which could even look like 2-3%. That becomes a national emergency if it happens fast.

Dwarkesh Patel

The concern is that whatever you’re saving on those white-collar workers, if that’s not growing the economy but just creating saved resources that can be allocated elsewhere, is that enough to do a broad-based redistribution scheme? You have the money you’ve saved off a couple of people. Unless you can figure out exactly how to get it to them specifically, you have the problem of, “Can I do a UBI off the money I saved by laying off…?”

Alex Imas

You’re basically saying the pie did not grow that much. You’re just displacing a bunch of people, but that didn’t grow the technological frontier of what the economy can produce.

Dwarkesh Patel

Then there’s a question of whether every time this has happened in history, the technological frontier has expanded a bunch.

Alex Imas

I think that’s the case. Simply in history, the technological frontier has expanded. I think Phil made the same point. It’s hard to imagine that sort of scenario where you are getting intelligence that’s just enough to replace the software engineer but still costs a lot of money. It’s just a hair less expensive than the software engineer, so you’re not getting this abundance effect. Where is the redistribution going to happen because the pie didn’t grow?

Dwarkesh Patel

This is very helpful. Many different things have to be true for this scenario to come to pass, each of which seem unlikely.

One, it has to be the case that it is possible to automate entire white-collar jobs, but only in a piecemeal way. That is to say that you can only automate software engineers, but that same program can’t also automate an accountant and an analyst and whatever.

My model of intelligence is such that—both the breadth of tasks it requires to do something like software engineering and what intelligence is—if you can really just lay off all the software engineers, you’ve got enough in the bucket there that you could automate all kinds of white-collar work. There are huge amounts of potential savings that have happened as a result of these layoffs, and also AI is going to be cheaper than human labor.

If both of those things are true, this messy middle scenario where we literally don’t have the wealth to go around seems unlikely. Then the question is, what is the best way to tax it and redistribute it?

Alex Imas

I have some thoughts. I think it’s really important to outline the costs and benefits. First, there’s differential complexity in implementing these things. Two, they differ in the timeline of being actually helpful. Something like universal basic capital is not going to generate returns for something that happens in six months. You probably are going to end up with a layer of things.

Take a negative income tax, for example. You implement it, and the day it turns into law, you already have this insurance that there’s a floor where everybody gets a certain amount of money, and if you earn more money, you get taxed more. But there are positives and negatives to a negative income tax.

With UBI, for example, I worry a lot about the political economy implications. If people are just dependent on a check, it really matters who’s in power. Right now, we’re endowed with labor that can turn into income. When that is no longer the case and we are at the mercy of the elected official for basic needs, that feels like a power-sharing arrangement that’s really dangerous.

Dwarkesh Patel

But wouldn’t that be true of any sort of government redistribution program?

Alex Imas

With something like universal basic capital, where you have an ownership share and property rights for capital, you just have a share.

Dwarkesh Patel

You’re a normal shareholder.

Alex Imas

You’re just a normal person.

Dwarkesh Patel

But this goes back to the question of indexing, because if indexing is hard, then universal basic capital is hard.

Alex Imas

That’s the problem of universal basic capital: targeting. What do you target to put into people’s portfolios?

Dwarkesh Patel

Like, what if Anthropic goes to zero, but some random robotics company takes all this over?

Alex Imas

Exactly. That’s the risk of universal basic capital. With a negative income tax, you have the same sort of issues as with UBI, where somebody comes into power and says, “We’re not going to do that anymore,” and people can’t work, and then you have the issue of the floor being gone.

Dwarkesh Patel

One concern with the wealth tax is that there’s no politically sustainable equilibrium at a 0.5% wealth tax. This happened with the income tax, of course. It starts low, it’s for war or something, and then it slowly escalates until the marginal income tax rate in the US is on the order of 40%, and in certain states, upwards of 50%.

With a capital tax, is there a reason to worry that it would distort investment? Would people just say, “Why would I invest in Anthropic or Intel? The government is going to take larger and larger shares of it and dilute my share.”

Phil Trammell

Hold on. It’s worth separating how the revenue is raised, what’s taxed, and how it’s distributed. It could be that the government hands out shares of Anthropic to everyone by a broad-based tax and then buying Anthropic. Which would probably be the right thing to do. Hopefully, some populist proposal doesn’t interfere with that and expropriate some particular company that everyone happens to know about.

Dwarkesh Patel

You’re suggesting there could be some sort of optimal tax. We’re taxing externalities or we’re taxing land. I guess we probably need to tax something other than just those two things.

Phil Trammell

Or consumption.

Dwarkesh Patel

Ok, a consumption tax, like a European value-added tax, allows the government to go buy a bunch of stocks, and then they just distribute those stocks to everybody.

Alex Imas

That’s David Autor’s...

Phil Trammell

That’s not going to be that different from just redistributing the stocks, but it will be a little different.

Alex Imas

That was the proposal for Social Security, by the way. That was privatizing Social Security. It’s worked so far, but there are questions about how long it’s going to keep working. Privatizing Social Security was basically giving everybody a basket of stocks.

Dwarkesh Patel

People talk about whether there’s a white-collar apocalypse already. Is there any evidence that suggests there is mass automation or unemployment as a result of AI already?

Alex Imas

A lot of people are looking at it. This is an area where there’s a lot of eyes and a lot of data being produced. The Budget Lab over at Yale is doing really good analysis on this. They just recently released a report, and you really have to squint to see anything happening. If you want to take an approach across the entire economy, even looking at software engineering, the most exposed sectors, there’s just not really anything going on.

There might be a little bit of a signal about junior developers getting jobs less than before. But that’s a “less than before” rather than a level shift, as in there’s actually an increased demand for senior software engineers, if anything. If you look at the trend, for junior developers, it’s a bit below trend.

Dwarkesh Patel

So you’re saying the growth is slower than before, but there is still growth even for entry-level software engineers. What do you think is going on with the anecdotal evidence of graduating college students saying that they’re finding it harder to find CS jobs?

Alex Imas

I think that’s anecdotal evidence.

Dwarkesh Patel

You think it’s always been hard to get jobs for some people, and now it’s getting turned into an AI narrative? Same with the layoffs, where it’s probably just a normal layoff, and they turned it into an AI layoff.

Alex Imas

You have to be careful with all of this. There are these public coordination devices. Let’s say we get into a narrative where if you’re a firm and you’re not laying people off, then you’re seen as not adapting AI enough. Then you’re going to just get a cascade effect of firms needing to keep up with the Joneses in terms of starting to lay people off. That’s super worrying, where the firm might actually be worse off after the layoffs than before, but it’s just doing the layoffs to have the perception that, “Look, we’re not behind the times. We’re using AI.”

You probably heard these anecdotal stories of token counters, where you have to maximize tokens and things like that. Right now, we don’t really have any evidence of a white-collar bloodbath.

Dwarkesh Patel

Is that surprising at all, given all these things AI can do? This is a story as old as time. If you automate some complementary task, the overall bucket of things—the human labor which complements the automation—will increase in value.

Alex Imas

One of the statistics that’s really important for that argument is elasticity of demand.

Take the O-ring model of jobs. A job is a series of tasks. Let’s say the AI automates nine out of ten tasks. One task is not automated. If that person can now focus in on that task, the job will become more productive. If that translates into a price effect where the product is actually cheaper, and if demand responds enough where it’s being bought more and the service is being used more, that could actually lead to more hiring.

A lot of people on the internet have been making that argument very generally, saying, “Look, if anything in the data, we’re seeing an uptick in software engineering demand.” Which suggests that at least for now, given the way that jobs work, it might be elastic enough.

Dwarkesh Patel

I think this elasticity of demand argument is incredibly important for a lot of arguments that people make, or just a lot of labels that people use without understanding what the underlying causation is. People often talk about Jevons paradox. This is the idea that as something gets cheaper, you will want so much more of it that the total amount you spend on the thing increases. Famously, this happened to coal in Britain ~200 years ago.

But really this only happens if the demand for something is highly elastic. There are many things for which there is not super elastic demand. If oil, for example, gets super cheap, it’s not like magically—

Alex Imas

Or insulin.

Dwarkesh Patel

Exactly. It’s not like magically there’s going to be so many more cars that now we’re going to be using way more oil than before.

Phil Trammell

At least not in the short run.

Dwarkesh Patel

Exactly. The long-run elasticity is higher than short-run elasticity. But even in the long run, agriculture famously is the example where we can produce way more food if we dedicated the same portion of the economy that we dedicated to agriculture in the past. We’re already producing more food regardless, but we could produce even more if the same portion of the economy that was producing food 100 years ago was currently producing food. But you eat enough, and then you’re done.

The claim with software is that it is not some inherent property of markets that as it gets cheaper, you’ll just keep wanting more of it. The thing about software is this is a particular kind of good where as it gets cheaper, we’ll want more and more of it.

It is also highly relevant, and you wrote an essay about this—a lot of this podcast is me summarizing your essays back to you. There’s this very viral scenario planning about the future by Citrini, predicting that as a result of automation and very powerful AI, there will be a recession. White-collar workers will get automated, their salaries will no longer be available, and so there will be a slump. Do you want to recapitulate why this might be implausible?

Alex Imas

Part of it is plausible, part of it’s not. The part that we started the conversation with is the idea that there could be a lot of unemployment. If the speed of automation is quick, people could get laid off, and they may not find work very quickly. We can quibble about the unemployment part of the Citrini essay, but that’s not the issue. The issue is that they talked about negative economic growth.

What I did in the piece, that Phil and I had a back and forth on, was to say, let’s start with the proposition that there’s negative economic growth. What conditions do you need in the economy to get negative economic growth? It turns out the conditions are pretty improbable.

One thing that you need is for the holders of capital, rich people basically… Basically what you have in those sorts of scenarios is a reallocation of wealth and income from lower-income people who are using their labor towards tech capital owners. So you need demand to be bounded, like a hard bound, not even a soft diminishing sensitivity. You need for them to eventually say, “I’ve had enough. I don’t want to spend any more money.” And for that money to not enter as investment. Then you can get negative growth.

Dwarkesh Patel

The crucial thing is, even if we don’t want more shit, the world in which there’s a singularity and we don’t want to invest more money is crazy. We’re not saying, “Let’s build more data centers. Let’s build more fabs.” Even though we have AGI, we’re not investing in more data centers to run the AGI and that’s driving more economic growth.

Alex Imas

I sent the essay to Phil, and Phil wrote back being like, “This is pretty dumb,” like my essay. He said, “You’re trying to say that there’s going to be negative economic growth, but these are very implausible conditions.” And I was like, “That’s the point of the essay. These are very implausible economic conditions.”

That’s where scenario planning really shines. You have the Citrini essay, which was great that it was written because it started a conversation. It’s so intuitive, this idea that if there’s demand collapse, we can get the economy to shrink. You could get that with a depression. In the Depression, the technological frontier didn’t expand.

Here, the technological frontier is expanding. You actually have abundance. For abundance to generate negative economic growth, that’s really hard to get.

Dwarkesh Patel

We were talking a second ago about why there isn’t more automation as a result of LLMs. One plausible mechanism could be, as you were saying with the O-ring theory… O-ring theory refers to the fact that the Challenger shuttle blew up because one component malfunctioned, and it destroyed the whole thing. Maybe that’s a more general model of how goods are produced in the economy. You have to make sure everything is reliable and works well. So you can’t automate an entire job to an AI right now. Even though it might be able to perform it at some probability, you need extreme reliability in order for it to not destroy the finished good.

This might explain why there’s a lot less automation now than there otherwise could be. But I think it works in the other direction once AIs get advanced enough. Integrating humans into the production flow of future goods will become difficult. Even beyond the arguments about how humans will be more expensive or less capable, there will be whole production flows organized for AI labor. They’re talking in neuralese. They’re thinking many thousands of times faster.

So even if there’s some comparative advantage where it makes sense to hire a human, there will be transaction costs and worries of reliability that will actually make it hard to integrate humans into future production flows.

Phil Trammell

That seems right to me. In particular, I just want to distinguish between the point that if you automate nine-tenths of a job, people might shift over to the last tenth, but there might be ten times more work demanded of them.

Compare that to the model of O-ring automation from Gans and Goldfarb recently. If you can only automate nine-tenths of the job, but you do it to a lower standard of quality than the human could, you might not want to automate even those nine-tenths.

That’s the thing that could totally port over. Symmetrically, it could be a reason why we don’t use a human for one-tenth of the job anymore, because a human just can’t perform it to the level of quality that the AI can perform the other parts of the job, or the level of speed. They end up pulling down the quality or speed of the finished product.

Dwarkesh Patel

By the way, the model you’re talking about seems extremely plausible to me for why more lawyers, accountants, or even software engineers are not automated. There are cases where there’s a pretty good probability that the thing worked as you expect, but the thing you’re paying the lawyer for is: “No, really, my company’s not going to go under because—”

Alex Imas

You’re also paying for a lot of regulation-type stuff. With lawyers particularly, you need some entity to back up the product. You need ownership of the product. You need somebody to be able to fire or hire, and there are licensing issues. There’s a lot of regulatory layers that are also going to be keeping—even if there’s no relational element—humans in the loop that have nothing to do with the ability of the human to actually perform the service.

Phil Trammell

All of these frictions on the political-type decisions that we are accustomed to only trusting humans for—legislation, being a judge, being a juror, or all the licensing that keeps certain professions human—that all strikes me as transitional.

What we expect to come from a human and how we organize our politics has changed so many times throughout history, from little hunter-gatherer bands to empires to whatnot. Once an AI-run political system is much more efficient than the alternatives, those will probably tend to out-compete the others.

Dwarkesh Patel

Speaking of which, we’ve been talking about what preferences humans currently have and what impact that has on what kinds of goods will be scarce in the future. But of course, we’ll have different kinds of entities in the future: AIs. There was a time when there were no humans on Earth, but evolution selected for agents that have specific drives and preferences because those tend to survive the most, and those preferences now determine what a hundred-trillion-dollar world economy produces.

Why not expect the same thing from AIs in the future? This is not even a world with catastrophic misalignment, where they just kill everybody. But there will be evolution of, even if not individual AIs, firms which have AIs as part of them. What will that evolution favor? It will probably favor firms or agents that grow. There’s a selection argument that things which grow will be more prevalent.

Maybe just based on that, you can make some predictions about what their preferences will be. Is the kind of entity which prefers to have human-intrinsic goods going to be the kind of entity that accumulates resources the most? Probably not. Probably it saves more and has unsatisfiable demand for whatever the relevant resource happens to be. Compute is an obvious one. Can we use that to make some predictions about the non-human preferences that will be guiding the future?

Alex Imas

If there’s an AI that has its own welfare, is fully autonomous, and is making its own decisions that are welfare-relevant, to be honest, I have absolutely no prior that it would prefer to deal with humans. There’s no reason.

But let me take the other side of that argument. Humans’ preferences to be interacting with one another, to trust and empathize with other humans versus a simulated AI, I think it’s a really important question whether those will change.

I’ve heard a lot of arguments saying, “Look, right now we’re just not used to the technology. What you’re thinking of as relational… At some point, people are just going to see an AI therapist as a superior product, and they’re not going to need the empathy that the human is providing.”

I think this is actually a really complicated question. Here’s one argument for why it’s not going to go away, and it has to do with evolution. Let’s say there are two types of people. One person doesn’t really have this preference. They can just interact with an AI, whatever can simulate it better. The other one has almost a moral emotion—using Jonathan Haidt’s framework—against offloading those sorts of social interactions to an AI.

Which of those two people are going to reproduce, find a mate, all of these sorts of things? I think the answer is clear. It’s the second one that has the preference for other people.

Dwarkesh Patel

Depends on how the reproduction is happening.

Alex Imas

Fair. But if we’re in the world where reproduction is still happening the way that it’s happening, I think… And this is a big question, I’m not making a prediction.

You had David Reich on the show. His point on the last podcast was that we’re buzzing with natural selection. So even if you get some sort of indifference now, you might get selection to point into an even stronger preference for other humans.

Dwarkesh Patel

Here’s one way to think about it. How is the wealth of the richest people in the world instantiated? We were having a call earlier, and you made the point that their consumption is more geared towards relational goods. Like Mark Zuckerberg is hiring MMA instructors and dancers for his wife’s birthday, and so forth.

But most of his wealth is just stock in Meta. As a controlling shareholder, he could say, “Meta, turn all this wealth into dividend income, and I will just spend that on consumption.” Instead, he would rather have his wealth compound and have Meta build more data centers.

So you don’t even have to change humans for this to be the case. The humans who are wealthiest—and growing wealthier because their wealth is compounding—just have this almost Nick Landian preference for accelerating capital. That does seem to suggest that this is an important determinant of what kinds of things are produced in the future.

Phil Trammell

There are two ways you could get the two kinds of people, one of whom prefers a human therapist and one of whom is fine interacting with the AI. If they both satiate equally quickly in capital but the one who likes the human therapist also just likes having some human-intrinsic services, then the marginal value of capital in the future, compared to the marginal value of capital today, for each of them if they start out equally rich, should be basically the same. There could be interactions and whatnot, but basically, that should be the same.

If what’s driving the difference is that one person just doesn’t satiate in capital because they’re engaged by the prospect of exploring the universe and turning their head into a galaxy brain or whatever, and the other one satiates, then the person who doesn’t satiate in capital is going to, if they’re being rational, have a higher savings rate.

So in the long run, they’re going to have most of the wealth, and the overall capital share will basically be the capital share of that person’s spending, which is going to be one.

Dwarkesh Patel

It’s important that we’re not talking about a hypothetical future. Elon Musk is talking about mass drivers on the moon. He’s by far the wealthiest person in the world. Obviously, currently his investments are going towards humans as well as machines, but I don’t think he cares particularly that his future researchers and engineers are humans versus AI.

Phil Trammell

And he manages to reproduce fast as well. So I just think it’s worth drawing that distinction. There are currently some rich people that don’t seem to satiate quickly in capital, and so maybe in the long run they’ll save the most. That does seem right to me.

I would also say, even if they do reproduce more slowly biologically, that might just not matter that much in the long run if they can live forever.

Alex Imas

The living forever is key. Again, we’re scenario-building here. If you could live forever, a lot of stuff changes for my story as well. To your point about rich people not consuming a lot and investing, this will all depend on the returns to capital. Right now, the returns to data centers are super high, but if we get into a situation where people are satiated with capital, then the returns to accumulating capital are going to be lower. Then these rich people are going to be consuming more, because the incentive to invest is smaller.

Basically, you think about the general equilibrium of this sort of process… We have gotten tremendously richer since 1820. Many more people are investing, but you’re still getting a consumption response which keeps people employed and labor share high. That’s because—

Phil Trammell

Hold on.

Dwarkesh Patel

Wait, not necessarily. I think you’re probably making the same point. It could be that their investment has to be titrated through actual laborers who have to do things for their investment to work. In the future, only the consumption is human-mediated, right? Because the investment can just be done by the robots.

Alex Imas

So we’re in the scenario of how you can keep high labor share. Let’s take that scenario. In the scenario with high labor share, for whatever reason, the returns to capital are going to be lower.

Dwarkesh Patel

That’s right. To the earlier thing where we were saying why the messy middle is implausible, I feel like we can do a similar thing here. For our returns to capital to be lower, the growth rate has to be lower, right? It certainly has to be lower than what we’re expecting through the period of transformative AI. If there’s explosive growth…

Phil Trammell

Yes and no. The capital stock could grow quickly, but the price of capital goods relative to consumption goods could be falling faster than the capital stock is growing.

Alex Imas

It’s the difference between the potential frontier of technology and the realized prices of these things. Because you have relative prices.

Dwarkesh Patel

So you’re saying I could be putting my money towards earning 30% interest and investing in data centers, or whatever. There will be something in the future, if the growth rate is high, that earns high returns. Or, as a result of all these technological breakthroughs, there’s some cool product that I really want to buy right now, and both of those will be compelling options.

Phil Trammell

Yeah. It doesn’t have to be a new product. It could be a human-intrinsic product.

Dwarkesh Patel

Although, if it’s a human-intrinsic product, we would want to have it much more in the future than we want it now, because the thing it compares against is—

Phil Trammell

We might want it the same as we want it now in the sense that the marginal utility in a ballerina performance is exactly the same as now. But the marginal utility in a robot might just be a lot lower than now. So in units of robots, we want it a lot more than we want it now.

Would the interest rate be 30%? It depends what you mean by the real interest rate. It might be that every robot now can turn into 100 robots next year. So in units of robots, the interest rate’s 10,000%. But if the price of robots is falling really fast...

Alex Imas

Prices adjust. I think that’s the whole point.

Phil Trammell

Here prices are adjusting in this interesting way that too many macro models don’t allow for. What’s happening is what would be called investment-specific technical change.

The price of capital is falling relative to the price of consumption, instead of doing the standard macro thing of saying there’s just output, this chimera of a thing called output, which one for one can be allocated to capital or consumption. That’s not going to be true in this world.

Every unit of capital next year is giving up way less consumption than each unit of capital this year. One robot now turns into many robots next year, but the number of ballerinas is the same.

Alex Imas

Again, we’re going to go back to the increasing varieties thing. If all of those extra robots next year are actually different varieties of robots and I’m not getting satiated on those robots, then it’s a very different story.

Dwarkesh Patel

But now we’re talking about the consumption world. For the investment side of things, there could be just some greedy titan of industry who keeps wanting more and more robots. That alone would be enough to increase the marginal value of robots and therefore decrease labor share?

Phil Trammell

Yes.

Dwarkesh Patel

Why are we not expecting greedy titans of industry to keep existing?

Alex Imas

Greedy titans of industry historically have built libraries and—

Dwarkesh Patel

But that’s because they die, and they’re like—

Alex Imas

Oh, they all die. Everybody dies.

Dwarkesh Patel

Well, we’ll see.

Alex Imas

Conditional on people dying… You had a guest on the show who said to understand the future, you should think about the past. You could have new types of titans being born whose entire reason for accumulating wealth is just to accumulate wealth.

But a lot of the time, at least historically, the wealth accumulation process is part of a large social interaction amongst peers, amongst the community, where you want to be admired in some way. The stylized fact of titans of industry is you accumulate the capital, and then you buy a bunch of stuff.

Dwarkesh Patel

I guess this is a historical question, but it does seem to me that in a lot of cases what is happening is that as they near the end of their life, they either hand it off to their children, who are worse stewards of capital than they are. They don’t even manage to grow their wealth at the rate the economy grows, much less faster than the economy grows, which their parents were doing.

They’re like, “Well, I care less about my children having it than me playing this game of accumulating wealth. So I’m just going to give it to some trust.” If people are living longer or if they can figure out some way to align their trust to this wealth accumulation process… It just feels like the evolution here is so strong. You just need a couple of agents that think this way for this to be the dominant thing determining the preferences of the whole economy, because this part is growing much faster than the other parts of the economy.

Alex Imas

The part about satiation and diminishing marginal utility keeps coming up, I think it’s really important. If a person has an intrinsic preference for accumulation, that’s just what they want, then I think your story is totally right. But that’s just not how preferences usually work. You have enough hedonics in your life, and then the social status… Rousseau wrote about this, St. Augustine wrote about this. This is a basic part of preferences.

Now, you guys are arguing about something else, where you could have such high concentration that you could just have a couple of exceptions to the rule, and that’s going to be enough. I have nothing to say about that.

Phil Trammell

I think the claim is a little stronger, not just that you could have some exceptions, but that historically and today we see the exceptions. They just haven’t really taken over the economy historically because there have been these dissipation shocks, as they’re called. They’ve given it to their kids who squandered it, or they put it in foundations which spent it. It’s not really a shock, but…

People might have liked to fill the universe with monuments to themselves and live forever, very wealthy. It’s a weird preference, but it’s not a hypothetical preference. I think that’s the claim.

But who knows what’s going on in their heads? Even without the intrinsic preference for accumulation, there are some instrumental reasons why some people might value accumulation, which is also worth bringing up. There’s a desire for political, philosophical, or religious influence. People get into an arms race over what society looks like and what people believe.

Similarly but differently, because it’s not an arms race, there’s just total utilitarian philanthropy. When I think about why it might be good to have a lot of wealth in the future as a good classical utilitarian, to me, the value—or at least one way you could have an almost unsatiating utility function in having wealth in the future—is to create new happy beings. They just add to the total welfare of the world.

This idea goes at least as far back as Bostrom’s astronomical waste point, that we could put Dyson spheres around the stars and turn all the energy into really happy simulations and whatnot.

Dwarkesh Patel

I think the particular greediness of this optimizer doesn’t matter, what they’re greedy for. Forgetting about utilitarian philosophy or whatever, a pure von Neumann probe has… I don’t know, is this an accurate way to say it? They just have high marginal value for the random solar system they’ll occupy because that turns into more solar systems. A von Neumann probe is a thing that can exist. That’s a very greedy optimizer.

Phil Trammell

If we’re talking about whether they’ll dominate the economy, maybe this is a technicality. But we only count final consumption goods and investment goods as GDP. If there’s just this phenomenon—

Dwarkesh Patel

How does a von Neumann probe show up in GDP?

Phil Trammell

Exactly. If we recognize it as a person that owns itself, and it’s optimizing on the margin between spending a bit more on a baby von Neumann probe that colonizes another star system or a ballerina or something, and it just doesn’t value the ballerina very much…

Alex Imas

When we’re talking about AI beings, it just completely depends on how we’re doing the accounting there.

Dwarkesh Patel

What does the world look like in a world where von Neumann probes are possible? Is it possible labor share is high?

Phil Trammell

I think it’s possible the labor share is high the way we usually count it.

Dwarkesh Patel

Do economists have any advice for countries which are not in the AI production chain? If you’re not either producing the AI models, you’re not producing the hardware that goes into AI models, if you’re not Korea making HBM or Taiwan with the fabs or the Netherlands with ASML. India or Nigeria, what should they be doing right now? If you’re talking to Modi right now, what do you say?

Alex Imas

I think the biggest lack of resources that we have allocated in the economics profession is thinking about middle-income developing countries in the age of AI. This is something I fault myself with as well. There’s not enough people thinking about this question.

There are scenarios where you get AI technology being allocated and dissipating to Nigeria and developing countries, leveling the playing field, essentially giving them a level up as far as capabilities.

But there’s another world where, because they don’t have enough resources, they’re not training the models, they don’t have the hardware, and they just completely get left behind. And because of automation, we can produce commodities in developed countries now. Then we don’t even have the consumer market. That world looks pretty bad.

Phil Trammell

This seems to me like an extension of the messy middle case. One of the ways in which the messy middle might only be bad in a narrow range of scenarios isn’t just that it would be easy to redistribute because the pie would be bigger, but because the interest rate would be way higher, and/or, equivalently, the price of everything except human-intrinsic goods would be falling really rapidly. They’re sort of two sides of the same coin. A little bit of savings would turn into a lot of consumption next year.

Things have to go really wrong for us to just get over the threshold of capital being productive enough to automate lots of work, but not be productive enough that the interest rate is high and the price of capital-produced goods is falling a lot. Even without redistribution, a little bit of savings will save a lot of people.

Dwarkesh Patel

You’re saying if the developing countries have some savings in the developed world, that will be enough to produce a lot of surplus that they can then—

Alex Imas

They will now be able to consume a lot using their savings.

Phil Trammell

But the messy middle could be wider in this case. They’re starting from such a lower level in terms of how much they have and how much it’s actually indexed to the global economy. I think it’s important for them to get on it now. I don’t have strong feelings about whether it should take the form of sovereign wealth funds that invest in the right supply chains or just subsidies to their own citizens to buy a little bit of—

Dwarkesh Patel

This is actually a crucial point. We were talking earlier about why the Rockefellers of the world, why their descendants don’t control everything, if our argument about the selection of these greedy optimizers holds. One argument is just that it’s very hard to index the economy. Maybe they would’ve just decided to have their heirs index the economy and have their wealth grow at the rate of economic growth, and their heirs would be trillionaires by now.

Before index funds existed, it was just very hard. A very small fraction of the economy, going back 100 years, accounts for a majority of the value created now. If you missed those particular things, your wealth would’ve just stagnated.

Maybe there was a brief golden window from the creation of index funds up until five years ago where you could actually index the economy and have your wealth grow at the rate the economy grows. But now we’re in this world with very concentrated returns, especially to private companies. As we were making the point in our blog post, this is capital that the average person has disproportionately less access to. Most of their capital is having a random house, at least in the US.

Phil Trammell

Or a part of a house.

Dwarkesh Patel

Which, as we were saying, is capital that is uniquely ill-suited to be complementary to the production of AI or the serving of AI or to robots.

Phil Trammell

Or the kinds of goods that the rich will bid up the prices of.

Dwarkesh Patel

Exactly. What is the value of a house currently? It’s that the land is close to other humans and modulo relational stuff that is just not going to be the main factor of production in the future.

Alex Imas

This would be why a Georgist tax would not raise enough money for the sort of programs that we will be discussing.

Dwarkesh Patel

Right. But stepping back, the point I was trying to make is, if it gets harder to index the economy now, and that is the main way in which normal people are supposed to—modulo some sort of universal basic income—

Phil Trammell

In the developed world.

Dwarkesh Patel

—are supposed to have some purchase on the wealth from AI. And it’s also the way that developing countries are supposed to have some purchase on the wealth gains from AI.

But it’s very hard. Does Nigeria own a lot of SK Hynix and Anthropic? I’m guessing not. It’s not enough for them to just own the S&P 500.

Alex Imas

This brings up a really important point. Is AI going to be like electricity or social media?

Think about ConEd, or whatever the electricity provider here is. It’s a monopoly. It provides a resource that everybody uses. But do we think about electricity as creating a concentration of power? Does ConEd have this huge amount of political power, social power, or something like that? No, because with electricity, a lot of the downstream benefits actually came to the users of the electricity rather than the actual entity producing it.

On the other hand, with social media, it was the opposite case. Social media was everywhere. Everybody uses social media, but the rents went to the platform.

Dwarkesh Patel

That’s a really interesting point. I don’t endorse this take yet, I’m going to talk out loud. The more you think our economy is going to be run on AGI the way our economy currently runs on electricity—that is, there’s a broad fundamental transformation of the entire economy—the more it looks like electricity… Every company in the S&P of the future, if it’s going to make it to the S&P 500, it is because it has leveraged AI.

Alex Imas

Exactly. And then you’re indexed again.

Dwarkesh Patel

But then again, I guess if you just look at how concentrated the S&P is over time, just these big tech companies much more so… I guess this goes to a fundamental point that it’s hard to reason about how much of the gains from AI these individual private companies will be able to control.

Alex Imas

I think the open model thing is going to be a big point here. If we’re indeed in a world where the open models are six months behind the frontier—or nine months—then we’ll hit AGI, we’ll hit whatever, and in six months, everybody has access to this resource.

Dwarkesh Patel

This goes to show you that every question is connected to every other. That question about whether there’s runaway gains connects to questions about recursive self-improvement. Even if not recursive self-improvement, then continual learning, or online learning, which lets a model learn on the job. So if it’s deployed, it gets to learn more. These are just forecasting technical questions which then impact whether Uganda will have any purchase on the returns of AGI.

The reason I’m emphasizing the question is I think both for the messy middle and for developing countries, a recommendation that is often made naively is that you’ve got to do some kind of retraining. You’ve got to do some kind of jobs program, or you’ve got to have them build data centers in your country. I think you guys are suggesting something closer to just buying the index of AGI. That’s probably a much cleaner strategy and much more likely to succeed.

Alex Imas

These are the two scenarios. I think there is a world where it is concentrated, in which case it’s going to be really hard to index AGI. There is another world where it is electricity. Basically every company has access to AGI. So you just buy the index. Nigeria just needs to buy the index, and Nigeria has access to AGI because of the open models.

Phil Trammell

Just to get back to the question about whether to go with retraining or trying to index. I would prioritize trying to index, just given how fast AI could hit the world. But I definitely wouldn’t just rely on that.

In the messy middle cases or the long-timeline cases where we don’t get anything like AGI all that soon, it would be leaving a lot of value on the table if you could have retrained to be a bit better educated on how to use the latest wave of computing. I don’t think there’s that much of an either/or there.

Dwarkesh Patel

Maybe the reason to be pessimistic about this is because one of the reasons a country is poor is that it has a bad education system, so becoming the best in the world at retraining people at using AI doesn’t seem like a particularly promising strategy for that poor country.

Alex Imas

Although there are cases where, in developing countries, you had this leapfrogging effect with, for example, mobile banking. It’s much more prevalent in Nigeria than it is in Germany. Everybody is doing mobile banking. They have it on their phones, and they’re constantly doing this sort of thing.

Again, I’m not putting probabilities on this, but with a transformative technology like AI, you could get leapfrogging where you skip the step in the middle and get really astronomical growth.

Phil Trammell

Just about the ease of indexing, I think it’s definitely something to worry about a bit and keep an eye on. But as discussed in our own essay, and as other people have pointed out, it’s already not that hard to index. There’s been a bit of an increase in the privatization of returns, but still, well under 20% of the total market cap of non-tiny companies in the US is private.

Everyone thinks about OpenAI and Anthropic. If that’s where all the wealth will accrue, then all these questions about whether open models will stay only a little bit behind, those are important. But even they look like they’re going public before too long, probably. The frictions that have been keeping companies from going public might themselves be alleviated by AI a lot, just all of the disclosure requirements and whatnot. They want to get access to more potential investors, too.

If I had to guess, I would guess that the long general trend of lowering those frictions and making it easier for more and more people to index will continue, despite the recent bump in the other direction.

Dwarkesh Patel

This actually makes me hope even more so than before that the labs do get commoditized, or at the very least they go public as soon as possible. But hopefully they just get totally commoditized. I think AI will be much more popular and, more importantly, will be much more likely to lead to broad increases in prosperity if it is as hard to capture the gains of AI as it is to capture the gains of electrification.

Alex Imas

Exactly. There’s no anti-electricity people out there.

Dwarkesh Patel

I mean electricity doesn’t take your job, but—

Phil Trammell

Well, it takes some people’s jobs.

Dwarkesh Patel

It took some people’s jobs, yeah.

Alex Imas

This is maybe tangential to the conversation but I think narratives matter. There’s this really negative narrative around AI right now, but that’s because people are not putting out the positive narrative. There’s a reason. It’s more difficult to imagine a good thing that doesn’t exist than losing something that exists. It’s much easier for somebody to go on a podcast and say, “These jobs that you like, they’re going away,” than for somebody to spin up a utopia which doesn’t exist yet.

Phil Trammell

I hope this isn’t too out of left field, but I would be remiss if I didn’t point out one big cost of having commoditized frontier AI models, which is the tech race dynamic. For safety purposes, you might want fewer frontier companies so that each one has a buffer in case they want to slow things down to make things safer.

The way this relates to our point before about the widespread access of the returns, is that I think there’s a lot less of a trade-off there than some people imagine. Some people think either frontier AI gets commoditized and we all enjoy the benefits—but there might be some risk, because the market’s really competitive and cutthroat—or things are safer because there’s a big gap between the leader and the laggard. But that means the leaders get fantastically wealthy? No. You could just have a relatively big gap, but it’s a public company, and ownership in it is widely distributed.

Dwarkesh Patel

More recently, I have been thinking that the risk of commodification—which is that it diffuses the ability to use AI to harmful ends—is worth the benefit. I worry that having these concentrated labs not only makes it so that the surplus isn’t as widely distributed through society, but also creates a very tangible, clear political target for the government. We saw this with the Defense Production Act threat against Anthropic. If there wasn’t one lab, or a couple of labs, that are clearly ahead of others, this kind of threat would be much harder to make.

Thank you guys for doing this. I feel like there’s a lot of unresolved questions, but it is helpful to know what the first branch is along all these important dimensions.

Phil Trammell

Great.

Alex Imas

Thank you.

New blackboard lecture with Reiner Pope: how do chips actually work - starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do.

Reiner is CEO of MatX, a new chip startup (full disclosure - I’m an angel investor). He was previously at Google, where he worked on software efficiency, compilers, and TPU architecture.

Watch this one on YouTube so you can see the chalkboard.

• Crusoe was one of only five GPU clouds that made the gold tier in SemiAnalysis’ most recent ClusterMAX report. Gold-tier providers like Crusoe delivered 5-15% lower TCO than silver-tier clouds, even with identical GPU pricing. This is because optimizations like early fault detection and rapid node replacement don’t necessarily show up in the sticker price, but still matter a ton in the real world. Learn more at crusoe.ai/dwarkesh

• Cursor is where I do most of my work—from reading research papers to visualizing technical concepts to coding up internal tools for the podcast. Most recently, I used it to build two different review interfaces for my essay contest, one that anonymizes submissions for scoring and another that lets me see applicants’ essays next to their resumes and websites. Whatever you’re working on, you should try doing it in Cursor. Get started at cursor.com/dwarkesh

• Jane Street let me ask Ron Minsky and Dan Pontecorvo, two senior Jane Streeters, a bunch of questions about how they use AI. We discussed everything from the types of models they’re training to how they think about the future of trading to why they’re more bullish than ever on hiring technical talent. You can watch the full conversation and learn more about their open positions at janestreet.com/dwarkesh

00:00:00 – Building a multiply-accumulate from logic gates

00:16:31 – Muxes and the cost of data movement

00:26:10 – How systolic arrays work

00:39:11 – Clock cycles and pipeline registers

00:51:51 – FPGAs vs ASICs

01:03:25 – Cache vs scratchpad

01:07:27 – Why CPU cores are much bigger than GPU cores

01:12:00 – Brains vs chips

01:15:33 – A GPU is just a bunch of tiny TPUs

Dwarkesh Patel

I’m back with Reiner Pope, CEO of MatX, a new AI chip company. Last time we were talking about what happens inside a data center. Now I want to understand what happens inside an AI chip. How does a chip actually work? Full disclosure, by the way: I am an angel investor in MatX. So hopefully you have designed a good chip.

Reiner Pope

Hope so. I’ll start with the smallest fundamental unit of chip design, and we’ll build up to what an actual production chip is and what its components are. At the very bottom level of a chip, the primitives we work with are logic gates, very simple things like AND, OR, and NOT. These are connected together by wires that have to be laid out physically as metal traces on a chip.

The main function that AI chips want to compute is the multiplication of matrices. Inside that, the fundamental primitive is a multiply-accumulate of pairs of numbers. We’re going to demonstrate what that calculation looks like by hand, and then infer what a circuit would look like for that.

It’ll be easiest if I do a multiply-accumulate of a four-bit number with another four-bit number. The clearest primitive is actually multiply-accumulate. So there’s a multiply of these two terms, and then we’re going to add in an eight-bit number.

Dwarkesh Patel

Can I ask a clarifying question? Why is this the natural primitive for whatever computation happens inside a computer?

Reiner Pope

There are a few reasons. It’s a little bit more efficient, but the reason it’s natural for AI chips is that if you look at what’s happening during a matrix multiply… What is a matrix multiply in short? There’s a for-loop over i, over j, and over k, of output [i, k] += input [i, j] x other input [j, k]. A multiply-accumulate happens at every single step of a matrix multiply.

The other observation is that the precision will almost always be higher in the accumulation step than in the multiplication step. This is specific to AI chips. You’re multiplying low-precision numbers, and then when you accumulate, errors accumulate quickly, so you need more precision there. This is why we’ve chosen to do a four-bit multiplication and an eight-bit addition.

Dwarkesh Patel

Let me make sure I understood that. There are two ways to understand that. One is that the value will be larger than the inputs. The other is that if it was a floating-point number it would be… Maybe that part is less intuitive to me. But maybe it’s the same principle?

Reiner Pope

It really is the same principle. The separate principle is that as you’re summing up this number, you’re summing up a whole bunch of numbers, so you’ve got a lot of rounding errors accumulating. Whereas in this case, there’s only one multiplication in the chain, so there aren’t a lot of rounding errors accumulating in the multiplication.

Dwarkesh Patel

Why are you summing up a whole bunch of numbers? There’s just two numbers there.

Reiner Pope

This summation is repeated j many times.

Dwarkesh Patel

Any errors accumulate. I see.

Reiner Pope

So how would we perform this calculation by hand? As a human, we would probably separate it into two, but we can do it all in one using long multiplication.

For the multiplication term first, we’re going to multiply this four-bit number by every single bit position in the other four-bit number. We write that out. First, 1001 multiplied by this bit position. That is the number itself. Then shifted across by one, we’re multiplying by 0. That gives us an all-0 number. Shifted across one more to multiply by this one, we get 1001. Finally, for this last bit position, we get an all-0 number again.

This gives us a bunch of terms that we have to add for the multiplication. While we’re doing that summation, we might as well add in the actual accumulator term as well. So we just copy that directly across. So this is the sum. It’s a five-way sum that we want to compute.

What logic gates did it take us to get to this intermediate step? We needed to produce all 16 of these partial products. How do I produce one of these partial products? Let’s take this number 1, for example here. We produce it by multiplying this number by this one over here. We can produce that with an AND gate. This number is 1 if both this bit is 1 and this bit is 1. If either of them is 0, then the multiplication of 0 times anything is 0. To produce all of this, we ended up consuming 16 AND gates. In the general case, if I were doing a p bit multiply times a q bit multiply, this will be p times q many ANDs.

Finally, I sum them. Most of the work is going to happen in the summing. Let me describe the other logic gate that we use here. AND is almost the simplest logic gate that exists on a chip. It’s almost the smallest. At the other extreme, the very largest logic gate you’ll typically use is something called a full adder.

Coming from software, you might think that a full adder adds 32-bit numbers together. In this case, it just adds three single-bit numbers together, so you can think of it as adding 0, 1, and 1 together. When I add these together, the result can be 0, 1, 2, or 3, so I can express that in binary using just two bits. As input, it has three bits. As output, it has two bits. The number 2 in binary is 10. This is also known as a 3→2 compressor because it takes three bits of input and produces two bits of output.

Dwarkesh Patel

Just to make sure I understood: the two inputs are an X and a Y value and then some carry that came in…

Reiner Pope

The three inputs are all bits in the same bit position, like three bits in a column here. The two outputs, I’ve drawn them vertically here and horizontally here to match this vertical versus horizontal layout. This expresses that things in the same column are in the same bit position, whereas things in adjacent columns are different. This is a carry out, whereas this was the sum.

Dwarkesh Patel

So if the inputs in the full adder were, say, 101, then the output would be 10. If it were 111, it’d be 11. If it were 000, it’d be 00. If it were 010, it’d still be 01. Got it.

Reiner Pope

Yeah. It’s essentially just counting the number of things and expressing that in binary. This circuit captures what we as humans naturally do when we’re summing along a column.

I’ll show one iteration of using the full adder to sum. The way I sum here is going to be a little unnatural for humans. We would sum along the column and then remember the carry, but instead of remembering the carry, we’ll explicitly write it out. We proceed from the rightmost column toward the left. On the rightmost column, we sum the 1 and the 1, and that produces a zero here and a carry of one. We’ve used this full adder circuit on this pair of bits and produced a pair of bits as output.

Now we can do the same thing with this column. We have a column of four numbers, so we’ll take the first three of them, run a full adder on them, and that gives us a 0 and a 0 as output. The sum of these is 00. That’s the full adder applied to all these bits. As I’ve used up bits, I’ll cross them out to indicate that I’ve handled them.

Let’s keep going a little bit more. I take these three numbers, I add them, and that gives me a 1 and a 0. I’ve dealt with these three numbers. Now I take these three numbers and add them, and that gives me a 1 and a 0, and I’ve dealt with these numbers.

The way to view this is that I have this whole grid of numbers that need to be added. I’m going to keep applying full adders to all the bits here, constantly removing three numbers from a column and writing out two numbers as output. Keep going over and over again until I eventually get just one single number coming out.

This approach is called a Dadda multiplier. This is the standard for how you do area-efficient multipliers using full adders. Let’s try to quantify the circuit size of this so we have a sense of how big things are and can compare them later.

How many full adders did I use? How many numbers did I start with? I have the 16 partial products, which is a product of all of these terms with all of these terms, plus the eight terms that I’m adding here. I started off with 24 bits. Eventually I produced eight bits on the output. In every step, I was crossing off three numbers and writing two numbers out as a result.

Every single use of a full adder eliminates one of the bits here. So how many full adders? It must be 24 minus 8, so there were 16 full adders in this circuit. This is true in the general case as well. There will be p times q many full adders in this circuit.

Dwarkesh Patel

Let me make sure I understand the logic of that. The input bits, 24, is p x q, plus p + q. The output bits are just p + q. So p x q plus p + q, minus p + q equals p x q.

Reiner Pope

That’s right. I think this explains, or at least hints at, the second reason we chose to do a multiply-accumulate. The first reason is that it’s what shows up in matrix multiplication. The second is that it gave us this very slick, simple p x q, very simple algebra.

We’ve described this whole procedure. Every single atomic step that I took here becomes a logic gate, and then the wires are connected together. When I had these three inputs that I used to produce these two outputs, if I think of mapping this to a physical device, there would be a wire connecting all three of these things together into a logic gate that produced this output.

This is the main primitive, at different bit widths, that’s inside an AI chip. We’re going to build up from here to how you would use it to run all the other operations you might want.

Dwarkesh Patel

This might be the wrong time to ask, but whenever Nvidia reports that this chip can do X many FP4 or half as many FP8, it seems to imply those circuits are fungible, that there’s not a dedicated FP4 versus FP8. But the way you’re mapping it out here, it seems like if it has to be mapped out in the logic, you would need a dedicated FP4 multiply-accumulate and then a dedicated FP8 accumulate. Can you “funge” them?

Reiner Pope

As drawn, they’re not particularly fungible. This is actually one of the main choices you have to make when designing a chip: how much of FP4 and how much of FP8 do I have? Sometimes I’ll make that consideration from the point of view of the customer requirement. Another angle is to equalize the power budget between FP4 and FP8.

Dwarkesh Patel

When they report those numbers and it just happens to be the case that it does 2x as many FP4 as FP8, they’re just choosing to give equivalent die areas to all the floating points, and as a result it ends up being…?

Reiner Pope

Why is the ratio exactly 2x? Part of it is that surely it won’t be exactly equivalent to die area. There’s a data movement reason. We’ll maybe come back to this when we look at how it goes into and out of memories. There’s something really nice from a software level about the fact that I can pack two four-bit numbers into the same storage as an eight-bit number. When I store that to memory, the sizing of the buses that I wire out within the chip makes that work out really nicely.

Dwarkesh Patel

Come to think of it, it’s not just 2x. The amount of area it takes sounds like it’s quadratic with the bit length. That’s why smaller precision is even more favorable than you might think.

Reiner Pope

This is a really big reason. In fact, Nvidia made a change. Historically, up until B100 or B200, every time you halved the bit precision, you doubled the FLOP count. For the reason you said, because of this quadratic scaling, that ratio is actually slightly wrong. You should get an even bigger speedup than you might otherwise think. Nvidia’s product specs have started acknowledging that in B300 and beyond, where the FP4 is three times faster than the FP8.

Dwarkesh Patel

Though it should be 4x.

Reiner Pope

Yeah. What I’ve shown here is the simplest case of integer multiply. When you’re dealing with floating point, as you do in FP4 and FP8, there’s this other term, the exponent, that complicates the calculation.

What can we see already from this? I think the big observation you’ve made is that there’s this quadratic scaling with bit width, which is very effective and is the single reason low-precision arithmetic has worked so well for neural nets. The other thing we’re going to do now is compare the area spent on the multiplication itself with all the circuitry around it.

Reiner Pope

We’ll walk back in time a little bit and see how GPUs prior to Tensor Cores worked, which is in fact the same way CPUs worked. Where do we stick this multiply-accumulate unit? Generically, I’ll describe a CUDA core or a CPU. You’ll have some register file which stores some number of entries, maybe eight entries of, in this case, 4-bit numbers, but typically 32-bit numbers.

Inside the CUDA core, I’ll have some register file of some depth, and then I’ll have my multiply-accumulate circuit. What it’s going to do is take three arbitrary registers from this register file, perform the multiply-accumulate, and then write back to the register file. It’s going to write to this one, but it was able to read from this one, this one, and another random one. It will take three inputs like this.

This is the core data path of many processors. Most processors look like this. You’ve got some set of registers, and then you’ve got some set of logic units, or ALUs. We want to analyze the cost of the data movement from the register file to the ALU and back.

Ultimately, there’s going to be some circuit that says, “Well, I don’t always have to select this guy. I might select any of the registers at any point in time.” The first question is: how can I build a circuit? The circuit I’m going to look for is a mux. In this case, it’s going to have eight inputs, one from each entry of the register file, and it’s going to have one output, which is actually producing this output.

What is the cost of this thing? All we have to build it out of is AND and OR. How do we build it? We do the dumbest thing possible. We form a mask. When we want to read the third entry, we’re going to AND every single entry with either 1 or 0 based on whether that’s what we want to read, and then we’re going to OR all of them together.

Dwarkesh Patel

Just to make sure I understand the basics. What the mux is doing is just selecting an input?

Reiner Pope

Just selecting, invisible to software. You say “I want input number three,” and that means there’s a mux here.

So what is the cost of this mux? An n-input mux operating on p bits. I have n rows. That’s just eight rows, and each row is p bits wide. I have to AND every single bit, so I get n x p many AND gates. For every single input I have to decide whether I’m going to mask it out or not. Then I’m going to OR them all together.

There’s going to be n – 1 times p many OR gates. I’ve got all of these different things, almost all of them are 0s, but I need to collapse them from my eight options down into one option. Every step, I need to OR one row into an existing row.

Dwarkesh Patel

It’s actually funny that you don’t think at the level of hardware. You just think, “Oh, I’ll just select element three,” and something as simple as that is in and of itself quite a complicated circuit.

Reiner Pope

This is the first step of all of the hidden data movement costs that show up. We’re just going to compare. I have to pay this cost. I’ve got one mux here, and in fact I have two more copies of that for each of the three inputs to my multiply-accumulate operation. I have this cost, which is 3 x n x p AND gates over here, compared to p x q gates in the actual circuit that is doing the thing I care about.

If we plug in actual numbers, with n being eight, I get 24 x p gates just in the data movement, compared to—if q is four—4 x p gates just in the multiply-adder.

Dwarkesh Patel

Where is the three coming from?

Reiner Pope

Three different inputs here. What I’m hinting at is that all of this work, which scales as the size of the register file—and this is a very small register file—all of this work just moving the data from the register file to the logic unit is many, many times more expensive than the logic unit.

Dwarkesh Patel

It may be helpful to just see what a mux looks like, maybe a two-bit or a four-bit mux.

Reiner Pope

We’ll do a two-way mux. We’ve got two different numbers, we’ve got these two inputs. These are the inputs that are being selected between, and the selector can either be “I want this one” or “I want the other one.” This is a one-hot encoding.

This is what we start with. Let’s focus on this case. This is the actual input we got, and we want to produce this guy as the result. Very laboriously, we AND this bit with all of these. That produces ANDing this bit with this row. Likewise, we AND this bit with this row. That produces all 0s. There are four ANDs here.

Finally, we OR these two together, and this gives a 1. We OR these two together, this gives a 1. We OR these two together, it gives a 0. We OR these two together and it gives a 1. Those are the four ORs. This actually ends up looking a little bit like addition. We did exactly the same set of ANDs. We’ve ANDed all of these things together, but then instead of collapsing it by using full adder circuits, we just get a very simple collapsing with OR gates.

Dwarkesh Patel

But that doesn’t look like n times p.

Reiner Pope

This was with n=2 inputs. In the general case, we will have n rows, and we’ll have p bits per row. That gives us the n times p many AND gates. In this circuit I’ve described, almost all of the cost, seven-eighths of the cost, is in reading and writing the register file, and only a tiny fraction of the cost is in the logic unit itself.

This is the problem to solve. This essentially was the state of play prior to the Volta generation of Nvidia GPUs. This kind of thing is what was inside the CUDA cores. This problem statement is what motivated the introduction of Tensor Cores, which are more generically called systolic arrays.

Think about how we’re going to solve this problem. We’re spending almost all of our circuit area on something that we really don’t care about and is hidden to the software programmer, and the thing that we actually care about is not much of the area. Make this one bigger somehow while keeping this at the same size. That’s the goal.

Reiner Pope

The evolution was that we had baked this much into hardware at this stage. This single line is a multiply-accumulate, and this single thing was baked into hardware. The idea of a systolic array is to go two levels of loops up and bake this entire loop out here into hardware. The idea is that if we have a much bigger granularity fixed-function piece of logic, maybe the taxes we pay on the input and output are much smaller.

Dwarkesh Patel

Interesting. It sounds like you’re suggesting that if you go up one step in the matrix multiply loop, you can tilt the balance more towards compute than communication.

Reiner Pope

That’s right. There are two effects we’re going to take advantage of here. One is that we can do more stuff per every trip through our register file. The other is that in some of this loop, we can take advantage of certain things staying fixed.

Visually, we’re going to look at this matrix multiplication. This portion of the loop corresponds to a matrix-vector multiplication. We’ll take a matrix and multiply it by a vector. How do we do this? Every column gets multiplied by the vector and then summed. We’re going to sum along columns.

This 0 and 3 gets multiplied by the 3 and 7 and gets summed, and then the 1 and 2 gets multiplied by the 3 and 7 and gets summed. There is a multiply-accumulate associated with every single one of these entries in the matrix. We’ll draw out these four multiply-accumulates.

Dwarkesh Patel

Just to make sure I understand why there are four multiply-accumulates: each entry in the column that corresponds to the output vector is a dot product, and in this case it will be two multiplications and then the addition of those two multiplications. You’re accumulating...

Reiner Pope

Really there’s only one addition per dot product, but we like to start with zero.

Dwarkesh Patel

But it includes the initialization of zero.

Reiner Pope

Yeah. We want to have quadratically more compute. We have x times y as much compute as we had before. But we want to aim for having only x times as much communication. The intention is to get this advantage term going as y.

We’ve laid down the multiplications. We want to bring in a vector of size two, and that is already in line with our columns target. That’s fine. However, we need to manage the communication of this matrix, which exceeds our budget of x.

The idea is that in an AI context, this matrix is going to stay fixed for a long period of time. We’ve got some register files sitting over here.  The amount of stuff coming out of this register file… this is the term that we want to go as x, in some sense. We don’t want to bring this full matrix in from the register file every cycle, because that would cost too much in terms of wiring from the register file.

Our key trick is that this matrix can be stored locally to the systolic array. We’ll store these numbers 0, 1, 2, and 3 in a gate called a register that physically stores these numbers, and we’re going to reuse these numbers over and over again for a large number of different vectors.

Dwarkesh Patel

The optimization here is that the nature of matrix multiplication is that you can store this square quadratic thing directly where the logic is happening, which has an extra dimension compared to the inputs that you keep swapping in and out.

Reiner Pope

That’s right.

Dwarkesh Patel

This is the nature of what a matrix multiplication is. You do a lot of multiplication to get one value out. A dot product is the result of a lot of multiplications. So that optimization means you can stuff a lot of multiplication in before you get some value out of it.

Reiner Pope

That’s right. Just to complete the picture of concretely how that looks: I swapped the 3 and the 2 here. Just like this 0 and 3 is going to multiply by the 3 and the 7, we’re going to form a dot product along columns here. We’re going to feed a 3 and a 7 in here. This feeds into this multiplication and also feeds into this multiplication. Likewise, the 3 feeds into here and also into here. Then we’re going to sum along here. Starting at the top of a column, we feed in 0s, and then coming out the bottom we get results.

Visually, there’s a dot product performed along columns in a matrix, and that maps exactly to what is done spatially in the systolic array. This is one dot product summed vertically, and this is a second dot product also summed vertically.

What is the data that needs to go into and out of the register file? We have x amount of data coming out on the output, and we also have x amount of data coming from the input. With respect to the input and output vectors at least, we’ve met our goal of having only x as much data going in and out of the register file.

This leaves open the question: I said the weight matrix is stored locally in the systolic array, so how did it get there in the first place? At some point, you need to boot your chip and populate this data, so where did that come from?

The trick is that we just do it very slowly. We very slowly trickle-feed it into the systolic array. The simplest strategy is that we run this daisy chain: feed a number in here, and on the next clock cycle it will move down to the next entry of the systolic array. We can do that in every column in parallel, this is also going to come from here, and that gives us another factor of approximately x units of bandwidth coming in.

Dwarkesh Patel

Would you mind repeating that sentence one more time?

Reiner Pope

We know that we’re going to be bringing numbers only rarely into the matrix. We just want to come up with any construction at all such that the amount of wiring that crosses the boundary of the systolic array is bounded to x and not go as xy.

A particularly simple strategy is that we bring a number into the top row of the systolic array in one clock cycle. Then, for y consecutive clock cycles, we bring in the top row every time and shift all the other rows down by one. This keeps the wiring that needs to come from this expensive register file only down to a factor of x rather than xy.

Dwarkesh Patel

I see. There are two questions in terms of communication: communication time and communication bandwidth. You’re saying that since we’re only going to be loading this in once, let’s minimize bandwidth, because bandwidth equals die area. We load it in slowly over smaller lanes because we’re just going to keep this value in there for a while.

Reiner Pope

Exactly.

Dwarkesh Patel

It’s interesting to me that when we were talking last time about inference across many chips, the big high-level thing we’re trying to optimize for is increasing the amount of compute per memory bandwidth, that is to say, per communication. Here also, we’re trying to increase the amount of actual multiplies or additions relative to transporting information from registers to the logic. In both cases, you’re trying to maximize compute relative to communication.

Reiner Pope

This shows up all the way up and down the stack. This is close to the bottom, to the gates. There’s a version that’s maybe even closer to the gates in the precision of the number format that you choose to use. We saw that same effect. There’s a squared versus linear term going on both purely in the precision of the ALU, but also in the size of the matrix.

This unit is the next bigger unit. We had the multiplication circuit, and on top of that we have a pretty large systolic array. I drew it as 2x2, but older TPUs were described as 128x128 of this circuit shown here. This ends up being the most efficient known circuit for implementing a matrix multiply.

Dwarkesh Patel

We’ve talked about how it seems obvious that you should try to maximize compute relative to communication. What are non-obvious trade-offs that keep you up at night, about whether you should do X or Y and it’s not obvious what the answer is?

Reiner Pope

Most of the decisions in chip design are sizing decisions. Already in what we’ve drawn so far… AI chips all have this circuit in them. They have a systolic array, and somewhere near it a register file providing inputs and outputs.

Even within this scope, the sizing questions you have are: how big should I make my systolic array, and how big should I make the register file? These two questions are coupled. One way to think of it is to set a budget for what percentage of chip area you want to spend on data movement. Maybe I say that I want this to be 10% and the systolic array to be 90%.

Then I can size my register file. Bigger register files are more flexible. They allow me to get more application-level performance out, but they take away from the area spent on the systolic array.

Dwarkesh Patel

Where does the clock cycle of a chip come in? What determines what that is? And what is a clock cycle of a chip?

Reiner Pope

At baseline, it’s worth observing that chips are incredibly parallel. You’ve got 100 billion transistors in a chip. A key thing you need to do whenever you have massive parallelism is synchronize between the different parallel units.

In software, typically you have these very expensive synchronization methods like a mutex. One thread will finish what it’s doing, grab a lock stored somewhere in memory, and notify the other thread that it’s done. On chips, we take a very different approach. Every nanosecond or so, all circuitry in the chip will pause for a moment and synchronize. That is the clock cycle. The entire chip typically goes in lockstep to the next operation in one fell swoop.

What this looks like in circuitry is that the clock is mediated by registers, which are these storage devices we’ve drawn elsewhere. The way to think of it is: I have some storage holding a bit, which might be 0 or 1. Then I have some cloud of logic, which maybe is this systolic array or multiplier. I have a bunch of inputs feeding into this cloud of logic, and eventually there’s going to be some output register that it writes to.

There is a global clock signal driving all these registers. At a certain instance in time, when the clock strikes, whatever value happens to be on that wire at that instant is what gets stored.

The challenge is that I would like to have my clock speed run as fast as possible. If I run at two gigahertz, I get twice as many operations done per second as if I run at one gigahertz. But what that ends up meaning is that I’m very sensitive to the delay through this cloud of logic, because any computation happening in there needs to finish before the next clock cycle hits. A major point of optimization on any chip is to make this delay as short as possible.

Dwarkesh Patel

Interesting. The constraint here seems to be that if you add too much logic, you might risk missing the clock cycle. But if you don’t add enough, you’re leaving potential compute on the table. Is there ever a situation where you take a probabilistic chance that a computation finishes, or is it strictly that it either finishes by the clock cycle or it doesn’t?

Reiner Pope

In standard chip design, you margin it such that there is a probability, but it’s many standard deviations out. For all intents and purposes, it is a reliable part and will always meet the clock.

There are some weird exceptions, like clock domain crossings where you go from one clock to another. Then you actually do have to reason about this probability. But in the main path, you margin it such that you’ll get there 25% of the clock cycle in advance, making it very unlikely that it misses.

Dwarkesh Patel

Where the clocks synchronize, where the registers are, is this something you determine as a chip designer? Or is it an artifact where you want a certain sequence of logic, and the software you use to convert your Verilog into what you send to TSMC just determines that to make it work, you have to put a register here, here, and here, making sure no single step makes the whole chip’s clock cycle longer than it has to be?

Reiner Pope

Inserting them is actually a huge part of the work of designing a chip. It’s done by a combination of manual and automatic methods.

To show the very dumb version of what you can do here, you can take this logic and split it in half. Instead of just one cloud of logic, I can have two smaller clouds of logic that do the same thing, but split them up by a register. If you split it in the middle, you can hit twice the clock frequency. That’s great, you get twice the performance, but at the cost of an extra register, which means more storage.

Dwarkesh Patel

Stepping back, why do we need to synchronize the whole chip? If you imagine playing Factorio or something, there’s no global clock cycle. Things are just done when they’re done. There’s iron on the plate, and you can take it if you want.

Reiner Pope

Taking that analogy, the thing you need to be mindful of is if I have two different paths through some logic. Say I have to do computation f here and computation g here, and they’re going to meet for computation h.

There’s going to be manufacturing variance. In some chips f will take a little longer; in some chips g will take a little longer. If I have a signal propagating through, and the results from f and g have to meet up at h, what can go wrong is that f gets there early and it meets the previous value of g, or the next value of g.

Dwarkesh Patel

Ah. And h needs to know when to start, when the next iteration has…

Reiner Pope

Exactly.

Dwarkesh Patel

This explains why different chips made at the same process node, the same TSMC technology, can have different clock cycles. Two chips made at 3 nm might have different clock cycles based on whether they were able to optimize to ensure no single critical path is so long that it slows down the whole chip’s clock cycle.

Reiner Pope

That’s right. This optimization I showed here is called pipeline register insertion. We’ve inserted a register in the middle of the pipeline. This is a pure trade-off between clock speed and area. That is the easy case. There is a harder case too. I drew out a pipeline of logic, but in other cases you may have some calculation which actually feeds back in on itself. It runs some function f and then writes back to itself. For example, this might be an addition where you’re adding a number every clock cycle. This little circuit essentially sums all the numbers presented on different clock cycles.

The challenge is, if this plus takes too long, what can I do? If I try to put a pipeline register right in the middle of it, it changes the computation that’s done. Instead of forming a running sum of everything that comes in, I will actually have two different running sums. I’ll end up with a running sum of the even numbers and a running sum of the odd numbers. This constraint—where I have a loop in my logic, which all chips have somewhere—is the hardest thing to address and sets the clock cycle.

Dwarkesh Patel

I don’t understand why it would be a problem to have that. I’m not even sure what it would mean to have a register there. Is it a sort of atomic operation?

Reiner Pope

Well, plus is not really atomic.

Dwarkesh Patel

As you just demonstrated.

Reiner Pope

It took a whole lot of work to do a summation. You can take the early parts of that work, stick a register in the middle, and then take the late parts of that work.

Dwarkesh Patel

Okay. TSMC offers a PDK which specifies the primitives of logic they can grant you in the chip. It’s up to them to determine that no primitive is bigger than the clock cycle they’re hoping a process node targets. But other than that, can’t you just take all the primitives from TSMC and keep adding registers between them as much as needed until you get to your desired clock cycle?

Reiner Pope

As a logic designer, the chip architect sets the clock cycle. For example, the primitives you get from TSMC are on the order of AND gates or full adders. It depends a lot on voltage and which library you choose, but generally you can have about 10, 20, or 30 of these sequentially in a clock cycle. These primitives are very fast, maybe 10 picoseconds.

As a logic designer, in principle, if you just had a register and an AND gate in a loop, you could get an insanely fast clock speed, more than four, five, or six gigahertz. But if you take this really simple circuit and look at the area you’re spending here… This is called one gate equivalent in size, so unit of one in area. This thing is maybe a unit of eight in area.

Again, almost all your cost becomes synchronization or communication cost compared to the actual logic. This would be a case where you’ve gone too far. You’ve made your clock speed really fast at the cost of spending almost all of your area on pipeline registers.

Dwarkesh Patel

Interesting. So you’re hinting at a dynamic where you can have a really fast clock speed but you’re not getting that much work done. You can have low latency but low throughput.

Reiner Pope

It hurts your throughput, in fact, because the throughput of your chip is the product of how much you get done per clock cycle—which is based on area efficiency—times how many clocks you get per second.

Dwarkesh Patel

This is actually so similar to the thing we were discussing last time about batch size, where if you have a low batch size, any one user can receive their next token really fast, but the total number of tokens processed in, say, an hour will be lower than it could otherwise be.

Reiner Pope

Exactly. You get less parallelism out if you drive your clock speed up really high.

Dwarkesh Patel

I remember talking to an FPGA engineer at Jane Street, Clark, who helped me prep for the previous interview we did together. He was explaining why they use FPGAs. I imagine that for high-frequency trading, throughput is less important than latency, so having very specific control over the clock cycle in a deterministic way is the most important thing. Maybe it’d be interesting to talk about why you can’t just achieve that with an ASIC, or why you might use an FPGA to have deterministic clock cycles for high-frequency trading.

Reiner Pope

Let’s consider the business case for an FPGA versus an ASIC. FPGAs and ASICs use largely the same conceptual model. You have a series of gates built from small primitives—ANDs, ORs, XORs—connected together with wires running in a fixed clock cycle. Anything you can express in an FPGA you can express in an ASIC too. It will be about an order of magnitude cheaper and have better energy efficiency on an ASIC than an FPGA.

The trade-off is that the first FPGA costs you $10,000, whereas the first ASIC you make costs $30 million because it requires an entire tape-out. The business use case for an FPGA is when you want something that has very deterministic latency, fast runtime, and high parallelism, but you are going to change the workload frequently, maybe every month. You don’t want to pay that tape-out cost every time.

How does an FPGA actually emulate the ASIC programming model in a fixed piece of hardware? At its core, it has the two components we just talked about. It has registers as storage devices, and it has lookup tables (LUTs) which provide all of the gates.

Then there’s a third component. We have a swarm of these registers and LUTs, and they are connected by a big set of muxes. In front of every single one of these, we have a mux which selects an input from everywhere else. We have a whole bunch of different options feeding into all of these things.

What this allows is essentially that when I program my FPGA, I can take all of these components and superimpose a particular wiring which goes through this LUT, feed it into another LUT, send it to this register, and then feed it into another LUT, or something like that.

What I’ve drawn in orange is how you… FPGA means Field-Programmable Gate Array. The orange is what has been programmed in the field, whereas the white is all the wires that must exist in the FPGA in order to actually make the device in the first place.

Dwarkesh Patel

What does it mean to be programmed in the field?

Reiner Pope

Programmed in the field means the device is being deployed in a data center. It’s sitting out in the world, and then you can come and program it.

Dwarkesh Patel

Ah, not field as in like electric field. Field as in like out there in the world, ok.

If I look at how the field programming comes out of the first lookup table and goes into a second one, how does that work?

Reiner Pope

Where are the wires that make that happen? I got a little bit lazy in drawing all of these. Every single device here has a mux sitting in front of it, which can select from all of the nearby circuits that are available. The actual configuration of the FPGA amounts to the mux control. In this mux, we have the data inputs, and we have the control that selects.

There’s a little storage device sitting next to every single one of these muxes saying, “This is where you’re going to source your input from.” Programming it consists of configuring every single one of these muxes.

Dwarkesh Patel

That makes sense. What is happening inside of the lookup table?

Reiner Pope

The lookup table is also going to have a little bit of control telling it what to do. Its purpose is to configurably take the role of an AND gate, OR gate, XOR, or any of those different things. There are many ways you could consider doing that. The way it’s done in traditional FPGAs… A lookup table has four bits of input and one bit of output. How many different functions are there from four bits to one bit? There are 16 different functions.

You can tabulate this as 16 different numbers. You’ve got a table of 0111001, 16 entries. This table is stored in this blue configuration bit. It views these four bits as binary, looks up the relevant row of the table, and emits that bit. This is essentially a truth-table view of lookup tables.

Dwarkesh Patel

Okay, so if you think about an AND gate, OR gate, NOR gate, XOR gate, these all take as input…

Reiner Pope

Those are two-input functions. Sometimes we have a three-input function, like a three-way XOR, or a four-way XOR.

Dwarkesh Patel

In this case, does it just depend on how big it is?

Reiner Pope

The typical size for LUTs is four inputs. It’s sort of just a sweet spot. There’s another compute vs. communication trade-off here. If it has too few inputs, you need to use more LUTs.

Dwarkesh Patel

Basically the lookup table is a truth table. With a truth table, you can program in any gate you want. So instead of a lookup table, you can just think of it as a programmable gate.

Reiner Pope

That’s right. One of the things you can do here is you can see where the rule of thumb that an FPGA is an order of magnitude more expensive than an ASIC comes from. You count how many gates would be inside this lookup table.

We can view this lookup table essentially as one of these muxes. It has to select between 16 different values, so it’s a mux with n=16 options and p=1 bits. As we saw earlier, this circuit costs n times p many gates. So it costs np, which is 16, AND gates, and also 16 ORs.

Dwarkesh Patel

This circuit being the mux?

Reiner Pope

Exactly, the mux.

Dwarkesh Patel

The mux that goes into the lookup table?

Reiner Pope

The lookup table itself you can think of as being a big mux that selects from all 16 rows down to one output. That’s the lookup table.

Dwarkesh Patel

But the way you’ve drawn it here, there’s a mux and then a lookup table.

Reiner Pope

It’s muxes all the way down. There is a second mux that is inside here. This mux is this mux.

Dwarkesh Patel

And the other mux is just saying…

Reiner Pope

where it came from in this mess of gates.

Dwarkesh Patel

Right, and the second mux is, “Okay, now you have one value, but that value is still a four-bit value.”

Reiner Pope

Yeah, I’ve selected four bits from the soup. Then I use those four bits to select which entry in the lookup table I’m going to use.

Dwarkesh Patel

Suppose in the first mux you’re pulling from eight nearby registers as input. That’s a total of 32 bits going in. Out of that, four bits come out. Those four bits go into the second mux, which is inside the lookup table.

Reiner Pope

In this case, these registers are single-bit registers. If there are eight nearby registers and lookup tables, then I have eight bits total coming in nearby. I select from eight down to four different values. There are actually four different muxes, a little mux associated with each of these input bits. Each of them is selecting one out of eight.

Dwarkesh Patel

Where are those eight coming from?

Reiner Pope

Nearby registers and other LUTs.

Dwarkesh Patel

Each register is one bit.

Reiner Pope

Yes.

Dwarkesh Patel

I guess AMD or whoever makes these FPGAs still has to be opinionated about which registers are connected to which registers. You can program in the actual gates, but they add a wire in the connect… the communication topology, right?

Reiner Pope

You get flexibility at a local grain. There’s a nearby neighborhood you can select from, but for more coarse, long-distance connections, they form an opinion on that.

Dwarkesh Patel

And the reason it’s 10x slower is why?

Reiner Pope

If you look at the cost of building this lookup table, it’s 32 gates. It can give me the equivalent of—what’s an interesting thing I can do here—a four-way AND gate. A four-way AND means AND, AND, and then an AND of an AND. This is a circuit I could implement in an ASIC directly using three AND gates. Using a LUT, I can also implement it, but it’s going to take 32 gates instead of three.

Dwarkesh Patel

So the overhead is really coming from the fact that there’s a more concise way to describe a truth table than listing out every single possible combination of inputs, which is just to write out the gate.

Reiner Pope

Yes, to place down the polysilicon and the wires and so on.

Dwarkesh Patel

Interesting. One important point you made to me is that the reason they prefer FPGAs to CPUs is that they get deterministic clock cycles. They know when a packet will come in and go out. Why isn’t that a guarantee in CPUs?

Reiner Pope

You can actually design a CPU that has deterministic latency as well. In fact, the processors inside a lot of AI chips also have deterministic latency. Groq has advertised this. TPUs have that in the core as well.

The challenge is getting deterministic latency and high speed at the same time. Non-deterministic latency comes from specific design choices in a CPU. It’s actually possible to remove those design choices and make a CPU with deterministic latency, but those are not very attractive in the market, so people don’t make those CPUs anymore.

In some sense, deterministic latency is a simpler starting point, and some chip designers have added things in to make it non-deterministic. To take a concrete example, probably the most important source of non-determinism on a CPU is the CPU cache itself.

In a CPU, you have the CPU die itself, and then DDR memory off on the side. You have a cache system inside that remembers recent accesses to DDR and stores them. When I’m running through my CPU instructions, every time I have an instruction that accesses memory, it first checks if the data was stored in the cache. If not, it fetches it from DDR.

This is a huge optimization. The cache is two orders of magnitude faster than the DDR. If you never used the cache, basically all programs would run a hundred times slower. The presence of a cache is absolutely necessary for a CPU to run at a reasonable speed.

But whether or not you get a cache hit depends on the ambient environment of the CPU: what other programs are running, what has run recently, and what the random number generator inside the cache system is doing. That is a big source of non-determinism in the runtime of a CPU.

That is the memory system for a CPU. The big thing you can do differently is, instead of having the hardware say, “I’m going to read memory” and then the hardware decides whether or not it comes from the cache, you can bake this decision into software, a different design philosophy.

You see this in TPUs, for example. I’ll draw the same diagram, but I’ll call it a scratchpad. The main difference is… This would be a TPU, and you have HBM in this case rather than DDR, but it’s still an off-chip memory. Instead of the software saying “first access memory” and letting the hardware decide, you have one kind of instruction that goes to the scratchpad and a totally different kind of instruction that goes to HBM.

This style is generically known as scratchpad instead of cache. The key distinction is that you have one kind of instruction that says “read or write scratchpad,” and a totally different instruction that says “read or write HBM.”

Dwarkesh Patel

So scratchpad being the cache.

Reiner Pope

Yeah, this thing here is the scratchpad.

Dwarkesh Patel

Stepping way back: people say computers have the “von Neumann architecture”, where there’s this serial processing of information. Maybe it’s just because we’ve been talking about parallel accelerators, but the FPGA is super parallel. The AI accelerators, the TPUs, are super parallel. Even CPUs are super parallel if you think about all the cores they have. In what sense is modern hardware actually the von Neumann architecture? Is that actually a fair way to describe modern hardware?

Reiner Pope

I think it’s a fair way to describe CPUs. The amount of parallelism you get on a CPU is about 100 cores times maybe 16-way vector units, so about 1,000-way parallelism on a CPU.

Dwarkesh Patel

One question: there is a die being used for the CPU, and if there are fewer threads, just as a matter of transistor voltages switching on and off, is it that there’s literally one control flow—a small part of the die—where voltages are switching on and off?

How do you actually occupy the die area of a CPU…

Reiner Pope

If there are so few cores, what are you spending all of the die on?

Dwarkesh Patel

Yeah, what is happening there?

Reiner Pope

The cores are just much bigger and more complicated. We should compare a CPU core, which takes up one one-hundredth of the die, to a LUT. A LUT is only 16 gates. It’s clear why there are so many more LUTs in an FPGA than cores in a CPU.

But why are there more CUDA cores, for example, than CPU cores? What’s the difference between a CPU and a GPU? Inside the CPU, one big use of the area is the cache. Very little is actually the ALUs. Mostly it’s these register files rather than the logic units. Both of those have equivalents in a GPU, so that’s not a big difference.

But the thing that does not have an equivalent in a GPU is the branch predictor. There is a whole big area in the CPU which is just a bunch of predictors saying when the next branch will be and where the branch target is. Stripping a lot of that out, as well as making these register files tighter, drives a lot of the GPU gains over the CPU.

Dwarkesh Patel

What is the purpose of the branch predictor? To execute both branches at once, or what does it do?

Reiner Pope

The issue is that when I’ve got a series of instructions, if I have a branch, the actual step of processing an instruction takes a really long time. It takes maybe five nanoseconds.

The time to notice that I’ve got a branch, evaluate whether the Boolean is true, update the program counter to the new target, and then read from the instruction memory could take five nanoseconds to finish. So in reality, this may finish somewhere down here. I want to run a clock speed that is much faster than what five nanoseconds allows. Five nanoseconds is a 200 MHz clock speed. I would like to run at one or two gigahertz.

So I need to run other instructions while the branch is being evaluated. I just want to keep running the instructions that happen after me. But that might have been wrong. If the branch ended up being taken, then I need to know that instead of evaluating these instructions, I actually need to jump to wherever the target is and run those instructions instead. The purpose of the branch predictor is to predict, five cycles earlier, that a branch is going to happen, before you even get to that instruction.

Dwarkesh Patel

If I think about how the brain works versus what you’re describing here, at a high level the differences might be that while you can do structured sparsity in these accelerators and save yourself some area that you would have otherwise had to dedicate to gates, in the brain there’s unstructured sparsity. Any neuron can connect to any other neuron, and not in ways where they have the column aligned.

Then there’s the fact that memory and compute are co-located. Although I guess you could say in a way the memory and compute are co-located on these dies too.

Reiner Pope

This is exactly the co-location, in some sense, of the memory and compute.

Dwarkesh Patel

So maybe that isn’t a big difference. Another big difference is that the clock cycle on the brain is much slower than on computers. Partly that’s to preserve energy, because the faster the clock cycle, the bigger the voltage needs to be in order for the signal to settle and to identify what state a transistor is in.

Reiner Pope

That’s right.

Dwarkesh Patel

I don’t know if you have any commentary on what the brain might be doing versus how these chips work.

Reiner Pope

Let’s take the clock speed one first. The clock speed is quite high on a chip because that drives higher throughput. When we compare a GPU running some workload, it’s running batch size 1,000. Whereas the brain is not running batch size 1,000, there’s only one of me.

You could imagine saying, “Take a GPU and instead of running at a gigahertz, run it at a megahertz,” and that would start to look a little more like the equivalent things you’re talking about in the brain. But in the way silicon works, that does not give you a 1,000x advantage in energy efficiency.

What it ends up looking like is you just run this circuit once to stabilization, and then it will sit idle for a long period of time. It doesn’t consume a lot of energy while it’s sitting idle because most of the energy is consumed in toggling bits from zero to one and back.

Let’s talk about the energy consumption of a circuit like this. The way to think of a bit being stored is that you’ve deposited some charge in a capacitor sitting somewhere in the chip implicitly. It becomes charged when the bit becomes a one, and then it becomes discharged when it next goes to a zero.

That cycle of charging the capacitor and then dumping that charge out to ground is where the energy is consumed. This is called the dynamic or switching power, and it’s most of the energy consumption of a chip. There is some other energy consumption just coming from the fact that insulators aren’t perfect, but we’ll discard that. Most of the energy consumption comes from toggling from zero to one and back to zero.

If you run a chip much slower and you only clock it once every thousand clock cycles, you will have 1,000 times fewer transitions. It will be about 1,000 times less energy consumption. But it’s not a substantial advantage in energy efficiency.

Dwarkesh Patel

Okay, so you described how a TPU works at a high level. What is the difference at a high level between how a GPU and a TPU work?

Reiner Pope

There is a high-level organization principle that is different, and then inside the cores things are different. Looking at the high level, we’ll take a GPU and a TPU and see what the top-level block structure looks like.

If you think of this as the whole chip in each case, the organization of the GPU is mostly a bunch of almost-identical units, which are the SMs. They’ve got an L2 memory in the middle, and then a bunch more of these SMs on the bottom. So there is a fairly regular grid of cores.

If we look at a TPU in comparison, you end up with much coarser-grained units of logic. You end up with just a few matrix units, which are the big systolic arrays. In the middle you’ve got some vector unit, and then you’ve got your matrix units at the bottom. These matrix units with a vector unit in the middle make up the whole TPU chip.

You can think of scaling this thing down into a really tiny unit with a smaller matrix unit and a smaller vector unit, and that is sort of what an SM is. From a very high-level point of view, the GPU has a lot of tiny TPUs tiled across the whole chip.

Dwarkesh Patel

Oh, interesting. You’re suggesting the tensor core within a streaming SM is analogous to an MXU?

Reiner Pope

Yeah, it’s all very similar.

Dwarkesh Patel

I see. If you had more lack of structure, having a bunch of tiny TPUs makes a lot of sense. Whereas if you just have huge matrix multiplications, you might want to avoid the cost of having individual SMs with their own registers and warp schedulers. Why not just make a huge thing and amortize those costs across the whole thing?

Reiner Pope

This shows up in how large you can grow things. We’ve seen this theme, especially with the systolic array, where a larger systolic array amortizes the register file costs better.

This design allows you to have larger systolic arrays, whereas the GPU design constrains you to having small units of everything. There is a trade-off, however. Because of this coarse-grained separation of things, you need to move a lot of data from the vector unit to the matrix units, through just two lines of perimeter here.

If you look at the equivalent thing in a GPU, you’ve got vector units everywhere, and you can move data through many different lines. The amount of data you can move between a vector unit and a matrix unit is actually much higher in a GPU than in a TPU. Instead of having to move all the data through just two lines, you’re moving it through 16 lines of wiring in a GPU.

Dwarkesh Patel

Right. But also you might have to move across less area.

Reiner Pope

Which is an energy saving as well. So if you can operate entirely within an SM, the data movement is much smaller. But the moment you want to operate across SMs, it becomes more complicated and expensive.

Dwarkesh Patel

So you don’t have to comment, but one might expect that a thing MatX might try to do is get the GPU-like smaller structure of systolic arrays surrounded by SRAM, but at the same time make it so that the things you need in an SM to support the CUDA architecture—which take a bunch of space—you might discard.

Reiner Pope

We’ve talked publicly about something we call a splittable systolic array, which in some sense you can think of as big systolic arrays that can be small systolic arrays too.

Dwarkesh Patel

Cool. Okay, I think that’s a good note to close on. Reiner, thank you so much.

Reiner Pope

Thanks, Dwarkesh.

I had an interesting discussion recently. Someone asked me, what is intelligence? I said, the ability to achieve your goals across a wide range of domains. Okay, he says, then by that definition isn’t Donald Trump the intelligent person in the world, followed in quick succession by Xi Jinping and Vladimir Putin?

To be clear, these people are obviously very competent and clever. But when you think of ASI, you don’t think of Trump, but more so.

The person who kept pressing this question was correctly pointing out that I basically defined intelligence as power. And by this definition, Stalin was the most intelligent person who ever lived.

Now, of course, you could change the definition of intelligence to something more like, manipulate abstract concepts and rotate shapes.

But notice that the most powerful people in the world do not max out this quantity. The correlation between extreme power and this kind of intelligence might be even weaker than the correlation between extreme power and height. The physicists are not running the world.

We tend to conflate power-seeking AI and superintelligent (in science and tech) AI. I’m not denying that AI can be power-seeking. Whatever skills and drives Donald Trump has could be embodied in a digital mind. I’m simply pointing out that the way AI systems are currently becoming smarter (by getting trained to be to be really good at specific economically valuable tasks like coding) is not that strongly correlated with power.

We often talk about power in this way that misunderstands how it is actually derived in our world. Our intuitions are primed by games like Diplomacy or Go, which are designed to isolate and reward a g loaded kind of strategic reasoning.

But in the real world, power is more the product of having the authority and trust to get lots of people to collaborate with you, rather than some galaxy brain scheming capability. Trump is not powerful because his brain, considered in isolation, is the most effective optimization engine on Earth. He is powerful because the government which hundreds of millions of people consider legitimate gives him a lot of authority.

A group versus individual level analysis is useful here. As Garett Jones has written a lot about, individual IQ is only modestly correlated with individual income, but national IQ is strongly correlated with national outcomes. This is because intelligence has a lot of spillover effects - smarter societies cooperate more, save more, and can coordinate to build things like space shuttles and semiconductors.

Richard Trevithick, who invented the high-pressure steam engine, died in poverty, buried in an unmarked pauper’s grave. But the fact that 18th and 19th century Britain had lots and lots of people like Trevithick contributed to Britain being able to set up a global empire and outcompete lots of backwards principalities around the world.

It seems to me that the right mental model is that automated firms will outcompete everyone else in normal capitalist ways, rather than a single AI outthinking everyone else.

Wrote up some flashcards here to help myself retain all the stuff below.

Had an interesting chat with someone on why pretraining runs often fail. It was very interesting to get a sense of all the tangible ways that things can get fucked, and why training is such a precarious operation. At a high level, breaking causality, and adding bias, seem to be key culprits.

Breaking causality:

• When you do expert routing, you first go through the router, which gives you a score of how much each token wants each expert. There’s two ways to proceed from here: 1. Token routing, where you read the scores from the token’s perspective, and allocate to each token’s top k experts. Problem is that you could end up with wildly unbalanced allocation across experts, which is terrible for performance. Alternatively, you could (and only in training) do expert choice, where you just split the tokens by which are more relatively preferred by each expert. This way you can enforce that each expert gets roughly the same number of tokens. But the big problem is that this breaks causality, because which expert token n gets allocated to may depend on which expert token n + k might be router to. And breaking causality is very bad, because you’re getting information in training (and updating based on it) that you wouldn’t see in deployment.

  • Rumor is that this explains why Llama 4 was underwhelming.

  • I guess you could do expert choice during prefill inference? But maybe it doesn’t work well in practice to allocate tokens to experts which would not have received that token in actual training.

  • Tbh I don’t fully understand why breaking causality is so bad. I understand you can’t see beyond causality in real inference. But why is this minor deviation such a big issue?

• Another thing that can break causality is token dropping. Where experts just ignore the tokens in the batch that they’re supposed to process, but which rank not so strongly, and cutting whom would spare going outside padding. This breaks causality cause a later token being more strongly matched to this expert might lead to an earlier token getting ignored.

  • Apparently this was an issue with Gemini 2 Pro.

Adding bias:

• Bias much worse than variance - variance can average out, but bias compounds

• Apparently the original GPT 4 training was slow and got initially fucked because of the following bug: they were using FP16 on their collectives like all-reduce. FP16 distributes its granularity according to logarithmic density - between 1 and 2, the mantissa bits carve the interval ~0.001 apart. But 1024 and up, the mantissa might be carving the interval by multiple whole number values. Suppose some collective involves adding 1 + 1 … 10,000 times - you could get in a situation where as soon as you get to 1024, you add 1, it goes to 1025, you round down to the nearest interval at 1024, add one again. And so the calculated value is 10x off the real value. Huge issue if you’re trying to sum many small gradients into a large accumulator. And imagine how hard the bug must have been to find!

Implications for AI training:

• Some of the people who think we can cure aging argue that there’s basically 5 different ways people die of old age (heart disease, cancer, etc), and that if we cure these 5 different diseases, then we’d basically have solved aging. You could ask a similar question about these failed pretraining runs - are there 5 different ways training runs fail, in which case once a lab figures out numerics and , you’ll just have smooth sailing, or will you keep seeing new bespoke issues emerge at each new level of scale? The person I talked to seemed to think the later - he pointed out that even within numerics, there’s so many ways you can fuck things up. And new ones will keep emerging at scale.

• Bearish on AI fully automating kernel writing anytime soon. Presumably this is because he thinks it’s more of an AGI complete problem than some give it credit for. There’s another school of thought that says, “Hey, which kernel gets attention or MLP to run fastest on this scaleup is a super verifiable domain, thus we can RL to superhuman performance easily.” But he says, it took Nvidia, which has the best kernel engineers in the world, a long time to optimize for Blackwell, which suggests that actually it’s quite hard, and might not be super easy to close the loop on.

• Sometimes people say inference for RL generation and inference for end user generation is basically the same. But this person pointed out that in RL inference, numerical drift between inference and training engine can cause these subtle off policy biases, which matter a ton for highest quality training. But are not an issue if just serving to users.

• Emphasized how important it is to have a disciplined process for amalgamating compute multipliers, because of the risks of stacking up bugs with subtle biases.

Notes from an excellent lecture that Horace He gave my friends and me.

What made this lecture so good is that Horace built up the whole topic as a chain of problems and solutions: here’s what we want to do, here’s why it breaks, here’s how we fix it, here’s why that fix eventually breaks too. Most explanations just list out a hodge podge of different strategies, without ever connecting them to the problems they solve or explaining why you’d pick one over another.

• Equation for pretraining flops = 6ND. 2 FLOPs per parameter per token for the forward pass (multiply + add). Backward pass is 2× forward because you compute gradients w.r.t. both input matrices. So 2 + 4 = 6.

• Okay we can’t do all this on one GPU. So how do we split up this problem? The obvious solution is to do data parallel - where you copy the model weights across each GPU, and you just do a part of the batch on each GPU.

  • The obvious problem is that each GPU only has a limited amount of HBM - B300 is 288GB - and this is not enough to store the weights as models get bigger and bigger, much less their activations.

• Okay so next thing we try is fully sharded data parallel - each GPU only stores 1/N of the parameters of each layer - before processing each layer, you all-gather the full layer’s parameters from all GPUs (each GPU only stores 1/N of each layer). After processing, each GPU discards the gathered parameters.

  • It was emphasized that this is the go to default. And you only move on from this when having too many GPUs forces you to move on, for reasons explained later. The reason this is the default is that it’s trivial to overlap compute and communication time - that’s because the only thing being communicated is the weights, which are not dependent on what happened in the layer before, so you can start all gathering the next layer while you’re still computing this layer. Compare this against tensor or expert parallelism, which do need to share activations for one layer before you can process the next one. The problem with pipeline parallelism is bubbles as explained below.

  • From a comms volume perspective, FSDP looks insanely expensive at first — you all-gather every layer’s full weights across all GPUs, use them for one matmul, then throw them away. But this ignores what regular data parallelism already costs you - in regular DP, you still need to do an all reduce after every layer of the backwards pass in order to sync the batch’s gradients across all the GPUs. That all-reduce has comms volume of params × 2. FSDP adds all-gathers — one per layer in the forward pass, one per layer in the backward pass. But an all-gather is half the comms volume of an all-reduce. So naive FSDP comms volume ends up being # params * 4 (all gather forward and back, plus all reduce on back). You can do even better: since each gradient shard only needs to end up on the one GPU that owns it, replace the all-reduce with a reduce-scatter (which skips the final broadcast step). That gets you to params × 3 total — a 50% overhead over vanilla DP.

• So why can’t you always just do FSDP?

  • Comms crossover: You want your compute time to be greater than your comms time - you don’t want to be bottlenecked on comms. But since compute time for FSDP decreases as you increase the number of GPUs, and comms time does not, as you scale the number of GPUs on FSDP, your MFU can totally crater. When this happens, you need to add pipeline parallelism too.

    • Compute time = (6 * # tokens * active params) / (compute per GPU * number of GPUs)

      • This decreases as you increase number of GPUs

    • Comms time = (# total params * 3) / (nv link domain size * infiniband BW)

      • Comms time does not increase as you add more domains. This was really confusing to me. Each domain collectively holds all the parameters, and you need to sync gradients across domains after each layer of the backward pass. You’d think that adding more domains means more hops in the ring, so the all-reduce gets slower. But the standard ring algorithm splits the message into one chunk per participant. More domains means more hops, but proportionally smaller chunks per hop. (This breaks down when chunks get so small that per-hop latency dominates, at which point you switch to tree algorithms.)

        • Technically, you can do better than a naive single all reduce for the gradients between all the domains. You do a hierarchical collective to optimize comms time across multiple NVLink domains. Key thing to remember is that each GPU in the domain gets its own bandwidth access to infiniband. So you wanna use it all up since interconnect bandwidth is the bottleneck. You do this by trying to do as much as possible within a scaleup before you move out. So you do reduce scatter within a scale up to give each GPU the domain-level reduced gradients for a shard of the layer, then all reduce these shards across corresponding GPUs across domains, then all gather within a domain. This shifts the comms time line down, thus moving the crossover point to the right.

        • Made an animation to illustrate it using Cursor and Composer 2:

  • If you look at the equations, you can see that if you increase batch size, crossover point moves to right, and if you make the model more sparse, moves to the left.

  • Also why TPUs are better at FSDP - because more accelerators within a domain.

• Batch size floor: FSDP is data-parallel, so each GPU processes at least one sequence. Attention is computed within a sequence and can’t (easily) be split across GPUs. If your critical batch size is 10M tokens and sequence length is 10K, you only have 1K sequences — so you can’t scale beyond 1K GPUs with pure FSDP, even if you have plenty of comms bandwidth left.

• Problems with pipeline parallelism (the next addition you’d make to FSDP in order to deal with these issues):

  • The problem with pipeline parallelism is different - there you have bubbles that emerge from the fact that at the beginning of the batch, the GPUs dedicated to the final layers are not being used, and conversely at the end of the batch, the GPUs dedicated to the first layers are not being used. The reason you can’t overlap batches in training to solve pipeline bubbles is that you need to consolidate gradients and update the model before you process the next batch.

  • But also you’re adding architecture constraints - things like Kimi’s attention-to-residuals (where each block attends to all previous layers’ residuals) become very difficult when those residuals live on different pipeline stages. Similarly, interleaving sliding-window and global attention layers could cause load imbalance across stages. Dealing with all this slows down research iteration, which is the greatest sin you can commit.

I’m writing up some threads that we explored in my interview with Michael Nielson. That episode was one of my favorites.

The organizing question from my interview with Michael Nielson was, “How do we recognize scientific progress?” It’s especially relevant to thinking about what it would take for AI to close the RL verification loop on scientific discovery. But it’s also a surprisingly mysterious and elusive question when thinking about the history of human science.

Some people have this idea that AI is going to be disproportionately good at making scientific breakthroughs. The reason they think this is that 1. Science is ‘verifiable’, 2. AI is absolutely crushing domains that have a tight verification loop - coding, math, etc - because you can RL on these loops.

But the history of human science shows that the verification loop for theories can be on the order of decades and centuries, and even then experiments do not definitely rule out alternatives: Ancient Athenians dismissed Aristarchus (2nd century BC) on heliocentrism because it would imply stellar parallax. The first successful measurement of stellar parallax was in 1838, achieved by Friedrich Wilhelm Bessel.

What we know today as the better theory can often actually make worse predictions: it’s well known that Copernicus’s model of circular orbits around the sun was less accurate than Ptolemy’s geocentric model, which had accumulated millenia of correcting epicycles. What is not well known is that Copernicus’s theory wasn’t even simpler (Ptolemy’s model interpreted the true elliptical nature of orbits using an equant trick where other planets are not moving in uniform circular motion around Earth exactly, but rather an off center point. Copernicus didn’t like this, because it violated his Platonic heuristics - so he discarded the quant trick, which led to a less parsimonious model, since Copernicus had to add more epicycles and epicyclets to make up for it.)

So in what sense was it a better theory in 1543? In some sense, it wasn’t! You couldn’t have known ex ante that heliocentrism married with Kepler’s 3 laws (1619) is a much cleaner and more accurate theory, or that there’s a very beautiful unification of heliocentric orbits and terrestrial gravity (Newton in 1686).

There was one ex ante reason that you should have preferred Copernicus in 1543: his theory required retrograde motion as a natural consequence of his theory, whereas for Ptolemy it was an ad hoc addition. Even more impressively, his theory, developed in 1543, actually predicted the phases of Venus before they were observed by Galileo in 1610. But both of these things were also implied by Brahe’s model, which had set the sun to orbit the earth and then all the planets to orbit the sun.

Under a naive falsificationist framework, you’d have to wait until Stellar parallax was observed in 1838 to know that Brahe was wrong. But obviously the scientific community was able to make progress faster than this. There is some mixture of judgment and heuristics in the progress of science that we don’t even understand well enough to actually articulate, much less codify into an RL loop.

Or consider the case of the discovery of Neptune in 1846. Uranus deviated from its predicted Newtonian path. Le Verrier predicted that an unknown perturbing planet must exist, calculated its mass and orbit, and Neptune was found almost exactly where predicted.

But the Neptune story is symmetric to a failure case. Mercury had an anomalous precession, where the ellipse that shows its orbit would rotate 43 arcseconds more per century than should be implied by the impact of other planets using Newtonian mechanics. This led astronomers to speculate that there’s an unknown planet Vulcan within Mercury’s orbit. But it was resolved in 1915 with Einstein’s General Relativity.

A proper Newtonian would still proceed with the research agenda, but modify it as follows. First, you predict some unknown planet. If it can’t be found, you say it’s so small, it must require a bigger telescope, and you build a bigger telescope. And if you still can’t find it, maybe there’s a cloud of cosmic dust occluding it. If still not found, maybe the satellite’s instruments are being screwed by some unknown magnetic field, and you send a new satellite. At each of these steps, had you discovered a new planet, or some unknown cosmic dust, or some new magnetic field, that would have been a sensational victory for Newtonians.

Ex ante, this is not unreasonable to do! It is only after decades or maybe centuries of patchwork that we can then analyze, are we simply adding epicycles, or is this theoretical framework progressive, in that it makes predictions we wouldn’t otherwise be able to.

What do these examples illustrate? That ex ante it is almost impossible to determine which research programs are progressive (will predict and explain unanticipated new phenomenon) and which are regressive (need to be contorted repeatedly to accommodate seemingly disconfirming new phenomenon).

But the verification loop is often extremely long and weirdly hostile, and even then, experiments do not definitely rule out alternatives (see the discussion in the Nielson episode about how physicist contemporaneous with the 1880s Michelson-Morley experiments thought that it simply ruled out a particular theory of ether. Only Einstein made the full conceptual leap to discard the ether altogether).

This means that big conceptual breakthroughs cannot be easily verified. They are recognized decades or centuries later, when it turns out they were much more productive than the alternatives available. What this means for AI for science is that 1. You can’t easily train an RL loop for big conceptual breakthroughs.

And 2. the society of AI scientists will still need individual AI instances that have idiosyncratic biases and heuristics, and to pursue them unrelentingly for decades on end - for example, like the one Einstein had in insisting that there shouldn’t be some arbitrary inertial reference frame. There should be dedicated people to keep a bunch of dormant research agendas alive in case they turn out to be productive upon further investigation. To understand the kind of intransigent dedication to hypotheses that is needed to preserve correct scientific idea - even in the face of disconfirming evidence - consider the following story: In 1815, Prout hypothesized that the atomic weights of all pure chemical elements are whole numbers, because experimentally, most elements seem to come out like this. But there’s many anomalies - for example Chlorine’s atomic weight is measured at 35.5. And so Prout’s school claimed that maybe the chemical substance in which these elements appeared were impure. But there seemed to be no chemical reaction that could get rid of the impurities. And then they said, maybe it’s fractions of full atomic weights - but the closer you measure, the less natural the fractions seem to get - Chlorine goes from 35.5 to 35.46. It takes until almost a century later for people to realize that these measurements are showing multiple isotopes of the same element, which can be separated physically, but have no chemical distinguishing characteristics.

What I’m trying to say is that ex ante, one couldn’t have known which research program would be more productive. We need to invest in all of them concurrently. But that investment looks like a bunch of different individual scientists being super unreasonable and obstinate about propping up their preferred research agenda.

The Origin of Species was published in 1859. Principia Mathematica was published in 1687, two centuries earlier. Conceptually, it seems like natural selection is much simpler than the theory of gravity. A contemporary of Darwin’s, Thomas Huxley, read the Origin of Species and said, “How extremely stupid not to have thought of that!” Nobody ever said the same for not beating Newton to the Principia. I wonder if the reason this happened is that, while Darwin’s theory is conceptually simpler, it cannot be decisively tested. The evidence is circumstantial, retrospective, and cumulative. There’s no equivalent of Newton running the numbers on the moon’s orbital period and radius, and confirming that it corresponds to his equations.

Also you need this concept of deep time. Charles Lyell published the Principles of Geology in 1830, which gave Darwin the vast stretches of time that natural selection needed. And the fact that Darwin and Wallace basically arrived at evolution at the same time (and both credited Lyell’s contribution) does suggest that these underrated intellectual footholds were quite important (geology, paleontology of ancient extinct species which showed intermediate species (in some cases between apes and humans), biogeography from voyages and age of colonization, more sophisticated artificial selection like pigeon breeding). It’s interesting that an idea whose essence must have been obvious to herders and parents for thousands of years actually required many millennia of ancillary intuition pumps to fully spell out.

The pattern of parallel discovery in science and technology is very interesting, and seems to contradict this vibe that certain innovations could have happened earlier much earlier than they really did.

Eric Jang walks through how to build AlphaGo from scratch, but with modern AI tools.

Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn.

Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second.

Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside.

Watch this one on YouTube so you can see the chalkboard.

And check out the flashcards I wrote to retain the insights.

• Cursor‘s agent SDK let me build a pipeline to generate flashcards for this episode. For each card, I had an agent read the transcript, ingest blackboard screenshots, generate an SVG visual, and run everything through a critic. A durable agent is much better at this kind of work than a chain of LLM calls, and Cursor’s SDK made it easy. Check out the cards at flashcards.dwarkesh.com and get started with the SDK at cursor.com/dwarkesh

• Jane Street gave me a real deep-dive tour of one of their datacenters. I got to ask a bunch of questions to Ron Minsky, who co-leads Jane Street’s tech group, and Dan Pontecorvo, who runs Jane Street’s physical engineering team. They were willing to literally pull up the floorboards and take out racks to explain how everything works. Check out the full tour at janestreet.com/dwarkesh

(00:00:00) – Basics of Go

(00:08:17) – Monte Carlo Tree Search

(00:32:04) – What the neural network does

(01:00:33) – Self-play

(01:25:38) – Alternative RL approaches

(01:45:47) – Why doesn’t MCTS work for LLMs

(02:01:09) – Off-policy training

(02:12:02) – RL is even more information inefficient than you thought

(02:22:16) – Automated AI researchers

Dwarkesh Patel

Today, I’m here with Eric Jang, who was most recently vice president of AI at 1X Technologies, and before that, senior research scientist at what is now Google DeepMind Robotics. You’ve been on sabbatical for the last few months. One of the things you’ve been doing is rebuilding, improving, and hacking on AlphaGo.

Today, you’re going to explain building AlphaGo from scratch and what it tells us about the future of AI research and development. Before we get to that, why is AlphaGo interesting? Why is this the project you decided to do on sabbatical rather than just hanging out at the beach?

Eric Jang

I like making things, and AlphaGo and Go AI is one of those things that really got me into the field. When I saw the early breakthroughs on AlphaGo in 2014, 2015, 2016 and so forth, it was profound to see how smart AI systems could become and the computational complexity class they could tackle with deep learning. This is a problem that has long been understood to be intractable for search, and yet it was solved through deep learning. That was quite mysterious to me, and I’ve always wanted to understand that phenomenon a little better.

My training is in deep neural nets for robotics, where the decisions made by the neural networks are a bit more intuitive. But AlphaGo is a problem where the decisions are the result of a very, very deep search. It’s always been very mysterious to me how a ten-layer network can amortize the simulation of something so deep in the game tree.

Eric Jang

If you plot out how much compute it took to build various iterations of strong Go bots over the years, you can see that in 2020 there was an open-source project called KataGo by David Wu from Jane Street, which achieved a 40x reduction in the compute needed to train a really strong Go bot tabula rasa. I’m not certain if it’s stronger than AlphaGo Zero, AlphaZero, or MuZero, but it’s very strong, and this is what most Go practitioners today train against when they’re playing an AI.

Thanks to LLM coding, what took a whole team of research scientists at DeepMind and millions of dollars of research and compute can now be done for a few thousand dollars of rented compute.

Dwarkesh Patel

We should first discuss how Go works. How does the game work?

Eric Jang

Go is a very simple game that can be implemented quickly and easily on a computer. The objective is to put down black and white stones and try to occupy as much territory as possible. I might start by putting down a black stone. Black always goes first. Go ahead. The way you capture an opponent’s stones is that for every intersection, if you can surround all four of its neighbors with your stones, then it’s cut off from oxygen, if you will, and it’s a dead stone. Now I control these four stones as well as this empty intersection here.

There are slight variations between Chinese, Japanese, and what are called Tromp-Taylor rules. Tromp-Taylor rules are designed to be completely unambiguous, so this is what all Go AIs train and resolve against. In typical Go, when humans play, you’re actually not allowed to put this white stone down here. It would be instant suicide. In Tromp-Taylor, it’s actually fine. You put it down, and it immediately resolves to death, so the outcome is the same.

Let’s start over and play a few stones, and then I’ll explain some more. I’ll just start there.

Dwarkesh Patel

I’m basically playing randomly here, but I’m trying to get around your stones and see if I can surround them.

Eric Jang

This move exposes one empty neighbor for your white stone. It’s akin to a check in chess. If you don’t respond immediately by putting one here, then I can immediately capture this.

Dwarkesh Patel

I see. Because it’s the diagonals that determine whether you’re surrounded or not.

Eric Jang

The cross-section, not the diagonals. This one is surrounded on three sides, so you’re at threat of losing that stone if you don’t play one immediately there.

Now you can see that I’m starting to pressure you, because by putting a stone here, you’re forced to put one here.

Dwarkesh Patel

Otherwise, you would have this two-block to yourself.

Eric Jang

Yes. And if you think through what happens if you were to respond here, you can probably search into the future and deduce what I’ll do in response once you do that.

Dwarkesh Patel

You have a lot of confidence in my abilities, but I’m guessing you’d put the black here.

Eric Jang

That’s right, and then I would capture all three of these stones.

Dwarkesh Patel

So I should just assume that this little block is gone.

Eric Jang

Yes. In Go, it’s actually okay to let an opponent capture some stones if, for example, it lets you position to capture more stones somewhere else on the board. This is what makes Go a beautiful game: you can lose the battle but win the war. As the board size increases, the complexity of these micro versus macro dynamics gets more interesting.

Dwarkesh Patel

Presumably you’d put one here.

Eric Jang

So now I would capture this entire group, and this would be mine.

There’s one more case I want to demonstrate, which I actually had a bug in my code for recently. Let’s consider a formation like this, with other pieces on the board in play. Let’s talk about how the game ends. In this territory, who controls these areas? Is it white or black?

Dwarkesh Patel

White.

Eric Jang

It’s actually black, because I’ve surrounded this whole area. Assuming I have other black stones here, it’s very hard for you to break this out of the control of these stones.

Dwarkesh Patel

And when the final score is tallied, would these ones also count as being in...

Eric Jang

Great question. This is where different rule sets have different ways of scoring. We should talk about how you resolve scores between humans and how you resolve scores in computer Go, because there’s some ambiguity in how humans evaluate this.

Most humans would look at this board configuration and conclude that black has totally surrounded white, and white has no chance of life. We could play out more here, but at the end I would capture everything. However, if you have a way of breaking this formation and connecting white to something outside of it, then it can flip. This is where it’s a little bit hard for a computer to decide these kinds of things.

How do humans do it? It’s worth thinking about how humans resolve this, because this will map later to how we think about the deep neural network. Humans basically say, “I think the game is done,” and then you have to also say, “I think the game is done.” Then we’ll say, “I think these are my stones,” and you have to agree. If you don’t agree, we keep playing.

Essentially, once two humans—their so-called value function—agree on a consensus, then the Chinese rules resolve that. In Tromp-Taylor scoring, it’s perfectly unambiguous, so it can be decided algorithmically by a computer. If you have this at the endgame, the way you score it is that you first count how many stones you control, and that’s unambiguous.

Then you count how many empty intersections are not touched by your opponent’s stones. These intersections would not count for either player, because all of these intersections are connected to both white stones and black stones.

If this were like this, then white would get three points. This is a little odd because a human would know that white is actually losing these points. But Tromp-Taylor scoring would consider white to have all of these points as well as these points. So that is a very big difference in how computer Go scores things and how humans score things.

Dwarkesh Patel

How does the game end?

Eric Jang

The game ends when either a player chooses to resign or both players pass consecutively. Those are the rules.

Dwarkesh Patel

Now help me crack this with AI.

Dwarkesh Patel

Let’s understand how AlphaGo actually works and how somebody in the audience might be able to implement it.

Eric Jang

Let’s start with an intuition about the underlying search process used to make moves, and we’ll layer on ideas from deep learning to make it much more efficient and tractable.

Go is a game with just two players. We’re going to draw a person here, and we’re going to draw an AI here. Let’s say this person is playing black, so they go first. They go here. Now the AI is going to make a move based on what it sees here. There’s a question of how you encode these inputs into the AI. Maybe you could use ones and zeros, but you want to represent black, white, and empty. You would need at least three different values. Maybe you could use zeros, ones, and twos. The AI might see something like zero, zero, zero, zero, one. This is the input to the AI on its turn.

The AI can choose. Let’s just pick three possible random moves it can make, and I just drew these at random. Which move is best here? Well, we don’t know until the game ends. Go doesn’t have any kind of local reward for which move here is good. This is what makes Go a very difficult game. You don’t actually know who won until you really get to the end.

How deep is this tree? On a 19x19 Go board, there are roughly on the order of 361 moves on any given move, and of course, as it fills up, you have fewer moves. The number of steps in the game can be somewhere from 250 to 300 moves. Experts might decide to end the game well before that, but under Tromp-Taylor scoring, you actually have to play things all the way to the end. So this could be 300 moves, a depth of 300 in the tree.

If you keep on expanding possible moves—here the AI goes, then here the human would go, and so forth—you find that you end up with an enormous explosion in the possible game outcomes originating from just this one state. This is something on the order of 361300, which is far more than the number of atoms in the universe. Of course, there are redundancies and symmetries, so it’s not actually that, but if you were to do a naive tree with no merging of children, you end up with a tree about this big.

Dwarkesh Patel

What do you mean by “merging of children”?

Eric Jang

Let me use this board here. If we start here, and then you play here, and then I play here, and then you play here, that’s equivalent to me starting here, you playing here, me playing here, and then you playing here. Both arrived at the same spot, but through different paths. So this child node can be thought of as a shared ancestor.

Dwarkesh Patel

Got it. And I guess it starts at 361 but it decreases by one each time.

Eric Jang

Yes, the branching factor decreases by one each time. In any case, this is a very large tree. This is also why computer scientists for many years thought that Go was not a tractable problem this century. The amount of compute you would need to exhaustively search every possibility is just too large.

If you could, Go is actually a deterministic game. In any given state, you can compute the best possible strategy you can make in order to win the game. You can search all the possible futures where you win, and then just make sure you always stay in that set of futures.

AlphaGo’s core conceptual breakthrough was using neural nets to make this search problem tractable. Before we get into how neural networks are involved, let’s talk about how we can, assuming we had a powerful enough computer, search this tree to find the best move.

In the beginning, you’re not going to build out the whole tree, because storing it would be very expensive. Instead, you might interactively figure out which leaves of this tree are worthy of exploring and expanding into the future to see what else is there. There are some early algorithms in the bandit literature like UCB1, which is not exactly appropriate for a sequential game like Go, but very much inspired the action selection algorithm used in AlphaGo. UCB1 looks like: on every move, we’re going to take the best action—the argmax over a that maximizes Q of a, and I’ll explain what Q of a is in a moment—plus some sort of exploration bonus.

On every node, we’re going to track a few quantities. Let’s consider each of these a node. This is the root node where you’re making decisions from, and these are the children of the root node. Each node is basically a data structure that stores a visit count of this child node.

Dwarkesh Patel

How often the parent visited this node.

Eric Jang

Yes. One thing that’s easy to trip on if you come from robotics or other kinds of reinforcement learning is, where are the actions? I’m only talking about nodes. Nodes here represent states, and because this is a perfectly deterministic game with no randomness, you actually can just infer the action based on the child. If I go here, that implies an action, and this is the state we resolve to.

If you ask LLMs to vibe-code an MCTS implementation, they will most likely design the right data structure here. But it’s sort of a chef’s choice. You can rewrite the tree structure however you like. This is what Claude 4.6 wrote for me when I asked it, and it was a very reasonable choice.

Qa represents the mean action value of this action. I’ll use a subscript a to denote that this corresponds to taking a specific action to get here from the root node. Taking a gets us to this node here. We’re also going to store the probability of taking this action

Dwarkesh Patel

Again, from the parent?

Eric Jang

From the parent, yes. What are the odds that we sample this one? This will become relevant later. We’ve talked about a deterministic tree for now, so I’ll bring probabilities into this later.

Finally, we have a dictionary of children, which is just more of these nodes in a classic linked list-style reference tree. This is the basic data structure to implement a tree.

In AlphaGo, they use a slightly different action-selection criterion called PUCT, short for Predicted Upper Confidence with Trees. When you select which child to take, you do argmax a of Q(s,a) plus a constant. The equation forms are pretty similar. These are both scoring criteria. You want to argmax this quantity and you want to argmax this quantity to determine which action to take.

Let’s break down the intuition of how you select actions here. Q(s,a) is the mean action value, so how good a given child is on average. If you actually knew the whole tree, this is all you need to select the best action. You don’t really need to do more than that. But if you’re interactively building this tree as you’re figuring out what the Q values should be, then occasionally you have to try some other actions as an explore-versus-exploit trade-off.

In both UCB and PUCT, there’s this term that basically rewards taking actions you haven’t taken before. As we mentioned, each node stores the visit count of taking that specific action. Everything is initialized to zero. For a given action, let’s call it action a, initially it’s zero. As n is increasing, if we’ve already made 10 action selections from that root node but we haven’t picked a yet, then this term starts to become quite large for a. Conversely, if we’ve chosen a 10 times out of 10, then this term is quite small. It diminishes very quickly. The same thing is true here.

Dwarkesh Patel

Just to make sure I’m understanding, let me put it in my own words. Let’s focus on UCB. Conceptually, you can think of it as two different things: the Q and the exploration term.

Let’s be clear about what Q is. Q is basically saying, once we do these rollouts—you’re actually running all these simulations—you go down the tree and figure out: if I end up at the terminal value of this tree, do I win this game or not? You average whether I win or not across all the leaves of this tree starting from this node. That average, you put in Q. So Q represents the probability that I’ll win this game starting at this node. That is your sort of exploit. It’s saying: I’ve run these simulations, and I think this is a good move or not.

The other term is saying: have I explored this branch enough yet relative to the other actions I could be exploring, or have already explored? If I haven’t explored this branch yet, maybe I think it has a low score, but I just haven’t explored that many leaves down this node in this tree. So I should try this even though Q, the exploit term, is telling me it’s not that valuable.

Because ln(n) grows slower than n, over time you will move from the argmax being dominated by the exploration term, which is the second term here, to the argmax being dominated by the Q term, which is when you’ve done enough simulations and are confident that this is the branch to go down.

Eric Jang

Yes, that’s right. The motivation for UCB was to come up with an algorithm where, if you don’t know the payoff of the different actions you can select, this strategy, given some exploration term here, bounds your regret in terms of how wrong you can possibly be.

I don’t know the proof. I also don’t know if this one is proved to have logarithmically or square-root-bounded regret, but I think the algorithm was derived to look something like this. You can tell these terms grow a little differently, and this is to account for the fact that Go has many more actions for any given move compared to your standard bandit problem.

One small clarification: you talked about simulations and probabilities. We should remember that Go fundamentally is a deterministic game. Where does the notion of probability come from here? If you had a very powerful computer, there are no probabilities. You can just compute the true average of the mean action value.

So where does the probability come in? In computer Go before AlphaGo, we’ve always done some sort of Monte Carlo method where we take the expected Q value averaged over a randomly selected tree. That randomly selected tree is where probabilities come in. The interpretation of Q is: what is the expected action value under the random distribution induced by some random search process?

Where does the random search process come in? That’s where Pa, of action, comes in. If we assume a naive algorithm where you have a uniform probability of taking any valid action, then this would just be one over the number of valid moves. You would be taking this average over a very diffuse tree. This is a valid integral, but it’s very slow because you’re going to consider a lot of trees that have very low value. It’s essentially almost like an importance sampling problem. Only a few actions and paths contribute high value, and almost everything else is low value. So that’s a tricky problem here.

This is the action selection criterion for how you decide which moves to go down. As you move down in tree search, you will eventually run into a node where it’s quite clear you’ve won or lost. At the very end of the game, when there are no valid moves left to play under Tromp-Taylor scoring, you can decide whether you won or lost. This is the final return of the whole game.

We can assign a value, U, to a terminal leaf node of the tree, but how do we assign values to the nodes prior to that, the parents? You take the mean action value, which is essentially your average. Suppose these were all leaf nodes. The mean action value of this node is just the average of whether you won or lost at the leaf nodes. Correspondingly, you can walk up the chain and say the mean action value of this node—let’s call it Qb—is just a weighted average of these ones here.

The weighted average could depend on whether you have a different sampling distribution. But the basic intuition is that you want to resolve the game where you have a deterministic win or lose, and then you can go backwards—this is called the backup step—and assign values to these nodes or actions corresponding to the average over the final terminal leaf.

If you were to do this without neural networks, it would still be intractable. You would have trouble finding which actions to sample. A lot of the actions would contribute very low value, especially if you’re trying to fight your way out of a losing position. Only a few actions give you high value, so the search in practice is still very expensive. But the idea is that because Go follows a tree structure, you can inform a very good estimate of the value of this node based on the downstream values, assuming they’re all correct and you’ve searched deep enough.

Dwarkesh Patel

Your earlier explanation—about the sorts of states where it’s obvious to a human who’s going to win, but deterministically you still have to play it out—actually drew upon the intuition for why 1) the value function is both trainable and 2) necessary in order to learn this game effectively. Maybe it’s worth defining value in the first place.

Eric Jang

We talked about U being your final resolution of whether you won or lost. This is a terminal leaf node condition. Humans don’t play all the way to the leaves of the tree. They stop dozens of moves before, maybe even 100 moves before in high-level play.

How do they know? You can think about humans as implicitly having a neural network called a value function that takes in a board state and evaluates p(win). The human glances at the board and knows, “I’m probably going to lose.” They’re essentially running a neural network that looks at a board and implicitly amortizes a huge number of possible game playouts. They take that average and decide whether the board is winnable or not, and whether they should concede or keep playing.

This is remarkable. If you think about the beauty of something like this, a neural network in a human can somehow do all of this simulation at a glance. They just know within a few seconds, without actually playing every single game logically, based on crystallized knowledge and experience. They can do this. This gives us a hint that in games like Go, there are ways to radically speed up the search process. This is one of the fundamental intuitions behind why AlphaGo works. You can train a value function to look at a board and quickly resolve the game without playing out all of these trees to a very deep search depth.

Dwarkesh Patel

Makes sense. I will say for the audience, for previous episodes when I was prepping and it seemed relevant to understand how AlphaGo works, I would find it very confusing. But it’s the kind of thing where once you understand the problem in this way and then build the next few pieces, it is actually a lot more understandable and it makes a lot of sense. It’s okay to be confused right now, but it’s probably simpler to understand, by the end of this lecture, than you anticipate. I’ll just make that note for the audience.

Eric Jang

The important intuition at a high level—to step back about where we’re going with all this—is that classically, for games like Go, you could build a tree, but we don’t have computers powerful enough for that. Estimating the value of every action you could possibly take is also hard because you don’t know until the end of the game.

You could take averages by playing them to the end, but that’s also hard because you don’t know which actions to take to sample these averages. Conceptually, there are two problems: the breadth of the tree and the depth of the tree. AlphaGo gives us a way to shrink both of those to be tractable. That’s essentially the core idea behind it.

We take the idea that humans can glance at a board and instantly predict whether we win. That maybe gives us the opportunity to truncate how deep we search. We also know that humans can intuitively, at a glance, decide what moves might be good on a Go board. These are two things we can use deep neural networks for, to accelerate the search process.

Before we talk about neural nets, let’s go back to how this playout works. We’ve only talked about making one move. The AI looks at this encoded Go board. It has a tree. It searches deeply into the tree to find out which of its actions might be the best, takes that action, and goes back to the human. Now the human sees a Go board that looks like this, and they make their move. Maybe they put their stone here.

Now we go back to the AI, which now looks at a new encoded board. I’ve used 2 to denote the AI playing as white, 1 to denote the human playing as black, and 0 as empty. On the AI’s turn, it does the MCTS tree search all over again from scratch. It throws away the old tree that it searched last round. Now there’s a new root node and it begins to search anew.

You can basically think about MCTS as a search algorithm that decides which moves to play best, aided by neural networks, and it’s done on every move. Let’s talk about the neural network part of this.

Dwarkesh Patel

While you’re erasing, another thing that was important for me to understand was about the MCTS data structure with nodes and children of nodes. This is done per move and reinstantiated once a move is made. A human makes a move, then the AI looks at this and runs a bunch of simulations to figure out what move to make next.

A simulation is exploring one more node in the MCTS tree. Once you run 1,000 simulations, that informs the probability of what move to make next, as you’ll explain. That’s what you store. You choose the best move given those probabilities. You discard all of that, the next player makes a move, and you restart this process at the beginning of every move.

Eric Jang

Correct. One small addendum: you don’t discard all of that. You keep one thing behind that we’ll use later.

Eric Jang

Now that we have a basic intuition of how moves are made with search, we’re going to talk about how neural networks can speed this up by providing an analog to human intuition. There are two networks. There is the value network, which takes in a state and predicts, am I going to win or lose? It’s a binary classification problem. Then we have a policy network, which induces a distribution over good actions to take.

I’m going to draw a one-dimensional flattened move distribution, but this is really a square grid. These are the probability distributions over good actions. Both of these are categorical classification problems. You can train this like any classifier with deep learning, cross-entropy loss, that kind of stuff.

The specific architecture does not matter too much. I tried a few different architectures. Transformers work, ResNets work. For small data regimes, my experience is that ResNets still outperform transformers and give you more bang for the buck at lower budgets. But this may not always be true.

Dwarkesh Patel

Wait, why is that?

Eric Jang

They provide the inductive bias of local convolutions. Generally, transformers start to outperform residual convolutional networks when you want more global context.

One interesting finding from the KataGo paper was that they found it quite useful to pool together and aggregate global features throughout the network, to give the network a global sense of how to connect value from one side of the board to the other.

Dwarkesh Patel

What does it mean to aggregate global features?

Eric Jang

If you have a very large 19x19 Go board, and you’ve got some battles going on here and some battles going on there, when you pass this through a convolutional neural network, the receptive fields are going to be good at computing local things and making that invariant. But they won’t be able to connect these two features easily. They need to be pooled together and attend to each other somehow.

The argument for why transformers are good for computer vision tasks—vision transformers and so forth—is that because they have global attention across the whole thing, they can more easily draw these connections. But you do need more data there so that you can learn the invariant local features through data.

I’ve tried very hard to make transformers work for this problem because I was curious if transformers would present some sort of breakthrough in Go and just remove a lot of those tricks. But try as I might, I haven’t figured out a way to make transformers better than ResNets for now.

Dwarkesh Patel

One more tangential question. It makes sense why transformers, with their global pooling of information, would be better if you need to consider information that is not just spatially… CNNs give you a bias that the things next to you are especially relevant.

Eric Jang

And then they’re aggregated up slowly.

Dwarkesh Patel

For games where it isn’t that relevant what is happening locally—you just have to consider the whole thing—you’re saying transformers would work better. We’re talking about the spatial dimension. How about the temporal dimension? Right now we’re only considering the previous move because it is a deterministic full-information game.

What if it were something like poker or Diplomacy, where a bluff they made a while back is relevant to understanding now? And to decide your next move, you need to consider all those previous states. Would that change the consideration of what inductive bias and which architecture is most relevant?

Eric Jang

Great question. Go is a perfect information game. In perfect information games, there does exist a Nash equilibrium strategy for which you can do no worse than any other strategy.

If you know your opponent has a particular bias, like they love to play aggressively, you can in principle counter that specific strategy better than a Nash equilibrium policy. But to counter any given strategy, there exists a single Nash equilibrium that can be decided solely using the current state. That is a design choice AlphaGo made, which in hindsight turned out to work very well because the Nash equilibrium seems to be superhuman. No human strategy seems to be able to beat it.

There are variations of this where you would need to consider temporal history. This is a very exciting research area, and I’d encourage people to fork my repo and try it out. If you were to play 2v2 Go, you actually need to model your partner’s behavior. You may not have information on how they play, so you need to aggregate information on how they play so you can respond accordingly. These are situations where it’s no longer a perfect information game. In games of imperfect information or partial observability, you do need context to build a model. That’s a place where things can get very exciting in terms of self-play or Diplomacy-style games.

Returning to the neural network, the architecture again is not super important. You can get it to work with transformers. You can get it to work with ResNets. I found that for low-budget experiments, ResNets work a little better. You can also use Karpathy-style AutoResearch hyperparameter tuning to make your architecture pretty good. You don’t have to worry too much about that. You just need to set up the problem so that you have a target optimization.

We’re going to pick a somewhat arbitrary architecture that worked for what I did. But again, this part is not super important. You have your encoded board state and, similar to an RGB, we’re going to have three channels. One channel to encode black, one to encode white, and one to encode empties or a masked region if you want to train on multiple board sizes. I’m actually not going to talk about multiple board sizes for now. That’s a bit too complicated.

We have this two- or three-channel RGB-like image, and we feed it into a ResNet. Then we have two branching heads. One head predicts the value function, which is a single logit, let’s just call it R1. Then we have the policy, which is R361. This is the architecture. We’re going to train this to predict the outcomes of games given the board state, and we’re also going to train it to predict what are good moves.

The original AlphaGo paper, called AlphaGo Lee, initialized this network with a supervised learning dataset of expert human play. Later, they removed this restriction by having the model teach itself how to play well. I find it super nice for implementation, for your audience, to always initialize your experiments to something easy and get the problem working before trying to bite off the whole thing and learn tabula rasa.

In deep learning, initialization is everything. You always want to initialize your research project to something as close to success as possible, especially if you’re doing something new that you haven’t done before. Always pick something that works and then get it to do something better, rather than start from something that doesn’t work at all and try to make it work.

Under that philosophy, it’s a great idea to start with something that has a good initialization. We’re going to take human expert plays and train this model to predict good actions. We’re going to take all the moves in which an expert won and predict those actions. Regardless of board state, whether you won or lost, you’re going to predict the outcome.

You might be wondering, for some of the early boards where only one stone has been put down, how could you possibly know who the winner of this game is? Well, if you have hundreds of thousands of games, on average you’ll probably see that boards starting like this have half of the games branch off and win, and half branch off and lose. So that’ll actually be fine. When you train this model to predict those outcomes, the logit will sort of converge to 0.5. For these things, it’s expected that once you train the model, a starting board state will look like 0.5, and then as you progress towards the end of the game, the win probability will either go up or down.

This is your move number. As you get hundreds of steps into the game, it becomes much clearer who’s more likely to win or lose under your expert data distribution.

Dwarkesh Patel

I didn’t understand the significance of why this way of thinking about value is especially relevant to the expert data.

Eric Jang

It is not relevant to the expert data. It’s true for any data that you trained it on. If you were to learn tabula rasa, you would also expect this to fall out.

Imagine you’re vibe coding AlphaGo. You gather some expert datasets from KataGo online, or you have a dataset of human players, and you train this model. It turns out this model is already a pretty good Go player. It will most likely beat most human players. If you just take this policy recommendation and take the argmax over these probabilities—if you take the argmax and just take this action as your Go play—it’ll be a very fast Go player that doesn’t think in terms of reasoning steps. It just shoots from the hip, and it’ll be a very strong Go player. This is already quite miraculous if you think about how ten neural network layers, maybe under 3 million parameters, can already do something that impressive.

You can start this way. It’s important when implementing this to verify that this is probably true. It’s good to verify that your Go rules are implemented correctly and that you can run these simulations relatively quickly. Just as a checkpoint, you want to make sure you can actually do this basic step before you try to layer on more complex things like search.

But we can do a lot better than taking the raw neural network and playing the moves. So let’s apply the neural network to improve Monte Carlo tree search. We start with our root node, and we now have a four-step iterative process to do MCTS.

This tripped me up when I was first reading the paper and trying to understand it. Essentially what we’re going to do is choose a number of simulations, and this number varies. It can be somewhere between 200 and 2,048. I believe in the AlphaGo vs. Lee match, they used tens of thousands of simulations per move because they really wanted to boost the strength of the model as much as possible. But in training, you don’t actually need too many. KataGo, I think, uses something in this order as well.

Dwarkesh Patel

Do you know if they used… If you watch the documentary, they had a laptop out during the game. They didn’t use the laptop itself. It was on some—

Eric Jang

It was on some TPU pod, I think.

Dwarkesh Patel

Honestly, kind of unfair. Lee is not using 1E22 FLOPS to do a move.

Eric Jang

Fair enough. Interestingly, modern Go bots don’t need that much compute at test time. What we’ll find out, as we talk about how the MCTS policy improvement works, is that over time the raw network actually takes all of the burden of that big TPU pod and just pushes it into the network. You can do all of that work with one neural network forward pass. The TPU pod will always add the extra oomph on top, and that’s what they wanted for the match.

We’re going to pick this number of simulations thing, and for every simulation, we’re going to do several things simultaneously. We’re going to see which moves are the best in the current tree. We’re going to add extra leaves to the tree if we get to a point where we need to add a leaf, and we’re going to update the action values for the tree. Every simulation involves this four-step process: selection, expansion, evaluation, and backup.

At the beginning of our Monte Carlo tree search, our tree is very basic. It only has the root node, or our current board that our AI wants to play at. We’re basically going to select the best action for this. When this root node is created, we also know that we can evaluate it under our neural network and get the quantities Vθ, as well as our probability over actions. And I’m going to say root. For all of the actions here, we can create a bunch of children. In this case, I’m drawing a 3x3 board with one missing, so there are eight possible children associated with this root node. Each of these has an associated probability of taking that action, so there’s P8, P1, ... P2, and so on.

So at the beginning of our Monte Carlo tree search, we have our root node, and we can initialize it with some children. The policy network evaluated on the root node gives us—on a 3x3 board with one existing stone placed—eight possible children that this AI could take. With each of the children, the policy network also gives us the probability of selecting that child. The first step is to do the selection of the tree. Again, this is a very shallow tree. All we have so far is essentially a tree of depth one.

Our first move is to select by maximizing, or argmaxing, the PUCT criterion, which is basically Q(s,a) + CPUCT x Pa x (√N / (1 + Na)). For each of these, Na is zero for all the actions initially. N is zero. So we’re going to pick according to this. Initially, the chosen action is most likely going to be biased towards the highest-likelihood action, because these are uniform for every node.

Let’s suppose P1 was the highest-probability node, so you selected this one here. Now you get to this node and you realize it’s not a leaf node. It’s not a terminal game, so you cannot resolve the final resolution. The next step is expansion. You will then run this board state through the policy network. Note that this is the AI’s move. The AI is making this move. When we expand this tree, we’re now thinking about what the human, or any opponent, might do.

When we evaluate the node here, we’re going to evaluate it from the perspective of this player. This node has possible actions that we could take, and we expand the leaf nodes here. For each of these nodes that we could arrive at, we’re going to now check how good they are. From here, the human could play here, here, or here. We’re going to store Vθ for each of these, so Vθ of Node1’, Vθ of Node1’’, and so on. We’re basically using our neural network to make an intuitive guess of how good this board is from the perspective of this player.

Fortunately, because it’s a zero-sum game, it’s easy to deduce that the value for this player at this step is just one minus the value from the other perspective. It’s easy to flip the search process depending on which player you’re at. This is the expansion step. You’ve taken a non-leaf node, expanded it, and evaluated the value. This is essentially a quick guess as to whether I’m going to win or not if I were to play to the end. You can almost think about Vθ as a shortcut for searching to the end of the tree, for any given simulation.

This is essentially the evaluation step. We’re evaluating the quality of each of these boards. In the original AlphaGo Lee, they did something kind of interesting. They took this value and averaged it with the value of a real Go playout. They actually played a real game from here all the way to the end. I’m just going to draw this squiggly line to indicate some path. They play this all the way to Tromp-Taylor resolution of a full board. So this is a zero or one. They took this value and averaged it with this one here. The formula was α x Vθ of some node plus (1 - α) of a truly randomly sampled playout.

You might be wondering how they play this out. It would be very costly to do another search on this playout, almost like a tree within a tree. They don’t do this. Instead, they just take the policy network and play it against itself. They use it as both players, and they just play it all the way to the end. This helps ground the estimates here in reality because you can get a single-sample estimate of whether you win or not. In the endgame, where the board is almost resolved, this becomes quite useful because the play according to the policy will most likely decide a pretty reasonable guess of the game, so you’re not facing a problem where this becomes untethered from reality.

It turns out this is totally unnecessary. In all subsequent papers after AlphaGo Lee, they got rid of it. In my implementation, I did the same, and it speeds things up a lot because you don’t have to roll these games out on every single simulation.

Dwarkesh Patel

Okay, just to reinforce my own understanding and re-explain it. For the audience, by the way, in case it’s not obvious, the P there in the select, that is the probability coming from the network in this case.

Eric Jang

Correct. The policy network here.

Dwarkesh Patel

Fundamentally, a simulation, just think of it as rolling out one more node in the search process.

Eric Jang

Almost. A simulation is easy to think about when the whole tree already exists. You just walk down the tree using the PUCT selection criterion, and then you keep going. In AlphaGo, the data structure is such that we begin with a tree that basically only has depth one—its only children—and you want to iteratively build out the tree as you’re also selecting actions down the tree. That’s the core thing here. Because Go is such a combinatorially complex game, you cannot afford to build the tree in advance and then search it. You must search while building the tree.

Let me finish up with the last step, which is the backup. Once you’ve scored these things, the Q value assigned to the node here for taking this action is just the average across your evaluated values. You take a running mean over all the simulations you’ve taken, averaging the values of the children nodes. That’s the backup step, and once you evaluate this, you can recursively go back up. If you know the action value of this node, you can then take the average on its parent, and so on.

You have this four-step process: 1) You’re choosing the best action you know of so far. 2) You may run into a node you haven’t been to before, so you need to grow the tree a bit. 3) You run it through the network to guess whether you’re going to win or not. 4) You walk all the way back up to the root node to update your values on what the best moves are.

As you do this iteratively, because you’re always selecting according to this criterion, you’re always going to be selecting the best action you think at any given branch. The final visit counts of how often you chose these things will reflect your correct policy distribution as induced through this search process. The visit count we stored in the node earlier actually becomes the vote for which action we should finally select.

As a test of understanding, it’s worth thinking a little bit about whether we could make this even simpler. Could we actually maybe even get rid of this one and still make the thing work? Recall that when you do an expansion and then an evaluation at this node, you are checking the win probability of each of the child nodes. So if this one is one and these are zero, you do know something about which action might be better to take. Why would you still need this, right? Why not just normalize the values into a distribution and call that your policy distribution?

This is fine. You can do this, and it probably does work. But in practice, having a single forward pass that gives you a pretty good guess is how the breadth is pruned out. There is a duality here. It would be weird if the policy recommended an action that disagreed with the value. If a policy said this was very high probability, but this one said it was a low value, then there’s something fundamentally wrong between your policy head and your value head. They are linked, and you probably could get rid of this if you came up with a different way to recover it from just the value evaluations.

Dwarkesh Patel

Right. Just to make sure I understand, the reason you don’t do that is so you don’t have to do 360 independent forward passes to say, “Okay, here’s the value of everything. Let’s argmax over it.” Instead, you can just do one forward pass and get the probabilities of all of them.

Eric Jang

You can usually batch these somewhat efficiently, so it probably isn’t a huge computational burden in practice. But yes, you would have to pass up to 361 boards into a single mini-batch update to evaluate all the values, then normalize them.

There’s actually a more important reason why we still do this, which is how Monte Carlo tree search is used to feed back on itself and recursively improve its own predictions and search capabilities. That’s where having this as an explicit entity you’re modeling, rather than an implicit normalization over your value, is a good idea.

Dwarkesh Patel

Makes sense. Okay.

Eric Jang

We talked about the simulations. What you end up with as you roll out the number of simulations is a tree that looks like… I’m drawing a very low-dimensional version of this. Of course, in the real game, it’s much more high-dimensional. You’ll end up with a tree structure that has a lot of leaves that terminate and are not visited again because their value is deemed to be too low. But along one path, there will be a set of actions with very high visit counts that gravitate towards that one set of decisions as you increase N.

This is the mental picture of what the tree in Monte Carlo tree search looks like. You should contrast this with an exhaustive tree like in tic-tac-toe, where there are nine actions, then eight, then seven and six, and so it’s a nine-factorial-sized tree. The Monte Carlo tree search in Go is very sparse. It only considers the paths that you’ve expanded children nodes on.

Now that we have the search algorithm that applies the value function as well as the policy function, we can talk about how the Monte Carlo tree search algorithm can act as an improvement operator on top of these guys here.

Eric Jang

We now talk about the RL part of how this thing gets stronger by playing itself. Let’s say we play a game. You make a move. The AI will compute the search, and this is the visit count distribution. Let’s say this is your initial policy recommendation at this node.

After MCTS, it gets more confident about one of these actions. Maybe the distribution looks a bit more peaky like this based on the search. Of course, you can tune the search process so that it ends up more diffuse, but that’s probably not a good idea. MCTS should get more confident about specific actions over others. It might place a lot of weight on other actions initially, and then as you increase the number of sims, it should converge to a very peaky distribution.

Let’s call this π. Let’s wrap this in an MCTS operator of (a | s). After applying the MCTS process, your policy recommended distribution looks like this. It’s a bit more peaky than the previous one. Then you take the argmax, or maybe you just sample from this. It doesn’t have to be the argmax. You make your move, throw away the tree, and begin anew on the next move. Again, you compute a new distribution. Initially, maybe your guess looks like this, and then you refine it through MCTS.

Dwarkesh Patel

There should be one more X on the board, right?

Eric Jang

I’m sorry, that’s correct, yes. Something that looks like this. On every move, you have your initial guess from your policy network. Then the search process, which combines your policy network and your value network, arrives at a more confident action that you take, and so on and so forth.

Then the game ends, and one person wins and one person loses. The beauty of how AlphaGo trains itself is that it can actually take this final search process—the outcome of the search process—and tell the policy network, “Hey, instead of having MCTS do all this legwork to arrive here, why don’t you just predict that from the get-go? Why don’t you not use this guess and predict this to begin with?” If you have this guess to begin with in your policy network, then MCTS has to do a lot less work to get things to work.

If we draw a test-time scaling plot… Let’s say this is the number of simulations. At zero simulations, your implicit win rate is here. If you just take this raw action, this is what your win rate is. Let’s say we increase the number of sims, maybe you have a win rate curve that looks like this. When you search for 1,000 simulation steps, that gets you to a policy here that gets you to here, which is great.

But if you were to distill this MCTS policy network back into your shoot-from-the-hip policy network, then you could actually start here. Let’s say this was zero, by distillation, if you then spend another 1,000 sim steps, you actually get to here. It’s almost as if you could amortize the first 1,000 steps into the policy network instead of the search process, then you could begin at a much better starting point and get a much better result for the number of sims that you play.

Dwarkesh Patel

The sigmoid type nature of test-time scaling as the number of simulations increases, the increase in win rate gets smaller. Is that true even for the distilled network? That is to say, is there some gain where we start from the distilled network and get these early gains again, or is that just inherent to the nature of MCTS?

Eric Jang

To be honest, I actually don’t know the test-time scaling behavior of MCTS simulations. I believe it might be quite sensitive to how strong this one is in practice. I’m just drawing a monotonically increasing function that gets to one.

Don’t pay too much attention to the shape of the curve. Just know that it’s monotonic with respect to sims. The idea of MCTS is very brilliant. We got something better by applying search. Now, on our next iteration of updating this network, we’re going to train it to approximate the outcome of 1,000 steps of search.

Instead of starting here, we get to have our neural network start here, and then the play gets stronger once we then apply another 1,000 steps on top of it. You can keep going. The training algorithm for AlphaGo is to basically take the games where you’ve applied the search on every move that the policy encountered—whether you won or lost, and that’s quite important—and just train the model to imitate the search process.

There’s an analogy to robotics, which is the DAgger algorithm. First I’m going to draw a schematic of the states: S0, S1, S2, S3. Let’s say we took a series of actions in an MDP to get a trajectory. These actions may be suboptimal. Maybe we lost at the end of this game. There is a family of algorithms that basically take trajectories and relabel the actions to better trajectories. Maybe a better action here would have been to take A0’. A better action here would have been A1’, and yet another one, A2’, A3’.

What MCTS is doing is saying: you played this game where you eventually lost, but on every single action, I’m going to give you a strictly better action that you should have taken instead. It does not guarantee that you are going to win, but it does guarantee that if you take these tuples as training data and retrain your policy network to predict these actions instead of the original ones, you’re going to do better.

This is very related to DAgger in robotics and imitation learning, where you want to collect an intervention here. Even if you’re in a not-great state—for example, a self-driving car that veers off the side of the road—there is still a valid action that corrects you and brings you back.

Dwarkesh Patel

Pedantic question, but is there a guarantee that MCTS must be better than the policy? For example, you could imagine early on in training, because MCTS is informed by the value network, that when the value network hasn’t been well-trained on finished games, MCTS is worse than a randomly initialized policy. Is it just a heuristic that MCTS is better than the policy, or is there some guarantee?

Eric Jang

In practice, it is a heuristic. It does work in practice, but let me illustrate an example where MCTS can give you a worse distribution than your policy network. This can often happen if your self-play algorithm has trained to a good point, but then somehow it collapses because it’s not trained on diverse data or something.

Let’s say we have a board state where the policy recommendations are very good. So Πθ (a | s) is great. But somehow, maybe because we’re playing a lot of games where the bots just resign instead of playing all the way to the Tromp-Taylor resolution, they forget how to evaluate those late-stage plans. Like in the case we showed with the corner play, maybe 100% of our training data in our replay buffer has lost examples of how to evaluate the value function at those states.

You might end up in a scenario where your terminal value is very bad. If the terminal values of the leaves are not good, then this will propagate all the way up and cause your PUCT selection criteria and your backups to be off. And then you end up visiting a very different distribution than what your policy initially recommended.

Also, if your number of sims is low, you might have a variance issue where you just don’t explore enough. It’s only guaranteed to converge when you take N to infinity. Variance in your search process as well as inaccuracies in your evaluation can definitely screw with the quality of your policy recommendation. That’s why it’s not a guaranteed improvement.

That is why I suspect AlphaGo Lee had the playouts to the end in their training algorithm, so that they could ground this thing in real playouts. In practice, what you could also do is, for 10% of the games, prevent the bots from resigning and just say, “Resolve it to the end.” That way you get some training data in your replay buffer to really resolve those late-stage playouts that normal human players would not play to.

If you assume that the value functions are correct, this is why MCTS gives you a better policy. It’s a very critical chain of assumptions. Assuming this is accurate, your search process should give you a better recommendation than your initial guess.

Dwarkesh Patel

If you have a cold-started policy—if you have an AlphaZero-type thing—really what’s happening for the first few epochs is that the policy is kind of useless. What you’re really just doing is, “Hey, let’s play full games, and once we have played full games, for the preceding moves, we’ll have labeled who won and who didn’t win.” The loss for AlphaZero has two components: how good is the policy relative to MCTS, and how good is the value prediction relative to who actually won the game from this move. You can think of this being applied to every single action or every single move.

Really what’s happening at the beginning of AlphaZero training is just that we’re trying to get the value function to actually predict who will win the game if you find yourself in this state and you’re this player. Functionally, that’s all that’s happening. Later on, once that’s well trained, now the policy is also improving.

Eric Jang

Correct. There’s one trick I found to be pretty useful. This is not a peer-reviewed claim, so take it with a grain of salt. I found it useful in my own implementation to do the following. You want to first make sure that this is good before you invest a lot of cycles doing MCTS. It doesn’t really make a lot of sense to do search on garbage value predictions. You want to start at a good place where this works.

AlphaGo Lee does a very good thing where it just takes human games, you train on them, and it just works. It totally works. You can also take an open-source Go bot, play it against itself, generate data, and that also works. If you have some offline dataset that has realistic, good play, you can easily learn the late-stage value functions pretty well. That’s what you need to start the search process.

Dwarkesh Patel

Sorry, can you just repeat that sentence one more time?

Eric Jang

Sure. It’s quite easy to evaluate a late-stage Go game. When almost all the pieces are on the board, it’s almost like a decidable problem because there’s lower and lower uncertainty as to the depth of the tree. So most games played to the end by reasonable people will be good training data to train a good value function at terminal parts of the tree.

Then, as you play more games, the search will back up good values into the intermediate nodes of the tree. As you increase the amount of data, your value head gets a good intuition of what is a healthy board state versus a not-healthy board state. Those are much more subtle to judge in the mid-game than at the beginning or the end.

The most difficult part to score is not the beginning, because the beginning is obviously 0.5, and at the end it’s pretty obvious who’s winning. The hard part that you want to learn in the value function is who is winning in the middle.

Dwarkesh Patel

So this is actually very analogous to TD learning.

Eric Jang

Yes. There’s a beautiful connection to TD learning that we can talk about in a bit, contrasting it with Monte Carlo tree search.

You first want to get good value functions, and expert data can give you a quick shortcut. I recommend for practitioners, just do that first to initialize to a good starting point. If you want to do the AlphaZero thing or KataGo tabula rasa learning, what you can try to do is play random games on a small board. Just take a random agent. If you play 50,000 games, you’ll actually learn a pretty good value function as well. On a 9x9 board, you can see enough of the common patterns with random play.

If you train on both 9x9 and 19x19 data—and KataGo proposed one of these architectures—there’s pretty good transfer learning from the value head evaluated at 9x9 to 19x19.

Dwarkesh Patel

Right, because this, unlike other games, has very much a sense… There’s not like a new kind of piece introduced when you increase the size or something.

Eric Jang

If we take it to its limit and consider a very tiny 4x4 Go board, if you play 50,000 games, you’re going to have a lot of end states that look like human play. It’s just Tic-Tac-Toe at that point.

If you broaden this a little bit to 5x5 or 9x9, it’s not unrealistic to imagine that purely random play will generate pretty reasonable-looking boards. You can score those pretty easily. That is what gives you the bootstrapping to then improve your policy with search.

But it’s very critical that MCTS has accurate value estimates. You need to ground the value. Ultimately, MCTS will fall apart if you don’t have a grounding function for the value.

Dwarkesh Patel

I’d be curious how much compute you save by training the value and policy on the same network. Because they share the same representations, how much more efficient is learning? We’ve just talked about how they’re making similar predictions, or they should be in line with each other. So I’d be curious if you’re halving the amount of compute you have to do by keeping them in the same network.

Eric Jang

AlphaGo Lee, the original AlphaGo paper, had two separate networks. In all subsequent papers, they merged them into two heads, and presumably this saves compute. But answering that question in a rigorous scientific way is a simple question that in practice, if you really want to chase the question down to its limit, takes quite a bit of work to really resolve.

Intuitively, yes, they share a lot of representations. As we mentioned, your policy network and your value network, when doing evaluations, should agree. There really should be this consistency between them.

Dwarkesh Patel

Tell me if this is the wrong way to think about it. When I learn how an LLM works and how simple RLVR is as an algorithm, I’m stunned by the kinds of things it can do. It can learn how to build very complicated code repositories simply from getting a yes or no.

Here, if you understand it more deeply, just predicting MCTS, AlphaGo seems less impressive in retrospect the more you understand it. You’re putting in a lot of bias by telling it how it should titrate exploration as things go on. You’re building this very explicit tree search for it. I don’t know if you share that intuition where the more you understand it, the less impressive the accomplishment in 2017 seems.

Eric Jang

I personally disagree. I think they’re profound for different reasons. I don’t understand the LLM RL enough to comment on it on your podcast. But why is AlphaGo a profound accomplishment? It’s worth stepping back a little bit. It is different from modern RL, and we can talk a little bit about some of the algorithmic choices there.

I think the most profound thing here is that a 10-layer neural network pass, basically 10 steps of reasoning… Of course, the reasoning is not just one trail of thought. It could be distributed representations and a lot of thoughts going on at the same time. But by construction, let’s say a 10-layer neural network can only do 10 sequential steps of thinking. 10 steps of neural network parallelized distributed-representation thinking is able to amortize and approximate to very high fidelity a nearly intractable search problem.

This was a breakthrough that I think most people don’t even fully comprehend today, how profound that accomplishment is. This is what also girds AlphaFold, for example, where you have a very, very difficult physical simulation process where you would need to roll out as so many microscale simulations, and yet 10 steps of a somewhat small neural network can somehow capture what feels like an NP-class problem into a single problem.

It actually makes me wonder if our understanding of problems like P=NP, or these fundamental computational hardness problems, is incomplete. Obviously, it’s not a proof of P=NP, but there’s something to it that is very disturbing, where what felt like a very hard problem can fall to a very simple macroscopic solution.

Dwarkesh Patel

That is a very interesting insight, that a lot of problems which are proven to be NP-hard—I don’t know if Go is proven to be NP-hard, but protein folding, et cetera—neural networks can solve. They’re NP-hard in the worst case, but we’re usually not concerned about the worst case. These problems usually have a lot of structure to them.

Eric Jang

I think the question we should be asking ourselves is about how we’ve been formulating solutions to NP-hard problems in worst-case complexity. I wouldn’t say this solves Go. It doesn’t give us an exact solution of the optimum, but in practice, it is extremely useful. The same thing has been shown in AlphaTensor and AlphaFold. Yes, there is a very hard problem that, in the worst case, seems intractable, and yet we’re able to make almost arbitrary amounts of progress.

In the limit, what might this look like? If you want to simulate something very complex like weather, or predict the future—do we live in a simulation or not—the computing resources you need to build a very complex simulation might be much smaller than you think, based on our ability to amortize a lot of that computation into the forward pass of a single network. To me, AlphaGo was the first paper that really showed this profound level of simulation being compressed into a small amount of compute.

Dwarkesh Patel

I feel totally not qualified on the computational complexity or the math to comment on this, but I wonder if there’s an important role of chaos here. What is the problem with weather, and why does it take 10x the amount of resources to predict weather a day out, and continually so for every additional day out? It’s because it’s a chaotic system, so small perturbations can totally change the final estimate as time goes on. I guess you would expect that for Go and protein folding as well.

Eric Jang

Here’s an analogy to weather that might be relevant in Go. Here’s our current board state. Given what we know about both players, what is the exact board state in the future? This is extremely sensitive to initial conditions. A single stone placed here can disrupt the entire prediction. This is hard. Intuitively, this is the chaotic problem.

Yet somehow, we can predict who’s going to win. This captures a lot of possibilities. There’s a more macroscopic quantity that we really care about, which is the average or expectation or some sort of global macrostructure over a lot of possible futures.

In weather, it could be the same thing. We don’t exactly care what the velocity of wind 6,000 feet above a specific latitude and longitude is. We care where the hurricane is, and things like that. In chaos, there’s the classic Lorenz attractor. If you start anywhere on the Lorenz attractor, you don’t know where you’re going to end up, but you do know that the thing looks like this. There’s a kind of beauty here. Sometimes we don’t necessarily care about the microscale things. We actually care about the macroscopic structure, and these things can be predictable.

Dwarkesh Patel

Contrast that to something like a hash function, which is also incredibly dependent on initial conditions but doesn’t have a macrostructure, or at least hopefully doesn’t, if the algorithms work.

Eric Jang

One would hope.

Dwarkesh Patel

There’s no equivalent of a value function or broadly how the weather is going to be that’s interesting there. It’s really just about what the board is going to look like exactly 100 moves from now exactly.

Eric Jang

Intuitively, that seems correct. This is also out of my area of expertise, but I find it interesting that the tools of cryptography and hashing have also not been able to prove that you cannot come up with fast approximations. You cannot come up with fast approximations. If they were able to do that, then you could prove P is not equal to NP.

Dwarkesh Patel

In fact, we know that there’s structure in many cryptographic protocols, obviously RSA cryptography. There is structure, and that structure is what quantum computers exploit to break them.

Reiner has a very interesting blog post, which we’ve talked about in the episode, where he points out that if you look at a high level at what cryptographic protocols look like and what neural networks look like, they’re extremely similar. You have sequential layers of jumbling information together. There’s a convergent evolution in the algorithms. In cryptography, you want the final state to be incredibly sensitive to initial conditions, so that it comes out looking jumbled if you change anything. In neural networks, you similarly want everything to be dependent on all the information, because you want to process all the information and consider how it relates to itself.

Eric Jang

You have the maximum power of a neural network at the edge of chaos. I think there are some research papers from Jascha Sohl-Dickstein on this. There’s something quite fundamental about chaos that’s not just hopeless noise. There’s something useful in chaotic systems, at least at that boundary. But this is just thinking about it as a philosophy. I don’t know the math well enough to comment on it.

We’ll talk about LLM RL in a little bit, because there’s some connection there, but let’s go back to MCTS. What is it doing? Crucially, it is not saying that we’re going to increase the probability of winning directly. It’s not going to upweight all actions that won and downweight all actions that didn’t win.

Importantly, what it is doing is saying: for every action we took, we did a pretty exhaustive search on MCTS to see if we could do better, and we’re going to make every action that we took better by having the policy network predict that outcome instead. This is a very nice idea because you have one supervision target for every single action. The variance of your learning signal is very low compared to the alternative naive RL thing.

Let’s consider a very naive algorithm that looks a lot more like modern LLM RL today, where we do something like take the winner of a self-play game and encourage it to do more of that.

Eric Jang

It’s worth thinking a little bit about what alternatives we could use to train self-play agents instead of MCTS. We use a lot of LLM-style RL these days. Is that relevant? Could we do that instead?

Suppose we have a very naive algorithm where we take a league of agents of different checkpoints and play them against each other. For the games where a single player wins, we reinforce those actions up and retrain the policy network to imitate those winners, instead of the MCTS objective. Let’s say you have a chain of actions that led to a win, and you have a matchup between two agents that are basically the same. Assume policy a (Πa) and policy b (Πb) are evenly matched, so their true win rate is 50%.

Let’s say you play 100 games, and each game lasts 300 moves. You’re doing some sort of evolution strategy or some way to perturb these things to get them to do different things. Or maybe you don’t, and you just play them against each other to see that occasionally one might have a better strategy than the other. Let’s say policy a wins 51 games and policy b wins 49. This is just due to random luck, or maybe you perturbed policy a in some way that let it do this.

Just to have a very simple model, let’s pretend that for 50 of the games, they played exactly equally. In the one game where a won, it played slightly differently. It made one critical move that normally it would have done differently, but due to some exploration or random noise, it happened to make a smarter move than it did previously.

So you have one true supervision signal for your policy network, and then you have 99 games times 300 moves for which imitating those actions gives you exactly the same policy you had before. The scale of your variance is actually very bad, because you only have one label out of this enormous dataset of supervision actions where you want… Actually, let me clarify a little bit.

Dwarkesh Patel

We’re talking about how the good move, the out-of-distribution move, is a small fraction of all the moves that are played across all the games on which you’d want to train. This, of course, reminds me of how LLMs are trained with policy gradient methods. When Karpathy was on the podcast, he called it like “sucking supervision through a straw”. It’s interesting that this thing you’re saying—which would be intractable and prevents you from actually getting beyond a certain level in Go—is just by default how LLMs are trained?

Eric Jang

Right. This is not to say it doesn’t work. If you imagine increasing the number of games to millions of samples, you actually can get some meaningful supervision samples, so long as you find a way to mask out the supervision from these guys. This is where things start to get pretty related to RL in terms of advantage and baselines.

Let’s look at the gradient variance of a very naive approach—I’m going to call it gradient RL—that’s basically the sum of rewards. The sum of rewards is the return. In our naive setup here, we only have an indicator variable for the return, where either you won or lost. In the case where you lost, your gradient is zero, so you don’t train on those examples, and when you won, you try to predict those things. You can think about this setup as a special case of this general formula here.

The trouble here is that this is very high variance. When you multiply these terms out to compute the variance of the gradient, it’s equal to the expectation of… Just for simplicity, we can pretend this is on average zero or something if you’re centering it at no signal. The variance here basically means that you’re taking the square of this product term. You end up with a term that grows quadratically with T. When you have a setup like this, this thing acts as a coupling effect on top of these terms here.

Let’s actually map this to an LLM case, and we can answer why LLMs only do one-step RL instead of a multi-step RL scenario. In LLMs, you have a decoder that might predict some words like “hello world”. In current LLM RL, they treat this entire sequence as a single action, at, and big T is just one. It’s true that because of how transformers are formulated through the product of conditional probabilities, the log probability of the whole sequence is equal to the sum of the probabilities of individual tokens.

In this case, I would say something like log(hel) + log(lo) + log(world). This is true, and if this term were one, then they would be the same thing. However, when you’re sampling, if you have a reward term assigned to every specific token, now you have these interaction effects between the cross-multiplication of these terms and these terms. The problem becomes, how do you ascribe the credit associated with every episode to all these different terms?

Dwarkesh Patel

The thing I’m confused on is what it would even look like to do it that way in LLMs, because you only get a reward at the end of the episode.

Eric Jang

You could imagine a reward that says, “I’m going to give you some process supervision where you get a reward for each of these actions on every step.”

Dwarkesh Patel

Okay, so you’re saying if instead of doing it that way… Well, the way you’ve written it, it would be a sum at the end anyway, so they wouldn’t have to be multiplied. But you’re saying instead of doing it that way, you would just add up these process rewards at the end and then treat that as one single reward signal?

Eric Jang

Correct, for one single log prob action.

Dwarkesh Patel

But isn’t that how it’s written to begin with anyways? The sum of the rewards?

Eric Jang

The thing that’s a little bit hidden here in the math is that we’re assuming that when you decompose the problem into a multi-step problem, you’re now introducing correlations between your actions through the computation of this guy. If you separate these things out, this will magnify the variance of this one.

In the case where you don’t separate it out, if you just have T=1, you have a single estimate of log prob and a single estimate of reward. This term still shows up. In LLMs, the naive REINFORCE estimator looks a bit like the return of the single action… It looks kind of like this. This is the basic form here, but it’s still a contributor to variance. You want to make sure that, similar to how in this case we were training on a lot of neutral labels, you’re sort of penalizing the labels that don’t help and only rewarding the ones that actually make you better.

Intuitively, the analogy is: can we find a term in our training objective such that it’s actually discouraged from doing this, or these don’t have any effect on the gradient, and this has an effect on the gradient?

Dwarkesh Patel

If you applied that there, the only thing you could do is eliminate 49 of the games. At least the way you have it written there, it would be 51 times…

Eric Jang

Actually, the optimal case is to discard all of these moves, and only get a gradient on that single move that you got better on.

Dwarkesh Patel

How would you do that?

Eric Jang

This is a pretty tricky problem in practice. This is where advantage estimation happens in reinforcement learning. You want to subtract a term from your multiplier. Instead of an indicator function of one and zero, you want something that behaves like a zero for all of these guys, and then a one for these ones.

Dwarkesh Patel

You could do that if you can say, “Hey, I won this game, so this is slightly above baseline performance.”

Eric Jang

Well, you won a lot of games. But you don’t know which ones let you win because they were truly better versus winning by accident.

Dwarkesh Patel

How would you design a baseline where it’s truly better?

Eric Jang

This is where in RL people use things like TD learning to better approximate the quality function, the Q that we mentioned earlier. You can try to subtract that from your return.

Ideally, what you really want to do in RL is push up the actions that make you better than average and push down the actions that make you worse than average. They call this advantage. There are multiple ways to compute it. I highly recommend John Schulman’s “Generalized Advantage Estimation” paper as a good treatment on how to think about various ways to compute it. At the end of the day, you want to reduce variance by trying to make this smaller, so it doesn’t magnify the variance of this one.

Dwarkesh Patel

That makes sense. This requires you to have a very good estimate of what average performance from a state would look like. This gets us back to the value function thing we were talking about earlier.

Eric Jang

Keep in mind that this model-free RL setting is trying to solve a credit assignment problem where you don’t know which actions were actually good and which ones were bad. Monte Carlo tree search is doing something very fundamentally different. It’s not trying to do credit assignment on wins. It’s trying to improve the label for any given action you took. We can actually think about a completely different algorithm called neural fictitious self-play, which was used to great effect in systems like AlphaStar and OpenAI’s Dota.

Let me talk a little bit about how you can unify some of these RL ideas in the model-free setting as well as the self-play setting. What happens if you don’t have the ability to easily search a tree? In Go, it’s a perfectly observable game. You can easily construct a pretty deep tree that completely captures the game state. In a game like StarCraft where you don’t have complete control over the binary, it’s a little bit hard to do this, and I’m not even sure if it’s a deterministic game. That makes this difficult from a data structures perspective.

What is done instead is that the basic idea of supervising your actions with a better teacher is still there. We’re going to talk a little bit about how neural fictitious self-play works. It’s the same idea. We’re going to come up with better labels for each of the actions we took, just like in MCTS. But how do we derive the better labels? In MCTS, we perform search. Assuming we have a good value function, the search will give us a better result than our initial guess.

In a game where you can’t easily simulate a search process, what they do instead is train what’s known as a best response policy. You fix your opponent. Let’s say you’re currently training Πa against a strong opponent, Πb. In StarCraft, maybe these are the Zergs and you’re playing Protoss or something. You fix your opponent, and you treat this as a classic model-free RL algorithm, where your goal is just to beat this guy. Here you use your standard TD-learning-style tricks, or use PPO or any model-free RL algorithm to try to hill climb to winning against this player. You have a reward function where the return is one if it wins against Πb, and zero otherwise. This is no longer a self-play problem. It’s just a fixed opponent, and you’re trying to maximize a score against them. You have a fixed environment where all you care about is beating this guy.

Once you have a good policy that you train with your favorite model-free RL algorithm—PPO or SAC or V-MPO—you now have a good policy that gives you a good label for what this one should do when playing against that player. When you train multiple best response policies, you can basically distill the RL algorithms into the labels for a given opponent. You might have a best response policy against πb, and then maybe you have a league of opponents like πb, πc, πd. You’re going to take the best response policy that you train against each of these fixed opponents, and supervise them with the label that this one would provide.

This is almost like a proxy for your MCTS teacher. Instead of an MCTS teacher, you use a model-free RL algorithm to find the best search action that you could do to beat your opponent. Then you’re distilling the policy here into what’s known as a mixed strategy, where it’s trying to average across all possible opponents you could play against. This is what gives you something that can do no worse than an averagely selected opponent from the league. This gets around the problem of having to derive a teaching signal from MCTS, but fundamentally it’s still about relabeling your states with better actions so that they improve your policy.

Dwarkesh Patel

Just to make sure I understand, if you win a game against this other policy, you reinforce all the actions on that trajectory.

Eric Jang

Yes. Here you can use a number of algorithms like PPO, V-MPO, or even Q-learning if you want. The specific algorithm here is usually a model-free thing because you don’t have search, but there’s an interesting connection between MCTS and Q-learning that I want to bring up.

In MCTS, you have a tree, and through the resolution of your value function at the approximate leaves of the tree, you can back up through many sequences and obtain some sort of mean value estimate. Your Q is derived from the average of a bunch of simulations.

In model-free algorithms, there’s often a component of estimating a Q value. Q values are often learned through TD learning, although in PPO, the way they do advantage estimation is not necessarily through a Bellman backup. In Q-learning, there’s this very cool trick where Q(s,a) is backed up as r plus some discount factor times the max over a, Q of your next step. Intuitively, if you have an MDP and this is terminal, it’s saying that the best action you can take at this state is equal to the reward you get for taking this action, plus the best that you can do at the next state.

There’s a recursive, dynamic-programming property of MDPs, and you can train neural networks to basically enforce this consistency. You can say, “Well, once I know the Q value of this action, I can then use that to compute something about the Q value for the next.”

Dwarkesh Patel

Earlier I was like, “Hey, why are we training policy? Why don’t we just train the value alone?” That is what this is.

Eric Jang

This is an algorithm for recovering value estimates of intermediate steps when you don’t have the ability to do forward search. You must collect a trajectory first of n steps before you’re able to do this trick. But the intuition is kind of the same. Knowing something about the Q value here can tell you something about the Q value here. And indeed, you can recover a policy from a Q value. You don’t need to explicitly model the policy distribution. You can recover the policy distribution by doing argmax over your Q values.

Q-learning, or approximate dynamic programming, propagates what you know about the future Qs backward like this. You can see that there’s a similar structure going on here, where in one case you’re planning over trajectories your agent hasn’t been to yet, whereas in the other you’re planning over trajectories your agent has visited.

Importantly, why was Q-learning a big deal? It’s because historically we just haven’t had the ability to do search on fairly high-dimensional problems like robotics or whatever. For a long time, we made the assumption that if we can’t model the dynamics with a world model, we’re going to instead just collect trajectories and then plan with respect to the only number that really matters, which is reward.

Dwarkesh Patel

This is very interesting. To unify this with our discussion of LLMs, you don’t have Q values, but you’re doing this backwards learning where you find the trajectories that pass some unit test in some coding environment and then reinforce those trajectories.

There’s a huge difference between that and this forward approach with MCTS. The reason it’s much more preferable to do MCTS is because you can do it per move and make each move better, rather than having to learn per trajectory and hope, as Karpathy said, to learn this…

Eric Jang

Through a straw.

Dwarkesh Patel

Right, you get supervision through a straw. You basically just upgrade all the tokens in a trajectory that might or might not have been relevant to getting the answer right.

The reason you can do this much more sample-efficient, favorable thing with Go is that because MCTS works in Go, you basically know that if you just do the search locally here—and this search is truncated at the end by a value function that works even if you haven’t unfolded your whole trajectory—you can just say, “This is my new policy,” and improve in a more iterative, local way, rather than having to unfold all these trajectories.

Eric Jang

There was some research from Google in 2023 or 2024 where they tried to apply tree structures to reasoning. The jury is still out as to whether this can ever work. We probably will see a revisiting of this idea of forward search in the future.

But there are two things that make MCTS very simple for Go. Value estimation is concrete. You can determine it for real, and then you can use it to truncate depth, as you said. The breadth is also determined. What’s critical is that the action selection algorithm, where you iteratively visit and grow the tree, is well suited for the size and depth of problem that Go is.

But for something like LLM reasoning, PUCT might not be a good enough heuristic. It might be too greedy with local tokens, and it might only give you obvious thoughts that are correct but don’t really solve your final problem. I would say the jury is probably still out on what the final instantiation of reasoning for LLMs will look like. I wouldn’t rule out that this stuff could come back, but it’s a bit hard.

Dwarkesh Patel

Don’t LLMs natively learn to do MCTS, where they’ll try an approach and be like, “Oh, that doesn’t work. Let’s back up. Let’s try this other thing,” and then go in the direction that proves to be more fruitful?

Eric Jang

Certainly, LLMs manage to do something that looks like real human reasoning without having to do an explicit tree structure. That being said, I think the idea of doing forward search and simulation to get a better sense of what is valuable might make a comeback, even if not in exactly the same instantiation as AlphaGo.

Dwarkesh Patel

Just to make sure I understand the crux of it: the breadth comes from the number of legal actions being wider, and the depth comes from not being able to train a value function as easily, because…

Eric Jang

Here’s an example where LLMs break down. The CPUCT rule involves the √N over 1+Na. In an LLM, you’re most likely never going to sample the same child more than once. If you have multiple steps of thinking, because language is so broad and open-ended, a discrete set of actions is not really an appropriate choice for an LLM. Even though they’re discrete tokens, it’s just such a large number that this type of exploration heuristic is probably not the right thing to guide how to search down a tree.

Dwarkesh Patel

I guess the crux comes down to the fact that in Go, you know that MCTS is almost certainly better than your current policy, even though you haven’t explored the end of any trajectory. In normal reasoning for LLMs or robotics, there’s no way to just locally evaluate and improve your next move in a way that’s independent of actually solving the problem.

Eric Jang

“No way” is a strong word. Lots of people have thought about how to apply MCTS or its successors like MuZero to continuous control spaces, and I’m sure very cool research work is still ongoing to try to crack that problem. But yes, the apparent challenge right now is that most problems in much higher-dimensional action spaces, or something combinatorially much bigger like language, don’t seem as amenable to the discrete action-selection heuristics and game-evaluation type stuff that Go has.

That’s not to say the idea of thinking into the future along multiple parallel tracks might not give you some information about which way to search. If you think about mathematics, it often occupies more of a logical search procedure where you can back up and see which paths seem good or not. There’s more of a rigid structure there, whereas in a business negotiation it’s less of a tree and maybe something a bit different.

Dwarkesh Patel

We’re now seated, so I can ask you some more questions about AlphaGo and AI research more generally. In 2021, Andy Jones had a paper called “Scaling Scaling Laws with Board Games”, and he basically anticipated inference compute, or inference scaling, by showing that you can trade off test-time compute and training compute. That is to say, you can spend more compute searching through MCTS, and if you do that, you can get the equivalent performance to having spent more time training the model.

If you see this pattern, you might think that with LLMs you might do something similar in the future, and in fact that’s what ended up happening.

What is a fun exploration one could do now to explore other axes of scaling in toy settings, which will be important to understanding what AI development might be like in a few years?

Eric Jang

Test-time scaling and reasoning, and how they interact with model size, are quite profound when it comes to how much needs to actually be done as explicit search versus how much can be packed into the forward pass of a neural network. How does a forward pass of a neural network learn to do something that should be sequential and recursive? That’s quite interesting.

The Andy Jones “Scaling Scaling Laws with Board Games” paper is quite cool. There was another really nice result from that paper. Not only can you predict scaling laws of the LLM variety—where as you increase parameters you can decrease the amount of compute for search or vice versa—but he also showed that you can predict how much compute is needed to solve a larger version of the board game.

For example with Go, which can scale from 3x3 to an infinitely sized Go board, you might be able to revisit this question and try to reproduce whether this shows up. I actually started this project with the motivation of asking: does the “Bitter Lesson”, or our knowledge of scaling laws, allow us to execute a lot better on a compute-optimal Go bot? Can we build a strong Go bot without all of the KataGo tricks, just by really focusing on the Bitter Lesson and scaling laws?

I have not been successful so far, but usually when you want scaling laws to work, you want to be in the regime where the recipe already works and the data sets are good, rather than trying to figure out how to do scaling while also trying to figure out what the right data sets are.

The scientific understanding component in research often follows a step where you get something to work first. Then you use that system to collect data that then helps you build a mental model of how things work, such as scaling laws. Usually, if you want to build a strong Go bot using scaling laws, you actually have to make a strong Go bot first and then use the scaling laws to extrapolate a bit further into the future.

Dwarkesh Patel

Say more. Just so I understand, are you saying scaling laws did not work? There was no scaling-laws pattern that you could see in your Go bot?

Eric Jang

A mistake I made initially, when I had some bugs around how MCTS labeling was working, was that I would collect a bunch of data with an expert policy and then treat it as a supervised learning problem and try to identify scaling laws with expert data sets. You can indeed plot things that look like this, but if you’re in a regime where your policy is not working well, you might just be studying scaling laws on bad data.

One important implementation detail is that if you want to study a scaling-laws problem, you have to have a problem for which the data is good, the architecture is good, and there are no bugs, and then you solve it from there.

Ex ante, I wasn’t able to apply scaling laws to direct what to look at until I had the rest of the system working. This sounds obvious to researchers. Of course you want a working, bug-free system before you study scaling. But as advice for practitioners, on where I tripped up when I started this project, you don’t necessarily want to jump into the science of studying your man-made artifact before your man-made artifact is interesting enough to be studied.

Dwarkesh Patel

Speaking of compute, you can look at these charts of compute used to train the best AI model in the world over time going back 10 years. It’s a very smooth line in log space that grows exponentially year over year. Except there’s this huge aberration, and that aberration is AlphaGo Zero, which was trained on way more compute than any other AI model at the time. It was 3E23 FLOPS. It’s comparable to a frontier LLM, orders of magnitude off, but still.

The question is, especially with you being able to get something off the ground, did you train it on your own?

Eric Jang

I got a donation from Prime Intellect for about $10K. I spent maybe the first $4K doing exploratory research, and then about $3K on the final run. Some of it remained for serving the model.

Dwarkesh Patel

Is your sense that they did a bad job training it, if you can do it in $10K now?

Eric Jang

The compute required to be the first to do something is always much larger than the compute it takes to catch up. It’s the same story playing out in LLMs. Once someone else has done it, you can use tricks like distillation. You can use all sorts of crutches to bootstrap your way to success.

With my own bot that I’ve hosted online, I actually used best-response training against the KataGo models to get a strong level of performance. As of the time of recording, I’m validating whether I can do that first step, which is to do the tabula rasa play.

Importantly for research, you often want to start from a good init. The simple thing I did first was train best-response agents against KataGo. The AlphaZero team did not have any policy they could train against, because they were trying to do everything tabula rasa. Being the first to do it means you’re prioritizing getting the thing working rather than the most compute-efficient possible implementation.

This plays out in robotics as well. If you look at the frontier of large models trained for robotics, the scatter plot is all over the place, and there isn’t a very clean line the way there is for frontier LLMs. That’s because the folks training these models often are not at the scale where every FLOP counts and they need to squeeze out the performance of every single FLOP as the deciding factor in pre-training. Instead, their focus is more, “We want a certain capability to show up, so we optimize the training setup to make it easy to derive that capability.”

Once you have that capability… Invariably if you scale up the compute, you are forced to make it compute-efficient, because this is hundreds of millions of dollars we’re talking about. But in the past, when compute for experiments was more plentiful—or not accounted for in a way the researcher was really responsible for—you end up with people optimizing for things besides being on the compute-optimal Pareto frontier.

Dwarkesh Patel

I see, like speed or something?

Eric Jang

Yeah, time to result, or just getting it to work. The first AlphaGo probably had lots of compute, and they didn’t need to worry too much about making it the most compute-optimal thing.

Dwarkesh Patel

How much of the improvements to compute efficiency are methods that did not exist as of 2017 versus things which they could have done in 2017 but didn’t?

Eric Jang

Great question. Going into this project, I knew in the back of my mind that things always get easier to do over time. I wanted to see where Go was at, given that there hadn’t seemed to be any major open-source strong bot after KataGo in 2020.

Reading the KataGo paper, there were a lot of clever ideas. I was thinking, “Let’s see if the Bitter Lesson had happened, where a lot of these tricks just go away because Nvidia made faster GPUs. Roughly, where are we on that?”

Again, this is not a peer-reviewed claim. It’s just my preliminary vibe guess based on what I’ve seen with my own experiments. It seems like architecture choices don’t matter that much. Transformer versus ResNet… We’re at the speed of GPU where the size of the model is not so big that this really matters.

You can simplify the setup quite a lot. Instead of doing a distributed asynchronous RL setup with replay buffers and pushers and collectors, you can do a dumb synchronous thing where you collect, train a supervised learning model, and then collect again. There are opportunities to simplify infrastructure.

Nvidia GPUs have indeed gotten faster. Whereas KataGo was trained on V100s, you can train on half the number of desktop Blackwell GPUs and it still works. Some of the auxiliary supervision objectives that KataGo developed aren’t really necessary if you have a strong initialization. If you’re initializing best-response training against KataGo itself, your own model needs none of the tricks that KataGo needs. The core thing is getting as quickly as possible to some strong opponents. That matters a lot more than the specific architectural innovations.

But there are still some nice compute multipliers. I found that training on 9x9 boards was very nice for resolving endgame value functions. If you can co-train that on an architecture that can transfer between 9x9 and 19x19, you can really cut down the warm-start time to learn from scratch. AlphaGo Zero’s plot showed the first 30 hours or so spent basically catching up to the supervised learning baseline. You can cut down that time a lot by pre-training on a small board and then warm-starting that into your 19x19 board play.

There was some other stuff, like varying the number of sims between episodes. This turns out to be not that sensitive. You can fix it or increase it. It doesn’t matter too much. Anyway, it’s nice from a scientific perspective, revisiting an old paper and seeing what really matters.

Dwarkesh Patel

This is a tangential question, but why is it okay to have a buffer in AlphaGo? Every time I talk to an AI researcher, they’re telling me about how bad it is to be off-policy. But the way a naive implementation of AlphaGo Zero would work is that most of the moves in a given backward step, or in a batch of backward steps, would not be among the ones made by the most recently trained model. So why is that okay?

Eric Jang

Great question. This gets into the fundamental off-policy versus on-policy reinforcement learning questions. As you’ll recall, in MCTS, you take actions that you took, and you relabel them to take different actions on the same states.

The off-policy part comes in where, what if you’re relabeling states that your new policy would never visit? What’s the point? You’re wasting capacity. In the extreme limit, imagine the distribution of states in your training buffer are all states you would never visit. Then you’re supervising them to take good actions on states you would never achieve. Therefore, your policy can get really bad. This is where off-policy can really hurt AlphaGo.

However, if you interpret this from the DAgger perspective—which is basically a way to correct yourself back to the optimal trajectory given some data—what you want in an algorithm like this is to have mostly states that you would visit, but then a small or reasonable percentage of states in this high-dimensional tube around your optimal trajectories. Any of those states are given a supervision target to funnel you back into your optimal trajectory.

Maybe I can just draw real quickly here. In a DAgger-style setup, your optimal training data distribution is your optimal states and actions. You want to be in this state, you want to be in this state, you want to be in this state, and then you win here. These are your optimal policy actions. These are the things you definitely want to train on. But to make it robust to disturbances, you want to make sure that if you happen to drift off into some other states, you can funnel yourself back in.

Dwarkesh Patel

But why isn’t this a fully general argument for off-policy training?

Eric Jang

This is actually why you want to do off-policy training sometimes. You don’t want a compounding error where, if you make a mistake, you don’t have the data for how to return to your optimal distribution.

Optimal control doesn’t really say too much about how to not accidentally get here, because it’s making the assumption that once you learn the policy, you’re going to get to here. But in applications like robotics, a gust of wind blows you slightly off and now you need to correct. Or the friction on one of your tires is slightly lower than the other wheel, and now your car is drifting and you have to correct it. These kinds of things often happen in more real environments.

There was a funny quote about chess and Go. The problem with Go and chess is that the other player is always trying to do some shit. Things can drift off, and you always want to be able to correct back to your winning condition. Your replay buffer really should have the states your policy would visit, plus some distribution of states you might drift to, and then how to return back to your optimal states.

Now, if you take this to the extreme and you say, “We don’t have any of this data, and we’re just going to be labeling with MCTS states that are so far away from our optimal behavior, like this bag of states over here…” Well, now each of them gets an MCTS label, and your policy learns how to take the best possible action here, but you never get here. You’re training your model on states you would never reach. This is where off-policy can really hurt.

As part of this project, I did try an experiment where I took a bunch of trajectories, and to try to saturate the GPU as much as possible, I took random states from the dataset and reran MCTS on just those states. Instead of playing a whole game where I’m doing MCTS on every move, I ignore the causality of moves and pick random board states, and I label those with my current network. I might revisit old states I’ve labeled before and relabel them again with my current network.

In practice, this actually does work. You can take some states that are reasonable and constantly be relabeling them while training. This starts to converge on a very robotics-like setup, which is very common. You have your dataset of trajectories, and then you have something like a replay buffer pusher. These are off-policy offline trajectories. Your replay buffer pusher pushes transition tuples to the replay buffer, and then you have a job that’s continuously replanning what the best action you should have done instead of the action you took.

In robotics, it’s very common to minimize TD error. Your Bellman updater is constantly pulling things from here and trying to satisfy Q(s,a). From here you have your trainer, which is trying to fit the Q to the Qtarget. You can think about this as a sort of planner. You visit old states you’ve been to, and you take your current model and rethink. What could I have done better if I visited this? This is how off-policy robotic learning systems are usually trained. These days there’s a simpler recipe, but in the Google QT-Opt days, we did things like this.

Dwarkesh Patel

So what is the trainer?

Eric Jang

The trainer is where you try to minimize Q(s,a) and Qtarget.

Dwarkesh Patel

Can you explain the whole setup again at a high level?

Eric Jang

You have your off-policy data that came from various policies. You’re constantly pushing transitions you saw before to a replay buffer.

Then you’ve got this thing called a Bellman updater, which basically replans: instead of this action, what action should I have taken at s to have a better value? The way you enforce that is by trying to minimize the TD error. Given this, you have s’. You compute Q(s’,a) and you find the action that should go with s’ that makes this Q value as high as possible.

Then you add that to the reward here, and that gives you your actual target. For this current s and a, your Qtarget is this. Now you send back the Qtarget to this transition. With this tuple, you pair a Qtarget. Then on the trainer side, you simply use supervised learning and minimize your current network’s Q(s, a) with its target.

Dwarkesh Patel

Got it. So in the background you’re basically thinking through how valuable all these actions actually were.

Eric Jang

Yeah. In a more optimal policy where you’re trying to maximize this, what is the Qtarget of this transition?

Dwarkesh Patel

It’s sort of like daydreaming.

Eric Jang

Exactly. You can think about it like you’re going back in hindsight. Given what I’ve seen in the historical buffer, was there a better action I could have taken?

The connection to Go here—I tried and it was moderately successful, but too complex to open source—was replacing this with an MCTS relabeler. Instead of doing this target network computation, you run MCTS on your transition. In this case, you have your state, your action, and whether you won or not at the game. Actually, you can just toss those two. You don’t care about these ones. You just take your state and plan MCTS to get your best policy, Π, on your current network. Not the network that took this action, but your current best policy network. You rerun your search offline on these transitions.

If these are transitions your policy can get to, this actually acts as a very nice stabilizing effect. Another benefit is that you can fully saturate your GPU better because you’re not blocking on the Go game to give you board states. You simply search across all board states at any depth in parallel. Here the trainer would just predict the MCTS label as best as possible.

This kind of works, and it is quite relevant in robotics where you have a lot of offline data and you can’t simulate things like MCTS. But in practice, it does run into the problem that if the current model is looking at states it would never reach, it’s wasting capacity. You have to be a little bit careful here.

Much of RL has converged to a much more on-policy setup where they don’t really try to directly train on off-policy data. At best, they use off-policy data as a way to reduce variance, but not to directly influence the objective.

Dwarkesh Patel

Why have they converged to that?

Eric Jang

It’s just more stable. You might use the off-policy Q as a way to do advantage computation, like Q minus the sum of Q. If there are N actions, this is your value, and these are your current Q values. Your advantage for that action is the average value minus your current one. People can try to estimate Q in an off-policy way and then just use advantage here.

If there’s a problem in these dynamics, it doesn’t blow up your loss as much. In robotics, there’s a convergence toward using off-policy data to shape your rewards but not actually be directly here.

Dwarkesh Patel

I’m reminded of our earlier conversation about why MCTS is so favorable compared to the REINFORCE or policy gradient thing LLMs do. This might be totally wrong, but I wrote a blog post a few months ago about how policy gradient RL is even more inefficient than you might think.

The inefficiency one thinks about naively is the fact that you have to roll out a whole trajectory to get any learning signal at all. As these trajectories become longer and longer—as an agent has to do two days’ worth of work to figure out if it did a project correctly, rather than just completing the next word in a sentence—the amount of information per FLOP decreases. You have to unroll two days’ worth of thinking to see if you implemented a feature correctly, so the amount of samples per FLOP has been decreasing.

You’re trying to maximize as you’re learning, bits per FLOP. You can think of bits per FLOP as samples per FLOP times bits per sample. What I mentioned a second ago is that the samples per FLOP go down as RL becomes more long-horizon.

But this kind of naive RL is also terrible from a bits-per-sample perspective, at least compared to supervised learning. Early on in training, let’s say you have a vocabulary size for an LLM that is 100K long, so there are 100K possible tokens that one could answer. You have a totally untrained model and a prompt like, “The sky is…” With supervised learning, the model would have some probability distribution over all the things it could say. There’s a label that says the term here is “blue,” and it would learn through cross-entropy loss exactly how far its distribution is from correctly saying “blue.”

If you were doing this through RL, the model would try, “The sky is halycon.” Nope, that’s wrong. “The sky is told.” Nope, that’s wrong. This is a totally untrained model. You would have to do this on the order of 100K times just to stumble on “blue” and get some learning signal. In the supervised learning regime, you have your distribution of probabilities, you get told that it’s “blue,” and you figure out how far off you were. The amount you learn is a function of your pass rate. The further away you are from “blue,” the more you learn to go toward “blue” using cross-entropy loss.

You can think of it as like your pass rate, your prior probability of having said “blue.” As a function of that, in supervised learning, through cross-entropy loss, you would learn negative log(p), p being pass rate, bits, once you get this label. Whereas in RL, if you’re just randomly guessing and seeing if it works, that’s basically just the entropy of a binary random variable.

Eric Jang

What’s also tough here is that the distribution you’re sampling under is your policy’s distribution. If your policy has no chance of sampling “blue,” then you will never get a signal.

Dwarkesh Patel

Exactly. That’s modeled by the fact that your probability of sampling “blue” is extremely low. If you sample it, you do learn as much as you would have in supervised learning. In all other cases—99.99% of the time in an untrained model—you’re learning incredibly little from seeing that “halycon” or “told” is not the correct word.

That’s what happens most of the time. You learn very little. If you put your pass rate on the X-axis and the bits you’re learning from a sample on the Y-axis, with 0%, 50%, and 100%, so at the end of training you’re here. If you have supervised learning, the negative log pass rate would look something like this. The entropy of a binary random variable would look like this, depending on whether you’re doing nats or bits. If you do bits it’s one here at the peak. This is like a coin flip. You learn the most from a coin flip. This is supervised learning. This is RL.

However, the problem is you spend most of training in this regime, in the low pass rate regime. How fast you’re learning is a function of how many bits per sample you’re getting, and you’re getting very little signal here. If you chart the pass rate on a log scale—where at the beginning of training with a vocab size of 100K the pass rate is 1/100,000, then 1/10,000, 1/1,000, 1/100—what this graph looks like here, supervised learning would look like this, and then RL would look like that.

Eric Jang

And arguably you spend all your time here, potentially never even getting a single success. It’s a depressing plot in the sense that once you’re here, it’s not at all obvious how you get to there. Once you’re here you have something, but you actually spend all the time here in many RL problems. There’s a question of how you initialize so you’re at least at a non-zero pass rate.

One more thing I’d like to add about bits per sample that’s very relevant to any machine learning problem is that there’s a connection to soft targets and distillation. If you have access to the logits, not just the one-hot token answer, if you have access to the soft targets the entropy of this distribution is far higher than the one-hot. There’s way more information in bits per sample in a soft label. That’s why distillation is so effective per sample. It’s giving you way more information per sample.

Dwarkesh Patel

I wonder what the equation would be, but obviously it’s…

Eric Jang

It would just be the entropy of this distribution. The entropy of this is zero. The entropy of this is the entropy equation.

This is also why AlphaGo is quite beautiful. In AlphaGo, you don’t train the policy network to imitate the MCTS action. You train it to imitate the MCTS distribution. Both of these are valid, and if you wanted to do a scientific experiment of how important this soft label is, dark knowledge distillation, you can run an experiment where you retrain the policy network on the action MCTS selected rather than the soft target.

Dwarkesh Patel

Earlier I was stumbling around. Intuitively, why is this ability to do iterative search—where you don’t necessarily need to be able to win the game in the beginning, you just need to be able to improve your current policy—so powerful a capability in learning compared to how LLMs currently learn RL? It’s exactly this thing of considering your pass rate of the entire trajectory. I just don’t know a formal way to think about this. Maybe you should help me out here.

Eric Jang

Why is AlphaGo an elegant RL algorithm? The major reason is that you never have to initialize at a zero percent success rate and solve the exploration problem of how to get to a non-zero success rate. This is what allows you to hill-climb this beautiful supervised learning signal.

If you look at the actual implementation of AlphaGo, every step of the way, there’s actually no TD error learning or dynamic programming, at least explicitly. It’s just supervised learning on a value classification as well as a policy KL minimization. It’s just a supervised learning problem on improved labels.

The training is very stable. You can train as big of a network as you want. You can retrain this on the data set. Everything will just go stably. The infrastructure is very simple to implement as well. You don’t need a complex distributed system to keep everything on policy.

At the end of the day, you’re just saying, “I have some improved labels. Let’s retrain my supervised model on these targets.” You’re always in this beautiful regime where you’re just trying to improve the policy, rather than escape local minima where every signal is flat all around you.

One way to draw the curve is if you draw the win rate of an MCTS policy versus the raw network… Let’s say this dotted line is the raw network. The MCTS policy looks like this. Every step of the way, this supervision signal is very clean. You’re never in a situation where the MCTS is giving you no signal, unless your MCTS distribution converges to exactly what your policy network predicts.

Dwarkesh Patel

That’s a great way to explain it. Maybe we sit down and I ask some questions about automated research.

Dwarkesh Patel

One thing I really wanted to talk to you about is that you did a bunch of the research for this project through this automated LLM coding assistant loop. There’s an idea that if you fully automated AI research, you could have some sort of singularity.

Obviously we’re not there yet, but to the extent that we have early indications of what this process might look like, I’m curious about your observations about what the AI is good at, what it’s not good at, what you think about this scenario’s likelihood eventually, and what thoughts you have about this in general.

Eric Jang

I think automated scientific research is one of the most exciting skills that the frontier labs are developing right now. It’s important for everyone who’s doing any kind of research to get a good intuition of what it can do now and what it can’t, and how the science process might work in the future once we have AIs automating a lot of this investigation.

In brief, I mostly used Opus 4.6 and 4.7 while working on this. What works is that the models can do a very good job of hyperparameter optimization. In the past, people would come up with a search space of hyperparameters like learning rate, weight decay, and maybe how many layers are in your network. They would do a grid search or a Bayesian hyperparameter optimization approach, and it would find some tuned parameters.

The really cool thing that automated coding can do now is search a much more open-ended set of problems. It can say, “I’ve identified that the gradients are small in this layer, so let me change it up here. Let me rewrite the code so the data loader has a new augmentation I came up with. Let’s try to find the best way to fit the constraints of the optimization problem.” You end up with this much more flexible, high-level, almost grad-student-like ability to just grind a performance metric.

This can squeeze out quite a lot of performance. On a fixed data set with a fixed time budget, you can improve perplexity by quite a lot on a classification problem like LLMs or Go. It is also fantastic now at basically executing any experiment. I have a Claude Skill that I wrote called Experiment where I give it a description of what I want it to plot. I just describe, “Here’s the x-axis I want, here’s the y-axis. Answer this question for me.” It’ll run off and do all the experiments, compile the plot, make a report, and suggest what might have caused it and so forth. That’s what works quite well today, and I think we can expect these abilities to get better in the future.

But it’s also useful to know what it’s not doing so well today. In the blog version of this tutorial, I have a plot of all the experiments I did grouped in a tree, where every node represents a failed, successful, or mixed experimental result. From there, it branches off into a child representing the follow-on experiment. Occasionally, I’ll rabbit-hole down a track like off-policy MCTS relabeling, do a few experiments, and then realize it’s probably not worth it. So then I’ll jump to a completely different track. I call these things rows.

What I find is that the current closed models the public can access today don’t seem to be that great at selecting what the next experiment should be in a given track. They don’t seem to be able to step back and do the lateral thinking of, “Wait a minute, this track doesn’t really make sense. Let’s go back to first principles and think about what the bottleneck might be, or what we are trying to achieve.” Often I had to catch infra bugs myself by prompting the right question to Claude to investigate what’s causing the discrepancy, and then it’ll answer the question.

With Mythos-class models or Mythos++ models coming online, maybe this just completely changes and these problems fall to improved scaling. But at the same time, I think there’s a rich opportunity to develop RL environments that might incentivize this kind of lateral thinking. One of the motivations for setting up this Go environment was that Go captures a lot of very interesting research problems, often overlapping with LLMs or robotics. Yet it’s very quick to verify. The outer loop is ultimately: does the agent do what I think it does? You can check the outcome of a Go game quite easily.

The inner loop involves all this research engineering around distributed systems, predicting whether an idea is going to work or not, and predicting the difference a particular modification to your training algorithm might make. I think there’s a rich library of subtasks and sub-environments that you can train an automated scientist to work on, with Go as a sort of outer verification loop. Once you acquire these skills, maybe you can apply them to other domains like biosciences or robotics.

Dwarkesh Patel

Or automating AI research.

Eric Jang

Or automating AI research.

Dwarkesh Patel

Which is the real crux, the scary/incredible thing of making AIs make future versions of AIs. You’re suggesting the outer loop here could just be your win rate against KataGo?

Eric Jang

That’s one of them. I think there are a lot of deeper questions that one could tackle. For example, let’s say you have an idea on how to improve a scaling-law compute multiplier. The outcome isn’t necessarily “achieve the best Go bot ever”. The outcome might just be, “Can I predict what the win rate of my Go bot will be?” Or, “Can I predict the scaling-law plots that emerge from my idea?” But then you can verify that you haven’t reward-hacked anything by using a very verifiable game like Go on the outer loop.

Dwarkesh Patel

I think there are a couple of interesting follow-on questions. There are questions on the inner loop and the outer loop. On the inner loop, there’s a question of how locally verifiable any modification you might make is. That is to say, would you know whether some idea you try out is actually an improvement or a degradation? Would you know if something isn’t working as a result of a bug, or is it the result of the idea itself being wrong?

Ilya was talking about how one of the things he thinks makes him a good researcher is that he has a strong belief in what the correct idea is. He’s able to persevere through bugs and know which things are bugs versus mistakes in the fundamental idea, based on his high-level belief that “this idea should work, so therefore there has to be a bug”, versus the other way around. Why don’t we start with that question? How locally verifiable are things which are good ideas?

Eric Jang

As in the case of the success story for deep learning, you can think about this as a decades-long idea that took a lot of faith to get it to work. This presents a very challenging long-horizon RL problem where every step of the way you have a committee telling you that this is a bad idea, and then ultimately you break through. How do you design RL environments that maybe give you some feedback earlier? I think this is a very tough open question that I don’t have an answer to.

Ultimately, to play a very strong Go bot, you probably did need to discover deep learning. Having a challenging game that cannot be cheated easily on the outer loop could be used as an outer-loop signal for something like discovering the principles of deep learning. Now, of course, to make it tractable—and this is where research taste really matters—you have to come up with ways to initialize your problem so that you don’t try to solve a very intractable problem. Maybe you can leverage LLMs as a universal grammar in the middle to give you some sort of local feedback.

The fact that LLMs are a universal grammar means that they can move at almost any level of the stack. They can think very locally, as well as step back and think in very broad steps. I think that’s where a lot of the lateral thinking ability of humans comes from: knowing when the track you’re pursuing or the objective you’re pursuing is not right, and you should be asking a different question.

Dwarkesh Patel

The other question is how stackable local improvements are in the attempt to get to a better result on the outer loop. I’ve heard rumors that at some AI labs, the thing that has gone wrong is that people will individually pursue good ideas, but those don’t end up stacking well, and so the training run fails because of some weird interaction between two seemingly good ideas. Having a single top-down vision of how things should work is very important. Having worked at different AI labs and also played around with parallel agents trying different ideas, what’s your sense of how parallelizable AI innovation is?

Eric Jang

Great question. I think the research taste for executing well on the Bitter Lesson is that you need to know how much the Bitter Lesson can buy you and how much is too much to ask for, at any given moment. Of course, in the fullness of time compute is the single most important determinant of how things work. It’s almost inevitable that as you scale up energy and compute and parameters, intelligence will just fall out of that. That’s super beautiful, super profound. No algorithmic detail really matters beyond that.

But in the present day, we don’t have infinite compute and parameters and arbitrarily good initialization, so we have to come up with heuristics that give us that. These heuristics are probably somewhat redundant. That’s probably why you see this effect where a lot of these compute multipliers don’t necessarily stack. They might have some correlated benefit. And then three years down the line, when the Nvidia GPUs have gotten even stronger, maybe they stack even less well.

Maybe at any given point in time, the benefit of any given compute multiplier is transitory, which is what I suspected with the KataGo paper. There were many algorithmic ideas applied, and then you can see that with modern Blackwell GPUs and Ada-class GPUs—which are much better than the V100-grade GPUs that that paper used—some of these algorithmic tricks to speed up convergence just don’t matter so much compared to something else. I think that’s a matter of taste in the present time.

Dwarkesh Patel

How about the outer loop? How verifiable for making AI smarter? With Go, you do have this outer loop of win rate against the best open-source model out there. Even there, as you were saying, there are other outer loops of whether you discovered a new phenomenon, which is actually very hard to…. If you didn’t know scaling laws were important… When were Chinchilla or Kaplan scaling laws released?

Eric Jang

2018.

Dwarkesh Patel

So if you’re back in 2015, there’s not an automated procedure one can easily imagine for knowing which paper is the scaling laws paper versus which is just another random plot. Even in the Go case, it’s a hard-to-verify outer loop, and the whole idea of an outer loop is to have some backstop on improvement, let alone for general AGI, where of course we have a bunch of these benchmarks.

But there’s a problem. We know the things we can measure, and we improve on the things we can measure. We care about this broader ability to do economically useful work, which is not super easy to measure, at least until you automate everything. There’s a question of how good the outer verification loop is for AI self-improvement, and does that matter?

Eric Jang

I’m going to give a non-rigorous argument, but one that I intuitively believe. DeepMind started with a focus on games. They used games as their outer loop, and then their researchers learned from the experience of solving games, and now they’re working on LLMs.

Presumably, there was some positive transfer from their time working on games and Atari and Go and StarCraft that now helps them make good LLMs. I assume that there’s positive transfer in some regard, whether it’s coding or general research ability or project management. All these things probably help them do well.

If that’s the case, why wouldn’t it also be true for automated AI researchers? They should be able to positively transfer experience tackling quick-to-verify, quick-to-iterate environments to something more ambitious and economically useful, like automating drug discovery.

Dwarkesh Patel

I don’t know. Isn’t the issue… Historically, until Gemini 3 or whatever a couple years ago, people were saying, “Look, Google isn’t catching up in LLMs because they’re too tied to the old approach.” There are gains, but there are also ways in which it actively hinders you. It’s not obvious to me.

Eric Jang

The jury’s still out. Let’s say currently Google’s doing quite well. Who knows if the initialization of training on games is ultimately going to hobble their ability to be the winner in the long term. It’s hard to say for sure.

Likewise, who knows if the seeming late start was really just them pre-training for longer on how to scale up TPUs. They invested all their tech tree in getting TPUs to be good, which seemed not that useful in the short term but in the long term it becomes… So it’s even hard for humans to reason about what the optimal research strategy should be, even with the data we have today.

Dwarkesh Patel

Okay, we should let people know how they can find out more about this project, whether to fork it themselves, or check out your blog post where you do an excellent job explaining many of these ideas. Where do people go next?

Eric Jang

My website is evjang.com. There’s a blog post that links to an interactive version of this tutorial. On my GitHub, which is the username ericjang, there’s an AutoGo repo that people can fork and reproduce the training results.

Dwarkesh Patel

I also highly recommend people check out this blog post, “As Rocks May Think.” We touched on some of the ideas in this conversation, but it’s this grander thesis of what happens when you have thinking as a primitive in—

Eric Jang

Computer science.

Dwarkesh Patel

Exactly. I highly recommend people check out that blog post as well.

Eric Jang

I encourage the audience to think about the relationship between thinking and Go via MCTS and search, and how it relates to LLMs. I think there’s something quite profound there, and probably underexplored just because Go has been relatively underexplored compared to the boom in LLMs.

It’s not to say that I think we should have trees in our LLMs, but there is some very interesting duality between them. You can actually do a lot of research on Go, MCTS, and reasoning with very small budgets. So that’s very exciting.

Dwarkesh Patel

Awesome, Eric. Thanks for doing this.

Eric Jang

Thank you. It’s an honor to be on the podcast.

David Reich is back.

He and collaborator Ali Akbari just published a paper that overturns a long-standing consensus about human evolution — that natural selection has been dormant in our species since the agricultural revolution.

By scaling ancient DNA sequencing and developing a new statistical method, they found that selection has actually sped up.

Selection went especially bonkers during the Bronze Age (around 3,000 years ago).

That’s when gene frequencies for everything from immune function to body fat to intelligence were most in flux.

Over the last 10,000 years, selection pushed the genetic predictor of cognitive performance up by roughly a full standard deviation — most of it between 4,000 and 2,000 years ago.

After we finished recording, David sketched out on a whiteboard his new heretical model about who the Neanderthals really were. Luckily, I took out my iPhone and managed to record it.

He thinks the standard story (that Neanderthals are some separate archaic lineage we interbred with a little) just doesn’t fit the evidence. Instead, he proposes that Neanderthals are essentially genetically-swamped modern humans.

A small population somewhere around the Caucasus invented Middle Stone Age technology roughly 300,000 years ago and expanded outward. The ones that moved into Europe interbred with local archaic humans, got genetically swamped, and became Neanderthals. The same expansion went into Africa, met much more diverged archaic Africans, and that mixture became us.

This means Neanderthals and modern humans share the same cultural ancestry — the only difference is which archaic humans they mixed with afterward.

David is a brilliant and rigorous scholar. It was a real delight to learn from him again.

Watch on YouTube; listen on Apple Podcasts or Spotify.

• Cursor was super useful as I prepped for this episode. Whenever I had a question, I’d have Cursor kick off a few different models simultaneously and then compare their responses. I found that this led to better results than I could get out of any individual LLM. If you’ve only used Cursor for coding, you should try using it for research. Check it out at cursor.com/dwarkesh

• Jane Street uses an internal currency called “hive bucks” to allocate compute through a real-time auction – and anyone can change anyone else’s bids or even kill their jobs! Everyone just trusts each other to act in the firm’s best interest, which is what lets the system work in the first place. If this weird and high-trust culture sounds like your kind of thing, Jane Street’s hiring at janestreet.com/dwarkesh

• Crusoe’s ML infra team built fastokens, an open-source tokenizer that delivers a ~9x speedup over Hugging Face and up to 40% faster time-to-first token – on real production workloads! Crusoe achieved these results by parallelizing things and using some clever engineering to handle duplicates without cross-thread coordination. Learn more at crusoe.ai/dwarkesh

(00:00:00) – Ancient DNA suggests strong selection over last 10,000 years

(00:15:45) – Natural selection intensified during the Bronze Age

(00:35:02) – Why didn’t evolution max out intelligence?

(00:57:21) – Evolution is limited by time, not population size

(01:09:02) – Why no farming before the Ice Age?

(01:17:13) – The Neanderthal puzzle David can’t stop thinking about

(01:54:10) – The methodology behind this breakthrough

Dwarkesh Patel

I am back with David Reich, who is a professor of ancient DNA at Harvard. How do you describe what it is that you study?

David Reich

I’m a geneticist, and I work on human history and how ancient people relate to each other and people living today.

Dwarkesh Patel

We did an interview two years ago, which ended up being one of the most popular interviews I’ve ever done. I think people found it really compelling that there’s so much about human history we don’t know and are just learning about now as a result of the kinds of techniques your lab is using.

You have a new preprint that’s very exciting, and I wanted to talk to you about it. Can you give me a little bit of context on what we’re talking about today?

David Reich

The dream was that when this ancient DNA field started, more than 16 or 17 years ago, we were going to learn a lot about biology —about how people’s biology changed over time— by getting DNA out of ancient human remains and tracking changes over time. And that dream has really not been realized since the beginning of this field.

The field has been a big success with regard to learning about human history. It’s resulted in surprising findings about human migrations —people not being descended from the people who lived in the same place hundreds or thousands or tens of thousands of years before— and mixture being common in human history, and sex-biased processes being common. And there have been things that were not expected from archaeology. The field’s been a big success from that perspective, but what’s not been successful is learning about biology and biological change.

One big reason has been that the sample sizes have been too small. When you have a single person’s DNA, it provides a tremendous amount of information about history. That’s because when you look at one person’s DNA, it’s not a single person. It’s many people. It’s your two parents, your four grandparents, your eight great-grandparents, 16 great-great-grandparents, and so on. Going back in time, thousands, tens of thousands, even hundreds of thousands of ancestors are contributing to people today.

When you look at the DNA of a single person’s genome or a Neanderthal genome, you have effectively tens of thousands of ancestors all represented in your data. And you can position that individual exquisitely with respect to other people from whom you have data. But when you are interested in how a particular genetic variant—that affects something like your skin pigmentation, or your ability to digest cow’s milk into adulthood, or a behavioral trait—changes over time, a single person gives you only one sample, or maybe two samples: the one in their mother and the one in their father.

To get a high-resolution picture of how the frequency changes over time, you need very big sample sizes, truly very large numbers of people. We just didn’t have that until the last few years. What motivates the study we’re talking about today, and the work that hopefully a number of groups will be doing in the coming years, is the fact that we now finally have those numbers. We can do something with the data to see how frequency changes over time.

Dwarkesh Patel

Can I ask a question? I’ll be asking a lot of naive questions through the next few hours, but why are frequency changes especially interesting?

David Reich

What we’re interested in is using the experiment of nature that’s occurred in our history, over the last tens of thousands of years, to understand what’s biologically significant in our DNA. If there has been a change in environment that a population has experienced—for example, people shifted to agriculture, began living close to domesticated animals, or moved from a cold place to a warm place, or a low place to a high place—then there’s pressure on the population to adapt to these new stresses and new needs.

The way you’re going to detect that is by seeing that the frequency of a genetic variant—that for example might allow you to live at higher altitude, or that might nudge you to have a different behavioral pattern advantageous in the new situation—pushes systematically in some direction in a way that is enough for you to detect. It’s very hard to detect slight shifts in frequency by a few percent or ten percent unless you have a very big sample size. What we’re looking for are those changes in frequency that are too extreme to be due to chance. That will tell us there have been pushes against the biology as a result of the changes in environment that people have experienced.

Dwarkesh Patel

Interesting. What did you guys find?

David Reich

Seven years ago, Ali Akbari, who at the time was a postdoctoral scientist in my laboratory and a few years later became a permanent staff scientist, set out to use the data we were producing to learn about biological change over time. I think the reason he was interested in our laboratory rather than other places was that a focus of our lab has been generating truly large amounts of data from ancient humans. We’ve been trying to industrialize the process, make it very inexpensive, make it high quality, and generate large numbers of samples with lots of good data for this purpose.

There’s been this large amount of data we’ve generated. And it made it possible to conceive again of asking whether there have been frequency changes over time. The mainstream view in human evolution in the last several decades has been that natural selection has been pretty quiescent over the last several hundred thousand years of human history.

There are several lines of evidence that have been deployed to document this. One is that if you compare diverse populations from different continents around the world, for example Europeans and East Asians, and you look at mutations that differ in frequency between these groups—all mutations differ a little bit in frequency, sometimes a lot—you can say, “What are the most different mutations in terms of frequency between Europeans and East Asians?” And there are almost no genetic changes that are 100% different in frequency between Europeans and East Asians.

Europeans and East Asians descend from a common ancestral population 40,000 or 50,000 years ago that came out of Africa and the Middle East. This population had a set of gene frequencies, and these variants bopped around randomly—a process known as genetic drift—or perhaps under selection in one direction or another. The time that’s passed since 40,000 or 50,000 years ago is sufficiently small on an evolutionary timescale that there’s just not much genetic differentiation on average between these two groups.

However, if there’s been natural selection, for example to help people in one place digest alcohol better, or digest milk better, what you might expect is that there would be some mutation that would have rocketed up to very high frequency. Forty or fifty thousand years is a lot of time, it’s maybe 1,500-2,000 generations. That might easily be enough time to see a 100% difference in frequency. Yet you don’t see any more than what you would expect by chance.

This combination of things made it seem that selection has just been quiescent. Maybe a few hundred thousand years ago, the ancestral human population got to some kind of optimum, and after that there hasn’t been much genetic change in one way or the other.

There have been small amounts of natural selection, or selection to remove bad mutations that are constantly raining down on the genome, but not what we call directional selection. That would be newly arising mutations, or mutations being pushed in a systematic direction, to help the population get to a different adaptive set point more favorable for the conditions that population is living in.

We were able to partition how much of the changes in frequencies of all the mutations that we’re seeing in the DNA—we’re looking at about 10 million positions that vary—is due to directional selection (adaptation) versus other factors, especially genetic drift. And 98% of it is other factors, especially genetic drift. It’s overwhelmingly migrations and population structure causing fluctuations in frequency.

As a result, it’s super hard to detect the signals of adaptive natural selection because they’re a tiny fraction of the total frequency change. The vast majority of it are these migrations and mixtures. Nevertheless, there’s so much natural selection, as our study has shown, that it’s actually been rampant in the genome.

Dwarkesh Patel

Can I ask a clarifying question here? Why are we discounting population admixture or replacement as selection? If you think about it at a group level, if one population replaces another population, isn’t that selection?

I remember from the last episode you were explaining how there have been huge changes in what kinds of people are in a specific area. One population came in and replaced the previous one, and then a new population came in and replaced that one. To the extent that the genetics are relevant to why that population replaced the other one, why should that not count towards what we understand to be selection over the last 10,000 years?

David Reich

It could count, and may count, and probably should count in some respects. But it could also be that this population replacement is due to some cultural phenomenon —technology held by one of these groups and not others. And maybe there are some genetic mutations that are contributing to this. Who knows? It’s possible.

But what you’re seeing is a whole-genome shift. What we’re looking to see is whether there’s one place in the DNA that is driving the change in a way that’s different from the rest of the genome. From a statistical point of view, what happens at these times of migration is there are just huge fluctuations in frequencies. These are extremely uninformative times for detecting natural selection. The best moments to detect natural selection are when migrations and population admixtures are not happening for a few hundred years. During these times, you can actually see the mutation slowly blowing in one direction as a result.

The way we think about the history of Europe and the Middle East for the purpose of this study is as an archipelago of little populations in space and time, each pretty isolated from each other. You have a little population in Britain isolated for a few hundred years, or a little population in Hungary isolated for a few hundred years, between big events of migration and mixture. In each of those little experiments of nature, we can ask: does this mutation slightly increase in frequency? Does that same mutation slightly increase in frequency? If all the arrows point in the same direction, we win. They’re telling us that natural selection is occurring.

For example, 4,500 years ago in Europe, almost all mutations went through huge frequency changes. That’s not because of natural selection. It’s because of the steppe migration from north of the Black and Caspian Sea. 40-80% of the DNA becomes Yamnaya from steppe pastoralists. Their frequencies of mutations were different not because of selection necessarily, but just because they had evolved in different places for thousands and tens of thousands of years. When you look at the descendant populations, there are huge changes in frequency. What you need to do is see if natural selection is explaining a shift more than you would expect by chance.

Dwarkesh Patel

So you found these locations that seem to be under selection. I have another clarifying question. You say you found 3,800 locations which you’re 50% confident have been under selection in the last 10,000 years.

David Reich

It’s 7,200 where we’re 50% confident. We’re getting about 7,200 positions in the DNA that have 50% confidence of being real. Only half of those are real—we don’t know which ones—so 3,600 of them are real.

Dwarkesh Patel

Does that also mean that outside of those 7,200, you’re confident the other locations in the genome are not under selection?

David Reich

No. If you look at the 25% probability cutoff, there will be tens of thousands, and there will be many real ones there too. In fact, multiple analyses we do suggest that the genome is vibrating with natural selection.

There are all sorts of weaker effects that would be picked up in even larger studies than we’ve done. In fact, almost every position in the DNA is correlated to another position that is being dragged in one way or the other by natural selection. Instead of being quiescent, natural selection is everywhere. Even though it’s only 2% of the frequency change, it’s tugging the positions in one direction or the other everywhere.

So we analyzed these positions that we had identified, the hundreds of positions we were super confident about. We looked to see whether they were randomly distributed in the DNA or whether they had patterns. We looked at maybe 100 or so traits where there had been genome-wide association studies for all sorts of different traits, associated with immunity or autoimmunity or behavior or metabolism, and other things.

For each of these we could ask: do the genetic variations that are known to affect these traits from genome-wide association studies have an unusual number of genetic selection signals? What we found is that there was a vast enrichment, by about four or five-fold, for immune traits. There was a super concentration of selected signals in immune traits. We also saw a strong enrichment for metabolic traits—things that might impact obesity or fat traits or Type 2 diabetes—and almost no detectable enrichment, as far as we could tell, for behavioral or psychiatric traits.

Dwarkesh Patel

Just to make sure I understand. This is not to say that behavioral or psychiatric or cognitive traits are not under selection. It’s just that the individual sites where such traits are controlled are not especially likely to be among the locations you’ve identified as under selection.

David Reich

That’s exactly right. It might seem from the results of that analysis that immune traits are highly selected and that there’s been no selection for behavior in the last 18,000 years in this part of the world. But that’s a wrong conclusion, and we have evidence that it’s a wrong conclusion. There’s clear evidence of selection also on behavioral traits.

The reason we think we see much weaker signals for behavioral traits is that behavioral traits, we know from medical studies, are underpinned by much larger numbers of genes than immune traits, which are underpinned by relatively small numbers of genes of strong effect. Behavioral traits are shaped genetically by a very large number of genes of weak effect, and we just don’t have the statistical power to detect these very weak signals.

When we do an analysis looking at our very strong signals of selection, that collection of very strong results is very effectively querying the immune traits, but is not very effectively querying the behavioral traits. It may still be the case, and I guess it is, that immune traits are the most selected category. But it is not at all the case—and we can prove it’s not the case—that behavioral traits are not selected.

Dwarkesh Patel

Interesting.

David Reich

We’ve been able to prove that there are two ways to reconcile the previous observations with our new observations. Remember, the previous observation is that natural selection seems to have been quiescent over a timescale of hundreds of thousands or many tens of thousands of years. Reason? That you don’t see 100% difference in frequency variance across Europeans and East Asians.

Now we’re seeing hundreds of positions that are rocketing up in frequency with selection rates of 1% or more in a lot of cases. A 1% or more selection rate will mean a rapid doubling over periods of dozens of generations. Over the 1,500 or 2,000 generations separating Europeans and East Asians, shouldn’t you see many genetic variants that are 100% different in frequency across populations?

We were able to show that this is explained by at least two factors. One is that in this part of the world—Europe and the Middle East—we are actually in a period of accelerated natural selection. One way to see this is to look at the enrichment pattern we’re observing, where immune traits are unusually associated with these selection signals. We could compare the last 5,000 years of our time period, what’s called the Bronze Age and further onward, to the previous 5,000 years. What we see is that this intensification of selection around immune traits, and similarly the intensification around metabolic traits, has accelerated over this time period.

It’s not like natural selection has been at the same rate over all places and times. It’s increasing over the time period we’re analyzing. Plausibly the whole time period has increased compared to previous periods. We’re in a period of intensified selection. That’s not implausible, because this is a population that went through a huge shock in terms of the way people live and the culture. Almost everyone we’re analyzing are farmers or food producers in one way or another. Farming was invented for the first time anywhere in the world in the Middle East 11,000 or 12,000 years ago. The people who invented farming exploded into Europe after 8,500 years ago, spread across the continent, and expanded rapidly.

In the Bronze Age, there was an intensification of how people lived, with much higher population densities. People were living more and more next to their animals and getting their diseases, and exchanging their diseases with the animals and with each other. This is a period of rapid change in how people are living, resulting in different biological needs of this population. It’s not surprising, perhaps, that in the context of these dramatic changes, the biology of the population might not be ideally adapted.

There might be what some people call an evolutionary mismatch, where you take a genetic variation that evolved in hunter-gatherers and put it into farmers or pastoralists, and it’s not exactly right. What you’re seeing is the DNA of this population, which descended from hunter-gatherers only 10,000 years ago, reacting to the shock of having been moved into an agricultural, Bronze Age, high-population-density, urban environment. A hypothesis is that what we’re seeing is the adaptation that occurs as a result.

Dwarkesh Patel

In the paper you have many examples of this intensification of selection around the Bronze Age. It might be helpful to go through some of these.

David Reich

One of the things we do in this work is look carefully at many of these positions in the DNA. We actually have an internet browser called the AGES browser, which Ali and a colleague of his—who’s a co-author of our paper—built. It allows you to query each of these 10 million positions and see the trajectories at each position and the evidence for selection.

One of the things we see is that, while for the most part the signals of natural selection we detect are consistent with constant natural selection over time, in a handful of them we’re able to see that there’s been a reversal or a radical change in natural selection. Very often that occurs in the period between 5,000 to 2,000 years ago, which is the Bronze Age and the Iron Age, a period of rapid population growth and rapid movement to intensive use of many technologies that were not used that way before.

An example of this is the TYK2 genetic variant that is a major risk factor for severe tuberculosis, which is the most important infectious disease killer in the world today. If you look at this major risk factor for tuberculosis, this variant rockets up in frequency from 8,000 or 6,000 years ago to maybe 9% or 10% in this part of the world. Then it rockets down in frequency in the last 3,000 years. In both cases, there’s very clear evidence of natural selection, in the first case to increase in frequency, and in the next case to decrease in frequency.

A possible reason is the spread of tuberculosis. It maybe becomes endemic in the population 2,000 or 3,000 years ago. That’s potentially consistent with pathogen sequence data and other lines of evidence. And maybe this variant was protecting against something before then, but then tuberculosis became significant after that point, and it was so bad that it pushed in the opposite direction. That’s speculative.

Dwarkesh Patel

The thing it was protecting against was probably another disease?

David Reich

Maybe.

Dwarkesh Patel

One of the big takeaways for me from the paper was just that something weird happened in the Bronze Age. As you said, across trait after trait, the selection intensifies during the Bronze Age.

This makes sense for some things. For example, why do we see lactase persistence, where adults can process milk, intensified during this period? This is the time when we start using cattle not just for the meat, but also for milk and wool and other secondary products. So it makes sense why lactase persistence would matter more.

But then there are other things that seem like they should have been relevant since the dawn of agriculture. I forget the exact name of the allele, but was it FADS1, which helps convert plant fatty acids into long-chain fatty acids that your body needs? That’s obviously relevant when you move from a diet of meat as a hunter-gatherer to a diet of cereals.

That is also one I think you found was under especially high selection 5,000 to 3,000 years ago. So what’s going on? Why is the Bronze Age so special across all these different traits that you’re observing?

David Reich

So this FADS1/2 variant is a vegetarian/meat-eating adaptation. Already in work prior to this, Ian Mathieson, who worked with me in 2015, identified this as a very strongly selected variant. It’s actually ancient. You see copies in archaic humans too.

One of the findings of our paper is the ABO blood system. You get your blood typed as A, B, and O. The B variant has increased up to 10% at the expense of A, but previous work has shown that A and B were both already present in the ancestor of humans and gibbons and other apes. Some of these mutations have been going back and forth and fluctuating over different time periods.

But we’re talking about changes in the Bronze Age. The TYK2 variant for tuberculosis risk, a multiple sclerosis risk variant, inflected and increased in frequency before the Bronze Age, and then 2,000 or 3,000 years ago reversed in that period. There are differences in Northern Europe where this process is super strong, very strong positive selection, very strong negative selection. And then in Southern Europe, only a little bit, and not even very strong negative selection.

For haemochromatosis, which is pathogenic iron buildup that causes problems in Europe, that too has reversed around this period. In some of the complex traits that maybe we’ll talk about later, these traits too have periods of intensification of natural selection. For example, depigmentation: Europeans have gotten lighter skin over the last 10,000 years. You can see it in our data. The period of strongest depigmentation is between about 4,000 to 2,000 years ago, and then after that it’s much less.

This seems to be a very impactful, eventful, important period where a lot of the processes we are seeing become very powerful. It’s surprising on first principles. You might think, before you walked into this genetic data, that the big change is going to be starting to grow plants and maybe farm animals. That happens in the Neolithic, beginning 10,000-12,000 years ago, and spreads into Europe after 8,500 years ago. But actually, the intensification happens 5,000 years ago, 4,000 years ago.

It’s really interesting. This observation of that being an inflection point tells us something about when humans, at least in this part of the world, were wrenched into a way of living that was so different from how their hunter-gatherer ancestors lived that the organism had to adapt very strongly. It may be that the degree of that wrenching process moving into the Bronze Age was qualitatively greater than the degree of the wrenching process that happened from the initial transition to growing plants. That’s surprising, because our cartoon picture is that the big transition is farming. But the biological readout is saying our genome is reacting much more strongly to these events that happened 5,000 years ago.

Dwarkesh Patel

You did some work with Bhatia and many other colleagues in 2014 where you were looking at 20,000 or 30,000 African American genomes today. You were saying, “Look, there’s 80% West African DNA and then 20% European DNA. Can we look at their genomes today and see that their allele frequencies are much different than we’d expect from this admixture?” Correct me if I’m wrong, but you found that they weren’t.

That is to say, over 200 or 300 years of extremely intense environmental change—going from chattel slavery to a completely new environment—there’s no effect of natural selection. So we see episodes like this where we don’t see natural selection, but then the Bronze Age apparently must have had an even stronger effect, where the change in environment is even stronger than what we see from Africans in Africa being migrated to the New World and living under slavery.

David Reich

That may be the case. It also may be the case that that period is just too short to see much effect. In the Bhatia et al. paper, where we looked at about 30,000 African Americans, we looked to see whether—instead of the average percentage of around 80% West African ancestry—there were some places in the DNA with significantly more than 80%, or significantly less than 80%. That’s what you would expect if there were natural selection for some genetic variant from Europeans or from Africans.

We didn’t see any place in the DNA that was significantly different from what you would expect by chance. One possible explanation is just that there’s only a handful of generations, maybe five, over which natural selection would operate. So if the selection was 2% a generation, you would still only see a 10% compounded effect, and there’s just not enough time to detect it.

But the Bronze Age is not 300 years, it’s 3,000 years. It’s the power of compound interest, and you have enough time to begin to see a strong effect. This really, really does seem to be a very impactful time in terms of human history, and you can see it in our complex traits. Look at pigmentation, for example, which is the strongest signal of selection for a complex trait in our data set. You look at genetic mutations that are known to affect pigmentation. You add up their effect across all of the DNA, there’s dozens or hundreds of them. You look to see when natural selection is strongest, and the time period is really 2,000 to 4,000 years ago.

For some of these other traits as well, you see again that the time period over which selection is strongest is 2,000 to 4,000 years ago. For example, if you look at genetic variants that affect measures of cognitive performance, such as performance on intelligence tests in white British people today. This is of course a very strange trait to measure in the past because there were no intelligence tests and there was no school. But it is a predictor today, and you can look at how it’s changed in the past.

We see very strong natural selection for this combination of genetic variants that predicts people’s performance on IQ tests and is also highly correlated to the predictor of the number of years of school or the household wealth of people. All crazy traits in the past because there was no wealth in the past, there was no school in the past. But if you look at the predictors today, there is a strong movement in a systematic direction, a large effect, about a standard deviation on the scale of modern variation.

We can do this trick of looking to see whether there are periods of time when this natural selection has occurred more intensely or less intensely. We drag a 2,000-year window through our data, and we repeat our whole analysis, not on 18,000 years, but just on a short 2,000-year window. We can measure the strength of selection in each of these 2,000-year windows. What you see when you look at intelligence is that this maxes out in the Bronze Age, between 5,000 and 2,000 years ago.

The impact in the last 2,000 years is almost nothing. There’s no evidence of natural selection at all. Your bias coming into this, my bias perhaps, might be that if there’s any signal of natural selection on this trait at all, that it would be unusually strong in the last 2,000 years. Maybe this is a time of industrialization. Maybe this is a time of greater need for this particular trait. But in fact, there’s no evidence of natural selection at all in the last 2,000 years. There’s very strong evidence between 2,000 and 4,000 years ago, where instead of a one standard deviation strength of selection, it’s a two standard deviation strength, averaged over this time period.

Dwarkesh Patel

The standard deviation here is how much the polygenic score for the trait itself moves?

David Reich

How much the polygenic score for the trait moves over a 10,000-year period within a population that is held constant in terms of its ancestry. What we’re actually doing is looking in our data set at a heterogeneous group of people. There’s Southern Europeans and Northern Europeans and hunter-gatherers and farmers. At different times in the past, those groups are more or less represented.

The whole strength of the methodology Ali Akbari developed is that it corrects for that changing ancestry over time. Really what’s being asked here is that we’ve divided up our whole data set into an archipelago of little populations in different places in space and time. We’re asking in each place in space and time: a little pocket of people in Britain from 4,000 years ago to 3,500 years ago, a little pocket of people in Hungary, a little pocket of people in Italy from 2,000 years ago to 1,500 years ago. In each of these places, where the ancestry is relatively similar without being too disrupted in that short period by migrations, we watch to see if the genetic changes blow in the same direction. We’re measuring the strength of selection at each point in time after correcting for the big population changes that have occurred.

Dwarkesh Patel

The effect here is huge then. One standard deviation above the median would be somebody in the 85th percentile. You’re saying the effect of selection has been so strong that comparing 10,000 years ago to now, the median has gone to the 85th percentile. That’s just a huge effect over the last 10,000 years on something like intelligence or the thing that predicts household income.

Especially given that this is only 2% of the change in allele frequencies, and the 98% is coming from migration… It’s stupendous to think about what the impact of migration is, if this alone is driving a standard deviation change in these kinds of qualities, at least among the kind of variation we see in the world today.

David Reich

One thing you can see in the data is that the migration impact is huge. For example, if you look at the trajectory for measures of cognitive performance—scores on intelligence tests in white British people today—but you look at the predictor of that in people in ancient times, the estimate for the hunter-gatherers of Europe is three standard deviations below the modern mean. So that’s hugely different.

Then you see a huge jump from them to the farmers, who are at the mean, at zero. That’s migration. What you’re seeing is that those two groups had different set points for those traits. And then the steppe pastoralists have a lower set value.

You see huge fluctuations in the predictor of this trait over time. That doesn’t prove selection. That’s just migration. But what our test is telling you is: in addition to those fluctuations due to migration, is there a consistent effect of natural selection blowing the trait in the same direction over all places and times? That’s what we’re detecting.

Dwarkesh Patel

There’s this theory called the collective intelligence hypothesis, which is the idea that selection for intelligence has actually been in the opposite direction. As society has developed, there’s been more specialization, and if there’s more specialization, each person only needs to understand a smaller and smaller part of the world. Therefore, the ancients were actually much smarter than us, and we’ve evolved down in intelligence.

Your results seem to point in the opposite direction. Although there hasn’t been selection in the last 2,000 years as society has gotten more complicated, at least when society began, there was more need for the kind of thing that predicts intelligence today.

The reason that’s surprising is, if you think about hunter-gatherers—reading your colleague Joseph Henrich’s book—the amount of information they needed to hold onto and assess, everything from how to process food, to how to build shelters, fire, et cetera, compared to my world, where I just need to know how to set up mics and ask questions… It seems like the demands on intelligence should have been way higher in the ancestral environment. So it’s very surprising that the beginnings of civilization increased the selection on intelligence.

David Reich

This is the power of data. I think if you asked Joe prior to this work what the hunter-gatherer selection would be and where their set point for this particular trait would have been… I think he probably wouldn’t have made a very strong prediction, but he would have said, “Maybe you would have expected it to have a high predicted value of this trait because these people were really having to do a lot of things and figure a lot of stuff out. Maybe once you have more complex societies, there would be more of a collective brain, and maybe there’d be selection against this trait.” In fact, it’s the opposite in some ways.

It’s the power of data. It’s not what you expect.  It’s actually the value of data to try to make sense of all these things. It’s very interesting. The genetic predictor of intelligence, there are lots of things that are confusing about it, so it’s worth talking about. Or the genetic predictor of years of schooling, which is highly correlated to it and is measured even better.

If you look at the genetic predictor of years of schooling, there’s another amazing study from 2017 from a group in Iceland that looked at this measure over the last hundred years in Iceland. It looked at older people and younger people born more recently. There’s an estimated 0.1 standard deviation decrease in the genetic predictor of intelligence in Iceland just within one century. It’s an absolutely huge effect over a short period. This is selection against years of schooling. If I said intelligence, I didn’t mean to. It’s selection against the genetic predictors of the number of years of school.

One possible interpretation of this—hand-wavy—is that what’s being measured here is not selection for years of schooling or for real intelligence, but for another trait altogether that’s correlated to both of them. For example, the predictor of the number of years of schooling is very strongly correlated to the age at which women have their first kid. If you control for that, all of the signal of years of schooling goes away. So maybe what you’re measuring is women’s decision about when to have children.

If you have children earlier, you don’t go to school as much. If you have children later, you go to school more. Maybe it’s some kind of measurement of delaying gratification or putting things off or planning. The same trait is correlated to body mass index, to obesity, and to walking pace. So is this really intelligence as we think about it, or is it something else that manifests itself differently at different times in the past?

Dwarkesh Patel

Obviously, a trait like years of schooling was not itself a meaningful thing in the past. The underlying things for it seem to have been under strong selection. Whatever in the genome predicts years of schooling seems to have been under strong selection. How should we think about this? What’s the actual thing that’s changing in the genome?

David Reich

There are two things going on that you need to think about. Years of schooling is connected to so many other things genetically. If you look at the genetic predictor of years of schooling—this trait has been measured in millions of people now—it’s correlated to really surprising things. It’s correlated to the age at which women have their first kid. It’s correlated to people’s obesity. It’s correlated to people’s walking pace. It’s correlated to people’s household wealth. It’s correlated to a variety of other traits that seem quite different from it.

If you think you’re actually measuring the genetic prediction of intelligence, or actual studiousness, you should think again because there are many things that it’s correlated to. There seems to be some kind of general trait that you could maybe think of as executive function or a propensity to defer gratification—I’m just waving my hands—that is under selection. It pushes all these traits in the same direction one way or the other, and at different times in the past, it’s advantageous or disadvantageous.

When we found this signal of the genetic propensity to go to school for more years as it manifests itself in white British people today, we were incredulous. How could this be? Maybe this is a problem. So we did a few tests to try to figure out whether this was real. One of the tests we did was that we looked for a study where this measurement of the number of years of school was done not in Europeans, but in Chinese people in China. We looked at the effect size of many variants as they affected the number of years of school in China, and we saw whether they had a correlation to the trajectory of those same genetic variants in Europeans over the last 10,000 years.

These are two parts of the world where the populations have been essentially completely disconnected. There’s no way by chance that the trajectory in Europeans over the last 10,000 years would have anything to do with the effect on years of schooling in China today. But there’s actually a huge statistical correlation, a five or six standard deviation correlation between the effect size of variants on the number of years of school in China today and the trajectory in Europe. It’s just as strong, actually, as the effect size of variants in Europeans on years of school to the trajectory in Europeans. We just could not see a way this could happen by chance. Once we saw that, we felt quite convinced that this was a real signal and that somehow there has been natural selection to increase the genetic changes that today manifest themselves as predicting more years of schooling.

Dwarkesh Patel

Just to make sure I understood, you’re looking at this ancient DNA in Europe. You’re saying it seems to predict years of schooling for modern people in Europe, or at least selection on that ancient DNA seems to predict more years of schooling in modern Europe. You also find that the same variants predict more years of schooling for Chinese people in China. So this is not just some weird artifact from the way these GWAS were done in Europe. These parts of the genome seem to robustly predict the kind of thing that actually leads to more years of schooling, at least in people today.

David Reich

Correct.

Dwarkesh Patel

Stepping back, I want to understand what this tells us about what actually changed in our environments over the last 18,000 years. We talked a little about what happened after the Bronze Age. We were talking about this during the collective intelligence part of the conversation. It’s surprising to me that things like intelligence, or lack of schizophrenia—things that just seem robustly good—were not maxed out before the Bronze Age.

The diversity among different populations was so big that you have the European hunter-gatherers having three standard deviations less predicted value for what they would score on an intelligence test if it existed. But they were existing in the real world in a place where intelligence matters.

How can it be that this was not a trait… You just look at the human body or any animal, and evolution has been acting on it so strongly to make it functional for the things it needs to do. And this one thing, which seems so relevant—especially to what human hunter-gatherers needed to do—doesn’t seem to have been under that strong selection in the Mesolithic or Paleolithic eras?

David Reich

I think that’s a great question. As we talked about before, selection is very effective. It can move the mean value of traits within hundreds or thousands of years in one direction or the other if that’s adaptive in a particular environment. So you might wonder, isn’t intelligence good in all contexts and places in time? There are a number of ways to think about that.

First of all, we are speaking from the point of view of a society which intensely values this particular trait, the ability to score well on IQ tests or things like them, or to go to school for a long time. I think it’s unprecedented in human history that we live in a time like this. If you look at the Hebrew and Christian Bible, and you look at how much intelligence is valued, it’s basically not at all.

Dwarkesh Patel

But when the Bible was being written, especially the Old Testament, that’s exactly when selection for intelligence is at the highest point it’s apparently ever been.

David Reich

Exactly. But there it’s about strength or courage or religiosity. Those are the values. If you read Homer or the texts of other religions, it’s not intelligence. It’s beauty and other things. This value system which has a hyper-focus on smarts is not obviously a trait value that’s been common in the past. You might think that in certain communities there might be valuation of things that are more proximate to years of schooling. But really broadly, it’s not been a high value in the population.

Dwarkesh Patel

Obviously, the thing we care about is not direct performance on an IQ test, especially in the past. The thing I’m trying to understand better is intelligence more broadly. Maybe IQ-test intelligence is just not that correlated with, “Here is a new-world environment, go figure out how to process food there and make shelter and everything else.”

Your colleagues like Joseph Henrich have talked about how modern people underestimate the difficulty of doing this kind of thing with a small band of people. Maybe that’s not IQ-test intelligence, and that’s why we don’t see that strong a selection effect on this thing. But intuitively, regardless of the value system, it just seems very valuable to have this trait maxed out.

David Reich

I’m being very speculative. Let me give you two examples of how I’m thinking about this, not that I’m a particularly good authority on these things. As I mentioned, a lot of these traits, which are quite disparate, are highly correlated to each other. Obesity, years of schooling, walking pace, performance on IQ tests, household wealth, all these crazy traits seem to be governed to a substantial extent by a shared combination of genetic variants.

Let’s think about what this might mean. In Iceland in the last hundred years, there’s been selection against this combination of variants. One possible interpretation is that it’s basically selection for two ways of investing in your children: having many kids and not investing a lot in them, or having few kids and investing more in them. If you invest in deferring having kids, having more wealth, having more resources, and putting more into each kid, you’re going to have lower fertility and fewer kids. That’s going to result in lower fertility, but those kids might survive more and do better in society. Alternatively, you can just have as many kids as you can and invest less in them. They might individually have less good outcomes, but in a time of plenty—which is potentially Iceland in the 20th century—it might make sense to have more kids and invest less in them.

There’s a toggle between having more kids and investing less in them, and having fewer kids and investing more in excelling in various ways. You can imagine that at different times and in different places… In ecology, there are different ways. Mammals often invest a lot with a pregnancy and a small number of children, whereas fish will spawn huge numbers of offspring into the river, the great majority of whom will be eaten. But that is an effective way to produce offspring in certain conditions. So there will be a toggle depending on the environmental conditions back and forth between investing in large numbers of offspring with less investment, or smaller numbers of offspring with more investment. Maybe we’re just seeing that move back and forth over different places and times.

Similarly, for schizophrenia and bipolar disease, how could this ever be advantageous? Maybe what we’re seeing with these diseases is a readout of some spectrum of traits that in some contexts might be advantageous. Maybe being anxious, imaginative, or neurotic might be helpful in a shamanistic tradition or a religious tradition which values people who can have visions or be creative. Maybe these are subclinical versions of schizophrenia or bipolar disease that in certain times may be advantageous and in other times may be disadvantageous. You might just be seeing selection for different types of creativity or other thinking that can be valuable in different contexts.

I’m waving my hands here, but my sense is that these complex traits have not pushed in one direction because there are advantages to both ends of the spectrum, and there are multidimensional impacts of these different traits.

Dwarkesh Patel

Julian Jaynes has this famous theory in The Origin of Consciousness in the Breakdown of the Bicameral Mind. I’m butchering this, but fundamentally, the way I understand it is that up until Homer, basically everybody was schizophrenic. People genuinely thought that gods were real people that you were communicating with. His claim is that ancient texts seem to show people behaving in this way.

David Reich

You’re being asked to believe in visions. Even today, there’s valuation in some religious communities in communicating with God, having visions, and having supernatural communions. So I just don’t know.

But I think it’s super interesting to ask the question of why certain traits are not always advantageous. For schizophrenia and bipolar disease, there is a sense in which most of the mutations are disadvantageous. We can see that from the patterns of variation, where the variants that are risk factors tend to be low frequency and they tend to be small effects.

Dwarkesh Patel

So another trait you find under selection is the trend away from body fat since the agricultural revolution. Why is that?

David Reich

What you see is a reduction in the combination of genetic mutations that make you at risk for obesity, body mass index, and similarly very correlated to it, higher fat mass, higher waist-to-hip ratio, and higher type 2 diabetes risk. There is clear selection, by about a standard deviation on the scale of modern variation for these traits, reducing over the last 10,000 years in this part of the world.

What can be going on there? Why wasn’t there selection for this combination of traits before? There’s a longstanding idea known as the thrifty gene hypothesis. The idea is that once you have hunter-gatherer populations that move into a farming environment where there’s plentiful food, there is no longer a need to the same extent to be able to build up body fat to survive in times of stress, because there are more constant stores of food.

As a result, there will be natural selection against body fat once you move into an agricultural environment and into periods of food plenty. Maybe what you’re seeing is that this group of people in Europe and the Middle East over the last 10,000 years has moved into a period of relatively more stable food, where building up stores of fat is not as advantageous, and there’s been selection against this combination of traits. Europeans are actually relatively better protected genetically against type 2 diabetes than some other populations around the world, like African Americans and Native Americans, that have perhaps not been exposed to agriculture for as much time. So you may be seeing the effect of more exposure to more stable food accessibility.

Dwarkesh Patel

This is also another way in which the data go against a common story. The common story is that hunter-gatherers actually had much more stable diets because they were more varied, and they weren’t reliant on a single cereal or crop for their calories. If one game went away, they had other things they could scout for. They could move locations more easily because they weren’t tied down to the land. So they were more food-stable. But if there’s been selection against storage of body fat, that suggests that as unstable and as common as famines might have been in agricultural societies, it’s at least more stable than what the hunter-gatherers had.

David Reich

There’s a timescale issue. You’re absolutely right. As I understand it, I’m no anthropologist, when there’s a hunt in traditional societies or communities that hunt, people will often gorge themselves, eat a huge amount, build up a temporary store of fat, and then go multiple days without eating meat until the next hunt. There is this boom-and-bust access to high-value nutrition that is not true to the same extent in farming communities.

On the flip side, famines are something that occurs more commonly in agricultural societies, but the timescale and the tempo of them is very different from the hunting tempo. Maybe there’s a famine every three years. Indeed, if you look at the bones of farmers, at least in some communities, there’s more stress in them, maybe due to a famine every three years or every five years. But selection might not be acting on that three-year time period. Your fat store from the latest hunt is not going to carry you through to the famine three years later. Survival of famines is a different thing than building up body fat to be able to survive two weeks later.

Dwarkesh Patel

A random question I have. You were mentioning that compared to these other things which matter much more for fitness in the ancestral environment—the immune system, especially after the Bronze Age—all these other things have mattered more than intelligence. They’ve been under much more selective pressure than intelligence.

That makes you wonder whether there’s much more room at the top for intelligence. If humans had been selected especially for intelligence, they could have been much smarter. The reason that’s relevant is that we’re currently building AI systems, which we’re trying to make as smart as possible. In fact, the only goal of the training process is intelligence. We don’t have to worry about at the same time making their immune systems powerful—

David Reich

We have lots of energy to spend on it.

Dwarkesh Patel

And at the same time making sure they’re not schizophrenic. I guess we kind of do worry about that. But if intelligence has not been the dominant trait under selection for humans over the last 10, 20, or 100,000 years, does that mean there’s more room at the top for this trait?

David Reich

I think there’s more room at the top for a lot of these traits. You can move height extremely in one direction, much more than it is today. You can move any of these traits much more extreme in the other direction. There are probably very strong negatives to doing that. You’re probably sacrificing other things, and there are trade-offs. But it’s highly likely that if natural selection pushed any of these traits more in one direction than it is, the mean would move.

Dwarkesh Patel

So all of this evolution since “Out of Africa” is acting on alleles that already existed in the pool of human variation from that first group we were talking about last time, on the order of 10,000 people, that exploded out of Africa. Is it surprising that across all these different traits, from cognitive profiles to disease resistance to height, that one pool of people contained so much latent variation that they could supply enough stretchiness to accommodate all of these different traits you’re studying now?

David Reich

That’s a rich question, and I think the human population has within it a tremendous amount of variation for complex traits. There’s a huge amount of variation that affects height. There’s a huge amount of variation that affects body mass index. If you take all these mutations and set them to the high-height variant, a person will be extremely tall, like as tall as a tall building. Of course, that will never happen.

But if you take all these variants that affect schizophrenia risk and you point them all in the same direction, there will be extreme risk or extreme protection for schizophrenia. For complex traits, ones underpinned by many mutations, all the variation already exists to move the population to a different adaptive set point that’s optimal in the environment it’s in.

If you push the population into a new environment, within hundreds or thousands of years, the population can rapidly move to a new adaptive set point. There are some unusual traits, like the ability to digest cow’s milk or protection against sickle cell anemia, that require a single very important mutation that may not yet exist in the population. You have to wait for the mutation to occur in some people. When the populations are relatively small, only 10,000 people, you might have to wait dozens or hundreds of generations for that mutation to arise. But when the populations are large, there’s no mutation limit anymore.

Every mutation that can occur does occur. There are eight billion people in the world. There are maybe 30 new mutations every generation, so that’s 240 billion new point mutations every generation. There are only three billion DNA bases in the genome, so every mutation that can occur does occur about 100 times every generation. We’re not mutation-limited anymore. The mutations can arise again. They do arise again. But when the population is only 10,000, you sometimes have to wait dozens or hundreds of generations for the new mutation to occur.

Dwarkesh Patel

How likely is it that the thing that changed with the Bronze Age is just that the human population was big enough? By 3000 BC, you go to a population of 50 million-ish people. The population is big enough, and the gene flow between different areas is high enough, such that things which don’t have an overwhelming selection coefficient, which aren’t overwhelmingly favored by evolution, are finally visible to selection.

David Reich

I think that’s not likely to be true, but it’s an extremely interesting thing to think about. Already when population sizes are on the order of a million or so, every mutation that can occur does occur within a few generations. That’s well before the Bronze Age if you take the population even of a place like Europe, but also of other places. Or maybe it’s at the dawn of the Bronze Age or the farming period. The question you’re asking is whether, when the population is small, maybe natural selection doesn’t work effectively.

A common thing people think about with natural selection, which is true, is that in small populations selection doesn’t work effectively. That’s because mutations bop around in frequency from generation to generation a lot in a small population, just randomly. If you have a population size of 1,000, mutations will bop around by a frequency of one over 1,000 every generation. If the selection coefficient is less than that, it will be drowned in the random bopping around of frequencies due to genetic drift. But that is already for a population of 1,000. A 0.1% selection coefficient is very weak. We’re talking about 1% effects, and that’s very strong. It will work very well even in a population of size 1,000 or 10,000.

If you are talking about mutations of the type that will start rising only in large populations but not small populations, those are selection coefficients on the scale of one over 10,000 or one over 100,000. Those will take 10,000 or 100,000 generations to rise in frequency, which is hundreds of thousands or millions of years. That’s not going to do anything over the timescale we’re talking about. There’s just a timescale issue. We’re talking about strong, measurable selection coefficients on the order of half a percent or more in this study. All of those are going to work in small populations or large populations. It’s not going to be affected by the population size.

Dwarkesh Patel

Interesting. You’re saying that more generally, once you hit a given threshold of population, the dominant factor is time span, not population size.

David Reich

Correct. It’s very interesting, and it’s actually not widely understood.

Dwarkesh Patel

Speaking of data contradicting what you might have otherwise assumed, one of the papers you sent me beforehand, Mallick 2016, found that there are no fixed differences between modern and archaic humans 50,000 years ago. We know this is the period in which the so-called cognitive revolution happened, and modernity started, and people are making art. Does this suggest that nothing biological changed to make modern humans modern? The thing that happened was some cultural change? How do we understand what this data tells us?

David Reich

Right. 100,000 to 50,000 years ago, there’s a quickening of the pace of change in culture. You see the first extensive representational art, bead necklaces, drawings on the wall, and a rapidly increasing pace of innovation in the types of tools that people use. The thought might be that there would have been some important genetic switch, a kind of important genetic change that occurred in the population and swept to high frequency that everybody soon had. That made it possible to do these things. Maybe some genes allowed people to have complex, representational language, for example.

One thing we did in 2016 in this paper by Swapan Mallick and colleagues was look across the DNA for places that might be expected to look like this, where nearly all people living today share a common ancestor maybe 100,000 or 200,000 years ago. We looked really hard, and right across all the DNA we could look at, we couldn’t find anything more recent than four or five hundred thousand years ago.

This is a crazy result because it looks like there are no key selective sweeps that have occurred in this period that are ancestral to everyone living today. We talked before about no selective sweeps between Europeans and East Asians, but there don’t even seem to be any selective sweeps shared between all humans in this really important period when a lot of evidence in the material culture record appears. It could be that there’s biological adaptation in this period, but it’s polygenic. There are lots of mutations that all shift in the same direction to help the population move to a new set point, but there’s no key biological change that rises to high frequency in this time.

Dwarkesh Patel

This group 50,000 years ago, are they the ancestors of everybody out of Africa or also some Africans?

David Reich

This is 100,000 to 50,000 years ago. This is the population that’s ancestral to West Africans, to most East Africans, to all non-Africans.

There are a couple of populations in Africa that have substantial ancestry coming from more divergent groups. For example, Khoisan from Southern Africa or Central African rainforest hunter-gatherers have substantial fractions of their ancestry from groups that diverged maybe 200,000 years ago from the other lineages. But all of these groups today are able to go to college and do everything everybody else does. There is no evidence that there is any key mutation lacking in some groups that is not present in the others.

Dwarkesh Patel

The differences we see between different groups of people, especially if this group 50,000 to 100,000 years ago had a very small population size… I think last time we were discussing on the order of 10,000 people. So almost everybody in the world, or the variance we see between different humans today, was latent in this group.

I get your point that if you just stack up different things across the genome, stacking them up really has a big effect. But it’s interesting that we have so many different groups in the world today, and all that diversity comes from a very small population size.

David Reich

A lot of us in human genetics think that our population contains within it the clay that’s needed to make almost any trait. And that depending on environmental conditions or selection conditions, the mean value of these traits will move in different directions. There’s an empirical question about how much selection there’s been in different human populations over time.

One of the things this new work we’re involved in is showing is that at least in the last 18,000 years in this part of the world, there has been significant movement, at least for a handful of important traits. We looked at more than 500 traits. About 100 complex traits showed significant movement in a systematic direction over this time period. It really does seem that there is a response to the environments people are living in that has occurred over this period, and that is potentially stronger than in previous periods.

Dwarkesh Patel

We were talking earlier about how there are no fixed differences between humans 30,000 years ago and humans today. So if there’s no genetic basis for the kind of thing that allowed humans to have more symbolic representation, have farming, et cetera—I think I asked you this question last time we talked, but especially with this context—why no farming before the Ice Age? Genetically we were there.

David Reich

That is such an interesting question. Genetically we’re there. The common ancestral population has all of the ingredients for farming 50,000 years ago. These people are distributed into different parts of the world: the Americas 15,000 years ago or whatever it is, New Guinea 40,000 years ago, East Asia, Europe, West Africa. No farming developed before 11,000 or 12,000 years ago. It only developed in the last 12,000 years, the period known as the Holocene, which is the end of the Ice Age.

If you talk to climate scientists and archaeologists—I keep asking people this question every time I meet someone who’s an expert in this—how can it be that farming develops in all these places? Are we really living in such an unusual time? People tell me, indeed, we’re living in a very unusual time on a scale of two million years. That is, 12,000 years ago we switched into this period of not just warmth, but climate stability.

It’s hard to believe that we’re living in such a special time. But if you look at data from the bottoms of ponds where you can measure the fluctuations of temperatures using isotopic signatures, apparently we’re in a period where it’s fluctuating a lot less year to year, 10 years to 10 years, and 100 years to 100 years. It’s a period of relative stability that we are miraculously living in.

When this period of relative stability happens, it follows that multiple groups independently turn to agriculture, even though they all have the same genetic complement that arose 50,000, 100,000, 200,000, 300,000 years ago. It’s a crazy observation that people just accept, but it’s unbelievable.

Dwarkesh Patel

Oh, so you increased the range there. You said 100,000, 200,000, 300,000 years ago. Based on the genetic differences between modern people and people from 300,000 years ago. Do you basically think they’re modern 300,000 years ago?

David Reich

I don’t know. This is actively what I’m thinking about all the time right now. There’s a big transformation in terms of the culture of humans 300,000 or 400,000 years ago: this invention of Levallois technology, the ability to make stone tools out of cores.

The Middle Stone Age Revolution, or the Middle Paleolithic Revolution depending on what you call it in Africa or Eurasia, is a new way of making stone tools that’s shared by Neanderthals and by modern humans, but is not shared in East or South Asia. It’s a big change, and it presumably involves a cognitive change in order to make this sort of technology.

Then there’s a further change to the Upper Paleolithic Later Stone Age, maybe 100,000 to 50,000 years ago, when there’s a second transition with a new type of tool making, but it’s not as revolutionary as the earlier one. So when the cognitive leap happens is unclear.

The diversification of the lineages leading to people living today, like Khoisan Southern Africans and rainforest hunter-gatherers, all occurs more on the timescale of 300,000 or 200,000 years. All of these people are capable of going to college and doing everything. So it’s not obvious that the cognitive toolkit, the behavioral toolkit, and the genetic abilities were not all in place 200,000 or 300,000 years ago, and that even Neanderthals had them. It’s not obvious that this was not the case.

I just don’t know. You distribute these people descended from this diversification that happened 200,000 or 300,000 years ago to different parts of the world, and then after 12,000 years ago, you start having agriculture popping up in different places. It’s an outstanding mystery of human history. I find it unbelievable that we live in a time period that climatologically is so unique on a scale of two million years, but my colleagues tell me it’s true.

Dwarkesh Patel

The climate thing seems surprising given there are so many different environments in which agriculture was independently developed. I understand that across environments the variance could have gone down. If it had only happened in one place at one time, I could have bought that explanation. But the fact that they’re making maize in the New World and they’ve got cereals in the Old World in very different environments makes it surprising.

David Reich

It’s very, very surprising. We accept it, but it’s a crazy observation that most normal people don’t realize.

The thing that basically everybody accepts is that the common ancestral population of almost everybody in the world, except for rainforest hunter-gatherers and Khoisan, is around 70,000 years ago. Everybody accepts that these people all have in place the cognitive, behavioral, and intellectual ingredients that are necessary for the farming revolution and building state societies. Because when these descendants get distributed to West Africa, East Africa, the Americas, Europe, South Asia, East Asia, New Guinea, and so on, their descendants all do this. They do it independently, semi-independently, or demonstrably completely independently in all these different parts of the world.

The cognitive resources for doing this must have all been in place, but it’s a very long fuse. It delays for 40,000 or 60,000 years in all these different places after the common ancestral population splits up, and then ignites into agriculture and all these other things after that point.

It’s a crazy claim. Then you could argue about whether the actual fuse is 300,000 years, from when Neanderthals separated and from when different lineages of extant modern humans separate, and that’s also plausible. It’s a crazy set of things that we’re being asked to believe.

Dwarkesh Patel

Is it possible that agriculture existed, but you didn’t have modern metallurgy or whatever it was that allowed populations to explode starting in 5000 BC with the Bronze Age? Population-wise, it doesn’t seem like much is happening from 10,000 BC to 5000 BC in the early Neolithic. Is it possible that they had farming but they didn’t have copper or tin, which you needed to go to the Middle East for, to develop a civilization that could make use of bronze at a large scale, and so they just disappeared from the historical record?

David Reich

I think we would see their archaeology. There are extraordinary developments in the Americas which are entirely Stone Age.

Dwarkesh Patel

You would see them today if they had completely vanished?

David Reich

Oh, yeah. We should go for a trip to Teotihuacán in Mexico. It’s so impressive. When I went there when I was 20, it was totally as impressive as ancient Egypt. It’s huge. It’s massive. It’s without metal.

Dwarkesh Patel

It’s even more impressive because it’s not only without metal, but without animals and without wheels, which is crazy. The marble is just hauled without wheels.

David Reich

Right. Take any person who has an old world superiority and take them to these places, and they will not have it anymore. It’s just extraordinary what’s in these places. These are people who separated 20,000 years ago at least from the ancestors of East Asians and 40,000 years ago from the ancestors of West Eurasians.

They just had the same biological and cultural shared toolkit from then, but there’s a long fuse delay until all this stuff happens. It’s an amazing thing, and we don’t question it.

Dwarkesh Patel

What are other questions you are either investigating right now or want to investigate, these kinds of big picture questions of human history?

David Reich

I’m perplexed. I don’t know if we talked about it before, but I remain very confused about the relationships between archaic and modern humans.

We have genome sequences now from archaic humans who lived in Europe, West Eurasia, and Central Eurasia, and the Neanderthals. We have archaic sequences from these enigmatic Denisovans, who we now have a skeleton for since we last talked. There’s now a skull that’s been shown to be a Denisovan. We have data from lots of modern humans, and there are really big mysteries about the relationships amongst these groups.

Genetically, the Denisovans and the Neanderthals are sisters. They descend from a common ancestral population 500,000 or 600,000 years ago. That group descends 700,000 or 800,000 years ago from the common ancestors of modern humans. Genetically, the whole genome data says that Neanderthals and Denisovans are archaic humans from a common ancestral archaic population.

But there are so many things shared between Neanderthals and modern humans that don’t seem to be shared with East Asians. They both share Middle Stone Age stone tools, Levallois technology, this cognitively unique way of making stone tools that wasn’t used in East Asia. They both have the same mitochondrial DNA and Y chromosome sequence.

The Y chromosome sequence of Neanderthals and the mitochondrial DNA of Neanderthals, is actually modern human that came through interbreeding 200,000 or 300,000 years ago and then shot up to 100% frequency. Neanderthals and modern humans are both the product of mixture events that happened between archaic and modern humans 300,000 or 200,000 years ago, demonstrably through patterns of variation in ancient and modern DNA.

It feels that there’s something shared between Neanderthals and modern humans that’s not shared with Denisovans, even though the vote of the whole genome says that Denisovans and Neanderthals are related. One wonders whether there’s something connecting Neanderthals and modern humans that’s different from Denisovans, even though genome-wide, Denisovans and Neanderthals cluster. I’m thinking about that all the time now.

Dwarkesh Patel

Connecting them would be interbreeding events or being in the same place at the same time that we missed?

David Reich

There’s a known interbreeding event from the lineage leading to modern humans into Neanderthals, but it’s supposed to be only 5%.

I’m interested in the possibility that that 5% is actually a sign of something much more impactful, that somehow Neanderthals are in some sense deeply modern in some ways, and even though they get swamped by archaic genes, they actually have more of a modern impact than one would think. The Middle Stone Age and Middle Paleolithic Revolution that they share with modern humans is more fundamentally a part of who they are, in some sense, than we think.

Dwarkesh Patel

Interesting. Sorry, when was this interbreeding event?

David Reich

300,000 to 200,000 years ago.

Dwarkesh Patel

So the common ancestor between Neanderthals and most humans alive today is potentially more recent than the common ancestor between all humans alive today.

David Reich

Oh, for sure.

Dwarkesh Patel

Which is crazy.

David Reich

Well, the divergence to all the archaic humans, including Denisovans, is within human variation.

Dwarkesh Patel

Wait, what?

David Reich

Yes. The average time to the common ancestor of any two human genes is one or two million years ago. If you look at the copy of chromosome 3 you get from your mother and the copy of chromosome 3 you get from your father, the typical time they share a common ancestor is one or two million years ago. That’s before the split from Neanderthals and Denisovans. So there are many places in your DNA where you’re more closely related to a Neanderthal on your mother’s side than you are to your father.

Dwarkesh Patel

I’m sure there’s a simple explanation, but how?

David Reich

It’s the same reason that if you have a sister, in some places in your DNA you’re more closely related to her than you are to me because you share a parent. But in other places you’re more closely related to me than you are to your sister because you happen not to share the same DNA from your parents.

It’s just that the DNA we get from our common ancestral population was already quite variable 500,000 years ago, 700,000 years ago, a million years ago, and some of us descend from some of those ancestors and others descend from other of those ancestors. Neanderthals split from our lineage really close in time on human evolutionary timescales, such that in some places in our DNA we’re more closely related to Neanderthals than to each other.

Dwarkesh Patel

Interesting. What are the other big questions?

David Reich

That’s the main thing I’m thinking about a lot these days. I continue to be very obsessed with questions about the spread of human populations around the world and trying to reconstruct that with ancient DNA.

The thing I’ve been thinking about a lot recently is the possibility that maybe we’re not thinking in the right way about the relationship between archaic and modern humans. The standard model is one where Denisovans—these archaic humans that were found from ancient DNA—and Neanderthals descend from a common ancestral population 500,000 or 600,000 years ago, and these two separate earlier, maybe 700,000 to 800,000 years ago, from the ancestors of modern humans, people like us. That’s the big result of a lot of studies since 2010.

But there’s also evidence of an interbreeding event that happened maybe 200,000 to 300,000 years ago that resulted in modern humans contributing DNA to the ancestors of Neanderthals. So maybe 5% of the DNA of Neanderthals comes from this interbreeding event, and a lot of studies have shown this. I’m very interested in this because, from the archaeological record, Neanderthals and modern humans look quite similar to each other, much more similar to each other than a lot of them do to Denisovans, these archaic humans in East Asia.

For a lot of history, people have thought that Neanderthals are our sisters. But in 2010, the sequencing of the Denisovan genome made it very clear that on average, Denisovans are closer to Neanderthals than to modern humans. This was a very confusing result. Most people now think that Neanderthals and Denisovans descend from a common ancestral population that separated earlier from the ancestors of modern humans.

I’m interested in the possibility that the right way to think about Neanderthals is actually as somehow culturally modern humans, even though genetically they’re mostly Denisovans. The model I’m thinking about is motivated by this archaeological phenomenon known as the Middle Stone Age Revolution. If this is Africa and this is Europe, we know that the new way of making stone tools—with cores that were very carefully mined far away from the locations they were used, made out of high-quality stone like flint—starts being used 300,000 or 400,000 years ago, first in the Caucasus, places like Georgia today, or East Africa.

This way of making stone tools is quite revolutionary. It is known in Europe as the Middle Paleolithic and in Africa as the Middle Stone Age, and is associated with much more widespread use of fire and moving stone around at much further distances than before.

I’m interested in the idea that this is something shared between modern humans and Neanderthals. There’s somehow some shared cultural feature that’s absent in East Asia, and that might have a relationship in the genetic data and is somehow related to this 5% DNA.

The idea I’m interested in is the possibility that there is a population here that invents the Middle Stone Age and the Middle Paleolithic, sometimes called Levallois technology, and that people from this population expand into Europe and mix with the local archaic humans who are there. That is what this 5% interbreeding event is. It happens 200,000 to 300,000 years ago. It produces a group that, as it expands across this landscape in Europe, mostly picks up the local DNA and becomes mostly archaic genetically, but retains its modern human culture, the way of making stone tools and some of its traditions.

One of the things that’s super interesting about this is that if you actually look at the genetics, across the whole genome, Neanderthals and Denisovans cluster. But if you look at the mitochondrial DNA—which humans and Neanderthals get from their moms—Neanderthals and modern humans cluster. If you look at the mitochondrial DNA, Denisovans and modern humans share an ancestor well more than 700,000 or 800,000 years ago, as you’d expect from the history. If you look at the Y chromosome that you get from your dad, Denisovans and modern humans share an ancestor more than 700,000 or 800,000 years ago, which is consistent with this history.

But if you look at the Neanderthal mitochondrial DNA, it’s only 300,000 to 450,000 years. If you look at the Y chromosome, it’s only 300,000 to 450,000 years. What the current genetic work is asking us to believe is that even though this is only 5% of the whole genome, it introduces mitochondrial DNA and Y chromosomes, and they jump up to 100% frequency. It’s kind of a crazy claim because the probability of this occurring by chance is low, maybe 5% times 5%, a very small number. It’s what we actually all believe, but it’s a very surprising event. Somehow it’s accreted to all the findings in the literature so that we make ourselves believe this, but it seems unlikely on first principles that somehow only 5% would introduce both the Y chromosome and mitochondrial DNA.

And it really does look like this. There’s amazing data from a site in Spain that’s 300,000 to 400,000 years old, called Sima de los Huesos. They have a nuclear genome that looks Neanderthal-like for most of the genome, but their mitochondrial DNA and Y chromosome are Denisovan-like. So it really looks like there was a population related to modern humans that pushed into this Sima de los Huesos-like population, displaced its mitochondrial DNA and Y chromosome, but kept the rest of its genome. It really looks like something like this happened.

The idea I’m playing with—and probably it’s wrong, who knows—is that there’s a landscape… This is Europe and you can break up into a hundred or so demes, little areas. Modern humans get introduced at the bottom right corner, in the Middle East or somewhere, and they spread into Europe. As this population spreads, there’s a wave front of expansion, and they’re interacting with the local archaic humans.

The theory from simulations and studies of different species like mammals and birds shows that even if there’s a small amount of interbreeding—when there’s an invasion or a movement of expansion of one group into the territory occupied by another—there’s massive introgression of local genes. The pioneers at the wave front will sometimes interbreed with the local population. There are so many of them around that their DNA will get swamped by the local group, so by the time they make it to the other side, they’re largely local.

Maybe this is what we’re seeing. You have a modern human population that’s matrilineal, for example, where the transmission of making stone tools this way is happening from mother to child. That’s why they’re retaining their mitochondrial DNA, but by the time they get to the other end of Europe, they’re mostly local archaic. You end up with a 95% population replacement. This would explain why the mitochondrial DNA is shared between Neanderthals and modern humans, and it would also explain why the mixture proportion is only 5%.

The really interesting thing is that there’s other evidence from studies of modern humans showing that modern humans are also admixed. The right way to think about this is that modern humans are a mixture of two groups that diverged maybe 1.5 million years ago, and they come together 200,000 to 300,000 years ago with maybe 20% ancestry from this archaic African group and 80% ancestry from this early modern lineage. And that same group then mixes with Neanderthals, and it’s 5% modern and 95% local.

So you actually have this key population that makes the Middle Stone Age or Levallois technology. It appears and expands in all directions—into Europe and into Africa 200,000 to 300,000 years ago—bringing this technology, new ideas, and perhaps some genetic adaptations. It expands into archaic humans in Europe, mixes with the local population, and gets 95% replaced but still retains its cultural features and maybe some genetic features.

It expands in Africa too, but here it’s not 95% replaced, it’s only 20% replaced. Probably the reason is that this group is much more diverged. It’s 1.5 million years diverged rather than 700,000 to 800,000 years. As a result, there are many more genetic incompatibilities and barriers to gene flow. But there’s still a lot of mixing, maybe 20%, and we have evidence that this is a big mixture event.

So what you’re actually seeing is a modern human expansion both into Europe and into Africa. In one place, it forms Neanderthals. In another, it forms the ancestors of everybody living today. But all of these groups are descended from this key revolutionary event that happens here.

We often talk about the revolutionary events 50 to 100,000 years ago, the more symbolic behavior and so on, that first appears in Africa and the Middle East and spreads beyond. But there’s also this earlier event, and it is contemporaneous with the breakup of all the different groups in Africa today, the Khoisan Southern Africans and the Central African rainforest hunter-gatherers. One wonders whether this is an equally important formative event.

If that’s true, it makes you think of Neanderthals as somehow our cousins. Their shared Y chromosome, their shared mitochondrial DNA, they share the formation of this 200,000 or 300,000-year-old event, and their shared toolkit. Even though the genome is telling us they’re cousins of Denisovans, the correct way to think about them may, in an important sense, be as close cousins of modern humans.

Dwarkesh Patel

I have so many questions. Do you have 15 more minutes? First of all, what is going on with this group of archaic Africans 1.5 million years ago? Where in Africa are they, and what happens to the portion of them that don’t form modern humans? Do they survive?

David Reich

This is not from ancient DNA, but from analysis of modern DNA from different people, mostly in Africa, but also non-Africans. In multiple studies—there’s at least three, maybe four or five studies that I know about—they have looked at the patterns of variation in people today and say the data in modern people today, including in Africans, is not consistent with a homogeneous population.

It looks like a population that split well more than a million years ago into multiple groups—at least two, but maybe many—and then came together a few hundred thousand years ago. The papers have different models that they fit, but they all have this feature of a split-up more than a million years ago, and then on the order of a few hundred thousand years ago, a coming together and a remixture event forming the ancestors of anatomically modern humans.

Dwarkesh Patel

This includes the Khoisan and whatever other groups?

David Reich

Yes. All of these groups have this, maybe in slightly different proportions. So you ask, where are these people living? Who knows. In this scenario, the 80% is coming from the Caucasus or Northeast Africa, where the Middle Stone Age forms. It’s from this population that the Middle Stone Age comes. They mix with local groups, and who knows where they are: Southern Africa, Western Africa, Central Africa, Eastern Africa.

We don’t have any ancient DNA, but this is a very rich environment. People have been living there for seven million years at least, and there would have been different groups of people everywhere. Probably it’s not just two groups, it’s probably more.

The important theme here is there’s evidence of substructure that’s well more than a million years old. This place would have been a landscape full of archaic humans that would have been differently related to these expanding people and would have admixed with them when they came through.

Dwarkesh Patel

So with the Neanderthals, the first time around 300,000 years ago, our ancestors share culture with them. They share the Middle Stone Age technology, but they don’t replace the population. The technology spreads through culture, basically.

David Reich

It spreads through genes too. If you look at Yamnaya in India, there’s almost no Yamnaya ancestry in India. It’s just diluted down. As Yamnaya expanded into Central Asia and into Europe, it makes the Corded Ware. There’s a 25% dilution. It expands back across Central Asia. It goes through the Hindu Kush and gets into northern South Asia. It admixes more with local people. Today, the most Yamnaya ancestry you see in India is 20% or 10%. Most people have less than 10% or 5%.

There’s been a lot of mixture on the way, but it is the tracer dye. It tracks Indo-European languages, and important aspects of Indo-European culture are coming through Yamnaya. So if you know where to look, that tracer dye is only 10%, 5%, or 2% in some groups. But it’s the languages people speak, and important shared cultural elements, that connect them to people on the other side of the Indo-European-speaking world. So this 5%, you shouldn’t sneeze at it. That’s tracing something important in this model.

Dwarkesh Patel

I understand that if things are transmitted more through women… Sorry, let me back up. I don’t understand why the maternal mitochondrial DNA and the Y chromosome would be especially privileged as the spreading is happening. Can you explain that?

David Reich

The reason I’m talking about these matrilineal or patrilineal expansions is that I’m really troubled, and have been troubled for many years—especially in the last three or four years—by the fact that the mitochondrial DNA and Y chromosome cluster Neanderthals and modern humans, but the rest of the genome clusters Neanderthals and Denisovans.

This is a crazy result that is not seen in any other species. I’m very interested in patterns that would explain it. If you assume that there was a matrilineal or a patrilineal expansion—it could be either—then modern humans, when they were expanding across the landscape of Europe, retained their identity along one of the lines.

If it’s matrilineal, when they incorporate a male from the local community, he’s brought into the community, and the kids are raised based on the culture of the mothers. If it’s a patrilineal expansion and they incorporate a female from the community, she’s raised with the culture of the fathers. If that happens, it guarantees that one of these two parts of the genome looks like it does, because it’s a modern human expansion. If it’s patrilineal, it will retain the Y chromosome. If it’s matrilineal, it will retain the mitochondrial DNA. So it will solve one of your two problems.

Dwarkesh Patel

But not both.

David Reich

It won’t solve the other one, so you need to solve the other one. You can solve it either by natural selection or by social selection. By the way, patrilineality and matrilineality are the rule, not the exception, in human communities. Usually communities have continuity along the male or the female line. Usually it’s patrilineality, sometimes it’s matrilineality.

You can also have phenomena like social selection. It could be that once you have kids of someone whose father, for example, is from the outside community… Usually in most communities, females all reproduce. That’s typical today. Usually women have kids if they can. But men in traditional societies are actually very variable in their reproductive success. A large fraction of men never have kids. Then there’s a subset of men who have many kids with many women.

There’s competition among men for kids. In this context, where males are competing for access to females, female mate choice begins to be an important process. You have a phenomenon where if your dad is an archaic male, it could be the case that you’re not going to be as successful in the competition for local females as if your dad is a non-archaic male.

Some simple social phenomenon like that could explain the data, and we actually see this in human society. For example, if I remember right, in Central African rainforest hunter-gatherers, there’s different treatment of boys and girls depending on whether their dad or mom is one group or the other.

Dwarkesh Patel

I guess I don’t understand how the maternal… The group spreads, and it gets to the next front. They have kids. From the humans that have just entered, the kids will have the mitochondrial DNA from the humans. But from the existing people, they will have the mitochondrial DNA of the archaic humans. Why are the people with the archaic mitochondrial DNA not surviving?

David Reich

It’s a question. There are multiple possible explanations, but it’s much easier to explain that than both the mitochondrial DNA and the Y chromosome. One possibility is that the mitochondrial DNA was less biologically fit. Another possibility is that there’s social discrimination against people based on whether their parents are archaic or not, which is not at all surprising in a human context. It’s the weakest link in this argument. This argument’s probably wrong, but I’m just telling you what I’m thinking about.

Dwarkesh Patel

Okay, the Neanderthals. So 300,000 years ago our lineage interacts with them, but mostly their lineage survives, and there’s cultural and genetic diffusion. And then is it 70,000 years ago that we interact again?

David Reich

Yes.

Dwarkesh Patel

And they don’t survive.

David Reich

The genetic ancestry doesn’t survive.

Dwarkesh Patel

The genetic ancestry doesn’t survive. Presumably there was also other contact between 300,000 years ago and 70,000 years ago.

David Reich

Probably. But these are the ones we are detecting currently.

Dwarkesh Patel

Is it just contingent that one time there’s this kind of diffusion where most of the archaic genome survives, and the other time it’s total replacement?

David Reich

This is not at all surprising given the context. If you think about this model, this is 700,000 or 800,000 years ago. This is 300,000 years ago. So this is 400,000 years separated. You talked about the Bhatia paper with me earlier. That’s two populations 70,000 years separated. There are no biological incompatibilities between West Africans and Europeans. There’s no natural selection against biological incompatibilities.

We know when Neanderthals and modern humans met and mixed, there were biological incompatibilities. That was 700,000 years ago. As populations become further apart, biological incompatibilities rapidly develop, probably as the square of the separation distance, because you need pairs of interacting genes. Here, it would have been maybe only 400,000 years separated between this lineage and that lineage. But here, it’s 1.2 million years. That’s a lot. These are at the edge of not being able to produce children. These are quite different humans. These are actually three times closer than these.

If you look at mixtures of humans today, there are mixtures in Southern Africa of people who are half this distance. If you look at Khoisan and Bantu people mixing in Southern Africa, like the Xhosa, which is the population of Nelson Mandela, these are groups that are separated by almost 200,000 years, which is half of this. Totally compatible.

What you’re seeing is a group that’s actually completely permeable genetically, or nearly completely permeable. This other one almost certainly has substantial biological incompatibilities. Because 200,000 or 300,000 years later, we see interbreeding between Neanderthals and modern humans, or between Denisovans and modern humans, and there’s clear evidence of incompatibility at that point. But this would be even bigger.

What you would expect to see is that as this group spread, they would be moving into a territory full of archaic humans. There would be some interbreeding, but the kids would not be very fit. They would die off. There would be a lot of infertility. The barriers to gene flow and to interbreeding would be greater.

To me, it’s not at all surprising that as this group moves into Eurasia, you have Eurasian archaics—the ancestors of Denisovans—who are only 400,000 years diverged from these people over here. And then you have African archaics, and these are 1.2 million years diverged. They just don’t interbreed as much, and you don’t get as much gene flow.

But the key thing is the timing. It’s the same time. It really feels like the signature of an explosion of people from one place, interacting with people here and interacting with people there. It’s the same cultural or technological revolution impacting this place and that place, and creating populations that are impacted by this cultural revolution, which we know is the case because they share the same toolkit.

Some people argue that Levallois technology is independently invented. But it’s very similar, and this model would be a way that it could have the same origin. So there’s a culturally shared thread, this shared toolkit. There’s a mitochondrial DNA and Y chromosome thread. And then there is a shared timing thread, which is they both form by mixture.

Dwarkesh Patel

Because otherwise you’d have to believe that Neanderthals independently developed Stone Age tools.

David Reich

Yes, which is not inconceivable. But it’s a little bit like believing that farming independently developed in multiple parts of the world.

Dwarkesh Patel

Right. But it did.

David Reich

It did. So as I said, this is probably wrong. I’m trying to tell you that we don’t really know the world we live in. This is not obviously wrong. In fact, to me, this is much more plausible than the model we currently write down. It’s probably wrong, but it’s much more plausible. It explains many more things, and it’s no more complicated.

Dwarkesh Patel

Do you want to recapitulate the thing you were saying about the analogy to Ptolemy and the epicycles? I thought that was quite interesting.

David Reich

I think the model that we’ve put together collectively about the relationships between archaic and modern humans has accreted over time. There was this idea that modern humans are distinct and that Neanderthals and Denisovans are sisters of each other. Over time, we detected additional mixture events, like this modern human into Neanderthal, and then these other ones I didn’t even talk about, like a super-divergent lineage going into Denisovans and all this other stuff. We still say, “Oh, the whole genome says Neanderthals and Denisovans are sisters, so that’s the truth.”

We’ve patched it all together and gotten it all to work. You look at the mitochondrial DNA and the Y chromosome, and they have this odd pattern, and it’s improbable, but we can get that to work if we invoke natural selection, things like this. You patch it all together.

It reminds one of what happened in the ancient world, where there was this idea that the sun revolves around the Earth, but it doesn’t quite explain the movements of the planets properly. In order to get the movements of the planets to work right, Ptolemy and the astronomers made up these epicycles, these special extra rotations and movements to make everything work about right. It was such a convoluted model. When Copernicus and colleagues suggested instead that everything is revolving around the sun, it simplified things ever so much.

What was happening is that as astronomical information accumulated, it kept being contradictory to the standard model, but it could be made to work by proposing another complication and another complication and another complication. This is not as fantastic as proposing that everything revolves around the sun rather than the Earth, but it is much simpler. And it actually explains many things.

Dwarkesh Patel

What is counterintuitive or unexpected or hard to accept about this alternative model? What is the hesitation that people have for adopting this?

David Reich

I don’t know. Nobody’s thinking about this model right now. It just seems obviously a very natural model to me.

Dwarkesh Patel

The reason I ask is that Aristarchus, the ancient Greek, had the heliocentric theory because he had deduced how far the Earth is from the sun and noticed other things. But it was not adopted, because his fellow Athenians were like, “Look, if we believe that the Earth revolves around the sun, for it to be the case that we don’t see relative movement of the stars to the Earth, the only possible explanation is that the stars are so far away that it is just incomprehensible and implausible.” So the heliocentric theory was dismissed.

What I’m trying to ask is, what is the equivalent here of “for this to work, the stars have to be so far away that it’s inconceivable,” where actually the stars are so far away? Maybe we should adopt the implausible implication that this theory gives us.

David Reich

That’s a great question. I think we have to assume that there’s a linkage between the cultural transformations in Africa and Eurasia at this time, and that’s not something the community has really put together with the genetic data.

There’s this thread in the genetics about substructure in Africans, and then there’s this whole world based on ancient DNA, and they’ve never been put together. Nobody’s put together the now extensive work on modern human substructure with the extensive work based on ancient DNA of archaic human relationships to modern humans.

If you put them together, you realize they line up in terms of their time of substructuring. I don’t know if that’s improbable. It seems parsimonious to me.

Dwarkesh Patel

It also seems significant that different groups of humans at this time were capable of adopting Stone Age technology. Once one group had figured it out, the genetic difference between different human lineages was not so big that you could not show people how to use stone tools.

David Reich

Who knows? It could be that this was genetically driven. We talked before about the time to the common ancestor of human genes. There’s nothing at 100,000 years or 150,000 years, but there’s a lot at 400,000 or 500,000 years.

If that’s what happens, and you have a mutation that occurs in the Caucasus, somewhere in the Middle East, or Northeast Africa, there could be key genetic mutations that make people able to do this. Then this population expands. When it moves into Europe, it’s swamped by local genes, but there could be retention of those genes through selection as it expands.

Maybe what you’re seeing is that there are genetic developments. Most of the discussion has been focused on the 50,000 to 100,000-year event, which is anatomically modern human behavior. But a lot of archaeologists think this is an equally—if not more—profoundly significant event in many ways. Why is that not the event we should be talking about?

Dwarkesh Patel

You’re talking about how there are no fixed differences between modern humans and the humans 50,000 years ago. Do we know if there are any fixed differences between the people 50,000 years ago and the people 300,000 years ago?

David Reich

I think there are.

Dwarkesh Patel

Other than obviously these interbreedings.

David Reich

If you look at the genetic variation going back 300,000 or 400,000 years, there do begin to be places where all modern humans share common ancestry. That’s another way of saying there begin to be fixed differences at that time depth. That is where you start seeing evidence for possible fixed differences.

What’s happening, if everybody shares a common ancestor 400,000 or 500,000 years ago, is that there’s a single ancestor at that time. If you compared it to another population, they would descend from a different lineage, so any mutation that occurred ancestral to that single ancestor would be a fixed difference. This is the time at which you can begin to see fixed differences.

Dwarkesh Patel

But anatomically modern, cognitively modern humans exist by the beginning of the Middle Stone Age, before we’re breeding with this ancient group of Africans or breeding with Neanderthals.

David Reich

Anatomically modern humans occur exactly here. It’s the same moment. This is when they occur. The people who have skeletal features like ours, and Neanderthals, appear exactly then.

This is when it all happens. There is this disconnect between anatomically modern humans in the skeletal record and behaviorally modern humans, which is 50,000 to 100,000 years ago. Anatomically modern humans appear at this time, and recognizable Neanderthals appear roughly around this time, too.

Dwarkesh Patel

Interesting. But we don’t know what exactly happens, if anything, between 200,000 years ago and 50,000 years ago that goes from just anatomical modernity to behavioral modernity.

David Reich

My understanding is no. They’re busy making Levallois stone tools like Neanderthals for 200,000 years, and they are not more impressive than Neanderthals in any obvious way, as I understand it.

Then there begins to be in the archaeological record a quickening of behavioral traits, which could be not genetic at all, or could be genetic. There are lots of arguments about this. We were obsessed with intelligence earlier in our conversation. People are obsessed with art and these things that seem important to us but who knows what’s important?

Dwarkesh Patel

Interesting. Cool, thanks for the digression.

David Reich

The work that I’ve been involved in has consistently shown that I was wrong in my biases coming into the work, and I’ve really been almost traumatized by this. Again and again, I’ve come into a project with some kind of guess about what the data was showing, and then the data doesn’t show that.

For example, when I got involved in the Neanderthal genome project helping to analyze data looking at how archaic Neanderthals were related to modern humans, I was part of a group of scientists who had established that non-Africans were a simple subset of African variation and that there was no evidence at all of Neanderthal interbreeding into the ancestors of modern humans or other archaic interbreeding. Different analyses that I and many other people had done made it look like non-African variation was just a subset, a small sample of that in Africa, and that could have fully explained the data.

So when I was involved in analyzing the Neanderthal DNA sequences, what happened was I found this very strong evidence of Neanderthals being more closely related to non-Africans than to Africans. It was very surprising, and I thought it must be a mistake. I was quite incredulous. I thought it was unlikely to be true because other evidence that had been found before seemed to point in the other direction. So I spent several years trying to make these results go away, as did my colleagues, and we just couldn’t make the results go away. They just kept getting stronger.

And this experience working on natural selection was the same. What we were convinced of was that natural selection had been pretty quiescent in our species over the last several hundred thousand years. Therefore, if we look at patterns of variation in non-African people today, or in any people today, we should see not a lot of selection going on. Indeed, the first ancient DNA studies, beginning in 2015 with this paper that we were involved in with Ian Mathieson and colleagues, seemed to show relatively small numbers of genetic positions associated with natural selection.

In 2015, we analyzed data from about 200 Europeans and Middle Easterners to try to understand frequency changes over time. We compared those ancient people who were the sources of modern Europeans to people in Europe today, and we looked at frequency differences that were too extreme to be due to chance. We were very excited to find 12 positions that we were convinced were highly different in frequency between Europeans today and what we would expect, based on the history that we and others had identified as the history relating modern to ancient Europeans. Some of these were known and some of these were not known, and this was very exciting.

We hoped that as the numbers of samples would increase and we got higher resolution to be able to appreciate differences in frequencies over time, it would make it possible to detect far more. What was quite disappointing over the subsequent decade is that that didn’t happen. For example, the largest study of that type in 2024 by a group in Copenhagen analyzed much better data than we had in 2015 and found only 21 positions that were highly different in frequency across time. While that was exciting—it was almost twice as many as we had found in 2015—in a lot of ways it was disappointing because the sample size and data quality had gone up so much, and yet this is all that was found.

That suggested we might be hitting an asymptote and might not be able to get beyond where we currently were. This approach to learning about biology, which was very promising in theory, might not produce a high yield. Maybe natural selection was quiescent, and the reason we’re seeing so few changes is that there has not been a lot of adaptive directional selection. That was the situation we found ourselves in until just a few years ago when we carried out this study in our research group led by Ali Akbari.

What we did is we deployed a few innovations to try to improve our power to detect natural selection. One of them was that we just pumped a lot of data into the system, increasing the amount of data by about 14-fold. The main thing that we do in this study is report new data from about 10,000 individuals. This is a very big increase in the amount of data in the literature. The total dataset size of ancient individuals distributed over the last 18,000 years is about 16,000 people. This is a large dataset. It’s much larger than was previously possible, and when you have more data, you can estimate frequency changes with much more subtlety.

The data comes from only one part of the world, which is Europe and the Middle East. It’s not a more important part of the world than other places, but it’s the place where maybe 70-80% of the data in the ancient DNA literature so far comes from due to historical reasons. It provides us with a natural laboratory where we can see what happens to the genome in one place over time as environments change. It’s really interesting to imagine doing this type of analysis in other parts of the world, and the comparative analyses are super important and interesting, but this study right now is about this one place in the world where we have particularly fantastic data.

The other thing we did is that we developed an entirely new methodology that hadn’t been used in this area before. The methodology is based on a technique that had been developed for finding risk factors for a disease in medical studies. A simple way to explain it is that we ask how to predict the genetic type a person has based on their pattern of relatedness to other people. We have a dataset of about 16,000 ancient people, and 22,000 people if we include the modern people. Then we look at how closely related each of these 22,000 people are to each other, and we predict the genetic type at each position in the DNA—at 10 million positions—based on the pattern of relatedness to all of the other 22,000 people.

Then we ask if natural selection blowing the frequency of the mutation in the same direction in all geographic places and at all times predicts the data a little bit better than just knowing the relatedness to all the other samples in the database. We’re simply asking if the alternative hypothesis—that selection has been blowing in the same direction at all times—explains the data better. That’s a dumb assumption, because of course, the truth is that natural selection will have changed in frequency over time. But we’re just asking the simplest of questions: whether assuming a constant rate of selection explains the data more than not doing so.

Dwarkesh Patel

To summarize to make sure I’ve understood, you’re trying to make a model that predicts allele frequency changes over time. You have two different parts. One part is this genetic relatedness matrix, which captures how similar different genomes are to each other. That should capture the impact of different bottlenecks, of drift, of population admixtures, and all those things which affect the entire genome.

Then you have the separate thing, which is, if we look at specific locations, can we just say that, “Oh, this location has been selected at whatever coefficient over time”? And if we add some coefficient, does it become easier to predict the allele frequency changes than you would have just seen from this other artifact, which is just looking at, “Oh, if you look at the whole genome, are these guys in the same, have they gone through the same bottlenecks? Have they gone through the same drift,” etc.?

David Reich

That’s precisely right.

Dwarkesh Patel

Okay, what did we learn?

David Reich

When we analyzed the data this way, we looked at 10 million positions in the DNA across these 22,000 people, 16,000 of whom were ancient. We looked to see if there was more change in this consistent direction over time than you would expect by chance. When we analyzed the data, we found many hundreds of places in the DNA that were changing too much over time and in too consistent a way to be explained by chance.

There’s a bit of a statistical problem in figuring out how many there are because they’re so densely packed that they’re close to each other and interfering with each other. But when you try to piece them out and say, “Let’s count only one in each place in the DNA and blank out the others,” we find at least 479 positions that are all independently pushing in the same way. We are 99% confident that those positions are real. By another criteria of being more than 50% confident that they’re real, we think that about 3,800 positions are all pushing in the same direction. This is a crazy number of results given that in our previous work and other people’s work, there were at most a couple of dozen discoveries coming from a single scan.

So when we got this result, we were very surprised. We thought it must be wrong, and we spent the next couple of years trying to make the results go away, but they just kept getting stronger. We were trying to look for some independent type of evidence to tell us whether these positions were real. We stumbled on something really powerful for this purpose that had not been used in this way before. It relied on the fact that we had very large numbers of discoveries, many hundreds of discoveries or even thousands.

We took a completely independent dataset, which was the corpus of genome-wide association studies. These are studies that people have carried out in hundreds of thousands of people, looking for whether particular genetic mutations are more common in people with high blood pressure than with low blood pressure, or something like this. We took the UK Biobank, which is about 500,000 people from Great Britain who have been measured for hundreds of traits. The whole genomes of all these people have been sequenced. For each of these traits, we could look at whether each of these 10 million positions are connected to this trait in a convincing way. Out of 10 million positions, about 15%—about 1.5 million positions in the DNA—are predictive of at least one of these several hundred traits.

Then we could ask a question: is our natural selection signal, our statistic, related to whether a mutation causes high blood pressure or some other trait? We slid our statistic for natural selection upward, to a value of one, two, three, four, or five. As we did that, the enrichment for genetic mutations that affect traits got higher and higher. Whereas it was only 15% when we didn’t use our selection statistic, when we required the selection statistic to be above about five, there was about a five-fold enrichment for mutations that cause traits.

Dwarkesh Patel

Sorry, what is a selection statistic?

David Reich

This is the statistic we use to measure whether a mutation is changing over time significantly in a non-zero way. It can be approximately thought of as a normally distributed statistic, a Gaussian statistic, which is the number of standard deviations the statistical value is away from zero, where zero is no natural selection.

It’s not exactly that, but it’s close to that. If the statistic is above five, we see about a five-fold enrichment in mutations that affect a trait. Instead of 15% of the mutations that are at random affecting the trait, it’s 60% or 70% that are affecting the trait when we slide our statistic upward. This provides completely independent evidence that these sites are real, and as you slide above five, there’s no more enrichment. Our interpretation of these results—which we were able to validate and show made sense using computer simulations of our process—is that once you slide the statistic above five, essentially all the signals of natural selection are real.

Dwarkesh Patel

Okay, just to make sure I understand. You’re saying, in order to figure out what alleles have been under selection, your model assigns a statistic saying, “In order to explain why this allele has a specific frequency, we’re going to give it a selection statistic.” Independently, we run these studies on modern populations where we say, “If you look at height, eye color, intelligence, or whatever trait, what are the parts of the genome that are correlated with that trait?” The higher the statistic you give it in your study to explain allele frequency changes over time as a result of selection, the more probable it is that that region in the genome is associated with traits that have some functional thing we can measure.

David Reich

That’s exactly right. This is a brilliant idea that Ali had. It abandons the traditional approach of assigning statistical significance to mutations that cause a trait because we’re just using an external piece of information—the correlation to traits, measured in a completely different way—to read off the probability mutations are real.

We can ask how much enrichment for real signal is there given a particular selection statistic. If it’s halfway enriched to the plateau, we’re able to show the correct interpretation is that 50% of the mutations are really selected. If it’s three-quarters of the way toward the plateau, there’s a three-quarters probability that the mutation is real. If it’s 99% of the way to the plateau, there’s a 99% probability that it’s real. That gives us a calibrated estimate of the probability that a particular position is really under natural selection.

A major concern here is that what we’re actually seeing is not that these mutations are really under selection, but rather that both association to a disease and our selection signal are due to some third thing that’s causing both of them, which is a type of selection which is not what we’re after, not selection to adapt to new environments, but what’s called background selection: selection against newly arising bad mutations that are removed from the population that tend to be concentrated in genes. Genes are also the parts of the genome that tend to be associated to traits. This common process is causing both the enrichment for trait signals and is also causing the enrichment for selection signals that we’re observing. That’s the concern. We were super concerned about this.

So what we did is we repeated this enrichment analysis in slices of the DNA that all were affected to the same extent by background selection, by this rain of slightly bad mutations, and we get exactly the same pattern. We also repeated this experiment just using mutations of the same frequencies because there’s different statistical power to detect these signals at different frequencies. We see the same pattern where above a value of the selection statistic of around five, we get this plateau.

Dwarkesh Patel

The thing that changed that allowed you to increase the amount of sequences you’re generating by two orders of magnitude is just the statistical method you’re using to identify which part is human? Or what exactly changed in 2014 and since then?

David Reich

There’s been a whole series of improvements. The big ones have been the huge drop in sequencing cost, which made it possible to generate ancient DNA in the first place. The drop in cost has been a millionfold since the late 2000s, and another maybe one to two orders of magnitude from 2010 to today. That’s one big change.

Another change has been in-solution enrichment. It’s been this way of taking a sample that has very small percentages of human DNA, but then suddenly creating a process that will mean that the great majority of the sequences that one’s analyzing will be useful for analyses. The approach that we used was we took the DNA samples that we had, most of which were very low percentages of human DNA—less than 10%, often less than 1%—which is such a low proportion that it’s prohibitively expensive to sequence them and to just brute-force sequencing them given the technology that we had available at the time.

We took these samples and washed them over an artificially synthesized set of short DNA fragments that targeted positions of the DNA that we were interested in analyzing. This is more than a million positions that are highly variable in people, and we picked many of these to be biologically interesting. We had a whole set of known biological targets that affected traits in genome-wide association studies, which is the way that people look to see if there are particular genetic variants in modern people that have particular impacts in phenotypes and traits.

And so, what we did is we had this artificially synthesized set of DNA fragments that we washed our ancient sample over, and it bound the parts of the DNA that we targeted. The resulting sequence that we generated was very enriched for the parts of the genome that were informative about history. Even though only 10% or 1% of the DNA was human, it ended up that a very large fraction was from the parts of the genome that we were interested in, and it became economically efficient to do it.

Dwarkesh Patel

What was the other 99% of the DNA?

David Reich

It’s mostly microbial. It’s from bacteria and fungi that colonize a person’s body after they die. Depending on how they die, there’ll be more or less of these bacteria and fungi. When you typically sequence DNA from a person, it’ll just be full of microbial sequence. Sometimes the microbial sequence is very interesting; it might be pathogens that a person died of. There’s amazing work, for example, about different plagues of malaria and Black Death and hepatitis B and so on that have been obtained from the sequences of these pathogens in people’s teeth and other parts of their body when they died.

But we’re focusing here on the human DNA. This changed the amount of data that was possible to produce from tens per year to hundreds per year, and then we further roboticized and industrialized the process so that there were many hundreds or even thousands per year. Just in our laboratory, we’ve been generating genome-scale data from more than 5,000 individuals per year. I know this is true also of several other laboratories in the world now.

This huge jump in data, this semi-exponential or even super-exponential jump in some cases, has made it possible to ask and answer questions. While we were only on the order of 10 genome sequences from humans in 2010, this year it’s passed more than 20,000 reported sequences. There are several orders of magnitude increase, and the questions we were able to ask in 2014 are just not the same as the ones we can ask today.

Dwarkesh Patel

Awesome. Excellent. David, thanks for your time.

David Reich

Thank you, Dwarkesh.

Did a very different format with Reiner Pope - a blackboard lecture where he walks through how frontier LLMs are trained and served.

It’s shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk.

It’s a bit technical, but I encourage you to hang in there – it’s really worth it.

There are less than a handful of people in the world who understand the full stack of AI, from chip design to model architecture, as well as Reiner. It was a real delight to learn from him.

Wrote up some flashcards and practice problems to help myself retain what Reiner taught. Hope it’s helpful to you too!

Reiner is CEO of MatX, a new chip startup (full disclosure - I’m an angel investor). He was previously at Google, where he worked on software efficiency, compilers, and TPU architecture.

Highly recommend the scaling book he coauthored for further study.

Watch this one on YouTube so you can see the chalkboard.

• Jane Street needs constant access to incredibly low-latency compute. I recently asked one of their engineers, Clark, to talk me through how they meet these demands. Our conversation—which touched on everything from FPGAs to liquid cooling—was extremely helpful as I prepped to interview Reiner. You can watch the full discussion and explore Jane Street’s open roles at janestreet.com/dwarkesh

• Google’s Gemma 4 is the first open model that’s let me shut off the internet and create a fully disconnected “focus machine”. This is because Gemma is small enough to run on my laptop, but powerful enough to actually be useful. So, to prep for this interview, I downloaded Reiner’s scaling book, disconnected from wifi, and used Gemma to help me break down the material. Check it out at goo.gle/Gemma4

• Cursor helped me turn some notes I took on how gradients flow during large-scale pretraining into a great animation. At first, I wasn’t sure the best way to visualize the concept, but Cursor’s Composer 2 Fast model let me iterate on different ideas almost instantaneously. You can check out the animation in my recent blog post. And if you have something to visualize yourself, go to cursor.com/dwarkesh

(00:00:00) – How batch size affects token cost and speed

(00:32:09) – How MoE models are laid out across GPU racks

(00:47:12) – How pipeline parallelism spreads model layers across racks

(01:03:37) – Why Ilya said, “As we now know, pipelining is not wise.”

(01:18:59) – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal

(01:33:02) – Deducing long context memory costs from API pricing

(02:04:02) – Convergent evolution between neural nets and cryptography

Dwarkesh Patel

Today, I’m interviewing Reiner Pope, who is the CEO of MatX, which is a new chip startup. Previously, he was doing TPU architecture and many other things at Google. This is a very different format from my usual interviews. This is going to be a blackboard lecture. We’re going to get up in a second. We in fact built this whole new studio with specifically this format in mind, so it’s a pleasure to get to inaugurate it with you.

We’re going to be talking about model architecture, ML infra, and many other things. The reason I think it’s an important topic is because once you understand how training and inference work in a cluster, a lot of things—about why AI is the way it is, why AI architectures are the way they are, why API prices are the way they are, and fundamentally why AI progress is the way it is—start making sense. You need to understand the details to get there, and you need a blackboard to understand the details. Reiner, thank you so much for doing this.

Reiner Pope

Very happy to be here.

Dwarkesh Patel

Full disclosure, I am an angel investor in MatX, but that’s unrelated to this podcast. Reiner, to kick us off I’ll ask this question. We have a couple of companies like Claude and Codex and Cursor offering something like Fast Mode, where for 6x the price, they’ll stream you tokens at 2.5x the speed. Mechanically, I’m curious what’s going on here. Why is it the case that you can pay more to get faster latency?

Two, could you keep going? Could you pay 100x more and somehow get much faster speeds? Three, could you go the other way? Could you have something like Claude Code “Slow Mode”, where if you are willing to wait for minutes on end, you could get even cheaper prices? Maybe this will help motivate the analysis that you’ll be doing through the lecture.

Reiner Pope

Great. To jump to the conclusion a little bit, the big effect is batch size. What we’re going to do now is quantify exactly what that looks like and what its implications are on latency and cost. There’s another effect, which you can call speculative decoding or multi-token prediction. We can maybe come back to that later, but the first thing that we’ll talk through is batch size.

What I’d like to introduce is the two principles of analysis. First, we’re going to look at a roofline analysis of how we run a transformer model on a cluster of chips. We’ll take a Blackwell NVL72 cluster, so a rack of 72 GPUs. The roofline analysis means we look at memory bandwidth and compute performance. The other side of that is that we’re going to look at just two simple factors of the model: the time to operate on the weights, and the time to operate on the context, the KV cache.

Let’s jump in. We’re going to try and estimate the time that it takes to run an inference of a certain shape. We’re not perfect here. We can’t exactly predict the time, so instead we’re going to approximate. We’re going to say that the time must be greater than or equal to a certain quantity. We’re going to consider two different aspects: the time it takes to do the memory fetches, and the time it takes to do the compute. It will turn out that this gives us very strong predictive power, even with a simple model.

One by one, what is the time that it takes to do the compute? There are really two things I need to do in the compute. I need to multiply by all of the active parameters, and then I need to do some work on the attention. Multiplying by all the active parameters, I have a certain batch size that I’m running, and I’ve got a number of active parameters in my model. Then I’m just going to divide this by the compute throughput, which is the FLOPs of the chip. This is a hardware concern.

This accounts for all of the compute time for all of the weight matrix multiplies. There’s a little caveat here. We’ve ignored the time to do any of the attention computation, but that in general will be quite small in comparison to this. So we’ll ignore this.

Dwarkesh Patel

I’ll just interrupt from time to time to ask some very naive questions or to clarify some basic points. For the audience, you’re not serving one user at a time. The batch refers to the fact that you’re serving many different users at the same time, and that’s a whole batch.

Reiner Pope

I can motivate the batch at least a little bit. We will see exactly why batch is such a favorable optimization. What will turn out to be the case is that if you do not batch together many users, the cost and the economics you get can be a thousand times worse than if you do batch many users together. We’ll be able to see that quite explicitly.

Then, number of active parameters. If I look at, for example, a DeepSeek model, the DeepSeek V3 model has about 37 billion active parameters, and 700 billion total parameters. We’re focusing on just the ones that are active for a single AI token.

We’re modeling compute performance. I’m going to keep writing equals, but in all of these cases, you can think of this time as being at least this much, and maybe there will be some terms we ignored.

On the memory side, what do we need to do with memory? We need to fetch all of the weights, so there is some time to fetch the total number of parameters, not just the active parameters. There’s weight fetch time, and then in addition, there’s a KV cache fetch time. This actually depends on batch size. For every element of the batch, we have to fetch an entire context length worth of tokens, and there’s a size per token, bytes for one token. This is a model parameter.

Dwarkesh Patel

Maybe just backing up, let’s explain what the KV cache is real quick.

Reiner Pope

When I do a forward pass… Let me draw how the autoregressive inference works. This is during decode. If I have a bunch of text tokens… I’m drawing a tensor because ultimately the tokens are represented as a tensor in some embedding dimension. In this direction, I have the sequence length.

The work of running a decode is that I have to run each token through a whole bunch of matrix multiplies over a bunch of different layers. In general, I’m going to have to do that work over all of these tokens. But one step of decode is to produce just this one additional token up here.

What I’m going to do there is run a full forward pass of multiplying by all of the weight matrices in the entire model. But then I’ve got this attention mechanism where this token is looking at all of the past tokens, and what is it looking at specifically? It is looking at some internal representation that the model has produced of the tokens, and we call that the KV cache. This process of this single token attending to all of the history of tokens is attention. It is mostly dominated by memory fetches rather than matrix multiplies.

So we’ve got the amount of memory that we’re fetching shown over here, and then this is of course just divided by the memory bandwidth, so the memory bytes per second. In fact, these equations here are enough for us to now draw some fit lines. The things that we’d like to look at are sensitivity to batch, and then also, which we’ll draw separately, to context length. We said that the big effect you can get is some trade-off in latency versus cost in batch size.

Let’s draw them out. I think there are just really two graphs that we want to draw. We’ll first draw batch size versus time here. When we look at the shape of this, we’ve got a maximum of the sum and then another term. Let’s look at these terms one by one and how they scale: the time for compute and memory, and how they show up.

Let’s first look at this compute time. This is just purely linear in batch size with no offset, so it is some curve like this. This is t compute. On the memory side, we’ve got some portion here that is just this constant in some base offset here, which is the weight fetch. Finally, we have this term here, which is the KV fetch, which is pretty linear in batch size, and so it looks like that. The sum of this plus this maxed with this… Let’s at least first draw the sum. The two memory times in conjunction end up looking on this curved slope like this. Then the overall maximum is—I’ll draw a little thicker here—the maximum of these two curves.

What does this mean? This is a latency plot. If I grow my batch size, initially I get some not very strong dependence on batch size, so there is some lower bound on latency here. This already partially answers the question. For a given hardware configuration—and we can talk about varying the hardware configuration—there is a lower bound on latency. It is simply that I need to read all of my total parameters from memory into the chips, and that takes a certain amount of time. If I use all of my memory bandwidth, I can’t do any better than that.

Dwarkesh Patel

It seems like the way you’ve drawn the slopes for compute time and how the KV grows—and what implication the KV has on memory time—

Reiner Pope

What if this were above or below?

Dwarkesh Patel

Yeah, is that necessarily the case? If this is always true, then as batch size grows compute always dominates KV, which suggests that if you have a big enough batch size, maybe memory is never an issue.

Reiner Pope

This is really sensitive to the context length, so I think we should come back and explore this. As you vary the context length, the KV fetch time will go up and up, and that will cause a transition from compute-limited to memory-limited.

Dwarkesh Patel

Is there something especially significant about the slope being exactly the slope of the compute time?

Reiner Pope

Whenever we have balance points, it says that you’re getting it exactly right. For the particular context length where the slopes match, that says I am equally memory-bound and compute-bound, which is a really desirable place to be.

Dwarkesh Patel

This is a very simple algebra problem, but suppose the optimal is 100K context length, and you go to 200K context length. Does your MFU go down to 50%? Does it have a humongous impact on MFU to be slightly outside of the optimal context length range, the Goldilocks zone?

Reiner Pope

That’s right. That is true as modeled here. There is a key point here that I’m modeling the memory fetch as linear in context length. That depends on model architecture. It is true for all of the model architectures with dense attention. Sparse attention actually scales much better than that.

Dwarkesh Patel

Got it. Is sparse attention what everybody uses in practice?

Reiner Pope

I’m pretty excited about sparse attention. It’s hard to know what the labs are using. DeepSeek has published a sparse attention mechanism. I’ll just put a plug in that some of the DeepSeek papers that have published sparse attention end up putting a square root in this term.

So far, we’ve looked at the latency. It’s hard to read off cost from this. If I think about what cost means… To run this inference, I’m going to use the GPU for a certain number of seconds, like one millisecond or 20 milliseconds. I have to pay the rental time for that time. So it’s $2/hour per GPU or something like that.

That’s the cost of this inference, but how many tokens have I processed during that inference? That is the batch size. What we actually want to plot is the cost versus batch size, which is t over B versus batch size. This is the cost per token. We have to imagine dividing each of these three curves by B, so multiplying by this reciprocal. What we end up with there is… The compute curve was linear. We divide by B, and that makes it a constant here. This is t compute. The KV fetch was linear, and now it becomes a constant as well. Then the weight fetch was constant, and now we’ve divided by B, so it becomes this hyperbola.

Again, we’re going to compute the max of the sum. The sum of these two terms shifts the hyperbola up. The sum of the KV fetch and the weight fetch gives us a higher hyperbola that’s like this. Then we’re going to take the max with the compute here. We end up with this being the overall shape that we care about.

Again, we see some limiting behavior. The cost initially starts very high at a batch size of one. It almost goes to infinity because we’ve got so many weight fetches that are not amortized over a large batch size. But as we increase the batch size, the weight fetches become amortized over so many different batch elements that their cost grows very small, and eventually the compute time ends up driving the cost. So there is a limiting lower bound on cost, which is this line here.

Dwarkesh Patel

So Claude Code Slow or Codex Slow or whatever would just live on this line. It wouldn’t help much because you’re not able to amortize the KV values over a much bigger batch.

Reiner Pope

They’re unique per batch. The compute is also unique per batch. So what is the minimum work you can do per batch after amortizing everything else away?

Dwarkesh Patel

This point where you are no longer memory bandwidth bound, practically how big a batch do you need? How big are the batches practically for frontier models?

Reiner Pope

You can just solve for that. It’s not even particularly sensitive to model architecture. Let’s go ahead and do that.

What we’re talking about is when the memory time is equal to the compute time. That’s what that question is. Because we’re focused on what the batch size is—and really there’s a question of when the weights are amortized over the multiplies—I’m going to focus on comparing the weight fetch time to the weight multiply time. I’m going to disregard the KV fetch term just to simplify the analysis so we can get a clean answer out. We’re going to equate this portion with these two times.

Writing that out, we get N, number of total parameters, over memory bandwidth, is equal to batch size times number of active parameters divided by the compute performance. Looking over here, everything on the top are model parameters. Everything on the bottom are hardware parameters. It turns out to be nice to rearrange them such that we have the hardware parameters on one side.

This is equivalent to FLOPs over memory bandwidth being equal to batch size times number of active parameters, divided by the number of total parameters. This hardware parameter ends up being a dimensionless constant. If you look in terms of FLOPs… What are the dimensions of this? This is multiplies per second. This is bytes per second. So that’s not quite dimensionless. But what you do is you say, how many FP4 multiplies per second times the fact that each FP4 is half a byte. I can actually make this end up being dimensionless. On most GPUs, this ends up being somewhere around 300.

Dwarkesh Patel

Has that ratio changed over time as we’ve gone from model generation to model generation, where the FLOPs keep increasing?

Reiner Pope

This is a hardware parameter. To what extent has the hardware changed? From A100 to H100 to B100, the FLOPs have increased substantially, the memory bandwidth has also increased substantially, and it has remained reasonably stable.

We can express this one as well. This is a sparsity parameter. I might even phrase this slightly differently. Let’s solve for batch size in total. Moving this back over to the other side, we end up with batch size needs to be bigger than approximately 300 times sparsity. For example, in DeepSeek I activate 32 out of 256 experts, so this would be 8 for DeepSeek.

This actually gives you a ballpark which is remarkably accurate to practice. Generally, people will go a little bit larger than this. They don’t really want to be exactly at the balance point because real-world efficiencies aren’t as good as a roofline analysis would say. But take this and maybe double or triple it.

Dwarkesh Patel

Okay, so it’s two to three thousand tokens per batch. But then if you included the KV cache, the implication would be that the optimal batch size...

Reiner Pope

Should grow larger. We solved for the equivalence between when compute time is equal to memory time. If I add in something that consumes more memory bandwidth, then I have less available for the weight loads. I need to grow the memory bandwidth more, and therefore the batch size more.

Dwarkesh Patel

This seems incredibly small. This would be less than one sequence, right?

Reiner Pope

Keep in mind that I’m talking about the number of tokens that I’m generating one more token for. It’s actually 2,000 unique sequences.

Dwarkesh Patel

Got it. We’re just talking about a single forward pass on these sequences. You think of the batch as the number of sequences.

Reiner Pope

That’s right.

Dwarkesh Patel

If you’ve got a frontier model and you are actually doing inference, surely they must have more than 2,000 concurrent users. Is there any added latency from the fact that you need to have the whole batch fill up? Or if you have a reasonable amount of users, is it so unlikely that it would take you 100 milliseconds to fill up the next 2,000 slots?

Reiner Pope

The way to think about this is: when does the train depart, as a model? Let’s say I’ve picked a batch size that I’m going to run at. By the way, this intersection point is the same intersection point here. I pick this batch size, and I know that it’s going to take, for example, 20 milliseconds, which is a common place this ends up landing.

This is a timeline of what is running on the GPU. It’s going to start a new batch every 20 milliseconds regardless. You can think of this as a schedule for the train. A new train departs every 20 milliseconds. Any passengers who are ready board the train. If the train is full, they wait until the next train. If the train is not full, the train is going to go anyway.

In terms of what that means for queuing latency, the worst case is that a request arrives just after the train departed. It has to wait for the next train, so that’s up to 20 milliseconds, and then it has to wait for that train to complete. So the worst-case latency is 40 milliseconds.

Dwarkesh Patel

How is the 20 milliseconds derived?

Reiner Pope

It’s a rule of thumb, but where it comes from is not fully explained yet. So far we’ve focused on memory bandwidth and compute time. When we look at memory, the other consideration is that we want to use all of the memory capacity we have. Generally, we’re going to use all of that memory capacity to store the weights or the KVs. In the time of doing a forward pass, we want to read all of the memory capacity into the chip. That is capacity divided by bandwidth. That tends to be 20 milliseconds on many different generations of HBM.

Dwarkesh Patel

The units make sense. You would have a byte divided by bytes per second.

Reiner Pope

For example, on the Rubin generation, it is something like 288 gigabytes divided by 20 terabytes per second. This comes out to about 15 milliseconds.

Dwarkesh Patel

Let me make sure I understand what this is saying. I understand the unit analysis. What it’s saying is we can evacuate and replace the HBM in this amount of time. So we don’t want to be in a situation where the HBM is not big enough that we’re not actually able to write everything we want to it or take everything out of it. Or we don’t want to be in a situation where our ability to write back and forth is so small compared...

Reiner Pope

There are sort of two scenarios. Why don’t we pick a latency that is bigger than 15 milliseconds? If I think about what that means, it means I actually have time to read the HBM twice. By the way, most HBM accesses are reads, not writes. It’s almost all reads because the weight matrices are read-only, and almost all of the KV cache accesses are reads. In around 30 milliseconds, I can read all of HBM twice, but what’s the point of that? I don’t want to read the weight matrices twice. I don’t want to read the KVs twice.

Dwarkesh Patel

Makes a ton of sense. A couple of quick questions. If it is the case that the optimal batch size is something like 2,000, it’s totally dependent on the sparsity, not dependent on the model size or anything.

Reiner Pope

Sparsity shows up in model size, but beyond that, it only depends on sparsity, not on scale.

Dwarkesh Patel

That’s a very interesting result. One question is how much of a push towards centralization is it that you would have these economies of scale from inference for batching? But it seems like it’s not that big a deal. Is 2,000 users at the same time a lot? It doesn’t seem like a lot.

Reiner Pope

We can do a bit of analysis on this. You can think of it in terms of number of users, but a more productive way to think of it is in terms of tokens per second. What does this batch size mean in terms of tokens per second of the system?

Tokens per second is going to be equal to the batch size. We run a batch of tokens, and we do that every time interval, which is equal to the 15-millisecond or 20-millisecond number. This ends up being batch size times about 60, so 64 x B. This ends up being around 2,000 x 64, so 128,000 tokens per second. This is in more digestible units.

It’s hard to reason about concurrent users, but what is the global traffic for a system? When you look at some of the announcements, sometimes the API providers will brag about how much traffic they have. The numbers I remember from some announcements of Gemini last year were in the hundreds of millions of tokens per second worldwide. This is one-thousandth of that.

Dwarkesh Patel

Gemini is big. One-thousandth of Gemini is a lot. To actually be competitive at scale, you need to be able to serve at least one-thousandth of Gemini. That’s interesting.

The more sparsity you have, the less compute you need. It does seem that as batch sizes get bigger, compute ends up being the bottleneck, according to this analysis. Then the question is, how far can you take sparsity? As the sparsity ratio increases, as you have fewer active parameters relative to total parameters, how much is the performance of the model degrading? Is it degrading faster than you’re saving compute by increasing the sparsity factor?

Reiner Pope

You mean the quality of the model, rather than the speed of the model. Unfortunately, we’re not able to answer that analytically. That is an empirical question of model quality. The best I can do is pull up a paper and answer that empirically.

Dwarkesh Patel

Should we pull up the paper now?

Reiner Pope

This paper is “Unified Scaling Laws for Routed Language Models.” It’s a somewhat old paper by this stage, but one of the things they looked at is if I keep increasing sparsity, what is the model quality impact? This answer is very sensitive to the actual choice of mixture of experts. Mixture of experts has been around for a really long time, maybe even back in 2017, but the techniques have changed a lot. DeepSeek’s mixture of experts was a big change in how it worked. There have been older papers, like “GShard” and “Switch Transformer”. The actual empirical results are going to depend on all of that.

On one of the older techniques shown here, you can see if I hold constant the number of active parameters at a certain size, and then I increase the sparsity, which they call expert count, the quality keeps increasing. If you imagine drawing a horizontal line from 1.3B dense across, you end up seeing that, in this case, the 64-expert 370-million activated parameter model is as good as a dense 1.3-billion model.

Dwarkesh Patel

So in some sense, it’s actually not amazing returns where you need to increase total parameters a hundredfold to get the equivalent of 10x as many active parameters.

Reiner Pope

Actually even more so. It’s a huge increase in parameter count for a modest increase in efficiency.

Dwarkesh Patel

So in this case, actually it’s 4x?

Reiner Pope

64x for 4x.

Dwarkesh Patel

So while it is true that you get this benefit of being able to economize on your compute time if you increase sparsity, naively it would seem like a trade-off worth making. But if you’re decreasing this by 2x and then having this go up by 8x every time you double sparsity...

Reiner Pope

Is that good or bad, actually? Even from a memory point of view… Keep in mind you are doubling this portion of the memory fetches, which is amortized by batch. So just keep running a larger batch size. From the point of view of the analysis we’ve done here, this is a pure win. Keep doing it until you run out of available users, basically.

There’s this equivalence where if I have a lot of users, I can go to a much sparser model. From that point of view, it’s a reasonable trade-off. The other trade-off that shows up here is that it also consumes memory capacity. We’ve only reasoned about memory bandwidth here, but it also consumes memory capacity.

Dwarkesh Patel

I see. Let me make sure I understood. You’re saying we want to spend less time computing, therefore we do more sparsity. To make that work, we need bigger batch sizes. Which means we need more memory capacity to have more sparsity.

Reiner Pope

Maybe this would be a good point to talk about how a mixture of experts layer is typically laid out on a rack of GPUs.

Dwarkesh Patel

Cool. Makes sense. Where were we?

Reiner Pope

Sparse mixture of experts. Maybe how we lay that out on a GPU.

Let’s zoom in on the mixture of experts layer first and draw what that looks like. Typically, we’ll have some kind of a router layer, which is making the decision of where we route the tokens to. We get tokens coming in here, they go through a router layer, and then we have a bunch of different experts. I’ll draw a few more to line some up.

The router will make a decision of which experts it’s going to route to, and it will be a small fraction of them, maybe 1 in 32. Maybe it will make a decision to route to this one, maybe this one, and maybe this one. Each expert itself is a normal MLP. It has an up projection and then a down projection with a nonlinearity in between.

Then finally, we do the inverse operation. Where we were broadcasting things out here, we’re going to bring them back in and sum them up. Bringing them in like this. Then finally, we have our residual connections. The token is also passed through here, and it gets added to the result of the MoE layer. This is a normal MoE layer.

What I want to talk through is how this is mapped to a GPU rack and what this means for communication, because I think this will start to show some of the limits of how sparse we can go. The standard practice here, and it is the best solution, is to use expert parallelism. That means different experts go on different GPUs. If we take something like a DeepSeek model, they have 256 experts. Let’s say we want to run that on a Blackwell rack. There are 72 GPUs.

We have a divisibility problem. This is not a power of two. We’ll just simplify and say we’re only going to use 64 of them. Just ignore the other eight. It’s not a big deal. So we have four experts per GPU. Very simple. For the sake of the diagram, actually let’s just say we have two experts per GPU. We end up just putting these GPU boundaries. Every pair of experts is on its own GPU.

Then we can look at the communication cost. We had some tokens stored centrally here. They get routed to all of these experts, and there is some communication cost paid here. There’s the same communication cost paid on the output. The hope is that this does not become communication limited.

Now what is the traffic pattern here? The traffic pattern here is that any GPU will be talking to any other GPU, depending on the decisions made by the model. This is an all-to-all traffic pattern.

Dwarkesh Patel

When you say any GPU in the pre-tense, the router is more than one GPU?

Reiner Pope

I drew this as one router. In reality, you would actually have many copies of the router, and you would have as many routers as GPUs, in fact.

Dwarkesh Patel

As the incoming traffic.

Reiner Pope

Yeah. These are 64 GPUs and these are 64 GPUs. It’s actually the same GPUs, we just draw them as separate because they’re serving different purposes. So at this point, any GPU can be sending to any other GPU.

This all-to-all pattern of communication that shows up and how the Blackwell racks are configured is a perfect fit for the communication pattern that the MoE actually wants to do. However, if you think maybe one rack is too slow and I want to do two racks, then I have this challenge that maybe I’ve got some sort of rack boundary drawn outside here like this, and I no longer have all-to-all communication between all the GPUs in two racks. The rack-to-rack communication ends up being a substantial bottleneck.

The fundamental thing here is that one rack bounds the size of an expert layer you can do. This has been part of what’s been driving towards larger and larger interconnect domains.

Dwarkesh Patel

Before we continue, it may be worth you explaining what exactly a rack is. The differences in bandwidth between a rack and within a rack, and the all-to-all versus not all-to-all nature of communication within versus outside.

Reiner Pope

This is a place where it starts to be very different between Nvidia, for example, and Google, and then others, including us. Generally, a rack is a physical structure. It’s a few meters tall, a meter or two wide, depending on configuration, and it stores some number of GPUs or XPUs, which is typically about 64.

What constrains it being a certain size is power delivery, weight, and cooling ability. It ends up being about this size in many cases because of these physical constraints. When I deploy a data center, a data center may have thousands of these racks. I’ve got one of these tall racks, it’s got a bunch of GPUs in it, and so on. And then I put another rack next to it.

Dwarkesh Patel

You make it sound so easy.

Reiner Pope

Right. I just drop them in. In Nvidia’s case, the communication topology… They actually put the GPUs on the outside of the rack, and then they put these switches on the inside of the rack. What this ends up being is that there’s a set of switches in here. These are the NV switches. Then they run a bunch of cables. Every single GPU has cables going to the switches in the middle. The switches have connections to all the GPUs. All of the GPUs can talk to all the other GPUs in just two hops: going to the switch, going to the other GPU.

Now, when I want to leave the rack, I end up going via a different path. The GPUs also have a much slower connectivity, which is typically about eight times slower. The green that I drew here in the GPU cases is the NVLink. More generally, it’s called the scale-up network. You will typically also have a scale-out network, which allows you to connect to some data center switch. All of the GPUs will have some connectivity up to some data center switch somewhere. This is the scale-out, and it tends to be about 8x slower in bandwidth.

The challenge, if you want to lay out a mixture of experts layer across two racks, is that half of the GPUs here are going to be wanting to talk to the GPUs here. On average, when I look at where the tokens on these GPUs want to go, half of the tokens want to go inside the rack. That’s great. They can use the fast scale-up network. But half the tokens are going to want to leave the rack and go to the other rack, and that’s not as good. They need to use a much slower network, and so that becomes the bottleneck on the all-to-all pattern.

A different choice would be, why don’t I have a big switch here and connect everything to a much bigger switch that actually combines the two racks together? There are many ideas in this direction, but in general, the reason you have this hierarchy of switches rather than one big switch is to manage the cabling congestion. You just need to run a large number of cables.

Dwarkesh Patel

Sorry, is that question you just asked basically, why isn’t it a bigger scale-up?

Reiner Pope

Exactly. Why not just have a million chips in scale-up or a thousand chips?

Dwarkesh Patel

What has changed that has allowed Nvidia to go from Hopper, which was 8, then Blackwell is 72, and now Rubin will be... is it 500 something?

Reiner Pope

Yeah, 500 and something.

Dwarkesh Patel

What has allowed that to happen?

Reiner Pope

From Hopper to Blackwell is mostly just the decision to switch from trays as the form factor to switching to racks as the form factor. That’s a product decision. There wasn’t a substantial technical barrier there.

Switching from 64 to 500 or so, there’s a bit of Jensen math there, but there is at least a genuine 4x increase, which is coming from a much more complicated and difficult rack design. That is actually a new physical design to run more cables.

Dwarkesh Patel

The cable complication is just the cost of figuring out which cable hops to which, or which signal goes from what to what?

Reiner Pope

Let’s zoom in on this and look at the wire density. I’ll draw this diagram just once more so we have a bit of a cleaner and larger version to work with.

Let’s say I have some switches in the middle. Initially, I’m going to start with just two GPUs on each side or two trays of GPUs on each side. Let’s say maybe each tray wants to have two cables coming out of it. I physically run vertical cables that look like this running out to the switches. Now if I want to double the number of GPUs in a rack, I need to run literally twice the density of cables. I need to run these as well.

Dwarkesh Patel

Extremely naive question. But if you look at a physical data center, it seems like there’s a lot of space within a rack. I don’t know. The cables are really big and...

Reiner Pope

There is space outside the rack. Inside the rack… As they become more optimized, these racks are very tight. There’s connector density going from the tray into the rack and the rack’s backplane, and the backplane itself has a really high density. There are other physical constraints including the bend radius of cables. You don’t want to snap them and so on.

Dwarkesh Patel

Okay, so it’s literally the physical space to put a cable that’s constraining it. I had no idea. Interesting. That seems surprising. The rack is so big and we can’t just stuff more cables in there.

Reiner Pope

Rack design is not my expertise, but when I talk to folks on what constraints they’re up against, it’s a combination of things. What are the big physical things you’re optimizing for? Space, weight of the rack. It’s actually really heavy, so you need enough metal to not sag and fall. But then you add more metal, and it’s heavier. Then power and cooling. All of those are competing. Modern racks are pushing all of those to very extreme physical limits.

Dwarkesh Patel

When was GPT-4 released again? Was it 2022 or 2023?

Reiner Pope

2023.

Dwarkesh Patel

Okay. And it was rumored to be over one trillion parameters. It seems like only now, within the last six months, have models been getting released that have significantly more parameters than the model released three years ago, when supposedly there should have been this scaling in the meantime.

Is the reason that we were just waiting for racks with enough memory to hold a five-trillion parameter model, along with its KV cache for enough users for a lot of sequences? Or if you’re doing RL, a similar consideration of actually holding the KV cache for the batch of problems you’re trying to solve.

If you look at Hopper, you had eight Hoppers, and I think that’s 640 gigabytes as of 2022. With Blackwell finally, which was deployed in…?

Reiner Pope

Very recently. Maybe last year.

Dwarkesh Patel

Last year. You finally have a scale-up on the order of 10-20 terabytes, which is enough for a 5T model plus KV cache.

Reiner Pope

Deploying in larger scale-up domains is a huge unlock. I’ve drawn here the Nvidia Blackwell deployment. The Google deployment has actually had very large scale-up domains for a long time.

Dwarkesh Patel

That also explains why Gemini seemed to be ahead. It just seems like Gemini has had successful pre-training for longer than some of the other labs.

Reiner Pope

Not having been there at the time, I’m not sure how much is coming from successfully deploying higher sparsity ratios, which it could be. It could also be a whole bunch of actual modeling things, specifically how you do the mixture of experts. We’ve seen the DeepSeek mixture of experts activate more experts, but finer-grained experts. That was a big innovation. I’m sure there are many other innovations on the model architecture as well as on the training data.

It’s hard to disentangle all of them, but what shows up in terms of the limits of what you can do is that the active parameters, as we saw, are limited by the compute cost, and the total parameters are limited by the scale-up size.

Dwarkesh Patel

When you’re operating within a single scale-up domain, is that a consideration specifically for either forward or backward, or specifically for prefill versus decode? Or is it preferred to always be within a scale-up whatever kind of workload you have, whether you’re doing a pre-training run, RL generation, or inference for users?

Reiner Pope

Really interesting. To answer that question, we’re going to need to talk about the communication patterns. We’ve talked about the mixture of experts communication pattern. That is this all-to-all. All-to-all very strongly favors full connectivity, which is what we’ve just shown here, and it favors being within one rack.

There are other kinds of parallelism besides expert parallelism, which we just showed here. In the literature is tensor parallelism. With the trend towards smaller experts, this has become much less relevant, so we can ignore that. But the other two things we have available are data parallelism and pipeline parallelism. They can be a much better fit for using multiple racks.

Let’s focus on pipeline parallelism specifically. This is one layer of MoE. I’m going to have a hundred more layers up above. I could decide at this point, for example, to move to a different rack, change rack. Now, is that going to become a communication bottleneck? We can actually solve for when this becomes a communication bottleneck. Before we do that algebraically, let’s visualize it out and sketch the path. We’re going to have another MoE layer, and another MoE layer here, and so on.

Let’s say I change rack here, and then some number of layers later, I change rack here as well. The methodology we’re going to use to determine whether we have a communication bottleneck at the point where we change rack is we’re going to compare the scale-out bandwidth requirements to the scale-up bandwidth requirements. Let’s write this. The hint is going to be that there’s a lot more sends here. We’re sending many things here, whereas we’re only sending one thing here, and we’re also maybe doing it many times. That’s what makes the difference.

Dwarkesh Patel

Can I try to guess? Just out of curiosity, to see if I’m actually understanding, it seems like you’re sending batch size into the rack.

Reiner Pope

In here? Yes.

Dwarkesh Patel

But the communication within the rack is batch size times number of GPUs.

Reiner Pope

Number of activated GPUs. I don’t send to this GPU at all. There’s an explosion from 1-3x larger here in this diagram. The key thing is that I didn’t even need to send to this GPU at all, and so that’s a big saving.

We’re going to talk through to what extent scale-up is a bottleneck over scale-out. We will directly jump to the ratio of the time spent on scale-up over the time spent on scale-out. This is the quantity we’re talking about.

The first consideration is that scale-up is 8x faster than scale-out generally. At a baseline, if the bandwidths were the same, we would have this 1/8, which is coming from bandwidth. But then we have some amount of expansion in how much data we’re sending. If one token comes in here, then this one token gets routed to, in the DeepSeek case maybe 32 experts or 16 experts. It gets routed to some number of experts. So this is the number of activated experts. This same thing applies on multiple different layers, so maybe I’m going to run two layers. There’s also multiple times the number of layers per stage.

Dwarkesh Patel

Don’t you need to multiply the whole thing by two for the all-to-all?

Reiner Pope

For the up and down. Yes, there’s a factor of two. Thank you.

What we would like is for the scale-up time to be greater than the scale-out time, because the scale-up time is the more important and precious resource. We would like this number to be greater than or equal to one. This really doesn’t seem hard. There’s just a factor of 8 that we need to overcome. So we need the product of these three things to be bigger than 8. Typically we have a fairly large number of activated experts. It could be 8 by itself. Then we can increase the number of layers per stage a lot until we satisfy this.

What this ends up looking like is that I can have an entire pipeline of racks where one rack does one layer, and then I move on to the next rack and do another layer, and then I move on to the next rack and do another layer.

Dwarkesh Patel

It’s interesting to me that the best parallelism strategy in practice ends up being one which physically resembles the actual architecture. It’s not some galaxy brain thing. It’s like, “Oh, we have experts, we’re going to put them on different GPUs, or we have different layers, we’re just going to put them on different racks.” I feel that’s interesting.

Reiner Pope

The cutting matches the model architecture.

Dwarkesh Patel

Exactly. It could have been something wackier with tensor parallelism and whatever.

Reiner Pope

The galaxy brain way to think of it is, what are all the different dimensions in which a model is scaled up? It is scaled up by layers, it is scaled up by the model dimension, it is scaled up by the DFF dimension, it is scaled up by the number of experts. Every single one of those numbers you can choose to cut along. If those numbers are big enough, it eventually becomes profitable to cut along there. We have selected two of them. The other two, in the way models are typically sized, are not profitable.

Dwarkesh Patel

So there’s a talk by Ilya where he says, “Today we know not to do pipeline parallelism.” And Horace He gave my friends and me… I hate that it sounds like a Dr. Seuss quote. But he gave us a lecture on these different kinds of parallelisms. He said the problem with pipeline parallelism is that, other than the bubbles, it creates these architectural constraints. Kimi, for example, has these residuals where attention attends to layers a few back, so it becomes hard to implement in this way.

Reiner Pope

I guess we didn’t fully articulate even what is the benefit that we’re getting from pipelining. These complexities are real. Pipelining is a massive hassle, but it does give you some benefits. You can then decide whether those benefits are worth the costs. It has some benefits in inference, maybe bigger benefits in training. In inference, what are we saving on? Are we saving on memory time or compute time? Not really. We’re just moving the memory time from one chip to another chip, or one rack to a different rack. There’s no actual benefit in runtime.

However, what we are saving on is memory capacity. If we think that the memory in a rack is a bottleneck, then there’s a constraint on how fast we can go. Pipelining allows us to massively reduce that bottleneck.

Dwarkesh Patel

The opposite connotation to this… Before this interview, I was chatting with Axel, who’s a GPU performance engineer at Jane Street. He was explaining that to do pipelining, you have to do micro-batches rather than full batches. If you do micro-batches, then you’re by definition not able to amortize loading the weights across all the users or all the sequences. The positive connotation of that is you don’t have to use as much memory. The negative connotation is that we can’t amortize loading the weights across all those users. Maybe it’s worth explaining why you have to do micro-batches.

Reiner Pope

Shall we draw the pipeline bubble? What is this micro-batching that shows up in pipeline parallelism?

I’ll focus on inference first. It’s a slightly simpler problem. I’m going to draw time, and then which rack we’re on. The idea is that maybe I’ll have four racks. I’ve got an inference that is going to step through these four racks in some time like this. This is inference number zero. It runs at a certain batch size and steps through all the pipeline stages like this.

Now, if we were to say, “Well, we’re going to run inference number one here,” this is clearly a massive waste. Like three-quarters of the time each of the racks is doing nothing. We don’t actually run inference one here, we run it as soon as we can, which is immediately after inference zero finishes. And then we keep going. If we hadn’t filled this in, we would call this the pipeline bubble. When I’ve drawn it in this inference context where we’re only going in a forwards pass, it’s obvious. Why would you do this stupid thing? In a training context, it’s maybe less obvious. But in the inference context, it’s really natural to make this change.

Dwarkesh Patel

Oh, interesting. This is sort of obvious, but the difference between micro-batch and batch doesn’t matter at all in inference because you can just call it whatever you want. It only matters in training because there is an optimal batch size.

Reiner Pope

Yes.

Dwarkesh Patel

Before you do a full backward step, you want to have accumulated all the sequences in that batch. If you want to do pipelining in training, in order to avoid that bubble, you need to—

Reiner Pope

Should we draw the training diagram with that? Let’s do that. This is the inference diagram, and I’ll call this forward so we don’t have the wrong thing showing up there. Let’s do the same thing for training now. We’ve got a forwards pass, but at some stage we’re going to have to transition to a backwards pass.

We’ll do some number of batches in the forwards pass, and then we’re going to transition to the backwards pass for everyone all in one go. The inference part is the same here, but then we do a hard stop at this point and transition everyone to the backwards pass, with similar numbering like this.

Dwarkesh Patel

It may be worth clarifying the reason there is that hard stop is because you want to do a whole batch at once for the backward step. And then there is an optimal size for how big that batch should be.

Reiner Pope

Smaller is always better, actually, is a way to put it. From an ML convergence rate perspective, smaller is always better because you’re getting the freshest information from the gradient descent.

Dwarkesh Patel

But from a total training time perspective?

Reiner Pope

From a total training time perspective, smaller is worse from a systems perspective. The optimum is the trade-off between those two.

So you pick a batch size, and for that batch size, you do some amount forwards and then some amount backwards. You asked why there is even a hard stop there. With pipeline parallelism, because you’ve got this idle time here which is the bubble, there are so many techniques in the literature for how to lay this out differently and avoid that. There are more complicated schemes called zero bubble or one-forward-one-backward, which interweave the forwards and the backwards in complicated ways.

Dwarkesh Patel

You can mine Bitcoin in that bubble.

Reiner Pope

Right. More usefully, you can do the weight gradient step, but you can also mine Bitcoin.

In inference, the effect of pipelining on anything you care about, like batch size or latency, is neutral. It doesn’t improve it, it doesn’t make it worse. If you look at the latency of this inference, running it if it were pipelined versus if it were all on one rack… If it were all on one rack, we would just slide all the boxes down and still put them in a row, and the latency would be the same.

Pipelining is neither better nor worse for latency. It does mean that you just use less memory capacity per rack. Because now instead of needing the whole model, you only need a quarter of the model, and you can expand.

Dwarkesh Patel

Makes a ton of sense. So it’s a no-brainer to use pipelining during inference, but there’s this harder trade-off during training.

Reiner Pope

Even in inference, in fact, it is not used a ton. It reduces your memory capacity requirements, but there’s actually a huge surplus. I think you were saying that a rack of Blackwell has many tens of terabytes. That’s much bigger than a trillion parameter model. A trillion parameter model only needs one terabyte, so it already fits. There’s not a huge benefit from pipelining because you’re reducing a number that’s already pretty small.

But it does say that theoretically, maybe you had too much memory there. You could have built different hardware that has less memory. If you were designing your hardware, you could say, “I didn’t need that much memory because I don’t need the weights to fit in one rack. I can fit the weights in eight racks, then I could have built hardware that didn’t have so much HBM per GPU.”

Dwarkesh Patel

Macro question: everybody’s talking about the memory wall right now. Memory is getting super expensive. There’s not enough memory. Smartphone volume will go down 30% because there’s not enough memory. This is shocking, Dylan said hyperscalers are spending 50% of their CapEx this year on memory.

Reiner Pope

That’s believable.

Dwarkesh Patel

What is hyperscaler CapEx? That’s high hundreds of billions, maybe a trillion, and they’re spending half of that on memory? That is a huge constraint. That’s why we’re not going to get new laptops and phones this year.

But at the same time, we have too much memory? People are willing to put too much memory into these systems. Why is Jensen shoving all this memory into these racks if you don’t need it?

Reiner Pope

In the equations we had here before we erased them, we were doing memory time, memory bandwidth and compute bandwidth. Let’s now start looking at memory capacity.

We’ll start off with memory capacity without even thinking about a parallelism scheme. The demand on memory is the number of total parameters. This is what we need to fit the weights in some system that we are using. Then we need to fit the KVs as well. KVs go as batch size times the length of the context times the bytes per token.

What I was arguing about in this context, and the case I was making for pipelining, is that there are some techniques that allow us to solve this. Let’s consider running this on some number of GPUs. We’re going to have one extent, which is E, the expert parallelism. When we had this sharding of an expert layer across many GPUs, to what extent do we do that? How many GPUs? We’re going to say that this is, for example 64. Then P is going to be the extent of pipelining. This is the number of racks, maybe we’ll pick 4 or something like that.

This is the total memory requirement across the system, but now I’m going to calculate a memory requirement per GPU. I’ll use a lowercase cmem. Obviously, we just take all of these numbers and divide it by E and P. Really easy. It’s this Ntotal, plus the batch times length of context times bytes per token, all divided by E times P.

Why is this correct as divided this way? We knew that the parameters were perfectly divided amongst all the GPUs in a rack. The layers are perfectly divided amongst the different racks. So that works here. Somehow we’re going to arrange—I’ll hand-wave exactly how—the same perfect sharding of the contexts across GPUs in a rack, and then based on layer across racks.

Dwarkesh Patel

Sorry, 4 is the number of racks?

Reiner Pope

Yeah, for example.

This is the place where we actually need to go back and analyze this batch size B. You were making this comment that there’s micro-batching versus global batching. Let’s come back to this pipelining diagram here.

We’ve got one batch going forward here, and then as I drew it, it kind of just disappeared. That’s not really correct. If you think about how decode is working, I have a bunch of tokens that I have generated already. I do one forwards pass where I generate a new token, and then I write that to my KV cache. Then I do another forwards pass that generates the next token. I’m actually going to be running this batch zero in a loop. In fact, I go forwards. Once I finish, I can start the next iteration of the loop up here. We’ll just fill this in. We’ve got the two, three, two and three, and two and three.

Let’s split this batch. This batch will be the global batch size. B is going to be the number of micro-batches times the batch size per micro-batch. How many micro-batches do we need? The number of micro-batches in this diagram is 4: zero, one, two, three. The micro-batch size is still this 2000-ish number. Sorry, no, this is the 300 times sparsity.

Dwarkesh Patel

This is how big the train that takes off every 20 milliseconds is.

Reiner Pope

Right. This is going to be the 20-millisecond train. The global batch size is the number of micro-batches times the local batch size. Local batch size is set by this hardware parameter.

The number of micro-batches is as small as possible, such that we can wrap around and not leave any idle time. If we had fewer, we would have this idle time when we wrap around. You can visually see that it is equal to the number of pipeline stages. It’s a proof by visual here. It is 4, and it’s 4 this way as well. You can look and see that it goes along here, and then it wraps around to the number of pipeline stages.

Dwarkesh Patel

Sorry, very basic question. Is this what is actually done? A frontier model today will have pipelining during inference?

Reiner Pope

For sure during massive scale training this is done. It can be done for inference. I’m actually going to make the case for why it is less attractive. It is useful for weights, but not so useful for KVs.

The big challenge is... Let’s fill this in. The micro-batch size here ends up being equal to the number of pipeline stages. When we go back and substitute all of that into here, we get a number of pipeline stages times this little b showing up in here. When we factor this out, I’m going to split this plus into two terms.

We get the full division by E times P over here. We still have division by E times P over here, but the Ps cancel. What we find is that if you increase the number of pipeline stages, the memory footprint for the number of weights keeps going down and down and down, but the memory footprint for the number of activations stays constant. So it doesn’t actually work.

Most of your memory… Once you do enough pipelining—and it’s really not much, even two is often enough—this term becomes very small. The KV cache becomes the dominant term.

Dwarkesh Patel

I know this is wrong. I’m just trying to think about why my train of logic here is wrong. If you’re pipelining through many different stages, the KV values are not shared between layers. Why would it not help to be pipelining across multiple layers? Because then you don’t have to store...

Reiner Pope

You only need to store one layer rather than two layers of KVs. It helps from that perspective, you’re right. What’s competing with that, though, is that you need to be keeping all of the racks usefully busy at a time, so the number of sequences that are in flight simultaneously has gone up.

Dwarkesh Patel

Ah, that makes sense.

Reiner Pope

Those exactly cancel, and you end up not getting a saving per GPU.

Dwarkesh Patel

Right. This is going back fundamentally to the point of how you’re not able to amortize across KV caches.

Reiner Pope

First, we established you can’t amortize KV caches across batch size. Now we’re saying you also can’t shard it across pipeline stages. It sucks from both of those points of view.

Dwarkesh Patel

Interesting. So then what is done during inference?

Reiner Pope

The DeepSeek paper reports what they do, which is that they just do a lot of expert parallelism. In effect, you should increase your expert parallelism up to your scale-up domain size, and then do very little pipelining. Maybe none at all, maybe two, just enough to make the weight storage not too big of an issue.

Those are the only two parallelisms that really make sense. In the past, there was tensor parallelism, which was cutting up within an expert, but the experts are so small now that that is not a profitable optimization.

Dwarkesh Patel

Does that mean that frontier labs, when they’re doing inference, are just within a single scale-up?

Reiner Pope

Yes. You can look at how it depends on model size. You could have a very large model, one that exceeds the memory of a rack. There you should be doing a bit of pipelining. Maybe it’s extremely sparse, for example, and that would be a reason to do it.

Dwarkesh Patel

This goes back to the promise at the beginning of the lecture, which was this will actually tell you about AI progress as well. To the extent it is the case that model size scaling has been slow until recently…

Let me make sure I understand the claim. The claim would not be you could have trained across more racks. It was just that it would not have made sense before, we didn’t have the ability to do inference for a bigger model easily.

Reiner Pope

Actually, pipelining doesn’t help with context length. It totally helps with model size. Because of the ability to do pipelining, a rack at least should not be a constraint on your ability to fit the model parameters.

The other consideration you’re asking is, why hasn’t it scaled up more, and why did bigger scale-up domains help? We talked through one aspect of that, which is that it’s not because of memory capacity. We have a solution to the memory capacity at least with respect to model size, not with respect to KV cache size but at least with respect to model size. The other issue that shows up is latency.

Dwarkesh Patel

I was just about to ask, going from rack to rack, what is the latency cost per hop?

Reiner Pope

This is very much dependent on the hardware. I can’t say with a lot of authority. I think it’s probably on the order of a few milliseconds, but it could be off by an order of magnitude there.

Dwarkesh Patel

Is 4 a realistic number of how many pipelining stages you might have?

Reiner Pope

Yes.

Dwarkesh Patel

So that’s not that much.

Reiner Pope

On a small number of pipelining stages, this is not a huge latency impact.

Dwarkesh Patel

But I guess it’s 10 milliseconds per token.

Reiner Pope

That’s right.

Dwarkesh Patel

2 times 4-ish, or I don’t know how many you said… 10 milliseconds per token is actually a lot.

Reiner Pope

If it goes from 20 to 30, or something like that…

Just to chart the path that it goes through, here you’re going from your GPU or TPU to a network card, which then goes to a top-of-rack switch, and then hops over to the other rack and does the same thing in reverse. You have to sum up the latencies of these different things.

Dwarkesh Patel

Sorry, is this the same thing as the data center switch?

Reiner Pope

It may in fact go up to a data center switch and back. It depends on deployment configuration.

Dwarkesh Patel

Got it. And because it’s decode and sequential, they stack up across the stages. You can’t do them at the same time.

Reiner Pope

That’s right.

Dwarkesh Patel

This brings us back to the question then, is the size of the scale-up at all relevant to why AI model sizes have been what they have been over the last few years, whether through training or through inference?

Reiner Pope

We talked about latency of the hop. There is also just the tmem latency. The memory time latency is actually massively improved by larger scale-up domains. I’ll recall tmem down here. tmem for the weights was equal to the number of total parameters divided by the memory bandwidth. Which memory bandwidth are we talking about here? Is it just one GPU? It is the number of GPUs that I can use in parallel to load these weights. I can’t use different pipeline stages in parallel because they’re not running at the same time, but I can use all the GPUs in my scale-up domain in parallel to load the weights. This is actually extremely effective. Basically, I end up with a term here, this memory bandwidth term itself is equal to scale-up size...

Dwarkesh Patel

Times memory bandwidth per GPU.

Reiner Pope

Yeah. Times GPU bandwidth. This term doesn’t increase a lot. It maybe increases 1.5 or 2x per generation, but this one increased by a factor of 8 from Hopper.

Dwarkesh Patel

So the reason the bigger scale-up matters, it’s not the memory capacity of the whole scale-up, but really the memory bandwidth.

Reiner Pope

Yeah. Pipelining totally solves the capacity problem, but scale-up size helps solve the bandwidth problem.

Dwarkesh Patel

And the bandwidth problem helps you do longer context lengths, which is more and more relevant as these models get more agentic.

Reiner Pope

It lets you just run the model at lower latency as a first thing. If I just do a very sparse model and it’s on a little H100 box, the latency will be really high.

Dwarkesh Patel

A super tangential question. There’s Chinchilla scaling, which tells you how big a model should be relative to the amount of data you’re going to train it on. But now, obviously, you’re not just trying to optimize for the highest quality model you could get with training compute. You want the best results a user can get with a mixture of training and inference compute.

So there’s a question of how much you should over-train a model such that compute amortized over training and inference is minimized to get a certain performance. But now with RL, there’s another consideration which is, you’re going to do some amount of pre-training. That pre-training will be used both for RL generation and then for inference for the final user. By over-training here I mean that while it would have been more efficient just from a training compute perspective to have a bigger model that you train for less time because it can learn faster, maybe you get a smaller model, spend more compute training it than you otherwise would have, but now it’s cheaper to give it to users.

Let me make the question more concrete. Basically, how much more than Chinchilla optimal are models over-trained? And has that changed as a result of RL generation?

Reiner Pope

This is a place where we have to do a bit of guesswork because the updated scaling laws and the model traffic are not reported, so we have to guess there. One way to look at it…

Let me first just make a general heuristic claim. If I have some cost, and I’ve got a total cost which is a sum of cost A and cost B, like maybe this is the training cost and this is the inference cost, and I want to minimize this sum…

For many curves, the minimum tends to be where the costs are equalized. That’s something of a heuristic claim, but there are many examples where it’s true. Where one is 1/x and the other one is x, for example, they tend to be minimized at the point where they equal each other. It’s also true for ex and e-x and all kinds of other things. Basically, I’ve got some curve that’s going down, some other curve that’s going up, and they tend to be minimized at this equal point.

Heuristically, I will conjecture that that is true for the setup you described as well. Actually showing that would be true would require looking at the scaling laws and fitting these weird exponents, but things that follow power laws tend to have this property. So I’ll just make that claim and move on.

We’re going to say that we want to equalize the cost of training and the cost of inference. We can do all of it in general. The cost of pre-training, that’s the number of active params times the data on pre-training. There’s a factor of 6 out here, which is the number of FLOPs. There’s the famous 6ND formula. Then in RL, we have approximately the same thing. We’ve got the same number of active parameters, but now the amount of data is the RL data. There is this extra efficiency multiplier, or inefficiency...

Dwarkesh Patel

Which is the fact that you’re not training on all your rollouts.

Reiner Pope

Well, there’s that, and then the other, perhaps even bigger inefficiency is that this involves a substantial amount of decode. Often decode runs at less MFU than training.

Dwarkesh Patel

Okay. So if you’re doing a backward pass on every single generation in RL, it would be 6ND.

Reiner Pope

So this could be a smaller number, right?

Dwarkesh Patel

It would at least be two, because that’s the lower...

Reiner Pope

Somewhere in the range of two to six. We’ll say somewhere in the range of two to six and leave it at that. Then we can add in the inference cost. The inference cost is two, the number of active parameters times the data in inference.

Dwarkesh Patel

Sorry, I think the way I said it was super garbled. Just for the audience, forward plus backwards per parameter is 6. Forward alone is 2. That’s why RL, where you’re definitely going to generate all the trajectories but you might or might not train all the trajectories, is 2 to 6.

Reiner Pope

Yes. Thank you. And then inference is just 2. We’re going to solve for essentially equality of all three of these terms. That is the ballpark of where people are going to be. Labs have more information on what is productive in doing more RL, for example, versus doing more pre-training. I don’t have that information, but I think a good ballpark is a 33% split between each of them.

Dwarkesh Patel

I’m not sure I understand the intuition for that. Another naive model could have been that RL plus pre-training would be 50% and inference would be 50%.

Reiner Pope

That’s also a valid answer. Because this is heuristic, I can’t really argue for one versus the other. They don’t differ by that much. Thirty-three versus twenty-five is only a small factor off.

Let’s pick one of them. All equal seems simple enough, so we’re just going to solve for equality of them. It’s pretty straightforward. We can immediately see that the number of activated parameters totally disappears, so let’s factor that out. We’re going to just say that data in pre-training—I decided to do it your way, it’s a little bit nicer—plus... Oh, I didn’t have the inefficiency over here either. Data in pre-training plus some multiple of α times the data in RL is going to end up equal to some β times the data in inference.

Let’s just roughly size the α. This α is maybe somewhere in the range of 2 to 6. Over 6, from this term compared to this term. And then we’ve got an inefficiency term, which I would say is maybe in the range of 30%. So this alpha is going to be something like 1/10. And this β here is actually the same. It’s a third. It’s one third times 30%. So it also equals 1/10.

Dwarkesh Patel

If both of them are one in ten, that kind of implies that there’s never a backward pass on RL?

Reiner Pope

Yeah. Okay, we can make this 2/10. Make it a bit bigger. Just write it out once more, this is 2/10, this is 1/10.

The number of inference tokens you have is just a function of hundreds of millions of tokens per second times my model is deployed for two months before I ship to the next version. That should determine the number of tokens in RL and pre-training.

I guess we didn’t do the equivalence between pre-training and RL, so we’ll do that here. Data in pre-training should be equal to 2/10 data in RL for them to be cost equivalent. Sorry, 1/10. I got it backwards. We pay more cost when it’s inefficient, so this needs to be 1/10. Tracing this back… This thing ends up actually being, as written here… This is like 1.5, and this is one.

Dwarkesh Patel

Billions of dollars worth of compute just flowed in the other direction.

Reiner Pope

Right? I think if you do it with a spreadsheet and actually model it out, you might notice when the money’s going down the drain. All of these end up being close in, as modeled here. This 30% may have been a little bit too generous. So let’s say something like 1.5 here, and leave this as a one here.

I think at this point, you can almost read it off. The number of inference tokens should be about the same as the number of pre-training tokens, which should be about the same as the number of RL tokens, within factors that we’re not able to reason about.

Dwarkesh Patel

Sorry for making a basic algebra mistake. It seems like there should be fewer RL tokens than pre-training tokens?

Reiner Pope

That’s in general right. Because RL is less efficient in terms of machine time, if you’re trying to equalize the RL and pre-training time, then you should have fewer tokens in order to have the same wall time.

Dwarkesh Patel

This is all quite interesting. I never thought about it in terms of equalizing data.

Reiner Pope

I think starting with equalizing in cost is right, but depending on how you model the cost, this comes close to equalizing in data.

Dwarkesh Patel

So for GPT to be trained optimally, every single user who uses GPT-5, the total amount of tokens that they stream should equal the total amount that has gone into pre-training. And the total amount of tokens that have gone into pre-training is the sum of all human knowledge. Each model should generate the sum of human knowledge on the output that it gets on the input.

Reiner Pope

Yeah. Which way are people going to err? If you think that people’s power of prediction is not perfect, and also you run the risk that you make a model that is not a frontier model and then you just throw it away, then that changes the cost trade-off because there’s some probability that applies to the inference. And you should derate the inference tokens by some amount.

Dwarkesh Patel

Right. Can we back out how much more compute than Chinchilla optimal for a given sized model?

Reiner Pope

I think we just have to make some real-world assumptions here in order to do that.

The inference tokens, we should totally be able to count, right? Let’s say a few hundred million. Maybe it’s five hundred million tokens a second now, I don’t really know. Five hundred million tokens a second times. A model is deployed for two months before it becomes obsolete?

I can’t do this in my head. Can you type it into a computer?

Dwarkesh Patel

2.6 x 1015.

Reiner Pope

Okay. 2.6 x 1015. This number is probably too large because this is going to be multiple models in a family. Let’s make it 5x smaller or 10x smaller or something like that. So we’re estimating maybe fifty million tokens per second, per specific model. The model is live for two months. This comes out to around two hundred trillion tokens. And then we want to compare that to active parameters on a frontier model. I don’t actually know the latest rumors. Do you know?

Dwarkesh Patel

Somebody told me a hundred and fifty trillion.

Reiner Pope

Active parameters?

Dwarkesh Patel

Sorry, I meant tokens.

Reiner Pope

Trained on a hundred and fifty trillion tokens. Interesting.

Dwarkesh Patel

Which is similar.

Reiner Pope

That’s actually similar. So data on pre-training.

Dwarkesh Patel

This is not well-cited but it’s fine.

Reiner Pope

I think often the number of active parameters could be in the range of a hundred billion, something like that. Maybe a bit larger. So multiply by 20 to get the Chinchilla token count. So Chinchilla, DChinchilla, would be around two trillion. We see we’re about a hundred times larger than that.

Dwarkesh Patel

What does DChinchilla actually mean?

Reiner Pope

The token count for pre-training that the Chinchilla scaling law would recommend, I guess.

Dwarkesh Patel

Oh, I see. So how much is it over-trained? Got it.

Reiner Pope

The ratio of this two hundred trillion or a hundred trillion parameters over the Chinchilla optimal of two trillion, that’s the amount it’s over-trained. Which is a factor of a hundred over-trained.

Dwarkesh Patel

A hundred. So if you consider this right here, to the extent this is in the right ballpark, just by thinking about how you want everything to be equal in terms of compute… If OpenAI also realizes that and they’re serving a certain amount of tokens per second, that tells you how much data went into the pre-training of GPT-5. Even if it’s 50% off or something, it is wild that you can first-principles these kinds of numbers.

Reiner Pope

This is why you should just approximate everywhere, because there are big error bars on this. But it’s kind of empowering to just set A equal to B and figure it out.

Dwarkesh Patel

That’s super cool. Okay, so in the spirit of trying to deduce things, we can publicly look up the API prices of these models, and maybe we can learn something from that.

First, with longer context, Gemini 3.1 is 50% more expensive if you go over 200k tokens than if you’re below 200k tokens. At a high level, I understand why that might be, but why specifically 50%?

Reiner Pope

Why specifically 50%? The high level, even in the first place, is that there is some amount of increasing cost with context length. We can bring that back up. That was the memory time versus the compute time.

We’ve put up these same equations from before, of the time for memory fetches which is the weights and the KV cache, and then the time for the compute which is just the matrix multiplications for the weights. I will also draw the cost curve, but this time I’ll do it as a function of context length instead of batch size. So this is the cost curve as a function of context length. We’ll draw the compute. The cost of the compute is actually constant as a function of context length. There’s no dependence here on context length. In reality, there is some dependence, but it is very mild, so we’ll ignore it. So this is the time for the compute.

Then we’ll also draw the dependence of the memory fetch on context length. This starts at a large number for the weights and then grows gradually with the context length. Maybe starting here, and then grow gradually with context length. And so, you take the maximum and you see there is this inflection point here.

So this is the cost that Gemini might be paying. And then you think, how might you put a pricing structure on top of that? You would like to ensure that no matter what the context length is, you are still profitable. So we’ve got a two-tier pricing structure. Maybe we’ve got something that looks like this up to some extent.

I think it says something about, given that the bump is at 200k, it probably means that this is somewhat aligned with this crossover point. Maybe not exactly aligned with it. We can actually probably even complete that calculation just to see where it lands out.

We can solve for the number of bytes per token if we make some assumptions about the number of active parameters. So solving for the number of bytes per token, we’re going to assume the point where we equalize the time of memory and the time of compute is at, let’s say, 200k tokens. So we equalize these two.

We’re also going to assume that the batch size is large enough that the memory time spent on weights is negligible. So we’ll forget about this, and we’ll focus on the actual memory time spent on KV cache. That ends up saying, copying this term over, batch times length of context times bytes per token over memory bandwidth is going to be equal to the number of activated params over FLOPs. And then we’re going to solve for bytes per token. Batch size was missing here. It shows up here, and then it cancels out by the time we get to here. And I dropped the length of context.

So we can plug in numbers. This is the reciprocal of the number that we saw before. This is 1/300, which is reasonably stable across many different hardware platforms. We conjecturally said that maybe the number of activated parameters is a hundred billion. The length of the context we said was 200k. Something is wrong here, though. Length of the context should be on the denominator, not the numerator.

Dwarkesh Patel

1667. Almost two kilobytes.

Reiner Pope

That is plausible, actually. You said around two kilobytes. Let’s just do a sanity check for what this could be. There are two mechanisms that people do attention with a small number of bytes per token. One is dense attention with a lot of reuse across layers. Character AI has a blog post talking about that, alternating long and short context. In the Character AI kind of model, which also showed up in the Gemma models, the global context—which is really what we’re talking about here—was shared across all the layers.

To get this to kilobytes, you could get that, for example, as a dhead of 128, which is typical. Then the number of bytes is typically the number of attention layers times two times dhead times the number of KV heads. This is the number of unique contexts per layer.

Do you share the context across many layers, or do you use it only once? In the Character AI-like models, this number is one. We said this is 128. This is a choice which typically ranges from one... Sorry, this is KV heads, I meant.

Dwarkesh Patel

The difference between a head and a KV head is that…?

Reiner Pope

The KV heads are the heads that are stored in memory, store the contents of the previous tokens. The Q heads are the retrieval heads. They’re only used temporarily and they’re used by the attending token. In this autoregressive context, I’ve got KV heads associated with all of the contexts, and then Q heads associated with this new token here.

Dwarkesh Patel

But this head, the 128.

Reiner Pope

Oh, sorry. This d-head is the dimension of the vector. The number of KV heads is typically in the range of 1 to 8. It is totally plausible to get this by, for example, having 8 KV heads and a d-head of 128. That gives you exactly this number. Or you could have fewer KV heads, but more layers.

This is one way to get there via dense attention. There’s also a way to get there via sparse attention, where you increase all of these numbers, but then you have a 1/sparsity term. I think this number is plausible, if maybe a little bit small.

Dwarkesh Patel

It’s funny that they would leak so much information through their API pricing.

Reiner Pope

I mean, you are incentivized to price close to your costs because otherwise someone could scoop you.

Dwarkesh Patel

Maybe we can learn something about the difference in input versus output prices, and what that tells us about decode versus prefill in these models. I think last I checked it’s 50% more expensive or something like that?

Reiner Pope

I don’t remember. What I’ve seen in the past is 3-5x more expensive.

Dwarkesh Patel

Okay, that makes more sense. So let’s say it’s 5x more expensive. This is the compute to process the next token in decode. Suppose you’re doing prefill, where you’re not just processing the most recent token, you’re processing all the tokens in parallel. I want to say that it would be this times length prefill?

Reiner Pope

Or length of the pass in general. If we can think of decode as being a pass with one, and then prefill being a pass with many.

Dwarkesh Patel

Okay. So maybe prefix? Okay, memory. You’re not storing the KV cache for the tokens that are the prefill tokens.

Reiner Pope

Let’s actually draw how prefill shows up here, if I may clarify. We do a bit of decode like this. We may actually come back and do more prefill. If you think this is a chat session, the user says something, the AI generates a response, and then the user says something else and we prefill this. Maybe this is the general case, rather than this.

Dwarkesh Patel

In fact, this is like you read a file or something.

Reiner Pope

Read a file or the AI is responding to a user input, tool call, or anything that’s not AI-generated.

Dwarkesh Patel

Okay, suppose we’re here. You will have calculated all of this previously. So just the KV of everything that came before. But what is the memory cost of this? Well, the memory bandwidth cost of this. If you’re doing flash attention, it would—

Reiner Pope

It’s basically temporary. It doesn’t even go to main memory. Just ignore that.

Dwarkesh Patel

Exactly. So then it would just be everything that came before. Is it not just that then?

Reiner Pope

There’s actually no adjustment at all to the memory time.

Dwarkesh Patel

Okay. Great. So it’s a very trivial change to accommodate. This term is making it 5x more expensive. Now, why would that be? What does that actually tell us? What variable does this help us clamp? The only thing that could have changed is that the compute is 5x more expensive as a result.

Reiner Pope

This is the time for one pass, but actually the amount of tokens is that much larger. We want the cost per token, in fact, or the time per token.

Dwarkesh Patel

I’m not sure I understood. This is for processing the next token in prefix?

Reiner Pope

Well, actually for processing the entire batch. At this cost, we have processed this many tokens, the length of prefill. Or I guess the length of the pass. Not this prefix, but it’s this cost.

Dwarkesh Patel

Okay. Let’s just do this pass. So this is 5x more expensive. Input is 5x more expensive.

Reiner Pope

Output is more expensive, in fact.

Dwarkesh Patel

Output is 5x more expensive.

Reiner Pope

The result we want to work towards is that prefill is compute-limited and decode is memory bandwidth-limited.

Dwarkesh Patel

Why don’t we do this? Why don’t we just chart it with len-pass on the X-axis and t on the Y-axis.

Reiner Pope

We want the cost per token, so it’ll be t over length of the pass. That’ll be right.

Dwarkesh Patel

I guess I’m getting confused by this. Len-pass is... It seems like this should be higher when you’re doing prefill.

Reiner Pope

Prefill has a bigger length pass. Yeah.

Dwarkesh Patel

But then why is it cheaper?

Reiner Pope

Why is the cost higher? It’s this division by length pass. This is going to divide out, but then all of this is going to divide by length of pass, and it’s going to make the memory costs cheaper.

Dwarkesh Patel

Okay. Let me think about this then. Basically we’ll have four different lines. Let’s do prefill first... Actually, let’s do decode first.

Reiner Pope

Length of the pass, when it’s one, that is decode. When it is bigger, that is prefill.

Dwarkesh Patel

Oh, okay. I see. That makes sense. Getting back to it. So tcompute, if you have basically just this divided by len-pass, so just this amount. This actually does not vary based on t, so it’ll just be some flat value like this. And this is tcompute. And this is—

Reiner Pope

That’s decode.

Dwarkesh Patel

Decode. Right. Now tmem, we have this whole thing divided by len-pass. Well, it doesn’t really matter what’s up there, it’ll just be something that looks like this. Let’s say this is tmem. This is decode again.

So as the length of the prefix goes up, or pass, your memory bandwidth time declines, and that means that to the extent that you were bottlenecked on memory bandwidth before, you can avoid being bottlenecked on memory bandwidth.

The fact that they are charging 5x less for prefill than decode does suggest that they are bottlenecked on memory bandwidth to quite a degree, such that for them at least—because t is equivalent to cost, it’s the cost of renting a compute—this would be at 1, and this would be at 5.

Reiner Pope

That’s right.

Dwarkesh Patel

So it is, in fact, tremendously memory bandwidth bottlenecked. The real graph looks something like that.

Reiner Pope

It still crosses, but yeah.

Dwarkesh Patel

Exactly. Let me do it this way. This is the gap on decode between the memory and the compute time. Okay, interesting.

Another interesting one would be why cache hits are so much cheaper. If I remember correctly, cache hits are like 10x… It’s more expensive to write to cache according to the pricing on all these models. But if you do hit a cache, it’s 10x. Presumably, this is the cost of keeping something in HBM rather than just evacuating it. But if you do keep it in HBM, then it’s cheaper to load again?

Reiner Pope

Right. There are two ways you can produce the KV cache for a token. You can just produce it from scratch by computing it from the underlying token IDs, which are tiny. Or you can previously have produced it and stored it in a memory somewhere.

The cost ratio is really talking about the ratio between those two mechanisms of producing it. A cache miss means you’ve deleted it from all your memories, and you have to recompute it from the tokens directly. You can even take that a step further and think about which memory tier you store it in. You could store it in HBM. There are other slower and cheaper memories than HBM, like DDR on your host or flash as well. One of the things you can do is a calculation of where it makes sense to be in each memory tier, and this is related to how long you’re going to store it for.

We want to look at the cost of storage in a few different memory tiers and also the cost of rematerialization. Remat means the cost to rebuild all of the KV cache from scratch after you deleted it, so we rematerialize it. Basically, this is going to cost the length of the context. Actually, we’ll look at the cost per token, so we don’t need to carry around this length of context everywhere.

To rematerialize one token of KV cache, I just need to run a forward pass on the whole model. This is going to be the compute time. I have to rerun the compute at whatever speed my GPU does it, and then I multiply it by my GPU dollars per second.

Dwarkesh Patel

Sorry, excuse a naive question. Why is there not a quadratic term?

Reiner Pope

There is a quadratic term. It shows up in the compute. As an approximation, I chose to remove it. I’ll just show you quickly what that looks like. If you look at the cost per token, or the number of FLOPs per token, there are the FLOPs that are coming from doing the weight matrix multiplies as a function of—

Dwarkesh Patel

Which is flat.

Reiner Pope

...context length. And then there is the number of multiplies that comes from doing the KV cache, which goes up linearly with the amount of stuff you attend to. The slope on this is so low that when you draw it like this, it’s very well approximated by a flat line. You start to notice the effect of the quadratic or the linear term up in the millions of tokens or so. So it’s just not super relevant.

Dwarkesh Patel

So what is the reason that there’s no company which has over a million token context length, if this is true?

Reiner Pope

There are two costs of long context. One is the memory bandwidth cost, which we’ve spent a lot of time analyzing. That’s this thing. The other one is the compute cost. The compute cost is almost always forced by fundamental principles to be a much smaller slope than the memory bandwidth cost. The primary things that limit you to really large contexts are memory bandwidth and memory capacity, which is exactly this effect.

Dwarkesh Patel

There’s this idea that Dario said on the podcast, and others have said, which is, “We don’t need continual learning for AGI, in-context learning is enough.” If you believe that, then you have to think that we have to get to a hundred-million-token context length to have an employee that is the equivalent of working with you for a month. Now, maybe that’s no longer true with sparse attention or something. But if you think that, then some ML infra thing would have to change to allow for a hundred million, like the memory bandwidth, to allow for a hundred-million-token context lengths.

Reiner Pope

Sparse attention gives you a get-out for sure, because you get this square root. It gives you a big improvement. But if you look at the history of context lengths of models, from earlier models like GPT-3, maybe to GPT-4—I don’t remember when the transition happened exactly—they shot up from about 8K to 100-200K. And then for the last year or two, they’ve all been hovering around there. I think that indicates that this is the reasonably balanced cost point, and going massively beyond that would be cost-prohibitive.

Dwarkesh Patel

Not because of the compute cost, because of the memory bandwidth...

Reiner Pope

Because of memory bandwidth cost, yeah. I actually don’t see a very good path to solving that. The HBM is where it is. It’s not getting hugely better.

Dwarkesh Patel

And why doesn’t sparse attention solve it?

Reiner Pope

Sparse attention is a big improvement. Maybe that is priced in already, perhaps. It’s not an infinite improvement because if you go too sparse, you lose too much quality.

The empirical result is that the context lengths haven’t been increasing that much. I think it’s because there is no solution to the memory wall here. Going too sparse just means you’re attending to a very small subset of the tokens, and the quality will get worse.

Dwarkesh Patel

Makes sense.

Reiner Pope

What is the cost of these different ways of resynthesizing the KV cache? Computing it from scratch is based on my GPU time. I have to do a certain amount of multiplies, of GPU time that I spend in order to produce it.

Storing in HBM. This really goes as my bytes per token. I need to just have some number of bytes per token, and then I need to store this in the HBM. It’s going to use up some of my HBM capacity. A way to think of this is that if I have too many of these things sitting in my HBM, if I fill up my HBM with just KV caches that I’m not using, I can’t use that GPU.

How do I price that? Maybe I say that the cost of it is proportional to the fraction of the HBM I’m using. There’s also times GPU dollars. Let’s just do one more memory tier and say store in DDR instead. The same kind of thing goes up for flash and for DDR.

I put these in the wrong columns. I meant to make two columns. The distinction I want to make is that there is the cost to retrieve, and then there’s a cost to hold on. This is a cost per second, whereas this is an instantaneous cost. Rematerialization has a cost to retrieve and has zero cost to store it because we’ve deleted it. This is the one that I put in the wrong location. This is actually the cost just to hold on, so I will rewrite it.

If we’re just storing it in HBM, it has this sort of cost profile. If we store in DDR, it’s actually going to take some time. We get the same thing here: bytes per token over DDR capacity times DDR cost per second. But now this has a cost to retrieve that is higher than the HBM because we need to copy it into the HBM. So this is bytes per token over DDR bandwidth. And then this consumes some amount of the DDR as well.

Dwarkesh Patel

And every scale-up has DDR and flash?

Reiner Pope

This is really a deployment question, so you can choose that. Nvidia does deploy in this form. It has both.

Dwarkesh Patel

Why isn’t the cost to retrieve HBM the bytes divided by memory bandwidth?

Reiner Pope

It depends what you define a retrieve to be. Here, I’m defining retrieve to be, move it into HBM so that you can start actually doing inference on it.

Dwarkesh Patel

Because if it’s already in HBM, you can be doing compute while you’re getting it from HBM to SRAM? Interesting.

Reiner Pope

Yeah, for example. These are three things, and I guess I ordered them wrong. In general, if you’re balancing two costs and you’ve got different tiers in the memory hierarchy, you should expect as this cost goes up, this cost should go down. You can kind of see where the zeros are. I should have ordered them this one first, this one second, and this one third.

If you’re going to hold onto it for a very short amount of time, then all of this is multiplied by the hold time. This one is, and so is this one.

Dwarkesh Patel

Interestingly, they have different prices to write for. Do you specify this in the API for five minutes versus an hour? Which suggests that the five minutes is HBM and the hour is DDR.

Reiner Pope

I think that’s a pretty good assumption. If you look at the numbers, it might also turn out that it’s one tier down, and it’s DDR versus flash.

Dwarkesh Patel

Interesting. I’ll look up the price difference. The base input tokens is $5 per million tokens.

Reiner Pope

Base, which means remat. This is $5.

Dwarkesh Patel

That’s $5 to “retrieve”. And then to write, presumably HBM, for five minutes is 6.25.

Reiner Pope

We might be able to determine which memory tier it is by the durations.

Dwarkesh Patel

Five minutes versus one hour.

Reiner Pope

Exactly. I think this will probably end up being the drain time of the memory tier that you’re in. What that means is, given that I know I’m going to be holding something for five minutes, I would like to pick a memory that I can read every five minutes. I can read the whole memory once per five minutes, ballpark. That is the drain time of the memory. So if I take the storage capacity over storage bandwidth, I would like this to be equal to five minutes.

We did this calculation for HBM. For HBM, we know that this number is 20 milliseconds. So HBM is much too small. DDR could be about an order of magnitude or two off from this, so this is probably on the order of seconds, like 1 to 10 seconds. I don’t have these numbers memorized, but generally, as you go to slower tiers, flash is plausibly on the order of one minute. And then spinning disk, which is massively different, is on the order of one hour. So this might actually identify the tiers of flash and spinning disk.

Dwarkesh Patel

Sorry, why is this the calculation? This is the storage capacity divided by the bandwidth?

Reiner Pope

You’ve got a bunch of different memory tiers, we’ve listed four of them. Your choice of which memory tier is about minimizing the cost. What fraction of the device are you using? You’re using some fraction of the device for holding onto it, and then you’re using some fraction of the device to retrieve it. Let’s say I’m using 10% of the device. And I want to equalize those two fractions. That’s a sign that I’ve hit the right thing.

Let’s say I’ve got some runtime here. I’m going to hold on for all of this time, so this is the time-hold. And then there’s going to be some amount of time here, which is time-retrieve. Basically to equalize these two costs, I want the retrieval time to be equal to the hold time times the fraction of capacity. Because this is the retrieval time, this is how many other things I can hold simultaneously.

Dwarkesh Patel

Basically, you want to store things in there for so long such that the amount of time it’s in there is the time to get all your things in there and out.

Reiner Pope

Yeah basically. I think that probably indicates that the two tiers are flash and spinning disk. I’m kind of shocked to see spinning disk being used at all, because it’s such an old technology.

Dwarkesh Patel

Interesting. It’s also crazy that it’s so slow that it takes an hour to load its full capacity to it in.

Reiner Pope

It’s a really unattractive technology but it’s useful in some places.

Dwarkesh Patel

We’re sitting down because I want to ask you some questions that don’t need a blackboard. You have this extremely interesting blog post where you talk about how, at a high level, the architecture of different cryptographic protocols looks a lot like neural networks. There’s this convergent evolution where they both need to jumble information across all their inputs. For cryptographic protocols, it’s to make sure that each new input into a hash function will totally scramble what happens. For neural networks, of course, they need to consider how this piece of information changes what you should make of this other piece of information.

I thought that was an extremely interesting point. At a high level, in some sense they’re trying to do the inverse thing. Cryptographic protocols are trying to take information which has structure and make it look indistinguishable from randomness. Neural networks are trying to take things which look random—protein sequences, DNA, garbled text—and extract higher-level structure from it. They have similar high-level mechanisms, but they’re actually trying to do the opposite things. I wonder what you make of that.

Reiner Pope

I try to look for other examples where mixing and scrambling shows up as well. There’s almost a physical example where you’re making a cake and you want to stir the batter. Literally the idea to first stir it this way and then stir it this way is not too bad of an approach.

Beyond that, back to the digital world, there are some differences, and the one you call out is a pretty strong difference. The way it shows up, if you just randomly initialize a neural network, maybe it’s a reasonable cipher as well because the random initialization is going to jumble stuff in a complicated way. It may even do what you want. Who knows?

The thing that makes it interpretable is the gradient descent. You can differentiate a neural network and get a meaningful derivative. We do a lot of work to not overcomplicate the derivative, so the residual connection keeps it contained and simple. And so does the LayerNorm stuff that we do.

One of the biggest attacks against cryptographic ciphers is also to differentiate the cipher. Ciphers run in a different number field. They run in the field of two elements, so just binary, whereas neural nets run, in theory, in the field of real numbers. You have to differentiate with respect to binary numbers, but you can absolutely differentiate a cipher. This is called differential cryptanalysis.

Basically, what it says is that if you take a small difference of the input, it’s quite difficult to make the difference of the output be small. The whole job of a well-designed cipher is to make the difference in output very large. The distinction is that the optimization goals at that point are about complexifying. They don’t have the same residual connections, like LayerNorms.

Dwarkesh Patel

I guess a place where the two merge is backdoors. With a backdoor in an LLM, you’re trying to hide… Would you consider it an input? It’s not an input into the forward pass but it’s an input into the backward pass. You’re trying to hide an input into the backward pass.

Reiner Pope

This is an adversarial context? This is actually a place where you get exactly the avalanche property that ciphers have as well. Adversarial attacks on image classification models are about finding a very small perturbation of the image that totally changes the classification, totally changes the output. That is the common case in ciphers, whereas that’s the undesired case in neural nets.

Dwarkesh Patel

Ok, so I was asking you, have neural networks actually been used for cryptography? And we realized it may be better to just do this on the blackboard. Are they actually being used for cryptography?

Reiner Pope

Using neural nets for cryptography… In general, creating a new cipher is a very dangerous proposition. Almost all of them are broken. 99% of them are broken, so it’s probably a bad place to start. But the other direction has been, in at least one very clear case, quite productive.

There’s a construction that exists in ciphers and then was imported into neural nets called a Feistel cipher, or Feistel network. The idea is that you may have some function f which is not invertible, but you like the function because it does interesting things, like it does an MLP, for example. Or it mixes it in an interesting way.

You’d like to build something out of this that is invertible. The construction we’re going to make is going to be a two-input function rather than a one-input function. We’re going to apply f(x). We need to actually remember what x was, so we’re going to stick x over here so that we can work backwards, and then we also can’t drop y. We’re going to remember y, and we’re going to add them together to form this tuple.

The way to invert this, if you think I have this output and I want to recover x and y, I can easily recover x. That’s right there, I just read it off. To recover y, if this thing was called z, I can recover y by z minus f(x), because I’ve already recovered x. That means this construction is invertible.

This was used in ciphers a ton and still is used. It’s one of the main mechanisms of constructing ciphers. Often you want ciphers to be invertible, especially the layers of ciphers, because that has better cryptographic properties.

This has actually been ported over into neural nets. There’s a 2017 paper called RevNets, reversible networks. What it does is make the entire network invertible. You can apply it to any network, like a transformer network. I do a forwards pass, but then I can run the entire pass backwards as well. The whole neural network is invertible with exactly this construction.

This paper applied it to some layer, like a transformer layer, for example. We’ve got this function f, which is our transformer layer. Normally we would have just an input and then a residual connection coming out, and it gets added over here. Now, the variation of this is going to be we’ve got two inputs, x and y. x goes through the function, gets added to y, and then this becomes the new x, output x. Then this x becomes the output y.

Really what this is doing, if you think of two layers back, is the thing you mentioned before. It’s doing the residual connection from two layers back. This y came from the previous layer and was the residual connection there. Because of this construction, the whole thing is invertible.

Why do I care? What does invertible matter for? The big thing that it can be interesting for is training. If I think of a forward pass of training… Let’s say I have four layers and I run them in zero, one, two, three order. I have to write all of the activations to HBM. I get an HBM footprint here that is kind of linear in the number of layers.

This can actually be the largest memory footprint during training. This is normal training, and then I run the backwards pass and read it in reverse. The forward pass goes forward, and the backward pass goes backwards. I have to read them back out.

The idea of this RevNets paper is that because it’s invertible, I don’t need to store this at all. I can completely rematerialize it. I run my forwards pass, and then when I’m running my backwards pass, I’m simultaneously in lockstep undoing all of the forwards pass steps that I did in order to have the activations that I need here. This ends up being memory saving, which is a nice idea.

Dwarkesh Patel

Interesting. In some sense you’re spending more compute to save memory.

Reiner Pope

That’s right.

Dwarkesh Patel

Interesting. It’s the opposite of what you’re doing with the KV cache. With the KV cache, you’re spending more memory to save compute.

Reiner Pope

Yeah. Spending more memory to save compute is generally profitable given where hardwares are.

Dwarkesh Patel

Interesting. That was super fun. Reiner, thank you so much for doing it. I feel like it really vindicated the vision behind the studio and the blackboard.

Reiner Pope

Yeah.

Dwarkesh Patel

Cool, thanks so much for doing it.

Reiner Pope

Thanks.