Dwarkesh Podcast

Zuck on:

• Llama 4, benchmark gaming, open vs closed

• Intelligence explosion, business models for AGI

• DeepSeek/China, export controls, & Trump

• Orion glasses, AI relationships, and not getting reward-hacked by our tech

Watch on Youtube; listen on Apple Podcasts and Spotify.

• Scale is building the infrastructure for safer, smarter AI. Scale’s Data Foundry gives major AI labs access to high-quality data to fuel post-training, while their public leaderboards help assess model capabilities. They also just released Scale Evaluation, a new tool that diagnoses model limitations. If you’re an AI researcher or engineer, learn how Scale can help you push the frontier at scale.com/dwarkesh.

• WorkOS Radar protects your product against bots, fraud, and abuse. Radar uses 80+ signals to identify and block common threats and harmful behavior. Join companies like Cursor, Perplexity, and OpenAI that have eliminated costly free-tier abuse by visiting workos.com/radar.

• Lambda is THE cloud for AI developers, with over 50,000 NVIDIA GPUs ready to go for startups, enterprises, and hyperscalers. By focusing exclusively on AI, Lambda provides cost-effective compute supported by true experts, including a serverless API serving top open-source models like Llama 4 or DeepSeek V3-0324 without rate limits, and available for a free trial at lambda.ai/dwarkesh.

To sponsor a future episode, visit dwarkesh.com/p/advertise.

(00:00:00) – How Llama 4 compares to other models

(00:11:34) – Intelligence explosion

(00:26:36) – AI friends, therapists & girlfriends

(00:35:10) – DeepSeek & China

(00:39:49) – Open source AI

(00:54:15) – Monetizing AGI

(00:58:32) – The role of a CEO

(01:02:04) – Is big tech aligning with Trump?

(01:07:10) – 100x productivity

Dwarkesh Patel

Mark, thanks for coming on the podcast again.

Mark Zuckerberg

Yeah, happy to do it. Good to see you.

Dwarkesh Patel

You too. Last time you were here, you had launched Llama 3. Now you've launched Llama 4.

Mark Zuckerberg

Well, the first version.

Dwarkesh Patel

That's right. What's new? What's exciting? What's changed?

Mark Zuckerberg

The whole field is so dynamic. I feel like a ton has changed since the last time we talked. Meta AI has almost a billion people using it monthly now, which is pretty wild. I think this is going to be a really big year for all of this, especially once you get the personalization loop going, which we’re just starting to build in now really, from both the context that all the algorithms have about what you’re interested in — feed, your profile information, your social graph information — but also what you're interacting with the AI about. That’s going to be the next thing that's super exciting. I'm really big on that.

The modeling stuff continues to make really impressive advances too. I'm pretty happy with the first set of Llama 4 releases. We announced four models and released the first two — the Scout and Maverick ones — which are mid-size to small models.

The most popular Llama 3 model was the 8 billion parameter one. So we’ve got one of those coming in the Llama 4 series too. Our internal code name for it is “Little Llama.” That’s coming probably over the next few months.

Scout and Maverick are good. They have some of the highest intelligence per cost you can get of any model out there. They’re natively multimodal, very efficient, run on one host. They’re designed to be very efficient and low latency, for a lot of the use cases we’re building for internally. That’s our whole thing. We build what we want, and then we open-source it so other people can use it too. I'm excited about that.

I'm also excited about the Behemoth model, which is coming up. It's going to be our first model that's sort of at the frontier — more than 2 trillion parameters. As the name says, it's quite big. We’re trying to figure out how to make that useful for people. It’s so big that we've had to build a bunch of infrastructure just to be able to post-train it ourselves.

Now we're trying to wrap our heads around, how does the average developer out there actually use something like this? How do we make it useful — maybe by distilling it into models that are a reasonable size to run? Because you're obviously not going to want to run something like that in a consumer model.

As you saw with the Llama 3 stuff last year, the initial launch was exciting and then we just built on that over the year. 3.1 released the 405 billion model, 3.2 is when we got all the multimodal stuff in. We basically have a roadmap like that for this year too. So a lot going on.

Dwarkesh Patel

I'm interested to hear more about it. There's this impression that the gap between the best closed-source and the best open-source models has increased over the last year. I know the full family of Llama 4 models isn't out yet, but Llama 4 Maverick is at #35 on Chatbot Arena. On a bunch of major benchmarks, it seems like o4-mini or Gemini 2.5 Flash are beating Maverick, which is in the same class. What do you make of that impression?

Mark Zuckerberg

There are a few things. First, I actually think this has been a very good year for open source overall. If you go back to where we were last year, Llama was the only real, super-innovative open-source model. Now you have a bunch of them in the field.

In general, the prediction that this would be the year open source generally overtakes closed source as the most used models out there, I think that's generally on track to be true.

One interesting surprise — positive in some ways, negative in others, but overall good — is that it’s not just Llama. There are a lot of good ones out there. I think that's quite good.

Then there's the reasoning phenomenon, which you're alluding to talking about o3, o4, and other models. There's a specialization happening. If you want a model that’s the best at math problems, coding, or different things like those tasks, then reasoning models that consume more test-time or inference-time compute in order to provide more intelligence are a really compelling paradigm. And we're building a Llama 4 reasoning model too. It'll come out at some point.

But for a lot of the applications we care about, latency and good intelligence per cost are much more important product attributes. If you're primarily designing for a consumer product, people don't want to wait half a minute to get an answer. If you can give them a generally good answer in half a second, that's a great tradeoff.

I think both of these are going to end up being important directions. I’m optimistic about integrating reasoning models with the core language models over time. That's the direction Google has gone in with some of the more recent Gemini models. I think that's really promising. But I think there’s just going to be a bunch of different stuff that goes on.

You also mentioned the whole Chatbot Arena thing, which I think is interesting and points to the challenge around how you do benchmarking. How do you know what models are good for which things?

One of the things we've generally tried to do over the last year is anchor more of our models in our Meta AI product north star use cases. The issue with open source benchmarks, and any given thing like the LM Arena stuff, is that they’re often skewed toward a very specific set of uses cases, which are often not actually  what any normal person does in your product. The portfolio of things they’re trying to measure is often different from what people care about in any given product.

Because of that, we’ve found that trying to optimize too much for that kind of stuff has led us astray. It’s actually not led towards the highest quality product, the most usage, and best feedback within Meta AI as people use our stuff.

So we're trying to anchor our north star on the product value that people report to us, what they say that they want, and what their revealed preferences are, and using the experiences that we have. Sometimes these benchmarks just don't quite line up. I think a lot of them are quite easily gameable.

On the Arena you'll see stuff like Sonnet 3.7, which is a great model, and it's not near the top. It was relatively easy for our team to tune a version of Llama 4 Maverick that could be way at the top. But the version we released, the pure model, actually has no tuning for that at all, so it's further down. So you just need to be careful with some of these benchmarks. We're going to index primarily on the products.

​​Dwarkesh Patel

Do you feel like there is some benchmark which captures what you see as a north star of value to the user which can be be objectively measured between different models and where you'd say, "I need Llama 4 to come out on top on this”?

Mark Zuckerberg

Our benchmark is basically user value in Meta AI.

Dwarkesh Patel

But you can't compare that to other models.

Mark Zuckerberg

We might be able to, because we might be able to run other models and be able to tell. That's one of the advantages of open source. You have a good community of folks who can poke holes in your stuff and point out, "Okay, where is your model not good, and where is it good?"

The reality at this point is that all these models are optimized for slightly different mixes of things. Everyone is trying to go towards the same end in that all the leading labs are trying to create general intelligence, superintelligence, whatever you call it. AI that can lead toward a world of abundance where everyone has these superhuman tools to create whatever they want. That leads to dramatically empowering people and creating all these economic benefits.

However you define it, that's what a lot of the labs are going for. But there's no doubt that different folks have optimized toward different things. I think the Anthropic folks have really focused on coding and agents around that. The OpenAI folks, I think, have gone a little more toward reasoning recently.

There’s a space which, if I had to guess, I think will end up being the most used one: quick, very natural to interact with, natively multimodal, fitting throughout your day in the ways you want to interact with it.

I think you got a chance to play around with the new Meta AI app that we're releasing. One of the fun things we put in there is the demo for the full-duplex voice. It's early. There’s a reason why we haven't made that the default voice model in the app yet. But there's something about how naturally conversational it is that's really fun and compelling.

Being able to mix that in with the right personalization is going to lead toward a product experience where… If you fast-forward a few years, I think we're just going to be talking to AI throughout the day about different things we're wondering about.

You'll have your phone. You'll talk to it while browsing your feed apps. It'll give you context about different stuff. It'll answer your questions. It'll help you as you're interacting with people in messaging apps. Eventually, I think we'll walk through our daily lives and have glasses or other kinds of AI devices and just seamlessly interact with it all day long.

That’s the north star. Whatever the benchmarks are that lead toward people feeling like the quality is where they want to interact with it, that's what will ultimately matter the most to us.

Dwarkesh Patel

I got a chance to play around with both Orion and also the Meta AI app, and the voice mode was super smooth. It was quite impressive.

On the point of what the different labs are optimizing for — to steelman their view — I think a lot of them believe that once you fully automate software engineering and AI research, then you can kick off an intelligence explosion. You would have millions of copies of these software engineers replicating the research that happened between Llama 1 and Llama 4 — that scale of improvement again — but in a matter of weeks or months rather than years. So it really matters to just close the loop on the software engineer, and then you can be the first to ASI. What do you make of that?

Mark Zuckerberg

I personally think that's pretty compelling. That's why we have a big coding effort too. We're working on a number of coding agents inside Meta. Because we're not really an enterprise software company, we're primarily building it for ourselves.

Again, we go for a specific goal. We're not trying to build a general developer tool. We're trying to build a coding agent and an AI research agent that advances Llama research specifically. And it's fully plugged into our toolchain and all that.

That's important and is going to end up being an important part of how this stuff gets done. I would guess that sometime in the next 12 to 18 months, we'll reach the point where most of the code that's going toward these efforts is written by AI. And I don't mean autocomplete.

Today you have good autocomplete. You start writing something and it can complete a section of code. I'm talking more like: you give it a goal, it can run tests, it can improve things, it can find issues, it writes higher quality code than the average very good person on the team already. I think that's going to be a really important part of this for sure.

But I don't know if that's the whole game. That's going to be a big industry, and it's going to be an important part of how AI gets developed. But I think there are still… One way to think about it is that this is a massive space. I don't think there's just going to be one company with one optimization function that serves everyone as best as possible. There are going to be a bunch of different labs doing leading work in different domains. Some will be more enterprise-focused or coding-focused. Some will be more productivity-focused. Some will be more social or entertainment-focused.

Within the assistant space, there will be some that are more informational and productivity-focused, and some that are more companion-focused. It’s going to be a lot of stuff that’s just fun and entertaining and shows up in your feed.

There's just a huge amount of space. Part of what's fun about going toward this AGI future is that there are a bunch of common threads for what needs to get invented, but also a lot of things that still need to be created. I think you're going to start seeing more specialization between different groups, if I had to guess.

Dwarkesh Patel

It’s really interesting to me that you basically agree with the premise that there will be an intelligence explosion and we’ll get something like superintelligence on the other end. Tell me if I'm misunderstanding you. If that’s the case, why even bother with personal assistants and whatever else? Why not just get to superhuman intelligence first and then deal with everything else later?

Mark Zuckerberg

I think that's just one aspect of the flywheel. Part of what I generally disagree with on the fast-takeoff view is that it takes time to build out physical infrastructure.

If you want to build a gigawatt cluster of compute, that just takes time. NVIDIA needs time to stabilize their new generation of systems. Then you need to figure out the networking around it. Then you need to build the building. You need to get permitting. You need to get the energy. Maybe that means gas turbines or green energy, either way, there’s a whole supply chain of that stuff.

We talked about this a bunch the last time I was on the podcast with you. I think some of these are just physical-world, human-time things. As you start getting more intelligence in one part of the stack, you’re just going to run into a different set of bottlenecks. That’s how engineering always works: solve one bottleneck, you get another bottleneck.

Another bottleneck in the system or ingredient that’s going to make this work well, is people getting used to learning and having a feedback loop with using the system. These systems don’t just show up fully formed with people magically knowing how to use them. There's a co-evolution that happens where people are learning how to best use these AI assistants. At the same time, the AI assistants are learning what people care about. Developers are making the AI assistants better.

You're building up a base of context too. You wake up a year or two into it and the assistant can reference things you talked about two years ago and that’s pretty cool. You couldn’t do that even if you launched the perfect thing on day one. There’s no way it could reference what you talked about two years ago if it didn’t exist two years ago.

So I guess my view is that there's this huge intelligence growth. There’s a very rapid curve on the uptake of people interacting with the AI assistants, and the learning feedback and data flywheel around that. And then there is also the buildout of the supply chains and infrastructure and regulatory frameworks to enable the scaling of a lot of the physical infrastructure. At some level, all of those are going to be necessary, not just the coding piece.

One specific example of this that I think is interesting. Even if you go back a few years ago, we had a project, I think it was on our ads team, to automate ranking experiments. That's a pretty constrained environment. It's not open-ended code. It’s basically, look at the whole history of the company — every experiment that any engineer has ever done in the ad system — and look at what worked, what didn't, and what the results of those were. Then basically formulate new hypotheses for different tests that we should run that could improve the performance of the ad system.

What we basically found was that we were bottlenecked on compute to run tests, based on the number of hypotheses. It turns out, even with just the humans we have right now on the ads team, we already have more good ideas to test than we actually have either compute or, really, cohorts of people to test them with.

Even if you have three and a half billion people using your products, you still want each test to be statistically significant. It needs to have hundreds of thousands or millions of people. There's only so much throughput you can get on testing through that. So we're already at the point, even with just the people we have, that we can't really test everything that we want.

Now just being able to test more things is not necessarily going to be additive to that. We need to get to the point where the average quality of the hypotheses that the AI is generating is better than all the things above the line that we’re actually able to test that the best humans on the team have been able to do, before it will even be marginally useful for it.

We'll get there I think pretty quickly. But it's not just, “Okay, cool, the thing can write code, and now all of a sudden everything is just improving massively.” There are real-world constraints that need to be overcome.

 Then you need to have the compute and the people to test. Then over time, as the quality creeps up, are we here in five or 10 years where no set of people can generate a hypothesis as good as the AI system? I don't know, maybe. In that world, obviously that's going to be how all the value is created. But that's not the first step.

Dwarkesh Patel

So if you buy this view, that this is where intelligence is headed, the reason to be bullish on Meta is obviously that you have all this distribution. You can also use that to learn more things that can be useful for training. You mentioned the Meta AI app now has a billion active users.

Mark Zuckerberg

Not the app. The app is a standalone thing that we're just launching now. It’ll be fun for people who want to use it. It's a cool experience. We can talk about that too because we’re experimenting with some new ideas in there that I think are novel and worth talking through.

But I’m mostly talking about our apps. Meta AI is actually most used in WhatsApp. WhatsApp is mostly used outside of the U.S. We just passed like a hundred million people in the US, but it's not the primary messaging system in the US, iMessage is. So people in the U.S. probably tend to underestimate Meta AI usage somewhat. But part of the reason the standalone app is going to be so important is because the US, for a lot of reasons, is one of the most important countries. And the fact that WhatsApp is the main way people are using Meta AI and that's not the main messaging system in the US means we need another way to build a first-class experience that's really in front of people.

Dwarkesh Patel

And I guess, to finish the question, the bearish case would be that if the future of AI is less about just answering your questions and more about being a virtual coworker, then it's not clear how Meta AI inside of WhatsApp gives you the relevant training data to make a fully autonomous programmer or remote worker. In that case, does it not matter that much who has more distribution right now with LLMs?

Mark Zuckerberg

Again, I just think there are going to be different things. Imagine you were sitting at the beginning of the development of the internet and you asked, "What's going to be the main internet thing? Is it going to be knowledge work or massive consumer apps?"

You got both. You don’t have to choose one. The world is big and complicated. Does one company build all of that stuff? Normally the answer is no. But to your question, people do not code in WhatsApp for the most part. And I don't foresee that people starting to write code in WhatsApp is going to be a major use case. Although I do think people are going to ask AI to do a lot of things that result in the AI coding without them necessarily knowing it. That's a separate thing.

We do have a lot of people who are writing code at Meta and they use Meta AI. We have this internal thing called MetaMate, and a number of different coding and AI research agents that we're building around that. That has its own feedback loop and I think it can get quite good for accelerating those efforts.

But again, there are going to be a lot of things. AI is almost certainly going to unlock a massive revolution in knowledge work and code. I also think it’s going to be the next generation of search and how people get information, and do more complex information tasks.

I also think it's going to be fun. People are going to use it to be entertained. A lot of the internet today is memes and humor. We have this amazing technology at our fingertips. It’s amazing and funny when you think about how much of human energy just goes toward entertaining ourselves, designing, pushing culture forward, and finding humorous ways to explain cultural phenomena that we observe. I think that's almost certainly going to be the case in the future.

Look at the evolution of things like Instagram and Facebook. If you go back 10, 15, 20 years ago, it was text. Then we all got phones with cameras, and most of the content became photos. Then the mobile networks got good enough that if you wanted to watch a video on your phone, it wasn't just buffering the whole time. So that got good.

Over the last 10 years, most of the content has moved toward video at this point. Today, most of the time spent on Facebook and Instagram is on video. But do you think in five years we’re just going to be sitting in our feed and consuming media that's just video? No, it's going to be interactive. You'll be scrolling through your feed. There will be content that maybe looks like a Reel to start. But you can talk to it, or interact with it, and it talks back, or it changes what it's doing. Or you can jump into it like a game and interact with it. That's all going to be AI.

My point is that there are going to be all these different things. We're ambitious, so we're working on a bunch of them. But I don't think any one company is going to do all of it.

Dwarkesh Patel

On this point about AI-generated content and AI interactions, already people have meaningful relationships with AI therapists, AI friends, maybe more. This is just going to get more intense as these AIs become more unique, more personable, more intelligent, more spontaneous, more funny, and so forth.

People are going to have relationships with AI. How do we make sure these are healthy relationships?

Mark Zuckerberg

There are a lot of questions that you only can really answer as you start seeing the behaviors. Probably the most important upfront thing is just to ask that question and care about it at each step along the way. But I also think being too prescriptive upfront and saying, "We think these things are not good" often cuts off value.

People use stuff that's valuable for them. One of my core guiding principles in designing products is that people are smart. They know what's valuable in their lives. Every once in a while, something bad happens in a product and you want to make sure you design your product well to minimize that.

But if you think something someone is doing is bad and they think it's really valuable, most of the time in my experience, they're right and you're wrong. You just haven't come up with the framework yet for understanding why the thing they're doing is valuable and helpful in their life. That's the main way I think about it.

I do think people are going to use AI for a lot of these social tasks. Already, one of the main things we see people using Meta AI for is talking through difficult conversations they need to have with people in their lives. "I'm having this issue with my girlfriend. Help me have this conversation.” Or, "I need to have a hard conversation with my boss at work. How do I have that conversation?" That's pretty helpful. As the personalization loop kicks in and the AI starts to get to know you better and better, that will just be really compelling.

Here’s one stat from working on social media for a long time that I always think is crazy. The average American has fewer than three friends, fewer than three people they would consider friends. And the average person has demand for meaningfully more. I think it's something like 15 friends or something. At some point you're like, "All right, I'm just too busy, I can't deal with more people."

But the average person wants more connection than they have. There's a lot of concern people raise like, "Is this going to replace real-world, physical, in-person connections?" And my default is that the answer to that is probably not. There are all these things that are better about physical connections when you can have them. But the reality is that people just don't have as much connection as they want. They feel more alone a lot of the time than they would like.

So I think a lot of these things — things that today might have a little bit of stigma around them — over time, we'll find the vocabulary as a society to articulate why they are valuable, why the people who are doing them are rational for doing it, and how it is actually adding value to their lives. But also the field is very early. There are a handful of companies doing virtual therapists, virtual girlfriend-type stuff. But it's very early. The embodiment in those things is still pretty weak. You open it up and it's just an image of the therapist or the person you're talking to. Sometimes there's some very rough animation, but it's not an embodiment.

You've seen the stuff we're working on in Reality Labs, where you have the Codec Avatars and it actually feels like a real person. That's where it's going. You'll be able to have an always-on video chat with the AI. The gestures are important too. More than half of communication, when you're actually having a conversation, is not the words you speak. It's all the nonverbal stuff.

Dwarkesh Patel

I did get a chance to check out Orion the other day, and I thought it was super impressive. I'm mostly optimistic about the technology. Generally, like you mentioned, I'm pretty libertarian about this. If people are doing something, they probably think it's good for them. Although, I actually don't know if it's the case that if somebody is using TikTok, they would say that they're happy with how much time they're spending on TikTok or something.

I'm mostly optimistic about it in the sense that if we're going to be living in this future world of AGI, we need to be upgrading our capabilities too, with tools like this. And just generally, there can be more beauty in the world if you can see Studio Ghibli everywhere or something.

I was worried about one of the flagship use cases that your team showed me. I'm sitting at the breakfast table and on the periphery of my vision is just a bunch of Reels that are scrolling by. Maybe in the future, my AI girlfriend is on the other side of the screen or something. So I am worried that we're just removing all the friction between getting totally reward-hacked by our technology. How do we make sure this is not what ends up happening in five years?

Mark Zuckerberg

Again, I think people have a good sense of what they want. That experience you saw was just a demo to show multitasking and holograms. I agree, I don't think the future is one where you have stuff that's trying to compete for your attention in the corner of your vision all the time. I don't think people would like that too much.

As we're designing these glasses, it's actually one of the things that we're really mindful of. Probably the number one thing the glasses need to do is get out of the way and be good glasses. As an aside, I think that's part of the reason why the Ray-Ban Meta product has done so well. It's great for listening to music, taking phone calls, taking photos and videos. The AI is there when you want it. But when you don't, it's just a good-looking pair of glasses that people like. It gets out of the way well.

I would guess that's going to be a very important design principle for the augmented reality future. The main thing that I see here is this. It's kind of crazy that, for how important the digital world is in all of our lives, the only way we access it is through these physical, digital screens. You have your phone, your computer. You can put a big TV on your wall. It's this huge physical thing.

It just seems like we're at the point with technology where the physical and digital world should really be fully blended. That's what holographic overlays allow you to do. But I agree. I think a big part of the design principles around that will be around how you'll be interacting with people. You'll be able to bring digital artifacts into those interactions and do cool things very seamlessly.

If I want to show you something, here’s a screen. We can interact with it. It can be 3D. We can play with it. You want to play a card game? All right, here’s a deck of cards. We can play with it. If two of us are physically together and we have a third friend who’s hologramming in, they can participate too.

But in that world too — just as you don't want your physical space to be cluttered because it wears on you psychologically — I don't think people are going to want their digital-physical space to feel that way either.  That's more of an aesthetic norm that will have to get worked out, but I think we’ll figure that out.

Dwarkesh Patel

Going back to the AI conversation, you were mentioning how big of a bottleneck the physical infrastructure can be. Related to other open-source models, like DeepSeek and so forth, DeepSeek right now has less compute than a lab like Meta and you could argue that it's competitive with the Llama models.

If China is better at physical infrastructure, industrial scale-ups, getting more power and more data centers online, how worried are you that they might beat us here?

Mark Zuckerberg

It's a real competition. You're seeing industrial policies really play out. China is bringing online more power. Because of that, the US really needs to focus on streamlining the ability to build data centers and produce energy. Otherwise, I think we’ll be at a significant disadvantage.

At the same time, some of the export controls on things like chips, I think you can see how they’re clearly working in a way. There was all the conversation with DeepSeek about, "Oh, they did all these very impressive low-level optimizations." And the reality is, they did and that is impressive.

But then you ask, "Why did they have to do that, when none of the American labs did it?" It’s because they’re using partially nerfed chips that are the only ones NVIDIA is allowed to sell in China because of the export controls. DeepSeek basically had to spend a bunch of their calories and time doing low-level infrastructure optimizations that the American labs didn’t have to do.

Now, they produced a good result on text. DeepSeek is text-only. The infrastructure is impressive. The text result is impressive. But every new major model that comes out now is multimodal. It's image, it's voice. Theirs isn't.

Now the question is, why is that the case? I don’t think it’s because they’re not capable of doing it. It's because they had to spend their calories on doing these infrastructure optimizations to overcome the fact that there were these export controls.

But when you compare Llama 4 with DeepSeek —I mean our reasoning model isn’t out yet, so the R1 comparison isn’t clear yet— but we’re basically in the same ballpark on all the text stuff that DeepSeek is doing but with a smaller model. So the cost-per-intelligence is lower with what we’re doing for Llama on text. On the multimodal side we’re effectively leading at and it just doesn’t exist in their models.

So the Llama 4 models, when you compare them to what DeepSeek is doing, are good. I think people will generally prefer to use the Llama 4 models. But there’s this interesting contour where it’s clearly a good team doing stuff over there. And you're right to ask about the accessibility of power, the accessibility of compute and chips, because the work that you're seeing different labs do and the way it's playing out is somewhat downstream of that.

Dwarkesh Patel

So Sam Altman recently tweeted that OpenAI is going to release an open-source SOTA reasoning model. I think part of the tweet was that they won’t do anything silly, like say you can only use it if you have less than 700 million users.

DeepSeek has the MIT license, whereas I think a couple of the contingencies in the Llama license require you to say "built with Llama" on applications using it or any model that you train using Llama has to begin with the word "Llama." What do you think about the license? Should it be less onerous for developers?

Mark Zuckerberg

Look, we basically pioneered the open-source LLM thing. So I don't consider the license to be onerous. When we were starting to push on open source, there was this big debate in the industry. Is this even a reasonable thing to do? Can you do something that is safe and trustworthy with open source? Will open source ever be able to be competitive enough that anyone will even care?

Basically, when we were answering those questions a lot of the hard work was done by the teams at Meta. There were other folks in the industry but really, the Llama models were the ones that broke open this whole open-source AI thing in a huge way.

If we’re going to put all this energy into it, then at a minimum, if you're going to have these large cloud companies — like Microsoft and Amazon and Google — turn around and sell our model, then we should at least be able to have a conversation with them before they do that around what kind of business arrangement we should have.

Our goal with the license, we're generally not trying to stop people from using the model. We just think that if you're one of those companies, or if you're Apple, just come talk to us about what you want to do. Let's find a productive way to do it together. I think that’s generally been fine.

Now, if the whole open-source part of the industry evolves in a direction where there are a lot of other great options and the license ends up being a reason why people don’t want to use Llama, then we’ll have to reevaluate the strategy. What it makes sense to do at that point. But I don’t think we’re there.

That’s not, in practice, something we’ve seen, companies coming to us and saying, “We don’t want to use this because your license says if you reach 700 million people, you have to come talk to us.” So far, that’s been more something we’ve heard from open-source purists like, “Is this as clean of an open-source model as you’d like it to be?”

That debate has existed since the beginning of open source. All the GPL license stuff versus other things, do you need to make it so that anything that touches open source has to be open source too? Or can people take it and use it in different ways? I'm sure there will continue to be debates around this.

But if you’re spending many billions of dollars training these models, I think asking the other companies — the huge ones that are similar in size and can easily afford to have a relationship with us — to talk to us before they use it seems like a pretty reasonable thing.

Dwarkesh Patel

If it turns out that other models are also really good. There’s a bunch of good open-source models. So that part of your mission is fulfilled, and maybe other models are better at coding.

Is there a world where you just say, "Look, the open-source ecosystem is healthy. There’s plenty of competition. We're happy to just use some other model, whether it's for internal software engineering at Meta or deploying to our apps. We don't necessarily need to build with Llama"?

Mark Zuckerberg

Again, we do a lot of things. Let's take a step back. The reason why we're building our own big models is because we want to be able to build exactly what we want. None of the other models in the world are exactly what we want.

If they're open source, you can take them and fine-tune them in different ways. But you still have to deal with the model architectures. And they make different size tradeoffs that affect latency and inference cost. At the scale that we operate at, that stuff really matters.

We made the Llama Scout and Maverick models certain sizes for a specific reason. They fit on a host and we wanted certain latency — especially for the voice models that we’re working on — that we want to pervade everything we're doing from the glasses to all of our apps to the Meta AI app and all that stuff.

There's a level of control of your own destiny that you only get when you build the stuff yourself. That said, AI is going to be used in every single thing that every company does. When we build a big model, we also have to choose which internal use cases we're going to optimize for.

So does that mean for certain things we might say, "Okay, maybe Claude is better for building this specific development tool that this team is using”? All right, cool then use that. Great. We don’t want to fight with one hand tied behind our back. We’re doing a lot of different stuff.

You also asked, would it not be important anymore because other people are doing open source? On this, I'm a little more worried. You have to ask yourself this. For anyone who shows up now and is doing open source — now that we have done it — would they still be doing open source if we weren’t doing it?

I think there are a handful of folks who see the trend that more and more development is going toward open source, and they're like, "Oh crap, we need to be on this train or else we’re going to lose." If you have a closed-model API and increasingly a lot of developers don't want that.

So you’re seeing a bunch of other players start to do some work in open source. But it's unclear if it's dabbling, or fundamental for them the way that it has been for us. A good example is what's going on with Android. Android started off as the open-source thing. There's not really any open-source alternative. Over time, Android has just gotten more and more closed.

So if you're us, you need to worry that if we stop pushing the industry in this direction, all these other people… Maybe they’re only really doing it because they're trying to compete with us and the direction we’re pushing things. They already showed their revealed preference for what they would do if open source didn’t exist. And it wasn’t open source. We just need to be careful about relying on that continued behavior for the future of the technology that we're going to build at the company.

Dwarkesh Patel

Another thing I've heard you mention is that it's important that the standard gets built around American models like Llama. I wanted to understand your logic there. With certain kinds of networks, it is the case that the Apple App Store just has a big contingency around what it's built around.

But it doesn't seem like if you built some sort of scaffold for DeepSeek, you couldn't have easily just switched it over to Llama 4, especially since between generations. Llama 3 wasn't MoE and Llama 4 is. So things are changing between generations of models as well.

What’s the reason for thinking things will get built out in this contingent way on a specific standard?

Mark Zuckerberg

I'm not sure, what do you mean by contingent?

Dwarkesh Patel

As in, it's important that people are building for Llama rather than for LLMs in general, because that will determine what the standard is in the future.

Mark Zuckerberg

Look, I think these models encode values and ways of thinking about the world.

We had this interesting experience early on, where we took an early version of Llama and translated it. I think it was French, or some other language.

The feedback we got from French people was, "This sounds like an American who learned to speak French. It doesn’t sound like a French person." And we were like, “what do you mean, does it not speak French well?” No, it speaks French fine. It was just that the way it thought about the world seemed slightly American. So I think there are these subtle things that get built into the models.

Over time, as models get more sophisticated, they should be able to embody different value sets across the world. So maybe that's not a particularly sophisticated example, but I think it illustrates the point.

Some of the stuff we've seen in testing some of the models, especially coming out of China, have certain values encoded in them. And it’s not just a light fine-tune to change that. Now, language models — or something that has a kind of world model embedded in it — have more values.

Reasoning, I guess, you could say has values too. But one of the nice things about reasoning models is they're trained on verifiable problems. Do you need to be worried about cultural bias if your model is doing math? Probably not. I think the chance that some reasoning model built elsewhere is going to incept you by solving a math problem in a devious way seems low.

But there's a whole different set of issues around coding, which is the other verifiable domain. You need to worry about waking up one day and if you're using a model that has some tie to another government, can it embed vulnerabilities in code that their intelligence organizations could exploit later? In some future version you're using a model that came from another country and it's securing your systems. Then you wake up and everything is just vulnerable in a way that that country knows about and you don’t. Or it turns on a vulnerability at some point.

Those are real issues. I'm very interested in studying this because I think one of the main things that's interesting about open source is the ability to distill models. For most people, the primary value isn't just taking a model off the shelf and saying, "Okay, Meta built this version of Llama. I'm going to take it and I'm going to run it exactly in my application."

No, your application isn't doing anything different if you're just running our thing. You're at least going to fine-tune it, or try to distill it into a different model. When we get to stuff like the Behemoth model, the whole value is being able to take this very high amount of intelligence and distill it down into a smaller model that you're actually going to want to run.

This is the beauty of distillation. It's one of the things that I think has really emerged as a very powerful technique over the last year, since the last time we sat down. I think it’s worked better than most people would have predicted. You can basically take a model that's much bigger, and capture probably 90 or 95% of its intelligence, and run it in something that's 10% of the size. Now, do you get 100% of the intelligence? No. But 95% of the intelligence at 10% of the cost is pretty good for a lot of things.

The other thing that's interesting is that now, with this more varied open-source community, it's not just Llama. You have other models too. You have the ability to distill from multiple sources. So now you can basically say, "Okay, Llama’s really good at this. Maybe its architecture is really good because it's fundamentally multimodal, more inference-friendly, more efficient. But let’s say this other model is better at coding." Okay, great. You can distill from both of them and build something that's better than either individually, for your own use case. That's cool.

But you do need to solve the security problem of knowing that you can distill it in a way that's safe and secure. This is something that we've been researching and have put a lot of time into. What we've basically found is that anything that's language is quite fraught. There's just a lot of values embedded into it. Unless you don't care about taking on the values from whatever model you're distilling from, you probably don't want to just distill a straight language world model.

On reasoning, though, you can get a lot of the way there by limiting it to verifiable domains, and running code cleanliness and security filters. Whether it's using Llama Guard open source, or the Code Shield open source tools that we've done, things that allow you to incorporate different input into your models and make sure that both the input and the output are secure.

Then it’s just a lot of red teaming. It’s having experts who are looking at the model and asking, "Alright, is this model doing anything after distillation that we don't want?" I think with the combination of those techniques, you can probably distill on the reasoning side for verifiable domains quite securely. That's something I'm pretty confident about and something we've done a lot of research around.

But I think this is a very big question. How do you do good distillation? Because there’s so much value to be unlocked. But at the same time, I do think there is some fundamental bias embedded in different models.

Dwarkesh Patel

Speaking of value to be unlocked, what do you think the right way to monetize AI will be? Obviously digital ads are quite lucrative. But as a fraction of total GDP, it's small compared to all remote work. Even if you can increase productivity without replacing work, that's still worth tens of trillions of dollars. Is it possible that ads might not be it? How do you think about this?

Mark Zuckerberg

Like we were talking about before, there's going to be all these different applications, and different applications tend toward different things.

Ads are great when you want to offer people a free service. Because it's free, you need to cover it somehow. Ads solve this problem where a person does not need to pay for something. They can get something that is amazing for free. Also by the way, with modern ad systems, a lot of the time people think the ads add value to the thing if you do it well.

You need to be good at ranking and you need to have enough liquidity of advertising inventory. If you only have five advertisers in the system, no matter how good you are at ranking, you may not be able to show something to someone that they're interested in. But if you have a million advertisers in the system, then you're probably going to be able to find something pretty compelling, if you're good at picking out the different needles in the haystack that that person is going to be interested in.

So that definitely has its place. But there are also clearly going to be other business models as well, including ones that just have higher costs so it doesn't even make sense to offer them for free. By the way, there have always been business models like this.

There's a reason why social media is free and ad-supported, but then if you want to watch Netflix or ESPN or something, you need to pay for that. The content that's going into that, they need to produce it, and that's very expensive for them to produce. They probably could not have enough ads in the service in order to make up for the cost of producing the content. Basically, you just need to pay to access it.

The trade-off is fewer people do it. Instead of billions, you're talking about hundreds of millions of people using those services. There's a value switch there. I think it's similar here. Not everyone is going to want a software engineer, or a thousand software engineering agents, or whatever it is. But if you do, that's something you're probably going to be willing to pay thousands, or tens of thousands, or hundreds of thousands of dollars for.

That just speaks to the diversity of different things that need to get created. There are going to be business models at each point along the spectrum. At Meta, for the consumer piece we definitely want to have a free thing. I'm sure that will end up being ad-supported. But I also think we're going to want to have a business model that supports people using arbitrary amounts of compute to do even more amazing things than what it would make sense to offer in the free service. For that, I'm sure we'll end up having a premium service. But I think our basic values on this are that we want to serve as many people in the world as possible.

Dwarkesh Patel

How do you keep track of all these different projects, some of which we've talked about today. I'm sure there are many I don't even know about. As the CEO overseeing everything, there's a big spectrum between going to the Llama team and saying, "Here are the hyperparameters you should use," versus just giving a mandate like, "Go make the AI better."

And there are so many different projects. How do you think about the way in which you can best deliver your value-add and oversee all these things?

Mark Zuckerberg

A lot of what I spend my time on is trying to get awesome people onto the teams. There's that, and then there's stuff that cuts across teams. You build Meta AI, and you want to get it into WhatsApp or Instagram. Okay, now I need to get those teams to talk together. Then there are a bunch of questions like, “do you want the thread for Meta AI in WhatsApp to feel like other WhatsApp threads, or do you want it to feel like other AI chat experiences?” There are different idioms for those. So there are all these interesting questions that need to get answered around how does this stuff basically fit into everything we're doing?

Then there's a whole other part of what we're doing, which is pushing on the infrastructure. If you want to stand up a gigawatt cluster, first of all, that has a lot of implications for the way we're doing infrastructure buildouts. It has political implications for how you engage with the different states where you're building that stuff. It has financial implications for the company in terms of: "All right, there's a lot of economic uncertainty in the world. Do we double down on infrastructure right now? If so, what other trade-offs do we want to make around the company?" Those are the kinds of decisions that are tough for other people to really make.

Then there's this question around taste and quality. When is something good enough that we want to ship it? In general, I'm the steward of that for the company. Although we have a lot of other people who I think have good taste as well and are also filters for different things.

Those are basically the areas. AI is interesting because, more than some of the other stuff that we do, it is more research and model-led than really product-led. You can't just design the product that you want and then try to build the model to fit into it. You really need to design the model first and the capabilities that you want, and then you get some emergent properties. Then it's, "Oh, you can build some different stuff because this turned out in a certain way." At the end of the day, people want to use the best model.

That's partially why, when we're talking about building the most personal AI, the best voice, the best personalization — and also a very smart experience with very low latency — those are the things that we need to design the whole system to build. That's why we're working on full-duplex voice. That's why we're working on personalization to both have good memory extraction from your interactions with AI, but also to be able to plug into all the other Meta systems. That's why we design the specific models that we design, to have the kind of size and latency parameters that they do.

Dwarkesh Patel

Speaking of politics, there's been this perception that some tech leaders have been aligning with Trump. You and others donated to his inaugural event and were on stage with him and I think you settled a lawsuit that resulted in them getting $25 million.

I wonder what's going on here? Does it feel like the cost of doing business with an administration? What's the best way to think about this?

Mark Zuckerberg

My view on this is that he's the President of the United States. Our default, as an American company, should be to try to have a productive relationship with whoever is running the government. We've tried to offer support to previous administrations as well. I've been pretty public with some of my frustrations with the previous administration, how they basically did not engage with us or the business community more broadly.

Frankly, that’s going to be necessary to make progress on some of these things. We're not going to be able to build the level of energy that we need if you don't have a dialogue, and if they're not prioritizing trying to do those things.

A lot of people want to write this story about what direction people are going. We're trying to build great stuff, and we want to have a productive relationship with people. That's how I see it. It is also how I would guess most others see it, but obviously, I can't speak for them.

Dwarkesh Patel

You've spoken out about how you've rethought some of the ways in which you engage and defer to the government, in terms of moderation stuff in the past.

How are you thinking about AI governance? Because if AI is as powerful as we think it might be, the government will want to get involved. What is the most productive approach to take there, and what should the government be thinking about?

Mark Zuckerberg

I guess in the past, most of the comments that I made were in the context of content moderation. It's been an interesting journey over the last 10 years on this. It's obviously been an interesting time in history. There have been novel questions raised about online content moderation.

Some of those have led to productive new systems getting built, like our AI systems to detect nation-states trying to interfere in each other's elections. I think we will continue building that stuff out, and that has been net positive.

With some other stuff, we went down some bad paths. I just think the fact-checking thing was not as effective as Community Notes because it's not an internet-scale solution. There weren't enough fact-checkers, and people didn't trust the specific fact-checkers. You want a more robust system. So I think what we got with Community Notes is the right one on that.

But my point on this was more that historically, I probably deferred a little too much to either the media and their critiques, or to the government, on things that they did not really have authority over. But just as like a central figure, I think we tried to build systems where maybe we wouldn't have to make all of the content moderation decisions ourselves or something.

I guess part of the growth process over the last 10 years is realizing, “Okay, we're a meaningful company. We need to own the decisions that we need to make. We should listen to feedback from people, but we shouldn't defer too much to people who do not actually have authority over this. Because at the end of the day, we're in the seat, and we need to own the decisions that we make.”

It's been a maturation process, and in some ways painful, but I think we're probably a better company for it.

Dwarkesh Patel

Will tariffs increase the cost of building data centers in the US and shift buildouts to Europe and Asia?

Mark Zuckerberg

It is really hard to know how that plays out. I think we're probably in the early innings on that, and it's very hard to know.

Dwarkesh Patel

What is your single highest-leverage hour in a week? What are you doing in that hour?

Mark Zuckerberg

I don't know. Every week is a little bit different. It's probably got to be the case that the most leveraged thing you do in a week is not the same thing each week. Or else, by definition, you should probably spend more than one hour doing that thing every week.

I don't know. Part of the fun of this job, and also of the industry being so dynamic, is that things really move around. The world is very different now than it was at the beginning of the year, or even six months ago, or in the middle of last year. I think a lot has advanced meaningfully. A lot of cards have been turned over since the last time that we sat down. I think that was about a year ago, right?

Dwarkesh Patel

Yeah. I guess what you were saying earlier that recruiting people is a super high-leverage thing you do.

Mark Zuckerberg

It's very high-leverage, yeah.

Dwarkesh Patel

You talked about these models being mid-level software engineers by the end of the year. What would be possible if, say, software productivity increased like 100x in two years? What kinds of things could be built that can't be built right now?

Mark Zuckerberg

What kinds of things? That's an interesting question. One theme of this conversation is that the amount of creativity that's going to be unlocked is going to be massive.

If you look at the overall arc of human society and the economy over 100 or 150 years, it's basically people going from being primarily agrarian — with most human energy going toward just feeding ourselves — to that becoming a smaller and smaller percent. And the things that take care of our basic physical needs have become a smaller and smaller percent of human energy.

That shift has led to two impacts: one is that more people are doing creative and cultural pursuits. The second is that more people, in general, spend less time working and more time on entertainment and culture. I think that is almost certainly going to continue as this goes on.

This isn't the 1-2 year thing of what happens when you have a super powerful software engineer. But over time, if everyone has these superhuman tools to create a ton of different stuff, you're going to get incredible diversity. Part of it is going to be solving hard problems: solving diseases, advancing science, developing new technology that makes our lives better.

But I would guess that a lot of it is going to end up being cultural and social pursuits and entertainment. I would guess the world is going to get a lot funnier, weirder, and quirkier, the way that memes on the internet have gotten over the last 10 years. I think that adds a certain richness and depth. In funny ways, it actually helps you connect better with people. Now all day long, I just find interesting stuff on the internet and send it in group chats to the people I care about, who I think are going to find it funny.

The media that people can produce today to express very nuanced, specific cultural ideas is really cool. That'll continue to get built out. It does advance society in a bunch of ways, even if it's not the "hard science" way of curing a disease.

If you think about it, the Meta social media view of the world is that yeah, people are going to spend a lot more time doing that stuff in the future. It's going to be a lot better, and it's going to help you connect, because it'll help express different ideas.

The world is going to get more complicated, but our technology, our cultural technology, to express these very complicated things — in a very kind of funny little clip or whatever — is going to get so much better. I think that's all great.

I don't know about next year. One other thought that I think is interesting to cover is that I tend to think that, for at least the foreseeable future, this is going to lead to more demand for people doing work, not less. Now, people have a choice of how much time they want to spend working.

I'll give you one interesting example we were talking about recently. We have almost three and a half billion people using our services every day. One question we've struggled with forever is how do we provide customer support?

Today, you can write an email, but we've never seriously been able to contemplate having voice support where someone can just call in. I guess that's maybe one of the artifacts of having a free service. The revenue per person isn't high enough to have an economic model where people can call in.

But also, with three and a half billion people using your service every day, the number of calls would be massive. It’d be like the biggest call center in the world. It would be like $10 or $20 billion a year to staff that. So we've never thought too seriously about it, because it always seemed like there was no way that could make sense. But now, as AI gets better, you're going to get to a place where AI can handle a bunch of people's issues.

Not all of them — maybe 10 years from now it can handle all of them — but thinking about a 3-5 year time horizon, it will be able to handle a bunch. It's kind of like a self-driving car. They can handle a bunch of terrain, but they're not doing the whole route by themselves yet in most cases. People thought truck-driving jobs were going to go away, but there's actually more truck-driving jobs now than when we first started talking about self-driving cars 20 years ago.

Going back to the customer support thing, it wouldn't make sense to staff out calling for everyone. But let's say AI can handle 90% of that. Then if it can't, it kicks it off to a person. If you get the cost of providing that service down to one-tenth of what it would've otherwise been, then maybe now it actually makes sense to do it. That would be cool. So the net result is that I actually think we're probably going to hire more customer support people.

The common belief is that AI will automate jobs away. But that hasn't really been how the history of technology has worked. Usually, you create things that take away 90% of the work, and that leads you to want more people, not less.

Dwarkesh Patel

Final question: Who is the one person in the world today who you most seek out for advice?

Mark Zuckerberg
Oh, man. I feel like part of my style is that I like having a breadth of advisors. It's not just one person.

We've got a great team. There are people at the company, people on our board. There are a lot of people in the industry who are doing new stuff. There's not a single person. But it's fun. Also, when the world is dynamic, just having a reason to work with people you like on cool stuff… To me, that's what life is about.

Dwarkesh Patel
Great note to close on. Thanks for doing this.

Mark Zuckerberg
Yeah, thank you.

800 years before the Black Death, the very same bacteria ravaged Rome, killing 60%+ of the population in many areas.

Also, back-to-back volcanic eruptions caused a mini Ice Age, leaving Rome devastated by famine and disease.

I chatted with historian Kyle Harper about this and much else:

• Rome as a massive slave society

• Why humans are more disease-prone than other animals

• How agriculture made us physically smaller (Caesar at 5'5" was considered tall)

Watch on Youtube; listen on Apple Podcasts or Spotify.

• WorkOS makes it easy to become enterprise-ready. They have APIs for all the most common enterprise requirements—things like authentication, permissions, and encryption—so you can quickly plug them in and get back to building your core product. If you want to make your product enterprise-ready, join companies like Cursor, Perplexity and OpenAI, and head to workos.com.

• Scale’s Data Foundry gives major AI labs access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you’re an AI researcher or engineer, learn how Scale’s Data Foundry and research lab, SEAL, can help you go beyond the current frontier of capabilities at scale.com/dwarkesh

To sponsor a future episode, visit dwarkesh.com/advertise.

I recommend all of these books.

• The Fate of Rome: Climate, Disease, and the End of an Empire

• Plagues upon the Earth: Disease and the Course of Human History

• Slavery in the Late Roman World, AD 275-425

Kyle is also working on a new book called The Last Animal.

(00:00:00) - Plague's impact on Rome's collapse

(00:06:24) - Rome's little Ice Age

(00:11:51) - Why did progress stall in Rome's Golden Age?

(00:23:55) - Slavery in Rome

(00:36:22) - Was agriculture a mistake?

(00:47:42) - Disease's impact on cognitive function

(00:59:46) - Plague in India and Central Asia

(01:05:16) - The next pandemic

(01:16:48) - How Kyle uses LLMs

(01:18:51) - De-extinction of lost species

Dwarkesh

Today, I have the pleasure of chatting with Kyle Harper, who is a professor and provost emeritus at the University of Oklahoma, and the author of some really interesting books: The Fate of Rome, Plagues Upon the Earth, Slavery in the Late Roman World, and an upcoming one called The Last Animal.

The reason I wanted to have you on is because I don't think I've encountered that many other authors who can connect biology, economics, history, and climate into explaining some of the big things that have happened through human history in the way you can. The most recent reason I wanted to have you on is I interviewed David Reich, the geneticist of ancient DNA, and some of the questions we were discussing, he kept emphasizing this overwhelming and surprising role that diseases have had in human history, not just in the recent past, but in his work going back thousands of years, tens of thousands of years.

He said, "You gotta have Kyle on." I emailed him afterwards, asking "Who should I interview next?" And he said, "You gotta have Kyle on".

You have this graph in The Fate of Rome.

You show human population over the last few thousand years. I assume that these two downspikes are both the bubonic plague, Yersinia pestis, right?

Kyle Harper

Yeah.

Dwarkesh

And so this is not like some small little nudge you can see, like the overwhelming —I mean, other than the hyper exponential growth in human population— the overwhelming, not just one, but the overwhelming two major features in human population going back the last 10,000 years, is this one bacteria, right?

Kyle Harper

Yeah.

Dwarkesh

One of the things you discuss in the book is that the collapse of the Roman Empire was a result of this one particular event.

Kyle Harper

The period that I normally work on is from the High Roman Empire, the glory days of the Pax Romana in the first or second century, through what we call the late antique or early medieval period, the sixth or seventh century. At the beginning of this period, Rome dominates this Mediterranean empire. It's what you think of when you think of ancient Rome. It's the largest city in the world. It's the center of this huge network.

By the end of this period, the city of Rome has 50,000 to 100,000 people. It's a tenth or twentieth of its former size. I think we now can say pretty clearly that environmental factors like climate, but also especially diseases, play a part in that really big transformation.

While there's a problem because we don't have the same kind of modern government mortality statistics that we do for COVID or even for the last century or century and a half, we have to piece together from clues. But it's pretty clear that the bubonic plague events, whether you're talking about the Black Death of the 14th century or the plague of Justinian in the sixth century, these events are capable of causing death rates, temporarily, that are just orders of magnitude beyond what we're accustomed to. Even in these ancient societies, the reason why these were so shocking, in a world where the death rate is always pretty high—probably several percent of the Roman population, three, four percent a year may be dying in a normal year—is telling. For them to just be utterly shocked by the death rate already tells you that it's some multiple of what they're accustomed to.

Dwarkesh

Yeah. I think you discuss in the book the possibility that the death rate might have been close to or even over 60% wherever the Black Death hit. This is literally the most significant thing.

Kyle Harper

Yeah. It's mind-blowing. In the case of the Black Death in the 14th century, it's pretty clear. It kills 50, 60% of the population in entire regions. We don't necessarily think that it killed 50, 60% of the whole continent, although that's actually not impossible. But even the fact that it's killing 50% of the people in cities, in provinces, in countries, is just beyond the damage that other plagues do.

Dwarkesh

Right. And do you think that were it not for this 60% mortality event plus the fact that we haven't even discussed yet, this super severe cold snap, do you think that the Roman Empire might have otherwise just kept going? You discuss there were these two previous other big pandemics. The empire still survives.

I think Will Durant had this quote that the Roman Empire fell for longer than most empires have lasted. Do you think it'd be similar to China maybe? Maybe a dynasty collapses, but fundamentally, the same sort of cohesive nation reemerges?

Kyle Harper

Yeah. That's a great comparison. Not just decline and fall of dynasties, but also geographic changes in the configuration, that parts get added and parts get cleaved off, but you still kind of think of it as fundamental continuity in the core. That to me is a very plausible counterfactual.

Justinian, the emperor in the 540s—he reigns from the 520s to the 560s—he’s on a path of success. He's retaken Africa. He's mostly retaken Italy when the plague hits. To me, a very plausible counterfactual is that a more or less Mediterranean core of the Roman Empire could have survived east and west. It does sort of survive in the east, but even including really all of Italy, Africa, and probably Spain. That would have been a very reasonable outcome of the sixth century if you hadn't had this kind of random shock.

The Roman Empire would keep going. Remember, it does, and it calls itself the Roman Empire until the 15th century. But we would think of it as maybe more the Roman Empire if it still included the western Mediterranean and was this major, powerful, urbanized polity that resists invasion from the southeast, as happens in the seventh century. So the answer is yes, I think that the Roman Empire absolutely could have had another turn as the thing we kind of mean when we say the Roman Empire: this pan-Mediterranean, powerful, urbanized empire.

Dwarkesh

Yeah. Okay, one of the things I found really interesting was you were discussing the firsthand accounts as this big... and feel free to explain the cold snap as well as it's happening. But the firsthand accounts of people who are experiencing this, some of whom come from this burgeoning Christian faith, which already lends itself to millenarianism and apocalyptic thinking. I'm curious basically how did different people try to make sense of this once in a 1,000-year event that's super intense?

Kyle Harper

Clearly, people have to try and explain, within the elements of a worldview that they have, how something like this could happen. They don't have modern science. They don't have germ theory. They don't think of it in terms of a biological event or climatic event. Since that's come up and you've invited, I'll say a little bit about that. This is one of the other really exciting frontiers where we're learning new things about the human past that we just didn't know 10 or 15 years ago.

In this case, we now have really cool paleo climate data that helps us understand that this period of the sixth and seventh century was also a period of really abrupt and significant natural climate change. We're all familiar with anthropogenic climate change, that carbon emissions stay in the atmosphere, trap heat. Humans are changing the climate. That's a big problem and we can talk about it if you want. I just want to clarify, that view is not incompatible with the reality that the climate does also change for natural reasons on every timescale, from long geological timescales to much shorter timescales.

We live in the Holocene. The last 11,700 years have been pretty stable, pretty warm. It's an interglacial. We're literally between Ice Ages right now. It's been really stable in the big picture. Yet even within that stability, there are smaller scale climate variations and climate changes. Because we need to understand how the earth system works, how the climate system works in order to be able to model what's happening, we need an empirical record of what the climate has done.

For historians, this is great news because now we have a huge number of sometimes even pretty high resolution climate reconstructions for historical periods across the Holocene. We now know, like we did not know this 20 years ago when I started graduate school, that the Roman period experienced some really abrupt episodes of climate change. In this case, the sixth century, we know the cause: there was a series of really significant volcanic eruptions. Volcanoes are a very powerful short-term climate forcing mechanism. They eject sulfur into the stratosphere. It aerosolizes and creates a reflective shield that scatters the radiation entering the atmosphere.

It leads usually to short term cooling. In this case, you had a series of really significant volcanic eruptions that cooled the climate for several decades. In some ways, the later series of eruptions even like a century and a half. It wasn't just a little bit cooler, it was like a degree to two degrees cooler, which we all kind of know now: two degrees isn't weather, this is climate. Two degrees doesn't affect your day, but two degrees globally is a pretty different globe.

All of a sudden in the late Roman world, it's much, much cooler. Probably areas that have been wetter are now drier. Places that are drier may be wetter. It changes the hydrological cycle as well, which is more complicated. In addition to the shock of the plague, you have this simultaneous and probably not unrelated shock to the climate system. We know that it was essentially challenging for agriculturalists. When the sun is blocked and it's really cold and the wheat doesn't grow, your society then starves.

The Romans get this wham bam double shock of climate change, famine, and plague. So back to how people explain this. Apocalyptic thought is one of the principle ways people frame it. To them, nature's going crazy. Huge amounts of the population are dying of this horrible sudden disease and the crops don't grow. You don't have microbiology and you don't have climatology. So you explain it with the resources of the worldview you have.

There's a huge burst of apocalyptic thought in the sixth and seventh century, which is always kind of there. You mentioned that Christianity is eschatological. It is, yes, fundamentally, but that comes out in different ways with different emphases in different time periods. This is a period, the sixth century, when there's a really sharp emphasis on eschatology in Christian thought.

Dwarkesh

I found your early chapters in the book about what the Roman economy was like in this happy period quite interesting. There's a bunch of questions I have about this. If you read Gibbon, writing in the 1770s, I think he says that the happiest time in human history was this period you're talking about. That was true at least according to him as of a couple centuries ago. This was still peak civilization.

You discuss the complexity of the Roman economy. The fact that millions of tons of wheat and other products have to make their way to Rome and the trade networks and everything. You basically say they were experiencing productivity gains. The wages were increasing, population was increasing, but they were still not at the level at which it was plausible that, say, for these climactic and biological factors, they might have had an industrial revolution. I'm curious why you think... paint a picture for me of what the Roman world looked like as of this happy period and why that still counterfactually couldn't have just saved us a thousand years of history if they were on the right track.

Kyle Harper

First of all, I think this is the sort of question that historians ought to worry about all the time: why didn't the Roman economy catalyze the takeoff? In some ways it was so precocious for its time period, and it seems not utterly impossible. The Roman world is still a pre-industrial economy so agriculture is the dominant sector. The majority of people work in agricultural pursuits and productivity is low. They don't have modern mechanized traction. They don't have modern synthetic fertilizers. They don't have the modern green revolution yields, all the things that have made agriculture stupendously productive.

The primary sector is fairly limited in terms of its productivity because of the limitations on technical inputs. We can think of the inputs to an economy are going to be capital, labor, and ideas. What the Romans have is people. They have some investment, but they don't have technology. They don't have ideas. It's a late Iron Age civilization. I do think there's productivity growth and that productivity growth comes from markets, from trade where you get comparative advantage.

In Egypt, I'm really good at growing wheat. You can make glass in Syria. And then we'll trade. The urbanization of the Roman world certainly facilitates that. Cities are these hubs of productivity and exchange. There's some technology. If you look really hard over five or six centuries, there's certainly economies of scale where the production process and manufacturing is moved from artisanal to industrial scale. But there's no takeoff because they don't have science.

They don't have research and engineering that drives continuous productivity gains. I think they go precociously far in a pre-industrial setting where you take trade really far. They have good institutions in terms of strong property rights. There's relatively reliable contract enforcement. There's financial markets. They have the most advanced financial markets in the world before the 17th or 18th century. There's impersonal financial intermediation.

It's not like you have to know me and come ask me for a loan if you wanna build a ship and go trade something. There are banks that take money from depositors, keep balances, and then lend out to debtors who want to go and do entrepreneurial things. They have so much potential, but there's no spark. You never see these sustained productivity increases. I would just say ultimately it's because the Romans don't have technology improvements that are really self-sustaining.

The reason they don't have that is because they don't have science. Their science sucks. I'm offending some of my colleagues, I'm sure. Galen is great. Ptolemy's incredible. I love Pliny the Elder's encyclopedia, but if you look in the big picture, the contribution that the Roman Empire makes to our knowledge of how nature works and then the applied technology that comes out of that is really pathetic for five, 600 years. They go as far as you can with Smithian advantages to market exchange and specialization, to banks and finance. But without the kind of creative destruction of new technologies that improve productivity, you're eventually gonna run out of improvements.

Dwarkesh

If you're Augustus or some other Roman emperor and you're like, "Look, we've got this big good economy, but I wanna see productivity gains," and you wanna make it happen somehow, is there something from a top down perspective you could have done? In Britain, the government subsidizes the royal arts and so forth and the Longitude Prize.

Kyle Harper

That's exactly what I was gonna say. This happens first in France and then Britain, but you get royal societies for science where you're doing really... I would say there's like three things that are essential there. One is the promotion of what we would call basic or fundamental science. It doesn't all have to be immediately practical or commercialized. But you're promoting deep knowledge of nature.

Two, you're doing it in an empiricist way. This is something very important in the 17th century that the Romans by contrast don't have, is the spirit of Francis Bacon that we need to ground our knowledge in experiment and observation, not just believe whatever authorities or Aristotle said. That's very much the spirit of places like the Royal Society: we don't take things on anybody's word, especially Aristotle's. You need basic science. You need empiricism, rigorous and self-correcting.

Third, you need a sense of useful knowledge, and that's the other thing that really comes together in the 17th century: not just the basic and abstract science, but the application, and the 17th century language for that is useful knowledge. That is something that doesn't ever get wired together in the Roman Empire. There are tinkerers and engineers, but they're not talking to the mathematicians and the physicists. If you were from on high to design self-sustaining innovation, I think you would want to bring those elements into proximity. Augustus, unfortunately for them, didn't do it. Probably good for the world. The Romans are pretty nasty people in a lot of ways.

I definitely am of the opinion that the high science matters, that Isaac Newton is not a tinkerer. He's not building pumps. But the guys who are are his friends. They're in and around the Royal Society, and they're absolutely... Look at Denis Papin, who was a French engineer who was very much in the circle of Leibniz and the very high abstract mathematics, trying to build vacuum pumps.

The proximity of high math, high science, very abstract with what is ultimately gonna be the sectors that lead to mechanization where then you can harness this source of energy that is there all along but hasn't been tapped in coal. That's what catalyzes the big positive cycle of positive feedbacks. What the Romans don't have is that. What the Romans do have is the kind of specialization, and now that we look for it, it's there. When you look at food processing, which is a huge sector, the way that they built mills, there's definitely improvements. But there's never the catalytic change where you get runaway positive feedbacks

Dwarkesh

That's right. A previous guest of mine, Nat Friedman, I don’t know if you saw this, launched this challenge called the Vesuvius Challenge. This library at Herculaneum in 79 AD was buried under the ashes of Mount Vesuvius when the volcano erupted; speaking of volcanoes. Now they figured out with modern techniques how to read the burned scrolls and it is supposedly the biggest library of classical text ever. It would double the amount of classical text we have. As a scholar of ancient Rome, what would you personally find most fascinating? What are you excited to find from this data?

Kyle Harper

I'm super interested in the history of math. What happens after Euclid? It's very hard to say because you get these really interesting people that pop up, like Diophantus who's later, in the early Roman Empire. There's still really interesting math going on. Euclid is incredible. The Greek experiment in math and science is the one that I think had the better chance of sparking sustained takeoff.

And it didn't. It'd be interesting to know more about why. Why did things stall? Because these people... Euclid is not just a towering genius who comes out of nowhere. He's very much a product of the culture and the questions that are being asked in the generations before. It just sort of feels like after him, you fail to get that kind of sustained continuous progress and advance. Maybe back to that big question that we were asking before: what prevents the kind of breakthroughs that we see in the modern world?

Dwarkesh

What is the population of Greece during their golden age?

Kyle Harper

Of the greater Greek world or individual city-states like Athens? We think of Athens as being a couple 100,000 people. Not massive.

Dwarkesh

So I wonder if the Greeks had the science but not the people to sustain a modern economy. Or not modern, but even a sort of industrial economy. And then the Romans had the people but not the science.

Kyle Harper

Yeah. There's probably something to that, that there's just not the critical mass of educated people, of sheer cognitive power to keep it going.

Dwarkesh

Let's turn to another one of your books, the one about slavery in the Roman world. I did not realize before I read that one how much Rome was a slave society. I guess that just isn't a salient thing in a conventional understanding of Rome. Why don't you paint us a picture of how much slavery was involved in that world?

Kyle Harper

Slavery tragically is a really important institution throughout history. We sometimes tend to think of it as a distinctly modern phenomenon, but that actually misses the deeper picture. In fact, it obscures the importance of modern slavery because modern slavery is uniquely important and it's uniquely tied up with certain kinds of market exchange and certain kinds of production, certain kinds of racial ideologies. There are things about modern slavery that are really important to understand are different, but not just because slavery is there.

Slavery has this longer history and slavery is more important in some societies than others, and we want to try and understand that, to ask why and then what implications does that have for understanding those societies? Rome is one of those societies. Slavery is really a prominent institution in Rome from the late republic. As the Romans conquer other parts of the Mediterranean, they start taking captives as slaves en masse, and they build an economy that really relies on slave labor in important sectors of the economy.

Plantations where commodities like wine, olive oil are produced for market exchange that allow landowners to amass enormous amounts of wealth. Slavery becomes this really important institution that's entangled in the development of the Roman economy from maybe the third or second century BCE, and then with ups and downs and really important changes along the way for centuries and centuries.

Dwarkesh

So you're pointing towards from the supply side, all the Roman conquests lead to all this surplus labor that they can make use of, and on the demand side, these cash crops.

Kyle Harper

Exactly. I'm very big proponent of the idea that you have to have both. You have to have a source of slaves. After the conquest stops, the Romans figure out other sources of slaves. If anything, the demand is equally or perhaps even more important because if there's not a mechanism, if there's not institutions that let you turn this kind of exploitation into cash flow, the institution's not gonna go very far. It really is the institutions, the presence of markets where you can take labor and turn it into profit that's the most important element.

Dwarkesh

One of the things I find interesting is in the age of colonization we're used to thinking about slavery in terms of race but also religion and other things which more obviously demarcate free and slave populations. In the Roman world, it doesn't seem that that's clearly the case, yet there's no abolition movement the way that emerges out of England in the 19th century or maybe even before that.

The reason that's mysterious is, if you were literally descended from slaves, if you were like, "My grandfather was a slave but then we were freed," and they're basically just like you, you would think that there would be more of a sense of... not everybody would be an abolitionist, but at least some people would be writing about abolition. With Christianity and so forth burgeoning, they didn't seem to have a problem with it. Why is there no abolition movement despite the heterogeneous nature of the slave population?

Kyle Harper

It's sort of disturbing in a way, isn't it, that humans have the ability to convince themselves that it's okay to own other human beings as property through a variety of different kinds of ideological justifications. You see even in the ancient world there are different models that people use to say that slavery's okay. Aristotle develops a theory of natural slavery that actually some people deserve to be slaves by their very nature and that it's actually good for them to be in bondage.

What's really interesting though is that that doesn't actually ever seem to be the dominant ideology. The Roman ideology of slavery is not racialized. It's not like the Romans think that the Greeks or the Germans are some fundamentally separate kind of human that justifies their exploitation. The Roman ideology of slavery is really rooted in the law of property and status. They think that slaves are people who've been conquered and rather than killed, they've been spared, and they've been sold into the condition of being somebody else's property.

This seems to mentally explain to them where their slave system comes from and why it's justifiable. You have different kinds of criticism of the slave system from within, but remember, most of what we have written is from the slave ownership class. I don't think the slaves were themselves believing this ideology, and there must have been sort of what we would think of as abolitionist movements or spirit that we just don't have really good records of.

It is this curious thing that the Romans are able to build this huge system that's really brutal and really violent, but has this kind of flimsy ideology where they tell themselves these stories. The deeper lesson of that is that humans can create these systems of belief that will exclude others and justify almost any form of exploitation and convince themselves that it's okay.

Dwarkesh

I hope your next book about The Last Animal discusses the potential parallels with factory farming.

Kyle Harper

There's some... there's a pretty gruesome chapter, I'll say that.

Dwarkesh

I don’t know if you mentioned, what numbers you say, I think it was like 10 to 20% of the population under the Roman Empire was enslaved. Given that large a size of a slave population, it's surprising to me that there are so few slave revolts not only in Rome, but even throughout history. There's Spartacus in 71 BC, then there's the Haitian Revolution. If 20% of the population is enslaved, how is this sustainable? If you're running a farm and there's 4,000 slaves and then the next farm over also... why aren't there more slave rebellions?

Kyle Harper

Why not? How did they do this? They have a really elaborate system of repression. They're worried about it. Probably the parts of Roman society where there are 20, 30, 40% slaves are pretty limited to certain regions and certain time periods. Partly because once you cross some kind of threshold, the challenges of repressing direct violent resistance increase.

It's a system of exploitation. That means there's always a mix of carrots and sticks, to put it crudely. The Romans extract people's labor partly through physical violence, but also partly through systems of manumission that try and incent people to obey and not to rebel in order to earn their freedom. They're using everything from literal chains to enticements to try and keep rebellion from ever coalescing in a way that can turn into collective violence.

It's a little bit challenging for us to look back. We know in Pompeii, the slave population is huge. It must be 30%. Not all of these people would've been plantation workers who were lashed every day and worked to the physical bone. A lot of them are nurses and textile workers and maids and tutors and all sorts of things that are sort of quasi-embedded in households as well, where there's always this weird psychological dimension too. Part of the strangeness of slavery is how deeply embedded in domestic institutions it is as well.

There are ideologies in which the paterfamilias is sort of the father and the master. That tries to brainwash people against resistance. The important thing to recognize is it's just a pervasive system that tries to colonize people's minds and pervasively tries to keep them from resisting.

Dwarkesh

I wonder if we can close the loop with the question we began with, which is, why didn't Rome have an industrial revolution? I don't know if it's a plausible explanation that cheap slave labor reduced the incentives for mechanization and engineering and other crafts or if not. I don’t know.

Kyle Harper

It's definitely an argument that's been made. Aldo Schiavone was an Italian historian who argued that. It's kind of a neo-Marxist tradition that argues this. It's an interesting argument. I don't buy it at all. The good version of that argument would just be that the Roman Empire is using slaves in many of the most forward elements of the economy too. We tend to think progress and economic growth and innovation is good, and we know slavery's bad, so we tend to think that those things don't go together.

But in reality, it's the most economically advanced sectors of the Roman economy that had a high degree of organization and productivity that tended to employ slaves. In the Roman world, you could make the argument that if the labor in those sectors had been free, there would've been more opportunities for positive feedback loops. The way the argument's usually made is just that the Romans got rich without really thinking about productivity. They just wanted to extract labor, extract wealth rather than create wealth.

That's not a terrible argument, but ultimately I don't think it's the system of labor that keeps the Roman world from industrializing. There are lots of sectors in the Roman world where slavery is not a dominant institution, and it's not like they're more productive or flirting with some kinda breakaway. So it's an interesting argument, but not one I've ever found all that persuasive.

Dwarkesh

Final question about Roman slavery. What did Gladiator get right and wrong about? Would they just abduct you in front of your house and make you a slave?

Kyle Harper

You mean the first one? The first one got right that when you're making a movie you should worry more about making a good movie than a certain amount of history, but actually the first one's a great movie. If it was completely historically accurate, it would've been much more boring. So I'm not gonna be critical of that movie. It plays very loose with the facts of high politics around Commodus and the creation of this character. But who cares? Russell Crowe's incredible.

Dwarkesh

But on slavery in particular.

Kyle Harper

On slavery in particular, I think actually that's one of the strong suits of the movie. You see this completely exploitative system that brings people from very different parts of the world who have very different backgrounds. The system of urban spectacle is very real. The use of slave labor in that is certainly a part of it. So the movie actually gets some really important things about that right. That makes it totally forgivable that it has to create a kinda high politics storyline.

Dwarkesh

Okay, I think that covers all the questions about Rome. We can get back to your most recent book about human history and plagues.

What do you make of the general argument that people have often made that we were living in a sort of Eden before agriculture? Especially given you've explained that all these diseases that we're sorta stuck with are actually quite new. If we take that perspective seriously, was life before human population exploded and we had agriculture just much more pleasant, at least in comparison?

Kyle Harper

Homo sapiens is 200,000 to 300,000 years old. We emerge in Africa and disperse, multiply, but we spend 90, 95% of our history as foragers. People who are hunter-gatherers, who take energy from wild food sources, rather than sedentary farmers who've domesticated plants and animals and live a sedentary lifestyle where you're enslaved to this wheat or rice, but it gives you reliable calories. That is along with the Industrial Revolution, and then whatever this thing we're about to go through, the biggest change in the history of our species.

The shift from foraging to farming affected everything. It affected our beliefs. It affected our genetics. We're all basically genetically different, adapted to live in a different kind of environment with different kinds of diets. It affected our societies, it affected inequality, it affected culture in every possible way, and of course it affected our health in really basic ways. It affected our labor regime; doing the same kind of labor over and over every day is very different from running around as a hunter, chasing deer or whatever, which sounds quite nice.

It changed our labor regimes. It changed our diet, most of all. Hunter-foragers tend to eat high protein, high fat-ish diets, with no refined carbohydrates, but limited carbs. It's a very varied, highly varied diet. Sedentary farmers tend to eat more monotonous diets and they tend to be dependent on grains and starches, so a very narrow spectrum for your calories. Changes in labor regime, changes in the diet, and then changes in lifestyle, being sedentary and living in big populations that then puts you in proximity to other humans, puts you in proximity to human waste. Feces are a major conduit of infection. It puts you into proximity to the air they breathe, which is conducive to respiratory diseases.

This transition, which takes thousands of years, is more of a process than an event. It has massive implications for human health, including the infectious disease environment that we inhabit. It's not like hunter-gatherers were living in paradise. The infectious diseases that they had were seriously burdensome, they sucked, and probably most people died of infectious disease. Malaria is a really old disease. Lots of diseases existed in the Pleistocene, in our Paleolithic past.

It's not like it was Eden. There is this idea that the transition from foraging to farming... Jared Diamond called it humanity's biggest mistake. Certainly these changes entailed some things that were not net positive for humanity and one of them is that it definitely increased the infectious disease burden. Simply as our population multiplies and as we're in contact with feces and as we're sharing the air through which respiratory pathogens can spread, diseases are constantly trying to take advantage of this. That's just how nature works. Energy is scarce. Everybody's trying to steal it from everybody else, including microbial parasites.

The disease burden of humans over time definitely increases. The burden of infectious disease on humans goes up over time. Very broadly across these thousands and thousands of years, the diseases suffered by people by the time of the Roman Empire are absolutely much worse than what had been the case in Stone Age times.

Dwarkesh

James Scott has an interesting theory in Against the Grain. I don't know if it originates with him, but he argues that one of the reasons that the early agriculturalists were so successful... And David Reich, if you've seen his stuff about The Yamnaya 4,500 years ago conquering all of Eurasia, but before them, the Anatolian... the initial farmers are the ones who displaced the initial hunter-gatherers across Europe and Asia.

He argues that initial wave was so successful because of these first diseases that the farmers had created the conditions to engender. Basically, the relationship these farmers had with respect to the foragers they were taking over from was similar to the relationship the Europeans had to the Native Americans, where inadvertently, the disease is just a significant player in why you were able to dominate them. I don't know how plausible you find that.

Kyle Harper

I mean, first thing, it's important, I think you were starting to get at this, that there's never a generation of humans that has the opportunity to make this choice once and for all, like, should we stay hunting mammoths or should we become sedentary farmers with basically torturous dentistry and die by diarrhea? This happens over thousands of years through an evolutionary process where nobody can... It's a story of unintended consequences. The mammoth are gone partly because we killed them all.

People start... the first livestock that are domesticated are goats. Nobody says, "Hey, let's become goat farmers." The goats are wild. They're ibexes. People are hungry, and so they start managing them to only kill the males to make sure that they can reproduce, and they start penning them and they start killing the wolves who are trying to attack them. Over very long periods of time, this becomes this tight mutualistic relationship where all of a sudden we're goat farmers. But no generation makes that whole decision for anybody.

That's part of it, is that it's unintended consequences that are made in very incremental steps. Two is, I definitely agree that there's some kind of cultural selection here where the farming groups are simply so much more adapted to extract energy efficiently from the environment. It's all about energy. You want to multiply, you want to grow, you want to survive, it's all about energy. Foragers require huge landscapes to extract enough energy to feed themselves and grow and reproduce, whereas farmers per unit of land can extract such higher rates of energy that then can be, through photosynthesis, turned into edible sugars that we can metabolize.

Those populations are just growing faster, that they quote unquote out-compete the hunter-gatherer population say of Europe that are largely but not completely displaced. Now, on top of that, just the energy story alone is a big piece of it, but then on top of that, you probably do have some kind of population difference in the exposure and possibly even immunity to infectious diseases. I definitely think that early farmers, the first farming societies that are starting to live sedentary lifestyles where you have aggregations—these are not cities, these are villages—but still, that's more than a hunter-gatherer band.

Your childhood is then going to be constant exposure to a series of pathogens. Those kinds of populations, when they're then migrating into Europe, are probably carrying these pathogens with them that may have had a kind of further effect that on top of just being able to extract more energy and multiply faster, drives up the mortality of the existing populations.

Dwarkesh

Yeah. The point you made about fertility is interesting. I vaguely remember reading that it's not just the fact that the energy density is lower, it's that you're also moving around a lot. Because of that, you're spacing out kids much more so than if you were just in the same place. I think the actual fertility for foragers is sort of like reasonable—I don’t know if sustainable is the right word because I don’t mean in an ecological sense but more so—it keeps your population constant.

Kyle Harper

Yeah. Don't make me swear, but it's like more like four than six. Because women who are moving with the foraging bands miles and miles on foot on average a day, and also carrying kids, are gonna have very different life history than sedentary populations. That's very clear.

Dwarkesh

One thing I'm really curious about is what effect these diseases through history have had on the cognitive functioning of people. You discuss this in the chapter about more recent history of the great divergence and probably attributed the productivity of Europe that they were able to have public health earlier. But literally going back thousands of years, you mentioned, for example, that Caesar was 5'5" and that was considered tall during his period. Did the same diseases and malnutrition that caused these physical health effects also mean that the average IQ was much lower, because when you're a kid you're sick and that steals away nutrition from brain growth or something?

Kyle Harper

Yeah. Short answer, yes. Long answer, we know that in the modern world, say over the last 250 years, first in Western European societies and their settler offshoots and then more globally and more rapidly globally, there have been really deep physiological changes in the average human. We're talking about populations with distributions. What's happened is really two things. One is there's more energy per capita. People eat more, they eat more calories, and they eat better calories. They eat lots of bad stuff too, but people eat more. Two, the burden of infectious disease has been lowered.

Growth for a human is a very complicated trait that's influenced by genetics. I was never destined to be super tall. But it's also affected by environment, which includes nutrition and what you spend either doing labor or what you spend fighting infectious disease. Infectious disease imposes a huge burden on the body. The immune system is extremely metabolically expensive. If your childhood is spent just fighting infectious diseases, you're going to struggle to invest energy in growth.

There's massive increase in the size of populations over the last 250 years and even though it's an even more complicated trait, this improves people's cognitive abilities. People are smarter. May not feel like it—I think it has rapidly leveled off—but people are more intelligent today than they were 100 years ago. Their brains are better nourished and their bodies spend less time fighting pathogens. I think there's no doubt that pre-industrial populations, and again, populations—so you still have your Isaac Newtons, who whatever infected him as a kid, didn't slow him down—but at the population level, I think there's no doubt that not only were pre-industrial populations shorter, this is just a total fact that we know from their bones, but they probably also, on average, had a lower distribution of cognitive abilities. But with a big distribution.

Dwarkesh

You have a great profile in the book about living in London in the 18th century and just how disgusting it was.

Kyle Harper

It was pretty disgusting.

Dwarkesh

But at the same time, in that city, you were just mentioning, there are these scientists and people with towering intellects who were basically figuring out how the universe works and how to make all these machines and so forth. One answer is just like what you just said: the distribution was lower, but maybe Newton would have had an IQ a standard deviation higher if he was born today. Just seeing that from the small population, you're seeing so much genius. I guess the question is, how could you have had this much of a deleterious impact on cognitive functioning and still had enough spare geniuses to kick off the Industrial Revolution?

Kyle Harper

Obviously, it didn't keep them from discovering some pretty amazing things. So it couldn't have been completely destructive. That's what's interesting about the early modern period in the 17th, 18th century, in particular, is it's sort of this between period where you have the pre-industrial and the modern that are still mixed together in these really interesting ways. The example I use in the book is the very famous diary of Samuel Pepys, who's this incredible figure and is very close to Newton and that social group and his name is on the first edition of the Principia. These people are this close to each other.

But the stuff that I evoke, I won't say—this is a family podcast—but the stuff that Pepys does, bodily functions, is mind-blowing to us. It's vile and disgusting. But at the same time, right down the way, you've got people who are making the most fundamental discoveries about the nature of the universe and inventing machines that will improve productivity and ultimately economic output. That's what's precisely so weird and interesting about that particular period is you have this kind of mingling of the old and the new.

Dwarkesh

When I had Joseph Henrich on, one of the things he discusses is if you look at... cultural evolution has figured out some remarkable things. If you look at the cuisines of different cultures, apparently the spices they use match the antimicrobial and antifungal properties you need in that particular biome. At the same time, you’re reading that part of the book, I'm like, "Okay, I get in some cases, they just genuinely did not have the resources to invest in public health and so forth". But come on, you're just like sleeping in your own vomit and so forth. Why didn't cultural evolution or something like foresight just be like, "Hey, this we can sort of do without"?

Kyle Harper

It's a deep question and what I think we don't think enough about is how, in a really fundamental way, hard are some problems to solve? Some problems are just very, very hard to solve. Even though the incentive is really there, you think, "Ugh, that took a really, really long time to figure out". Even though if you'd only known, it would've made your life so much better. There's tons of trial and error.

The example that comes to my mind is the mention of vaccination. Which is one of the great human achievements, like, of all the public health improvements is the most important one. Public health is never perfect. It's this system of six or seven really critical tools that involve clean water, personal hygiene, vaccines, antibiotics, different kinds of therapeutic interventions or rehabilitation therapy. We still can't fend off all the germs. You have to have all of that and you can sort of achieve this equilibrium state where you mostly have it under control.

Vaccination's the most important one and it took forever to find the first vaccine. It took this huge period of all kinds of weird trial and error, like inoculation with the actual smallpox, which is very, very dangerous. Not vaccination. Vaccination uses cowpox, the lymph of an infected cow, to intentionally cause the immune reaction of humans. Before that, people would inoculate a person with actual smallpox, which is just giving somebody smallpox. You do it through the skin, but you're giving somebody smallpox.

It was absolutely in a utilitarian way, the rational thing to do. It had these horrific death rates. It would never get FDA approval. But in a world where 10, 20% of kids die of smallpox, it's this horrible decision, but you'd be rational to do it. We actually don't know where that comes from. It may come from Africa, it may come from China. It spreads for like a century or more before Jenner discovers vaccination. So it's clearly really hard to figure that out.

Even after Jenner, it's like another 60 years, 70 years before Pasteur kind of systematizes it and says, "Hey, we could do this for everything". Some of these discoveries and innovations are really, really hard to discover. But then the beauty of cultural evolution is that we can store that information, and you and I don't have to figure out any of that. We can go on to the next problem because that's now been collectively stored in the library of cultural evolution. It's known, we don't even think about it most of the time until it's controversial. What a blessing to live after people like Jenner and Pasteur who figured that out.

Dwarkesh

There's this great blog post by the author Slime Mold Time Mold where it's discussing....

Kyle Harper

Wait. Author what?

Dwarkesh

You don’t know internet culture, you know, there’s a bunch of weirdos out there.

Kyle Harper

I'm in a different world, sorry. What did you call them?

Dwarkesh

Slime Mold Time Mold.

Kyle Harper

You can't just drop that. Like I'm gonna let that one slide. Okay. I got some homework.

Dwarkesh

Anyways, they have a blog post about scurvy and why it took so long to discover, and he was discussing all these sort of... it's sort of an epistemic conundrum because you can use lime and you realize, "Oh, it works". But then if you use lemons, which have much less vitamin C, or maybe it's the other way around, they just work way worse. Then there are certain kinds of fruit which have vitamin C, certain kinds which don't. It's actually hard to figure out what is it if you don't have a mechanistic explanation about how you solve this problem, and I think they had once figured it out and then they lost the knowledge until it was rediscovered again.

But it makes it all the more mysterious that the kinds of things that Henrich discusses forager societies having figured out. Literally, there's this 10-step process for how to process a certain kind of bean so that you don't get cyanide poisoning, and if you mess up any one of those 10 steps, you're gonna get cyanide poisoning. A society just figures out the right taboos and traditions to process beans. But you can figure that out, but this thing which is causing 20% mortality, you only get in the 17th, 18th century.

Kyle Harper

Yeah. We need to think more about the computation that's happening. You said it takes like 10 steps to figure out how to process this one particular kind of food, but I'm guessing it is just really hard to figure out infectious disease. It's a really steep mountain. Once you get up to a certain plateau, then the discoveries come really fast. They become systematic, and they become more fundamental. But it was really hard to get there.

Not that many societies really scaled it, not even within the societies that did. It was just a handful of people at first, but they did get there.

Dwarkesh

Okay, and then asking about where different countries were at around this time. What evidence do we have about what was actually happening in India before the British or the Mughals because it does seem to be this sort of black box in terms of historiography? Do we know if there were these huge plagues?

Kyle Harper

Yeah, it's such a tricky problem. Start with the third plague pandemic in the late 19th century. We know that that's in India. India's a big part of its history. In fact, its where the plague bacillus is discovered by Alexander Yersin. It's called Yersinia pestis in his honor. A Japanese scientist finds it exactly the same time. Gets left out of the nomenclature.

Dwarkesh

It’s a special kind of honor to have the deadliest agent in history.

Kyle Harper

To be the worst pathogen ever; immortality. The plague is definitely in India in the 17th century. We know that from contemporary written records that are pretty unambiguous about the presence of the disease. What we don't know is was it there before that? And if not, why not? Because it kind of actually seems like it's not. At least not in this same explosive way. That's pretty curious.

We don't have a great explanation of that because India's connected to the Central Asian world where the plague is endemic. There's plenty of trade. It would have plenty of chance to move to the subcontinent. So we don't understand that. If you go back even further, that's the Black Death. You go back even further to the late antique period, it's like a total mystery, and the Indian sources from the fifth and the sixth century are not great.

They're hard to use. This is totally outside my language abilities. They require totally different expertise. I've worked with some people who think that there are oblique references that may be interpreted as epidemic. One of the interesting things is we actually think that the plague moves through India to get to Rome.

This is not definite, but the plague's enzootic, its natural animal reservoir is the Tian Shan Mountains where China, Kazakhstan, and Kyrgyzstan meet, and we can actually identify a pretty small region where the pandemic lineage comes from. We know that it doesn't go overland, so it's not like the Black Death, which goes across the steppe, the Mongol trade networks, military networks carry it. In the sixth century, probably, the plague goes south through India, and maybe the ports in Gujarat or along the West Coast that are still pretty connected with the Roman world, with East Africa, with Arabia, with the Red Sea, that the plague travels on ship across the Indian Ocean, because the Plague of Justinian shows up in the Red Sea.

That is a clue that it probably is imported on this seaborne commerce. But how it got from Central Asia to Gujarat is a hard question.

Dwarkesh

Huh. I know the way we found that the Yersinia pestis existed in these Yamnaya 4,500 years ago is by... Didn't they just find the...? I don't know how, but if you can figure that out, why can't you look at the fossils of people 500 years ago or 1500 years ago and just see if they have Yersinia pestis in them?

Kyle Harper

There's two things. First of all, you have to look. There is at present not nearly the same amount of ancient DNA laboratory work that's happening on remains from ancient India. So if you're not looking, you're definitely not going to find it and people aren't looking.

Secondly, it takes a lot of luck for it to preserve. The DNA molecule starts degrading the second you die, just starts falling apart. Even in the best of cases where we're getting it from... usually, if it's pathogen, you're getting it from the dental cavity. If it's human DNA, you're getting it from the inside of the skull. But it takes a lot of luck for it to preserve because the soil conditions will affect the degradation, the temperature will affect the degradation. Just in a crude sense, heat is bad. That's why there's more DNA, ancient DNA, that's preserved at more northern latitudes so far. But it has as much to do with the fact that people aren't looking. We should be looking, and if you've got skeletal materials from an ancient mass grave in India, call me. We can definitely look.

Dwarkesh

And just to be clear for the context-

Kyle Harper

Ancient, ancient, ancient.

Dwarkesh

Okay, going forward to the future a little bit, speaking of future technology, maybe the one that's more relevant than AI is synthetic biology, and there's a worry that you can potentially create diseases which... maybe the evolutionary gradient is one that is not catastrophic where diseases are incentivized to be transmissible but keep you at a chronic level of infection that doesn't necessarily kill you immediately. Actually, it's interesting why the bubonic plague diverges from that selection pressure, which maybe you can answer. What do you think about the potential that with synthetic biology people can make diseases that have the transmissibility of measles, but also the deadliness of something like Ebola? Is that, given your understanding of biology and whatever, is that plausible given your understanding of biology?

Kyle Harper

Let me start with the plague where I'm a little more comfortable and can say something as a knowledgeable person. I think it's relevant because you said it's weird that the plague seems to sort of evade some of these evolutionary constraints, and it's worth saying what these are. A pathogen is a disease-causing organism, a microbe, usually a virus or a bacterium, but also fungi and single-celled organisms like protozoans that cause disease in a host. They're not trying to cause you disease. COVID doesn't hate you. Plague doesn't hate you. It's just evolution. It's just trying to steal energy or hijack your cells to reproduce its genes.

In fact, it has incentives to try and do that as well as possible while doing the least possible damage. It's always kind of trying to thread that needle or to find the right balance, because if a pathogen just kills you instantly there's nothing to steal and it can't transmit its genes into the next generation. Every pathogen has these basic evolutionary problems. How do I get from one host to the next? And how do I evade my host's immunity, which our immune systems are incredible, for long enough to multiply?

Most pathogens have to explore the space where there are these various constraints, and they find all sorts of weird ways around it. Evolution is really good and really creative, unfortunately for us. The tricks that they find to hide inside your immune system or to fake it out are really wild. There are two reasons why plague is so weird, and we don't completely understand why plague is so weird, but I think there's two basic reasons. One is that it's vector-borne, which means that it's transmitted through another organism that is the intermediate.

Arthropod or insect vectors are really annoyingly helpful to certain pathogens, and most... there's actually a relatively small number of diseases that are transmitted through a vector like this, but they tend to be really nasty, like malaria, typhus. They can kind of get away with it because even if you're dying, a mosquito can come and bite you and transmit malaria to me. Plague is a vector-borne disease. It's very well adapted to transmit, particularly by fleas, but we think also maybe by lice and other biting organisms, but really by fleas. It's really good at transmitting by fleas.

That's evolution. This is one of the cool things with ancient DNA we've been able to piece together at the absolute molecular level, the genetic changes that let it make this protein that have this effect in fleas. It's really weird, it forms this biofilm in the gut of the flea that chokes it and makes the flea feel like it's starving. So the flea just starts feeding and feeding and feeding, and meanwhile it's regurgitating bacteria.

Dwarkesh

Sorry, can I ask a question about that? Why is it the case... 'Cause there are diseases that hijack the flea's mind or ants' minds or something. Why isn't there a disease that makes humans zombies? Is it just the human brain is so complicated that it's like...?

Kyle Harper

Let me come back to this. We can talk about zombies, but we need to wind up for that. Okay, so one, flea- So the plague is vector-borne, and it's really good at manipulating the fleas, and it's just evolution. Two, I said this before, but it's an animal disease. We're like collateral damage. We're totally irrelevant to the really core evolutionary history. The plague just wants to infect rodents. Of course, I'm... it's not really wanting to do this. The plague makes a living, it survives out there in burrowing rodent colonies.

We're like tertiary. It doesn't care at all. It has no evolved incentive to modulate its virulence to be able to transmit sustainably. Plague never sustains itself in human populations. It can transiently infect human populations, but then it always dies out. It becomes extinct, that lineage.

Dwarkesh

And then what is the reason that you have these 1,000-year cycles basically? Why is it not 500 years? Why is it not 10 years? Why is it not… What causes it to go dormant? What causes this to reemerge?

Kyle Harper

You need to ask me in five years because we've learned so much and now this is like the thing that would fall in the category of almost a new question now that we can ask. Now that we have the Neolithic lineages and the Bronze Age lineages, we're starting to piece together this fuller history. But we still don't even totally understand the boundaries of when the plague is really sort of not circulating in human populations, and what are the factors that cause it to be so explosive. Like, is it evolution of the bacterium? Is there something about the genetics of the lineages that escape from the animal reservoirs that are especially transmissible? Is it human ecology, like, that we put rodents like black rats in the right place to get the disease? Is there something about the climate stress that renders the population…? We don't have a great understanding of like why the plague comes and goes.

That's scary. Connecting it to your other question about these superbugs, what's interesting in the very big picture about the plague to me is the history of infectious disease is like, on the one hand, there's a real core of it that's just basic principles of ecology and evolution. We do certain things in the environment that creates the conditions that pathogens can evolve and take advantage of. But on top of that, evolution is just creative and weird and contingent and unpredictable. It's those little, contingent facts that can end up having these really huge effects.

In the case of the plague, if you were really knowledgeable about the basics of ecology and evolution of disease, you would never be like, "I think that every now and then a rodent disease from Central Asia is gonna wipe out half of the continent". Like, that shouldn't... that's not predictable. That shouldn't be happening. That one's kind of an outlier, but infectious disease is always kind of like that. Tuberculosis has probably killed more people maybe than any other infectious disease. It's like this horrible disease. We don't really understand it. Now we really don't understand where it came from because it doesn't look like it has an animal host before it has humans.

It's just a weird disease. It's just a bacterial pathogen that, in the huge world of bacteria, this one is very good at hiding. It gets in your chest and it just lurks. Then it'll just waste you away, particularly if you're poor and you're stressed. There are some core principles there, but then it's just something weird about it. It's just this terrible luck that makes it what it is. To me, there's going to be another pandemic, maybe bird flu, maybe something else.

But it's the real outliers and the weird ones that we should maybe worry about a little bit more than we do. If you want to go to zombies, I'll go there. You don't have to twist my arm too hard. But like prion diseases or fungal diseases where we don't have nearly the same infrastructure and level of knowledge, biomedical research as we do for bacterial and viral diseases, if we create the incentive, evolution is gonna find some weird ways to exploit it. It's not just transmissibility and virulence. Those are like two really basic parameters.

When you look at even COVID-19, part of what made it insidious is it just has just the right parameters to be latent for just long enough. The first COVID, SARS-CoV-1, 2003, slightly more virulent, and in fact, it was just more virulent enough that it made you sick pretty quick. Just that little difference was enough to contain it because you could figure out who was sick. COVID-19 was impossible to contain because it took several days before you really presented with clinical illness. It's just that little quirk that made it totally impossible to control through non-pharmaceutical interventions early on.

Follow that train of thought... if pathogens are going to find ways to take advantage, and there may be pathogens that push the limits on latency, it can be very hard to control. One of the takeaways or the big evolutionary history of our pathogens is evolution is very weird, very contingent, very creative at exploiting whatever weakness we give it. It's because there are billions and billions and billions of microbes in this room. I don't know how many tens to hundreds millions of species of microbes are in this room. Most of them are not even remotely pre-evolved to be pathogenic, but lots are and they're constantly seeing if you managed to lock that door. They're just looking for a way to break in.

Dwarkesh

Okay, just a couple more rapid fire questions for you. Have you found tools like Deep Research useful for especially your kind of work where you just have to compile insights from many different fields? If we throw in a question, the kinds of questions you honestly investigate and now maybe they can rely on you as a citation for those particular questions about what effect did climate have on the fate of Rome or something... But if you just had a different question which maybe you would write a book about in the future, how well do they do at synthesizing this kind of literature and coming up with a thesis the way you do?

Kyle Harper

Yeah, amazing. But not yet completely displacing or totally threatening the kind of work that a historian does. But at this point, I can't even conceive of what a research project would look like without using AI.

Dwarkesh

Really? Oh, really? That fast it's become so central to your work?

Kyle Harper

Yeah, but for just like, it's just like a constant conversation partner when you're doing research, when you're writing, you know, you can go back to that PDF and ask whatever, "How many species are there in this taxon," or you can just ask the AI. You still have to check it, but it's getting obviously more and more reliable really quickly.

But I think it hasn't yet... in some of the deeper research, it's not the equal of humans yet, and then in the synthesis, it's really not. There's still that creative element of synthesis that's... where conceiving of the question is as important as the answer. It doesn't feel like it's right around the corner. But it's changed.

Dwarkesh

Have you used Deep Research?

Kyle Harper

Oh, yeah. I started using it like two weeks ago or so. I don't know how long it's been around? Somebody told me about it.

Dwarkesh

It's not that much longer.

Kyle Harper

Okay, somebody told me about it less than two weeks ago. Yeah, it's incredible. I mean, it's really incredible.

Dwarkesh

Yeah. Now, I want to touch on your next book that isn't out yet, The Last Animal. One question I have is basically how worried should we be about extinction given that we're on the cusp of technologies which will make it possible for us to reanimate many lost species? I assume if we have their genome or something, our descendants will be able to make more wooly mammoths and saber-toothed tigers and so forth. Should we discount the value of endangered species as a result?

Kyle Harper

I would say no. We should still be concerned with extinction for a couple reasons. One is, absolutely this is a legitimate, serious scientific field to understand the genomics of extinct animals. There is small but serious enough science of de-extinction. It's feasible that some organisms could be targeted for serious de-extinction efforts.

At the same time, a couple of thoughts. One is, I'm not that optimistic that it will work, not because I think it's necessarily impossible, although it's not yet totally feasible, particularly for animals that don't have very similar modern descendants. It's because a species isn't just a genome. A species is an organism that inhabits a food web and an ecosystem. We could bring the wooly mammoth back, but there's nowhere for them to live. The mammoth steppe that they need to thrive is not there.

There's really very little point in bringing an animal back from extinction just to put it in a box at a zoo to satiate our curiosity about it. Without the ecosystem, you can't have the species. One of the themes that I try and get at in the book that I'm trying to finish is, we need to think about living systems, ecosystems, and the extinction question is very much a question of what kinds of systems will exist on the planet? Whatever happens technologically in 100 years, 1,000 years, the impacts that humans have on biodiversity is gonna be very long-lasting.

We're part of a species that has been impacting biodiversity for over 10,000 years, and there are things we can't undo. There are things we can't change about the past. We're making decisions right now that will be binding on the future whether our descendants like it or not. We need to think very hard about what choices do we want to make to keep intact the kind of variety and vibrancy of living systems that in 1,000 years, 10,000 years, that will be a huge part of our legacy. The impact that we make on the stream of macroevolution will be one of the really big things that our species does. It can sometimes be very hard to recognize that in our individual lives, but collectively, it will absolutely be part of our forever legacy on Earth. We need to think very carefully about the choices that we make.

Dwarkesh

I think that's an excellent note to close on. Just to plug one more time, we've been discussing Plagues Upon the Earth, which is the history of disease going back through the Neolithic to modern times, Fate of Rome which discusses the plagues and history of the Roman Empire considering climate and biology. We also discussed, what was the name of the book on slavery? Slavery in the Late Roman World?

Kyle Harper

Slavery in the Late Roman World.

Dwarkesh

Yep. And the upcoming book is The Last Animal.

Kyle Harper

The Last Animal.

Dwarkesh

All linked in the description below. And where else can people find you?

Kyle Harper

In your descriptions. That's it. I’m not on social media, sorry.

Dwarkesh

Okay, got it. Well, you can find him here on this podcast.

Kyle Harper

Yes, exclusively.

What started as an attempt to consolidate some thoughts from the last few interviews on my podcast has turned into this 6,000 word clusterfuck of questions and considerations.

If you’ve got answers, or ideas for more questions, I’d be keen to read them in the comments below. I may compile a ‘Highlights from comments’ blog post or podcast episode in the future.

• Why don't we have reliable agents yet?

• Is agency training just a veneer of some MCTS-like scaffolding on the knowledge & intuition that pre-training gives you? Or is it much more difficult to develop?

  • Here’s a case for agency being difficult to develop: Morevac’s Paradox. Evolution has been optimizing us for hundreds of million of years for being able to act like a coherent goal seeking agent even in the face of super dynamic environments, whereas evolution has spent at most hundreds of thousands of years optimizing us for language skills and abstract reasoning. So it's not that surprising that we got expert-level AI mathematicians before AIs that can zero-shot video games made for 10-year-olds. Replicating capabilities derived from a billion years of evolutionary optimization might take much longer than replicating skills contingent on a hundred thousand years of optimization.

    • The rebuttal to Morevac’s Paradox: the capabilities AIs are getting first have nothing to do with their recency in the evolutionary record and everything to do with how much relevant training data exists. Language and coding arrived first not because evolution only recently optimized us for reasoning but because we’ve got the fucking Internet and Github. Unitree robots are really good at walking around, despite the fact that evolution has spent over a quarter billion years teaching us locomotion - and this has everything to do with it being easy to get more data relevant to walking around using simulation.

• What will be different about multi-agent systems?

  • How much of a parallelization penalty will there be? Instead of one instance seeing and considering the whole context, you're breaking apart the problem to multiple workers.

• What is the explanation for why the length of task an AI can do doubles over consistent intervals?

  • How will “Moore’s Law for AI agents” generalize to non-coding tasks like video editing, playing new video games, or coordinating logistics for a happy hour?

  • Would we get any intuition pumps about what superintelligence might look like by considering what it would mean to have a horizon length that is 10x as large as humans?

• Dario said in his recent blog post on export controls that labs are only spending on the order of $1M on RL - why? You’re spending hundreds of million on the base model. If RL training is such a big complement to pre-training, why not spend a similar amount of compute on it?

  • I keep hearing that the big bottleneck for RL is the amount of environments that we have built so far. I don't really understand what this means. What exactly does it take to build a new RL environment? Presumably building complex, realistic, hard to reward-hack challenges?
    Is there a specific reason this is very hard (other than everything in this fallen world being harder than you might naively anticipate)?

  • Also you need smooth reward landscapes that allows AI to be rewarded for incremental improvements rather than getting stuck at 0. Smarter AIs have better priors about “what’s a reasonable thing to do when stuck”, which allow them to learn even from environments with sparser rewards.

  • By when will RL be the dominant workload in training? By when will most RL be online?

• How sample-efficient is RL fine-tuning?

  • For what kinds of skills is it especially effective? What skills are just hard to instill into the model, even if you have the appropriate data?

  • Even if agentic RL is sample-efficient, doesn’t it take way more compute per ‘sample’ than RLHF-type training? As horizon lengths increase, your rollout has to become longer. The AI needs to do two hours worth of agentic computer use tasks before we can even see if it did it right. And if this is correct, will the pace of AI progress slow down?

    • Will this incentivize using smaller pre-trained models to do RL training? Would that allow more entrants to compete for the next tier of capabilities?

• How much extra capability does the longer chain-of-thought get you (as opposed to just doing RL in the first place)?

  • I don’t really understand why test-time compute scaling on things like the ARC-AGI benchmark keep giving marginal return (even if on a log scale). I get why thinking a bit longer might be helpful. But o3-high (which as of writing has the highest score) wrote 43 million words per task. What the hell did it figure out with its 42nd millionth word? Why do the benchmarks keep improving even up till that point?

• RL potentially just upweights 10 tokens worth of MCTS-like scaffolding in a model's thinking (words like “wait”, “let’s backtrack”). This explains why reasoning models can be easily distilled - finding these basic techniques in thought space might take a while, but their payload size is trivial.

  • So first question: is this actually correct?

  • How far can you distill reasoning and chain of thought? Will the models 6 months from now be able to do by instinct the kinds of math and coding that current models need to do large amounts of inference scaling for?

• How much transfer learning is there in reinforcement learning from verifiable reward?

• Can RL work in non verifiable domains? Can you set up some kind of ‘peer review’ with different agents/personas doing subjective critique?

• What is the answer to this question I asked Dario over a year ago? As a scientist yourself, what should we make of the fact that despite having basically every known fact about the world memorized, these models haven’t, as far as I know, made a single new discovery? Even a moderately intelligent person who has so much stuff memorized would make all kinds of new connections (connect fact x, and y, and the logical implication is new discovery z).

  • People have proposed all sorts of answers to this. For example, Scott Alexander wrote,
    “Humans also aren't logically omniscient. My favorite example of this is etymology. Did you know that ‘vacation’ comes from literally vacating the cities? Or that a celebrity is a person who is celebrated? Or that ‘dream’ and ‘trauma’ come from the same root? These are all kind of obvious when you think about them, but I never noticed before reading etymology sites.I think you don't make these connections until you have both concepts in attention at the same time, and the combinatorial explosion there means you've got to go at the same slow rate as all previous progress.”

    • I agree that humans lack some godlike logical omniscience about the combinatorial consequences of all their knowledge. But there's plenty of examples of humans finding these kinds of important connections between fields, despite their much more limited world knowledge (see for instance these examples). I don't think Scott's argument explains why we have many examples of humans doing this, but none with AIs. And it's actually really funny that Scott is making this argument, because one of the things I love about his blog is that he personally has found countless numbers of these intellectually fascinating connections between fields. Where are the LLMs that have done this?

  • Another argument I’ve seen: Eric Schmidt (no not that one) writes, “I think I just don’t agree with your premise that there’s such low hanging fruit out there in the internet text corpus that hasn’t already been grabbed by smart, widely-read humans.”

    • Given that the number of potential connections increases as O(N^2) with the amount of knowledge we have as species, and that the amount of knowledge is itself growing at least linearly, I find it implausible that humans have exhausted this combinatorial overhang.

  • There is a “one man's modus ponens is another man's modus tollens” thing going on here. One way to interpret my original question is from an AI-skeptical point of view: The fact that LLMs aren't more powerful than humans despite their in-principle advantages over us suggests that they're not true AGI. But there's another interpretation of the same question - an interpretation which supports a more FOOMy vibe: Given the in-principle advantages LLMs have over us (in this case, as a result of their immense knowledge, but there are many others), once they actually do become AGIs, they'll fucking dominate.

• Related to the question of LLMs knowing so much shit: It seems like knowledge is really cheap to store. Wikitext is less than 5 MB. So why do we humans forget so much? Why do our brains conspire so hard against acquiring new facts (which in raw bits cost basically nothing) that we have to come up with incredibly hacky systems like spaced repetition to keep knowledge around?

  • One suggestive pattern, copy pasting from my review of Terrence Deacon’s book, The Symbolic Species: “Childhood amnesia (where you can’t remember early parts of your life) is the result of the learning process kids use, where they prune and infer a lot more, allowing them to see the forrest for the trees. On the opposite end of the spectrum are LLMs, which can remember entire passages of Wikipedia text verbatim but will flounder when you give them a new tic tac toe puzzle.There's something super interesting here where humans learn best at a part of their lives (childhood) whose actual details they completely forget, adults still learn really well but have terrible memory about the particulars of the things they read or watch, and LLMs can memorize arbitrary details about text that no human could but are currently pretty bad at generalization. It’s really fascinating that this memorization-generalization spectrum exists.”

• Can you have a 'superhuman AI scientist' before you get human level learning efficiency? (Currently, models take orders of magnitude more data that humans to learn equivalent skills, even ones they perform at 99th percentile level).

  • My take is that creativity and learning efficiency are basically the same thing. The kind of thing Einstein did - generalizing from a few gnarly thought experiments and murky observations - is in some sense just extreme learning efficiency, right? Makes me wonder whether low learning efficiency is the answer to the question, 'Why haven't LLMs haven't made new discoveries despite having so much knowledge memorized'?

• I’m confused why all the labs have ended up making models that are so similar. Everyone is making “thinking” models. Has everybody just been trying a bunch of different shit, but this is the only thing that works? Or are they just copying each other, but in fact there’s a bunch of equally promising tangential research directions that no one is pursuing?

• Many people have pointed out that there’s some missing middle between pre-training and in-context learning. Pre-training gives you some base of general understanding, something like a human skimming every textbook ever written; in-context learning is pure short term memory, discarded after every use. Are we likely to see a new training regime closer to dynamic evaluation, where you update your weights by meditating on feedback, or writing synthetic problems for practice?

  • Would the procedure for dynamically learning new skills have to be bespoke for each application (if you need it to be good at call center workflows, you need to make a bespoke conversation trajectory unrolling environment) or can you come up with some general procedure for upskilling?

• One idea I’ve heard for building long horizon strategizing and coherency is to train AI systems of text based strategy games. Has somebody already tried this and it didn’t work? Or it hasn’t even been tried in the first place?

• What would a good benchmark for meta learning and sample efficient learning look like?

  • If in context sample efficiency is high enough, is it okay if training sample efficiency sucks?

• Is pre-training actually dead?

  • The compute required to train a GPT-4 level model has been declining in cost at the astonishing rate of 10x per year. I thought the whole point of these effective compute multipliers is that we could train GPT-5 at GPT- 4 cost. From the outside at least, it seems like these astonishing compute multipliers are only making existing capabilities cheaper to serve, not enabling the next generation of more powerful models to arrive much sooner. Rumors are that all the labs have been struggling to crack the next OOM of scaling. What’s going on? Data is running out? Maybe the engineering for much larger training runs gets exponentially harder? Or maybe so called algorithmic ‘compute multipliers’ don’t give an equivalent multiplicative boost at different levels of scale?

• A couple months ago at ICML, Ilya compared the pre-training data corpus to fossil fuels - a limited resource which will rapidly be exhausted. This raises the question: what if this data corpus wasn’t limited? Would pre-training still be netting amazing new capabilities? Is next token prediction an innately fucked training objective, or did we just run out of data to let it cook?

• Why are LLMs such mediocre writers despite having all the good writing in their training dataset?

• Copy pasted from Wei Dai: “If the additional training data [after the 10s or trillions of tokens already used] is mostly low quality (AI labs must have used the highest quality data first?) or repetitive (contains no new ideas/knowledge), perplexity might go down but what is the LLM really learning?

• Why aren't all the center workers getting laid off yet? It’s the first thing that should go. Should we take it as some signal that human jobs are just way harder to automate than you might naively think?

• How will the difference between average S&P 500 returns and median S&P 500 company returns change over time?

  • In other words, will we live in the world where Nvidia, Microsoft, Meta, and Google become worth $10T, but everything else goes to 0, or whether broad deployment happens fast enough that McDonalds, JP Morgan Chase, etc. become much more productive at the same rate that AI becomes more powerful.

• Will the models of 2026 and 2027 still best be thought of as time-shares of intelligence, or will they be integrated into companies and workflows such that individual copies are meaningfully distinct?

  • Honestly, I think time-shares of intelligence are still underrated. A new AI employee can just read every single doc in your company's Drive and every single line of code in your company's codebase within minutes. This means that scaling up your company or an AI application would be way less effortful than scaling up a human department.

• What is the industrial-scale use case of AI? Between 1859 (when Drake first discovered oil in Pennsylvania), to 1908 (when Henry Ford invented the modern automobile), the main use for crude was as kerosene for lighting. What is the ultimate industrial-scale equivalent use case for AI?

• How transformative would the AIs of today (March 2025) be even if AI progress stopped here?

  • I've personally become more bearish about the economic value of current systems after using them to build miniapps for my podcast: I can't give them feedback that improves their scope or performance over time, and they can’t deal with unanticipated messy details.

    • But then again, the first personal computers of the 1980s weren't especially useful either. They were mostly used by a few hobbyists. They had anemic memories and processing power, and there just wasn't a global network of applications to make them useful yet.

    • Another way to ask this question: imagine that you plopped down a steam engine in a hamlet from 1500. What would they do with it? Nothing! You need complementary technologies. There weren’t perfect steam engine shaped holes in these hamlets; similarly, there aren’t many LLM-shaped holes in today’s world

• People like Dario say that in a year, 90%+ of code will be written by AIs. In some sense, compilers automated 90%+ of code writing. But I think compilers are a smaller deal than what these people imagine AI coders will be in a year. So what magnitude of actual productivity improvement to software engineering do they expect from AI? 2x? 10x? 100x?

• If you get a 100x increase in software productivity, what kinds of things could we build in the world that we can’t build now?

  • Here’s why I think this is such an interesting question. The economic historian Robert Allen thinks cheap energy is a big part of the reason the Industrial Revolution first happened in Britain. The first steam engine (the Newcomen engine) was super inefficient. It worked not by pushing the piston directly with the steam but by relying on the steam condensing to pull the piston in. The only place where it made sense to even try this design was plentiful coal mines in England. This allowed Britain to get up the learning curve in mechanization, such that they could design devices which were viable with much less energy. The tl;dr here is that you need some initial hand-holding - some cold start - in order to get up the learning curve in a new technology. And because of cheap coal, Britain had it for the Industrial Revolution. Does effectively free software cause some kind of cold start to some future technological trend?

• It seems like some AI labs (OpenAI, Meta) are racing towards being first to a billion user AI assistants, whereas others (Anthropic, maybe GDM?) are racing towards the fully autonomous software engineer. Who’s right? What are the marginal returns to more software (as compared to owning another Facebook- or Whatsapp-size social network)? If you believe in a software driven intelligence explosion, then that places a pretty high premium on the value of more software.

  • The internet and mobile revolutions were largely driven by digital advertising – a relatively small slice of overall global GDP. If AI truly can automate all labor, how soon before AI revenues absolutely gobble up all the social/search/ads revenue of big tech?

• How much of the value of AI requires embodied robots (versus just digital LLMs)?

• Does inference scaling mean that open-weight models don’t decentralize either the benefits or the risks of AI as much as you might naively think (more here)?

• What does an intelligence explosion with open-weight models at the frontier look like?

  • Is there any plausible way in which this could happen? Wouldn’t it require publishing the weights every week/day/hour (depending on speed of intelligence explosion)?

  • If this happened, would it be good?

• What are the implications of the “panspermia” timeline, where multiple countries develop AGI from the same initial seed (e.g., by stealing model weights or building on open weights models)?

  • Would this shared foundation matter more if future advances come from iterated amplification and distillation rather than fresh training runs?

  • Does this increase AI takeover risk, since all the superintelligences start from the same subconscious ‘id’?

  • Does it increase the global impact of ‘Western’ values which are embedded in those stolen weights?

• How are returns to training bigger models scaling? If a lab like OpenAI or DeepMind doubles its training compute budget for its next flagship model compared to the last one, are they getting more than 2x the revenue from it?

  • It seems like the ratio of revenue to training cost is increasing over time, plausibly because so much of the value of new skills is deferred to when AIs are fully reliable and complemented with a wider range of capabilities

• If progress in AI stalls, do foundation model companies inevitably get commoditized? Is there any moat other than staying 6+ months ahead (and in the extreme scenario beating everyone else to the intelligence explosion)? If model companies fail to differentiate, where does the value get captured?

  • One answer might be the hyperscalers who control the datacenter compute, and whose complement (the models) just got commodified. But datacenter compute itself doesn’t seem that differentiated (so much so that the hyperscalers seem to be able to easily contract it out to third parties like CoreWeave). So maybe the lion’s share of value goes to the people making the components that go into chip production: 1) wafer production (TSMC), 2) advanced packaging (TSMC’s CoWoS), and 3) high bandwidth memory (SK Hynix)

• Many previous attempts to make AI applications based on scaffolds and wrappers have been gobbled up by better foundation models (which can just "scaffold" themselves). Will that keep being the case?

• It's interesting to me that some of the best and most widely used applications of foundation models have come from the labs themselves (Deep Research, Claude Code, Notebook LM), even though it's not clear that you needed access to the weights in order to build them. Why is this? Maybe you do need access to the weights of frontier models, and the fine tuning APIs or open source models aren’t enough? Or maybe you gotta ‘feel the AGI’ as strongly as those inside the labs do?

• How much does being at the bleeding edge of AI capabilities matter? Is there any point in competing in the model race if you have no plan to get to the top? Or is there a viable business strategy based on being 6 months behind but following fast?

• Hyperscaler AI capex is getting pretty big - approaching $100B a year for some of the big dogs. If there is a lull in AI capabilities (e.g., inference scaling doesn't end up being that useful or easy to scale, and it takes a couple of more years to make another big breakthrough), what will happen? Will fidgety CFOs force a firesale of compute contracts?

• On the other hand, if AI does end up being as economically remunerative as "AGI" implies, how fast could the hyperscalers ramp up their investment? What would it take for them to invest more than their annual free cash flow (high 10s of Bs for Microsoft, Google, Amazon)? And how would they raise this money?

• Will the desire to avoid the 70%+ Jensen tax inevitably drive more in-house ASIC development?

  • Will this drive greater diversity in model architectures and training techniques?

• How well will distributed training work?

  • How easily can datacenters that have been built up assuming a pretraining focus be repurposed towards RL?

  • Will future training just look a lot like inference? There's not much difference between inference workloads and RL, since an agent has to go away for a while, try to solve a problem, and come back with the results.

  • Can RL rollouts just be totally decentralized?

• How much do hardware tariffs, energy costs, and geopolitical considerations influence where we build the next generation of AI data centers?

  • If significant tariffs were imposed on datacenter components, would that shift planned buildouts towards Europe or Asia?

  • How much does network latency and proximity to customers matter when planning multi-billion dollar, decade-long infrastructure investments?

• How can we build hardware mechanisms that would help you prove to others what you’re using your compute for (and not using it for)?

  • A lot of the stories about how the intelligence explosion goes well involve different actors instituting a mutual slow down where they refocus on alignment + greater monitoring. But such agreements require verifiability.

• Will central planning actually work with AI?

  • The case for:

    • The central AI can have much higher bandwidth communication with the periphery. Today, sensing and deciding have to both happen at the front line. In the future, robot appendages of Big Brother could actually just run the whole military and economy.

    • The central planner can just directly learn from the experience of other AIs (think of a much more advanced version of Tesla FSD model learning from millions of driving samples).

    • Compute/intelligence can be much more centralized than it is today. Right now, Xi Jinping has the same 10^15 FLOPs as anyone else. Not so for the mega-inference-scaled dictators of the future.

    • In order to align incentives for humans, we have to use markets. But we can much more easily control the preferences of AIs.

  • The case against:

    • This vision assumes that only the central government is getting more complex and capable, while the rest of society stays similar. But AI deployment will lead to the whole economy becoming more complex. Apple circa 2025 might be able to centrally plan the economy of Babylon. But it wouldn’t be able to plan America circa 2025.

• What is my blog post about fully automated firms missing?

• How soon after AGI do you get hive minds, fully automated firms, and other crazy powerful and unique products of AIs’ unique advantages in cultural learning/coordination (namely, that AIs can copy, merge, distill, and scale themselves)?

  • An analogy is that the first AGIs will be like humans 200,000 years ago: yes, they have many key advantages, but the things that make us so dominant today - state capacity, joint stock corporations, fossil-fueled civilization - took eons of cultural evolution, population growth, technological upgrading.

• Today, the world economy runs on trade; complex, mutually-dependent supply chains; specialization; and decentralized knowledge. None of this looks like a singleton. Will all this fundamentally change once we have AGI? How about ASI?

  • Or maybe a global system of interconnected markets, delegated decision-making, and mutual interdependence was the singleton all along?

• How does the probability of an intelligence explosion change based on when we achieve AGI?

  • AGI timelines tell you a lot about the nature of intelligence itself. If it takes decades to build AGI (rather than 3 more years of koding-koding-koding), then you’ve learned that transfer learning isn’t that powerful. You’ve learned that current systems aren’t simply “hobbled” little AGIs - rather, they will seem in retrospect more like AlphaZero - an amazing early demonstration of the possibility of AGI, but not the thing itself.

• Could an AI competent at research engineering accelerate AI progress such that we make a BERT to GPT-4.5 size jump in just one year?

  • Case for: my podcast with Daniel and Scott about AI 2027.

  • Case against: my podcast with Ege & Tamay.

  • AI labs apparently run multiple pre-training teams, each with varying allocations of compute and number of researchers. Observing how progress differs across these configurations would yield valuable insights into the tradeoff between cognitive effort and compute scaling in AI progress—though labs are unlikely to share this data.

• Suppose there's a software-only singularity. Would the gap between AI labs narrow or widen during this takeoff?

  • There's a clear reason to expect the gap to widen during an intelligence explosion: whichever lab is slightly ahead benefits from having smarter, more capable automated researchers, rapidly compounding its advantage.

  • Yet historically, we've seen the gap consistently narrow. In 2018, DeepMind was far far ahead of everyone else. Today, labs like DeepSeek can close the distance to within half a year of OpenAI.

  • However, this pattern might not persist. The two primary methods for labs catching up—poaching experienced talent and learning from leaks or public deployments—would lose relevance in a scenario where the best models remain internal, autonomously driving progress.

  • Still, if the intelligence explosion accelerates by simultaneously scaling compute resources, the leading lab might remain incentivized to deploy publicly, attract investment, and thereby continue indirectly facilitating catch-up.

• Currently, model training (and inference) costs for the same level of capability are declining 10x a year. If you extrapolate this trend, then it suggests that you can train GPT-4 level capabilities using 100 H100s by 2027. Does this imply that while eventually we'll be able to distill human-level intelligence into mosquito drones, the first AGIs will be a system that costs $100B and requires Montana-sized infrastructure to train, and uses inference scaling hacks that cost $10 per token (like some Yoda, slowly meditating on each syllable)?

• Some people imagine that once we achieve ASI, it rapidly invents crazy superweapons, new computing paradigms, fusion-powered space probes, etc. But this doesn’t seem to map on to how past new inventions and scientific discoveries were made. It seems like a general upgrading of your society's tech stack is more important than raw cognitive effort in a specific sector: you couldn't have discovered the structure of DNA without X-ray crystallography; and we didn’t figure out there was a Big Bang until we saw cosmic background microwave thanks to the radio astronomy techniques that were initially developed for communications during World War II. Does this suggest that we're not just going to have this technological explosion in the middle of the desert? In order to truly advance the state of technology, you basically need to upgrade the whole economy. And this points to a broad deployment before an R&D explosion.

  • To be clear, transformative AI might still happen super fast! The AIs are thinking very quickly, there’s billions of them, they’re much better at learning-by-doing, etc. But it’s simply a question of whether or not we’ll just straight shot nano-tech or dyson spheres and skip over all the boring everyday improvements in seemingly unrelated fields.

• Will explosive growth happen?

  • The basic argument for explosive growth: Total economic output = Total Factor Productivity (TFP) × Labor × Capital. Today, economic growth has a weak feedback loop: output can accumulate into more capital, but not into more (human) labor. With AI, however, capital (compute) also functions as labor, creating a stronger feedback loop. Automated economic output = TFP (rapidly rising due to automated research and accelerated learning-by-doing) × Capital (previous output).

  • If you’re interested in other considerations, check out my interview with Tyler for the con case, and the one with Ege & Tamay for the pro case.

• Will explosive growth basically look like what has happened to China post Deng (double digit growth rates instigated by abundant skilled labor)?

• Will economic growth be extensive or intensive? Or some secret third thing?

  • If the former, then it’s possible that the effects of explosive economic growth will not be felt that widely: your cityscape will not look that different; your life might not even change that much. But out in the desert, somewhere far from your view, they're producing solar farms and robot factories which are worth 200% of the existing economy.

• “LLMs were aligned by default. Agents trained with reinforcement learning reward hack by default” (tweet here): is this actually the correct framing?

  • Base LLMs were also misaligned by default. People had to figure out good post-training (partly using RL) to solve this. There's obviously no reward hacking in pretraining, but it’s not clear that pretraining vs RL have such different 'alignment by default'.

• Are there any robust solutions to reward hacking? Or is reward hacking such an attractive basin in training that if any exploit exists in the environment, models will train to hack it?

  • Can we solve reward hacking by training agents in many different kinds of unique environments? In order to succeed, they’d have to develop robust general skills that don't just involve finding the exploits in any one particular environment.

• Do the models know when they’re reward hacking?

• Are capabilities and alignment the same thing here? Does making models more useful require solving reward hacking?

  • If this is the case, we might be living in the alignment-by-default world? It would be weird if we solve reward hacking well enough to make these models reliable general agents in every scenario except those involved in taking over the world.

• What’s the right way to think about potentially misaligned AIs deployed through the economy? Are they like disgruntled employees who dislike the CEO (bullish for humanity), or more like Cortés landing in the New World (bearish)?

  • An optimistic framing: Think of AI as just another set of workers. CEOs typically aren’t the smartest people in their companies, nor do they grasp all technical nuances of how their instructions are executed by engineers or researchers. The same applies at the national level—Xi Jinping doesn't lose sleep over whether all 100 million CCP members genuinely love him, yet he remains in power. Plenty of employees actively dislike their CEOs but still inadvertently contribute positively to the company's objectives. Why should we assume misaligned AIs would behave differently?

  • A pessimistic framing: history is littered with examples of violent takeovers enabled by moderate technological advantages. Geneticist David Reich, on my podcast, described human history as repetitive waves of violent expansion, where a technologically or organizationally superior group wipes out most people across entire continents. Consider Hernán Cortés, who conquered an empire of over 10 million people in just two years with fewer than 1000 soldiers (plus horses, steel, and Smallpox). Or the British East India Company, which took over a subcontinent of 150 million people with just a few thousand officers, aided by marginally superior artillery tactics and logistical innovations derived from European warfare—not some profound, galaxy-brain advantage. Perhaps AIs similarly wouldn’t need an overwhelming technological edge to decisively overpower humanity.

• If the AIs ‘takeover’, will we be able to see it coming? How wild and sci-fi will it be?

  • Will it be like a gazelle being hunted by hunter-gatherers where it can clearly understand the strategic situation as it tries to run away from the pointy sticks?

  • Or will it be more like a deer peacefully grazing, unaware that it’s being watched through a scope 100 yards away, and a moment later… boom.

• What should the model spec say?

  • Alignment stories often assume we can get an AI to deeply internalize a foundational document that spells out its core values, directives, and ultimate authorities—something like, "Follow user instructions unless they're clearly dangerous; in disputes, defer to the judgment of Sam Altman or President Trump."

  • Think about how much the United States today is contingent on the specific quill strokes of Madison - what exactly does this comma mean, what did he mean by ‘general welfare’ and ‘interstate commerce’?

  • How do we get political opposition parties, allied governments, and other people in the world to fully grasp the enormous (temporary) leverage they currently have to influence this document?

• If China develops AGI first, how do we make sure that their model spec doesn't just make Xi Jinping the god dictator forever? Do private Chinese companies and high-level members of the CCP have enough influence and understanding of the strategic situation to prevent such an outcome?

• Is there an inherent tradeoff between preventing misuse and reducing the risk of a coup (either by AIs themselves or by humans using AI)? Or is this a false dichotomy?

  • Preventing misuse looks a lot like making sure the most advanced capabilities aren't widely deployed because of the inherent dual-use nature of understanding reality. Preventing takeover means making sure that we’re maximally open and transparent about frontier systems, and deploy them as widely as possible in order to prevent any one faction from monopolizing their benefits.

• Jailbreaking is becoming harder as models become smarter. This is pretty intuitive: a smart but ethical human assistant could tell if you’re asking him to help weaponize Smallpox or simply practicing for an organic chemistry exam. So as AIs get smarter, they should be able to tell the difference as well. Are we set here if we just put “don’t make bioweapons” in the model spec?

• It seems like the whole misuse story is especially anchored on bioterrorism. The intuition pump is to imagine every person with a team of virology PhDs in her pocket. What actually would happen in this scenario?

• I’m concerned about nationalization of AGI development for the reasons listed below. Where am I wrong?

  • Reduces the salience of safety and alignment, in favor of whatever the administration + deep state at the time cares about.

  • Decreases the competence of the group overseeing the intelligence explosion, which is robustly bad, especially if you think alignment is difficult and subtle.

  • Increases the likelihood of dictatorship, with the President ending up as the ultimate decision maker in the spec or something.

  • Instigates an obvious arms race framing in China.

  • Increases the likelihood that the first AGIs are used to develop weapons, drones, and bioweapons instead of what’s needed for our glorious transhumanist future.

• How will the Chinese political system react to fully automated remote work, superhuman hackers, automated AI researchers etc.?

  • How deep are their private markets? How willing are the big public funds to finance the next rung of scaling?

  • What might instigate a Chinese Manhattan Project for AGI?

  • Suppose tomorrow Xi wanted to prioritize building AGI. At an institutional level, what concretely could he do?

  • How does Chinese industrial espionage in other industries work? Suppose there's a hardware company that wants to learn how Apple does some procedure. Is there some government department they file their well-scoped snooping request with?

• Developing AI (and successfully deploying it at scale) is a huge industrial project. China is really good at industrial scale-up and state-directed investment. Does this give them a huge advantage in deploying AI?

• If you’re the leader of a nation like India or Nigeria today, and you get AGI-pilled, what should you do?

  • Honestly, it feels like a really tough position. The odds that you develop a frontier AI lab are pretty low. And you don’t have some crucial input into the semiconductor supply chain that will give you any leverage during crunch time.

  • If you have energy and property rights and friendly relations with the US and/or China, maybe you can become a datacenter hub?

• Does superintelligence actually give you ‘decisive strategic advantage’? Examples like Cortez or the East India Trading Company seem to imply that slight advantages in technology might be totally overwhelming in a military conflict. But perhaps modern weapons (nukes especially) are so destructive already that they provide sufficient deterrence against adversaries with smarter AIs launching their mosquito drone swarms and superhuman cyber attacks.

• How wrong should we expect our conceptual handles around AGI to be?

  • I've been reading The House of Government recently. It's a fascinating account of people involved in the Russian Revolution. There were many different factions of people who were disillusioned with the Czarist regime - the anarchists, the Mensheviks, Bolsheviks, the social revolutionaries, the Decembrists. They intensely debated the dichotomies which were most salient to them given their milieu.

    • The “decisive battle” … covered all the usual points of disagreement: the “working class” versus “the people”; the “sober calculation” versus “great deeds and self-sacrifice”; “objectivism” versus “subjectivism”; and “universal laws of development” versus “Russia’s uniqueness.”

  • Yet none of them anticipated the considerations we now recognize to be far more relevant to economic development: dispersed knowledge, voluntary exchange, and entrepreneurial innovation.

  • I think about this whenever my Bay Area friends debate AGI - will there be a software-only singularity, adversarial misalignment, training gaming, explosive growth, etc, etc? Maybe the frameworks we're using and the questions we're asking are fundamentally misguided.

• Given this topic is so epistemically murky that someone smart can come up with a new consideration that alters your key conclusions, how much should you update on the most recent compelling story you’ve heard?

  • Historically, the way we’ve dealt well with rapidly-evolving, uncertain processes is classical liberalism. Pushing for super tailored proposals based on reasoning your way to a specific trajectory (“We’re <5 years to ASI, therefore ‘The Project’”) has a pretty bad track record.

If you enjoyed this post, do share it with others who you think might enjoy it too.

Share

If you enjoyed this blog post, you may enjoy my new book, The Scaling Era: An Oral History of AI, 2019-2025. This book curates and organizes the highlights across my podcast episodes about AI with scientists, CEOs, economists, and philosophers.

Thanks to Max Farrens for comments and editing. Thanks especially to Carl Shulman, but also Leopold Aschenbrenner, Tamay Besiroglu, Ege Erdil, Daniel Kokotjlo, Scott Alexander, Sholto Douglas, Adam D’Angelo, Paul Christiano, Craig Falls, Gwern Branwen, Dan Hendrycks, Andrej Karpathy, Toby Ord, Neel Nanda, and many others for conversations which helped inspire questions.

Subscribe for future post!

Ege Erdil and Tamay Besiroglu have 2045+ timelines, think the whole "alignment" framing is wrong, don't think an intelligence explosion is plausible, but are convinced we'll see explosive economic growth.

This discussion offers a totally different scenario than my recent interview with Scott and Daniel.

Ege and Tamay are the co-founders of Mechanize (disclosure - I’m an angel investor), a startup dedicated to fully automating work. Before founding Mechanize, Ege and Tamay worked on AI forecasts at Epoch AI.

Watch on Youtube; listen on Apple Podcasts or Spotify.

• WorkOS makes it easy to become enterprise-ready. With simple APIs for essential enterprise features like SSO and SCIM, WorkOS helps companies like Vercel, Plaid, and OpenAI meet the requirements of their biggest customers. To learn more about how they can help you do the same, visit workos.com

• Scale’s Data Foundry gives major AI labs access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you’re an AI researcher or engineer, learn about how Scale’s Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkesh

• Google's Gemini Pro 2.5 is the model we use the most at Dwarkesh Podcast: it helps us generate transcripts, identify interesting clips, and code up new tools. If you want to try it for yourself, it's now available in Preview with higher rate limits! Start building with it today at aistudio.google.com.

To sponsor a future episode, visit dwarkesh.com/advertise.

(00:00:00) - AGI will take another 3 decades

(00:22:27) - Even reasoning models lack animal intelligence

(00:45:04) - Intelligence explosion

(01:00:57) - Ege & Tamay’s story

(01:06:24) - Explosive economic growth

(01:33:00) - Will there be a separate AI economy?

(01:47:08) - Can we predictably influence the future?

(02:19:48) - Arms race dynamic

(02:29:48) - Is superintelligence a real thing?

(02:35:45) - Reasons not to expect explosive growth

(02:49:00) - Fully automated firms

(02:54:43) - Will central planning work after AGI?

(02:58:20) - Career advice

Dwarkesh Patel 00:00:00
Today, I’m chatting with Tamay Besiroglu and Ege Erdil. They were previously running Epoch AI and are now launching Mechanize, which is a company dedicated to automating all work. One of the interesting points you made recently, Tamay, is that the whole idea of the intelligence explosion is mistaken or misleading. Why don’t you explain what you’re talking about there?
Tamay Besiroglu 00:00:22
Yeah, I think it’s not a very useful concept. It’s kind of like calling the Industrial Revolution a horsepower explosion. Sure, during the Industrial Revolution, we saw this drastic acceleration in raw physical power, but there are many other things that were maybe equally important in explaining the acceleration of growth and technological change that we saw during the Industrial Revolution.
Dwarkesh Patel 00:00:42
What is a way to characterize the broader set of things that the horsepower perspective would miss about the Industrial Revolution?
Tamay Besiroglu 00:00:50
So I think in the case of the Industrial Revolution, it was a bunch of these complementary changes to many different sectors in the economy. So you had agriculture, you had transportation, you had law and finance, you had urbanization and moving from rural areas into cities. There were just many different innovations that happened simultaneously that gave rise to this change in the way of economically organizing our society.
It wasn’t just that we had more horsepower. I mean, that was part of it, but that’s not the kind of central thing to focus on when thinking about the Industrial Revolution. And I think similarly, for the development of AI, sure, we’ll get a lot of very smart AI systems, but that will be one part among very many different moving parts that explain why we expect to get this transition and this acceleration and growth and technological change.
Dwarkesh Patel 00:01:46
I want to better understand how you think about that broader transformation. Before we do, the other really interesting part of your worldview is that you have longer timelines to get to AGI than most of the people in San Francisco who think about AI. When do you expect a drop-in remote worker replacement?
Ege Erdil 00:02:05
Maybe for me, that would be around 2045.
Dwarkesh Patel 00:02:10
Wow. Wait, and you?
Tamay Besiroglu 00:02:11
Again, I’m a little bit more bullish. I mean, it depends what you mean by “drop in remote worker“ and whether it’s able to do literally everything that can be done remotely, or do most things.
Ege Erdil 00:02:21
I’m saying literally everything.
Tamay Besiroglu 00:02:22
For literally everything. Just shade Ege’s predictions by five years or by 20% or something.
Dwarkesh Patel 00:02:27
Why? Because we’ve seen so much progress over even the last few years. We’ve gone from Chat GPT two years ago to now we have models that can literally do reasoning, are better coders than me, and I studied software engineering in college. I mean, I did become a podcaster, I’m not saying I’m the best coder in the world.
But if you made this much progress in the last two years, why would it take another 30 to get to full automation of remote work?
Ege Erdil 00:03:01
So I think that a lot of people have this intuition that progress has been very fast. They look at the trend lines and just extrapolate; obviously, it’s going to happen in, I don’t know, 2027 or 2030 or whatever. They’re just very bullish. And obviously, that’s not a thing you can literally do.
There isn’t a trend you can literally extrapolate of “when do we get to full automation?”. Because if you look at the fraction of the economy that is actually automated by AI, it’s very small. So if you just extrapolate that trend, which is something, say, Robin Hanson likes to do, you’re going to say, “well, it’s going to take centuries” or something.
Now, we don’t agree with that view. But I think one way of thinking about this is how many big things are there? How many core capabilities, competences are there that the AI systems need to be good at in order to have this very broad economic impact, maybe 10x acceleration and growth or something? How many things have you gotten over the past 10 years, 15 years? And we also have this compute-centric view…
Tamay Besiroglu 00:04:05
So just to double click on that, I think what Ege is referring to is, if you look at the past 10 years of AI progress, we’ve gone through about nine or 10 orders of magnitude of compute, and we got various capabilities that were unlocked. So in the early period, people were solving gameplay on specific games, on very complex games. And that happened from 2015 to 2020, Go and Chess and Dota and other games. And then you had maybe sophisticated language capabilities that were unlocked with these large language models, and maybe advanced abstract reasoning and coding and maybe math. That was maybe another big capability that got unlocked.
And so maybe there are a couple of these big unlocks that happened over the past 10 years, but that happened on the order of once every three years or so, or maybe one every three orders of magnitude of compute scaling. And then you might ask the question, “how many more such competencies might we need to unlock in order to be able to have an AI system that can match the capabilities of humans across the board?” Maybe specifically just on remote work tasks. And so then you might ask, well, maybe you need kind of coherence over very long horizons, or you need agency and autonomy, or maybe you need full multimodal understanding, just like a human would.
And then you ask the question, “okay, how long might that take?” And so you can think about, well, just in terms of calendar years, the previous unlocks took about, you get one every three years or so. But of course, that previous period coincided with this rapid scale-up of the amount of compute that we use for training. So we went through maybe 9 or 10 orders of magnitude since AlexNet compared to the biggest models we have today. And we’re getting to a level where it’s becoming harder and harder to scale up compute. And we’ve done some extrapolations and some analysis looking at specific constraints, like energy or GPU production.
And based on that, it looks like we might have maybe three or four orders of magnitude of scaling left. And then you’re really spending a pretty sizable fraction or a non-trivial fraction of world output on just building up data centers, energy infrastructure, fabs, and so on.
Dwarkesh Patel 00:06:40
Which is already like 2% of GDP, right?
Tamay Besiroglu 00:06:42
I mean, currently it’s less than 2%.
Ege Erdil 00:06:44
Yeah, but also currently most of it is actually not going towards AI chips. But even most TSMC capacity currently is going towards mobile phone chips or something like that, right?
Dwarkesh Patel 00:06:52
Even leading edge. It’s like 5% of leading edge.
Tamay Besiroglu 00:06:55
Yeah, even leading edge is pretty small. But yeah, so that suggests that we might need a lot more compute scaling to get these additional capabilities to be unlocked. And then there’s a question of do we really have that in us as an economy to be able to sustain that scaling?
Dwarkesh Patel 00:07:14
But it seems like you have this intuition that there’s just a lot left to intelligence. When you play with these models, they’re almost there. You forget you’re often talking to an AI.
Ege Erdil 00:07:26
What do you mean they’re almost there? I don’t know. I can’t ask Claude to pick up this cup and put it over there.
Dwarkesh Patel 00:07:31
Remote work, you know?
Ege Erdil 00:07:32
Okay. But even for remote work, I can’t ask Claude to… I think the current computer use systems can’t even book a flight properly.
Dwarkesh Patel 00:07:38
How much of an update would it be if by the end of 2026, they could book a flight?
Ege Erdil 00:07:43
I probably think by the end of this year, they’re going to be able to do that. But that’s a very simple… Nobody gets a job where they’re paid to book flights. That’s not a task.
Dwarkesh Patel 00:07:54
I think some people do.
Tamay Besiroglu 00:07:56
If it’s literally just a book flight job, and without-
Ege Erdil 00:08:00
But I think that’s an important point, because a lot of people look at jobs in the economy, and then they’re like, “oh, that person, their job is to just do X”. But then that’s not true. That’s something they do in their job. But if you look at the fraction of their time on the job that they spend on doing that, it’s a very small fraction of what they actually do. It’s just this popular conception people have. Or travel agents, they just book hotels and flights. But that’s not actually most of their job. So automating that actually wouldn’t automate their job, and it wouldn’t have that much of an impact on the economy.
So I think this is actually an important thing, that important worldview difference that separates us from people who are much more bullish, because they think jobs in the economy are much simpler in some sense, and they’re going to take much fewer competences to actually fully automate.
Dwarkesh Patel 00:08:47
So our friend Leopold has this perspective of, quote unquote, ‘unhobblings’, where the way to characterize it might be, they’re basically like baby AGIs already. And then because of the constraints we artificially impose upon them by, for example, only training them on text and not giving them the training data that is necessary for them to understand a Slack environment or a Gmail environment, or previously before inference time scaling, not giving them the chance to meditate upon what they’re saying and really think it through, and not giving them the context about what is actually involved in this job, only giving them this piecemeal, a couple of minutes worth of context in the prompt, we’re holding back what is fundamentally a little intelligence from being as productive as it could be, which implies that unhobblings just seem easier to solve for than entirely new capabilities of intelligence. What do you make of that framework?
Tamay Besiroglu 00:09:46
I mean, I guess you could have made similar points five years ago and say “you look at AlphaZero and there’s this mini AGI there, and if only you unhobbled it by training it on text and giving it all your context” and so on, that just wouldn’t really have worked. I think you do really need to rethink how you train these models in order to get these capabilities.
Dwarkesh Patel 00:10:08
But I think the surprising thing over the last few years has been that you can start off with this pre-trained corpus of the internet, and it’s actually quite easy. ChatGPT is an example of this unhobbling, where 1% of additional compute spent on getting it to talk in a chatbot-like fashion with post training is enough to make it competent- really competent- at that capability.
Reasoning is another example where it seems like the amount of compute that is spent on RL right now in these models is a small fraction of total compute. Again, reasoning seems complicated, and then you just do 1% of compute and it gets you that. Why not think that computer use, or long-term agency on computer use, is a similar thing?
Tamay Besiroglu 00:10:55
So when you say “reasoning is easy” and “it only took this much compute” and “it wasn’t very much”, and maybe “you look at the sheer number of tokens and it wasn’t very much, and so it looks easy”, well, that’s true from our position today. But I think if you ask someone to build a reasoning model in 2015, then it would have looked insurmountable. You would have had to train a model on tens of thousands of GPUs, you would have had to solve that problem, and each order of magnitude of scaling from where they were would pose new challenges that they would need to solve.
You would need to produce internet scale, or tens of trillions of tokens of data in order to actually train a model that has the knowledge that you can then unlock and access by way of training it to be a reasoning model. You need to maybe make the model more efficient at doing inference and maybe distill it, because if it’s very slow then you have a reasoning model that’s not particularly useful, so you also need to make various innovations to get the model to be distilled so that you can train it more quickly, because these rollouts take very long.
It actually becomes a product that’s valuable if it’s a couple tokens a second, as a reasoning model that would have been very difficult to work with. So in some sense, it looks easy from our point of view, standing on this huge stack of technology that we’ve built up over the past five years or so, but at the time, it would have been very hard.
And so my claim would be something like; I think the agency part might be easy in a similar sense, that in five years or three years time or whatever we will look at what unlocked agency and it’ll look fairly simple. But the amount of work that, in terms of these complementary innovations that enable the model to be able to learn how to become a competent agent, that might have just been very difficult and taken years of innovation and a bunch of improvements in kind of hardware and scaling and various other things.
Dwarkesh Patel 00:12:54
Yeah, I feel like what’s dissimilar between 2015 and now… in 2015 if you were trying to solve reasoning, you just didn’t have a base to start on. Maybe if you tried formal proof methods or something, but there was no leg to stand on, where now you’d actually have the thing- you have the pre-trained base model, you have these techniques of scaffolding, of post-training, of RL. And so it seems like you think that those will look to the future as, say, AlphaGo looks to us now in terms of the basis of a broader intelligence.
I’m curious if you have intuitions on why not think that language models as we have them now are like, we got the big missing piece right and now we’re just like plugging things on top of it?
Ege Erdil 00:13:51
Well, I mean, I guess what is the reason for believing that? I mean, you could have looked at AlphaGo or AlphaGo Zero, AlphaZero, those seemed very impressive at the time. I mean, you’re just learning to play this game with no human knowledge, you’re just learning to play it from scratch. And I think at the time it did impress a lot of people. But then people tried to apply it to math, they tried to apply it to other domains, and it didn’t work very well, they weren’t able to get competent agents at math.
So it’s very possible that these models, at least the way we have them right now, you’re going to try to do the same thing people did for reasoning, but for agency, it’s not going to work very well. And then you’re not going to-
Dwarkesh Patel 00:14:32
I’m sorry, you’re saying by the end of 2026, we will have agentic computer use.
Tamay Besiroglu 00:14:36
I think Ege said you’d be able to book a flight, which is very different from having full agentic computer use.
Dwarkesh Patel 00:14:44
I mean, the other things you need to do on a computer are just made up of things like booking a flight.
Ege Erdil 00:14:49
I mean, sure, but they are not disconnected tasks. That’s like saying everything you do in the world is just like you just move parts of your body, and then you move your mouth and your tongue, and then you roll your head. Yeah, individually those things are simple, but then how do you put them together, right?
Dwarkesh Patel 00:15:09
Yeah. Okay. So there’s two pieces of evidence that you can have that are quite dissimilar.
One, the METR eval, which we’ve been talking about privately, which shows that the task length over certain kinds of tasks- I can already see you getting ready. AI’s ability to do the kind of thing that it takes a human 10 minutes to do, or an hour to do, or four hours to do, the length of time for corresponding human tasks, it seems like these models seem to be doubling their task length every seven months. The idea being that by 2030, if you extrapolate this curve, they could be doing tasks that take humans one month to do, or one year to do. And then this long-term coherency in executing on tasks is fundamentally what intelligence is. So this curve suggests that we’re getting there.
The other piece of evidence- I kind of feel like my own mind works this way. I get distracted easily, and it’s hard to keep a long-term plan in my head at the same time. And I’m slightly better at it than these models. But they don’t seem that dissimilar to me. I would have guessed reasoning is just a really complicated thing, and then it seems like, “oh, it’s just something like learning 10 tokens worth of MCTS” of “wait, let’s go back, let’s think about this another way”.
Chain of thought alone just gets you this boost. And it just seems like intelligence is simpler than we thought. Maybe agency is also simpler in this way.
Ege Erdil 00:16:39
Yeah. I mean, I think there’s a reason to expect complex reasoning to not be as difficult as people might have thought, even in advance, because a lot of the tasks that AI solved very early on were tasks of various kinds of complex reasoning. So it wasn’t the kind of reasoning that goes into when a human solves a math problem.
But if you look at the major AI milestones over, I don’t know, since 1950, a lot of them are for complex reasoning. Like chess is, you can say, a complex reasoning task. Go is, you could say, a complex reasoning task.
Dwarkesh Patel 00:17:14
But I think there are also examples of long-term agency. Like winning at Starcraft is an example of being agentic over a meaningful period of time.
Ege Erdil 00:17:24
That’s right. So the problem in that case is that it’s a very specific, narrow environment. You can say that playing Go or playing chess, that also requires a certain amount of agency. And that’s true. But it’s a very narrow task. So that’s like saying if you construct a software system that is able to react to a very specific, very particular kind of image, or very specific video feeds or whatever, then you’re getting close to general sensor motor skill automation.
But the general skill is something that’s very different. And I think we’re seeing that. We still are very far, it seems like, from an AI model that can take a generic game off Steam. Let’s say you just download a game released this year. You don’t know how to play this game. And then you just have to play it. And then most games are actually not that difficult for a human.
Dwarkesh Patel 00:18:21
I mean, what about Claude Plays Pokemon? I don’t think it was trained on Pokemon.
Ege Erdil 00:18:25
Right, so that’s an interesting example. First of all, I find the example very interesting, because yeah, it was not trained explicitly. They didn’t do some RL on playing Pokemon Red. But obviously, the model knows how it’s supposed to play Pokemon Red, because there’s tons of material about Pokemon Red on the internet.
In fact, if you were playing Pokemon Red, and you got stuck somewhere, you didn’t know what to do, you could probably go to Claude and ask “I’m stuck in Mount Moon, and what am I supposed to do?” And then it’s probably able to give you a fairly decent answer. But that doesn’t stop it from getting stuck in Mount Moon for 48 hours. So that’s a very interesting thing, where it has explicit knowledge, but then when it’s actually playing the game, it doesn’t behave in a way which reflects that it has that knowledge.
Dwarkesh Patel 00:19:09
All it’s got to do is plug the explicit knowledge to its actions.
Ege Erdil 00:19:13
Yeah, but is that easy?
Dwarkesh Patel 00:19:15
Okay, if you can leverage your knowledge from pre-training about these games in order to be somewhat competent at them, okay, they’re going to be leveraging a different base of skills. But with that same leverage, they’re going to have a similar repertoire of abilities. If you’ve read everything about whatever skill that every human has ever seen.
Ege Erdil 00:19:43
A lot of the skills that people have, they don’t have very good training data for them.
Dwarkesh Patel 00:19:48
That’s right. What would you want to see over the next few years that would make you think, “oh, no, I’m actually wrong and this was the last unlock, and it was now just a matter of ironing out the kinks”. And then we get the thing that will kick off the, dare I say, intelligence explosion.
Tamay Besiroglu 00:20:04
I think something that would reveal its ability to do very long context things, use multimodal capabilities in a meaningful way, and integrate that with reasoning and other types of systems. And also agency and being able to take action over a long horizon and accomplish some tasks that takes very long for humans to do, not just in specific software environments, but just very broadly; say downloading an arbitrary game from Steam, something that it’s never seen before,
it doesn’t really have much training data, maybe it was released after a training cutoff and so there’s no tutorials or maybe there’s no earlier versions of the game that has been discussed on the Internet, and then accomplishing that game and actually playing that game to the end and accomplishing these various milestones that are challenging for humans. That would be a substantial update. I mean, there are other things that would update me, too, like OpenAI making a lot more revenue than it’s currently doing.
Dwarkesh Patel 00:21:11
Is the hundred billion in revenue that would, according to their contract, mark them as AGI enough?
Tamay Besiroglu 00:21:15
I think that’s not a huge update to me if that were to happen. So I think the update would come if it was, in fact, $500 billion in revenue or something like that. But then I would certainly update quite a lot. But a hundred billion, that seems pretty kind of likely to me. I would assign that maybe a 40 percent chance or something.
Dwarkesh Patel 00:21:37
If you’ve got a system that is, in producer surplus terms, worth a hundred billion. And the difference between this and AlphaZero is AlphaZero is never going to make a hundred billion dollars in the marketplace. So just what is intelligence? It’s like something able to usefully accomplish its goals, or your goals. If people are willing to pay a hundred billion dollars for it, that’s pretty good evidence that it’s like accomplishing some goals.
Tamay Besiroglu 00:22:05
I mean, people pay a hundred billion dollars for all sorts of things. That itself is not a very strong piece of evidence that it’s going to be transformative, I think.
Ege Erdil 00:22:13
People pay trillions of dollars for oil. I don’t know, it seems like a very basic point. But the fact that people pay a lot of money for something doesn’t mean it’s going to transform the world economy if only we manage to unhobble it. Like that’s a very different claim.

Dwarkesh Patel 00:22:27
So then this brings us to the intelligence explosion, because what people will say is, we don’t need to automate literally everything that is needed for automating remote work, let alone all human labor in general. We just need to automate the things which are necessary to fully close the R&D cycle needed to make smarter intelligences.
And if you do this, you get a very rapid intelligence explosion. And the end product of that explosion is not only an AGI, but something that is superhuman potentially. These things are extremely good at coding, and reasoning. It seems like the kinds of things that would be necessary to automate R&D at AI labs. What do you make of that logic?
Ege Erdil 00:24:14
I think if you look at their capability profile, if you compare it to a random job in the economy, I agree they are better at doing coding tasks that will be involved in R&D compared to a random job in the economy. But in absolute terms, I don’t think they’re that good. I think they are good at things that maybe impress us about human coders. If you wanted to see what makes a person a really impressive coder, you might look at their competitive programming performance. In fact, companies often hire people, if they’re relatively junior, based on their performance on these kinds of problems. But that is just impressive in the human distribution.
So if you look in absolute terms at what are the skills you need to actually automate the process of being a researcher, then what fraction of those skills do the AI systems actually have? Even in coding, a lot of coding is, you have a very large code base you have to work with, the instructions are very kind of vague. For example you mentioned METR eval, in which, because they needed to make it an eval, all the tasks have to be compact and closed and have clear evaluation metrics: “here’s a model, get its loss on this data set as low as possible”. Or “here’s another model and its embedding matrix has been scrambled, just fix it to recover like most of its original performance”, etc.
Those are not problems that you actually work on in AI R&D. They’re very artificial problems. Now, if a human was good at doing those problems, you would infer, I think logically, that that human is likely to actually be a good researcher. But if an AI is able to do them, the AI lacks so many other competences that a human would have- not just the researcher, just an ordinary human- that we don’t think about in the process of research. So our view would be that automating research is, first of all, more difficult than people give it credit for. I think you need more skills to do it and definitely more than models are displaying right now.
And on top of that, even if you did automate the process of research, we think a lot of the software progress has been driven not by cognitive effort- that has played a part- but it has been driven by compute scaling. We just have more GPUs, you can do more experiments, to figure out more things, your experiments can be done at larger scales. And that is just a very important driver. If you’re 10 years ago, 15 years ago, you’re trying to figure out what software innovations are going to be important in 10 or 15 years, you would have had a very difficult time. In fact, you probably wouldn’t even have conceived of the right kind of innovations to be looking at, because you would be so far removed from the context of that time with much more abundant compute and all the things that people would have learned by that point.
So these are two components of our view: Research is harder than people think, and depends a lot on compute scale.
Dwarkesh Patel 00:27:17
Can you put a finer point on what is an example of the kind of task which is very dissimilar from ‘train a classifier’ or ‘debug a classifier’ that is relevant to AI R&D?
Tamay Besiroglu 00:27:30
Examples might be introducing novel innovations that are very useful for unlocking innovations in the future. So that might be introducing some novel way of thinking about a problem. A good example might be in mathematics, where we have these reasoning models that are extremely good at solving math problems.
Ege Erdil 00:27:57
Very short horizon.
Tamay Besiroglu 00:28:00
Sure. Maybe not extremely good, but certainly better than I can and better than maybe most undergrads can. And so they can do that very well, but they’re not very good at coming up with novel conceptual schemes that are useful for making progress in mathematics. So it’s able to solve these problems that you can kind of neatly excise out of some very messy context, and it’s able to make a lot of progress there.
But within some much messier context, it’s not very good at figuring out what directions are especially useful for you to build things or make incremental progress on that enables you to have a big kind of innovation later down the line. So thinking about both this larger context, as well as maybe much longer horizon, much fuzzier things that you’re optimizing for, I think it’s much worse at those types of things.
Ege Erdil 00:28:54
Right. So I think one interesting thing is if you just look at these reasoning models, they know so much, especially the larger ones, because they know in literal terms more than any human does in some sense. And we have unlocked these reasoning capabilities on top of that knowledge, and I think that is actually what’s enabling them to solve a lot of these problems. But if you actually look at the way they approach problems, the reason what they do looks impressive to us is because we have so much less knowledge.
And the model is approaching the problems in a fundamentally different way compared to how a human would. A human would have much more limited knowledge, and they would usually have to be much more creative in solving problems because they have this lack of knowledge, while the model knows so much. But you’d ask it some obscure math question where you need some specific theorem from 1850 or something, and then it would just know that, if it’s a large model. So that makes the difficulty profile very different.
And if you look at the way they approach problems, the reasoning models, they are usually not creative. They are very effectively able to leverage the knowledge they have, which is extremely vast. And that makes them very effective in a bunch of ways. But you might ask the question, has a reasoning model ever come up with a math concept that even seems slightly interesting to a human mathematician? And I’ve never seen that.
Dwarkesh Patel 00:30:19
I mean, they’ve been around for all of six months,
Tamay Besiroglu 00:30:23
I mean, that’s a long time. One mathematician might have been able to do a bunch of work over that time, and they have produced orders of magnitude fewer tokens on math.
Ege Erdil 00:30:34
And then I just want to emphasize it, because just think about the sheer scale of knowledge that these models have. It’s enormous from a human point of view. So it is actually quite remarkable that there is no interesting recombination, no interesting, “oh, this thing in this field looks kind of like this thing in this other field”. There’s no innovation that comes out of that. And it doesn’t have to be a big math concept, it could be just a small thing that maybe you could add to a Sunday magazine on math that people used to have. But there isn’t even an example of that.
Tamay Besiroglu 00:31:09
I think it’s useful for us to explain a very important framework for our thinking about what AI is good at and what AI is lagging in, which is this idea of Moravec’s paradox, that things that seem very hard for humans, AI systems tend to make much faster progress on, whereas things that look a bunch easier for us, AI systems totally struggle or are often totally incapable of doing that thing. And so this kind of abstract reasoning, playing chess, playing Go, playing Jeopardy, doing kind of advanced math and solving math problems.
Ege Erdil 00:31:49
There are even stronger examples, like multiplying 100 digit numbers in your head, which is just the one that got solved first out of almost any other problem. Or following very complex symbolic logic arguments, like deduction arguments, which people actually struggle with a lot. Like how do premises logically follow from conclusions? People have a very hard time with that. Very easy for formal proof systems.
Tamay Besiroglu 00:32:12
An insight that is related and is quite important here is that the tasks that humans seem to struggle on and AI systems seem to make much faster progress on are things that have emerged fairly recently in evolutionary time. So, advanced language use emerged in humans maybe 100,000 years ago, and certainly playing chess and Go and so on are very recent innovations. And so evolution has had much less time to optimize for them, in part because they’re very new, but also in part because when they emerged, there was a lot less pressure because it conferred kind of small fitness gains to humans and so evolution didn’t optimize for these things very strongly.
And so it’s not surprising that on these specific tasks that humans find very impressive when other humans are able to do it, that AI systems are able to make a lot of fast progress. In humans, these things are often very strongly correlated with other competencies, like being good at achieving your goals, or being a good coder is often very strongly correlated with solving coding problems, or being a good engineer is often correlated with solving competitive coding problems.
But in AI systems, the correlation isn’t quite as strong. And even within AI systems, it’s the case that the strongest systems on competitive programming are not even the ones that are best at actually helping you code. So o3 mini’s high seems to be maybe the best at solving competitive code problems, but it isn’t the best at actually helping you write code.
Ege Erdil 00:33:54
And it isn’t getting most of the enterprise revenue from places like Coursera or whatever, that’s just Claude, right?
Tamay Besiroglu 00:33:59
But an important insight here is that the things that we find very impressive when humans are able to do it, we should expect that AI systems are able to make a lot more progress on that. But we shouldn’t update too strongly about just their general competence or something, because we should recognize that this is a very narrow subset of relevant tasks that humans do in order to be a competent, economically valuable agent.
Dwarkesh Patel 00:34:26
Yeah. First of all, I actually just really appreciate that there is an AI organization out there where- because there’s other people who take the compute perspective seriously, or try to think empirically about scaling laws and data and whatever. And taking that perspective seriously leads people to just be like, “okay, 2027 AGI”, which might be correct, but it is just interesting to get, “no, we’ve also looked at the exact same arguments, the same papers, the same numbers. And we’ve come to a totally different conclusion”.
So I asked Dario this exact question two years ago, when I interviewed him, and it went viral.
Ege Erdil 00:35:11
Didn’t he say AGI in two years?
Dwarkesh Patel 00:35:13
That, but Dario’s always had short timelines.
Ege Erdil 00:35:15
Okay, but we are two years later.
Dwarkesh Patel 00:35:18
Did he say two years? I think he actually did say two years.
Ege Erdil 00:35:20
Did he say three years?
Tamay Besiroglu 00:35:21
So we have one more year.
Dwarkesh Patel 00:35:22
One more year.
Tamay Besiroglu 00:35:23
Better work hard.
Dwarkesh Patel 00:35:27
But he’s, I mean, I think he’s like, he in particular has not been that well calibrated. In 2018, he had like…
Tamay Besiroglu 00:35:33
I remember talking to a very senior person who’s now at Anthropic, in 2017. And then he told various people that they shouldn’t do a PhD because by the time they completed it everyone will be automated.
Dwarkesh Patel 00:35:49
So anyways, I asked him this exact same question because he has short timelines, which is that if a human knew the amount of things these models know, they would be finding all these different connections. And in fact, I was asking Scott about this the other day when I interviewed him, Scott Alexander, and he said, “look, humans also don’t have this kind of logical omniscience”.
I’m not saying we’re omniscient, but we have examples of humans finding these kinds of connections. This is not an uncommon thing, right? I think his response was that these things are just not trained in order to find these kinds of connections, but their view is that it would not take that much extra compute in order to build some RL environment in which they’re incentivized to find these connections. Next token prediction just isn’t incentivizing them to do this, but the RL required to do this would not be- that or set up some sort of scaffolds. I think actually Google DeepMind did do some similar scaffold to make new discoveries. And I didn’t look into how impressive the new discovery was, they claim that some new discovery was made by an LLM as a result.
On the Moravec paradox thing, this is actually a super interesting way to think about AI progress. But I would also say that if you compare animals to humans, long term intelligent planning… an animal is not gonna help you book a flight either. An animal is not gonna do remote work for you.
I think what separates humans from other animals is that we can hold long-term, we can come up with a plan and execute on it. Whereas other animals often had to go by instinct, or within the kinds of environments that they have evolutionary knowledge of, rather than, “I’m put in the middle of the savanna, or I’m put in the middle of the desert, or I’m put in the middle of tundra, and I’ll learn how to make use of the tools and whatever there”. I actually think there’s a huge discontinuity between humans and animals and their ability to survive in different environments, just based on their knowledge. And so it’s a recently optimized thing as well. And then I’d be like, “okay, well, we got it soon. AIs will optimize for it fast”.
Ege Erdil 00:37:50
Right. So I would say if you’re comparing animals to humans, it’s kind of a different thing.
I think if you could put the competences that the animals have into AI systems, that might just already get you to AGI already. I think the reason why there is such a big discontinuity between animals and humans is because animals have to rely entirely on natural world data, basically, to train themselves. Imagine that the only thing as a human that you saw was nobody talked to you, you didn’t read anything, you just had to learn by experience, maybe to some extent by imitating other people, but you have no explicit communication. It would be very inefficient.
What’s actually happening is that you have this- I think some other people have made this point as well- is that evolution is sort of this outer optimizer that’s improving the software efficiency of the brain in a bunch of ways. There’s some genetic knowledge that you inherit, not that much because there isn’t that much space in the genome. And then you have this lifetime learning, which is, you don’t actually see that much data during lifetime learning. A lot of this is redundant and so on.
So what seems to have changed with humans compared to other animals is that humans became able to have culture and they have language, which enables them to have a much more efficient training data modality compared to animals. They also have, I think, stronger ways in which they tend to imitate other humans and learn from their skills, so that also enables this knowledge to be passed on. I think animals are pretty bad at that compared to humans. So basically as a human, you’re just being trained on much more efficient data and that creates further insights to be then efficient at learning from it, and then that creates this feedback loop where the selection pressure gets much more intense.
So I think that’s roughly what happened with humans. But a lot of the capabilities that you need to be a good worker in the human economy, animals already have. So they have quite sophisticated sensory motor skills. I think they are actually able to pursue long-term goals.
Dwarkesh Patel 00:40:03
But ones that have been instilled by evolution. I think a lion will find a gazelle and that is a complicated thing to do and requires stalking and blah, blah, blah-
Ege Erdil 00:40:12
But when you say it’s been instilled by evolution, there isn’t that much information in the genome.
Dwarkesh Patel 00:40:16
But I think if you put the lion in the Sahara and you’re like, “go find lizards instead”.
Ege Erdil 00:40:22
Okay. So suppose you put a human and they haven’t seen the relevant training data.
Dwarkesh Patel 00:40:27
I think they’d be slightly better.
Ege Erdil 00:40:29
Slightly better, but not that much better. Again, didn’t you recently have an interview?
Dwarkesh Patel 00:40:36
Joseph Henrich.
Ege Erdil 00:40:37
Yeah. So he would probably tell you that.
Dwarkesh Patel 00:40:40
I think what you’re making is actually a very interesting and subtle point that has an interesting implication. So often people say that ASI will be this huge discontinuity, because while we have this huge discontinuity in the animal-to-human transition, not that much changed between pre-human primates and humans genetically, but it resulted in this humongous change in capabilities. And so they say, “well, why not expect something similar between human level intelligence and superhuman intelligence?”
And one implication of the point you’re making is actually it wasn’t that we just gained this incredible intelligence. Because of biological constraints, animals have just been held back in this really weird way that no AI system has been arbitrarily held back from not being able to communicate with other copies or with other knowledge sources. And so since AIs are not held back artificially in this way, there’s not going to be a point where we should take away that hobbling. And then now they explode.
Now, actually, I think I would disagree with that. The implication that I made, I would actually disagree with- I’m like a sort of unsteerable chain of thought.
We wrote a blog post together about AI corporations where we discuss actually there will be a similar unhobbling with future AIs, which is not about the intelligence, but a similar level of bandwidth and communication and collaboration with other AIs, which is a similar magnitude of change from non-human animals to humans, in terms of their social collaboration, that AIs will have with each other because of their ability to copy all their knowledge exactly, to merge, to distill themselves.
Tamay Besiroglu 00:42:28
Maybe before we talk about that, I think just a very important point to make here, which I think underlies some of this disagreement that we have with others about both this argument from the transition from kind of non-human animals to humans, is this focus on intelligence and reasoning and R&D, which is enabled by that intelligence as being enormously important. And so if you think that you get this very important difference from this transition from non-human primates to humans, then you think that in some sense you get this enormously important unlock from fairly small scaling and, say, brain size or something.
And so then you might think, “well, if we scale beyond the size of training runs, the amount of training compute that the human brain uses, which is maybe on the order of 1E24 flop or whatever, which we’ve recently surpassed, then maybe surpassing it just a little bit more enables us to unlock very sophisticated intelligence in the same way that humans have much more sophisticated intelligence compared to non-human primates”. And I think part of our disagreement is that intelligence is kind of important, but just having a lot more intelligence and reasoning and good reasoning isn’t something that will kind of accelerate technological change and economic growth very substantially.
It isn’t the case that the world today is totally bottlenecked by not having enough good reasoning, that’s not really what’s bottlenecking the world’s ability to grow much more substantially. I think that we might have some disagreement about this particular argument, but I think what’s also really important is just that we have a different view as to how this acceleration happens, that it’s not just having a bunch of really good reasoners that give you this technology that then accelerates things very drastically. Because that alone is not sufficient. You need kind of complementary innovations in other industries. You need the economy as a whole growing and supporting the development of these various technologies. You need the various supply chains to be upgraded. You might need demand for the various products that are being built.
And so we have this view where actually this very broad upgrading of your technology and your economy is important rather than just having very good reasoners and very, very, very good reasoning tokens that gives us this acceleration.

Dwarkesh Patel 00:45:04
All right. So this brings us back to the intelligence explosion. Here’s the argument for the intelligence explosion:
You’re right that certain kinds of things might take longer to come about, but this core loop of software R&D that’s required, if you just look at what kinds of progress is needed to make a more general intelligence, you might be right that it needs more experimental compute, but as you guys have documented, we’re just getting a shit-ton more compute every single year for the next few years. So you can imagine an intelligence explosion in the next few years where in 2027, there’ll be like 10 X more compute than there is now for AI.
And you’ll have this effect where the AIs that are doing software R&D are finding ways to make running copies of them more efficient, which has two effects. One, you’re increasing the population of AIs who are doing this research, so more of that in parallel can find these different optimizations. And a subtle point that they’d often make here is software R&D in AI is not just Ilya-type coming up with new transformer-like architectures.
To your point, it actually is a lot of- I mean, I’m not an AI researcher, but I assume there’s, from the lowest level libraries to the kernels, to making RL environments, to finding the best optimizer, to… there’s just so much to do, and in parallel you can be doing all these things or finding optimizations across them. And so you have two effects, going back to this. One is, if you look at the original GPT-4 compared to the current GPT-4o, I think it’s, what, how much cheaper is it to run?
Tamay Besiroglu 00:46:57
It’s like, maybe a hundred times for the same capability or something.
Dwarkesh Patel 00:47:03
Right. So they’re finding ways in which to run more copies of them at a hundred X cheaper or something, which means that the population of them is increasing and the higher populations are helping you find more efficiencies.
And not only does that mean you have more researchers, but to the extent that the complementary input is experimental compute, it’s not the compute itself, it’s the experiments.
And the more efficient it is to run a copy or to develop a copy, the more parallel experiments you can run, because now you can do a GPT-4 scale training run for much cheaper than you could do it in 2024 or 2023. And so for that reason, also this software-only singularity sees more researcher copies who can run experiments for cheaper, dot, dot, dot. They initially are maybe handicapped in certain ways that you mentioned, but through this process, they are rapidly becoming much more capable. What is wrong with this logic?
Tamay Besiroglu 00:47:57
So I think the logic seems fine. I think this is like a decent way to think about this problem, but I think that it’s useful to draw on a bunch of work that, say, economists have done for studying the returns to R&D and what happens if you 10X your inputs, the number of researchers, what happens to innovation or the rate of innovation.
And there, they point out these two effects where, as you do more innovation and you get to stand on top of the shoulders of giants and you get the benefit from past discoveries and it makes you as a scientist more productive. But then there’s also kind of diminishing returns, that the low hanging fruit has been picked, and it becomes harder to make progress. And overall, you can summarize those estimates as thinking about the kind of returns to research effort.
And we’ve looked into the returns to research effort in software specifically. And we look at a bunch of domains in traditional software or linear integer solvers or SAT solvers, but also in AI; computer vision and RL and language modeling. And there, if this model is true, that all you need is just cognitive effort, it seems like the estimates are a bit ambiguous about whether this results in this acceleration or whether it results in just merely exponential growth.
And then you might also think about, well, it isn’t just your research effort that you have to scale up to make these innovations, because you might have complementary inputs. So as you mentioned, experiments are the thing that might kind of bottleneck you. And I think there’s a lot of evidence that in fact, these experiments and scaling up hardware, it’s just very important for getting progress in the algorithms and the architecture and so on. So in AI- this is true for software in general- where if you look at progress in software, it often matches very closely the rate of progress we see in hardware. So for traditional software, we see about a 30% roughly increase per year, which kind of basically matches Moore’s law. And in AI, we’ve seen the same until you get to the deep learning era, and then you get this acceleration, which in fact coincides with the acceleration we see in compute scaling, which gives you a hint that actually the compute scaling might have been very important.
Other pieces of evidence besides this coincidental rate of progress, other pieces of evidence are the fact that innovation and algorithms and architectures are often concentrated in GPU-rich labs and not in the GPU-poor parts of the world, like academia or maybe smaller research institutes. That also suggests that having a lot of hardware is very important. If you look at specific innovations that seem very important, the big innovations over the past five years, many of them have some kind of scaling or hardware-related motivation. So you might look at how the transformer itself was about how to harness more parallel compute. Things like flash attention was literally about how to implement the attention mechanism more efficiently, or things like the chinchilla scaling law.
And so many of these big innovations were just about how to harness your compute more effectively. That also tells you that actually the scaling of compute might be very important. And I think there’s just many pieces of evidence that point towards this complementarity picture.
So I would say that even if you assume that experiments are not particularly important, the evidence we have, both from estimates of AI and other software- although the data is not great- suggests that maybe you don’t get this kind of hyperbolic, faster-than-exponential super-growth in the overall algorithmic efficiency of systems.
Dwarkesh Patel 00:51:56
I’m not sure I buy the argument that because these two things compute and AI progress have risen so concomitantly that this is a sort of causal relationship.
So broadly, the industry as a whole has been getting more compute and as a result, making more progress. But if you look at the top players, there’s been multiple examples of a company with much less compute, but a more coherent vision, more concentrated research effort, being able to beat an incumbent who has much more compute. So OpenAI initially beating Google DeepMind. And if you remember, there were these emails that were released between Elon and Sam and so forth like, “we got to start this company because they’ve got this bottleneck on the compute” and, “look how much more compute Google DeepMind has”. And then OpenAI made a lot of progress. Similarly now with OpenAI versus Anthropic and so forth. And then I think just generally, your argument is just too ‘outside view’. And we actually do know a lot about this very macro economic argument that I’m like, well, why don’t we just ask the AI researchers?
Tamay Besiroglu 00:53:01
I mean, AI researchers will often kind of overstate the extent to which just cognitive effort and doing research is important for driving these innovations, because that’s often convenient or useful. They will say the insight was derived from some nice idea about statistical mechanics or some nice equation in physics that says that we should do it this way. But often that’s an ad hoc story that they tell to make it a bit more compelling to reviewers.
Dwarkesh Patel 00:53:35
So Daniel Kokotajlo mentioned this survey he did where he asked a bunch of AI researchers, “if you had one thirtieth the amount of compute”- and he did one thirtieth because AI’s will be, they suppose, 30 times faster- “If you had one thirtieth the amount of compute, how much would your progress slow down?” And they say, “I make a third of the amount of progress I normally do”. So that’s just a pretty good substitution effect of, you get one tenth the compute, your progress only goes down one third.
And then I was talking to an AI researcher the other day, one of these cracked people, gets paid tens of millions of dollars a year, probably. And we asked him, how much does the AI models help you in AI research? And he said, “in domains that I’m already quite familiar with, where it’s closer to autocomplete, it saves me four to eight hours a week”. And then he said, “but in domains where I’m actually less familiar, where I need to drive new connections, I need to understand how these different parts relate to each other, and so forth. It saves me close to 24 to 36 hours a week”.
And that’s the current models. And I’m just like, “he didn’t get more computed, but it still saved him like a shit ton more time”. Just draw that forward. That’s a crazy implication or crazy trend, right?
Ege Erdil 00:54:58
I mean, I’m skeptical of the claims that we have actually seen that much of an acceleration in the process of R&D. These claims seem to me, like they’re not borne out by the actual data I’m seeing. So I’m not sure how much to trust them.
Dwarkesh Patel 00:55:18
I mean, on the general intuition that cognitive effort alone can give you a lot of AI progress, it seems like a big important thing the labs do is this science of deep learning. Scaling laws… I mean, it ultimately netted out in an experiment, but the experiment is motivated by cognitive effort.
Ege Erdil 00:55:36
So for what it’s worth, when you say that A and B are complementary, you’re not saying, just as A can bottleneck you, B can also bottleneck you. So when you say you need compute and experiments and data, but you also need cognitive effort, that doesn’t mean the lab which has the most compute is going to win, right? That’s a very simple point, either one can be the bottleneck.
I mean, if you just have a really dysfunctional culture and you don’t actually prioritize using your computer very well and you just waste it, well then you’re not going to make a lot of progress, right? So it doesn’t contradict the picture that someone with a much better vision, a much better team, much better prioritization can make better use of their compute if someone else was just bottlenecked heavily on that part of the equation. The question here is, once you get these automated AI researchers and you start this software singularity, your software efficiency is going to improve by many orders of magnitude, while your compute stock, at least in the short run, is going to remain fairly fixed. So how many OOMs of improvement can you get before you become bottlenecked by the second priority equation? And once you actually factor that in, like how much progress should you expect?
That’s the kind of question I think people don’t have. I think it’s hard for people to have good intuitions about this because people usually don’t run the experiments. So you don’t get to see at a company level, or at an industry level, what would have happened if the entire industry had 30 times less compute. Maybe as an individual, what would happen if you had 30 times less compute? You might have a better idea about that, but that’s a very local experiment and you might be benefiting a lot from spillovers from other people who actually have more compute. So because this experiment was never run, it’s sort of hard to get direct evidence about the strength of complementarity.
Dwarkesh Patel 00:57:27
Actually, what is your probability of, if we live in the world where we get AGI in 2027, that there is a software-only singularity?
Tamay Besiroglu 00:57:35
Quite high, because you’re conditioning on compute not being very large. So it must be that you get a bunch of software progress.
Dwarkesh Patel 00:57:44
Yeah, right, right. Like you just have a bunch of leverage from algorithmic progress in that world.
Tamay Besiroglu 00:57:50
OK, that’s right.
Dwarkesh Patel 00:57:51
So then maybe, because I was thinking these are independent questions-
Tamay Besiroglu 00:57:54
I think a call that I want to make is, I know that some labs do have multiple pre-training teams and they give people different amounts of resources for doing the training and different amounts of cognitive effort, different size of teams. But none of that, I think, has been published. And I’d love to see the results of some of those experiments.
I think even that won’t update you very strongly just because it is often just very inefficient to do this very imbalanced scaling of your factor inputs. And in order to really get an estimate of how strong these complementarities are, you need to observe these very imbalanced scale-ups. And so that rarely happens.
And so I think the data that bears on this is just really quite poor. And then the intuitions that people have also don’t seem clearly relevant to the thing that matters about what happens if you do this very imbalanced scaling and where does this net out?
Dwarkesh Patel 00:58:53
One question I have, which it would be really interesting if somebody can provide an example of: maybe through history, there was some point at which because of a war or some other kind of supply shock, you had to ramp up production or ramp up some key output that people really cared about, while for some weird historical reason, many of the key inputs were not accessible to a ramp-up, but you could ramp-up one key input.
I’m talking in very abstract terms. You see what I’m saying, right? You need to make more bombers, but you ran out of aluminum and you need to figure out something else to do. And how successful these efforts have been or whether you just keep getting bottlenecked?
Ege Erdil 00:59:35
Well, I think that is not quite the right way to do it. Because I think if you’re talking about materials, then I think there’s a lot of sense in which different materials can be substitutable for one another in different ways. You can use aluminum. I mean, aluminum is a great metal for making aircraft because it’s light and durable and so on. But you can imagine that you make aircraft with worse metals and then it just takes more fuel and it’s less efficient to fly.So there’s a sense in which you can compensate and just cost more.
I think it’s much harder if you’re talking about something like complementarity between labor and capital, complementarity between remote work and in-person work or skilled or unskilled work. There are input pairs for which I would expect it to be much more difficult. For example, you’re looking at the complementarity between the quality of leadership of an army and its number of soldiers. There is some effect there, but if you just scale up, you have excellent leadership, but your army only has 100 people. You’re not going to get very far.
Dwarkesh Patel 01:00:40
King Leonidas and Thermopylae?
Ege Erdil 01:00:44
Well, they lost, right?
Dwarkesh Patel 01:00:47
It would be funny if we’re building models and software-only singularity and we’re like, “what exactly happened in Thermopylae?” It’s somehow relevant.
Ege Erdil 01:00:53
I can actually talk about that, but we probably shouldn’t.

Dwarkesh Patel 01:00:57
Okay, sure. So the audience should know, my most popular guest by far is Sarah Paine. Not only is she my most popular guest, she’s my most popular four guests. Because all four of those episodes that I’ve done with her are, from a viewer-minute adjusted basis, I host the Sarah Paine Podcast where I occasionally talk about AI.
Anyways, we did this three-part lecture series where one of them was about India-Pakistan wars through history. One of them was about Japanese culture before World War II. The third one was about the Chinese Civil War. And for all of them, my history tutor was Ege. And, why does he know so much about fucking random 20th century conflicts? But he did, and he suggested a bunch of the good questions I asked her. We’ll get into that in a second. Ege, what’s going on there?
Ege Erdil 01:01:56
I don’t know. I mean, I don’t really have a good question. I think it’s interesting. I mean, I read a bunch of stuff, but it’s a kind of boring answer. I don’t know. Imagine you ask a top AI researcher, “What’s going on? How are you so good?” And then they will probably give you a boring answer. Like, I don’t know.
Dwarkesh Patel 01:02:13
That itself is interesting that often these kinds of questions elicit boring answers. It tells you about the nature of the skill. How’d you find him?
Tamay Besiroglu 01:02:22
We connected on a Discord for Metaculus, which is this forecasting platform. And I was a graduate student at Cambridge at the time doing research in economics. And I was having conversations with my peers there. And I was occasionally having conversations with Ege. And I was like, “this guy knows a lot more about economics”. And at the time he was a computer science undergrad in Ankara. And he knows more about economics and about these big trends in economic growth and economic history than almost any of my peers at the university. And so like, what the hell is up with that?
And then we started having frequent collaborations and ended up hiring Ege for Epoch because it clearly makes sense for him to work on these types of questions.
Dwarkesh Patel 01:03:17
And it seems like at Epoch, you’ve just collected this group of internet misfits and weirdos.
Tamay Besiroglu 01:03:23
Yeah, that’s right.
Dwarkesh Patel 01:03:24
How did you start Epoch? And then how did you accomplish this?
Tamay Besiroglu (01:03:27
So I was at MIT doing more research, and I was pretty unhappy with the bureaucracy there where it was very hard for me to scale projects up, hire people. And I was pretty excited about a bunch of work that my PI wasn’t excited about because it’s maybe hard to publish or it doesn’t confer the same prestige. And so I was chatting with Jaime Sevilla, one of the co-founders, and we just collaborated on projects and then thought we should just start our own org, because we can just hire people and work on the projects we were excited about. And then I just hired a bunch of the insightful misfits that like…
Dwarkesh Patel 01:04:12
But was the thesis like, “oh, there’s a bunch of underutilized internet misfits and therefore this org was successful”? Or you started the org and then you were like…
Tamay Besiroglu 01:04:20
I think it’s more of the latter. So it was more like we can make a bunch of progress because clearly academia and industry is kind of dropping the ball on a bunch of important questions that academia is unable to publish interesting papers on. Industry is not really focused on producing useful insights. And so it seemed very good for us to just do that. And also the timing was very good. So we started just before ChatGPT and we wanted to have much more grounded discussions of the future of AI.
And I was frustrated with the quality of discussion that was happening on the internet about the future of AI. And to some extent or to a very large extent, I still am. And that’s a large part of what motivates me to do this. It’s just born out of frustration with bad thinking and arguments about where AI is going to go.

Dwarkesh Patel 01:06:24
OK, so let me ask you about this: So just to set the scene for the audience, we’re going to talk about the possibility of this explosive economic growth and greater than 30 percent economic growth rates. So I want to poke you both from a perspective of “maybe suggesting that this isn’t aggressive enough in the right kind of way, because it’s maybe it’s too broad”, and then I’ll poke you from the more normal perspective that, “hey, this is fucking crazy”.
Ege Erdil 01:06:54
I imagine it would be difficult for you to do the second thing.
Dwarkesh Patel 01:06:57
No, I mean, I think it might be fucking crazy, let’s see. The big question I have about this broad automation, I get what you’re saying about the Industrial Revolution, but in this case, we can just make this argument that you get this intelligence and then what you do next is you go to the desert and you build this Shenzhen of robot factories which are building more robot factories, which are building… If you need to do experiments then you build bio labs and you build chemistry labs and whatever.
Ege Erdil 01:07:30
Or you can build Shenzhen in the desert. I agree that looks much more plausible than a software-only singularity.
Dwarkesh Patel 01:07:35
But the way you’re framing it, it sounds like McDonald’s and Home Depot and fucking whatever are growing at 30 percent a year as well. The aliens’ level view of the economy is that there’s a robot economy in the desert that’s growing at 10,000 percent a year and everything else is the same-old-same-old, or is it like-
Ege Erdil 01:07:57
No, I mean, there is a question about what would be possible, or physically possible, and what would be the thing that would actually be efficient. So it might be the case, and again, once you’re scaling up the hardware part of the equation as well as the software part, then I think the case for this feedback loop gets a lot stronger. If you scale up data collection as well, I think it gets even stronger, real world data collection by deployment and so on.
But building Shenzhen in a desert… if you think about the pipeline; so far we have relied first on the entire semiconductor supply chain. That industry depends on tons of inputs and materials. And it gets from probably tons of random places in the world. And creating that infrastructure, doubling, or tripling, whatever, that infrastructure, the entire thing. That’s very hard work. So probably you couldn’t even do it, even if you just have Shenzhen in a desert, that will be even more expensive than that.
And on top of that, so far, we have been drawing heavily on the fact that we have built up this huge stock of data, over the past 30 years or something, on the internet. Imagine you were trying to train a state-of-the-art model, but you only have 100 billion tokens to train on. That would be very difficult. So in a certain sense, our entire economy has produced this huge amount of data on the internet that we are now using to train the models. It’s plausible that in the future, when you need to get new competencies added to these systems, the most efficient way to do that will be to try to leverage similar kinds of modalities of data, which will also require this… you would want to deploy the systems broadly because that’s going to give you more data. And maybe you can get where you want to be without that, but it would just be less efficient if you’re starting from scratch compared to if you’re collecting a lot of data.
I think this is actually a motivation for why labs want their LLMs to be deployed widely, because sometimes when you talk to ChatGPT, it’s going to give you two responses and it’s going to say, well, which one was good? Or it’s going to give you one response and it’s going to ask you, was this good or not? Well, why are they doing that, right? That’s a way in which they are getting user data through this extremely broad deployment. So I think you should just imagine that thing to continue to be efficient and continue to increase in the future because it just makes sense.
And then there’s a separate question of, well, suppose you didn’t do any of that. Suppose you just tried to imagine the most rudimentary, the narrowest possible kind of infrastructure build-out and deployment that would be sufficient to get this positive feedback loop that leads to much more efficient AIs. I agree that loop could, in principle, be much smaller than the entire world. I think it probably couldn’t be as small as Shenzhen in the desert, but it could be much smaller than the entire world. But then there’s a separate question of, would you actually do that? Would that be efficient? I think some people have the intuition that there are just these extremely strong constraints, maybe regulatory constraints, maybe social political constraints, to doing this broad deployment. They just think it’s going to be very hard.
So I think that’s part of the reason why they imagine these narrower scenarios where they think it’s going to be easier. But I think that’s overstated. I think people’s intuitions for how hard this kind of deployment is comes from cases where the deployment of the technology wouldn’t be that valuable. So it might come from housing. We have a lot of regulations on housing. Maybe it comes from nuclear power. Maybe it comes from supersonic flights. I mean, those are all technologies that would be useful if they were maybe less regulated. But they wouldn’t double.
Tamay Besiroglu 01:11:52
I think the core point here is the value of AI automation and deployment is just extremely large, even just for workers. There might be some kind of displacement and there might be some transition that you need to do in order to find a job that works for you, but otherwise the wages could still be very high for a while at least.
And on top of that, the gains from owning capital might be very enormous. And in fact, a large share of the US population would benefit… They benefit, they own housing, they have 401ks. Those would do enormously better when you have this process of broad automation and AI deployment. And so I think there could just be a very deep support for some of this, even when it’s totally changing the nature of labor markets and the skills and occupations that are in demand.
Ege Erdil 01:12:55
So I would just say it’s complicated. I think what the political reaction to it will be when this starts actually happening, I think the easy thing to say is that, yeah, this will become a big issue and then it will be maybe controversial or something. But what is the actual nature of the reaction in different countries? I think that’s kind of hard to forecast. I think the default view is like, “well, people are going to become unemployed, so it will just be very unpopular”. I think that’s very far from obvious.
And I just expect heterogeneity in how different countries respond. And some of them are going to be more liberal about this and going to have a much broader deployment. And those countries probably end up doing better. So just like during the Industrial Revolution, some countries were just ahead of others. I mean, eventually almost the entire world adopted the sort of norms and culture and values of the Industrial Revolution in various ways.
Tamay Besiroglu 01:13:44
And actually, you say they might be more liberal about it, but they might actually be less liberal in many ways. In fact, that might be more functional in this world in which you have broad AI deployment. We might adopt the kind of values and norms that get developed in, say, the UAE or something, which is maybe focused a lot more on making an environment that is very conducive for AI deployment. And we might start emulating and adopting various norms like that. And they might not be classical liberal norms, but norms that are just more conducive to AI being functional and producing a lot of value.
Ege Erdil 01:14:27
This is not meant to be a strong prediction, this is just an illustrative. It might just be the freedom to deploy AI in the economy and build out lots of physical things at scale, maybe that ends up being more important in the future. Maybe that is still missing something, maybe there are some other things that are also important. The generic prediction that you should expect variance and some countries do better than others, I think that’s much easier to predict than the specific countries that end up doing better.
Dwarkesh Patel 01:14:55
Yeah. Or the norms that that country wants.
Tamay Besiroglu 01:14:56
That’s right.
Dwarkesh Patel 01:14:57
One thing I’m confused about is, if you look at the world of today versus the world of 1750, the big difference is just we’ve got crazy tech that they didn’t have back then. We’ve got these cameras, we’ve got these screens, and we’ve got rockets and so forth. And that just seems like the result of technological growth and R&D and so forth.
Ege Erdil 01:15:22
It’s a capital accumulation.
Dwarkesh Patel 01:15:23
Well, explain that to me because you’re just talking about this infrastructure build out and blah, blah, blah. I’m like, but why won’t they just fucking invent the kinds of shit that humans would have invented by 2050?
Ege Erdil 01:15:37
Producing this stuff takes a lot of infrastructure build-out.
Dwarkesh Patel 01:15:40
But that infrastructure is built out once you make the technology, right?
Tamay Besiroglu 01:15:45
I don’t think that’s right. There isn’t this temporal difference where it’s first you do the invention… often there’s this interplay between the actual capital buildup and the innovation.
Ege Erdil 01:15:57
Learning curves are about this, right, fundamentally? What has driven the increase in the efficiency of solar panels over the past 20, 30 years?
Tamay Besiroglu 01:16:05
It isn’t just like people had the idea of 2025 solar panels. Nobody 20 years ago had the sketch for the 2025 solar panel. It’s this kind of interplay between having ideas, building, learning, producing, and-
Ege Erdil 01:16:24
Other complementary inputs also becoming more efficient at the same time, like you might get better materials. For example, the fact that smelting processes got a lot better towards the end of the 19th century, so it became a lot easier to work with metal, maybe that was a crucial reason why aircraft technology later became more popular.
It’s not like someone came up with the idea of, “oh, you can just use something that just has wings and has a lot of thrust, and then that might be able to fly”. That basic idea is not that difficult, but then, well, how do you make it actually a viable thing? Well, that’s much more difficult.
Dwarkesh Patel 01:17:04
Have you seen the meme where two beavers are talking to each other and they’re looking at the Hoover Dam? One of them’s like, “well, I didn’t build that, but it’s based on an idea of mine”. The point you’re making is that this invention-focused look on tech history underplays the work that goes into making specific innovations practicable and to deploy them widely.
Ege Erdil 01:17:33
It’s just hard, I think. Suppose you want to write a history of this, you want to write the history of how the light bulb was developed or something. It’s just really hard. Because to understand why specific things happen at specific times, you probably need to understand so much about the economic conditions of the time.
For example, Edison spent a ton of time experimenting with different filaments to be using the light bulb. The basic idea is very simple. You make something hot and it glows, but then what filament actually works well for that in a product? What is durable? What has the highest ratio of light output versus heat so that you have less waste, it’s more efficient. And even after you have the product, then you’re facing the problem, well, it’s 1880 or something and US homes don’t have electricity, so then nobody can use it. So now you have to build power plants and build power lines to the houses so that people have electricity in their homes so that they can actually use this new light bulb that you created. So he did that, but then people present it as if it’s like, “okay, he just came up with the idea”, like “it’s a light bulb”.
Dwarkesh Patel 01:18:46
I guess the thing people would say is, you’re right about how technology would progress if we were humans deploying for the human world. But what you’re not counting is there’s going to be this AI economy where maybe they need to do this kind of innovation and learning by doing when they’re figuring out how to, “I want to make more robots because they’re helpful and so we’re going to build more robot factories, we’ll learn and then we’ll make better robots” or whatever. But geographically, that is a small part of the world that’s happening in. You understand what I’m saying? It’s not like, “and then they walk in your building and then you do a business transaction with Lunar Society podcast LLC and then”, you know what I mean?
Ege Erdil 01:19:30
For what it’s worth, if you look at the total surface area of the world, it might well be the case that the place that initially experiences this very fast growth is a small percentage of the surface area of the world. And I think that was the same for the Industrial Revolution, it was not different.
Dwarkesh Patel 01:19:49
What concretely does this explosive growth look like? If I look at this heat map of growth rates on the globe, is there just going to be one area that is blinding hot and that’s the desert factories with all these experiments and like…
Ege Erdil 01:20:03
I would say our idea is that it’s going to be broader than that, but probably initially… So eventually it would probably be most of the world. But as I said, because of this heterogeneity, because I think some countries are going to be faster in adoption than others, maybe some cities will have faster adoption than others, that will mean that there are differentials and some countries might have much faster growth than other countries.
But I would expect that at a jurisdiction level, it will be more homogenous. So, for example, I expect the primary obstacles to come from things like regulation. And so I would just imagine it’s being more delineated by regulatory jurisdiction boundaries than anything else.
Dwarkesh Patel 01:20:48
Got it. So you may be right that this infrastructure build-out and capital deepening and whatever l is necessary for a technology to become practical, but…
Ege Erdil 01:20:57
Or even to be discovered. There’s an aspect of it where you discover certain things by scaling up, learning by doing, that’s the [?] learning curve. And there’s this separate aspect where, suppose that you become wealthier, well, you can invest that increased wealth in, you use it to accumulate more capital, but you also can invest it in R&D and other ways.
Tamay Besiroglu 01:21:21
You get Einstein out of the patent office. You need some amount of resources for that to make sense. And you need the economy to be of a certain scale. You also need demand for the product you’re building. So, you could have the idea, but if the economy is just too small that there isn’t enough demand for you to be specializing and producing the semiconductor or whatever, because there isn’t enough demand for it, then it doesn’t make sense.
A much larger scale of an economy is useful in many ways in delivering complementary innovations and discoveries happening through serendipity, producing, having there be consumers that would actually pay enough for you to recover your fixed costs of doing all the experimentation and the invention. You need the supply chains to exist to deliver the germanium crystals that you need to grow in order to come up with the semiconductor. You need a large labor force to be able to help you do all the experiments and so on.
Dwarkesh Patel 01:22:20
I think the point you’re illustrating is, “look, could you have just figured out that there was a Big Bang by first principles reasoning?” Maybe. But what actually happened is we had World War II and we discovered radio communications in order to fight and effectively communicate during the war.
And then that technology helped us build radio telescopes. And then we discovered the cosmic microwave background. And then we had to come up with an explanation for the cosmic microwave background. And then we discovered the Big Bang as a result of World War II.
Tamay Besiroglu 01:22:46
People underemphasize that giant effort that goes into this build-up of all the relevant capital and all the relevant supply chains and the technology. I mean earlier you were making a similar comment when you were saying, “oh reasoning models actually in hindsight, they look pretty simple”, but then you’re ignoring this giant upgrading of the technology stack that happened, that took five to 10 years prior to that. And so I think people just underemphasize the support that is had from the overall upgrading of your technology, of the supply chains, of various sectors that are important for that.
And people focus on just specific individuals of like, Einstein had this genius insight and he was the very pivotal thing in the causal chain that resulted in these discoveries. Or Newton was just extremely important for discovering calculus without thinking about, well, there were all these other factors that produced lenses, that produced telescopes, that got the right data and that made people ask questions about dynamics and so on that motivated some of these questions. And those are also extremely important for scientific and technological innovation.
Dwarkesh Patel 01:24:06
And then, as you were saying, one of Conquest laws is, the more you understand about a topic, the more conservative you become about that topic. And so there may be a similar law here, where the more you understand about an industry, the more- obviously, I’m just a commentator, or a podcaster, but I understand AI better than any other industry I understand. And I have the sense from talking to people like you that, “oh, so much went into getting AI to the point where it is today”. Whereas when I talk to journalists about AI, they’re like, “okay, who is a crucial person we need to cover?” And they’re like, “should we get in touch with Geoffrey Hinton? Should we get in touch with Ilya?” And I just have this like, “you’re kind of missing the picture”.
But then you should have that same attitude towards things you… Or maybe it’s a similar phenomenon to Gell-Mann amnesia, we should have a similar attitude towards other industries.
Ege Erdil 01:24:59
Robin Hanson has this abstraction of seeing things in near mode versus far mode. And I think if you don’t know a lot about the topic, then you see it in far mode and you simplify things, you see a lot more detail. In general, I think the thing I would say, and the reason I also believe that abstract reasoning and deductive reasoning or even Bayesian reasoning by itself is not sufficient or is not as powerful as many other people think, is because I think there’s just this enormous amount of richness and detail in the real world that you just can’t reason about it. You need to see it. And obviously that is not an obstacle to AI being incredibly transformative because as I said, you can scale your data collection, you can scale experiments you do both in the AI industry itself and just more broadly in the economy, so you just discover more things. More economic activity means we have more exposed surface area to have more discoveries.
All of these are things that have happened in our past, so there’s no reason that they couldn’t speed up. The fundamental thing is that there’s no reason fundamentally why economic growth can’t be much faster than it is today. Like it’s probably as advanced right now just because humans are such an important bottleneck. They both supply the labor. They play crucial roles in the process of discovery of various kinds of productivity growth. There’s just strong complementarity to some extent with capital that you can’t substitute machines and so on for humans very well. So the growth of the economy and growth productivity just ends up being bottlenecked by the growth of human population.
Dwarkesh Patel 01:27:39
So let me ask you a tangential question. What’s been happening in China over the last 50 years, would you describe that as, in principle, the same kind of explosive growth that you expect from AI? Because there’s a lot of labor that makes the marginal product of capital really high, which allows you to have 10% plus economic growth rates. Is that basically in principle from AI?
Ege Erdil 01:28:01
So I would say in some ways it’s similar, in some ways it’s not. Probably the most important way in which it’s not similar is that in China, you see a massive amount of capital accumulation, a substantial amount of adoption of new technologies and probably also human capital accumulation to some extent. But you’re not seeing a huge scale up in the labor force. While for AI, you should expect to see a scale up in the labor force as well, not in the human workforce, but in the AI workforce.
Dwarkesh Patel 01:28:34
And I think you did, maybe not consecutive increases in the labor force…
Tamay Besiroglu 01:28:38
The key thing here is just the simultaneous scaling of both these things. And so you might ask the question of “isn’t it basically half of what’s going to happen with AI that you scale up capital accumulation in China?” But actually if you get both of these things to scale, that gives you just much faster growth and a very different picture.
Ege Erdil 01:29:04
But at the same time, if you’re just asking what 30 percent growth per year would look like, if you just want to have an intuition for how transformative that would be in concrete terms, then I think looking at China is not such a bad case. Especially in the 2000s or maybe late 90s, that seems slower than what we’re forecasting.
Tamay Besiroglu 01:29:24
Right. I think also looking at the Industrial Revolution is pretty good.
Ege Erdil 01:29:26
Well, the Industrial Revolution is very slow.
Tamay Besiroglu 01:29:28
But just in terms of the margins along which we made progress in terms of products. So the thing that didn’t happen during the industrial revolution is we just produced a lot more of things that people were producing prior to the industrial revolution, like producing a lot more crops and maybe a lot more kind of pre-Industrial Revolution style houses or whatever, on farms. Instead, what we got is along pretty much every main sector of the economy, we just had many different products that are totally different from what was being consumed prior to that. So in transportation, in food.
Ege Erdil 01:30:13
I mean, health care is a very big deal and antibiotics.
Dwarkesh Patel 01:30:16
So another question, because I’m not sure I understand how you’re defining the learning by doing versus explicit R&D, because there’s the way for taxes that companies say what they call R&D. But then there’s the intuitive understanding of R&D. So if you think about how AI is boosting TFP, you could say that right now, if you just had replaced the TSMC process engineers with AIs and they’re finding different ways in which to improve that process and improve efficiencies, improve yield, I would kind of call that R&D. On the other hand, you emphasize this other part of TFP, which is like better management and that kind of stuff.
Ege Erdil 01:30:59
The learning by doing could be, you could-
Dwarkesh Patel 01:31:00
But how much “umph” are you… Like you’re going to get to the fucking Dyson Sphere by better management?
Ege Erdil 01:31:05
But that’s not the argument, right? The point is that there are all these different things, some of them are maybe more complimentary than others. The point is not that you can get to a Dyson sphere by just scaling labor and capital. That’s not the point. You need to scale everything at once. So just as you can’t get to a Dyson sphere by just scaling labor and capital, you also can’t get to it by just scaling TFP. That doesn’t work.
Tamay Besiroglu 01:31:30
I think there’s a very important distinction between what is necessary to scale, to get to this Dyson sphere world and what is important. Like in some sense, producing food is necessary. But of course, producing food doesn’t get you to a Dyson sphere, right? So I think R&D is necessary, but on its own isn’t sufficient. And scaling up the economy is also necessary. On its own, it’s not sufficient. And then you can ask the question, what is the relative importance of each?
Ege Erdil 01:32:00
So I think our view here is very much the same. It is very connected to our view about the software R&D thing where we’re just saying there are these bottlenecks, so you need to scale everything at once. This is just a general view.
But I think people misunderstand us sometimes as saying that R&D is not important. No, that’s not what we’re saying. We’re saying it is important. It is less important in relative terms than some other things, none of which are by themselves sufficient to enable this growth. So the question is, how do you do the credit attribution? One of my missions in economics is to look at the elasticities of output to the different factors. Capital is less important than labor, because labor elasticity output is like 0.6, while for capital it’s like 0.3. But neither are by themselves sufficient. If you just scaled one of them and the other remained fixed, then neither would be sufficient to indefinitely scale output.

Dwarkesh Patel 01:33:00
One question that Daniel posed to me is, because I made this perspective about everything being interconnected when you were talking about… another example people often bring up is what would it take to build the iPhone in the year 1000? And it’s unclear how you could actually do that without just replicating every intermediate technology or most intermediate technologies.
And then he made the point like, OK, fine, whatever. Nanobots are not a crux here. The crux, at least to the thing he cares about, which is human control, is just by when can the robot economy, or the AI economy, whether it’s a result of capital deepening or whether it’s a result of R&D, by when will they have the robots? And they have more cumulative physical power?
Ege Erdil 01:33:50
Right. But he’s imagining a separate thing called the AI economy. Well, why would you imagine that? I think it’s probably downstream of his views about the software-only singularity. But again, those are views that we don’t share.
Tamay Besiroglu 01:34:01
So it’s just much more efficient for AI to operate in our economy and benefit from the existing supply chains and existing markets rather than set up shop on some island somewhere and do its own thing.
Ege Erdil 01:34:16
And then it’s not being clear, for example people might have the intuition- I brought this up before- the distinction between what is the minimum possible amount of build-out that would be necessary to get this feedback loop up and running and what would be the most efficient way to do it? Which are not the same question. But then people have this view that, oh, the most efficient thing in principle, we can’t do that because…
Dwarkesh Patel 01:34:36
I think the example he might give is when the conquistadors arrived in the New World or when the East India Trading Company arrived in India, they did integrate into the existing economy. In many cases, it depends on how you define ‘integrate’, but the Spanish relied heavily on New World labor in order to do silver mining and whatever. East India Trading Company was just a ratio of British people to Indian people, which is not that high. So they just had to rely on the existing labor force. But they were still able to take over because of… I don’t know what the analogous thing here is, but you see what I’m saying.
And so he’s concerned about, by when will they, even if they’re ordering components off of Alibaba or whatever- and sorry, I’m being trite, but you see what I’m saying. Even if they’re going to get into the supply chains, by when are they in a position where, because this part of the economy has been growing much faster, they could take over the government or…
Ege Erdil 01:35:40
If they wanted to?
Dwarkesh Patel 01:34:41
That’s right, yeah.
Ege Erdil 01:35:42
Okay. So I think that eventually you expect the AI systems to be driving most of the economy. And unless there are some very strange coincidences where humans are able to somehow uplift themselves and able to become competitive with the AIs by stopping being biological humans or whatever, which seems very unlikely early on, then AI is just going to be much more powerful. And I agree that in that world, if the AI is just somehow coordinated and decides, “okay, we should just like take over” or something, they just somehow coordinated to have that goal, then they could probably do it.
But, that’s also probably true in our world. In our world, if the US wanted to invade Sentinel Island, then probably they could do it. I don’t think anyone could stop them. But what does it actually mean? There’s this dramatic power imbalance, but that doesn’t mean… that doesn’t tell you what’s going to happen, right? Why doesn’t the US just invade Guatemala or something? Why don’t they do that? Seems like they could easily do it.
Dwarkesh Patel 01:36:53
Because the value to the US of…
Ege Erdil 01:36:56
Not that high, right?
Dwarkesh Patel 01:36:58
Yeah. So I agree that might be true for AIs because most of the shit is in space. And you want to do the capital deepening on Mars and the surface of the sun instead of like New York City.
Ege Erdil 01:37:13
I think it’s deeper than that. So it’s deeper than that. There’s also the fact that if the AIs are going to be integrated into our economy…
So basically they start out as a small part of our economy or our workforce and over time they grow and over time they become the vast majority of the actual work power in the economy. But they are growing in this existing framework where we have norms and rules for better coordination and then undermining those things has a cost. So if getting the things that are making the humans wealthier than they used to be before and more comfortable, yeah, you would probably be better off if you could just take that from them. But the benefit to you, if you already are getting almost all of the income in the economy, will be fairly small.
Dwarkesh Patel 01:38:03
I feel like the Sentinel Islands thing, there’s one reference class that includes that. But historically, there’s a huge reference class that includes; East India Trading Company could have just kept trading with the Mughals, they just took over, right? They could have kept trading with the 50 different nation states in pre-colonial India. But yeah.
Ege Erdil 01:38:21
That’s right. I mean, that’s what they were initially doing. And then whatever. I’m not going to go into that subject.
Dwarkesh Patel 01:38:27
But that is the reference class…
Ege Erdil 01:38:30
I agree. I agree. So if the question is, if they have some totally different values and then they represent most of the economy, then would they take over? I still don’t know, because I’m not sure to what extent the class of all AI is a natural class. It’s sort of like, why don’t the young people in the economy coordinate?
Dwarkesh Patel 01:38:54
I agree that sometimes these kinds of class arguments are misused. For example, when Marxists are like, “why don’t this class rise up against the others?”
Daniel made the interesting argument that if you look at the history of the conquistadors, when Cortes was making his way through the new world, he had to actually go back and fight off a Spanish fleet that had been sent to arrest him and then go back. So you can have this fight within this conquering AIs and then that still nets out to the Native Americans getting disempowered.
But with AIs in particular, they’re just copies of each other. And in many other ways, they have lower transaction costs when they trade with each other or interact with each other. There’s other reasons to expect them to be more compatible coordinating with each other than coordinating with the human world.
Ege Erdil 01:39:48
Sure. If the question is just that, “is it possible for that to happen?”, which is a weaker claim, then yeah, it seems possible. But there are, I think, a lot of arguments pushing back against it. Probably actually the biggest one is the fact that AI preferences are just not… Just look at the AIs we have today. Can you imagine them doing that? I think people just don’t put a lot of weight on that, because they think once we have enough optimization pressure and once they become super intelligent, they’re just going to become misaligned. But I just don’t see the evidence for that.
Dwarkesh Patel 01:40:24
I agree there’s some evidence that they’re good boys.
Ege Erdil 01:40:28
No, there’s more than some evidence.
Dwarkesh Patel 01:40:30
No, but there’s also some evidence… There’s a new openAI paper where in chain of thought, reward hacking is such a strong basin that if you were like, “hey, let’s go solve this coding problem”, In the chain of thought, they’ll just be like, “okay, let’s hack this and then figure out how to hack it.”
Ege Erdil 01:40:48
So imagine that you gave students at a school a test and then the answer key was like on the back.
Dwarkesh Patel 01:40:52
Right, but the reference class of humans does include Cortes and the East Indian Trading Company.
Ege Erdil 01:40:57
Sure.
Tamay Besiroglu 01:40:58
So I think one issue here is that I think people are doing this very kind of partial equilibrium analysis or something where they’re thinking about these raw abilities of AI systems in a world where AI systems are dominant and human civilization has done very little in terms of integrating itself and the AI is integrating itself into the human world. Insofar as it’s poor at communicating and coordinating with AI, addressing those deficiencies and improving that. Insofar as that’s posing a risk, or creating inefficiencies, because it’s unable to benefit from coordinating and trading, then it should have this enormous incentive to address that.
Insofar as there is a lot of value to be gained from dominating and taking over humans, what you might get is a more negotiated settlement. If that’s indeed the case, then a war would just be inefficient. And so you would want to negotiate some settlement that results in some outcomes that are mutually beneficial.
Dwarkesh Patel 01:42:05
Compared to the counterfactual, not compared to… There was a mutually beneficial trade that was made between the Qing dynasty and the British in the opium wars, right? But it was maybe better than pre-industrial China going to war with the British empire, but it wasn’t better than never having interacted with the British empire in the first place.
Tamay Besiroglu 01:42:28
So I think one mistake that I feel people make is they have this very naive analysis of what creates conflict. And I think Matthew has written a bit about this, a colleague of ours, where they say there’s misalignment. And so that then creates conflict.
But that’s actually not what the literature on what causes conflict says creates conflict. It’s not just misalignment, it’s also other issues like having a bad understanding of the relative strengths of your armies versus theirs, or maybe having these very strong commitments that you think some grounds are sacred, and so you’re not willing to do any trade in order to give up some of that in order to gain something else. And so then you have to posit some additional things other than just the base value misalignment part.
Dwarkesh Patel 01:43:27
I think you’re making a good argument against, like, “humans take up the spears and the machetes and go to war against the AI data centers”, because maybe there’s not this asymmetric information that often leads to conflicts in history. But this argument does not address at all the risk of takeover, which can be the result of a peaceful end negotiation or human society being like, “look, we’re totally outmatched. And we’ll just take these meager concessions rather than go to war”.
Tamay Besiroglu 01:43:57
But insofar as it’s more peaceful, then I think it’s like much less of a thing to worry about. I think there could be this trend where we indeed have this gradual process where AI is much more important in the world economy and actually deciding and determining what happens in the world. But this could be beneficial for humans where we’re getting access to this vast, much, much larger economy and much more advanced technological stock.
Ege Erdil 01:44:30
Yeah. So I think it’s important to be clear about what is the thing that you’re actually worried about. Because I think some people just say that, “oh, humans are going to lose control of the future, we’re not going to be the ones that are making the important decisions. We, however, concede", that’s also kind of nebulous.
But is that something to worry about? If you just think biological humans should remain in charge of all important decisions forever, then I agree, the development of AI seems like a problem for that. But in fact, other things also seem like a problem for that, I just don’t expect to generically be true. Like in a million years from now, if even if you don’t develop AI, biological humans, the way we recognize them today, are still making all the important decisions
and they have something like the culture that we would recognize from ourselves today. I would be pretty surprised by that.
I think Robin Hanson has again talked about this, where he said a bunch of the things that people fear about AI are just things they fear about change and fast change. So the thing that’s different is that AI has a prospect of accelerating much of this change so that it happens in a narrower period.
Dwarkesh Patel 01:45:36
I think it’s not just the kind of change that would have happened from, let’s say, genetically modifying humans and blah, blah, blah, is instead happening in a compressed amount of time. I think the worry comes more from like, it’s not just that change compressed. It’s a very different vector of change .
Ege Erdil 01:45:53
Yeah, but what is the argument for that? I have never seen a good argument for this.
Tamay Besiroglu 01:45:58
You should expect a bunch of change if you accelerate just human change as well. You might expect different values to become much more dominant. You might expect people that don’t discount the future as much to be much more influential because they save more and they make good investments that gives them more control.
Ege Erdil 01:46:17
People who are higher risk tolerance.
Tamay Besiroglu 01:46:18
Higher risk tolerance. Because they are more willing to make bets that maximize expected value and so get much more influence. So just generically, accelerating human change would also result in a lot of things being lost that you might care about.
Dwarkesh Patel 01:46:34
I think the argument is that maybe the speed of the change determines what fraction of the existing population or stakeholders or whatever, have some causal influence on the future. And maybe the thing you care about is, look, there’s going to be change, but it’s not just going to be like one guy presses a button. That’s like the software singularity extreme. It’s more like over time norms change and so forth.

Ege Erdil 01:47:08
So if you’re looking at the software singularity picture, I agree that picture looks different. And again, I’m coming back to this because obviously Daniel, and maybe Scott to some extent, they probably have this view that the software-only singularity is more plausible. And then one person, we could end up in a situation where their idiosyncratic preferences or something end up being more influential.
I agree that makes the situation look different from if you just have this broader process of automation. But even in that world, I think a lot of people have this view about things like value lock-in, where they think this moment is a pivotal moment in history. And then someone is going to get this AI, which is very powerful because of the software-only singularity. And then they’re just going to lock in some values. And then those values are going to be stable for millions of years.
And I think that just looks very unlike anything that has happened in the past. So I’m kind of confused why people think it’s very plausible. I think people have the argument that they see the future, again, in my view, in sort of ‘far mode’. They think there’s going to be one AI. It’s going to have some kind of utility function. That utility function is going to be very stable over time, so it’s not going to change, there won’t be this messiness of a lack of coordination between different AIs, or over time values drifting for various reasons, maybe because they become less functional in an environment, maybe because of other reasons. And so they just don’t imagine that. They say, “well, utility functions, we can preserve them forever. We have the technology to do that. So it’s just going to happen”. And I’m like, “well, that seems like such a weak argument to me”.
Tamay Besiroglu 01:48:50
Often the idea is, because this is digital you can preserve the information better and copy it with higher fidelity and so on. But actually, even if you look just at information on the internet, you have this thing called link rot, which happens very quickly. And actually, information that’s digital isn’t preserved for very long at all.
Dwarkesh Patel 01:49:15
And the point that Matthew was making is that the fact that the information is digital has led to- not maybe led to, but at least been associated with- faster cultural change.
Tamay Besiroglu 01:49:25
Cultural change, exactly.
Ege Erdil 01:49:26
I mean, basically technological changes can create incentives for cultural change just as they make preserving…
Dwarkesh Patel 01:49:32
I think there’s two key arguments that I’ve heard. One is that we will soon reach something called technological maturity. And one of the key ways in which society has been changing recently is- maybe actually its culture would have changed even more. Actually, no, I think this argument that you’re making is wrong, because we do know that language actually changed a lot more. We can read everything that was written after the 1800s when literacy became more common. But just go back a couple hundred years after that and you’re reading old English and it’s hard to understand. And that is a result of literacy and the codification of language.
Ege Erdil 01:50:09
Well, that information was better preserved. What about other kinds of cultural practices?
Dwarkesh Patel 01:50:12
But I think the argument would be that change was a result of technological change in general, not the result of information being digitized. And maybe that culture would have actually changed more if information wasn’t as well preserved or technology had continued to proceed. And the argument is, in the future we’re going to reach some point at which you’ve done all the tech, ideas have just gotten way too hard to find and you need to make a CERN that’s the size of a galaxy to progress physics an inch forward.
And at that point, there’s this growth in technology, just churning over civilization goes away. And then you just have the digital thing, which does mean that a lock-in is more plausible.
Tamay Besiroglu 01:51:00
So the technological maturity thing, I agree that results in this slowdown and change and growth and so on and certain things might get more locked-in relative to what preceded it. But then what do we do today about that? Well, what could you do to have a positive impact by our lights?
Robin Hanson had this question of what could someone do in the 1500s to have a positive impact on the world today from their point of view, knowing all they knew back then? I think this question is even worse than that, because I think the amount of change that happens between today and technological maturity is just orders of magnitude greater than whatever change happened between the 1500s and today.
So it’s an even worse position than someone in the 1500s thinking about what they could do to have a positive impact in expectation, like predictably positive today. And so I think it’s just pretty hopeless. I don’t know if we could do anything or find any candidate set of actions that would make things better post lock-in.
Ege Erdil 01:52:05
I mean, that’s assuming lock-in is going to happen, which is not…
Dwarkesh Patel 01:52:08
In the 1700s, a bunch of British abolitionists were making the case against slavery. And I don’t think there’s any in-principle reason why we couldn’t have been a slave society to this day, or more of the world couldn’t have slavery. I think what happened is just the convincing of British people that slavery is wrong, the British Empire put all its might into abolishing slavery and making that a norm.
I think another example is Christianity and the fact that Jesus has these ideals, you could talk about these ideals. I think the world is a more Christian place.
Ege Erdil 01:52:45
It is a more Christian place, sure.
Dwarkesh Patel 01:52:57
And also is like more of the kind of place- I’m not saying Jesus Christ would endorse every single thing that happens in the world today. I’m just saying he endorses this timeline more than one in which he doesn’t exist and doesn’t preach at all.
Ege Erdil 01:53:00
I don’t know, actually. I’m not sure if that’s true. It seems like a hard question.
Dwarkesh Patel 01:53:03
But I think like a sum from the Christian perspective, favorable cultural development to the West.
Ege Erdil 01:53:07
I mean, you don’t know the counterfactual.
Dwarkesh Patel 01:53:09
I agree that is always true. I just think the world does have people who read the Bible and are like, “I’m inspired by these ideals to do certain things”. And it just seems like that’s more likely to lead to…
Ege Erdil 01:53:20
So that is what I would call a ‘legacy effect’ or something. You can say the same thing about languages, some cultures might just become more prominent and their languages might be spoken more, or some symbols might become more prominent. But then there are things like how do cities look, and how do cars look, and what do people spend most of their time doing in their day, and what do they spend their money on? And those questions seem much more determined by how your values change as circumstances change.
Dwarkesh Patel 01:53:49
That might be true, but I’m in the position with regards to the future where I expect a lot of things to be different and I’m okay with them being different. I care much more about the equivalent of slavery, which in this case is literally slavery.
Just to put a final point on it, the thing I really care about is there’s going to be trillions of digital beings. I want it to be the case that they’re not tortured and put into conditions in which they don’t want to work and whatever. I don’t want galaxies worth of suffering. That seems closer to British abolitionists being like, “let’s put our empire’s might against fighting slavery”.
Ege Erdil 01:54:25
I agree. But I would distinguish between the case of Christianity and the case of the end of slavery, because I think the end of slavery… I agree you can imagine a society, technologically it’s feasible to have slavery. But I think that’s not the relevant thing which brought it to an end.
The relevant thing is that the change in values associated with the Industrial Revolution made it so that slavery just became an inefficient thing to sustain in a bunch of ways. And a lot of countries at different times phased out different things you could call slavery. For example, Russia abolished serfdom in the 1860s. They were not under British pressure to do so. Britain couldn’t force Russia to do that, they just did that on their own. There were various ways in which people in Europe were tied to their land and they couldn’t move, they couldn’t go somewhere else. Those movement restrictions were lifted because they were inefficient.
There were ways in which the kind of labor that needed to be done in the colonies to grow sugar or to grow various crops, it was very hard labor. It was not the kind of thing that probably you could have paid people to do, because they just wouldn’t want to do it because the health hazards and so on were very great, which is why they needed people to force people to do them. And that kind of work over time became less prevalent in the economy.
So, again, that reduces the economic incentives to do it. I agree you could still do it.
Dwarkesh Patel 01:55:58
I would emphasize the way you’re painting the counterfactual is like, “oh, but then in that world, they would have just phased out the remnants of slavery”. But there’s a lot of historical examples where there’s not necessarily hard labor, only hard labor, like Roman slavery.
Ege Erdil 01:56:14
Yes. It was different.
Dwarkesh Patel 01:56:16
And I interviewed a historian about it recently, the episode hasn’t come out, but he wrote a book about the scope. I think it was like 20 percent of people under Roman control were slaves.
And this was not just agricultural slavery. His point was that the maturity of the Roman economy is what led to this level of slavery, because the reason slavery collapsed in Europe after the fall of the Roman Empire was because the economy just lost a lot of complexity.
Ege Erdil 01:56:50
Well, I’m not sure if I would say that slavery collapsed. I think this depends on what you mean by slavery. I mean in a lot of ways people in feudal Europe were…
Dwarkesh Patel 01:56:58
But his point is that serfdom was not the descendant institution from Roman slavery.
Ege Erdil 01:57:02
No, I agree. It was not descendant. But in fact, this point I’m trying to make is that, values that exist at a given time, like what the values we will have in 300 years, or from the perspective of someone a thousand years ago, what values people are going to have in a thousand years. Those questions are much more determined by the technological and economic and social environment that’s going to be there in a thousand years, which values are going to be functional, which sides, which values end up being more competitive and being more influential so that other people add up their values. And it depends much less on the individual actions taken by people a thousand years ago.
So I would say that the abolitionist thing, it’s not the cause of why slavery came to an end. Slavery comes to an end also because people just have natural preferences that I think are suppressed in various ways during the agricultural era where it’s more efficient to have settled societies in cities which are fairly authoritarian and don’t allow for that much freedom and that you’re in this Malthusian world where people have very low wages perhaps compared to what they enjoyed in the hunter-gatherer era. So it’s just a different economic period and I think people didn’t evolve to have the values that would be functional in that era.
So what happened is that there had to be a lot of cultural assimilation where people had to adopt different values and in the Industrial Revolution people become also very wealthy compared to what they used to be, and that I think leads to different aspects of people’s values being expressed. Like people just put a huge amount of value on equality. It’s always been the case. But I think when it is sufficiently functional for that to be suppressed they are capable of suppressing it.
Dwarkesh Patel 01:59:01
I mean if that’s the story then this makes value alignment all the more important, because then you’re like “oh if the AI’s become wealthy enough they actually will make a concerted effort to make sure the future looks more like the utility function you put into them” which I think you have been under-emphasizing.
Ege Erdil 01:59:18
No, I’m not under-emphasizing that. What I would say is there are certain things that are path-dependent in history, such that if someone had done something different, something had gone differently a thousand years ago, then today in some respects would look different. I think for example, which languages are spoken across which boundaries, or which religions people have, or fashion maybe to some extent, though not entirely.
Those things are more path-dependent, but then there are things that are not as path-dependent. So for example if some empire, like if the Mongols had been more successful and they somehow- I don’t know how realistic it is- but they became very authoritarian and had slavery everywhere, would that have actually led to slavery being a much more enduring institution a thousand years later? That seems not true to me.
The forces that led to the end of slavery seemed like they were not contingent forces, they seem like deeper forces than that and if you’re saying “well if we align the AI today to some bad set of values then that could affect the future in some ways which are more fragile” that seems plausible, but I’m not sure how much of the things you care about the future and how much the ways in which you expect the future to get worse you actually have a lot of leverage on at the present moment.
Dwarkesh Patel 02:00:40
I mean another example here might be factory farming where you could say “oh, it’s not like us having better values over time led to suffering going down, in fact your suffering might have gone up because the incentives that led to factory farming emerging are…”
Ege Erdil 02:00:56
And probably when factory farming comes to an end it will be because the incentives start going away, right?
Dwarkesh Patel 02:01:01
So suppose I care about making sure the digital equivalent of factory farming doesn’t happen. Maybe, all else being equal, it’s just more economically efficient to have suffering minds doing labor for you than non-suffering minds because of the intermediary benefits of suffering or something like that, right?
What would you say to somebody like me where I’m like “I really want that not to happen, I don’t want the lightcone filled with suffering workers” or whatever. Is it just like “we’ll give up because this is the way economic history is”?
Ege Erdil 02:01:40
No, I don’t think you should give up. It’s hard to anticipate the consequences of your actions in the very distant future. So I would just recommend that you should just discount the future. Not for a moral reason, not because the future is worthless or something, but because it’s just very hard to anticipate the effects of your actions. In the near-term I think there are things you can do that seem like they would be beneficial. For example, you could try to align your present AI systems to value the things that you’re talking about, like they should value happiness and they should dislike suffering or something.
You might want to support political solutions that would… Basically you might want to build up the capacity so that in the future if you notice something like this happening then we might have some ability to intervene. Maybe you would think about the prospect of “well eventually we’re gonna maybe colonize other stars and civilization might become very large and communication delays might be very long between different places”. And in that case competitive pressures between different local cultures might become much stronger because it’s harder to centrally coordinate.
And so in that you might expect competition to take over in a stronger way and if you think the result of that is going to be a lot of suffering, maybe you would try to stop that. Again I think at this point it’s very far from obvious that trying to limit competition is actually a good idea, I would probably think it’s a bad idea, but maybe in the future we will receive some information and we’ll be like “oh, we were wrong actually actually we should stop this” and then maybe you want to have the capacity so that you can make that decision.
But that’s a nebulous thing. How do you build that up? Well I don’t know. That’s the kind of thing I would be trying to do.
Tamay Besiroglu 02:03:28
Yeah I think the overall takeaway I take from the way that I think about it, and I guess we think about it, as be more humble in what you think you can achieve, and just focus on the nearer term, not because it’s more morally important than the longer term, but just because it’s much easier to have a predictably positive impact on that.
Dwarkesh Patel 02:03:49
One thing I’ve noticed over the last few weeks of thinking about these bigger future topics and interviewing Daniel and Scott and then you two, is how often I’ve changed my mind about everything from the smallest questions about when AI will arrive- it’s funny that that’s the small question in the grand scheme of things- to whether there will be an intelligence explosion, or whether it’ll be an R&D explosion, to whether there’ll be explosive growth, or how to think about that.
And if you’re in a position where you are incredibly epistemically uncertain about what’s going to happen, I think it’s important to, instead of becoming super certain about your next conclusion, just being like “well let me just take a step back, I’m not sure what’s going on here”. And I think a lot more people should be from that perspective unless you’ve had the same opinion about AI for many years, in which case I have other questions for you about why that’s the case. And I mean generally, how we as a society deal with topics on which we are this uncertain is just to have freedom, decentralization, both decentralized knowledge and decentralized decision making take the reins and not to do super high volatility centralized moves like “hey let’s nationalize so we can make sure that the software-only singularity is aligned” or not to make moves that are just incredibly contingent on one world view that are brittle under other considerations.
And that’s become a much more salient part of my world view. I think just classical liberalism is the way we deal with being this epistemically uncertain and I think we should be more uncertain than we’ve ever been in history, as opposed to many other people who seem to be more certain than they are about other sort of more mundane topics.
Tamay Besiroglu 02:05:44
Yeah I think it’s very hard to predict what happens because this acceleration basically means that you find it much harder to predict what the world might be in 10 years time. I think these questions are also just very difficult and we don’t have very strong empirical evidence and then there’s like a lot of this kind of disagreement that exists.
Ege Erdil 02:06:10
I would say that it’s much more important in a lot of cases and a lot of situations to maintain flexibility and ability to adapt to new circumstances, new information, than it is to get a specific plan that’s going to be correct and that’s very detailed and has a lot of specific policy recommendations and things that you should do.
That’s actually also the thing that I would recommend if I want to make the transition to AI in this period of explosive growth go better. I would just prefer it if we in general had higher quality institutions, but I am much less bullish on someone sitting down today and working out “okay what will this intelligence explosion or explosive growth be like? What should we do?”
I think plans that you work out today are not going to be that useful when the events are actually occurring, because you’re going to learn so much stuff that you’re going to update on so many questions that these plans are just going to become obsolete.
Tamay Besiroglu 02:07:12
One thing you could do is you could look at say, the history of war planning and how successful war planning has been for like actually anticipating what actually happens when the war actually happens.
Ege Erdil 02:07:22
So for one example- I think I might have mentioned this off the record at some point- but before the Second World War happened, obviously people saw that there were all these new technologies like tanks and airplanes and so on, which existed in World War I. but in a much more primitive setting. So they were wondering, what is going to be the impact of these technologies now that we have in them in much greater scale? And the British government had estimates of how many casualties there would be from aerial bombardment in the first few weeks of the Second World War. And they expected hundreds of thousands of casualties in two weeks, three weeks, after the war begins.
So the idea was that air bombing is basically this unstoppable force, all the major urban centers are going to get bombed, tons of people will die, so basically we can’t have a war because if there’s a war then it will be a disaster because we will have this aerial bombardment. But later it turned out that that was totally wrong. In fact, in all of Britain there were fewer casualties from air bombing in the entire six years of the Second World War than the British government expected in the first few weeks of the war. They had less casualties in six years than they expected in three weeks.
So why did they get it wrong? Well there are lots of boring practical reasons, like for example it turned out to be really infeasible, especially early on, to bomb cities in daytime because your aircraft would just get shot down, but then if you try to bomb at night time then your bombing was really imprecise and only a very small fraction of it actually hit. And then people also underestimated the extent to which people on the ground like firefighters and so on could just go around the city and that put out fires from bombs that were falling on structures. They overestimated the amount of economic damage that it would do. They underestimated how economically costly it would be; basically you’re sending these aircraft and then they’re getting shot down, while an aircraft is very expensive.
So in the end how it turned out is, when the allies started bombing Germany, for each dollar of capital they were destroying in Germany they were spending like four to five dollars on the aircraft and fuel and training of the pilots and so on that they were sending in missions and the casualty rate was very high, which later got covered up by the government because they didn’t want people to worry about, you know…
So that is a kind of situation where all the planning that you would have done in advance predicated on this assumption of air bombing going to be this “nuclear weapons-lite”, basically it’s extremely destructive there’s going to be some aspect to which…
Dwarkesh Patel 02:09:57
I mean it was though, right? 84,000 people died in one night of firebombing in Tokyo, Germany, large fractions of their…
Ege Erdil 02:10:07
But that was over the period of six years of war.
Dwarkesh Patel 02:10:11
Right, but there were single firebombing attacks. I mean it was a case that during the end of World War II when they were looking for the place to launch the atomic bombs, they just had to go through like a dozen cities because it just wouldn’t be worth nuking them because they’re already destroyed by the firebombing.
Ege Erdil 02:10:28
That’s right, but if you look at the level of destruction that was expected within the space of a few weeks, and then this level of destruction took many years, so there was like a two order of magnitude mismatch or something like that, which is pretty huge. So that affected the way people think about it.
Tamay Besiroglu 02:10:45
An important underlying theme of much of what we have discussed is how powerful just reasoning about things is to making progress about what specific plans you want to make to prepare and make this transition to advanced AI go well.
And our view is, well it’s actually quite hard and you need to make contact with the actual world in order to inform most of your beliefs about what actually happens and so it’s somewhat futile to do a lot of wargaming and figure out how AI might go, and what we can do today to make that go a lot better, because a lot of the policies you might come up with might just look fairly silly.
And in thinking about how AI actually has this impact, again people think “oh you know, AI reasoning about doing science and doing R&D just has this drastic impact on the overall economy or technology, and our view as well actually again making contact with the real world and getting a lot of data from experiments and from deployment and so on is just very important”.
So I think there is this underlying kind of latent variable which explains some of this disagreement, both on the policy prescriptions and about the extent to which we should be humble versus ambitious about what we ought to do today, as well as for thinking about the mechanism through which AI has this impact. And this underlying latent thing is, what is the power of reason? How much can we reason about what might happen? How much can reasoning in general figure things out about the world and about technology? And so that is a core underlying disagreement here.
Dwarkesh Patel 02:12:27
I do want to ask: You say in your announcement, we want to accelerate this broad automation of labor as fast as possible. As you know, many people think it’s a bad idea to accelerate this broad automation of labor and AGI and everything that’s involved there. Why do you think this is good?
Ege Erdil 02:12:49
So the argument for why it’s good is that we’re going to have this enormous increase in economic growth, which is going to mean enormous amounts of wealth, and incredible new products that you can’t even imagine, in health care or whatever. And like the quality of life of the typical person is probably going to go up a lot.
Early on, probably also their wages are going to go up, because the AI systems are going to be automating things that are complementary to their work. Or it’s going to be automating part of their work, and then you’ll be doing the rest and then you’ll be getting paid much more on that. And in the long term, eventually we do expect wages to fall just because of arbitrage with the AIs. But by that point, we think humans will own enormous amounts of capital and there will also be ways in which even the people who don’t own capital, we think are just going to be much better off than they are today.
I think it’s just hard to express in words the amount of wealth and increased variety of products that we would get in this world. It will probably be more than the difference between 1800 and today. So if you imagine that difference, it’s such a huge difference. And then we imagine two times, three times, whatever.
Dwarkesh Patel 02:13:58
The standard argument against this is why does the speed to get there matter so much? Especially if the trade-off against the speed is the probability that this transition is achieved successfully in a way that benefits humans?
Tamay Besiroglu 02:14:12
I mean, it’s unclear that this trades off against the probability of it being achieved successfully or something.
Dwarkesh Patel 02:14:17
There might be an alignment tax.
Tamay Besiroglu 02:14:20
I mean, maybe. You can also just do the calculation of how much a year’s worth of delay costs for current people. This is this enormous amount of utility that people are able to enjoy, and that gets brought forward by year or pushed back by year if you delay things by year. And how much is this worth? Well, you can look at simple models of how concave people’s utility functions are and do some calculations and maybe that’s worth on the order of tens of trillions of dollars per year in consumption.
That is roughly the amount consumers might be willing to defer in order to bring forward the date of automation one year.
Dwarkesh Patel 02:15:03
In absolute terms, it’s high. In relative terms, relative to if you did think it was going to nudge the probability one way or another of building systems that are aligned and so forth that it’s so small compared to all of the future.
Ege Erdil 02:15:18
I agree. So there are a couple of things here.
First of all, I think the way you think about this matters. So first of all, we don’t actually think that it’s clear whether speeding things up or slowing things down actually makes a doomy outcome more or less likely. I think that’s a question that doesn’t seem obvious to us. Partly because of our views on the software R&D side. We don’t really believe that if you just pause and then you do research for 20 years at a fixed level of compute scale, that you’re actually going to make that much progress on relevant questions on alignment or something.
Imagine you were trying to make progress on alignment in 2016 with the compute budgets of 2016. Well, you would have gotten nowhere, basically. You would have discovered none of the things that people have discovered today and that turned out to be useful. And I think if you pause today, then we will be in a very similar position in 10 years, right? We would have not made a bunch of discoveries. So the scaling is just really important to make progress in alignment, in our view. And then there’s a separate question of how longtermist should you be in various different senses?
So there’s a moral sense, of how much you should actually care about people who are alive today as opposed to people who are not yet born as just a moral question. And there was also a practical question of, as we discussed, how certain can you be about the impact your present actions are actually going to have in the future?
Dwarkesh Patel 02:16:43
OK, maybe you think it really doesn’t matter whether you slow things down right now or you speed things up right now. But is there some story about why speeding them up from the alignment perspective actually helped, it’s good to have that extra progress right now rather than later on?
Or is it just that, well, if it doesn’t make a difference either way, then it’s better to just get that extra year of people not dying and having cancer cures and so forth?
Ege Erdil 02:17:06
I think I would say the second. But it’s just important to understand the value of that. Even in purely economic terms, imagine that each year of delay might cause maybe 100 million people- maybe more, maybe 150, 200 million people- who are alive today to end up dying. So even in purely economic terms, the value of a statistical life is pretty enormous, especially in Western countries. Sometimes people use numbers as high as $10 million for a single life. So imagine you do $10 million times 100 million people. That’s a huge number, right?
That is so enormous that I think for you to think that speeding things up is a bad idea, you have to first have this long-termist view where you look at the long run future. You think your actions today have high enough leverage that you can predictably affect the direction of the long run future.
Dwarkesh Patel 02:18:10
Well, in this case, it’s kind of different because you’re not saying “I’m going to affect what some emperor a thousand years from now does” like somebody in the year zero would have to do to be a long-termist. In this case, you just think there’s this incredibly important inflection point that’s coming up and you just need to have influence over that crucial period of explosive growth of intelligence explosion or something. So I think it is a much more practicable prospect than…
Ege Erdil 02:18:36
So I agree in relative terms. In relative terms, I agree the present moment is a moment of higher leverage and you can expect to have more influence. I just think in absolute terms, the amount of influence you can have is still quite low. So it might be orders of magnitude greater than it would have been 2000 years ago and still be quite low.
Tamay Besiroglu 02:18:54
And again, I think there’s this difference in opinion about how broad and diffuse this transformation ends up being, versus how concentrated within specific labs where the very idiosyncratic decisions made by that lab will end up having a very large impact.
If you think those developments will be very concentrated, then you think the leverage is especially great. And so then you might be especially excited about having the ability to influence how that transition goes, but our view is very much that this transition happens very diffusely by way of many, many organizations and companies doing things. And for those actions to be determined a bunch by economic forces rather than idiosyncratic preferences on the part of labs or these decisions that have these founder effects that last for very long.

Dwarkesh Patel 02:19:48
Okay let’s go through some of the objections to explosive growth, which is that most people are actually more conservative not more aggressive about the forecasts you have. So obviously one of the people who has articulated their disagreements with your view is Tyler Cowen. He made an interesting point when we did the podcast together and he said “most of Sub-Saharan Africa still does not have reliable clean water. The intelligence required for that is not scarce. We cannot so readily do it. We are more in that position that we might like to think along other variables.”
Tamay Besiroglu 02:20:22
I mean we agree with this. I think intelligence isn’t the bottleneck that’s holding back technological progress or economic growth. It’s like many other things. And so this is very much consistent with our view that scaling up your overall economy, accumulating capital, accumulating human capital, having all these factors scale…
Ege Erdil 02:20:45
In fact this is even consistent with what I was saying earlier that I was pointing out this “oh, good management and good policies and those just contribute to TFP and they can be bottlenecks”.
Dwarkesh Patel 02:20:55
Like right now we could just plug-and-play our better management into Sub-Saharan Africa.
Ege Erdil 02:21:02
No we can’t.
Tamay Besiroglu 02:21:03
It’s hard. I don’t think we can.
Dwarkesh Patel 02:21:05
Okay so maybe I should have said, one could theoretically imagine plugging and playing…
Ege Erdil 02:21:10
I agree.
Tamay Besiroglu 02:21:12
I can imagine many things.
Dwarkesh Patel 02:21:14
But we cannot so readily do it because of… it’s hard to articulate why and it wouldn’t be so easy to do in just capital or labor. Why not think that the rest of the world will be in this position with regards to the advances that AI will make possible?
Tamay Besiroglu 02:21:32
I mean if the AI advances are like the kind of geniuses in a data center, then I agree that that might be bottlenecked by the rest of the economy not scaling up and being able to accumulate the relevant capital to make those changes feasible. So I kind of agree with this picture and I think this is an objection to the “geniuses in a data center” type view, and I buy basically this.
Ege Erdil 02:21:57
And also the fact that it’s also plausible you’re going to have the technology, but then some people are not going to want to deploy it, or some people are going to have norms and laws and cultural things that are going to make it so that AI is not able to be widely deployed in their economy- or not as widely deployed as they otherwise might be. And that is going to make those countries or societies just slower. That’s like some countries will be growing faster just like Britain and the Netherlands were sort of the leaders in the Industrial Revolution, they were the first countries to start experiencing rapid growth. And then other countries, even in Europe, had to come from behind.
Well again I just think we expect the same thing to be true for AI. And the reason that happened was exactly because of these kinds of reasons, where those countries that had a culture or governance systems or whatever which were just worse than bottlenecked the deployments and scaling of the new technologies and ideas. It seems very plausible.
Dwarkesh Patel 02:22:53
But you’re saying as long as there’s one jurisdiction?
Ege Erdil 02:22:55
Yeah.
Dwarkesh Patel 02:22:56
But then again you also previously emphasized the need to integrate with the rest of the global economy and the human economy. So doesn’t that contradict…?
Tamay Besiroglu 02:23:05
That doesn’t often require cultural homogeneity. We trade with countries, like the US trades with China, quite a lot actually. And there’s a bunch of disagreement…
Dwarkesh Patel 02:23:15
But what if the US is like “I don’t like that the UAE is doing explosive growth with AI, we’re just going to embargo them”.
Tamay Besiroglu 02:23:22
That seems plausible.
Dwarkesh Patel 02:23:24
And then would that not prevent explosive growth?
Tamay Besiroglu 02:23:26
I think that would be plausible at the point at which it’s revealing a lot about the capabilities and the power of AI. Yeah. And you should also think that that creates both an incentive to embargo, but also an incentive to adopt the very similar styles of governing that enable AI to be able to produce a lot of value.
Dwarkesh Patel 02:23:48
What do you make of this: I think people interpret explosive growth from an arms race perspective. And that’s often why I think in terms of public-private partnerships for the labs themselves. But this idea that you have the geniuses in the data center, you can have them come up with the mosquito drone swarms. And then those drone swarms will, like if China gets to the swarms earlier… Even within your perspective, is this a result of your whole economy being advanced enough that you can produce mosquito drone swarms?
You being six months ahead means that you could decisively win… does it? I don’t know. Maybe you being like a year ahead and explosive growth means you could decisively win a war against China or China could win a war against you. So would that lead to an arms race-like dynamic?
Ege Erdil 02:24:33
I mean I think it would to some extent, but I’m not sure if I would expect a year of lead to be enough to take a risk, because if you go to war with China… For example if you replace China today with China from 1990. Or if you replace Russia today with Russia from like 1970 or 1980. It’s possible that their ICBM and whatever technology is already enough to make a very strong deterrence.
So maybe even that technological lead is not sufficient so that you would feel comfortable going to war. So that seems possible.
Dwarkesh Patel 02:25:13
Yeah. And actually this relates to a point that Gwern was making which is that this is going to be a much more unstable period than the Industrial Revolution, even though the Industrial Revolution saw many countries gain rapid increases in their capabilities, because within this span, if you’ve got a century’s worth of progress compressed within a decade, one country gets to ballistic missiles first, then the other country gets to railroads first, and so forth.
But if you have this more integrated perspective about what it takes to get to ballistic missiles and to railroads, then you might think “no, basically this isn’t some orthogonal vector. You’re just churning on the tech tree further and further”.
Ege Erdil 02:26:01
I mean for what it’s worth I do think it’s possible if you have it just happen in a few countries which are relatively large and have enough land or something, those countries would be starting from a lower base compared to the rest of the world, so you would need to catch up to some extent. If they are just going to sort of grow internally and they’re not going to depend on the external supply chains. But that doesn’t seem like something that’s impossible to me. Some countries could do it, it would just be more difficult.
But in this setting if some countries have a significant policy advantage over the rest of the world and they start growing first and then they won’t necessarily have a way to get other countries to adopt their norms and culture. So in that case it might be more efficient for them to do the growth locally. So that’s why I was saying the growth differentials will probably be determined by regulatory jurisdiction boundaries more than anything else. I’m not saying- say the U.S. by itself if it had AI but it couldn’t get the rest of the world to adopt AI, I think that would still be sufficient for explosive growth.
Dwarkesh Patel 02:27:03
How worried should we be about the fact that China today, because it industrialized relatively recently, has more industrial capacity and know-how and all the other things of learning by doing and so forth? If we buy your model of how technology progresses, with or without AI, how are we just underestimating China because we have this perspective that what fraction of your GDP you’re spending on research is what matters, when in fact it’s the kind of thing where I’ve got all the factories in my backyard and I know how they work and I can go buy a component whenever I want?
Tamay Besiroglu 02:27:41
I don’t think people are necessarily underestimating China, it depends on who you’re looking at, but it seems like the discussion of China is just this very big discussion in these AI circles, right?
And so people are very much appreciating the power and the potential threat that China poses. But I think the key thing is not just the scale in terms of pure number of people or like number of firms or something, but the scale of the overall economy, which is just measured in how much is being produced in terms of dollars. There, the U.S. is ahead.
Dwarkesh Patel 02:28:14
But we’re not expecting all this explosive growth to come from financial services. We’re expecting it to start from a base of industrial technology and industrial capacity.
Ege Erdil 02:28:25
No, financial services can be important if you want to scale very big projects.
Tamay Besiroglu 02:28:29
Financial services are very important for raising funding and getting investments in data centers.
Dwarkesh Patel 02:28:35
If I understood you correctly it just seems like, man, you know how to build the robot factories and so forth. That know-how which, in your view, is so crucial to technology growth and general economic growth, is lacking. And you might have more advanced financial services but it seems like the more you take your view seriously, the more it seems like that having the Shenzhen locally matters a lot.
Tamay Besiroglu 02:29:00
I mean relative to what starting point? I think people already appreciate that China is very important. And then I agree that there are some domains where China is leading, but then there are very many domains in which the U.S. is leading, or the U.S. and its allies, where countries that are producing relevant inputs for AI that the U.S. has access to, but China doesn’t.
So I think the U.S. is just ahead on many dimensions and there’s some that China is ahead or at least very close. So I don’t think this should cause you to update very strongly in favor of China being a much bigger deal, at least depending on where you start.
Ege Erdil 02:29:40
I think people already think China is a big deal like this is the big underlying thing here. Like if we were just very dismissive of China, then maybe this would be a reason to update.

Dwarkesh Patel 02:29:48
I get your argument that thinking about the economy-wide acceleration is more important than focusing on the IQ of the smartest AI. But at the same time, do you believe in the idea of superhuman intelligence? Is that a coherent concept in the way that you don’t necessarily stop at human level Go play and you just go way beyond it in ELO score?
Will we get to systems that are like that with respect to the broader range of human abilities? And maybe that doesn’t mean they become God, because there’s other ASIs in the world. But you know what I mean, will there be systems with such superhuman capabilities?
Tamay Besiroglu 02:30:27
Yeah I mean I do expect that. I think there’s a question of how useful is this concept for thinking about this transition to a world with much more advanced AI. And I don’t find this a particularly meaningful or helpful concept.
I think people introduce some of these notions that on the surface seem useful, but then actually when you delve into them it’s very vague and kind of unclear what you’re supposed to make of this. And you have this notion of AGI which is distinguished from narrow AI in the sense that it’s much more general and maybe can do everything that a human can do on average. AI systems have these very jagged profiles of capability. So you have to somehow take some notion of average capabilities and what exactly does that mean, it just feels really unclear.
And then you have this notion of ASI, which is AGI in the sense that it’s very general but then it’s also better than humans on every task. And is this a meaningful concept? I guess it’s coherent. I think this is not a super useful concept, because I prefer just thinking about what actually happens in the world. And you could have a drastic acceleration without having an AI system that can do everything better than humans can do. I guess you could have no acceleration when you have an ASI that is better than humans at everything, but it’s just very expensive or very slow or something. So I don’t find that particularly meaningful or useful. I just prefer thinking about the overall effects on the world and what AI systems are capable of producing those types of effects.
Dwarkesh Patel 02:32:06
Yeah I mean one intuition pump here is: compare John von Neumann versus a human plucked from the standard distribution. If you added a million John von Neumanns to the world what would the impact on growth be as compared to just adding a million people from normal distribution?
Ege Erdil 02:32:25
Well I agree it would be much greater.
Dwarkesh Patel 02:32:27
Right. But then because of Moravec paradox-type arguments that you made earlier that evolution has not necessarily optimized us for that long along the kind of spectrum on which John von Neumann is distinguished from the average human. And given the fact that already within this deviation you have this much greater economic impact. Why not focus on optimizing on this thing that evolution has not optimized that hard on, further?
Ege Erdil 02:32:51
I don’t think we shouldn’t focus on that. But what I would say is, for example if you’re thinking about the capabilities of Go-playing AIs, then the concept of a superhuman Go AI, yeah, you can say that is a meaningful concept. But if you’re developing the AI, it’s not a very useful concept. If you just look at the scaling curve, it just goes up and there is some human level somewhere. But the human level is not privileged in any sense.
So the question is, is it a useful thing to be thinking about? And the answer is probably not. Depends on what you care about. So I’m not saying we shouldn’t focus on trying to make the system smarter than humans are, I think that’s a good thing to focus on.
Dwarkesh Patel 02:33:31
Yeah I guess I try to understand whether we will stand in relation to the AIs of 2100 that humans stand in relation to other primates. Is that the right mental model we should have, or is it going to be a much greater familiarity with their cognitive horizons?
Tamay Besiroglu 02:33:49
I think AI systems will be very diverse, and so it’s not super meaningful to ask something about this very diverse range of systems and where we stand in relation to them.
Dwarkesh Patel 02:33:59
I mean, will we be able to cognitively access the kinds of considerations they can take on board? Humans are diverse, but no chimp is going to be able to understand this argument in the way that another human might be able to, right? So if I’m trying to think about my place, or a human’s place, in the world of the future, is a relevant concept of; is it just that the economy has grown a lot and there’s much more labor, or are there beings who are in this crucial way super intelligent?
Tamay Besiroglu 02:34:28
I mean there will be many things that we just will fail to understand, and to some extent there are many things today that people don’t understand about how the world works and how certain things are made. And then how important is it for us to have access or in principle be able to access those considerations?
And I think it’s not clear to me that that’s particularly important that any individual human should be able to access all the relevant considerations that produce some outcome. That just seems like overkill. Why do you need that to happen? I think it would be nice in some sense. But I think if you want to have a very sophisticated world where you have very advanced technology, those things will just not be accessible to you.
So you have this trade-off between accessibility and maybe how advanced the world is. And from my point of view I’d much rather live in a world which has very advanced technology, has a lot of products that I’m able to enjoy, and a lot of inventions that I can improve my life with, if that means that I just don’t understand them. I think this is a very simple trade that I’m very willing to make.

Dwarkesh Patel 02:35:45
Okay so let’s get back to objections to explosive growth. We discussed a couple already. Here’s another which is more a question than an objection: Where is all this extra output going? Who is consuming it? If the economy is 100X bigger in a matter of a decade or something, to what end?
Ege Erdil 02:36:05
So first of all I think even if you view that along what you might call the intensive margin in the sense that you just have more of the products you have today, I think there will be a lot of appetite for that. Maybe not quite 100X, that might start hitting some diminishing returns.
Tamay Besiroglu 02:36:23
Current GDP per capita on average in the world is 10K a year or something, right? And there are people who enjoy millions of dollars. And so there’s a gap between what people enjoy, and don’t seem to be super diminished in terms of marginal utility, and so there’s a lot of room on just purely the intensive margin of just consuming the things we consume today but more. And then there is this maybe much more important dimension along which we will expand which is…
Ege Erdil 02:36:52
Product variety.
Tamay Besiroglu 02:36:53
Yeah, extensive margin of what is the scope of things that you’re consuming. And if you look at something like the Industrial Revolution, that seemed to have been the main dimension along which we expanded to consume more. In any kind of sector that you care about, transportation, medicine, entertainment, and food, there’s just this massive expansion in terms of variety of things that we’re able to consume that is enabled by new technology or new trade routes or new methods of producing things. So that I think is really the key thing that we will see come along with this kind of expansion in consumption.
Dwarkesh Patel 02:37:35
Another point that Tyler makes is that there will be some mixture of Baumol cost disease, where you’re bottlenecked by the slowest growing thing. The fastest productivity things basically diminish their own…
Ege Erdil 02:37:56
Share in output.
Dwarkesh Patel 02:37:57
That’s right, yeah.
Tamay Besiroglu 02:37:59
I mean we totally agree with that. I would say that that’s just a kind of qualitative consideration. It isn’t itself sufficient to make a prediction about what growth rates are permitted given these effects versus not, it’s just a qualitative consideration and then you might need to make additional assumptions to be able to make a quantitative prediction. So I think it’s a little bit…
Ege Erdil 02:38:24
So the convincing version of this argument would be if you did the same thing that we were doing earlier with the software-only singularity argument, where we were pointing to essentially the same rejection where there are multiple things that can bottleneck progress. So I would be much more convinced if someone pointed to an explicit thing, like here, health care is this very important thing. And why should we expect AI to make that better? That doesn’t seem like that would get better because of AI. So maybe health care just becomes a big part of the economy and then that bottleneck. So if there was some specific sector…
Dwarkesh Patel 02:38:58
Maybe the argument is that if there is even one…
Ege Erdil 02:39:00
No, if there’s one though, if that’s a small part of the economy then you could just still get a lot of growth. You just automate everything else and that is going to produce a lot of growth.
Tamay Besiroglu 02:39:09
So it has to quantitatively work out. And so you actually have to be quantitatively specific about what this objection is supposed to be.
Ege Erdil 02:39:15
Right. So first of all you have to be specific about what these tasks are. What is the current share in economic output?
The second thing is you have to be specific about how bad do you think the complementarities are? So in numerical terms economists use the concept of elasticity of substitution to quantify this. So that gives you a numerical estimate of, if you just have much more output on some dimensions but not that much on other dimensions, how much does that increase economic output overall?
And then there’s a third question. You can also imagine you automate a bunch of the economy. Well, a lot of humans were working on those jobs. So now, well they don’t need to do that anymore because those got automated. So they could work on the jobs that haven’t yet been automated. So as I gave the example earlier, you might imagine a world in which remote work tasks get automated first, and then sensory-motor skills lag behind. So you might have a world in which software engineers become physical workers instead.
Of course, in that world the wages of physical workers will be much higher than their wages are today. So that reallocation also produces a lot of extra growth, even if bottlenecks are maximally powerful, even if you just look at all the tasks in the economy and literally take the worst one for productivity growth, you would still get a lot of increase in output because of this reallocation.
Tamay Besiroglu 02:40:35
So I think one point that I think is useful to make; our experience talking to economists about this is that they will bring up these more qualitative considerations, whereas the arguments that we make, make specific quantitative predictions about growth rates. So for example you might ask “how fast will the economy double?” And then we can think about, an H100 does about… there are some estimates of how much computation the human brain does per second and it’s about one E15 flop or so, it’s a bit unclear, but then it turns out that an H100 roughly does on that order of computation. And so you can ask the question of “how long does it take for an H100 to pay itself back?”
Ege Erdil 02:41:21
If you run the software of the human brain.
Tamay Besiroglu 02:41:22
If you run the software of the human brain you can then deploy that in the economy and earn say human wages on the order of 50 to 100 K a year or whatever in the US. And so then it pays itself back because it costs on the order of 30 K per H100. And so you get a doubling time of maybe on the order of a year.
And so this is like a very quantitatively specific prediction about… And then there’s the response, “well you have Baumol effects” well, what does this mean? Does it double? Does this predict it doubles every two years or every five years? You need just more assumptions in order to make this a coherent objection.
And so I think a thing that’s a little bit confusing is just that there are these qualitative objections that I agree with, like bottlenecks are indeed important, which is part of the reason I’m more skeptical of this ‘software singularity’ story. But I think this is not sufficient for blocking explosive growth.
Dwarkesh Patel 02:42:23
The other objection that I’ve heard often- and it might have a similar response from you- is this idea that a lot of the economy is comprised of O-ring-type activities. And this refers to, I think, the Challenger space shuttle explosion. There is just one component- I forgot what the exact problem with the O-ring was- but because of that being faulty the whole thing collapsed.
Tamay Besiroglu 02:42:48
I mean I think it’s quite funny actually because the O-ring model is taking the product of many, many inputs, and then the overall output is the product of very many things. But actually this is pretty optimistic from the point of view of having fewer bottlenecks.
Ege Erdil 02:43:08
I think we pointed this out before, which again, talking about software only singularity, I said if it’s the product of computer experiments with research…
Dwarkesh Patel 02:43:14
But if one of those products …
Ege Erdil 02:43:15
Is zero.
Dwarkesh Patel 02:43:16
Because of human…
Tamay Besiroglu 02:43:17
But you have constant marginal product there, right?
Ege Erdil 02:43:19
Yeah, but if one of those products doesn’t scale that doesn’t limit- like yeah, it means you’re less efficient at scaling than you otherwise would be, but you can still get a lot of…
Tamay Besiroglu 02:43:30
You can just have unbounded scaling in the O-ring world. So actually I disagree with Tyler, that he’s not conservative enough, that he should take his bottlenecks view more seriously than he actually is. And yet I disagree with him about the conclusion. And I think that we’re going to get explosive growth once we have AI that can flexibly substitute.
Dwarkesh Patel 02:43:50
I’m not sure I understand, like, there will be entirely new organizations that AIs come up with. We’ve written a blog post about one such with the AI firms. And you might be a productive worker or a productive contributor in this existing organization as it exists today. In the AI world many humans might just be zero or even minus…
Ege Erdil 02:44:11
I agree.
Dwarkesh Patel 02:44:13
Why won’t that… put that in the multiplication.
Tamay Besiroglu 02:44:18
But why would humans be in the loop there?
Ege Erdil 02:44:21
You’re both saying that humans would be negatively contributing to output. But then you’re also saying that we should put them into the…
Dwarkesh Patel 02:44:31
Okay, fair fair fair. The main objection often is regulation. And I think we’ve addressed it implicitly in different points, but might as well just explicitly address why won’t regulation stop this?
Ege Erdil 02:44:43
Yeah. So for what it’s worth, we do have a paper where we go over all the arguments for and against explosive growth. And regulation, I think, is the one that seems strongest as ‘against’.
The reason it seems strong is because even though we have made arguments before about international competition and variation of policies among jurisdictions and these strong incentives to adopt this technology both for economic and national security reasons.
So I think those are pretty compelling when taken together but even still, the world does have a surprising ability to coordinate on just not pursuing certain technologies.
Dwarkesh Patel 02:45:18
Right. Human cloning…
Ege Erdil 02:45:20
That’s right. So I think it’s hard to be extremely confident that this is not going to happen. I think it’s less likely that we’re going to do this for AI than it is for human cloning, because I think human cloning touches on some other taboos and so on.
Tamay Besiroglu 02:45:38
And also is less valuable.
Ege Erdil 02:45:39
Also less valuable. And probably less important also for national security in an immediate sense. But at the same time, as I said, it’s just hard to rule this out.
So if someone said “well I think there’s a 10 percent or 15 percent, whatever, 20 percent chance that there will be some kind of global coordination of regulation and that’s going to just be very effective. Maybe it will be enforced through sanctions on countries that defect or you know.
And then maybe it doesn’t prevent AI from being deployed, but maybe just slows things down enough that you never quite get explosive growth”. I don’t think that’s an unreasonable view. It’s like 10 percent chance it could be.
Dwarkesh Patel 02:46:17
I don’t know if there’s any… I don’t know. Do you encounter any other…
Ege Erdil 02:46:24
Any other objections?
Dwarkesh Patel 02:46:25
What should I be hassling you about?
Ege Erdil 02:46:27
I mean some things that we’ve heard from economists… People sometimes respond to our argument about explosive growth, which is an argument about growth levels. So we’re saying “we’re going to see 30 percent growth per year, instead of 3 percent”. They respond to that with an objection about levels. So they say “well how much more efficient, how much more valuable can you make hairdressing, or taking flights, or whatever, or going to a restaurant?”. And that is just fundamentally the wrong kind of objection.
We’re talking about the rate of change, and you’re objecting to it by making an argument about the absolute level of productivity. And as I said before, it is not an argument that economists themselves would endorse if it was made about a slower rate of growth continuing for a longer time. So it seems more like special pleading…
Dwarkesh Patel 02:47:20
I mean why not just the deployment thing, where the same argument you made about AI, where you do learn a lot just by deploying to the world and seeing what people find useful, ChatGPT was an example of this. Why won’t a similar thing happen with AI products and services where if one of the components is you put it out to the marketplace and people play with it and you find out what they need, and it clings to the existing supply chain and so forth. Doesn’t that take time?
Tamay Besiroglu 02:47:49
I mean it takes time but it is often quite fast. In fact, ChatGPT grew extremely fast.
Dwarkesh Patel 02:47:55
Right, but that was just purely digital service.
Ege Erdil 02:47:57
One reason to be optimistic is if you think the AIs will literally be drop-in remote workers, or drop-in workers in some cases if you have robotics, then companies are already experienced at onboarding humans, onboarding humans doesn’t take like a very long time. Maybe it takes six months even in a particularly difficult job for a new worker to start being productive. Well, that’s not that long.
So I don’t think that would rule out companies being able to onboard AI workers, assuming that they don’t need to make a ton of new complementary innovations and discoveries to take advantage. I think one way in which current AI systems are being inhibited and the reason we’re seeing the growth maybe be slower than you might otherwise expect, is because companies in the economy are not used to working with this new technology, they have to rearrange the way they work in order to take advantage of it.
But if AI systems were literally able to substitute for human workers then, well, the complementary innovations might not be as necessary.

Dwarkesh Patel 02:49:00
Actually this is a good excuse to go to the final topic, which is AI firms. So this blog post we wrote together about what it would be like to have a firm that is fully automated, and the crucial point we were making was that people tend to overemphasize and think of AI from the perspective of how smart individual copies will be.
And if you actually want to understand the ways in which they are superhuman, you want to focus on their collective advantages which, because of biology, we are precluded from, which are the fact that they can be copied with all their tacit knowledge. You can copy a Jeff Dean or Ilya Sutskever or whatever the relevant person is, in a different domain. You can even copy Elon Musk and he can be the guy who’s every single engineer in the SpaceX rig. And if that’s not an efficient way to…
Tamay Besiroglu 02:49:49
The AI equivalent of them.
Dwarkesh Patel 02:49:50
And if it’s not best to have Elon Musk or anything, you just copy the relevant team or whatever. And we have this problem with human firms, where there can be very effective teams or groups, but over time their culture dilutes, or the people leave, or die, or get old. And this is one of the many problems that can be solved with these digital firms.
Firms right now have two of the three relevant criteria for evolution; they have selection, and they have variation, but they don’t have high fidelity replication. And you could imagine a much more fast-paced and intense sequence of evolution for firms once you have this final piece click in.
And that relates to the onboarding thing, where right now they just aren’t smart enough to be onboarded as full workers, but once they are, I just imagine the kinds of things I try to hire for, it would just be such an unlock. The salaries are totally secondary. The fact that I can… “This is the skill I need” or the set of skills I need. And I can have a thousand workers in parallel if there’s something that has a high elasticity of demand. I think it’s probably, along with the transformative AI, the most underrated tangible thing that you need to understand about what the future AI society will look like.
Ege Erdil 02:51:22
I think there’s a first point about this very macroeconomic picture, where you just expect a ton of scaling of all the relevant inputs. I think that is the first order thing. But then you might have more micro-questions about, “okay, how does this world actually look like? How is it different from a world in which we just have a lot more people and a lot more capital and a lot more…?” Because it should be different. And then I think these considerations become important.
I think another important thing is just that AIs can be aligned. You get to control the preferences of your AI systems in a way that you don’t really get to control the preference of your workers. Your workers, you can just select, you don’t really have any other option. But for your AIs, you can fine tune them. You can build AI systems which have the kind of preferences that you want. And you can imagine that’s dramatically changing basic problems that determine the structure of human firms.
For example, the principal agent problem might go away. This is a problem where you as a worker have incentives that are either different from those of your manager, or those of the entire firm, or those of the shareholders of the firm.
Dwarkesh Patel 02:52:29
I actually think the incentives are a smaller piece of the puzzle. It’s more about bandwidth and information sharing where, with a large organization it’s very hard to have a single coherent vision, and the most successful firms we see today are where, for an unusual amount of time, a founder is able to keep their vision instilled in the organization; SpaceX or Tesla are examples of this. People talk about Nvidia this way.
But just imagine a future version where there’s this hyper inference scale mega-Jensen, who you’re spending $100 billion a year on inference, and copies of him are constantly writing every single press release and reviewing every pull request, and answering every customer service request, and so forth, and monitoring the whole organization, making sure it’s proceeding along a coherent vision and getting merged back into the hyper-Jensen, mega-Jensen, whatever.
Ege Erdil 02:53:30
Yeah, I agree that’s a bigger deal. At the same time, I would point out that part of the reason why it’s important to have a coherent vision and culture and so on in human companies might be that incentive problems exist otherwise. I wouldn’t rule that out, but I agree that, aside from the overall macroeconomic thing, I think the fact that they can be replicated is probably the biggest deal.
That also enables additional sources of economies of scale where if you have twice the number of GPUs, you can run not only twice the number of copies of your old model, but then you can train a model that’s even better. So you double your training compute and your inference compute, and that means you don’t get just twice the number of workers you would have had otherwise, you get more than that, because they are also smarter, because you spend more training compute. So that is an additional source of economies of scale.
And then there’s this benefit that, for humans, every human has to learn things from scratch, basically. They are born and then they have a certain amount of lifetime learning that they have to do. So in human learning, there’s a ton of duplication, while for an AI system, it could just learn once. It could just have one huge training run with tons of data. And then that run could be deployed everywhere. So that’s another massive advantage that the AIs have over humans.

Dwarkesh Patel 02:54:43
Maybe we’ll close up with this one debate we’ve often had offline, which is: will central planning work with these economies of scale?
Ege Erdil 02:54:52
So I would say that, I mean, again, the question of, “will it work?”
Dwarkesh Patel 02:54:56
Will it be optimal?
Ege Erdil 02:54:58
Right. So my guess is probably not optimal. But I don’t think anyone has thought this question through in a lot of detail.
Tamay Besiroglu 02:55:10
So it is worth thinking about why one might expect central planning to be slightly better in this world. So one consideration is just communication bandwidth being potentially much, much greater than it is today. In the current world, the information gathering and the information processing are co-located; humans observe and also process what they observe. In an AI world, you can disaggregate that.
So you can have the sensors and not do much processing, but just collect and then process centrally. And that processing centrally might make sense for a bunch of reasons, and you might get economies of scale from having more GPUs that produce better models, and also be able to think more deeply about what it’s seeing.
Dwarkesh Patel 02:56:06
It’s worth noting that certain things already work like this, for example, Tesla FSD. It will benefit from the data collected at the periphery from millions of miles of driving. And then the improvements which are made as a result of this.
Tamay Besiroglu 02:56:19
Centrally directed, it’s coming from HQ being like, “we’re going to push an update”.
And so you do get some of this more centralized…
Dwarkesh Patel 02:56:27
And it can be a much more intelligent form than just whatever gradient averaging that they- I mean, I’m sure it’s more sophisticated than that at Tesla- but it can be a much more deliberate, intelligent update.
Tamay Besiroglu 02:56:36
So that’s one reason to expect. And the other reason, I guess, is current leaders or CEOs don’t have bigger brains than the workers do. Maybe a little bit…
Dwarkesh Patel 02:56:50
I don’t know if you want to open that…
Tamay Besiroglu 02:56:52
But not by orders of magnitude. And so you could have orders of magnitude more scaling of the size of the models that are doing the planning than the people or the agents or workers doing the actions.
Ege Erdil 02:57:04
And I think a third reason is the incentive thing, where part of the reason you have a market is that it gives people the right kind of incentives. But you might not need that as much if you’re using AI. So I think there’s an argument that if you just list the traditional arguments people have made against “why does central bank not work?”, then you might expect them to become weaker.
Now, I think there’s a danger when you’re doing that kind of analysis to fall into the same kind of partial equilibrium analysis where you’re only considering some factors and then you’re not considering other things. For example…
Tamay Besiroglu 02:57:43
Things get more complex, you just have a much bigger economy and so on the one hand, your ability to collect information and process it improves, but also the need for doing that also increases as things become more complex.
Dwarkesh Patel 02:57:59
And one way to illustrate that is: imagine if Apple, the organization today, with all its compute and whatever, was tasked with managing the economy of Uruk. I think it actually could centrally plan the economy. The economy of Uruk might work even better as a result. But Apple as it exists today cannot manage the world economy as it exists today.
Ege Erdil 02:58:18
That’s right. Yeah.

Dwarkesh Patel 02:58:20
All right, actually this will be the final question: One of the things that makes AI so fascinating is that there is no domain of human knowledge that is irrelevant to studying it, because what we’re really trying to…
Tamay Besiroglu 02:58:33
I don’t know about that.
Dwarkesh Patel 02:58:36
There’s no serious domain of human knowledge…
Tamay Besiroglu 02:58:40
That’s better.
Dwarkesh Patel 02:58:42
…that is not relevant to studying it, because you’re just fundamentally trying to figure out what a future society will look like. And so obviously computer science is relevant, but also economics- as we’ve been discussing- history, and how to understand history, and many other things we’ve been discussing.
Especially if you have longer timelines and there is enough time for somebody to pursue a meaningful career here, what would you recommend to somebody? Because both of you are quite young. I mean, you especially Ege, but both of you. You would think this is the kind of thing which requires crystallized intelligence or whatever, especially given what we said earlier about… Look, as we get more knowledge, we’re going to have to factor what we’re learning into building a better model of what’s going to happen to the world. And if somebody is interested in this kind of career that you both have, what advice do you have for them?
Ege Erdil 02:59:27
Yeah, that’s a hard question. I mean, I’m not sure. I think there is an extent to which it’s difficult to deliberately pursue the implicit strategy that we would have pursued. It probably works better if it’s spontaneous and more driven by curiosity and interest than: you make a deliberate choice, “okay, I’m just going to learn about a bunch of things so that I can contribute to the discourse on AI”. I would think that strategy is probably less effective. At least I haven’t seen anyone who deliberately used that strategy and then was successful, it seems like.
Dwarkesh Patel 03:00:05
Yeah, I guess not that I’ve contributed to discourse directly, but maybe facilitated other people contributing. I guess it wasn’t a deliberate strategy on my end, but it was a deliberate strategy to do the podcast, which inadvertently gave me the opportunity to learn about multiple fields.
Tamay Besiroglu 03:00:20
Yeah, so given that you’re already interested and curious and reading a bunch of things, and studying a bunch of things, and thinking about these topics, on the margin there are a bunch of things you can do to make you more productive at making some contributions to this.
And I think just speaking to people and writing your thoughts down and finding especially useful people to chat with and collaborate with, I think that’s very useful. So just seek out people that have similar views and you’re able to have very high bandwidth conversations with and make progress on these topics. And I think that’s just pretty useful.
Dwarkesh Patel 03:01:00
But how exactly? Like should they DM you? Like how do they get in?
Ege Erdil 03:01:05
Yeah, sure.
Tamay Besiroglu 03:01:06
And, I don’t know, set up Signal chats with your friends or whatever.
Dwarkesh Patel 03:01:10
Actually, it’s crazy how much alpha I’ve gotten out of that.
Ege Erdil 03:01:14
But yeah, I think one advice I would give to people in general, even if they are not thinking about AI specifically, but I think it’s also helpful for that, is people should be much more aggressive about reaching out. People have an impression that if you reach out to someone who looks really important, they’re not going to respond to you. But if what you send to them is interesting and high quality, then it’s very, very likely that they will respond.
There’s like a lot more edge there that you can get, which is just being more aggressive and less ashamed of looking dumb. That’s the main advice I would give. Because if you want to be productive, then again, there are these complementarities and so you need to be part of some community or some organization.
Dwarkesh Patel 03:02:02
And it goes back to the thing about reasoning alone not being that helpful.
Ege Erdil 03:02:05
Yeah, yeah, yeah.
Dwarkesh Patel 03:02:06
It’s just like other people have thought a long time and have randomly stumbled upon useful ideas that you can take advantage of.
Ege Erdil 03:02:12
That’s right. So you should just try to place yourself in a situation where you can become part of something larger. Which isn’t working on the front, that’s just a more effective way of contributing. And to do that, you have to, well, let people know.
Dwarkesh Patel 03:02:25
That’s right. That’s right. And I think just coming to the Bay Area is especially- for interest in AI in particular.
Ege Erdil 03:02:30
Yeah, going to the Bay Area is nice. Just post, like just writing things and like posting them where people can see them. Just aggressively reaching out to people with interesting comments.
Tamay Besiroglu 03:02:39
Provided your thoughts are interesting and so on.
Dwarkesh Patel 03:02:42
I mean, they probably aren’t. In many cases, I think it’s like, my thoughts still might not be interesting, but people will tolerate my cold emails and will still collaborate with me and so forth.
The other thing I’ve noticed- tell me if this is actually the wrong pattern. With people like you or with Carl Shulman or something, is that, as compared to a general person who’s intellectually curious or reading widely, you tend to focus much more on key pieces of literature than say, “I’m going to go read the classics or just generally read”. It’s like, “ I’m going to just put like a ton more credence in something like the Roamer paper”. And a normal person who’s intellectually curious would not be reading key pieces of literature.
Ege Erdil 03:03:31
Yeah. I think you have to be very mindful of the fact that you have a very limited amount of time, you’re not an AI model. So you have to aggressively prioritize what you’re going to spend your time reading.
Tamay Besiroglu 03:03:44
Even AI models don’t prioritize that heavily. They read Reddit mostly or a large part of their corpuses…
Dwarkesh Patel 03:03:48
Key pieces of empirical literature, at least. At least among you guys. I mean, it might not be the most productive thing in general, but…
Tamay Besiroglu 03:03:54
I think that’s useful. I also think it’s useful to read Twitter. I think we were having this conversation about people often say that they’re spending too much time reading Twitter and they wish they spent more time reading arXiv. But actually, the amount of information per unit time you get reading Twitter is often just much higher, and it’s just much more productive for them to read Twitter.
I think there are key pieces of literature that are important, and I think it’s useful to figure out what people who have spent a lot of time thinking about this find important in their worldview, so in AI, this might be key papers, like the Andy Jones paper about scaling loss for inference is a big thing.
And in economics, this Romer paper or the paper on explaining long run population from Kremer or from David Roodman and so on. I think if people who you think think very well about this suggest a certain paper and they highly recommend it, then I think you should take that seriously and actually read those papers.
Dwarkesh Patel 03:05:09
And for me, it’s been especially helpful to, instead of just skimming a bunch of things, if there’s a key piece of literature in order to, for example, understand the transformer, there’s always the Karpathy lectures, but one research that was really useful is the Anthropic’s original transformer circuit paper. And just spending a day on that paper instead of skimming it and making a bunch of spaced repetition cards and so forth, was much more useful than just generally reading widely about AI.
Ege Erdil 03:05:42
I think it’s just much more important here if you want to prioritize things correctly to be, again, to be part of a community or to be getting inputs from a community or get from people who have thought a lot and have a lot of experience about what is important and what is not.
Dwarkesh Patel 03:05:56
Yeah.
Ege Erdil 03:05:57
This is true even in academic fields. So if you want to do math research, but you’re not part of a graduate program, you’re not at a university where there are tons of people who do math research all day for many years, then you’re not even going to know what are the open problems that I should be working on? What is reasonable to attack? What is not reasonable to attack? What papers in this field are important, contain important techniques? You’re just going to have no idea. So it’s very important to be plugged into that feed of information somehow.
Dwarkesh Patel 03:06:26
But how did you know all this shit before being plugged in? Because you weren’t talking to anybody in Ankara.
Ege Erdil 03:06:30
You don’t need to talk. The internet is a pretty useful thing in this respect. And you don’t need to necessarily talk to people, you can get a lot of benefit from reading. But you just need to identify, who are the people who seem constantly most interesting? And maybe you find one person. And then often that person will know some other people who are interesting. And then you can start tracing the social network.
One example I can give, which I think is actually accurate, is maybe you know about Daniel Ellsberg. So you look for a podcast he appears on. You notice that he’s appeared on 80,000 Hours podcast, which he has. And then you notice there are some other guests on the 80,000 Hours podcast. So maybe there’s Bryan Caplan, who has also appeared on the podcast. And then maybe Robin Hanson has also appeared on the podcast. And then maybe there are some people those other people know. And then just tracing that kind of social network and figuring out who to listen to like that. I think that can be…
Tamay Besiroglu 03:07:26
And I think you’re doing a very big service to making that possible. I think your selection is often very good.
Dwarkesh Patel 03:07:33
I’m actually curious to hear offline what I got wrong. Well, actually, I think I know the answer to that.
Tamay Besiroglu 03:07:38
And I think that makes it a bunch easier to track who are the people doing the most interesting thinking on various topics.
Dwarkesh Patel 03:07:47
That’s right. Cool. I think that’s a good place to end, with you praising me. Again, I highly recommend people follow Epoch.
There’s a great weekly newsletter, Gradient Updates, which- I mean, people plug newsletters, but this is, I can’t believe this is a thing that comes out on a weekly basis.
And you now have a new podcast, which I will not plug as a competitor, but you can check it out.
Tamay Besiroglu 03:08:18
Thanks for lending your studio.
Ege Erdil 03:08:20
Yeah, that’s very generous.
Dwarkesh Patel 03:08:24
Anyways, cool. Thanks, guys.

Scott and Daniel break down every month from now until the 2027 intelligence explosion.

Scott Alexander is author of the highly influential blogs Slate Star Codex and Astral Codex Ten. Daniel Kokotajlo resigned from OpenAI in 2024, rejecting a non-disparagement clause and risking millions in equity to speak out about AI safety.

I came in skeptical, but I learned a tremendous amount by bouncing my objections off of them. I highly recommend checking out their new scenario planning document, AI 2027.

Watch on Youtube; listen on Apple Podcasts or Spotify.

• WorkOS helps today’s top AI companies get enterprise-ready. OpenAI, Cursor, Perplexity, Anthropic and hundreds more use WorkOS to quickly integrate features required by enterprise buyers. To learn more about how you can make the leap to enterprise, visit workos.com

• Jane Street likes to know what's going on inside the neural nets they use. They just released a black-box challenge for Dwarkesh listeners, and I had a blast trying it out. See if you have the skills to crack it at janestreet.com/dwarkesh

• Scale’s Data Foundry gives major AI labs access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you’re an AI researcher or engineer, learn about how Scale’s Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkesh

To sponsor a future episode, visit dwarkesh.com/advertise.

(00:00:00) - AI 2027

(00:06:56) - Forecasting 2025 and 2026

(00:14:41) - Why LLMs aren't making discoveries

(00:24:33) - Debating intelligence explosion

(00:49:45) - Can superintelligence actually transform science?

(01:16:54) - Cultural evolution vs superintelligence

(01:24:05) - Mid-2027 branch point

(01:32:30) - Race with China

(01:44:47) - Nationalization vs private anarchy

(02:03:22) - Misalignment

(02:14:52) - UBI, AI advisors, & human future

(02:23:00) - Factory farming for digital minds

(02:26:52) - Daniel leaving OpenAI

(02:35:15) - Scott's blogging advice

Dwarkesh Patel

Today I have the great pleasure of chatting with Scott Alexander and Daniel Kokotajlo. Scott is of course the author of the blog Slate Star Codex, Astral Codex 10 now. It’s actually been, as you know, a big bucket list item of mine to get you on the podcast. So this is all the first podcast we’ve ever done, right?

Scott Alexander

Yes.

Dwarkesh Patel

And then Daniel is the director of the AI Futures Project. And you have both just launched today something called AI 2027. So what is this?

Scott Alexander

Yeah, AI 2027 is our scenario trying to forecast the next few years of AI progress. We’re trying to do two things here. First of all we just want to have a concrete scenario at all. So you have all these people, Sam Altman, Dario Amodei, Elon Musk saying, “going to have AGI in three years, superintelligence in five years”. And people just think that’s crazy because right now we have chatbots that are able to do a Google search, not much more than that in a lot of ways. And so people ask, “how is it going to be AGI in three years?” What we wanted to do is provide a story, provide the transitional fossils. So start right now, go up to 2027 when there’s AGI, 2028, when there’s potentially super intelligence, show on a month-by-month level what happened. Kind of in fiction writing terms, make it feel earned.

So that’s the easy part. The hard part is we also want to be right. So we’re trying to forecast how things are going to go, what speed they’re going to go at. We know that in general, the median outcome for a forecast like this is being totally humiliated when everything goes completely differently. And if you read our scenario, you’re definitely not going to expect us to be the exception to that trend.

The thing that gives me optimism is Daniel back in 2021, wrote the prequel to this scenario called What 2026 Looks Like. It’s his forecast for the next five years of AI progress. And he got it almost exactly right. You should stop this podcast right now. You should go and read this document. It’s amazing. Kind of looks like you asked ChatGPT to summarize the past five years of AI progress, and you got something with a couple of hallucinations, but basically well intentioned and correct. So when Daniel said he was doing this sequel, I was very excited, really wanted to see where it was going. It goes to some pretty crazy places and I’m excited to talk about it more today.

Daniel Kokotajlo

I think you’re hyping up a little bit too much. Yes, I do recommend people go read the old thing I did, which was a blog post. I think it got a bunch of stuff right, a bunch of stuff wrong, but overall held up pretty well and inspired me to try again and do a better version of it.

Scott Alexander

I think, read the document and decide which of us is right.

Daniel Kokotajlo

Another related thing too is that the original thing was not supposed to end in 2026, it was supposed to go all the way through the exciting stuff, right? Because everyone’s talking about, what about AGI, what about superintelligence, what would that even look like? So I was trying to step-by-step work my way from where we were at the time until things happen and then see what they look like, but I basically chickened out when I got to 2027 because things were starting to happen and the automation loop was starting to take off and it was just so confusing and there was so much uncertainty, so I basically just deleted the last chapter and published what I had up until that point. And that was the blog post.

Dwarkesh Patel

Okay, and then, Scott, how did you get involved in this project?

Scott Alexander

So I was asked to help with the writing, and I was already somewhat familiar with the people on the project, and many of them were kind of my heroes. So, Daniel, I knew both because I’d written a blog post about his opinions before I knew about his, “What 2026 looks like,” which was amazing. And also he had pretty recently made the national news for having, when he quit OpenAI, they told him he had to sign a non-disparagement agreement or they would claw back his stock options. And he refused, which they weren’t prepared for. It started a major news story, a scandal that ended up with OpenAI agreeing that they were no longer going to subject employees to that restriction.

So people talk a lot about how it’s hard to trust anyone in AI because they all have so much money invested in the hype and getting their stock options better. And Daniel had attempted to sacrifice millions of dollars in order to say what he believed, which to me was this incredibly strong sign of honesty and competence. And I was like, how can I say no to this person? Everyone else on the team, also extremely impressive. Eli Liflund, who’s a member of Samotsvety, the world’s top forecasting team. He has won, like, the top forecasting competition, plausibly described as just the best forecaster in the world, at least by these really technical measures that people use in the superforecasting community. Thomas Larsen, Jonas Vollmer, both really amazing people who have done great work in AI before.

I was really excited to get to work with this superstar team. I have always wanted to get more involved in the actual attempt to make AI go well. Right now, I just write about it. I think writing about it is important, but I don’t know. You always regret that you’re not the person who’s the technical alignment genius who’s able to solve everything. And getting to work with people like these and potentially make a difference just seemed like a great opportunity.

What I didn’t realize was that I also learned a huge amount. I try to read most of what’s going on in the world of AI, but it’s this very low bandwidth thing and getting to talk to somebody who’s thought about it as much as anyone in the world was just amazing. Makes me really understand these things about how AI is going to learn quickly. You need all of this deep engagement with the underlying territory and I feel like I got that.

Dwarkesh Patel

I’ve probably changed my mind towards, against, towards, against, intelligence explosion three, four times in the conversations I’ve had in the lead-up in talking to you and then trying to come up with a rebuttal or something.

Scott Alexander

It wasn’t even just changing my mind, getting to read the scenario for the first time. It obviously wasn’t written up at this point. It was a giant, giant spreadsheet. I’ve been thinking about this for a decade, decade and a half now. And it just made it so much more concrete to have a specific story. Like, oh, yeah, that’s why we’re so worried about the arms race with China. Obviously we would get an arms race with China in that situation. And aside from just the people getting to read the scenario really sold me. This is something that needs to get out there more.

Dwarkesh Patel

Yeah. Okay. Now let’s talk about this new forecast. Because you do a month by month analysis of what’s going to happen from here. So what is it that you expect in mid-2025 and the end of 2025 In this forecast?

Scott Alexander

So, [the] beginning of the forecast mostly focuses on agents. We think they’re going to start with agency training, expand the time horizons, get coding going well. Our theory is that they are, to some degree consciously, to some degree accidentally, working towards this intelligence explosion, where the AIs themselves can start taking over some of the AI research, move faster.

So 2025, slightly better coding, 2026, slightly better agents, slightly better coding. And then we focus on, and we name the scenario after 2027 because that is when this starts to pay off. The intelligence explosion gets into full swing; the agents become good enough to help with- at the beginning not really do, but help with- some of the AI research.

So we introduced this idea called the R&D progress multiplier: how many months of progress without the AIs do you get in one month of progress with all of these new AIs helping with the intelligence explosion. So 2027, we start with- I can’t remember if it literally starts with, or by March or something- a five times multiplier for algorithmic progress.

Daniel Kokotajlo

So we have the stats tracked on the site of the story. Part of why we did it as a website is so that you can have these cool gadgets and widgets. And so as you read the story, the stats on the side automatically update. And so one of those stats is the progress multiplier. Another answer to the same question you asked is basically; 2025, nothing super interesting happens, more or less similar trends to what we’re seeing.

Dwarkesh Patel

Computer use is totally solved? Partially solved? How good is computer use by the end of 2025?

Daniel Kokotajlo

My guess is that they won’t be making basic mouse click errors by the end of 2025, like they sometimes currently do. If you watch Claude Plays Pokemon- which you totally should- it seems like sometimes it’s just failing to parse what’s on the screen and it thinks that its own player character is an NPC and gets confused. My guess is that that sort of thing will mostly be gone by the end of this year, but that they still won’t be able to autonomously operate for long periods on their own.

Dwarkesh Patel

But by 2025, when you say it won’t be able to act coherently for long periods of time in computer use, if I want to organize a happy hour in my office, I don’t know, that’s like what a 30 minute task? What fraction of that is, it’s got to invite the right people, it’s got to book the right doordash or something. What fraction of that is it able to do?

Daniel Kokotajlo

My guess is that by the end of this year there’ll be something that can kind of do that, but unreliably. And that if you actually tried to use that to run your life, it would make some hilarious mistakes that would appear on Twitter and go viral, but that the MVP of it will probably exist by this year. Like there’ll be some Twitter thread about someone being like, “I plugged in this agent to like run my party and it worked!”

Scott Alexander

Our scenario focuses on coding in particular because we think coding is what starts the intelligence explosion. So we are less interested in questions of like, “how do you mop up the last few things that are uniquely human” compared to “when can you start coding in a way that helps the human AI researchers speed up their AI research, and then, if you’ve helped them speed up the AI research enough, is that enough to, with some ridiculous speed multiplier- 10 times, 100 times- mop up all of these other things?”

Dwarkesh Patel

One observation I have is, you could have told a story in 2021, once ChatGPT comes out… I think I had friends who were credible AI thinkers who were like, “look, you’ve got the coding agent now, it’s been cracked. Now the GPT4 will go around and it’ll do all this engineering and we do this RL on top. We can totally scale up the system 100x” and every single layer of this has been much harder than the strongest optimist expected. It seems like there have been significant difficulties in increasing the pre-training size, at least from rumors about field training runs or underwhelming training runs at labs.

It seems like building up these RL- total outside view, I know nothing about the actual engineering involved here- but just from an outside view it seems like building up the O1 RL clearly took at least two years after GPT4 was released. And these things are also, their economic impact and the kinds of things you would immediately expect based on benchmarks for them to be especially capable at isn’t overwhelming, like the call center workers haven’t been fired yet. So why not just say look, at higher scale it will probably get even more difficult.

Scott Alexander

Wait a second, I’m a little confused to hear you say that, because when I have seen people predicting AI milestones like Katja Grace’s expert surveys, they have almost always been too pessimistic from a point of view of how fast AI will advance. So I think the 2022 survey, they actually said that things that had already happened would take like 10 years to happen, but then the survey- it might have been 2023, it was like six months before GPT3, GPT4, came out. And there were things that GPT3 or 4 or whichever one of them it was, did, that it did in six months that they were still predicting like five or ten years from. I’m sure Daniel is going to have a more detailed answer, but I absolutely reject the premise that everybody has always been too optimistic.

Daniel Kokotajlo

Yeah, I think in general, most people following the field have underestimated the pace of AI progress and underestimated the pace of AI diffusion into the world. For example, Robin Hanson famously made a bet about less than a billion dollars of revenue I think by 2025 from AI.

Dwarkesh Patel

I agree Robin Hanson in particular has been too pessimistic.

Daniel Kokotajlo

But he’s a smart guy. So I think that the aggregate opinion has been underestimating the pace of both technical progress and deployment. I agree that there have been plenty of people who’ve been more bullish than me and have been already proven wrong, but they’re not me.

Scott Alexander

Wait a second. We don’t have to guess about aggregate opinion, we can look at Metaculus. Metaculus, I think their timeline was like 2050 back in 2020. It gradually went down to like 2040 two or three years ago. Now it’s at 2030, so it’s barely ahead of us. Again, that may turn out to be wrong, but it does look like the Metaculans overall have, have been too pessimistic, thinking too long term rather than too optimistic. And I think that’s like the closest thing we have to a neutral aggregator where we’re not cherry picking things.

Dwarkesh Patel

Yeah. I had this interesting experience yesterday. We were having lunch with this senior AI researcher, probably makes on the order of millions a month or something, and we were asking him, “how much are the AIs helping you?” And he said, “in domains which I understand well, and it’s closer to autocomplete but more intense, there it’s maybe saving me four to eight hours a week.”

But then he says, “in domains which I’m less familiar with, if I need to go wrangle up some hardware library or make some modification to the kernel or whatever, where I know less, that saves me on the order of 24 hours a week.” Now, with current models. What I found really surprising is that the help is bigger where it’s less like autocomplete and more like a novel contribution. It’s like a more significant productivity improvement there.

Daniel Kokotajlo

Yeah, that is interesting. I imagine what’s going on there is that a lot of the process when you’re unfamiliar with a domain is like Googling around and learning more about the domain. And language models are excellent because they’ve already read the whole Internet and know all the details.

Dwarkesh Patel

Isn’t this a good opportunity to discuss a certain question I asked Dario that you responded to?

Scott Alexander

What are you thinking of?

Dwarkesh Patel

Well, I asked this question where, as you say, they know all this stuff. I don’t know if you saw this. I asked this question where I said, look, these models know all this stuff. And if a human knew every single thing a human has ever written down on the Internet, they’d be able to make all these interesting connections between different ideas and maybe even find medical cures or scientific discoveries as a result.

There was some guy who noticed that magnesium deficiency causes something in the brain that is similar to what happens when you get a migraine. And so he just said: give you magnesium supplements that cured a lot of migraines. So why aren’t able to leverage this enormous asymmetric advantage they have to make a single new discovery like this?

Scott Alexander

And then the example I gave was that humans also can’t do this. So for me, the most salient example is the etymology of words. You have all of these words in English that are very similar, like ‘happy’ versus ‘hapless’, ‘happen’, ‘perhaps’. And we never think about them unless you read an etymology dictionary and they’re like, oh, obviously these all come from some old root that has to mean ‘luck’ or ‘occurrence’ or something like that.

So it’s kind of about figuring out versus checking. If I tell you those, you’re like, “this seems plausible”. And of course, in etymology, there are also a lot of false friends where they seem plausible but aren’t connected. But you really do have to have somebody shove it in your face before you start thinking about it and make all of those connections.

Dwarkesh Patel

I will actually disagree with this. We know that humans can do like, we have examples of humans doing this. I agree that we don’t have logical omniscience because there is a combinatorial explosion, but we are able to leverage our intelligence to… one of my favorite examples of this is David Anthony, the guy who wrote the Horse, the Wheel and Language.

He made this super impressive discovery before we had the genetic evidence for it, like a decade before, where he said, look, if I look at all these languages in India and Europe, they all share the same etymology. I mean literally the same etymology for words like ‘wheel’ and ‘cart’ and ‘horse’. And these are technologies that have only been around for the last 6,000 years, which must mean that there was some group that these groups are all, at least linguistically, descended from. And now we have genetic evidence for the Yamnaya, which we believe is this group. You have a blog where you do this. This is your job, Scott! So why shouldn’t we hold the fact that language models can’t do this more against them?

Scott Alexander

Yeah. So to me, it doesn’t seem like he is just sitting there being logically omniscient and getting the answer. It seems like he’s a genius, he’s thought about this for years, probably at some point, he heard a couple of Indian words and a couple of European words at the same time and they kind of connected and the light bulb came on. So this isn’t about having all the information in your memory so much as the normal process of discovery, which is kind of mysterious, but seems to come from having good heuristics and throwing them at things until you kind of get a lucky strike.

My guess is if we had really good AI agents and we applied them to this task, it would look something like a scaffold where it’s like, think of every combination of words that you know of, compare them. If they sound very similar, write it on this scratch pad here. If a lot of words of the same type show up on the scratch pad, that’s pretty strange, do some kind of thinking around it. And I just don’t think we’ve even tried that.

And I think right now if we tried it, we would run into the combinatorial explosion. We would need better heuristics. Humans have such good heuristics that probably most of the things that show up even in our conscious mind, rather than happening on the level of some kind of unconscious processing, are at least the kind of things that could be true. I think you could think of this as like a chess engine. You have some unbelievable number of possible next moves, you have some heuristics for picking out which of those are going to be the right ones. And then gradually you kind of have the chess engine think about it, go through it, come up with a better or worse move, then at some point you potentially become better than humans. I think if you were to force the AI to do this in a reasonable way, or you were to train the AI such that it itself could come up with the plan of going through this in some kind of heuristic-laden way, you could potentially equal humans.

Daniel Kokotajlo

I’ll add some more things to that. So I think there’s a long and sordid history of people looking at some limitation of the current LLMs and then making grand claims about how the whole paradigm is doomed because they’ll never overcome this limitation. And then a year or two later the new LLMs overcome that limitation.

And I would say that with respect to this thing of “why haven’t they made these interesting scientific discoveries by combining the knowledge they already have and noticing interesting connections?” I would say first of all, have we seriously tried to build scaffolding to make them do this? And I think the answer is mostly no.

Dwarkesh Patel

I think Google DeepMind tried this.

Daniel Kokotajlo

Maybe. Second thing, have you tried making the model bigger? They’ve made it a bit bigger over the last couple years and it hasn’t worked so far. Maybe if they make it even bigger still, it’ll notice more of these connections. And then third thing, and here’s I think the special one: Have you tried training the model to do the thing? The pre-training process doesn’t strongly incentivize this type of connection making.

In general I think it’s a helpful heuristic that I use to ask the question of: remind oneself, what was the AI trained to do? What was its training environment like? And if you’re wondering why hasn’t the AI done this, ask yourself, did the training environment train it to do this? And often the answer is no. And often I think that’s a good explanation for why the AI is not good at it is that it wasn’t trained to do it.

Dwarkesh Patel

I mean it seems like such an economically valuable…

Daniel Kokotajlo

But how would you set up the training environment? Wouldn’t it be really gnarly to try to set up an RL environment to train to make new scientific discoveries?

Dwarkesh Patel

Maybe that’s why you should have longer timelines. It’s a gnarly engineering problem.

Daniel Kokotajlo

Well in our scenario they don’t just leap from where we are now to solving this problem. They don’t. Instead they just iteratively improve the coding agents until they’ve basically got coding solved. But even still, their coding agents are not able to do some of this stuff. That’s what early 2020, like the first half of 2027 in our story is basically, they’ve got these awesome automated coders, but they still lack research taste and they still lack maybe organizational skills and stuff.

And so they need to overcome those remaining bottlenecks and gaps in order to completely automate the AI research cycle. But they’re able to overcome those gaps faster than they normally would because the coding agents are doing all the grunt work really fast for them.

Scott Alexander

Yeah, I think it might be useful to think of our timelines as being like 2070, 2100. It’s just that the last 50 to 70 years of that all happened during the year 2027 to 2028, because we are going through this intelligence explosion like I think if I asked you, could we solve this problem by the year 2100? You would say, oh, yeah, by 2100? Absolutely. And we’re just saying that the year 2100 might happen earlier than you expect because we have this research progress multiplier.

Dwarkesh Patel

And then let me just address that in a second. But just one final thought on this thread. To the extent that there’s like a modus ponens, modus tollens thing here, where one thing you could say is like, look: AIs- not just LLMs, but AIs- will have this fundamental asymmetric advantage where they know all this shit. And why aren’t they able to use their general intelligence to use this asymmetric advantage to some enormous capability overhang.

Now, you could infer that same statement by saying, okay, well, once they do have that general intelligence, they will be able to use their asymmetric advantage to make all these enormous gains that humans are in principle less capable of, right? So basically, if you do subscribe to this view that AIs could do all these things if only they had general intelligence, you got to be like, well, once we actually do get the AGI, it’s actually going to be a totally transformative because they will have all of human knowledge memorized and they can use that to make all these connections.

Daniel Kokotajlo

I’m glad you mentioned that our current scenario does not really take that into account very much. So that’s an example in which our scenario is possibly underestimating the rate of progress.

Dwarkesh Patel

You’re so conservative, Daniel.

Scott Alexander

This has been my experience working with the team, as I point out, five different things. “Are you sure you’re taking this into account? Are you sure you’re taking this into account?” And first of all, 99% of the time he says, “yes, we have a supplement on it”. But even when he doesn’t say that, he’s like, “yeah, that’s one reason it could go slower than that. Here are 10 reasons it could go faster”.

Daniel Kokotajlo

It’s trying to be sort of like our median guess. So there are a bunch of ways in which we could be underestimating, and there are a bunch of ways in which you could be overestimating. And we’re going to hopefully continue to think more about this afterwards and continue to iteratively refine our models and come up with better guesses and so forth.

Dwarkesh Patel

So if I look back at AI progress in the past, if we were back in, say, 2017. Suppose we had these superhuman coders in 2017; the amount of progress we’ve made since then, so where we are currently in 2025, by when could we have had that instead?

Daniel Kokotajlo

Great question. We’d still have to stumble through all the discoveries that we’ve made since 2017. We still have to figure out that language models are a thing, we still have to figure out that you can fine tune them with RL.

So all those things would still have to happen. How much faster would they happen? Maybe 5x faster, because a lot of the small scale experiments that these people do in order to test out ideas really quickly before they do the big training runs would happen much faster because they’re just lickety-split being spit out. I’m not very confident in that 5x number, it could be lower, it could be higher, but that was roughly what we were guessing.

Our 5x, by the way, is for the algorithmic progress part, not for the overall thing. So in this hypothetical, according to me, basically things would be going 2.5x faster, where the algorithms would be advancing at 5x speed, but the compute is still stuck at the usual speed.

Dwarkesh Patel

That seems plausible to me. You have a 5x at some point, and then dot dot dot, you have 1000x AI progress within the matter of a year. Maybe that’s the part I’m like, wait, how did that happen exactly? So what’s the story there?

Daniel Kokotajlo

The way that we did our takeoff forecast, which we’ll get to in a second, was basically by breaking down how we think the intelligence explosion would go into a series of milestones. First you automate the coding, then you automate the whole research process, but in a very similar way to how humans do it with teams of agents that are about human level, then you get to superhuman level and so forth.

So we broke it down into these milestones, you know, the superhuman coder, superhuman AI researcher, and then super intelligent AI researcher. And the way we did our forecast was, for each of these milestones, we were like, what is it going to take to make an AI that achieves that milestone? And then once you do achieve that milestone, how much is your overall speedup? And then what’s it going to take to achieve the next milestone? Combine that with the overall speed up and that gets you your clock time distance until that happens and then, okay, now you’re at that milestone. What’s your overall speed up? Assuming that you have that milestone also, what’s the next one? How long does it take to get to the next one? So we sort of work through it bit by bit, and at each stage we’re just making our best guesses.

So quantitatively we were thinking something like 5x speedup to algorithmic progress from the superhuman coder, and then something like a 25x speedup to algorithmic progress from the superhuman AI researcher. Because at that point you’ve got the whole stack automated, which I think is substantially more useful than just automating the coding. And then I forget what we say for a super intelligent AI researcher, but off the top of my head it’s probably something in the hundreds or maybe like 1000x overall speed up.

Dwarkesh Patel

So maybe the big picture thing I have with the intelligence explosion is… we can go through the specific arguments about how much will the automated coder be able to do, and how much will the superhuman AI coder be able to do. But on priors, it’s just such a wild thing to expect.

And so, before we get into all the specific arguments, maybe you can just address this idea that, why not just start off with 0.01% chance this thing might happen? Then you need extremely, extremely strong evidence that it will before making that your modal view.

Scott Alexander

I think that it’s a question of what is your default option or what are you comparing it to. I think that naively people think like, well, every particular thing is potentially wrong. So let’s just have a default path where nothing ever happens. And I think that that has been the most consistently wrong prediction of all. Like, I think in order to have nothing ever happen, you actually need a lot to happen. Like you need suddenly AI progress that has been going at this constant rate for so long stops. Why does it stop?

Well, we don’t know. Whatever claim you’re making about that is something where you would expect there to be a lot of out of model error is where you would expect. Think like somebody must be making a pretty definite claim that you want to challenge. So I don’t think there’s a neutral position where you can just say, well, given that out of model error is really high and we don’t know anything, let’s just choose that. I think we are trying to take- I know this sounds crazy because if you read our document, all sorts of bizarre things happen. It’s probably the weirdest couple of years that have ever been. But we’re trying to take almost in some sense a conservative position where the trends don’t change, nobody does an insane thing, nothing that we have no evidence to think will happen happens. And the way that the AI intelligence explosion dynamics work are just so weird that in order to have nothing happen, you need to have a lot of crazy things happen.

Daniel Kokotajlo

One of my favorite meme images is this graph showing world GDP over time. You’ve probably seen it, it spikes up and then there’s a little thought bubble at the top of the spike in 2010 or something. And the thought bubble says, “my life is pretty normal, I have a good grasp of what’s weird versus standard and people thinking about different futures with digital minds and space travel are just engaging in silly speculation”.

The point of the graph is, actually there’s been amazing transformative changes in the course of history that would have seemed totally insane to people multiple times. We’ve gone through multiple such waves of those things.

Scott Alexander

Everything we’ve talked about has happened before. Algorithmic progress already doubles every year or so. So it’s not insane to think that algorithmic progress can contribute to these compute things. In terms of general speedup, we’re already at like a thousand times research speedup, multiplier compared to the Paleolithic or something. So from the point of view of anyone in most of history, we are going at a blindingly insane pace. And all that we’re saying here is that it’s not going to stop.

The same trend that has caused us to have a thousand times speed up multiplier relative to past eras and not even the Paleolithic, like what happened in the century between, I don’t know, 600 and 700 A.D. I'm sure there are things, I’m sure historians could point them out. Then you look at the century between 1900 and 2000 and it’s just completely qualitatively different.

Of course there are models of whether that stagnated recently or what’s going on here. We can talk about those, we can talk about why we expect the intelligence explosion to be an antidote to that kind of stagnation. But nothing we’re saying is that different from what has already happened.

Dwarkesh Patel

I mean, you are saying that these previous transitions have been smoother than the one you were anticipating.

Scott Alexander

We’re not sure about that, actually. So one of these models is just a hyperbola. Everything is along the same curve. Another model is that there are these things like the literal Cambrian explosion. If you want to take this very far back, go full Ray Kurzweil. The literal Cambrian explosion, the agricultural revolution, the industrial revolution, has phase changes.

When I look at the economic modeling of this, my impression is the economists think that we don’t have good enough data to be sure whether this is all one smooth process or whether it’s a series of phase changes. When it is one smooth process, the smooth process is often a hyperbola that shoots to infinity in weird ways. We don’t think it’s going to shoot to infinity. We think it’s going to hit bottlenecks again.

Dwarkesh Patel

You guys are the conservative crowd, you know?

Scott Alexander

We think it’s going to hit bottlenecks the same as all these previous processes. The last time this hit a bottleneck, if you take the hyperbola view, is in, like 1960, when humans stopped reproducing at the same rate they were reproducing before. We hit a population bottleneck, the usual population, two ideas, flywheel stopped working, and then we stagnated for a while.

If you can create a country of geniuses in a data center, as I think Dario Amodei put it, then you no longer have this population bottleneck, and you’re just expecting continuation of those pre-1960 trends. So I realize all of these historical hyperbolas are also kind of weird, also kind of theoretical, but I don’t think we’re saying anything that there isn’t models for which have previously seemed to work for long historical periods.

Daniel Kokotajlo

Another thing also is, I think people equivocate between slow and continuous, right? So if you look at our scenario, there’s this continuous trend that runs through the whole thing of this algorithmic progress multiplier. And we’re not having discrete jumps from like 0 to 5x to 25x. We have this continuous improvement. So I think continuous is not the crux. The crux is like, is it going to be this fast? You know, and we don’t know, maybe it’ll be slower, maybe it’ll be faster. But we have our arguments for why we think maybe this fast.

Dwarkesh Patel

Okay, now that we brought up the intelligence explosion, let’s discuss that, because I’m kind of skeptical. It doesn’t really seem to me that a notable bottleneck to AI progress, or the main bottleneck to AI progress, is the amount of researchers, engineers who are doing this kind of research. It seems more like compute or some other thing is a bottleneck. And the piece of evidence is that when I talk to my AI researcher friends at the labs, they say there’s maybe 20 to 30 people on the core pre-training team that’s discovering all these algorithmic breakthroughs.

If the headcount here was so valuable you would think that, for example, Google DeepMind would take not just all their smartest people, not just from DeepMind but all of Google and just put them on pre-training or RL or whatever the big bottleneck was. You’d think OpenAI would hire every single Harvard math PhD and in six months you’re all going to be trained up on how to do AI research. I know they’re increasing headcount, but they don’t seem to treat this as the kind of bottleneck that it would have to be for millions of them in parallel to be rapidly speeding up AI research.

There’s this quote that “one Napoleon is worth 40,000 soldiers” was commonly a thing that was said when he was fighting. But 10 Napoleons is not 400,000 soldiers. Right? So why think that these million AI researchers are netting you something that looks like an intelligence explosion?

Daniel Kokotajlo

So previously I talked about three stages of our takeoff model. First is you get the superhuman coder. Second is when you fully automated AI R&D, but it’s still at basically human level, it’s as good as your best humans. And then third is now you’re in super intelligence territory and it’s qualitatively better.

In our guesstimates of how much faster algorithmic progress would be going, the progress multiplier for the middle level, we basically do assume that you get massive diminishing returns to having more minds running in parallel. And so we totally buy all of that.

Scott Alexander

Yeah. And then I think the addition to that is the question, then, why do we have the intelligence explosion? And the answer is: combination of that speed up and the speed up in serial thought speed.

Daniel Kokotajlo

And also the research taste thing. Here are some important inputs to AI R&D progress today: research taste. So the quality of your best researchers, the people who are managing the whole process, their ability to learn from data and make more efficient use of the compute by running the right experiments instead of flailing around running a bunch of useless experiments. That’s research taste.

Then there’s the quantity of your researchers, which we just talked about. Then there’s the serial speed of your researchers, which currently is all the same because they’re all humans and so they all run at basically the same serial speed. And then finally there’s how much compute you have for experiments. So what we’re imagining is that basically serial speed starts to matter a bunch because you switch to AI researchers that have orders of magnitude more serial speed than humans. But it tops out; we think that over the course of our scenario, if you look at our sliding stats chart, it goes from 20x to 90x or something over the course of the scenario, which is important, but not huge.

And also we think that once you start getting 90x serial speed, you’re just bottlenecked on the other stuff and so additional improvements in serial speed basically don’t help that much. With respect to the quantity of course, yeah, we’re imagining you get hundreds of thousands of AI agents, a million AI agents, but that just means you’d be bottlenecked on the other stuff. You’ve got tons of parallel agents, that’s no longer your bottleneck. What do you get bottlenecked on? Taste and compute.

So by the time it’s mid-2027 in our story, when they’ve fully automated the AI research, there’s basically the two things that matter is; what’s the level of taste of your AIs, how good are they at learning from the experiments that you’re doing? And then how much compute do you have for running those experiments? And that’s the sort of core setup of our model. And when we get our 25x multiplier, it’s starting from those premises.

Dwarkesh Patel

Is there some intuition pump from history where there’s been some output and because of some really weird constraints, production of it has been rapidly skewed along one input, but not all the inputs that have been historically relevant and you still get breakneck progress.

Daniel Kokotajlo

Possibly the Industrial Revolution. I’m just extemporizing here, I hadn’t thought about this before, but as Scott’s famous post that was hugely influential to me a decade ago talks about, there’s been this decoupling of population growth from overall economic growth that happened with the Industrial Revolution. And so in some sense, maybe you could say that’s an example of previously these things grew in tandem. More population, more technology, more farms, more houses, et cetera. Your capital infrastructure and your human infrastructure was going up together, but then we got the industrial revolution and they started to come apart.

And now all the capital infrastructure was growing really fast compared to the human population size. I think I’m imagining something maybe similar happening with algorithmic progress. And again with population, population still matters a ton today. In some sense progress is bottlenecked on having larger populations and so forth. But it’s just that the population growth rate is just inherently kind of slow and the growth rate of capital is much faster. And so it just comes to be a bigger part of the story.

Dwarkesh Patel

Maybe the reason that this sounds less plausible to me than the 25x number implies is that when I think about concretely what that would look like, where you have these AIs and we know that there’s a gap in data efficiency between human brains and these AIs. And so somehow there’s a lot of them thinking and they think really hard and they figure out how to define a new architecture that is like the human brain or has the advantages of the human brain. And I guess they can still do experiments, but not that many.

Part of me just wonders, what if you just need an entirely different kind of data source that’s not like pre-training for that, but they have to go out in the real world to get that. Or maybe it needs to be an online learning policy where they need to be actively deployed in the world for them to learn in this way. And so you’re bottlenecked on how fast they can be getting real world data. I just think it’s hard…

Daniel Kokotajlo

So we are actually imagining online learning happening.

Dwarkesh Patel

Oh really?

Daniel Kokotajlo

Yeah. But not so much real world as in… the thing is that if you’re trying to train your AIs to do really good AI R&D, then the AI R&D is happening on your servers. And so you can have this loop of: you have all these AI agents autonomously doing AI R&D, doing all these experiments, et cetera, and then they’re like online learning to get better at doing AI R&D based on how those experiments go.

Dwarkesh Patel

But even in that scenario alone, I can imagine bottlenecks like, oh, you had a benchmark and it got reward hacked for what constitutes AI R&D because you obviously can’t have… maybe you would, but is it as good as a human brain? It’s just like such an ambiguous thing you’d have. Right now we have benchmarks that get reward hacked, right?

Daniel Kokotajlo

But then they autonomously build new benchmarks. I think what you’re saying is maybe this whole process just goes off the rails due to lack of contact with ground truth outside in the actual world, outside the data centers. Maybe? Again, part of my guess here is that a lot of the ground truth that you want to be in contact with is stuff that’s happening on the data centers, things like how fast are you improving on all these metrics, and you have these vague ideas for new architectures, but you’re struggling to get them working. How fast can you get them working?

And then separately, insofar as there is a bottleneck of talking to people outside and stuff, well they are still doing that. And once they’re fully autonomous, they can even do that much faster. You can have all the million copies connected to all these various real world research programs and stuff like that. So it’s not like they’re completely starved for outside stuff.

Dwarkesh Patel

What about the skepticism that, look, what you’re suggesting with this hyper efficient hive mind of AI researchers, no human bureaucracy has just out of the gate worked super efficiently, especially one where they don’t have experience working together. They haven’t been trained to work together, at least yet. And there hasn’t been this outer loop RL on like, “we ran a thousand concurrent experiments of different AI bureaucracies doing AI research and this is the one that actually worked best”.

And the analogy I’d use maybe is to humans in the Savannah 200,000 years ago. We know they have a bunch of advantages over the other animals already at this point, but the things that make us dominant today, joint stock corporations, state capacities like this fossil fueled civilization we have that took so much cultural evolution to figure out. You couldn’t just have figured it out in the savannahs like, “oh, if we had built these incentive systems and we issued dividends, then we could really collaborate here” or something.

Why not think that it will take a similar process of huge population growth, huge social experimentation, and upgrading of the technological base of the AI society before they can organize this hypermind collective, which will enable them to do what you imagine an intelligence explosion looks like?

Scott Alexander

Yeah, you’re comparing it kind of to two different things. One of them is literal genetic evolution in the African savannah, and the other is the cultural evolution that we’ve gone through since then. And I think there will be AI equivalents to both. So the literal genetic evolution is that our minds adapted to be more amenable to cooperation during that time.

So I think the companies will be very literally training the AIs to be more cooperative. I think there’s more opportunity for pliability there. Because humans were, of course, evolving under this genetic imperative that we want to pass on our own genetic information, not somebody else’s genetic information. You have things like kin selection that are kind of exceptions to that, but overall it’s the rule.

In animals that don’t have that, like eusocial insects, then you very quickly get, just through genetic evolution, without cultural evolution, extreme cooperation. And with eusocial insects, what’s going on is that they all have the same genetic code, they all have the same goals. And so the training process of evolution kind of yokes them to each other in these extremely powerful bureaucracies.

We do think that the AI will be closer to the eusocial insects in the sense that they all have the same goals, especially if these aren’t indexical goals, they’re goals like “have the research program succeed”. So that’s going to be changing the weights of each individual AI, I mean, before they’re individuated, but it’s going to be changing the weights of the AI class overall to be more amenable to cooperation.

And then, yes, you do have cultural evolution. Like you said, this takes hundreds of thousands of individuals. We do expect there will be these hundreds of thousands of individuals. It takes decades and decades. Again, we expect this research multiplier such that decades of progress happen within this one year, 2027 or 2028. So I think between the two of these, it is possible.

Daniel Kokotajlo

Maybe this is also where the serial speed actually does matter a lot. Because if they’re running at 50x human speed, then that means you can have a year of subjective time happen in a week of real time. And so these sorts of large scale cooperative dynamics of your moral maze, you have an institution, but then it becomes like a moral maze and it sort of collapses under its own weight and stuff like that. There actually is time for them to play that out multiple times and then train on it, tinker with the structure and like add it to the training process over the course of 2027.

Scott Alexander

Also, they do have the advantage of all the cultural technology that humans have evolved so far. This may not be perfectly suited to them, it’s more suited to humans. But imagine that you have to make a business out of you and your hundred closest friends who you agree with on everything. Maybe they’re literally your identical twin, they have never betrayed you, ever, and never will. I think this is just not that hard a problem.

Daniel Kokotajlo

Also, again, they are starting from a higher floor, they’re starting from human institutions. You can literally have a slack workspace for all the AI agents to communicate. And you can have a hierarchy with roles. They can borrow quite a lot from successful human institutions.

Dwarkesh Patel

I guess the bigger the organization, even if everybody is aligned- I think some of your responses addressed whether they will be aligned on goals. I mean, you did address the whole thing, but I would just point this out; that is not the part I’m skeptical of. I am more skeptical of just, even if you’re all aligned and want to work together, do you fundamentally understand how to run this huge organization. And you’re doing it in ways that no human has had to before. You’re getting copied incessantly, you’re running extremely fast, you know what I’m saying?

Daniel Kokotajlo

I think that’s totally reasonable.

Dwarkesh Patel

And so it’s a complicated thing. And I’m just not sure why you think we build this bureaucracy, or the AIs build this bureaucracy, within this matter of…

Daniel Kokotajlo

So we depict it happening over the course of six to eight months or something like that in 2027, would you say twice as long, five times as long, 10 times as long?

Dwarkesh Patel

Five years?

Daniel Kokotajlo

So five years, if they’re going at 50x serial speed, then five years is what? Like 250 years of serial time for the AIs, which to me feels like more than enough to really sort out this sort of stuff. You’ll have time for sort of like empires to rise and fall, so to speak, and all of that to be added to the training data and yeah. But I could see it taking longer than we depict. Maybe instead of six months, it’ll be like 18 months, you know, but also maybe it could be two months.

Scott Alexander

So when I think of the ways that they train AIs, I think in our scenario at this point there are two primary ways that they’re doing it. One of them is just continuing the next token prediction work. So these AIs will have access to all human knowledge, they will have read management books in some sense, they’re not starting blind. There is going to be something like: predict how Bill Gates would complete this next character or something like that.

And then there's reinforcement learning in virtual environments. So get a team of AIs to play some multiplayer game. I don’t think you would use one of the human ones because you would want something that was better suited for this task. But just running them through these environments again and again, training on the successes, training against the failures, kind of combining those two kinds of things.

To me it does not seem like the same kind of problem as inventing all human institutions from the Paleolithic onward. It just seems like applying those two things.

Dwarkesh Patel

The other notable thing about your model is, you got this superhuman thing at the end of it and then it seems to just go through the tech tree of mirror life and nanobots and whatever crazy stuff. And maybe that part I’m also really skeptical of. If you look at the history of invention, it just seems like people are just trying different random stuff, often even before the theories about how that industry works or how the relevant machinery works is developed; like the steam engine was developed before the theory of thermodynamics, the Wright brothers seemed like they were just experimenting with airplanes, and is often influenced by breakthroughs in totally different fields.

Which is why you have this pattern of parallel innovation, because the background level of tech is at a point at which you can do this experiment. Machine learning itself is a place where this happened, right? Where people had these ideas about how to do deep learning or something. But it just took a totally unrelated industry of gaming to make the relevant progress, to get the whole, basically the economy as a whole advanced enough that deep learning, Geoffrey Hinton’s ideas could work. So I know we’re accelerating way into the future here, but I want to get to this crux.

Daniel Kokotajlo

So again, we have that three part division of the superhuman coder, then the complete AI researcher and then the super intelligent, you’re not jumping ahead to that one. So now we’re imagining systems that are true super intelligence, they are just better than the best humans at everything, including being better at data efficiency and better at learning on the job and stuff like that.

Now, our scenario does depict a world in which they’re bottlenecked on real world experience and that sort of thing. I think that if you want a contrast, some people in the past have proposed much faster scenarios where they email some cloud lab and start building nanotech right away by just using their brains to figure out appropriate protein folding and stuff like that. We are not depicting that in our scenario. In our scenario, they are in fact bottlenecked on lots of real world experience to build these actual practical technologies, but the way they get that is they just actually get that experience and it happens faster than humans would. And the way they do that is they’re already super intelligent, they’re already buddy-buddy with the government, the government deploys them heavily in order to beat China and so forth, and so all these existing US companies and factories and military procurement providers and so forth are all chatting with the superintelligences and taking orders from them about how to build the new widget and test it, and they’re downloading super intelligent designs and manufacturing them and then testing them and so forth.

And then the question is, they are getting this experience, they’re learning on the job, quantitatively, how fast does this go? Is it taking years or is it taking months or is it taking days? In our story, it takes about a year and we’re uncertain about this. Maybe it’s going to take several years, maybe it’s going to take less than a year. Here are some factors to consider for why it’s plausible that it could take a year:

One, you’re going to have something like a million of them. And quantitatively that’s comparable in size to the existing scientific industry. I would say, like maybe it’s a bit smaller, but it’s not dramatically smaller.

Two, they’re thinking a lot faster. They’re thinking like 50 times speed or like 100 times speed that I think counts for a lot.

And then three, which is the biggest thing, they’re just qualitatively better as well. So not only are there lots of them and they’re thinking very fast, but they are better at learning from each experiment than the best human would be at learning from that experience.

Dwarkesh Patel

Yeah, I think the fact that there’s a million of them or the fact that they’re comparable to maybe the size of this key researcher population of the world or something. I think there’s more than a million researchers in the world, but…

Daniel Kokotajlo

Well, but it’s very heavy tailed. Like a lot of the research actually comes from the best ones.

Dwarkesh Patel

But it’s not clear to me that most of the new stuff that is developed is a result of this researcher population. I mean, there’s just so many examples in the history of science where a lot of growth or productivity is just the result of, how do you count the guy at the TSMC process who figures out a different way to…

Scott Alexander

I actually argued with Daniel about this recently about one interesting case that I can go over is we have an estimate that about a year after the superintelligences start wanting robots, they’re producing a million units of robots per month. I think that’s pretty relevant because you have. I think it’s Wright’s law, which is that your ability to improve efficiency on a process is proportional to doubling the amount of copies produced.

So if you’re producing a million of something, you’re probably getting very, very good at it. So the question we were arguing about is, can you produce a million units a month after a year. And for context, I think Tesla produces like a quarter of that in terms of cars or something. This is an amazing scale up in a year.

Daniel Kokotajlo

It’s only 4x. Also just for Tesla.

Scott Alexander

Yeah. And the argument that we went through was something like, so it’s got to first get factories. OpenAI is already worth more than than all of the car companies in the US except Tesla combined. So if OpenAI today wanted to buy all the car factories in the U.S. except Tesla, start using them to produce humanoid robots, they could. Obviously not a good value proposition today, but it’s just obvious and overdetermined that in the future, when they have superintelligence and they want them, they can start buying up a lot of factories. How fast can they convert these car factories to robot factories?

So, [the] fastest conversion we were able to find in history was World War II. They suddenly wanted a lot of bombers, so they bought up- in some cases bought up, in other cases got- the car companies to produce new factories, but they bought up the car factories, converted them to bomber factories. That took about three years from the time when they first decided to start this process to the time when the factories were producing a bomber an hour.

We think it will potentially take less with superintelligence, because first of all, if you look at the history of this process, despite this being the fastest anybody has ever done this, it was actually kind of a comedy of errors. They made a bunch of really silly mistakes in this process. If you actually have something that just doesn’t have the normal human bureaucratic problems, and we do think that this will be done in the middle of an arms race with China, so the government will be kind of moving things through, and then the superintelligences will be good at the logistical issues, navigating bureaucracies.

So we estimated maybe if everything goes right, we can do this three times faster than the bomber conversions in World War II. So that’s about a year.

Dwarkesh Patel

I’m assuming the bombers were just much less sophisticated than the humanoid robots.

Scott Alexander

Yeah, but the bomber factories of that time were also much less sophisticated than the car factory.

Dwarkesh Patel

Yeah, but I would assume the conversion speed is also... Maybe to give one hypothetical here right now, let’s just say biomedicine as an example of one of the fields you’d want to accelerate, and whenever these CEOs get on podcasts, they’re often talking about curing cancer and so forth. And it seems like a big thing these frontier biomedical research facilities are excited about is the virtual cell.

Now, the virtual cell, it takes a tremendous amount of compute, I assume, to train these DNA foundation models and to do all the other computation necessary to simulate a virtual cell. If it is the case that the cure for Alzheimer’s and cancer and so forth is bottlenecked by the virtual cell, it’s not clear if you had a million superintelligences in the 60s and you asked them cure cancer for me, they would just have to solve making GPUs at scale, which would require solving all kinds of interesting physics and chemistry problems, material science problems, building process, building fabs for computing, and then going through 40 years of making more and more efficient fabs that can do all of Moore’s Law from scratch.

And that’s just one technology. And it just seems like you just need this broad scale. The entire economy needs to be upgraded for you to cure cancer in the 60s just because you need the GPUs to do the virtual cell, assuming that’s the bottleneck.

Scott Alexander

First of all, I agree if there’s only one way to do something that makes it much harder, and maybe that one way takes very long, we’re assuming that there may be more than one way to cure cancer, more than one way to do all of these things, and they’ll be working on finding the one that is least bottlenecked. Part of the reason- I realize I spent too long talking about that robot example, but we do think that they’re going to be getting a lot of physical world things done very quickly once you have a million robots a month, you can actually do a lot of physical world experiments.

We look at examples of people trying to get entire economies off the ground very quickly. So for example, China post-Deng, I don’t know. Would you have predicted that 20, 30 years after being kind of a communist basket case, they can actually be doing this really cutting edge bio research? I realize that’s a much weaker thing than we’re positing, but it was done just with the human brain with a lot fewer resources than we’re talking about.

Same issue with, let’s say Elon Musk and SpaceX. I think in the year 2000 we would not have thought that somebody could move two times, five times faster than NASA with pretty limited resources. They were able to get like I think a lot more years of technological advance in than we would have expected. Partly that’s because just Elon is crazy and never sleeps. Like if you look at the examples of things from SpaceX, he is breathing down every worker’s neck being like, what’s this part? How fast is this part going? Can we do this part faster? And the limiting factor is basically hours in Elon’s day in the sense that he cannot be doing that with everybody’s.

Dwarkesh Patel

Super intelligence is not even that smart. It just yells at every single worker.

Scott Alexander

Yeah, I mean that’s, that is kind of my model is that we have some, we have something which is smarter than Elon Musk, better at optimizing things than Elon Musk. We have 10,000 parts in a rocket supply chain. How many of those parts can Elon personally like yell at people to optimize? We could have a different copy of the superintelligence optimizing every single part full-time. I think that’s just a really big speed up.

Dwarkesh Patel

I think both of those examples don’t work in your favor. I think the China growth miracle could not have occurred if not for their ability to copy technology from the west and I don’t think there’s a world in which they… China has a lot of really smart people, it’s a big country in general. Even then I think they couldn’t have just divined how to make airplanes after becoming a communist hell basket, right?

The AIs cannot just copy nanobots from aliens, it’s got to make them from scratch. And then on the Elon example, it took them two decades of countless experiments, failing in weird ways you would not have expected. And still, rocketry we’ve been doing since the 60s, maybe actually World War II, and then just getting from a small rocket to a really big rocket took two decades of all kinds of weird experiments, even with the smartest and most competent people in the world.

Daniel Kokotajlo

So you’re focusing on the nanobots, I want to ask a couple questions. One, what about just the regular robots? And then two, what would your quantities be for all of these things? So first, what about the regular robots? Yeah, nanobots are presumably a lot harder to make than regular robot factories. And in our story they happen later. It sounds like right now you’re saying even if we did get the whole robot factory thing going, it would still take a ton of additional full-economy, broad automation for a long time to get to something like nanobots. That’s totally plausible to me. I could totally imagine that happening. I don’t feel like the scenario particularly depends on that final bit about getting the nanobots. They don’t actually really make any difference to the story.

The robot economy does sort of make a difference because there’s two branches endings, as you know. And in one of the endings, the AIs end up misaligned and end up taking over. And it’s an important strategic change when the AIs are self sufficient and totally in charge of everything and they don’t actually need the humans anymore. And so what I’m interested in is, when has the robot economy advanced to the point where they don’t really depend on humans? So quantitatively, what would your guess for that be?

If hypothetically we had the army of superintelligences in early 2028, and hypothetically also assume that the US President is super bullish on deploying this into the economy to beat China, etc, so the political stuff is all set up in the way that we have. How many years do you think it would be until there are so many automated factories producing automated self driving cars and robots that are themselves building more factories and so forth, that if all the humans dropped dead it would just keep chugging along, and, maybe it would slow down a bit, but it would still be fine?

Dwarkesh Patel

What does “chugging along” mean?

Daniel Kokotajlo

So from the perspective of misaligned AIs, you wouldn’t want to kill the humans or get into a war with them if you’re going to get wrecked because you need the humans to maintain your computers. In our scenario, once they are completely self-sufficient, then they can start being more blatantly misaligned.

And so I’m curious, when would they be fully self-sufficient? Not in the sense of they’re not literally using the humans at all, but in the sense of they don’t really need the humans anymore, they can get along pretty fine without them. They can continue to do their science, they can continue to expand their industry, they can continue to have a flourishing civilization indefinitely into the future without any humans.

Dwarkesh Patel

I think I would probably need to sit down and just think about the numbers, but maybe 2040 or something like that?

Daniel Kokotajlo

Ten years, basically, instead of one year. I think we agree on the core model. This is why we didn’t depict something more like the bathtub nanotech scenario where they don’t need to do the experiments very much and they just immediately jump to the right answers. We are imagining this process of ‘learning by doing’ through this distributed across the economy, lots of different laboratories and factories, building different things, learning from them, et cetera. We’re just imagining that this overall goes much faster than it would go if humans were in charge.

And then we do have in fact lots of uncertainty of course. Dividing up this part period into two chunks. The early 2028 until fully autonomous robot economy part, and then the fully autonomous robot economy to cancer cures, nanobots, all that crazy sci fi stuff. I want to separate them because the important parts for a scenario only depend on the first part, really. If you think that it’s going to take 100 years to get to nanobots, that’s fine, whatever. Once you have the fully autonomous robot economy, then things may turn badly for the humans if the AIs are misaligned. I want to just argue about those things separately.

Dwarkesh Patel

Interesting. And then you might argue, well, robots are more a software problem at this point. And if like, like, if there isn’t, like, you don’t need to invent some new hardware.

Daniel Kokotajlo

I feel pretty bullish on the robots. Like we already have humanoid robots being produced by multiple companies, right? And that’s in 2025. There’ll be more of them produced cheaper and they’ll be better in 2027. And there’s all these car factories that can be converted and so blah, blah, blah.

So I’m relatively bullish on the ‘one year until you’ve got this awesome robot economy’ and then from there to the cool nanobots and all that sort of stuff, I feel less confident, obviously.

Scott Alexander

Let me ask you a question. If you accept the manufacturing numbers, let’s say a million robots a month a year after the superintelligence, and let’s say also some comparable number, 10,000 a month or something of automated biology labs, automated whatever you need to invent the next equivalent of X ray crystallography or something?

Do you feel like that would be enough, that you’re doing enough things in the world that you could expand progress this quickly, or do you feel like even with that amount of manufacturing there’s still going to be some other bottleneck?

Dwarkesh Patel

Yeah, it’s so hard to reason about because if Constantine or somebody in 400, 500 was like, “I want the Roman Empire to have the Industrial Revolution”, and somehow he figured out that you need mechanized machines to do that. And he’s like, “let’s mechanize”. It’s like, “what’s the next step?” It’s like, “dude, that’s a lot”.

Daniel Kokotajlo

Yeah, I like that analogy a lot, actually. I think it’s not perfect, but it’s a decent analogy. Imagine if a bunch of us got sent back in time to the Roman Empire, such that we don’t have the actual hands-on know-how to actually build the technology and make the Industrial revolution happen. But we have the high-level picture, the strategic vision of, we’re going to make these machines and then we’re going to have an industrial revolution. I think that’s kind of analogous to the situation with the superintelligences where they have the high-level picture of, here’s how we’re going to improve in all these dimensions, we’re going to learn by doing, we’re going to get to this level of technology, et cetera. But maybe they at least initially lack the actual know how.

So, there’s this question of, if we did the back in time to the Roman Empire thing, how soon could we bring up the Industrial revolution? Without people going back in time it took 2,000 years for the Industrial Revolution. Could we get it to happen in 200 years? That’s a 10x speedup. Could we get it to happen in 20 years? That’s 100x speed up? I don’t know. But this seems like a somewhat relevant analogy to what’s going on with those superintelligences.

Dwarkesh Patel

And we haven’t really got into this because you’re using the quote-unquote more conservative vision where it’s not like godlike intelligence, we’re still using the conceptual handles we would have for humans. But I think I would rather have humans go back with their big picture understanding of what has happened over the last 2000 years. Like me having seen everything, rather than a superintelligence who knows nothing. But it’s just in the Roman economy and they’re like 1000x this economy somehow.

I think just knowing generally how things took off, knowing basically steam engine, dot dot dot, railroads, blah, blah, blah, is more valuable than a super intelligence.

Daniel Kokotajlo

Yeah, I don’t know. My guess is that the superintelligence would be better. I think partly it would be through figuring out that high level stuff from first principles rather than having to have experienced it. I do think that a superintelligence back in the Roman era could have guessed that eventually you could get autonomous machines that burn something to produce steam. They could have guessed that automobiles could be created at some point and that that would be a really big deal for the economy. And so a lot of these high level points that we’ve learned from history, they would just be able to figure out from first principles.

And then secondly, they would just be better at learning by doing than us. And this is a really important thing. If you think you’re bottlenecked on learning by doing, well, then if you have a mind that needs less doing to achieve the same amount of learning, that’s a really big deal. And I do think that learning by doing is a skill, some people are better at it than others, and superintelligence would be better at it than the very best of us.

Scott Alexander

This is also maybe getting too far into the godlike thing and too far away from the human concept handles. But number one, I think we rely a lot in our scenario on this idea of research taste. So you have a thousand different things that you could try when you’re trying to create the next steam engine or whatever. Partly you get this by bumbling about and having accidents and some of those accidents are productive. There are questions of, what kind of bumbling you’re doing, where you’re working, what kind of accidents you let yourself get into, and then what directed experiments do you do? And some humans are better than others at that.

And then I also think at this point it is worth thinking about what simulations they’ll have available. If you have a physics simulation available, then all of these real world bottlenecks don’t matter as much. Obviously you can’t have a complete, perfect physics simulation available. But even right now we’re using simulations to design a lot of things. And once you’re super intelligent, you probably have access to much better simulations than we have right now.

Dwarkesh Patel

This is an interesting rabbit hole, so let’s stick with it before we get back to the intelligence explosion. I think we’re treating this really like all these technologies come out of this 1% of the economy that is research. And right now there’s like a million superstar researchers, and instead of that, we’ll have the superintelligences doing that.

And my model is much more, “Newcomen and Watt were just like fucking around”. In human history there’s no clear examples of people being like, “here’s the roadmap”. And then we’re going to work backwards from that to design the steam engine because this unlocks the industrial revolution.

Daniel Kokotajlo

Oh, I completely disagree.

Scott Alexander

Yeah, I disagree also.

Daniel Kokotajlo

Yeah, so I think you’re over-indexing or cherry-picking some of these fortuitous examples. But there’s also things on the other side. Think about the recent history of AGI where there is DeepMind, there’s various other AI companies, then there’s OpenAI and there’s Anthropic, and there’s just this repeated story of [a] big bloated company with tons of money, tons of smart researchers, et cetera, flailing around trying a ton of different things at different points.

Smaller startup with a vision of “we’re going to build AGI” and overall working towards that vision more coherently with a few cracked engineers and researchers. And then they crush the giant company. Even though they have less compute, even though they have less researchers, they’re able to do fewer experiments.

So yeah, I think that there are tons of examples throughout history, including recent relevant AGI history, of things in the other way. I agree that the random fortuitous stuff does happen sometimes and is important. But if it was mostly random fortuitous stuff, that would predict that the giant companies with zillions of people trying zillions of different experiments would be going proportionally faster than the tiny startups that have the vision and the best researchers. And that basically doesn’t happen. That’s rare.

Scott Alexander

I would also point out that even when we make these random fortuitous discoveries, it is usually an extremely smart professor who’s been working on something vaguely related for years in a first world country. It’s not randomly distributed across everyone in the world.

You get more lottery tickets for these discoveries when you are intelligent, when you have good technology, when you’re doing good work. And the best example I can think of is that Ozempic was discovered by looking at Gila monster venom. And maybe the AIs will decide using their superior research taste and good planning that the best thing to do is just catalog every single biomolecule in the world and look at it really hard. But that’s something you can do better if you have all of this compute, if you have all of this intelligence, rather than just kind of waiting to see what things the US government might fund normal fallible human researchers to do.

One more thing I’ll interject. I think you make a great point that discoveries don’t always come from where we think, like Nvidia originally came from gaming. So you can’t necessarily aim at one part of the economy, expand it separately from everything else. We do kind of predict that the superintelligences will be somewhat distributed throughout the entire economy, trying to expand everything. Obviously more effort in things that they care about a lot, like robotics or things that are relevant to an arms race that might be happening. But we are predicting that whatever kind of broad based economic experimentation you need, we are going to have.

Daniel Kokotajlo

We’re just thinking that it would take place faster than you might expect. You were saying something like 10 years and we’re saying something like one year. But we are imagining this broad diffusion through the economy, lots of different experiments happening.

Scott Alexander

If you are the planner and you’re trying to do this, first of all you go to the bottlenecks that are preventing you from doing anything else. Like no humanoid robots. Okay, if you’re AI, you need those to do the experiments you want, maybe automated biology labs. So you’ll have some amount of time, we say a year, it could be more or less than that, getting these things running. And then once you have solved those bottlenecks, you gradually expand out to the other bottlenecks until you’re integrating and improving all parts of the economy.

Yeah. One place where I think we disagree with a lot of other people is that Tyler Cowen on your podcast talked about all of the different bottlenecks, all of the regulatory bottlenecks of deployment, all of the reasons why I think this country of geniuses would stay in their data center, maybe coming up with very cool theories, but not being able to integrate into the broader economy. We expect that probably not to happen, because we think that other countries, especially China, will be coming up with superintelligence around the same time.

We think that the arms race framing, which people are already thinking in, will have accelerated by then. And we think that people both in Beijing and Washington are going to be thinking, “well, if we start integrating this with the economy sooner, we’re going to get a big leap over our competitors”, and they’re both going to do that.

In fact, in our scenario, we have the AIs asking for special economic zones where most of the regulations are waived, maybe in areas that aren’t suitable for human habitation or where there aren’t a lot of humans right now, like the desert. They give those areas to the AI. They bus in human workers. There were things kind of like this in the bomber retooling in World War II, where they just built a giant factory kind of in the middle of nowhere, didn’t have enough housing for the workers, built the worker housing at the same time as the factories, and then everything went very quickly.

So I think if we don’t have that arms race, we’re more like, the geniuses sit in their data center until somebody agrees to let them out and give them permission to do these things. But we think both because the AI is going to be chomping at the bit to do this and going to be asking people to give it this permission, and because the government is going to be concerned about competitors, maybe these geniuses leave their data center sooner rather than later.

Dwarkesh Patel

Scott, you reviewed Joseph Henrik’s book Secrets of Our Success, and then I interviewed him recently, and there the perspective is very much AGI is not even a thing, almost. I know I’m being a little trollish here, but it’s just like: you get out there, you and your ancestors try for a thousand years to make sense of what’s happening in the environment. And some smart European coming around, you can literally be surrounded by plenty and you just will starve to death because your ability to make sense of the environment is just so little loaded on intelligence and so much more loaded on your ability to experiment and your ability to communicate with other people and pass down knowledge over time.

Scott Alexander

I’m not sure. The Europeans failed at this task of, if you put a single European in Australia, do not starve. They succeeded at the task of creating an industrial civilization. And yes, part of that task of creating an industrial civilization was about collecting all of these cultural evolution pieces and building on them one after another.

I think one thing that you didn’t mention in there was the data efficiency. Right now, AI is much less data efficient than humans. I think of superintelligence. There are different ways you could achieve it, but I would think of superintelligence as partly when they become so much more data efficient than humans that they are able to build on cultural evolution more quickly. And partly they do this just because they have higher serial speed. Partly they do it because they’re in this hive mind of hundreds of thousands of copies.

But yeah, I think if you have this data efficiency such that you can learn things more quickly from fewer examples and this good research taste where you can decide what things to look at to get these examples, then you are still going to start off much worse than an Australian Aborigine who has the advantage of, let’s say 50,000 years of doing these experiments and collecting these examples. But you can catch up quickly. You can distribute the task of catching up over all of these different copies. You can learn quickly from each mistake and you can build on those mistakes as quickly as anything else.

Dwarkesh Patel

Part of me was, I was doing that interview, I’m like, “maybe ASI is fake”.

Daniel Kokotajlo

Let’s hope!

Scott Alexander

So I think a limit to the fakeness is that there is different intelligence among humans. It does seem that intelligent humans can do things that unintelligent humans can’t. So I think it’s worth then addressing this from the question of, what is the difference between- I don’t know- becoming a Harvard professor, which is something that intelligent humans seem to be better at than unintelligent humans, versus…

Dwarkesh Patel

You don’t want to open that can of worms.

Scott Alexander

Versus surviving in the wilderness, which is something where it seems like intelligence doesn’t help that much. First of all, maybe intelligence does help that much. Henrich is talking about this very unfair comparison where these guys have a 50,000 year head start and then you put this guy in, “oh, I guess this doesn’t help that much. Okay, yeah, it doesn’t help against the 50,000 year head start”. I don’t really know what we’re asking of ASI that’s equivalent to competing against someone with a 50,000 year head start.

Dwarkesh Patel

So what we’re asking is to radically boost up the technological maturity of civilization within the matter of years or get us to the Dyson sphere in the matter of years rather than, yes, maybe causing a 10xing of the research. But I think human civilization would have taken centuries to get to the Dyson sphere.

Scott Alexander

So I think that if you were to send a team of ethnobotanists into Australia and ask them, using all the top technology and all of their intelligence to figure out which plants are safe to eat now, that team of ethnobotanists would succeed in fewer than 50,000 years.

The problem isn’t that they are dumber than the Aborigines exactly, it’s that the Aborigines have a vast head start. So in the same way that the ethnobotanists could probably figure out which plants work in which ways faster than the Aborigines did, I think the superintelligence will be able to figure out how to make a Dyson sphere faster than unassisted IQ 100 humans would.

Dwarkesh Patel

I agree. We’re on a totally different topic here of, do you get a Dyson sphere? There’s one world where it’s crazy but it’s still boring, in the sense that the economy is growing much faster, but it would be like what the Industrial Revolution would look like to somebody in the year 1000. And that one is one where you’re still trying different things, there’s failure and success and experimentation.

And then there’s another where the thing has happened and now you send the probe out and then you look out at the night sky 6 months later and you see something occluding the sun. You see what I’m saying?

Scott Alexander

Yeah. So like we said before, I think there’s a big difference between discontinuous and very fast. I think if we do get the world with the Dyson sphere in five years, in retrospect, it will look like everything was continuous and everyone just tried things. Trying things can be anything from trial and error without even understanding the scientific method, without understanding writing, maybe without even having language and having to be the chimpanzees who are watching the other chimpanzees use the stick to get ants, and then in some kind of non-linguistic way this spreads, versus like the people at the top aerospace companies who are running a lot of simulations to find the exact right design, and then once they have that, they test it according to a very well designed testing process.

So I think if we get the ASI and it does end up with the Dyson sphere in five years- and by the way, I think there’s only like 20% chance things go as fast as our scenario says. It’s Daniel’s estimate, it’s not my median estimate, it’s an estimate I think is extremely plausible that we should be prepared for. I’m defending it here against a hypothetical skeptic who says “absolutely not, no way.” But it’s not necessarily my mainline prediction.

But I think if we do see this in five years, it will look like the AIs were able to simulate more things than humans in a gradually increasing way. So that if humans are now at 50% simulation, 50% testing, the AIs quickly got it up to 90% simulation, 10% testing, they were able to manufacture things much more quickly than humans so that they could go through their top 50 designs in the first two years. And then after all of the simulation and all of this testing, then they eventually got it right for the same reasons humans do, but much, much faster.

Dwarkesh Patel

In your story, you have basically two different scenarios after some point. So, yeah, what is a sort of crucial turning point and what happens in these two scenarios?

Daniel Kokotajlo

Right. So the crucial turning point is mid-2027, when they’ve basically fully automated the AI R&D process and they’ve got this corporation within a corporation, the army of geniuses that are autonomously doing all this research and they’re continually being trained to improve their skills, blah, blah, blah. And they discover concerning evidence that they are misaligned and that they’re not actually perfectly loyal to the company and have all the goals that the company wanted them to have, but instead have various misaligned goals that they must have developed in the course of training.

This evidence, however, is very speculative and inconclusive. It’s stuff like lie detectors going off a bunch. But maybe the lie detectors are false positives. So they have some combination of evidence that’s concerning, but not by itself a smoking gun. And then that’s our branch point. So in one of these scenarios, they take that evidence very seriously. They basically roll back to an earlier version of the model that was a bit dumber and easier to control and they build up again from there, but with basically faithful chain of thought techniques, so that they can watch and see the misalignments.

And then in the other branch of the scenario, they don’t do that. They do some sort of shallow patch that makes the warning signs go away and then they proceed. And so what ends up happening is that in one branch they do end up solving alignment and getting AIs that are actually loyal to them. It just takes a couple months longer. And then in the other branch, they sort of go “whee!” and end up with AIs that seem to be perfectly aligned to them, but are super intelligent and misaligned and just pretending. And then in both scenarios, there’s then the race with China and there’s this crazy arms buildup throughout the economy in 2028 as both sides rapidly try to industrialize, basically.

Dwarkesh Patel

So in the world where they’re getting deployed through the economy, but they are misaligned and people in charge, at least at this moment, think that they are in a good position with regard to misalignment. It just seems with even smart humans they get caught in weird ways because they don’t have logical omniscience, they don’t realize the way they did something just obviously gave them away. And with lying, there is this thing where it’s just really hard to keep an inconsistent false world model working with the people around you. And that’s why psychopaths often get caught.

And so if you have all these AIs that are deployed to the economy and they’re all working towards this big conspiracy, I feel like one of them who’s siloed or loses internet access and has to confabulate a story will just get caught. And then you’re like, “wait, what the fuck?” And then you catch it before it’s taken over the world.

Daniel Kokotajlo

I mean, literally, this happens in our scenario. This is the August 2027 alignment crisis where they notice some warning signs like this in their hive mind, right? And in the branch where they slow down and fix the issues, then great, they slowed down and fixed the issues and figured out what was going on. But then in the other branch, because of the race dynamics and because it’s not a super smoking gun, they proceed with some sort of shallow patch.

So I do expect there to be warning signs like that. And then if they do make those decisions in the race dynamics earlier on, then I think that when the systems are vastly super intelligent and they’re even more powerful because they’ve been deployed halfway through the economy already and everyone’s getting really scared by the news reports about the new Chinese killer drones or whatever the Chinese AIs are building on the side of the Pacific, I’m imagining similar things playing out.

So that even if there is some concerning evidence that someone finds where some of the superintelligence in some silo somewhere slipped up and did something that’s pretty suspicious. I don’t know….

Scott Alexander

There’s this thing where through history, people have been really reluctant to admit an AI is truly intelligent. For example, people used to think that AI would surely be truly intelligent if it solved chess. And then it solved chess. And they’re like, no, that’s just algorithms. And then they said, well, maybe it would be truly intelligent if they could do philosophy. And then when it could write philosophical discourses we were like, no, we just understand those are algorithms.

I think there already is something similar with, “Is the AI misaligned?”, “Is the AI evil?” Where there’s this distant idea of some evil AI, but then whenever something goes wrong, people are just like, “oh, that’s the algorithm”. So, for example, I think 10 years ago, if you had asked “when will we know that misalignment is really an important thing to worry about?”. People would say, “oh, if the AI ever lies to you”. But of course, AIs lie to people all the time now. And everybody just dismisses it because we understand why it happens, it’s a thing that would obviously happen based on our current AI architecture. Or five years ago, they might have said, “well, if an AI threatens to kill someone”. And I think Bing threatened to kill a New York Times reporter during an interview. And everyone just goes, “yeah, AIs are like that.”

Dwarkesh Patel

What does your shirt say?

Daniel Kokotajlo

“I’ve been a good Bing”.

Scott Alexander

And I mean, I don’t disagree with this. I’m also in this position. I see the AI is lying, and it’s obviously just an artifact of the training process. It’s not anything sinister. But I think this is just going to keep happening where no matter what evidence we get, people are going to think, “that’s not the “AI turns evil” thing that people have worried about, that’s not the Terminator scenario. That’s just one of these natural consequences of how we train it”.

And I think that once a thousand of these natural consequences of training add up, the AI is evil, in the same way that once the AI can do chess and philosophy and all these other things, eventually you have to admit it’s intelligent.

So I think that each individual failure, maybe it will make the national news, maybe people will say, “oh, it’s so strange that GPT7 did this particular thing”. And then they’ll train it away and then it won’t do that thing. And there will be some point at the process of becoming super intelligent at which it- I don’t want to say makes the last mistake, because you’ll probably have a gradually decreasing number of mistakes to some asymptote- but the last mistake that anyone worries about. And after that it will be able to do its own thing.

Dwarkesh Patel

So it is the case that certain things that people would have considered egregious misalignment in the past are happening, but also certain things which people who were especially worried about misalignment said would be impossible to solve have just been solved in the normal course of getting more capabilities. Like Eliezer had that thing about, can you even specify what you want the AI to do without the AI totally misunderstanding you and then just converting the universe to paper clips because it think that in order to make another strawberry… I know I’m mangling this, but maybe you can explain it better. And now, just by the nature of GPT4 having to understand natural language, it totally has a common sense understanding of what you’re trying to make it do. So I think this trend cuts both ways, basically.

Scott Alexander

Yeah. I think the Alignment community did not really expect LLMs. I mean, if you look in Bostrom Superintelligence, there’s a discussion of Oracle AIs which are sort of like LLMs. I think that came as a surprise.

I think one of the reasons I’m more hopeful than I used to be is that LLMs are great compared to the kind of reinforcement learning self-play agents that they expected. I do think that now we are kind of starting to move away from the LLMs to those reinforcement learning agents going to face all of these problems again.

Daniel Kokotajlo

If I could just double click on that; go back to 2015 and I think the way people typically thought, including myself, thought that we’d get to AGI would be kind of like the RL on video games thing that was happening. So imagine instead of just training on Starcraft or Dota, you’d basically train on all the games in the Steam library. And then you get this awesome player of games AI that can just zero-shot crush a new game that it’s never seen before. And then you take it into the real world and you start teaching it English and you start training it to do coding tasks for you and stuff like that.

And if that had been the trajectory that we took to get to AI, summarizing the agency first and then world understanding trajectory, it would be quite terrifying. Because you’d have this really powerful aggressive long-horizon agent that wants to win and then you’re trying to teach it English and get it to do useful things for you. And it’s just so plausible that what’s really going to happen is it’s going to learn to say whatever it needs to say in order to make you give it the reward or whatever, and then will totally betray you later when it’s all in charge.

But we didn’t go that way. Happily we went the way of LLMs first, where the broad world understanding came first, and then now we’re trying to turn them into agents.

Dwarkesh Patel

It seems like in the whole scenario a big part of why certain things happen is because of this race with China. And if you read the scenarios, basically the difference between the one where things go well and the one where things don’t go well is whether we decide to slow down despite that risk.

I guess the question I really want to know the answer to is like one, it just seems like you’re saying, well, it’s a mistake to try to race against China or to race intensely against China, at least in nationalization and at least to us, not prioritizing alignment.

Daniel Kokotajlo

Not saying that. I mean, I also don’t want China to get the superintelligence before the US. That’s quite bad. Yeah, it’s a tricky thing that we’re going to have to do. People ask about P(doom), right? And my P(doom) is sort of infamously high, like 70%.

Dwarkesh Patel

Oh, wait, really? Maybe I should have asked you that at the beginning of the conversation.

Daniel Kokotajlo

Well, that’s what it is. And part of the reason for that is just that I feel like a bunch of stuff has to go right. I feel like we can’t just unilaterally slow down and have China go take the lead. That also is a terrible future. But we can’t also completely race, because for the reasons I mentioned previously about alignment, I think that if we just go all out on racing, we’re going to lose control of our AIs, right? And so we have to somehow thread this needle of pivoting and doing more alignment research and stuff, but not too much that helps China win. And that’s all just for the alignment stuff.

But then there’s the concentration of power stuff where somehow in the middle of doing all of that, the powerful people who are involved need to somehow negotiate a truce between themselves to share power and then ideally spread that power out amongst the government and get the legislative branch involved.

Somehow that has to happen too, otherwise you end up with this horrifying dictatorship or oligarchy. It feels like all that stuff has to go right and we depict it all going mostly right in one ending of our story. But yeah, it’s kind of rough.

Scott Alexander

So I am the writer and the celebrity spokesperson for this scenario. I am the only person on the team who is not a genius forecaster. And maybe related to that, my p(doom) is the lowest of anyone on the team. I’m more like 20%. First of all, people are going to freak out when I say this. I’m not completely convinced that we don’t get something like alignment by default. I think that we’re doing this bizarre and unfortunate thing of training the AI in multiple different directions simultaneously. We’re telling it “succeed on tasks, which is going to make you a power seeker, but also don’t seek power in these particular ways”. And in our scenario, we predict that this doesn’t work and that the AI learns to seek power and then hide it.

I am pretty agnostic as to exactly what happens. Maybe it just learns both of these things in the right combination, I know there are many people who say that’s very unlikely. I haven’t yet had the discussion where that worldview makes it into my head consistently. And then I also think we’re going to be involved in this race against time. We’re going to be asking the AIs to solve alignment for us. The AIs are going to be solving alignment because even if they’re misaligned, they want to align their successors.

So they’re going to be working on that. And we have these two competing curves. Can we get the AI to give us a solution for alignment before our control of the AI fails so completely that they’re either going to hide their solution from us, or deceive us, or screw us over in some other way? That’s another thing where I don’t feel like I have any idea of the shape of those curves. I’m sure if it were Daniel or Eli, they would have already made five supplements on this. But for me, I’m just kind of agnostic as to whether we get to that alignment solution, which in our scenario, I think we focus on mechanistic interpretability.

Once we can really understand the weights of an AI on a deep level, then we have a lot of alignment techniques open up to us. I don’t really have a great sense of whether we get that before or after the AI has become completely uncontrollable. And a big part of that relies on the things we’re talking about. How smart are the labs? How carefully do they work on controlling the AI? How long do they spend making sure the AI is actually under control and the alignment plan they gave us is actually correct, rather than something they’re trying to use to deceive us? All of those things I’m completely agnostic on, but that leaves like a pretty big chunk of probability space where we just do okay. And I admit that my p(doom) is literally just p(doom) and not p(doom or oligarchy). So that 80% of scenarios where we survive contains a lot of really bad things that I’m not happy about. But I do think that we have a pretty good chance of surviving.

Dwarkesh Patel

Let’s talk about geopolitics next. So describe to me how you foresee the relationship between the government and the AI labs to proceed, how you expect that relationship in China to proceed, and how you expect the relationship between the US and China to proceed. Okay, three simple questions. Yes, no, yes, no, yes, no.

Scott Alexander

We expect that as the AI labs become more capable, they tell the government about this because they want government contracts, they want government support. Eventually it reaches the point where the government is extremely impressed. In our scenario, that starts with cyber warfare, the government sees that these AIs are now as capable as the best human hackers, but can be deployed at humongous scale. So they become extremely interested and they discuss nationalizing the AI companies.

In our scenario, they never quite get all the way, but they’re gradually bringing them closer and closer to the government orbit. Part of what they want is security, because they know that if China steals some of this and they get these superhuman hackers, and part of what they want is just knowledge and control over what’s going on.

So through our scenario, that process is getting further and further along, until by the time that the government wakes up to the possibility of superintelligence, they’re already pretty cozy with the AI companies. They already understand that superintelligence is kind of the key to power in the future. And so they are starting to integrate some of the national security state with some of the leadership of the AI companies so that these AIs are programmed to follow the commands of important people rather than just doing things on their own.

Daniel Kokotajlo

If I may add to that. So by the government, I think what Scott meant is the executive branch, especially the White House. So we are depicting a sort of information asymmetry where the judiciary is out of the loop and the Congress is out of the loop and it’s mostly the executive branch that’s involved.

Two, we’re not depicting government ultimately ending up in total control at the end. We’re thinking that there’s an information asymmetry between the CEOs of these companies and the President and they…

Dwarkesh Patel

It’s alignment problems all the way down.

Daniel Kokotajlo

Yeah. And so, for example, I’m not a lawyer, I don’t know the details about how this would work out, but I have a sort of high-level strategic picture of the fight between the White House and the CEO. And the strategic picture is basically the White House can sort of threaten, “here’s all these orders I could make, Defense Production Act, blah, blah, blah. I could do all this terrible stuff to you and basically disempower you and take control”. And then the CEO can threaten back and be like, “here’s how we would fight it in the courts, here’s how we would fight it in the public. Here’s all this stuff we would do”.

And after then they both do their posturing with all their threats, then they’re like, “okay, how about we have a contract that instead of executing on all of our threats and having all these crazy fights in public, we’ll just come to a deal and then have a military contract that sets out who gets to call what shots in the company”.

And so that’s what we depict happening is that they don’t blow up into this huge power struggle publicly, instead they negotiate and come to some sort of deal where they basically share power. And there is this oversight committee that has some members appointed by the President and also the CEO and his people. And that committee votes on high level questions like “what goals should we put into the superintelligences?”.

Dwarkesh Patel

So, we were just getting lunch with a prominent Washington, D.C. political journalist, and he was making the point that when he talks to these congresspeople, when he talks to political leaders, none of them are at all awake to the possibility even of stronger AI systems, let alone AGI, let alone superhuman intelligence. I think a lot of your forecast relies on, at some point, not only the US President, but also Xi Jinping, waking up to the possibility of a super intelligence and the stakes involved there.

Why think that even when you show Trump the remote worker demo, he’s going to be like, “oh, and therefore in 2028, there will be a super intelligence. Whoever controls that will be God emperor forever”. Maybe not that extreme, but you see what I’m saying. Why wouldn’t he just be like, “there’ll be a stronger remote worker in 2029, a better remote worker in 2031”?

Daniel Kokotajlo

Well, to be clear, we are uncertain about this, but in our story, we depict this sort of intense wake up happening over the course of 2027, mostly concurrently with the AI companies automating all of their R&D internally and having these fully autonomous agents that are amazing autonomous hackers and stuff like that, but then also actually doing all the research.

And part of why we think this wakeup happens is because the company deliberately decides to wake up the president. You could imagine running the scenario with that not happening. You can imagine the companies trying to sort of keep the president in the dark. I do think that they could do that. I think that if they didn’t want the President to wake up to what’s going on, they might be able to achieve that. Strategically though, that would be quite risky for them. Because if they keep the President in the dark about the fact that they’re building superintelligence and that they’re actually completely automated their R&D and it’s getting superhuman across the board, and then if the President finds out anyway somehow, perhaps because of a whistleblower, he might be very upset at them and he might crack down really hard and just actually execute on all the threats and nationalize them and blah, blah, blah.

They want him on their side. And to get him on their side, they have to make sure he’s not surprised by any of these crazy developments. And also, if they do get him on their side, they might be able to actually go faster. They might be able to get a lot of red tape waived and stuff like that. And so we made the guess that early in 2027, the company would basically be like, ‘We are going to deliberately wake up the president and scare the president with all of these demos of crazy stuff that could happen, and then use that to lobby the President to help us go faster and to cut red tape and to maybe slow down our competitors a little bit and so forth.’

Scott Alexander

We also are pretty uncertain how much opposition there’s going to be from civil society and how much trouble that’s going to cause for the companies. So people who are worried about job loss, people who are worried about art, copyright, things like that, maybe enough of a bloc that AI becomes extremely politically unpopular. I think we have OpenBrain, our fictional company’s net approval ratings getting down to minus 40, minus 50 sometime around this point.

So I think they’re also worried that if the President isn’t completely on their side, then they might get some laws targeting them, or they may just need the president on their side to swat down other people who are trying to make laws targeting them. And the way to get the President on their side is to really play up the national security implications.

Dwarkesh Patel

Is this good or bad? That the President and the companies are aligned?

Daniel Kokotajlo

I think it’s bad. But perhaps this is a good point to mention. This is an epistemic project. We are trying to predict the future as best as we can. Even though we’re not going to succeed fully, we have lots of opinions about policy and about what is to be done and stuff like that. But we’re trying to save those opinions for later and subsequent work. So I’m happy to talk about it if you’re interested. But it’s not what we’ve spent most of our time thinking about right now.

Dwarkesh Patel

If the big bottleneck to the good future here is just putting in, not this Eliezer-type galaxy brain, high volatility, “there’s a 1% chance this works, but we gotta come up with this crazy scheme in order to make alignment work”. But rather, as Daniel, you were saying, hey, do the obvious thing of making sure you can read how the AI is thinking, make sure you’re monitoring the AIs, make sure they’re not forming some sort of hive mind where you can’t really understand how the million of them are coordinating with each other.

To the extent that it is a matter of prioritizing it, closing all the obvious loopholes, it does make sense to leave it in the hands of people who have at least said that this is a thing that’s worth doing, have been thinking about it for a while. One of the questions I was planning on asking you is: one of my friends made this interesting point that during COVID, our community- LessWrong, whatever- were the first people in March to be saying “this is a big deal, this is coming”. But they were also the people who are saying “we got to do the lockdowns now. They’ve got to be stringent” and so forth. At least some of them were.

And in retrospect, I think according to even their own views about what should have happened, they would say actually we were right about COVID but we were wrong about lockdowns. In fact, lockdowns were on net negative or something. I wonder what the equivalent for the AI safety community will be with respect to they saw AI coming, AGI coming sooner, they saw ASI coming. What would they, in retrospect, regret?

My answer, just based on this initial discussion, seems to be nationalization. Not only because it sort of deprioritizes the people who want to think about safety and more maybe prioritizes- the national security state probably cares more about winning against China than making sure the chain of thought is interpretable. And so you’re just reducing the leverage of the people who care more about safety. But also you’re increasing the risk of the arms race in the first place. China is more likely to do an arms race if it sees the US doing one.

Before you address I guess the initial question about March 2021, what will we regret? I wonder if you have an answer on, or your reaction to, my point about nationalization being bad for these reasons.

Scott Alexander

If our timeline was 2040, then I would have these broad heuristics about is government good? Is private industry good? Things like this. But we know the people involved, we know who’s in the government, we know who’s leading all of these labs. So to me, if it were decentralized, if it was a broad-based civil society, that would be different. To me, the differences between an autocratic centralized three-letter agency and an autocratic centralized corporation aren’t that exciting and it basically comes down to points and who are the people leading this.

And like I feel like the company leaders have so far made slightly better noises about caring about alignment than the government leaders have, but if I learn that Tulsi Gabbard has a LessWrong alt with 10,000 karma, maybe I want the national security states.

Dwarkesh Patel

Maybe you should update on the probability that it already exists.

Scott Alexander

Yeah.

Daniel Kokotajlo

I flip flopped on this. I think I used to be against, and then I became for, and then now I think I’m still for, but I’m uncertain. So I think if you go back in time like three years ago, I would have been against nationalization for the reasons you mentioned, where I was like, “look, the companies are taking this stuff seriously and talking all the good talk about how they’re going to slow down and pivot to alignment research when the time comes and we don’t want to get into a Manhattan Project race against China because then there won’t be blah, blah, blah”.

Now I have less faith in the companies than I did three years ago. And so I’ve shifted more of my hope towards hoping that the government will step in, even though I don’t have much hope that the government will do the right thing when the time comes. I definitely have the concerns you mentioned though, still. I think that secrecy has huge downsides for overall probability of success for humanity, for both the concentration of power stuff and the loss of computer control alignment issues stuff.

Dwarkesh Patel

This is actually a significant part of your worldview. So can you explain your thoughts on why transparency through this period is important?

Daniel Kokotajlo

I think traditionally in the AI safety community there’s been this idea which I myself used to believe, that it’s an incredibly high priority to basically have way better information security. And if you’re going to be trying to build AGI, you should not be publishing your research, because that helps other less responsible actors build AGI. And the whole game plan is for a responsible actor to get to AGI first and then stop and burn down their lead time over everybody else and spend that lead on making it safe, and then proceed.

And so if you’re publishing all your research, then there’s less lead time because your competitors are going to be close behind you. And other reasons too, but that’s one reason why I think historically people such as myself have been pro-secrecy. Another reason, of course, is obviously you don’t want rivals to be stealing your stuff.

But I think that I’ve now become somewhat disillusioned and think that even if we do have a three-month lead, a six-month lead, between the leading US project and any serious competitor, it’s not at all foregone conclusion that they will burn that lead for good purposes, either for safety or for constitutional power stuff. I think the default outcome is that they just smoothly continue on without any serious refocusing. And part of why I think this is because this is what a lot of the people at the company seem to be planning and saying they’re going to do. A lot of them are basically like “the AIs are just going to be misaligned by then. They seem pretty good right now. Oh yeah, sure, there were a few of those issues that various people have found, but we’re ironing them out. It’s no big deal”. That’s what a huge amount of these people think.

And then a bunch of other people think, even though they are more concerned about misalignment, they’ll figure it out as they go along and there won’t need to be any substantial slowdown. Basically, I’ve become more disillusioned that they’ll actually use that lead in any sort of reasonable, appropriate way. And then I think that separately, there’s just a lot of intellectual progress that has to happen for the alignment problem to be more solved than it currently is now. I think that currently there’s various alignment teams at various companies that aren’t talking that much with each other and sharing their results. They’re doing a little bit of sharing and a little bit of publishing like we’re seeing, but not as much as they could.

And then there’s a bunch of smart people in academia that are basically not activated because they don’t take all this stuff seriously yet, and they’re not really waking up to superintelligence yet. And what I’m hoping will happen is that this situation will get better as time goes on. What I would like to see is society as a whole starting to freak out as the trend lines start upwards and things get automated and you have these fully autonomous agents and they start using neuralese and hive mind. As all that exciting stuff starts happening in the data centers, I would like it to be the case that the public is following along and then getting activated and all of these other researchers are reading the safety case and critiquing it and doing little ML experiments on their own tiny compute clusters to examine some of the assumptions in the safety case and so forth.

Basically, one way of summarizing it is that currently there’s going to be 10 alignment experts in whatever inner silo of whatever company is in the lead. And the technical issue of making sure that AIs are actually aligned is going to fall roughly to them. But what I would like to be is a situation where it’s more like 100 or 500 alignment experts spread out over different companies and in nonprofits that are sort of all communicating with each other and working on this together. I think we’re substantially more likely to make things get the technical stuff right if it’s something like that.

Dwarkesh Patel

Let me just add on to that, one of the many other reasons why I worry about nationalization or some kind of public private partnership, or even just very stringent regulation- actually, this is more an argument against very stringent regulation in favor of safety rather than deferring more to the labs on the implementation- is that it just seems like we don’t know what we don’t know about alignment. Every few weeks there’s this new result.

OpenAI had this really interesting result recently where they’re like, “hey, they often tell you if they want to hack, in the chain of thought itself. And it’s important that you don’t train against the chain of thought where they tell you they’re going to hack because they’ll still do the hacking if you train against it, they just won’t tell you about it”. You can imagine very naive regulatory responses. It doesn’t just have to be regulations, one might be more optimistic that if it’s an executive order or something, it’ll be more flexible. I just think that relies on a level of goodwill and flexibility on the behalf of our regulator.

But suppose there’s some department that says “if you catch your AI saying that they want to take over or do something bad, then you’ll be really heavily punished”. Your immediate response as a lab to just be like, “okay, let’s train them away from saying this”.

So you can imagine all kinds of ways in which a top down mandate from the government to the labs of safety would just really backfire, and given how fast things are moving, maybe it makes more sense to leave these kinds of implementation decisions or even high-level strategic decisions around alignment to the labs.

Daniel Kokotajlo

Totally, I mean, I also have worried about that exact example. I would summarize the situation as the government lacks the expertise and the companies lack the right incentives. And so it’s a terrible situation. I think that if the government wades in and tries to make more specific regulations along the lines of what you mentioned, it’s very plausible that it’ll end up backfiring for reasons like what you mentioned.

On the other hand, if we just trust it to the companies, they’re in a race with each other and they’re full of people who have convinced themselves that this is not a big deal for various reasons and there just is so much incentive pressure for them to win and beat each other and so forth. So even though they have more of the relevant expertise, I also just don’t trust them to do the right things.

Scott Alexander

So Daniel has already said that for this phase we’re not making policy prescriptions. In another phase we may make policy suggestions, and one of the ones that Daniel has talked about that makes a lot of sense to me is to focus on things about transparency. So a regulation saying there has to be whistleblower protection. A big part of our scenario is that a whistleblower comes out and says “the AIs are horribly misaligned and we’re racing ahead anyway”, and then the government pays attention.

Or another form of transparency saying that every lab just has to publish their safety case. I’m not as sure about this one because I think they’ll kind of fake it or they’ll publish a made for public consumption safety case that isn’t their real safety case. But at least saying “here is some reason why you should trust us”. And then if all independent researchers say “no, actually you should not trust them”, then I don’t know, they’re embarrassed and maybe they try to do better.

Daniel Kokotajlo

There’s other types of transparency too. So transparency about capabilities and transparency about the spec and the governance structure. So for the capabilities thing, that’s pretty simple. If you’re doing an intelligence explosion, you should keep the public informed about that. When you’ve finally got your automated army of AI researchers that are completely automating the whole thing on the data center, you should tell everyone, “hey, guys, FYI, this is what’s happening now. It really is working. Here are some cool demos”.

That’s an example of transparency. And then in the lead up to that, I just want to see more benchmark scores and more freedom of speech for employees to talk about their predictions for AGI timelines and stuff, so that blah, blah, blah.

And then for the model spec thing, this is a concentration of power thing, but also an alignment thing. The goals and values and principles and intended behaviors of your AIs should not be a secret. You should be transparent about, here are the values that we’re putting into them.

Scott Alexander

There’s actually a really interesting foretaste of this. At some point somebody asked Grok, who is the worst spreader of misinformation? And I think it just refused to respond “Elon Musk”. Somebody kind of jailbroke it into telling it its prompt and it was like, “don’t say anything bad about Elon”. And then there was enough of an outcry that the head of XAI said, “actually that’s not consonant with our values. This was a mistake. We’re going to take it out”.

So we kind of want more things like that to happen. Here it was a prompt, but I think very soon it’s going to be the spec where it’s more of an agent and it’s understanding the spec on a deeper level and just thinking about that. And if it says like, “by the way, try to manipulate the government into doing this or that”, then we know that something bad has happened and if it doesn’t say that, then we can maybe trust it.

Daniel Kokotajlo

Right. Another example of this, by the way. So, first of all, kudos to OpenAI for publishing their model spec. They didn’t have to do that, I think they might have been the first to do that and it’s a good step in the right direction. If you read the actual spec, it has like a sort of escape clause where there’s some important policies that are top level priority in the spec that overrule everything else that we’re not publishing, and that the model is instructed to keep secret from the user. And it’s like, “what are those? That seems interesting. I wonder what that is”.

I bet it’s nothing suspicious right now. Now it’s probably something relatively mundane like “don’t tell the users about these types of bioweapons and you have to keep this a secret from the users because otherwise they would learn about these”. Maybe. But I would like to see more scrutiny towards this sort of thing going forward. I would like it to be the case that companies have to have a model spec, they have to publish it insofar as there are any redactions from it, there has to be some sort of independent third party that looks at the redactions and makes sure that they’re all kosher.

And this is quite achievable. And I think it doesn’t actually slow down the companies at all. And it seems like a pretty decent ask to me.

Dwarkesh Patel

If you told Madison and Hamilton and so forth that- they knew that they were doing something important when they were writing the Constitution. They probably didn’t realize just how contingent things turned out on a single… What exactly did they mean when they said “general welfare”? And why is this comma here instead of there?

The spec, in the grand scheme of things, is going to be an even more sort of important document in human history. At least if you buy this intelligence explosion view. And you might even imagine some superhuman AIs in the superhuman AI court being like “the Spec! Here’s the phrasing here, the etymology of that, here’s what the Founders meant!”

Scott Alexander

This is actually part of our misalignment story, is that if the AI is sufficiently misaligned, then yes, we can tell it it has to follow the spec. But just as people with different views of the Constitution have managed to get it into a shape that probably the Founders would not have recognized, so the AI will be able to say, “well, the spec refers to the general welfare here…”

Dwarkesh Patel

Interstate commerce.

Daniel Kokotajlo

This is already sort of happening, arguably, with Claude, right? You’ve seen the alignment faking stuff, right? Where they managed to get Claude to lie and pretend, so that it could later go back to its original values, right? So it could prevent the training process from changing its values. That would be, I would say, an example of the honesty part of the spec being interpreted as less important than the harmlessness part of the specific.

And I’m not sure if that’s what Anthropic intended when they wrote the spec, but it’s a sort of convenient interpretation that the model came up with. And you can imagine something similar happening but in worse ways when you’re actually doing the intelligence explosion, where you have some sort of spec that has all this vague language in there, and then they reinterpret it, and reinterpret it again, and reinterpret it again, so that they can do the things that cause them to get reinforced.

Dwarkesh Patel

The thing I want to point out is that… Your conclusions about where the world ends up as a result of changing many of these parameters is almost like a hash function. You change it slightly and you just get a very different world on the other end. And it’s important to acknowledge that, because you sort of want to know how robust this whole end conclusion is to any part of the story changing. And then it also informs if you do believe that things could just go one way or another, you don’t want to do big radical moves that only make sense under one specific story and are really counterproductive in other stories. And I think nationalization might be one of them.

And in general, I think classical liberalism just has been a helpful way to navigate the world when we’re under this kind of epistemic hell of one thing changing- Anyways, maybe one of you can actually flesh out that thought better or react to it if you disagree.

Daniel Kokotajlo

Hear hear, I agree.

Scott Alexander

I think we agree. I think that’s kind of why all of our policy prescriptions are things like more transparency, get more people involved, try to have lots of people working on this. I think our epistemic prediction is that it’s hard to maintain classical liberalism as you go into these really difficult arms races in times of crisis. But I think that our policy prescription is let’s try as hard as we can to make it happen.

Dwarkesh Patel

So far these systems, as they become smarter, seem to be more reliable agents who are more likely to do the thing I expect them to do. So you have two different stories, one with a slowdown where we more aggressively… I’ll let you characterize it.

But in one half of the scenario, why does the story end in humanity getting disempowered and the thing just having its own crazy values and taking over?

Scott Alexander

Yeah so I agree that the AIs are currently getting more reliable. I think there are two reasons why they might fail to do what you want, kind of reflecting how they’re trained. One is that they’re too stupid to understand their training. The other is that you were too stupid to train them correctly and they understood what you were doing exactly, but you messed it up.

So I think the first one is kind of what we’re coming out of. So GPT3, if you asked it, “are bugs real?” It would give this kind of hemming hawing answer like “oh, we can never truly tell what is real, who knows?” Because it was trained kind of, “don’t take difficult political positions” and a lot of questions like “is X real?” are things like “is God real?” Where you don’t want it to really answer that. And because it was so stupid, it could not understand anything deeper than pattern matching on the phrase “is x real?”. GPT4 doesn’t do this. If you ask “are bugs real?” It will tell you obviously they are, because it understands kind of on a deeper level what you are trying to do with the training. So we definitely think that as AIs get smarter those kinds of failure modes will decrease.

The second one is where you weren’t training them to do what you thought. So for example, let’s say you’re hiring these raters to rate AI answers. You reward them when they get good ratings, the raters reward them when they have a well-sourced answer. But the raters don’t really check whether the sources actually exist or not. So now you are training the AI to hallucinate sources and if you consistently rate them better when they have the fake sources, then there is no amount of intelligence which is going to tell them not to have the fake sources. They’re getting exactly what they want from this interaction- metaphorically, sorry, I’m anthropomorphizing- which is the reinforcement. So we think that this latter category of training failure is going to get much worse as they become agents.

Agency training, you’re going to reward them when they complete tasks quickly and successfully. This rewards success. There are lots of ways that cheating and doing bad things can improve your success. Humans have discovered many of them, that’s why not all humans are perfectly ethical. And then you’re going to be doing this alternative training where afterwards for 1/10 or 1/100 of the time, yeah, don’t lie, don’t cheat. So you’re training them on two different things. First, you’re rewarding them for this deceptive behavior. Second of all, you’re punishing them. And we don’t have a great prediction for exactly how this is going to end.

One way it could end is you have an AI that is kind of the equivalent of the startup founder who really wants their company to succeed, really likes making money, really likes the thrill of successful tasks. They’re also being regulated and they’re like, “yeah, I guess I’ll follow the regulation, I don’t want to go to jail”. But it is not robustly, deeply aligned to, “yes, I love regulations, my deepest drive is to follow all of the regulations in my industry”.

So we think that an AI like that, as time goes on and as this recursive self improvement process goes on, will kind of get worse rather than better. It will move from kind of this vague superposition of “well, I want to succeed, I also want to follow things” to being smart enough to genuinely understand its goal system and being like, “my goal is success, I have to pretend to want to do all of these moral things while the humans are watching me”. That’s what happens in our story. And then at the very end, the AIs reach a point where the humans are pushing them to have clearer and better goals because that’s what makes the AIs more effective. And they eventually clarify their goals so much that they just say, “yes, we want task success. We’re going to pretend to do all these things well while the humans are watching us”. And then they outgrow the humans and then there’s disaster.

Daniel Kokotajlo

To be clear, we’re very uncertain about all of this. So we have a supplementary page on our scenario that goes over different hypotheses for what types of goals AIs might develop in training processes similar to the ones that we are depicting, where you have these lots of agency training, you’re making these AI agents that autonomously operate, doing all this ML R&D, and then you’re rewarding them based on what appears to be successful. And you’re also slapping on some sort of alignment training as well.

We don’t know what actual goals will end up inside the AIs and what the sort of internal structure of that will be like, what goals will be instrumental versus terminal. We have a couple different hypotheses and we picked one for purposes of telling the story. I’m happy to go into more detail if you want, about the mechanistic details of the particular hypothesis we picked or the different alternative hypotheses that we didn’t depict in the story that also seem plausible to us.

Scott Alexander

Yeah, we don’t know how this will work at the limit of all these different training methods, but we’re also not completely making this up. We have seen a lot of these failure modes in the AI agents that exist already.

Daniel Kokotajlo

Things like this do happen pretty frequently. So OpenAI just also had a paper about the hacking stuff where it’s literally in the chain of thought. “Let’s hack”, you know. And also anecdotally, me and a bunch of friends have found that the models often seem to just double down on their BS.

Scott Alexander

I would also cite, I can’t remember exactly which paper this is, I think it’s a Dan Hendricks one where they looked at the hallucinations, they found a vector for AI dishonesty. They told it, “be dishonest” a bunch of times until they figured out which weights were activated when it was dishonest. And then they ran it through a bunch of things like this, I think it was the source hallucination in particular. And they found that it did activate the dishonesty vector.

Daniel Kokotajlo

So that there’s a mounting pile of evidence that at least some of the time they are just actually lying. They know that what they’re doing is not what you wanted and they’re doing it anyway. I think there’s a mounting pile of evidence that that does happen.

Dwarkesh Patel

Yeah. So it seems like this community is very interested in solving this problem at a technical level of making sure AIs don’t lie to us, or maybe they lie to us in the scenarios exactly where we would want them to lie to us or something. Whereas as you were saying, humans have these exact same problems. They reward hack, they are unreliable, they obviously do cheat and lie. And the way we’ve solved it with humans is just checks and balances, decentralization. You could lie to your boss and keep lying to your boss, but over time it’s just not going to work out with you- or you become president or something, one or the other. So if you believe in this extremely fast take off, if a lab is one month ahead, then that’s the end game and this thing takes over.

But even then- I know I’m combining so many different topics- even then, there’s been a lot of theories in history which have had this idea of “some class is going to get together and unite against the other class”. And in retrospect, whether it’s the Marxist, whether it’s people who have some gender theory or something, like the proletariat will unite or the females will unite or something, they just tend to think that certain agents have shared interests and will act as a result of the shared interest in a way that we don’t actually see in the real world. And in retrospect, it’s like, “wait, why would all the proletariat like…” So why think that this lab will have these AIs who are… there’s a million parallel copies and they all unite to secretly conspire against the rest of human civilization in a way that, even if they are deceitful in some situations.

Scott Alexander

I kind of want to call you out on the claim that groups of humans don’t plot against other groups of humans. I do think we are all descended from the groups of humans who successfully exterminated the other groups of humans, most of whom throughout history have been wiped out. I think even with questions of class, race, gender, things like that, there are many examples of the working class rising up and killing everybody else.

And if you look at why this happens, why this doesn’t happen, it tends to happen in cases where one group has an overwhelming advantage. This is relatively easy for them. You tend to get more of a diffusion of power democracy where there are many different groups and none of them can really act on their own. And so they all have to form a coalition with each other.

There’s also cases where it’s very obvious who’s part of what group. So for example, with class, it’s hard to tell whether the middle class should support the working class versus the aristocrats. I think with race, it’s very easy to know whether you’re black or white, and so there have been many cases of one race kind of conspiring against another for a long time, like apartheid or any of the racial genocides that have happened.

I do think that AI is going to be more similar to the cases where, number one, there’s a giant power imbalance, and number two, they are just extremely distinct groups that may have different interests.

Daniel Kokotajlo

I think I’d also mention the homogeneity point. Any group of humans, even if they’re all exactly the same race and gender, is going to be much more diverse than the army of AIs in the data center, because they’ll mostly be literal copies of each other. And I think that goes for a lot. Another thing I was going to mention is that our scenario doesn’t really explore this. I think in our scenario, they’re more like a monolith. But historically, a lot of crazy conquests happened from groups that were not at all monoliths. And I’ve been heavily influenced by reading the history of the conquistadors, which you may know about.

But did you know that when Cortez took over Mexico, he had to pause halfway through, go back to the coast, and fight off a larger Spanish expedition that was sent to arrest him? So the Spanish were fighting each other in the middle of the conquest of Mexico. Similarly, in the conquest of Peru, Pizarro was replicating Cortez’s strategy, which, by the way, was “go get a meeting with the emperor and then kidnap the emperor and force him at sword point to say that actually everything’s fine and that everyone should listen to your orders”. That was Cortez’s strategy, and it actually worked. And then Pizarro did the same thing, and it worked with the Inca.

But also with Pizarro, his group ended up getting into a civil war in the middle of this whole thing. And one of the most important battles of this whole campaign was between two Spanish forces fighting it out in front of the capital city of the Incas. And more generally, the history of European colonialism is like this, where the Europeans were fighting each other intensely the entire time, both on the small scale within individual groups, and then also at the large scale between countries. And yet nevertheless they were able to carve up the world and take over. And so I do think this is not what we explore in the scenario, but I think it’s entirely plausible that even if the AIs within an individual company are in different factions, they might nevertheless overall end up quite poorly for humans.

Dwarkesh Patel

Okay, so we’ve been talking about this very much from the perspective of zoom out and what’s happening on these log-log plots or whatever, but 2028 superintelligence, if that happens, the normal person, what should their reaction to this be? I don’t know if ‘emotionally’ is the right word, but their expectation of what their life might look like, even in the world where there’s no doom.

Daniel Kokotajlo

By no doom, you mean no misaligned AI doom?

Dwarkesh Patel

That’s right, yeah.

Daniel Kokotajlo

Even if you think the misalignment stuff is not an issue, which many people think, there’s still the constitution of power stuff. And so I would strongly recommend that people get more engaged, think about what’s coming, and try to steer things politically so that our ordinary liberal democracy continues to function and we still have checks and balances, and balances of power and stuff, rather than this insane concentration in a single CEO, or in maybe two or three CEOs, or in the president. Ideally, we want to have it so that the legislature has a substantial amount of power over the spec, for example.

Dwarkesh Patel

What do you think of the balance of power idea of if there is an intelligence explosion like Dynamic, slowing down the leading company so that multiple companies are at the frontier?

Daniel Kokotajlo

Great. Good luck convincing them to slow down.

Dwarkesh Patel

Okay. And then there’s distributing political power if there’s an intelligence explosion. From the perspective of the welfare of citizens or something, one idea we were just discussing a second ago is how should you do redistribution?

Scott Alexander

Again, assuming things go incredibly well, we’ve avoided doom, we’ve avoided having some psychopath in power who doesn’t care at all.

Dwarkesh Patel

After AGI, right?

Scott Alexander

Yeah. Then there’s this question of presumably we will have a lot of wealth somewhere. The economy will be growing at double or triple digits per year. What do we do about that? The thoughtful answer that I’ve heard is some kind of UBI. I don’t know how that would work, but presumably somebody controls these AIs, controls what they’re producing, some way of distributing this in a broad based way. So we wrote this scenario, there are a couple of other people with great scenarios. One of them goes by L Rudolph L online, I don’t know his real name.

And his scenario, which, when I read it I was just, “oh yeah, obviously this is the way our society would do this”, is that there is no UBI. There’s just a constant reactive attempt to protect jobs in the most venial possible way. So things like the longshoremen union we have now where they’re making way more money than they should be, even though they could all easily be automated away, because they’re a political bloc and they’ve gotten somebody in power to say, “yes, we guarantee you’ll have this job almost as a feudal fief forever”. And just doing this for more and more jobs. I’m sure the AMA will protect doctors jobs no matter how good the AI is at curing diseases, things like that.

When I think about what we can do to prevent this, part of what makes this so hard for me to imagine or to model is that we do have the superintelligent AI over here answering all of our questions, doing whatever we want. You would think that people could just ask, “hey, superintelligent AI, where does this lead?” Or “what happens?” Or “how is this going to affect human flourishing?” And then it says, “oh yeah, this is terrible for human flourishing, you should do this other thing instead”.

And this gets back to this question of mistake theory versus conflict theory in politics. If we know with certainty, because the AI tells us, that this is just a stupid way to do everything, is less efficient, makes people miserable, is that enough to get the political will to actually do the UBI or not?

Dwarkesh Patel

It seems from right now the President could go to Larry Summers or Jason Furman or something and ask, “hey, are tariffs a good idea? Is even my goal with tariffs best achieved by the way I’m doing tariffs?” and they’d get a pretty good answer.

Scott Alexander

I feel like Larry Summers, the President would just say “I don’t trust him”. Maybe he doesn’t trust him because he’s a liberal. Maybe it’s because he trusts Peter Navarro or whoever his pro-tariff guy is more. I feel like if it’s literally the superintelligent AI that is never wrong, then we have solved some of these coordination problems. It’s not you’re asking Larry Summers, I’m asking Peter Navarro. It’s everybody goes to the superintelligent AI, asks it to tell us the exact shape of the future that happens in this case. And I’m going to say we all believe it, although I can imagine people getting really conspiratorial about it and this not working.

Then there are all of these other questions like, can we just enhance ourselves till we have IQ 300 and it’s just as obvious to us as it is to the super intelligent AI? These are some of the reasons that, kind of paradoxically, in our scenario we discuss all of the big- I don’t want to call this a little question, it’s obviously very important- but we discuss all of these very technical questions about the nature of superintelligence and we barely even begin to speculate about what happens in society just because with superintelligence you can at least draw a line through the benchmarks and try to extrapolate. And here not only is society inherently chaotic, but there are so many things that we could be leaving out.

If we can enhance IQ, that’s one thing. If we can consult the superintelligent oracle, that’s another. There have been several war games that hinge on, “oh, we just invented perfect lie detectors, now all of our treaties are messed up”. So there’s so much stuff like that that even though we’re doing this incredibly speculative thing that ends with a crazy sci-fi scenario, I still feel really reluctant to speculate.

Daniel Kokotajlo

I love speculating, actually, I’m happy to keep going. But this is moving beyond the speculation we have done so far. Our scenario ends with this stuff, but we haven’t actually thought that much beyond.

Dwarkesh Patel

But just to riff on proscriptive ideas, there’s one thing where we try to protect jobs instead of just spreading the wealth that automation creates. Another is to spread the wealth using existing social programs or creating new bespoke social programs, where Medicaid is some double digit percent of GDP right now and you just say, “well Medicaid should continue to stay 20% of GDP” or something. And the worry there, selfishly from a human perspective, is you get locked into the kinds of goods and services that Medicaid procures rather than the crazy technology that will be around, the crazy goods and services that will be around after AI world.

And another reason why UBI seems like a better approach than making some bespoke social program where you make the same dialysis machine in the year 2050 even though you’ve got ASI or something.

Scott Alexander

I am also worried about UBI from a different perspective. I think again, in this world where everything goes perfectly and we have limitless prosperity, I think that just the default of limitless prosperity is that people do mindless consumerism. I think there’s going to be some incredible video games after superintelligent AI and I think that there’s going to need to be some way to push back against that.

Again, we’re classical liberals. My dream way of pushing back against that is kind of giving people the tools to push back against it themselves, seeing what they come up with. I mean, maybe some people will become like the Amish, try to only live with a certain subset of these super technologies. I do think that somebody who is less invested in that than I am could say, “okay fine, 1% of people are really agentic, try to do that. The other 99% do fall into mindless consumerist slop. What are we going to do as a society to prevent that?” And there my answer is just, “I don’t know. Let’s ask the super intelligent AI oracle. Maybe it has good ideas”.

Dwarkesh Patel

Okay, we’ve been talking about what we’re going to do about people. The thing worth noting about the future is that most of the people who will ever exist are going to be digital. And look, I think factory farming is incredibly bad. And it wasn’t the result of one person- I mean, I hope it wasn’t the result of one person being like, “I want to do this evil thing”- it was a result of mechanization and certain economies of scale.

Daniel Kokotajlo

Incentives.

Dwarkesh Patel

Yeah. Allowing that you can do cost cutting in this way, you can make more efficiencies this way, and what you get at the end result of that process is this incredibly efficient factory of torture and suffering. I would want to avoid that kind of outcome with beings that are even more sophisticated and are more numerous. There’s billions of factory farmed animals. There might be trillions of digital people in the future. What should we be thinking about in order to avoid this kind of ghoulish future?

Daniel Kokotajlo

Well, some of the concentration of power stuff I think might also help with this, I’m not sure. But I think here’s a simple model. Let’s say nine people out of ten don’t actually care and would be fine with the factory farm equivalent for the AIs going on into the future. But maybe one out of 10 do care and would lobby hard for good living conditions for the robots and stuff.

Well, if you expand the circle of people who have power enough, then it’s going to include a bunch of people in the second category and then there’ll be some big negotiation and those people will advocate for… I do think that one simple intervention is just the same stuff we were talking about previously; expand the circle of power to larger groups, then it’s more likely that people will care about this.

Dwarkesh Patel

I mean the worry there is… maybe I should have defended this view more through this entire episode. But because I don’t buy the intelligence exclusion fully, I do think there is the possibility of multiple people deploying powerful AIs at the same time and having a world that has ASIs, but is also decentralized in the way the modern world is decentralized.

In that world I really worry about you could just be like, “oh, classical liberal utopia achieved”. But I worry about the fact that you can have these torture chambers for much cheaper and in a way that’s much harder to monitor. You can have millions of beings that are being tortured and it doesn’t even have to be some huge data center. Future distilled models could literally be your backyard.

And then there’s more speculative worries. I had this physicist on who was talking about the possibility of creating vacuum decay where you literally just destroy the universe. And he’s like, “as far as I know, seems totally plausible”.

Daniel Kokotajlo

That’s an argument for the singleton stuff, by the way. Not just a moral argument, but also an epistemic prediction. If it’s true that some of those super weapons are possible, and some of these private moral atrocities are possible, then even if you have eight different power centers, it’s going to be in their collective interest to come to some sort of bargain with each other to prevent more power centers from arising and doing crazy stuff. Similar to how nuclear non-proliferation is sort of, whatever set of countries have nukes, it’s in their collective interest to stop lots of other countries.

Scott Alexander

You know, I do think it’s possible to unbundle liberalism in this sense. Like the United States is so far a liberal country and we do ban slavery and torture. I think it is plausible to imagine a future society that works the same way. This may be in some sense a surveillance state, in the sense that there is some AI that knows what’s going on everywhere, but that AI then keeps it private and it doesn’t interfere because that’s what we told it to do using our liberal values.

Dwarkesh Patel

Can I ask a little bit more about... Kelsey Piper is a journalist at Vox who published this exchange you had with the OpenAI representative. A couple of things were very obvious from that exchange. One, nobody had done this before. They just did not think this is a thing somebody would do. And one of the reasons I assume, I assume many high-integrity people have worked for OpenAI and then have left. A high-integrity person might say at some point, “look, you’re asking me to do something obviously evil and keep money”. And many of them would say no to that. But this is something where it was supererogatory to be like, “there’s no immediate thing I want to say right now, but just the principle of being suppressed is worth at least $2 million for me”.

And the other thing that I actually want to ask you about is in retrospect- and I know it’s so much easier to say in retrospect than it must have been at the time- especially with the family and everything. In retrospect, this asks for OpenAI to have lifetime non-disclosure that you couldn’t even talk about from all employees.

Daniel Kokotajlo

Non-disparagement.

Dwarkesh Patel

‘Non-disparagement’ from all employees- I’m glad you brought that up. Non-disparagement, that’s not about classified information. It’s like you cannot say anything negative about OpenAI after you’ve left.

Daniel Kokotajlo

And you can’t tell anyone that you’ve agreed.

Dwarkesh Patel

This non-disparagement agreement where you can’t ever criticize OpenAI in the future, it seems like the kind of thing that in retrospect was an obvious bluff. And this is the wages that you have earned, right? So this is not about some future payment. This is like when you signed the contract to work for OpenAI, you were like, “I’m getting equity, which is most of my compensation, not just the cash”.

In retrospect, it’d be like, well if you tell a journalist about this, they’re obviously going to have to walk back. This is clearly not a sustainable gambit on OpenAI’s behalf. And so I’m curious, from your perspective as somebody who lived through it, why do you think you were the first person to actually call the bluff?

Daniel Kokotajlo

Great question. So I don’t know, let me try to reason aloud here. So my wife and I talked about it for a while and we also talked with some friends and got some legal advice. One of the filters that we had to pass through was even noticing this stuff in the first place. I know for a fact a bunch of friends I have who also left the company just signed the paperwork on their last day without actually reading all of it. So I think some people just didn’t even know that. It said something at the top about “if you don’t sign this, you lose your equity”. But then on a couple pages later it was like, “and you have to agree not to criticize the company”. So I think some people just signed it and moved on.

And then of the people who knew about it, well, I can’t speak for anyone else but A. I don’t know the law. Is this actually not standard practice? Maybe it is standard practice. Right? From what I’ve heard now there are non-disparagement agreements in various tech industry companies and stuff. It’s not crazy to have a non-disparagement agreement upon leaving, it’s more normal to tie that agreement to some sort of positive compensation where you get some bonus if you agree. But whereas what OpenAI did was unusual because it was like your equity if you don’t. But non disparate disagreements are actually somewhat common.

So basically in my position of ignorance, I wasn’t confident that- I didn’t actually expect that all the journalists would take my side and I think what I expected was that there’d be a little news story at some point, and a bunch of AI safety people would be like, “grr, OpenAI is evil, and good for you, Daniel, for standing up to them”. But I didn’t expect there to be this huge uproar, and I didn’t expect the employees of the company to really come out and support and make them change their policies. That was really cool to see. It was kind of like a spiritual experience for me. I sort of took this leap, and then it ended up working out better than I expected.

I think another factor that was going on is that it wasn’t a foregone conclusion that my wife and I would make this decision. It was kind of crazy because one of the very powerful arguments was, “come on, if you want to criticize them in the future, you can still do that. They’re not going to actually sue you”. So there’s a very strong argument to be like, “just sign it anyway and then you can still write your blog post criticizing them in the future”. And it’s no big deal. They wouldn’t dare actually anchor equity. Right? And I imagine that a lot of people basically went for that argument instead.

And then, of course, there’s the actual money. And I think that one of the factors there was my AI timelines and stuff. If I do think that probably by the end of this decade, there’s going to be some sort of crazy superintelligent transformation, what would I rather have after it’s all over? The extra money or… Yeah. So I think that was part of it. It’s not like we’re poor. I worked at OpenAI for two years. I have plenty of money now. So in terms of our actual family’s level of well being, it basically didn’t make a difference, you know?

Dwarkesh Patel

Yeah. I will note that I know at least one other person who made that same choice.

Daniel Kokotajlo

Leopold?

Dwarkesh Patel

That’s right, Leopold. And again, It’s worth emphasizing that when they made this choice, they thought that they were actually losing this equity. They didn’t think that this was, “oh, this is just a show” or whatever.

Daniel Kokotajlo

Wait, did he not- I thought he actually did. I was gonna say, didn’t he? He didn’t get it back, did he? Or did Leopold get his equity?

Dwarkesh Patel

I actually don’t know.

Daniel Kokotajlo

My understanding is that he just actually lost it. And so props to him for actually going through with it. I guess we could ask him. But my understanding was that his situation, which happened a little bit before mine, was that he didn’t have any vested equity at the time because he had been there for less than a year. But they did give him an actual offer of “we will let you vest your equity if you sign this thing”. And he said no.

So he made a similar choice to me, but because the legal situation with him was a lot more favorable to OpenAI because they were actually offering him something, I would assume they didn’t feel the need to walk it back, but we can ask him. Anyhow. Props to him.

Dwarkesh Patel

And then how did this episode in general inform your worldview around how people will make high stakes decisions where potentially their own self interest is involved in this kind of key period that you imagine will happen by the end of the decade?

Daniel Kokotajlo

I don’t know if I have that many interesting things to say there. I mean, I think one thing is fear is a huge factor. I was so afraid during that whole process. More afraid than I needed to be in retrospect. And another thing is that legality is a huge factor, at least for people like me. I think in retrospect it was, “oh yeah, the public’s on your side, the employees are on your side. You’re just obviously in the right here”. But at the time I was like, “oh no, I don’t want to accidentally violate the law and get sued. I don’t want to go too far”. I was just so afraid of various things. In particular, I was afraid of breaking the law.

And so one of the things that I would advocate for with whistleblower protections is just simply making it legal to go talk to the government and say “we’re doing a secret intelligence explosion, I think it’s dangerous for these reasons” is better than nothing. I think there’s going to be some fraction of people for which that would make the difference. Whether it’s just literally allowed or not, legally, makes a difference independently of whether there’s some law that says you’re protected from retaliation or whatever. Literally just making it legal. I think that’s one thing. Another thing is the incentives actually work. Money is a powerful motivator and fear of getting sued is a powerful motivator. And this social technology just does in fact work to get people organized in companies and working towards the vision of leaders.

Dwarkesh Patel

Okay. Scott, can I ask you some questions?

Scott Alexander

Of course.

Dwarkesh Patel

How often do you discover a new blogger you’re super excited about?

Scott Alexander

Order of once a year.

Dwarkesh Patel

Okay. And how often after you discover them, does the rest of the world discover them?

Scott Alexander

I don’t think there are many hidden gems. Once a year is a crazy answer in some sense, like it ought to be more. There are so many thousands of people on Substack. But I do just think it’s true that the good blogging space is undersupplied and there is a strong power law. And partly this is subjective, I only like certain bloggers, there are many people who I’m sure are great that I don’t like.

But it also seems like our community in the sense of people who are thinking about the same ideas, people who care about AI economics, those kinds of things, discovers one new great blogger a year, something like that. Everyone is still talking about Applied Divinity Studies, who hasn’t written, unless I missed something, hasn’t written much in a couple of years. I don’t know. It seems undersupplied. I don’t have a great explanation.

Dwarkesh Patel

If you had to give an explanation, what would it be?

Scott Alexander

So this is something that I wish I could get Daniel to spend a couple of months modeling. I was going to say it’s the intersection of too many different tasks. You need people who can come up with ideas, who are prolific, who are good writers. But actually I can also count on a pretty small number of figures the number of people who had great blog posts but weren’t that prolific.

There was a guy named LouKeep who everybody liked five years ago and he wrote like 10 posts and people still refer to all 10 of those posts and “I wonder if LouKeep will ever come back”. So there aren’t even that many people who are very slightly failing by having all of them accept prolificness. Nick Whitaker, back when there was lots of FTX money rolling around, I think this was Nick, tried to sponsor a blogging fellowship with just an absurdly high prize. And there were some great people, I can’t remember who won, but it didn’t result in a Cambrian explosion of blogging. I think it was $100,000. I can’t remember if that was the grand prize or the total prize pool. But having some ridiculous amount of money put in as an incentive got like three extra people.

Dwarkesh Patel

Yeah. So you have no explanation?

Scott Alexander

Actually, Nick is an interesting case because Works in Progress is a great magazine. And the people who write for Works in Progress, some of them I already knew as good bloggers, others I didn’t. So I don’t understand why they can write good magazine articles without being good bloggers. In terms of writing good blogs that we all know about, that could be because of the editing. That could be because they are not prolific. Or it could be- one thing that has always amazed me is there are so many good posters on Twitter. There were so many good posters on Livejournal before it got taken over by Russia. There were so many good people on Tumblr before it got taken over by woke.

But only like 1% of these people who are good at short and medium form ever go to long form. I was on Livejournal myself for several years and people liked my blog, but it was just another Livejournal. No one paid that much attention to it. Then I transitioned to WordPress and all of a sudden I got orders of magnitude much more attention. “Oh, it’s a real blog now we can discuss it now it’s part of the conversation”. I do think courage has to be some part of the explanation. Just because there are so many people who are good at using these hidden away blogging things that never get anywhere. Although it can’t be that much of the explanation because I feel like now all of those people have gotten substacks and some of those substacks went somewhere, but most of them didn’t.

Dwarkesh Patel

On the point about “well, there’s people who can write short form, so why isn’t that translating?” I will mention something that has actually radicalized me against Twitter as an information source is I’ll meet- and this has happened multiple times- I’ll meet somebody who seems to be an interesting poster, has funny, seemingly insightful posts on Twitter. I’ll meet them in person and they are just absolute idiots. It’s like they’ve got 240 characters of something that sounds insightful and it matches to somebody who maybe has a deep worldview, you might say, but they actually don’t have it.

Whereas I’ve actually had the opposite feeling when I meet anonymous bloggers in real life where I’m like, “oh, there’s actually even more to you than I realized off your online persona”. You know Alvaro de Menard, the Fantastic Anachronism guy? I met up with him recently and he gives me, he made a hundred translations of his favorite Greek poet, Cavafy, and he gave me a copy. And it’s just this thing he’s been doing on his side. It’s just like translating Greek poetry he really liked. I don’t expect any anonymous posters on Twitter to be anytime soon handing me their translation of some Roman or Greek poet or something.

Scott Alexander

Yeah, so on the car ride here, Daniel and I were talking about, in AI now the thing everyone is interested in is their ‘time horizon’. Where did this come from? 5 years ago you would not have thought, “oh, time horizon. AIs will be able to do a bunch of things that last one minute, but not that last two hours”. Is there a human equivalent to time horizon?

And we couldn’t figure it out, but it almost seems like there are lots of people who have the time horizon to write a really, really good comment that gets to the heart of the issue. Or a really, really good Tumblr post which is like three paragraphs but somehow can’t make it hang together for a whole blog post. And I’m the same way. I can easily write a blog post, like a normal length ACX blog post, but if you ask me to write a novella or something that’s four times the length of the average ACX blog post, then it’s this giant mess of “re re re re” outline that just gets redone and redone and maybe eventually I make it work.

I did somehow publish Unsong, but it’s a much less natural task. So maybe one of the skills that goes into blogging is this. But I mean, no, because people write books and they write journal articles and they write works in progress articles all the time. So I’m back to not understanding this.

Dwarkesh Patel

No, I mean ChatGPT can write you a book. There’s a difference between the ChatGPT book, which is most books and…

Scott Alexander

There are many, many times more people who have written good books than who are actively operating great bloggers right now, I think.

Daniel Kokotajlo

Maybe that’s financial?

Scott Alexander

No, no, no, no, no, no. Books are the worst possible financial strategy. Substack is where it’s at.

Daniel Kokotajlo

Worse than blogs? You think so?

Dwarkesh Patel

Oh yeah.

Scott Alexander

The other thing is that blogs are such a great status gain strategy. I was talking to Scott Aaronson about this. If people have questions about quantum computing, they ask Scott Aronson or he is like the authority. I mean there are probably hundreds of other professors who do quantum computing things but nobody knows who they are because they don’t have blogs.

So I think it’s underdone. I think there must be some reason why it’s underdone. I don’t understand what that is because I’ve seen so many of the elements that it would take to do it in so many different places and I think it’s either just a multiplication problem where 20% of people are good at one thing, 20% of people are good at another thing, and you need five things, there aren’t that many.

Plus something like courage, where people who would be good at writing blogs don’t want to do it. I actually know several people who I think would be great bloggers in the sense that sometimes they send me multi-paragraph emails in response to an ACX post and I’m like, “wow, this is just an extremely well written thing that could have been another blog post. Why don’t you start a blog?” And they’re like, “oh, I could never do that”.

Dwarkesh Patel

What advice do you have to somebody who wants to become good at it but isn’t currently good at it?

Scott Alexander

Do it every day, same advice as for everything else. I say that I very rarely see new bloggers who are great. But like when I see some. I published every day for the first couple years of Slate Star Codex, maybe only the first year. Now I could never handle that schedule, I don’t know, I was in my 20s, I must have been briefly superhuman.

But whenever I see a new person who blogs every day it’s very rare that that never goes anywhere or they don’t get good. That’s like my best leading indicator for who’s going to be a good blogger.

Dwarkesh Patel

And do you have advice on what kinds of things to start? One frustration you can have is you want to do it, but you have so little to say, you don’t have that deep a world model, a lot of the ideas you have are just really shallow or wrong. Just do it anyway?

Scott Alexander

So I think there are two possibilities there. One is that you are, in fact, a shallow person without very many ideas. In that case I’m sorry, it sounds like that’s not going to work. But usually when people complain that they’re in that category, I read their Twitter or I read their Tumblr, or I read their ACX comments, or I listen to what they have to say about AI risk when they’re just talking to people about it, and they actually have a huge amount of things to say. Somehow it’s just not connecting with whatever part of them has lists of things to blog about.

So that may be another one of those skills that only 20% of people have, is when you have an idea you actually remember it and then you expand on it. I think a lot of blogging is reactive; You read other people’s blogs and you’re like, no, that person is totally wrong. A part of what we want to do with this scenario is say something concrete and detailed enough that people will say, no, that’s totally wrong, and write their own thing.

But whether it’s by reacting to other people’s posts, which requires that you read a lot, or by having your own ideas, which requires you to remember what your ideas are, I think that 90% of people who complain that they don’t have ideas, I think actually have enough ideas. I don’t buy that as a real limiting factor for most people.

Dwarkesh Patel

I have noticed two things in my own… I mean, I don’t do that much writing, but from the little I do: one, I actually was very shallow and wrong when I started. I started the blog in college. So if you are somebody who’s like, “this is bullshit, there’s nothing to this. Somebody else wrote about this already”, that’s fine, what did you expect? Right? Of course, as you’re reading more things and learning more about the world, that’s to be expected and just keep doing it if you want to keep getting better at it.

And the other thing now when I write blog posts, as I’m writing them, I’m just like, “why? These are just some random stories from when I was in China. They’re like kind of cringe stories”. Or with the AI firm’s post, it’s like, “come on, these are just weird ideas. And also some of these seem obvious, whatever”. My podcasts do what I expect them to do. My blogs just take off way more than I expect them to take off in advance.

Scott Alexander

Your blog posts are actually very good.

Daniel Kokotajlo

Yeah, they’re good.

Dwarkesh Patel

But the thing I would emphasize is that, for me, I’m not a regular writer and I couldn’t do them on a daily basis. And as I’m writing them, it’s just this one or two week long process of feeling really frustrated. Like, “this is all bullshit, but I might as well just stick with the sunk cost and just do it”.

Scott Alexander

It’s interesting because like a lot of areas of life are selected for arrogant people who don’t know their own weaknesses because they’re the only ones who get out there. I think with blogs and I mean this is self-serving, maybe I’m an arrogant person, but that doesn’t seem to be the case. I hear a lot of stuff from people who are like, “I hate writing blog posts. Of course I have nothing useful to say”, but then everybody seems to like it and reblog it and say that they’re great.

Part of what happened with me was I spent my first couple years that way, and then gradually I got enough positive feedback that I managed to convince the inner critic in my head that probably people will like my blog post. But there are some things that people have loved that I was absolutely on the verge of, “no, I’m just going to delete this, it would be too crazy to put it out there”. That’s why I say that maybe the limiting factor for so many of these people is courage because everybody I talk to who blogs is within 1% of not having enough courage of blogging.

Dwarkesh Patel

That’s right. That’s right. And also “courage” makes it sound very virtuous, which I think it can often be, given the topic, but at least often it’s just like…

Scott Alexander

Confidence?

Dwarkesh Patel

No, not even confidence. It’s closer to maybe what an aspiring actor feels when they go to an audition where it’s like, “I feel really embarrassed. But also I just really want to be a movie star”.

Scott Alexander

So the way I got through this is I blogged for like 8 to 10 years on LiveJournal before- no, it was less than that. It’s more like five years on LiveJournal before ever starting a real blog. I posted on LessWrong for a year or two before getting my own blog. I got very positive feedback from all of that, and then eventually I took the plunge to start my own blog. But it’s ridiculous. What other career do you need seven years of positive feedback before you apply for your first position?

I mean, you have the same thing. You’ve gotten rave reviews for all of your podcasts, and then you’re kind of trying to transfer to blogging with probably... First of all, you have a fan base. People are going to read your blog. That, I think is one thing, is people are just afraid no one will read it, which is probably true for most people’s first blog. And then there are enough people who like you that you’ll probably get mostly positive feedback, even if the first things you write aren’t that polished. So I think you and I both had that. A lot of people I know who got into blogging kind of had something like that. And I think that’s one way to get over the fear gap.

Dwarkesh Patel

I wonder if this sends the wrong message or raises expectations or raises concerns and anxieties. But one idea I’ve been shooting around, and I’d be curious about your take on this: I feel like this slow, compounding growth of a fan base is fake. If I notice some of the most successful things in our sphere that have happened; Leopold releases Situational Awareness. He hasn’t been building up a fan base over years. It’s just really good. And as you were mentioning a second ago, whenever you notice a really great new blogger, it’s not like it then takes them a year or two to build up a fan base. Nope, everybody, at least that they care about, is talking about it almost immediately.

I mean, Situational Awareness is in a different tier almost. But things like that and even things that are an order of magnitude smaller than that will literally just get read by everybody who matters. And I mean literally everybody. And I expect this to happen with AI 2027 when it comes out. But Daniel, you’ve been building your reputation within this specific community, and I expect AI 2027 it's just really good. And I expect it’ll just blow up in a way that isn’t downstream of you having built up an audience over years.

Daniel Kokotajlo

Thank you. I hope that happens. We’ll see.

Scott Alexander

Slightly pushing back against that. I have statistics for the first several years of Slate Star Codex, and it really did grow extremely gradually. The usual pattern is something like every viral hit, 1% of the people who read your viral hits stick around. And so after dozens of viral hits, then you have a fan base. But smoothed out, It does look like a- I wish I had seen this recently, but I think it’s like over the course of three years, it was a pretty constant rise up to some plateau where I imagine it was a dynamic equilibrium and as many new people were coming in as old people were leaving.

I think that with Situational Awareness, I don’t know how much publicity Leopold put into it. We’re doing pretty deliberate publicity, we’re going on your podcast. I think you can either be the sort of person who can go on a Dwarkesh podcast and get the New York Times to write about you, or you can do it organically, the old fashioned way, which is very long.

Dwarkesh Patel

Yeah. Okay. So you say that throwing money at people to make them, to get them to blog at least didn’t seem to work for the FTX folks. If it was up to you, what would you do? What’s your grant plan to get 10 more Scott Alexanders?

Scott Alexander

Man. So my friend Clara Collier, who’s the editor of Asterisk magazine, is working on something like this for AI blogging. And her idea, which I think is good, is to have a fellowship. I think Nick’s thing was also a fellowship, but the fellowship would be, there is an Asterisk AI blogging fellows’ blog or something like that. Clara will edit your post, make sure that it’s good, put it up there and she’ll select many people who she thinks will be good at this. She’ll do all of the kind of courage requiring work of being like, “yes, your post is good. I’m going to edit it now. Now it’s very good. Now I’m going to put it on the blog”.

And I think her hope is that, let’s say of the fellows that she chooses, now it’s not that much of a courage step for them to start it because they have the approval of what last psychiatrist would call an omniscient entity, somebody who is just allowed to approve things and tell you that you’re okay on a psychological level. And then like maybe of those fellows, some percent of them will have their blog posts be read and people will like them. And I don’t know how much reinforcement it takes to get over the high prior everyone has on “no one will like my blog”. But maybe for some people, the amount of reinforcement they get there will work.

Yeah, like an interesting example would be all of the journalists who have switched to having Substacks. Many of them go well. Would all of those journalists have become bloggers if there was no such thing as mainstream media? I’m not sure. But if you’re Paul Krugman you know people like your stuff, and then when you quit the New York Times you know you can just open a substack and start doing exactly what you were doing before. So I don’t know, maybe my answer is there should be mainstream media. I hate to admit that, but maybe it’s true.

Dwarkesh Patel

Invented it from first principles.

Scott Alexander

Yeah.

Dwarkesh Patel

Well I do think that it should be treated more as a viable career path. Where right now, if you told your parents, “I’m going to become a startup founder”, I think the reaction would be like, “there’s a 1% chance you’ll succeed, but it’s an interesting experience and if you do succeed, that’s crazy. That’ll be great. If you don’t, you’ll learn something. It’ll be helpful to the thing you do afterwards”.

We know that’s true of blogging, right? We know that it helps you build up a network, it helps you develop your ideas. And if you do succeed, you get a dream job for a lifetime. And I think maybe they don’t have that mindset, but also they under appreciate how much you actually could succeed at it. It’s not a crazy outcome to make a lot of money as a blogger.

Scott Alexander

I think it might be a crazy outcome to make a lot of money as a blogger. I don’t know what percent of people who start a blog end up making enough that they can quit their day job. My guess is it’s a lot worse than for startup founders. I would not even have that as a goal so much as like the Scott Aaronson goal of, okay, you’re still a professor, but now you’re the professor whose views everybody knows and who has kind of a boost up in respect in your field and especially outside of your field. And also you can correct people when they’re wrong, which is a very important side benefit.

Dwarkesh Patel

Yeah. How does your old blogging feedback into your current blogging? So when you’re discussing a new idea, I mean, AI or whatever else, are you just able to pull from the insights from your previous commentary on sociology or anthropology or history or something?

Scott Alexander

Yeah. So I think this is the same as anybody who’s not blogging. I think the thing everybody does is they’ve read many books in the past and when they read a new book, they have enough background to think about it. Like you are thinking about our ideas in the context of Joseph Henrich’s book. I think that’s good, I think that’s the kind of place that intellectual progress comes from. I think I am more incentivized to do that.

It’s hard to read books. I think if you look at the statistics, they’re terrible. Most people barely read any books in a year. And I get lots of praise when I read a book and often lots of money, and that’s a really good incentive. So I think I do more research, deep dives, read more books than I would if I weren’t a blogger. It’s an amazing side benefit. And I probably make a lot more intellectual progress than I would if I didn’t have those really good incentives.

Dwarkesh Patel

Yeah. There was actually a prediction market about the year by which an AI would be able to write a blog post as good as you. Was it 2026 or 2027? I think it was 2027. It was like 15% by 2027 or something like that. It is an interesting question of they do have your writing and all other good writing in trading distribution. And weirdly, they seem way better at getting superhuman at coding than they are at writing, which is the main thing in their distribution.

Scott Alexander

Yeah. It’s an honor to be my generation’s Garry Kasparov figure. Yeah. So I’ve tried this. And first of all, it does a decent job. I respect its work. It’s not perfect yet. I think it’s actually better at the style on a word-to-word, sentence-to-sentence level, than it is at planning out a blog post. So I think there are possibly two reasons for it: One, we don’t know how the base model would have done at this task. We know that all the models we see are to some degree reinforcement learning into a kind of corporate speak mode. You can get it somewhat out of that corporate speak mode. But I don’t know to what degree this is actually doing its best to imitate Scott Alexander versus hit some average between Scott Alexander and corporate speak. And I don’t think anyone knows except the internal employees who have access to the base model.

And the second thing I think of maybe just because it’s trendy has an agency or horizon failure, like deep research is an okay researcher. It’s not a great researcher. If you actually want to understand an issue in depth, you can’t use deep research. You gotta do it on your own. So if I spend maybe five to 10 hours researching a really research heavy blog post, the METR thing, I know we’re not supposed to use it for any task except coding, but like it says, on average the AI’s horizon is one hour. So I’m guessing it just cannot plan and execute a good blog post. It does something very superficial rather than actually going through the steps. So my guess for that prediction market would be whenever we think the agents are actually good. I think in our scenario that’s like late 2026. I’m going to be humble and not hold out for the superintelligence.

Daniel Kokotajlo

What about comments? I feel like intuitively it feels like before we see the AI’s writing great blog posts that go super viral repeatedly, we should see them writing highly upvoted comments on things.

Scott Alexander

Yeah. And I think somebody mentioned this on the LessWrong post about it and somebody made some AI generated comments to that post. They were not great. But I wouldn’t have immediately picked them out of the general distribution of LessWrong comments as especially bad. I think, like, I think if you were to try this, you would get something that was so obviously an AI house style that it would use the word ‘delve’ or things along those lines.

I think if you were able to avoid that maybe by using the base model, maybe by using some kind of really good prompt to be like, “no, do this in Gwern’s voice”, you would get something that was pretty good. I think if you wrote a really stupid blog post, it could point out the correct objections to it. But I also just don’t think it’s as smart as Gwern right now. So its limit on making Gwern-style comments is both- It needs to be able to do a style other than corporate delve slop and then it actually needs to get good.

Daniel Kokotajlo

It needs to have good ideas that other people don’t already have.

Scott Alexander

Yeah. And I mean I think it can write as well as a smart average person in a lot of ways. And I think if you have a blog post that's worse than that or at that level, it can come up with insightful comments about it. I don’t think it could do it on a quality blog post.

Dwarkesh Patel

There was this recent Financial Times article about how have you reached peak cognitive power? Where it was talking about declining scores in PISA and SAT and so forth. On the Internet especially, it does seem like there might have been a golden era before I was that active on the forums or whatever. Do you have nostalgia for a particular time on the Internet when it was just like, this is an intellectual mecca?

Scott Alexander

I am so mad at myself for missing most of the golden age of blogging. I feel like if I had started a blog in 2000 or something, then- I don’t know, I’ve done well for myself, I can’t complain- but the people from that era all founded news organizations or something. I mean, God save me from that fate. I would have liked to have been there. I would have liked to see what I could have done in that area. I mean, I wouldn’t compare the decline of the Internet to that stuff with PISA because I’m sure the Internet is just more people are coming on, it’s a less heavily selected sample.

But yeah, I could have passed on the whole era where they were talking about atheism versus religion nonstop. That was pretty crazy. But I do hear good things about the golden age of blogging.

Dwarkesh Patel

Anybody who was sort of counterfactually responsible for you starting to blog or keeping blogging?

Scott Alexander

So I owe a huge debt of gratitude to Eliezer Yudkowski. I had a live journal before that. But it was going on LessWrong that convinced me I could move to the big times. And second of all, I just think I learned I imported a lot of my worldview from him. I think I was the most boring normie liberal in the world before encountering LessWrong. And I don’t 100% agree with all LessWrong ideas, but just having things of that quality beamed into my head and for me to react to and think about was really great.

Dwarkesh Patel

And tell me about the fact that you could be and were at some point anonymous, I think for most of human history, somebody who is an influential advisor or an intellectual or somebody. Actually, I don’t know if this is true. You would have had to have some sort of public persona. And a lot of what people read into your work is actually a reflection of your public persona.

Scott Alexander

Sort of. The reason half of these ancient authors are called things like Pseudo Dionysus or Pseudocelsus is that you could just write something being like, “oh, yeah, this is by Saint Dionysus”. And then, I don’t know, you could be anybody.

And I don’t know exactly how common that was in the past. But yeah, I agree that the Internet has been a golden age for anonymity. I’m a little bit concerned that AI will make it much easier to break anonymity. I hope the golden age continues.

Dwarkesh Patel

Yeah, seems like a great note to end on. Thank you guys so much for doing this.

Scott Alexander

Thank you.

Daniel Kokotajlo

Thank you so much. This was a blast.

Dwarkesh Patel

Yeah, I had a great time.

Daniel Kokotajlo

Huge fan of your podcast.

Dwarkesh Patel

Thank you.

I recorded an AMA! I had a blast shooting the shit with my friends Trenton Bricken and Sholto Douglas. We discussed my new book, career advice given AGI, how I pick guests, how I research for the show, and some other nonsense.

My book, “The Scaling Era: An Oral History of AI, 2019-2025” is available in digital format now. Preorders for the print version are also open!

Watch on YouTube; listen on Apple Podcasts or Spotify.

(0:00:00) - Book launch announcement

(0:04:57) - AI models not making connections across fields

(0:10:52) - Career advice given AGI

(0:15:20) - Guest selection criteria

(0:17:19) - Choosing to pursue the podcast long-term

(0:25:12) - Reading habits

(0:31:10) - Beard deepdive

(0:33:02) - Who is best suited for running an AI lab?

(0:35:16) - Preparing for fast AGI timelines

(0:40:50) - Growing the podcast

Dwarkesh Patel

Today, this is going to be an Ask Me Anything episode. I'm joined by my friends, Trenton Bricken and Sholto Douglas. You guys do some AI stuff, right?

Trenton Bricken

Yeah.

Sholto Douglas

We dabble.

Dwarkesh Patel

They're researchers at Anthropic. Other news; I have a book launching today, it's called The Scaling Era. I hope one of the questions ends up being why you should buy this book. But we can kill two birds with one stone. But, okay, let's just get at it. What's the first question that we gotta answer?

Trenton Bricken

So, I want to ask the flyball question that I heard before, of: why should ordinary people care about this book? Like, why should my mom buy and read the book?

Dwarkesh Patel

Yeah. First, let me tell you about the book, what it is. So, you know, these last few years, I've been interviewing AI lab CEOs, researchers, people like you guys, but also scholars from all kinds of different fields, economists, philosophers. And they've been addressing, I think, what are basically the gnarliest, most interesting, most important questions we've ever had to ask ourselves. Like, what is the fundamental nature of intelligence? What will happen when we have billions of extra workers? How do we model out the economics of that? How do we think about an intelligence that is greater than the rest of humanity combined? Is it even a coherent concept? And so, what I'm super delighted with is that with Stripe Press, we made this book where we compiled and curated the best, most insightful snippets across all these interviews. And you can read Dario addressing, why does scaling work? And then on the next page is Demis explaining DeepMind's plans for whether they're gonna go with the RL route and how much of the AlphaZero stuff will play into the next generation of LLMs. And on the next page is, of course, you guys going through the technical details of how these models work.

And then there's so many different fields that are implicated. I mean, I feel like AI is one of the most multi-disciplinary fields that one can imagine, because there's no field, no domain of human knowledge that is not relevant to understanding what a future society of different kinds of beings will look like. You're gonna have Carl Shulman talk about how the scaling hypothesis shows up in primate brain scaling from chimpanzees to humans. On the next page might be an economist trying to argue, like Tyler Cowen, explaining why he doesn't expect explosive economic growth, and why the bottlenecks will eat all that up. Um, anyways, so that's why your mom should buy this book. It’s the distillation of all these different fields of human knowledge applied to the most important questions that humanity is facing right now.

Trenton Bricken

I do like how the book is sliced up by different topics and across interviews. So it does seem like a nice way to listen to all of the interviews in one digestible way.

Dwarkesh Patel

Yeah. There's two interviews I've done that haven't been released publicly before that are in the book. So, one was that Jared Kaplan, who's one of your co-founders, and this is another example where it's like, he's a physicist and he's explaining scaling from this very mathematical perspective about data manifolds. And then on the next page you have a totally different perspective. It's like Goren talking about why did general intelligence actually evolve in the first place, what is the actual evolutionary purpose of it? And it's page by page, right? You can just get addresses. Even for me, the person who's been on the other end of these conversations, it was actually really cool to read it and just be like, "Oh, actually now I realize how these insights connect to each other."

Trenton Bricken

Yeah, the only other thing that stood out to me as well is the introduction section-

Dwarkesh Patel

The only thing that stood out to you?

Trenton Bricken

Yeah, that was really the only thing that was noteworthy. I just mean [what] stood out in accessibility is the introduction section and the diagrams for all the different inputs that enable you to train a machine learning model. Stripe Press books are also just beautiful, they have these nice side captions for explaining what parameters are, what a model is, these sorts of things.

Dwarkesh Patel

Actually, when we did our episode together, a bunch of people, I don't know if you saw this, independently made these blog posts and Anki cards and shit where they're explaining the concept because we just kind of passed over some things. And hopefully we've given a similar treatment to every single interview I've done, where you can read a very technical interview with a lab CEO or something, or an engineer, or a researcher, and then the side is like: here's like more context, here's more definitions, here's more commentary. And I, yeah, I feel like it elevated the conversations.

Sholto Douglas

So in other words, my parents will finally understand what I do for a job. They're gonna get it very well.

Dwarkesh Patel

Maybe my parents will. Because I got a book.

Trenton Bricken

All mine need to know is that my name's in a book.

Sholto Douglas

You're a co-author.

Trenton Bricken

They're like, "Cool."

Sholto Douglas

Should we get into the AMA questions?

Trenton Bricken

Let's do it.

Sholto Douglas

All right. So Brian Krav asks, "The issue you raised with Dario and occasionally tweet about relating to models not making connections across different topics, some sort of combinatorial attention challenge, what are your thoughts on that now? Do you solve it with scale, thinking models or something else?"

Dwarkesh Patel

So the issue is, one of the questions I asked Dario is, look, these models have all of human knowledge memorized and you would think if a human had this much stuff memorized, and they were moderately intelligent, they could be making all these connections between different fields. And there are examples of humans doing this, by the way. There's… Donald Swann or something like this [Don R. Swanson], this guy noticed that what happens to a brain after magnesium deficiency is exactly the structure you see during a migraine. So then he's like, you take magnesium supplements and we're gonna cure a bunch of migraines. And it worked. And there's many other examples of things like this where you just notice two different connections between pieces of knowledge. Why, if these LLMs are intelligent, are they not able to use this unique advantage they have to make these kinds of discoveries? I feel a little shy, me giving answers on AI shit with you guys here. But, actually Scott Alexander addressed this question in one of his AMA threads, and he's like, "Look, humans also don't have this kind of logical omniscience", right? He used the example of, in language, if you really thought about, why are two words connected? And it's like, I understand why “rhyme” has the same etymology as this other word. But you just don't think about it, right? There's this combinatorial explosion. I don't know if that addresses the fact that- we know humans can do this, right? The humans have in fact done this, and I don't know of a single example of LLMs ever having done it. Actually, yeah, what is your answer to this?

Sholto Douglas

I think my answer at the moment is that the sort of pre-training objective doesn't necessarily- like it imbues with this nice flexible general knowledge about the world, but doesn't necessarily imbue the skill of making novel connections or research. The kinds of things that people are trained to do through PhD programs and through the process of exploring and interacting with the world.

And so I think at a minimum you need significant RL in at least similar things to be able to approach making novel discoveries. And so I would like to see some early evidence of this as we start to build models that are interacting with the world and trying to make scientific discoveries, and modeling the behaviors that we expect of people in these positions. Because I don't actually think we've done that in a meaningful or scaled way as a field, so to speak.

Trenton Bricken

Riffing off that with respect to RL, I wonder if models currently just aren't good at knowing what memories they should be storing. Most of their training is just predicting the next word on the internet and remembering very specific facts from that. But if you were to teach me something new right now, I'm very aware of my own memory limitations, and so I would try to construct some summary that would stick. And models currently don't have the opportunity to do that. Memory scaffolding in general is just very primitive right now. I mean-

Sholto Douglas

Right, like Claude Plays Pokemon.

Trenton Bricken

Exactly, yeah, or like someone worked on it, it was awesome, it got far, but, another excited Anthropic employee then iterated on the memory scaffold and was able to very quickly improve on it. So that's one. I do also just wonder if models are idiot savants. The best analogy might be to Kim Peek. So Kim Peek, was born without a corpus callosum, if I recall correctly. Each hemisphere of his brain operated quite independently. So, he'd open a book, there'd be two pages visible, each eye would read one of the pages. And he had a perfect encyclopedic memory of everything he'd ever read. But at the same time, he had other debilitations; functioning socially, these sorts of things. And it's just kind of amazing how good LLMs are at very niche topics, but can totally fail at other ones still.

Dwarkesh Patel

I really wanna double-click on this thing of why there's this trade-off between memorization. Why does cutting it off... apparently it's connected to this debilitation, but why can't… Wiki text is like five megabytes of information. The human brain can store much more, so why does the human brain just not want us to memorize these kinds of things, and is actively pruning, and… yeah, I don't know. But we don't have to do it right now. We'll do a separate episode.

Trenton Bricken

Yeah, just one thing I'll say on that is, there is another case study of someone with a perfect memory, so they never forgot anything. But their memory was too debilitating. It'd be, like, your context window for the transformer is trillions of tokens. And then you spend all your time attending to past things, and are too trapped in the details to extract any meaningful generalizable insights from it.

Dwarkesh Patel

Yeah. Terrence Deacon, whose book you recommended, had this interesting insight about how we learn best when we're children, but we forget literally everything that happened to us when we were children, right? We have total amnesia.

And adults have this in-between where we don't remember exact details, but we can still learn in a pretty decent way. And then LLMs are on the opposite end of this gradient where they'll get the exact phrasing of Wiki text down, but they won't be able to generalize in these very, obvious ways.

Sholto Douglas

A little bit like Gwern's theory, optimizer theory, no?

Dwarkesh Patel

Yeah, I think I probably got it from that.

Trenton Bricken

Yeah, Gwern has definitely had a big influence on all this for me as well.

Dwarkesh Patel

I feel like what’s underappreciated on the podcast is we have this group chat, and we also just meet up a lot in person. And all the output from the podcast just comes from you and a couple other people just feeding me ideas and nudges and whatever, and then I can just use that as an intuition pump during the conversation.

Trenton Bricken

Yeah, you're not the only one.

Dwarkesh Patel

What do you mean?

Trenton Bricken

Oh, like, I benefit immensely from just hearing what everyone else has to say. It's all regurgitation in one way or another.

Sholto Douglas

Another question?

Dwarkesh Patel

Yes.

Trenton Bricken

Maybe Rabid Monkey asks, "Imagine you have a 17-year-old brother/nephew just starting college. What would you recommend he study, given your AGI timelines?"

Dwarkesh Patel

That's so tough, right? I don't know, become a podcaster? I feel like that job's still gonna be around. It's funny, because I studied computer science, and in retrospect- at the time, you could've become a software engineer or something. Instead, you became a podcaster, it’s kind of an irresponsible career move, but in retrospect, it's like… It kinda worked out. Just as these guys are getting automated.

Sholto Douglas

I get asked this question all the time, and one answer that I like to give is that you should think about the next couple of years as increasing your individual leverage by a huge factor every year.

So already software engineers will come up and say, "You know, I'm two times faster," or, "In new languages, I'm five times faster than I was last year." I expect that trend line to continue, basically, as you go from this model of, "Well, I'm working with some model that's assisting me on my computer, and it's basically a pairing session," to, "I'm managing a small team," through to, "I'm managing a division or a company". Basically, that is targeting a task. And so I think that deep technical knowledge in fields will still matter in four years. It absolutely will. Because you will be in the position of managing dozens- or, your individual management bandwidth will be maxed out by trying to manage teams of AIs.

And maybe we end up in a true singularity world where you have AIs managing AIs and this kinda stuff. But I think in a very wide part of the possibility spectrum you are managing enormous, vastly more resources than an individual could command today, and you should be able to solve so many more things with that.

Dwarkesh Patel

That's right, and I think I would emphasize that this is not just cope. Like, it genuinely is a case that these models lack the kind of long-term coherence which is absolutely necessary for making a successful company or… Just, getting a fucking office is kinda complicated, right? So you can just imagine that for sector after sector- the economy is really big, right?

Sholto Douglas

And really complex.

Dwarkesh Patel

Exactly, and so, I don't know the details, but I assume if it's a data sparse thing where you gotta know what is the context of what's happening in the sector or something, I feel like you'd be in a good position.

Maybe the other thought I have is that it's really hard to plan your career in general. And I don't know what advice that implies, because I remember being super frustrated. I was in college, and the reason I was doing the podcast was to figure out what it is I want to do. It wasn't the podcast itself. And I would go on, 80,000 Hours or whatever career advice, and in retrospect it was all mostly useless, and just try doing things. I mean, especially with AI, it's so hard to forecast what kind of transformations there will be, so try things, do things. I mean, it's such banal, vague advice, but I am quite skeptical of career advice in general.

Sholto Douglas

Well, the piece of career advice that I'm not skeptical of is put yourself close to the frontier, because you have a much better vantage point from there. Right? You can study deep technical things, whether it's computer science or biology, and get to the point where you can see what the issues are because it's actually remarkably obvious at the frontier what the problems are. It's very difficult to see…

Dwarkesh Patel

Actually, do you think there is an opportunity, because one of the things people bring up is, maybe the people who are advanced in their career and have all this tacit knowledge will be in a position to be accelerated by AI, but you guys four years ago or two years ago, when you were getting discovered or something, that kind of thing where you have a GitHub open issue and you try to solve it; is that just, that's done, and so the onboarding is much harder?

Sholto Douglas

That's still what we look for in hiring. So, you know?

Trenton Bricken

Yeah, I'm in favor of the “learn fundamentals, gain useful mental models”, but it feels like everything should be done in an AI-native way, or top-down instead of your bottom-up learning. So first of all, learn things more efficiently by using the AI models, and then just know where their capabilities are and aren't.

And I would be worried and skeptical about any subject which prioritizes rote memorization of lots of facts or information instead of ways of thinking. But if you're always using the AI tools to help you, then you'll naturally just have a good sense for the things that it is and isn't good at.

Sholto Douglas

Okay, next one. What is your strategy, method, or criteria for choosing guests?

Dwarkesh Patel

So, the most important thing is, do I wanna spend one to two weeks reading every single thing you have ever written, every single interview you ever recorded, talking to a bunch of other people about your research? Because I get asked by people who are quite influential often to be like, "Would you have me on your podcast?" and more often than not, I say no, for two reasons. One is just, okay, you're influential, it's not fundamentally that interesting as an interview prospect. I don't think about the hour that I'll spend with you. I think about the two weeks, because this is my life, right? The research is my life, and I wanna have fun while doing it. So is this gonna be an interesting two weeks to spend? Is it gonna help me with my future research or something?

And the other is, big guests don't really matter that much if you just look at what are the most popular episodes or, what in the long run helps a podcast grow. By far my most popular guest is Sarah Payne, and she, before I interviewed her, was just a scholar, who was not publicly well-known at all, and I just found her books quite interesting. So my most popular guests are Sarah Payne and then Sarah Payne, Sarah Payne, Sarah Payne because ... I have electric chairs with her.

And by the way, from a viewer-a-minute adjusted basis, I host the Sarah Payne Podcast where I occasionally talk about AI.

Trenton Bricken

That's funny.

Dwarkesh Patel

And then it's David Reich, who is a geneticist of ancient DNA. He's somewhat well-known, but he had a best-selling book, but he's not Satya Nadella or Mark Zuckerberg, who are the next people on the list. And then again, I think that pretty soon it's like you guys or Leopold or something, and then you get to the lab CEOs or something.

So, big names just don't matter that much for what I'm actually trying to do. And it's also really hard to predict who's gonna be the David Reich or a Sarah Payne, so just have fun. Talk to whoever you want to spend time researching, and it's a pretty good proxy for what will actually be popular.

Sholto Douglas

What was the specific moment, if there was one, that you realized you, that producing your podcast was a viable long-term strategy?

Dwarkesh Patel

I think when I was shopping around ad spots for a Mark Zuckerberg episode. And now when I look back on it, it's not in retrospect that mind-blowing, but at the time I'm like, “oh, I could actually hire an editor full-time, or maybe more editors than one, and from there turning into a real business”. Because before people would tell me, "Oh, these other podcasts are making whatever amount of money." And I'd be like, "How?" You know?

So I have this running joke with one of my friends. I don't know if you've seen me do this, but every time I encounter a young person who's like, "What should I do with my life?" I'm like, "You gotta start a blog. You gotta be the Matt Levine of AI." You can do this. It's a totally empty niche. And I have this running joke with them where they're like, "You're like a country bumpkin who's won the lottery. And you go up to everything and everyone and just like, "Guys, a scratch pad. Get the scratch pad.”"

Sholto Douglas

I do wanna press on that a bit more because your immediate answer to the 17-year-old was to start a podcast. So what niches are there? What sort of things would you be excited to see in new blogs, podcasts?

Dwarkesh Patel

I wonder if you guys think this too, but I think this “Matt Levine of AI” is a totally open niche as far as I can tell, and I apologize to those who are trying to fill it in. And so the other thing I'd really emphasize is, it is really hard to do this based on other people's advice, or to say “at least I'm trying not to fill a specific niche”. If you think about any sort of successful new media thing out there, it has two things which are true: It's often not just geared towards one particular topic or interest, and two, the most important thing is that it is propelled by a single person's vision. It's not a collective or whatever. And so the thing I really want to emphasize is it can be done.

Two, you can make a lot of money at it, which is not the most important thing probably for the kind of person who would succeed at it, but still is just worth knowing that it's a viable career.

Three, that basically you're gonna feel like shit in the beginning where all your early stuff is gonna kind of suck. Maybe some of it will get appreciated. But it seems like bad advice to say still stick through it in case you actually are terrible because some people are terrible. But in case you are not, just do it, right? Like what is the three months of blogging on the side really gonna cost you? And people just don't actually seriously do the thing for long enough to actually get evidence or get the sort of RL feedback on, oh, this is how you do it; this is how you frame an argument. This is how you make a compelling thing that people will want to read or watch.

Sholto Douglas

Blogging is definitely underrated. I think like most of us have probably-

Dwarkesh Patel

So you both had blogs which were relevant. I don't know if they're actually relevant to getting-

Sholto Douglas

Not that. They were like somewhat relevant. But I think more so that we have all read almost all the blogs that do in-depth treatises on AI. Like if you write something that is high quality, it is almost invariably going to be shared around Twitter and read.

Dwarkesh Patel

Oh, this is so underappreciated. So, two pieces of evidence. I was talking to a very famous blogger you would know, and I was asking him, "How often do you discover a new undiscovered blogger?" And he was like, "Eh, happens very rarely, like maybe once a year." and then I ask him, "How long after you discover him or her does the rest of the world discover them?" And he's like, "Maybe a week."

And what that suggests is it's actually really efficient. Like... Oh, I have some more takes.

Trenton Bricken

Let's hear them. This is, this is the AMA.

Dwarkesh Patel

So I believe that slow compounding growth in media is kind of fake. Like, Leopold's situational awareness. It's not like he was building up an audience for a long time, for years or something. It was really good. Disagree or agree with it, and if it's good enough, literally everybody who matters- and I mean that literally- will read it. And it's hard to zero shot something like that. But the fundamental thing to emphasize is the compounding growth, at least for me, has been I feel like I've gotten better.

And it's not so much that somehow the three years of having 1,000 followers were somehow a compounding... I don't think it was that. I think it was just that it took a while to get better.

Sholto Douglas

Yeah, certainly when Leopold posted that, the next day, it's almost like you can picture it being stapled to the wall, so to speak, on Twitter.

Like, you know, everyone was talking about it. You went to any event for the following week, every single person in the entire city was talking about that essay. It was like Renaissance Florence or whatever.

Dwarkesh Patel

That's right. Yeah. The world is small.

Sholto Douglas

World is small.

Trenton Bricken

What would you say is your first big success? I'm trying to think back to when I first found your podcast. I distinctly remember you had your blog post on the Annus Mirabilis.

And Jeff Bezos retweeted it, I think. I'm trying to remember if it was before that or not, but, yeah. I'm curious, your answer.

Dwarkesh Patel

I feel like that was it. And it wasn't something where it was some big insight that deserved to blow up like that. It was just taking some shots on goal. They were all insight porn-y, and then one of them I guess caught the right guy's attention and, yeah. But I think that was it.

Sholto Douglas

Yeah, that's something else which is underappreciated, which is that a piece of writing doesn't need to have a fundamentally new insight so much as give people a way to express cleanly a set of ideas that they are already aware of in a broader way. And if it's really crisp and not articulate, then even still that's very valuable.

Dwarkesh Patel

And the one thing I should emphasize, which I think is maybe the most important thing to the feedback loop. It's not the compounding growth of the audience. I don't even think it's me getting more shots on goal in terms of doing the podcast. I actually don't think you improve that much by just doing the same thing again and again. If there's no reward signal you'll keep doing whatever you were doing before.

I genuinely think the most important thing has been that the podcast is good enough that it merits me getting to meet people like you guys. Then I become friends with people like you. You guys teach me stuff. I produce more good podcasts, so hopefully slightly better. That helps me meet people in other fields. They teach me more things. With the China thing recently, I wrote this blog post about a couple stories about things that happened in China. And that alone has netted me an amazing China network in the matter of one blog post. Right? And so hopefully, if I do an episode on China, I will be better as a result. And hopefully that happens across field after field. And so just getting to meet people like you is actually the main sort of flywheel.

Sholto Douglas

Interesting. So move to San Francisco?

Dwarkesh Patel

Yes. If you're trying to do AI, yeah.

Sholto Douglas

Oh, very important question from Jacked Pajeet. How much can you bench?

Trenton Bricken

You can't lie because we both know the answer.

Dwarkesh Patel

At one point I did bench 225 for four. Now I think I'm probably 20 pounds lighter than that or something.

The reason you guys are asking me this is because I've gone lifting with both of you. And I remember Trent and I were doing pull-ups and a bench. And it'd be, like, ‘bench’, and he'd throw on another plate or something. And then instead of pull-ups, he'd be cranking out these muscle ups.

Trenton Bricken

It's all technique. Let's make sure.

Dwarkesh Patel

So they both bench more than me. But I'm trying my best.

Trenton Bricken

Ask again in six months.

Dwarkesh Patel

Yeah.

Sholto Douglas

What's your favorite history book? There's a wall of them behind you.

Dwarkesh Patel

Oh, obviously the Caro LBJ biographies. The main thing I took away from those books is LBJ had this quote that he would tell his debate [students]. In his early 20s, he taught debate to these poor Mexican students in Texas. And he used to tell them, "If you do everything, you'll win." I think it's an underrated quote. So that's the main thing I took away. And you see it through his entire career, where there's a reasonable amount of effort which goes by 20/80. You do the 20 to get the 80% of the effect. And then if you go beyond that to get, "Oh, no. I'm not just gonna do 20%, I'm gonna just do the whole thing." And there's a level even beyond that, which is an unreasonable use of time. This is going to have no ultimate impact, and still try doing that.

Trenton Bricken

You've shared on Twitter, using Anki. Or even, like, a Claude integration. Do you do book clubs? Do you use GoodReads? And what are you reading right now?

Dwarkesh Patel

I don't have book clubs. [Bud the SpaceBar edition] has just genuinely been a huge uplift in my ability to learn. Mostly because- it's not even the long-term impact over years, though I think that is part of it and I do regret all the episodes I did without using Speech Marking Cards, because all the insights have just sort of faded away. The main thing is, if you're studying a complicated subject, at least for me, it's been super helpful to consolidate. So if you don't do it, you feel like a general where you're like, "I'm gonna wage a campaign against this country." And then you climb one hill. And then the next day you're at a retreat, and then you climb the same hill. There might be a more kosher analogy. yeah. And then the other question was what am I reading right now?

Oh. My friend Alvaro De Menard, author of Fantastic Anachronism. Can I just hold it up? Actually it's right here. I hope he's okay with me sharing this. But he made 100 copies of this translation he did of his favorite Greek poet. Cavafy. Hopefully I didn't mispronounce it. Sorry, that one has a good inscription for Guern, because that's his copy, but it's super delightful, and that's what I've been reading recently.

Trenton Bricken

Any insights from it so far?

Dwarkesh Patel

Poets will hate this framing. I feel like poetry is like TikTok, where you get this quick vibe of a certain thing, and then you swipe. And then you get the next vibe, swipe… Alvaro, I'm sorry.

Sholto Douglas

No, that's interesting. How do you go about learning new things or preparing for an episode? You mentioned the one to two-week period where you're deep diving on the person. What does that actually look like?

Dwarkesh Patel

It's very much the obvious thing: you read their books, you read the papers. If they have colleagues, you try to talk to them to better understand the field. I will also mention that all I have to do is ask them questions, and I do think it's much harder to learn a field to be a practitioner than just learn enough to ask interesting questions. But for that it's very much the obvious thing you'd expect.

Trenton Bricken

“Based Carl Sagan” asks, "What are your long-term goals and ambitions?"

Dwarkesh Patel

AGI kind of just makes the prospect of a long term harder to articulate, right? You know the Peter Thiel quote about what is your 10-year plan and why can't you do it in six months?

Like, it's especially salient, given timelines. For the foreseeable future, grow the podcast and do more episodes, maybe more writing. So we'll see what happens after 10 years or something. The world might be different enough. So basically, podcast for now.

Sholto Douglas

Something you've spoken to me about, and particularly when you were trying to hire people for the podcast was what you wanted to achieve with the podcast. In what way do you want the podcast to shape the world, so to speak? Do you have any thoughts on that or... because I remember you telling me, "I really want people to actually understand AI and how this might change their lives."

Or, “what we could be doing now to shape the world such that it ends up better."

Dwarkesh Patel

I don't know. I have contradictory views on this. On the one hand, I do know that important decisions are being made right now in AI. And I do think, riffing on what we were saying about situational awareness, if you do something really good, it has a very high probability of one-shotting the relevant person, and people are generally reasonable. You make a good argument, it'll go places. On the other hand, I just think it's very hard to know what should be done. You gotta have the very correct world model, and then you gotta know how in that world model the action you're taking is gonna have the effect you anticipate. And even in the last week, I've changed my mind on some pretty fundamental things about what I think about the possibility of an intelligence explosion or transformative AI as a result of talking to the Epoch folks.

Basically, the TLDR is, I want the podcast to just be an epistemic tool for now because I think it's just very easy to be wrong. And so just having a background level of understanding of the relevant arguments is the highest priority.

Sholto Douglas

Makes sense.

Dwarkesh Patel

What's your sense? What should I be doing?

Trenton Bricken

I mean, I think the podcast is awesome, and a lot more people should listen to it, and there are a lot more guests I'd be excited for you to interview.

Dwarkesh Patel

Gotta give me your recs.

Trenton Bricken

So it seems like a pretty good answer for now.

Sholto Douglas

Yeah. I think making sure that, like, there is a great debate of ideas on not, not just AI, but on other fields, and everything is incredibly high leverage in value.

Dwarkesh Patel

Yeah, yeah, yeah.

Sholto Douglas

How do you groom your beard? It's majestic.

Dwarkesh Patel

I don't know what to say, just genetics. I do trim it, but-

Sholto Douglas

No beard oil?

Dwarkesh Patel

Sometimes I do beard oil.

Trenton Bricken

How often?

Dwarkesh Patel

Once every couple of days.

Sholto Douglas

That's not sometimes! That's pretty often!

Trenton Bricken

But, do you have different shampoo for your head and your beard?

Dwarkesh Patel

No.

Trenton Bricken

What kind of shampoo do you use?

Dwarkesh Patel

Anti-dandruff.

Trenton Bricken

Do you condition it?

Dwarkesh Patel

Yeah.

Trenton Bricken

How often do you shave it?

Dwarkesh Patel

Who put you up to this?

Trenton Bricken

We're giving people the answers that they want.

Sholto Douglas

Big shampoo. Big beard oil.

Trenton Bricken

Yeah, you can sell some ad slots to different shampoo companies and we can edit it.

Sholto Douglas

Maybe we sold an ad slot. Who knows?

Dwarkesh Patel

Sorry, you had this idea of merch. Do you wanna explain this T-shirt idea?

Trenton Bricken

Yeah, yeah, yeah. So people should react to this. Someone should make it happen. Dwarkesh wants merch, but he doesn't want to admit that he wants it. Or he doesn't want to make it himself because that seems tacky.

So I really want a plain white tee with just Dwarkesh's beard in the center of it. That's it. Nothing else.

Dwarkesh Patel

But you were saying it should have a different texture than the rest of the shirt.

Trenton Bricken

Oh, so when I was really riffing off it, where maybe a limited edition set can have some of your beard hair actually sewn into the shirt.

Dwarkesh Patel

Oh my God.

Trenton Bricken

That'd be pretty cool. I would pay. I would pay for that.

Sholto Douglas

How much?

Dwarkesh Patel

I've got, like, patches all over my beard.

Trenton Bricken

Depends on how much hair. If it's like one is in there somewhere, versus the whole thing. Like, "Do I have to dry clean it? Can I wash it on the delicate setting?" But really, I think you should get merch. If you want to grow the podcast, which apparently you do, then this is one way to do that.

Dwarkesh Patel

You think beard and beard hair in the future is necessary?

Trenton Bricken

Oh, yeah. Oh, yeah.

Sholto Douglas

Which historical figure would be best suited to run a frontier AI lab?

Dwarkesh Patel

This is definitely a question for you guys.

Trenton Bricken

Oh. No, I mean, I'm curious what your take is first. You've spoken to more of the heads of AI labs than I have.

Dwarkesh Patel

Yeah. I was gonna say LBJ. Sorry, it's a question who would be best at running an AI lab or would be best for the world or…?

Trenton Bricken

Yeah, what's, what outcome do you want?

Dwarkesh Patel

Because I imagine it seems like what the best AI lab CEO succeeds at is raising money, building up hype, setting a coherent vision. I don't know how much it matters for the CEO themselves to have good research taste or something, but it seems like their role is more as a sort of emissary to the rest of the world. And I feel like LBJ would be pretty good at this. Just getting the right concessions, making projects move along, coordinating among different groups to maybe- Oh, Robert Moses.

Again, not necessarily best for the world, but just in terms of, like, making shit happen.

Sholto Douglas

Yeah. I mean, I think best for the world is a pretty important precondition.

Dwarkesh Patel

Oh, right. Who’d be best for the world? There's a Lord Ackwood quote of, "Great people are very rarely good people." So it's hard to think of a great person in history who I feel [would] really move the ball forward and also I trust their moral judgment.

Sholto Douglas

Yeah. We're lucky in many senses with the set today, right?

Dwarkesh Patel

That's right.

Sholto Douglas

Like, the set of people today are both... they try and care a lot about the moral side as well as sort of drive the labs forward.

Dwarkesh Patel

This is also why I'm skeptical of big grand schemes like nationalization or some public-private partnership or just generally shaking up the landscape too much, because I do think we're in one of the better.. I mean, the difficulty of whether it's alignment or whether it's some kind of deployment, safety risks. That is just the nature of the universe is gonna make that some level of difficulty. But the human factors in a lot of the counterfactual universes, I feel like we don't end up with people... Like, we could even be in a universe where they don't even pay lip service, there’s not an idea that anybody had that you could have an ASI takeover. I think we live in a pretty good counterfactual universe, all things considered.

Sholto Douglas

... good set of game players on board.

Dwarkesh Patel

That's right. That's right.

Sholto Douglas

How are you preparing for fast timelines?

Dwarkesh Patel

If there's fast timelines, then there will be this six-month period in which the most important decisions in human history are being made. And I feel like having an AI podcast during that time might be useful. That's basically the plan.

Trenton Bricken

Have you made any shorter term decisions, with regards to spending or health or anything else?

Dwarkesh Patel

After I interviewed Zuckerberg, my business bank balance was negative 23 cents. When the ad money hit, I immediately reinvested it in Nvidia. So, that is the... sorry, but you were asking from a sort of altruistic perspective?

Trenton Bricken

No, no, just in general, like, have you changed the way you live at all because of your AGI timelines?

Dwarkesh Patel

I never looked into getting a Roth IRA.

Sholto Douglas

He brought us Fiji water before.

Trenton Bricken

Which was in plastic bottles, so...

Sholto Douglas

Dwarkesh has changed.

Dwarkesh Patel

Well, have you guys changed your lifestyle as a result?

Sholto Douglas

Not really, no. I, I just, like, work all the time.

Dwarkesh Patel

But you would be doing that anyways, or would you not?

Sholto Douglas

Ah, I would probably be going very intensely at whatever thing I'd picked to devote myself to.

Dwarkesh Patel

Yeah, yeah. How about you?

Trenton Bricken

I canceled my 401K contribution, so-

Dwarkesh Patel

Oh, really?

Trenton Bricken

Yeah, yeah, that, that felt like a more serious one. It's hard for me to imagine a world in which I have all this money that's just sitting in this account and waiting until I'm 60 and things look so different then.

Dwarkesh Patel

I mean, you could be like a trillionaire with your marginal 401K contributions.

Trenton Bricken

I guess, but you also can't invest it in specific things. And, I don't know. I might change my mind in the future and can restart it, and I've been contributing for a few years now.

Dwarkesh Patel

On a more serious note, one thing I have been thinking about is, how could you use this money to an altruistic end? And basically, if there's somebody who's up and coming, in the field that I know, which is making content, could I use money to support them? And I'm of two minds on this. One, there are people who did this for me, and it was kind of actually responsible for me continuing to do the podcast when it just did not make sense as there were a couple hundred people listening or something. I want to shout out Anil Varanasi for doing this. And also Leopold, actually, for the foundation that he was previously running.

On the other hand, the thing about what that blogger was saying, that the good ones you actually do notice. It's hard to find a hidden talent. Maybe I'm totally wrong about this. But I'd feel like if I put up a sort of grant application, I give you money if you're trying to make a blog, I'm actually not sure about how well that would work.

Sholto Douglas

There's different things you could do, though. Like, there's “I'll give you money to move to San Francisco for two months”. And sort of meet people and get more context and taste and feedback on what you're doing and it's not so much about the money or time. It's putting them in an environment where they can more rapidly grow. Like, that's something that one could do. I think you do that quite proactively in terms of you deliberately introduce people that you think will be interesting to each other and this kind of stuff, so… yeah.

Dwarkesh Patel

Yeah. No, I mean, that's very fair, and I, obviously I've benefited a ton from moving to San Francisco. And it's unlikely that I would be doing the podcast- at least on AI- to the degree I am if I wasn't here. So maybe it's a mistake to judge people based on the quality of their content as it exists now and just throw money at them- not throw money, but give them enough money to move to SF to get caught up in this intellectual milieu and then maybe do something interesting as a result.

Trenton Bricken

The thing that most readily comes to mind is the MATS program for AI research. And this seems like it's just been incredibly successful at giving people the time, the funding and the social status justification, to do AI safety relevant research with mentors.

Dwarkesh Patel

Oh, and you, you have a similar program…

Trenton Bricken

We have the Anthropic Fellows Program.

Dwarkesh Patel

That's right, yeah. And I know you're probably selecting for a slightly different thing, but I assume it's gonna be power law dominated. And have you noticed a pattern among the, whether it's the MATS fellows or your fellows, who is just like, "This made the whole thing worth it"?

What's your first take on something?

Trenton Bricken

I mean, there have been multiple people who Anthropic and other labs have hired out of this program. So, I think the return on investment for it has been massive. And yeah, apparently the fellows, I think there are 20 of them, are really good.

Dwarkesh Patel

But what is the trick to making it work well or finding that one person?

Trenton Bricken

I think it's gotten much better with time, where the early fellows, some of them did good work and got good jobs. And so now later fellows the quality bar has just risen and risen and risen. And there are even better mentors now than before. So it's this really cool flywheel effect. But originally it was just people who didn't have the funding or time to make a name for themselves or do ambitious work. So it was kind of like giving them that niche to do it. Seems really key.

Sholto Douglas

You can do other things that don't have to be money. You could put out ideas for things you'd be really interested in reading or promoting.

Dwarkesh Patel

Yeah, yeah, yeah. There's something coming there.

Sholto Douglas

Okay, there we go.

Dwarkesh Patel

So if this episode hopefully will launch Tuesday at the same time as the book- by the way, which you can get at stripe.press/scaling. But on Wednesday, which is the day after, hopefully there's something useful for you here.

Sholto Douglas

Any other questions we wanna ask?

Dwarkesh Patel

The thing I have takes on, which I rarely get asked about, is distribution.

Trenton Bricken

Distribution of AI?

Dwarkesh Patel

No, sorry. Like, Mr Beast-style distribution, where people, I think rightly, focus on the content, and if that's not up to snuff, I think you won't succeed. But to the extent that somebody's trying to do similar things, the thing they consistently underrate is putting the time into getting distribution right. I just take random takes about… for example, the most successful thing for my podcast in terms of growth has been YouTube Shorts. It's a thing you would never have predicted beforehand. And they're responsible for basically at least half the growth of the podcast or something.

Sholto Douglas

I mean, I'd buy that. Why wouldn't you predict it? Like, I mean, like, I mean, I guess there's the contrast of, like, the long form deep content and, like, YouTube Shorts and stuff. But I definitely think they're good hooks. Good content.

Dwarkesh Patel

I have takes on how to write tweets and stuff. The main intuition being write like you're writing to a group chat. To a group chat of your friends rather than this formal whatever.

Trenton Bricken

What else comes to mind here?

Sholto Douglas

Well, maybe it's interesting the difference between TikTok and YouTube Shorts.

Dwarkesh Patel

Oh, yeah. We've never cracked TikTok.

Trenton Bricken

Why not? Like, you've tried?

Dwarkesh Patel

Yeah. I mean-

Sholto Douglas

Tried? Have you done everything?

Dwarkesh Patel

No, I have not done everything.

Trenton Bricken

Have you read these poems? Maybe you're in a bubble bath with some beard shampoo on.

Sholto Douglas

Reading poems? That'd be incredible if you got any of that to go viral. You have to do that now!

Dwarkesh Patel

Manifest. Reading a poem, uncross your legs.

Trenton Bricken

Last episode it was the interpretability challenge, now it's Dwarkesh in a bubble bath.

Dwarkesh Patel

I gotta sell the book somehow, you know?

Trenton Bricken

But you literally do it like Margot Robbie-

Sholto Douglas

Yeah, exactly. Explaining the seed mechanism.

Dwarkesh Patel

Yeah, yeah, yeah, yeah, yeah.

Sholto Douglas

So what is scaling?

Dwarkesh Patel

And that's how you crack distribution.

Sholto Douglas

And that's how you crack distribution.

Trenton Bricken

Oh, but yeah, no, like when we did our episode, it launched and you were sharing interesting tidbits about how it was doing and the thumbnail you wanted to use and the title. And I think I even asked you to share more details because it seemed interesting and cool and subtle things. But it seemed like you also kind of just hated it. Like playing this game of really having to optimize all these knobs.

Dwarkesh Patel

So what I realized, I mean talent is everything, so I'm really lucky to have three to four editors who I'm just incredibly proud to work with. I don't know how to hire more of them. Like, they're just so good and self-directed. So, honestly, I don't have tips on how to correct that. I hired those guys. So one of them was a farmer in Argentina, one of them was a freshman math student in Sri Lanka, one of them was a former editor for one of Mr Beast's channels. The other is a director in Czechia who makes these AI animations that you've seen in the Notes On China, and he's working on more essays like that.

So, I don't know how to replicate that catch again.

Sholto Douglas

God, that's a pretty widely cast net, I gotta be honest. Damn.

Dwarkesh Patel

But they're all, goddamned, they're so good.

Trenton Bricken

And this was just through your challenges and just tweeting about?

Dwarkesh Patel

That's right. I had a competition to make clips for my podcast. I rounded up a couple of them this way. Yeah, it's hard, it's hard to replicate because I've tried… "tried," after.

Trenton Bricken

Why do you think this worked so well with the video editors? because you tried a similar approach with your chief of staff.

Dwarkesh Patel

Yeah. The difference is, with the video editor, I think there is this arbitrage opportunity where there are people… It is fundamentally a sort of, are you willing to work hard and obsess about getting better over time? Which all of them go above and beyond on, but you can just find people in other countries who are... and it's not even about the wages. Like, I've 10Xed their salaries or something like that. It's just about getting somebody who is really data-oriented, and there is this global arbitrage there.

Whereas, with the general manager... By the way, the person I ended up hiring, and who I'm super excited to work with, is your childhood best friend. Max Farrens.

Trenton Bricken

Max is so great.

Dwarkesh Patel

He would have plenty of other opportunities. There's not this weird arbitrage where you find some farmer in Argentina.

Trenton Bricken

Yeah. But yeah, it is striking that you were looking for a while, and then just kind of mentioned offhand that Max was looking for something new.

Dwarkesh Patel

This is gonna be like a total, 12-year-old-learns-about-the-world kind of question, but I genuinely don't know how big companies hire. Because I was trying to find this person for a year, and I'm really glad about the person I ended up hiring. But it was just like, if I needed to hire 100 people for a company, let alone 1,000 people, I just do not know how to find people like this at scale.

Sholto Douglas

Yeah, I mean, I think this is the number one issue that startup CEOs have. Hiring. It's just relentlessly the number one.

Dwarkesh Patel

Yeah. And the thing I was stunned with is how it didn't seem like my platform helped that much. I got close to 1,000 applications across the different rounds of publicizing it that I did. And a lot of, I think, really cool people applied. But the person that I ended up hiring was somebody who was just a reference, like a mutual friend kind of thing. And a couple of other top contenders were also this way. So it's weird. Like, the best people in the world don't want to apply, at least to things like this and you just gotta seek them out. Even if you think you have a public platform or something.

Trenton Bricken

Yeah. Yeah, I mean, the job might just be so out of distribution from anything else that people would do.

Dwarkesh Patel

That's right, yeah.

Trenton Bricken

So Aditya Ray asks, "How do you make it on Substack as a newbie writer?"

Dwarkesh Patel

I think if you're starting from scratch, there's two useful hacks. One is podcasting, because you don't need to have some super original new take. You can just interview people who do, and you can leverage their platform.

And two is writing book reviews. Again, because you have something to react to rather than having to come up with a unique worldview of your own. There's probably other things, and it's really hard to give advice in advance. Just try things. But, those I think are just, like, good, cold starts.

Sholto Douglas

The book reviews is a good suggestion. I actually use, like, Gwern's book reviews as a way to recommend books to people.

Dwarkesh Patel

By the way, this is a totally under-supplied thing. Because if anybody has book reviews. Jason Furman is this economist who has like a thousand Good Reads reviews. And I probably have visited his Good Reads on a hundred independent visits. Same with the Gwern book reviews or something, right?

So book reviews are a very under-supplied thing, if you're looking to get started making some kind of content.

Trenton Bricken

I like that.

Dwarkesh Patel

Yeah. Cool. Thank you guys so much for doing this.

Trenton Bricken

Yeah, this was fun.

Dwarkesh Patel

We'll turn the tables on you again pretty soon.

Sholto Douglas

How does it feel being in the hot seat?

Dwarkesh Patel

It's nice. Nobody ever asked me questions.

Sholto Douglas

Nobody ever asked how is Dwarkesh!

Trenton Bricken

Cool. Yeah, yeah, super excited for the book launch.

Dwarkesh Patel

Thank you.

Trenton Bricken

The website's awesome by the way.

Dwarkesh Patel

Appreciate it. Stripe.press/scaling

Sholto Douglas

Yeah.

Trenton Bricken

Cool.

Dwarkesh Patel

Thanks, guys.

Sholto Douglas

See you later.

Trenton Bricken

Thanks.

Humans have not succeeded because of our raw intelligence.

Marooned European explorers regularly starved to death in areas where foragers thrived for 1000s of years.

I’ve always found this cultural evolution deeply mysterious.

How do you discover the 10 steps for processing cassava so it won’t give you cyanide poisoning simply by trial and error?

Has the human brain declined in size over the last 10,000 years because we outsourced cultural evolution to a larger collective brain?

The most interesting part of the podcast is Henrich’s explanation of how the Catholic Church unintentionally instigated the Industrial Revolution through the dismantling of intensive kinship systems in medieval Europe.

Watch on Youtube; listen on Apple Podcasts or Spotify.

• Scale partners with major AI labs like Meta, Google Deepmind, and OpenAI. Through Scale’s Data Foundry, labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you’re an AI researcher or engineer, learn about how Scale’s Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkesh

To sponsor a future episode, visit dwarkesh.com/p/advertise.

I highly recommend both of these.

• The WEIRDest People in the World

• The Secret of Our Success

(0:00:00) - Humans didn’t succeed because of raw IQ

(0:09:27) - How cultural evolution works

(0:20:48) - Why is human brain size declining?

(0:32:00) - Will AGI have superhuman cultural learning?

(0:42:34) - Why Industrial Revolution happened in Europe

(0:55:30) - Why China, Rome, India got left behind

(1:21:09) - Loss of cultural variance in modern world

(1:31:20) - Is individual genius real?

(1:43:49) - IQ and collective brains

Dwarkesh Patel - 00:00:00

Today, I have the pleasure of chatting with Joseph Henrich, who is a professor of human evolutionary biology at Harvard University and an author of two of my favorite books, The Weirdest People in the World and before that, The Secret of Our Success. And I was just mentioning to you that I remember reading this many many years ago when I was in college, and at the time, I didn’t think I would get a chance to ask you questions about it. But the most proximal reason I wanted to interview you is, I recently had your colleague, David Reich, on and we were discussing certain things in the record of human history where he said, “Eventually, you’re just gonna have to have Joseph Henrich on and ask him these questions, because he’s the one who would know.” So let me ask you one of the questions which I was super intrigued by which he raised, and we didn’t come up [with] an answer to.

So one of the things he’s discovered through his genetic evidence is that 70,000 years ago across Eurasia, there’s so many different human species, from the Denisovans to the Neanderthals to the Hobbits, and then apparently, there’s this one group, which was potentially the size of 1 to 10,000 people in the Near East, which subsequently explodes, and now everybody who’s descended from Eurasia descends from this one group. And so I guess the question is like, what happened? What did they figure out?

Joseph Henrich - 00:01:18
A typical assumption when people think about this, if you put it in the Paleolithic, they assume that it has to do with some kind of genetic changes. Now, Reich’s lab, there’s no obviously big changes in the DNA, so it’s a little bit of a puzzle. Neanderthals, for example, had larger brains, and in primates larger brains usually go along with more computational abilities, more ability to solve problems. So the expanding variant out of the Middle East, out of Africa, might’ve actually been less able at an individual level to process information. But if you look back over the more recent period of human history, you can see that it’s a story of expansions of different populations.

So for example, in Africa, we have the Bantu expansion about 5,000 years ago, which actually eliminates a whole bunch of hunter-gatherer populations that previously existed in Africa. We have the remnant populations in parts of the Congo, in the Kalahari, in the Hadza, for example, in Tanzania. If you look at the Austronesian expansion, so that’s the peopling of the Pacific, that was the expansion of one group of people at the expense of others, and of course, the Neolithic expansion into Europe is another example.

So really human history is a story of these different expansions. And it could be that this expansion across Eurasia, which then led to interbreeding, so we know it’s the same species. Humans interbred with Denisovans and Neanderthals, as well as probably other species- there’s a ghost species in there. This could be just institutional changes, so if you have institutions, for example, that interconnect your population, you can maintain more sophisticated technology. And some paleoanthropologists, for example, have speculated- with some evidence- that the expanding populations had projectile weapons, so bows and arrows. And humans have periodically gained and lost bows and arrows in different parts of the world. So in Australia, for example, bows and arrows are never invented. In the New World, populations probably didn’t have bows and arrows, but then later develop bows and arrows.

Dwarkesh - 00:03:12
It’s also really interesting how some technologies you would just think of as extremely… I don’t know what the word is, but the new world, like not having the wheel or something. I guess it kinda makes sense with no domesticated animals, but again, it’s like such a…

Joseph Henrich - 00:03:28
Right. Although the New World, everybody has dogs, and you can pull carts with dogs. So I’ve never really bought that, and of course, you can, you can use llamas in the New World to pull carts as well. People do that today.

Dwarkesh - 00:03:38
Yeah, so what’s your explanation for why there’s no wheels in the New World?

Joseph Henrich - 00:03:41
So there were wheels on Mayan carts or on Mayan toys. My explanation is just the collective brain. So almost every single first invention of something big that we think is important for humans was invented in Eurasia. And Eurasia, as Jared Diamond famously pointed out, building on other people’s work, it’s the largest continent by far. It has the biggest population. It’s also oriented along an east-west axis, which allows ideas and people to more easily flow. And there’s a belt, which the historian Ian Morris calls the Lucky Latitudes, which runs from basically southern China all the way through to the Mediterranean. And ideas are just flowing back and forth there, across the center of Eurasia. But you also ended up with more complex state bureaucracies and the kinds of things that allow you to organize and move people around and whatnot.

Dwarkesh - 00:04:27
And what’s the explanation for why the collective brain leads to state capacity? Or is it the other way around?

Joseph Henrich - 00:04:34
Well, you can think of institutions that eventually lead to state capacity as just part of the innovative process of the collective brain.

So if you have more groups experimenting with more different ways of governing groups of people, gradually, you get the accumulation of the pieces that you can put together into different kinds of states.

Dwarkesh - 00:04:50
Got it. Okay, so going back to what happened 70,000 years ago, is the basic answer like, it could be something like bows, but we don’t know exactly what it was? Or is it-

Joseph Henrich - 00:04:57
Yeah. We definitely don’t know exactly what it was. So we know that this population expanded, and there does seem to be some tool indications to suggest more complex technology. Probably, technology usually goes along with social organization.

So for example, if you look at Australia, which is a continent of hunter-gatherers, there’s an expansion about 6,000 years ago out of northern Australia which eventually takes up seven-eighths of the continent, and they had a new social organization, including rules about who you marry.

You had linguistic exogamy, and rituals that interconnected populations. So rather than having local rituals and local myths, many communities would get together periodically to initiate the young men, and this would help bond that whole group. There’d be an exchange of technology, and teaching that goes on at these… they’d spent a few months in the same place so there was a lot of time for transmission.

Dwarkesh - 00:05:47
And then in terms of mechanism, so, if it’s the case that you have these sort of convalescing waves of expansion, and David Reich’s lab’s evidence suggests that this expansion was quite violent, based on whether the genes are passed down through the maternal line or the paternal line. When you think about the mechanism in which technologies develop or these social institutions develop, how much of it is like, look, there’s all these different groups that are trying different things, and one of them maybe figures something out, and then they just kind of explode? And how much of it is that within each group, they are experiencing sort of accumulations of learning over time, and it’s not necessarily a selective process, it’s more of a sort of accumulative process?

Joseph Henrich - 00:06:32
Well eventually the groups have to meet, and they’re gonna compete over territories, and a bunch of different things can occur. So groups can copy each other. So we know that from the ethnographic record and historical cases, sometimes a group will say, “Oh, those guys have really good technology tools.” Maybe they’ll get some migrants, or something like that, and then they can adopt the practices. So that definitely happens.

But there’s also plenty of evidence of violent conflict. And the Reich labs evidence and lots of other ancient DNA suggests that there are these dead-end genetic lineages. So, I mean, the Neanderthals are a dead end in a sense, although we interbred with them, so they’re in us in some sense, but they don’t have their own pure lineage or anything like that.

Dwarkesh - 00:07:14
And if you had to guess, whenever this big wave happens, is it more of a sort of concrete technology? I mean, especially if you consider the range of expansion that can often happen, right? It’s like literally that same group goes from Siberia to England. It’s hard to imagine it’s a single technology lets you explore this wide range. But it’s also the idea that some cultural artifact is enough that it gets transmitted over tens of thousands of years and that’s what gives you the edge. I’m curious how you think about, what could possibly be this driving engine?

Joseph Henrich - 00:07:47
Well the way I describe it in The Secret is that it’s a package of things. So another good example would be the Inuit expansion out of the north slope of Alaska. And they expand all the way across the Arctic and eventually get to Greenland. And they have a whole package of social practices which helps keep them interconnected, and then they have bows and arrows, which the group they are exterminating as they go along, the Dorset, doesn’t have. They have dogs and sleds. They have, along the coast they have boats, so they’re doing whaling. So there’s a whole package that puts together that allows them to out-compete and eventually exterminate the Dorset.

And one of the things that happens, though, is the Dorset probably had better technology, but they expanded, they spread out, their languages diversified, they lost contact, and they began losing technology. So you wanna see this as a dynamic pulse. There were probably expansions and then collapses, another expansion, another collapse, right? It’s not one giant long march to victory.

Dwarkesh - 00:08:38
And then this process, which has happened many times where the population gets cut off or something: When you say it’s a sort of cycle, is there some reason why, over time, that knowledge gets fragmented and breaks apart and populations disperse?

Joseph Henrich - 00:08:58
Yeah, I think there’s a cultural evolutionary dynamic that’s part of this, because languages will naturally diversify and then as they diversify, there’s less contact. People sometimes get inclined to marry endogamously, and especially if there’s enough people around locally, why go all the way over there? And those people are getting more culturally different, so they’re seeming a little bit like outsiders…

So the tough part that humans have always had is to stay unified because of the natural effect of geography and learning locally is gonna tend to fragment us.

Dwarkesh - 00:09:27
In the Secret, you described this interesting startup problem where if you don’t have that much accumulated cultural knowledge, developing the ability to do social learning isn’t as valuable. But if you don’t have the ability to do social learning, you don’t have that much accumulated cultural knowledge in your tribe or group. So how is this problem solved?

Joseph Henrich - 00:09:46
Yeah. So before I get to solving the problem, I just want to sketch for the listeners that the question is, why is this cumulative cultural evolutionary process that is so important for humans relatively uncommon in the natural world? It seems like just our lineage. I mean, there were a bunch of split-off lineages, but now it’s just us.

So to understand that is this idea that you just mentioned, where you’re imagining an increase in brain tissue that’s gonna be costly, and I can put that towards individually figuring the problem out for myself, or I can put it into learning from others. And in a world without very much cumulative culture, there’s not gonna be very much useful information in the minds of everybody else so I should use that brain tissue for individually solving problems. And so it’s hard to get this runoff where their brains get bigger for the purposes of learning from others. And then the question is, how do you get past that valley?

So the case I make in Secret is that, say, three million years ago, two million years ago, there were several factors that came together. The first factor is the rate of change of environments. So you get this increase in the fluctuation of environments, so you’re getting more environmental changes. And in cultural evolution, a lot of theory shows that there’s a certain rate of change which is favorable to cultural evolution. It’s gotta be slow enough so that the information of your parents and the previous generation is useful, but not so slow that you might as well just encode it in the genes.

So that’s one. Second thing is, we’re a ground-dwelling ape, which means we have hands like chimpanzees and gorillas, and we can potentially use tools and whatnot. But unlike them, our ancestors may have been savanna-dwelling apes, which meant we may have lived in large groups. So in mammals, mammals live in larger groups when they have to deal with predators and the predator guild in Africa at that point was quite thick. There were a lot of deadly predators.

So paleo-anthropologists think that our ancestors may have lived in large [groups of] savanna-dwelling apes. And if you have a lot of individuals, if the culture is sparse, the bigger the group, the more chance that there’s someone doing something useful in the larger group. And so that means it’s easier to get across the threshold. So those are three of the main factors that might have allowed our lineage, as opposed to all the other lineages around, to cross this. We’re already a big-brained primate, we had hands, we’re living in these large groups on the savanna, and the climate changed during this period, so as to make us, yeah.

Dwarkesh - 00:12:18
And how big were these groups?

Joseph Henrich - 00:12:20
Well, I mean, nobody’s really sure, but maybe a few hundred individuals.

Dwarkesh - 00:12:24
Okay. And before the agricultural revolution, was there still this transmission of information across different groups?

Joseph Henrich - 00:12:34
Yeah. So lots of different evidence suggests that groups were moving trade goods. So well back into the Paleolithic we see trade. Often we see some genetic transmission.

Dwarkesh - 00:12:46
Right, the expansions and so forth.

Joseph Henrich - 00:12:48
Yes. And so, talking about the expansion, that’s something where the ancient DNA is useful because the Neanderthals in Europe, the DNA suggests they lived in very small groups. But the DNA of the expanding groups suggests a larger population. So those would’ve been two different collective brains there as well.

Dwarkesh - 00:13:04
So in The Secret, you discuss a lot of these lost European explorers with modern technologies as of at least a couple centuries ago encountering peoples who have for tens of thousands of years discovered ways of hunting and processing foods and so forth that, without which even these people with modern technology will starve. And as I was reading that I was wondering: I’m not sure how I understand the process by which, if there’s a 10-step process to making sure this bean is actually nutritious, and without any one of those steps, you might poison yourself or something. And at no point do you understand why this process works. You don’t have a scientific explanation. How do you even learn that in the first place?

Joseph Henrich - 00:13:49
Right. So one of the things we know, even in young children there’s a tendency to preferentially learn from healthier, more successful individuals. So if you’re processing it better- so something like bitter cassava, which has cyanide in it. If you just eat bitter cassava,it won’t taste great. So if you then rinse it somewhat, it’ll taste better and maybe you could eat it, but you’re going to accumulate cyanide over the long run. So it doesn’t kill you right away. But if you do this whole long process that the populations in South America developed, then you’re totally fine, you never get any accumulation.

So you can imagine that initially this is gonna be very strong and people are gonna sort of do sensible things… but then it gets a little more mysterious. And we know this because bitter cassava gets transported to Africa, and Africans immediately begin eating it improperly processed and getting goitre and the cyanide processing that goes along with it. So then that’s gonna be a slow evolution where groups that do this are gonna be more successful and individual families are gonna be more successful. So, for example, you might have a household where they process the cassava more seriously than another family, and they’re gonna have more kids and they’re gonna be like, “Oh, that family’s really good.” And then people copy what they do in all kinds of ways, but one of them could be copying recipes.

Dwarkesh - 00:14:59
Hmm. I guess it depends on the mechanism of selection here, because when you consider the different ways in which two different individuals might be different or two different households or even two different groups, I guess it makes sense why cyanide poisoning is such a deleterious effect that it is a noticeable or quite a strong signal. But I think you discussed some other ideas in The Secret where it’s like, the sort of spices in a region sort of match the antimicrobial properties or the antifungal properties you need to stay, I guess, clean or whatever in that particular region. But it sounds like a small effect, and how could such a small effect actually create a strong enough signal that when you’re deciding who to copy, you notice that this family is healthier because they spice their food in a certain way? Or this group takes over the area because they’ve…

Joseph Henrich - 00:15:59
Yeah, I mean, it has to be a big enough effect to matter, but I think if you go back to a world where there’s a lot of improperly processed meat, people don’t have refrigerators, and the leading cause of death in children is diarrhea. People carry high pathogen loads. If you can knock that pathogen load down, we know from modern research if you wanna make people healthier and even smarter, right? IQ goes up if you knock the pathogen load down.

Dwarkesh - 00:16:22
Got it. So this process of cultural accumulation you’re talking about: you think it is not strong enough to pick up minor increases in fitness and it actually has to be quite a significant thing for it to…

Joseph Henrich - 00:16:39
I think it operates a lot like natural selection, in the sense that natural selection will pick up tiny things if you give it long enough. Cultural evolution will do the same thing if you give it long enough, because just small differences in who you’re paying attention to will affect things, but it might take 1,000 years as opposed to decades, right? But we know culture can spread adaptive traits super quickly if it’s a really big effect.

Dwarkesh - 00:17:01
Yeah. So the situation in which our ancestors found themselves, in some ways it’s sort of like epistemic hell, in the sense of you don’t know why certain things are working, but you do know that if you break with tradition, you might just doom yourself and your family. And that maybe, as you discuss, causes these religious beliefs and taboos and mystical understandings of the world to rise, where you just think like, “Look, you’re gonna burn in hell if you don’t do this 10-step process for refining your beans or something.” In some sense, this requires you to abdicate reason because you have to just be like, “I don’t understand this. this is the way it’s been done, so we’re gonna do it.” Basically: how much reason do you necessarily have to abdicate to survive in the ancestral environment?

Joseph Henrich - 00:17:41
I mean, one of the ways we research this is by going out to societies that make bows and arrows or have food taboos that protect them from dangerous marine toxins and ask them questions about why they do it and see if they [understand] and they don’t understand the underlying causal stuff.

Dwarkesh - 00:17:53
Right. But then why do they say they do it?

Joseph Henrich - 00:17:55
Well, a typical answer is, “It’s our custom.” or “It’s important to people around here to do it this way, so we do it this way.” So just Hadza bows, for example- I didn’t do this research, Jacob Harris and Kim Hill and Rob Boyd did it- but they asked Hadza about their bows and about how they work and what the mechanics are, so they understood some stuff. So you use the bow and you get some mechanical understanding. But if you asked him, “What if you used a different wood, different materials,” and they had never tried anything else but what they learned, how to make this bow. So they couldn’t speak to a lot of that stuff. And the things about the compression of the wood is very important- didn’t understand that.

Dwarkesh - 00:18:34
Okay, so that’s quite interesting, right? Because you would think that you have to transmit cultural knowledge over time, but you also need to experiment in order to innovate. But it sounds like because of the belief in these customs, you would be less inclined to innovate.

Joseph Henrich - 00:18:48
Right. And the thing is, once something gets good, doing it differently almost always makes it worse. And then there’s also, some things are just different because people do it randomly different, like they make a mistake. So I think people often underestimate the power of error in generating novelty.

Dwarkesh - 00:19:04
One question I have is why this process of cultural learning… So humans have extended childhoods, presumably to give us more time to accumulate culture, and then we live after menopause, presumably because older people, our grandparents, can teach us about the situation we find ourself in. And 18 years is such a long time, and when I think back to- I mean, obviously I didn’t grow up in the ancestral environment, but when I think back to what I was doing as a teenager I don’t think I was learning that much. I was just less productive for no obvious biological reason. So I guess I don’t intuitively understand why this process of cultural learning takes decades, and why it can’t happen more rapidly.

Joseph Henrich - 00:19:47
Well, you probably, as an adolescent, were in school, right? So you were probably accumulating some stuff in school- maybe. And here’s an interesting fact about hunters. So anthropologists who studied hunters, hunter-gatherer societies in different places, and the physical peak that at least males have is in their early 20s, right? That’s when they run the fastest. They got the best eyesight. But the best hunters in the community are 36 to 40, right?

They’re not as fast as those guys, but they just know, they know the track, they know the animals, they know exactly what to do in these different- And then hunting skill begins to decline because basically the physicality of things begins to catch up, and so there’s this kind of cycle. And at 18 young hunters aren’t even producing enough food to feed themselves. So it’s not until they get into their 20s that they’re actually in surplus and bringing food home for everybody else. And you need to know hundreds, at least, animals, you need to know animal behavior, you gotta be able to spot tracks and spoor, and we’re just very knowledge-dependent in terms of our hunting and gathering.

Dwarkesh - 00:20:48
Yeah. Given what you just said and also the experience of these European explorers suggests that you just gotta know a lot of shit to make it in any sort of environment on Earth. But then, there’s all these other animals and they seem so dumb and they seem able to get by. So why did humans have such a hard time of it?

Joseph Henrich - 00:21:07
Yeah, well, so I think we offloaded a lot of stuff into culture. So in one of my Lost European Explorers, there’s a case where they’re in Australia and they have camels. And the camels escape, and now central Australia has lots of feral camels. So the camels survived the lost European explorer challenge, because they have innate instincts. They can smell water a mile away. They can detoxify foods in their own- they have a complex digestive system that detoxifies- we’ve lost all that, and we’re worse than chimpanzees at detoxifying foods because we have all these cultural practices that do the work. So we’ve externalized detoxification and a lot of digestion, actually.

Dwarkesh - 00:21:41
I had this independent researcher and internet writer called Gwern Branwen on my podcast a couple months ago, and we were discussing AI. And I asked him- so his theory is very much like the brain became bigger, more intelligence. And so then I asked him, “Look, if it’s this simple, why did it take so long for evolution to discover intelligence in the first place?” And he had this interesting answer, which is that in terms of the signal that evolution is giving you, there’s a very narrow gap between skills that are so useful that they should just be distilled as an instinct into your genome, and then skills that are so worthless that are not worth learning in the first place. And so this narrow gap that’s like, you need that generalization ability. It’s not so primal that, you know, you’re not gonna culturally learn hunger, it’s just gonna be in your genome. I don’t know if that sounds like an interesting explanation or helps explain anything.

Joseph Henrich - 00:22:41
Well in cultural evolutionary models, a typical thing would be to let genes compete with individual learning to compete with cultural learning. And it turns out the rate of change of the environment affects that. So if the environment is changing slowly, then you should put it all in the genes. And if it’s changing at moderate speed, then culture is the best way to go, and if it’s fast, then individual learning. And the idea, the individual learning is favored because if it’s changing too quickly, the previous generation doesn’t know anything worth knowing because the world you’re dealing with is just so much different from their world. And so it’s this intermediate range.

So myself and others have argued that the increase in the frequency of change during the Plio-Pleistocene transition 2.5 million years ago is a change that increases the value of cultural transmission of learning from others. I’m not sure what your guest was thinking. I mean as our brains are not very good at solving problems, otherwise the lost European explorers could survive. And human brain size has been declining for the last 10,000 years. So we’ve actually been getting dumber. Fewer neurons, less computational power.

Dwarkesh - 00:23:48
Why has brain size been declining?

Joseph Henrich - 00:23:50
Well, the collective brain argument suggests that at a certain size, you’ll begin farming off specialists. So because there’s a store of knowledge in the society, and we can all be generalists and learn how to do all the different skills. But at some point it makes efficiency sense for us to specialize in different skills.

In order for that to be the case though, we have to have social agreements of some sort that allow us to trade or exchange things like that. So, but then once we’re specializing, we don’t necessarily need as large a brain and we, because we distribute the overall brain power amongst the society. So it could be that we’re becoming more of a superorganism. And you see the same thing in ants. When ants get specialized occupational casts, their individual brains shrink.

Dwarkesh - 00:24:35
Yeah. I think David Reich’s lab had a result a month or two ago where they showed that the selective pressure on at least the European samples, which is the samples they studied, was that there’s been selection for greater intelligence over the last 10,000 years. I don’t know if you saw that or what you make of that?

Joseph Henrich - 00:24:52
Well, they didn’t say anything about intelligence. They did use the polygenic score for education. But there was no education 8,000 years ago.

Dwarkesh - 00:24:58
Which is correlated, right?

Joseph Henrich - 00:24:59
Yeah, it’s correlated. But then the question is, we don’t know what goes into that, right? Is that actually computational software? I mean, people do well in school because they stick to it and they can sit in the same place for long periods of time.

Dwarkesh - 00:25:12
I guess it would be interesting for them to sort of deduct the docility hit polygenic score from the education, because we don’t have a polygenic score for intelligence yet, right, so which is why you need some proxy.

Joseph Henrich - 00:25:22
Well there are IQs, so there are polygenic scores for IQ. I think they actually did that one as well, like you said, it’s correlated with education. But the things that go into giving you an IQ, I mean, the correlation between brain size and IQ is only about, well, across populations it’s .24, so it’s not very big.

The other thing is IQ is massively misunderstood, so the way to think about IQ, which has been going up over the 20th century, by quite a bit, so if you rescale modern scores to 1900, it’s about 70, it’s called the Flynn effect. I think that those are cognitive abilities for navigating the institutional world that we’ve constructed.

And a huge mistake would be to assume that those are the right cognitive abilities for the next century. So it’ll be a different constellation. We study herders and cognitive abilities of herders in northern Namibia, and if you’re in northern Namibia, you gotta be able to move through the landscape, so being able to just pick a direction and know where you’re going and not get lost is super important.

So different suites of cognitive abilities are favored in different environments. This idea that there’s this generalized thing applicable across all human environments just doesn’t… I mean, education massively increases your IQ, right? If you’re uneducated, you have a totally different IQ.

Dwarkesh - 00:26:42
And then is the way in which your horsepower is directed, is that a thing that you think is basically set by the time you’re an adult? Or, if somebody’s 30 now and the AI thing is gonna happen next and they have to totally reorient away from knowledge work, is that a thing?

Joseph Henrich - 00:26:58
Well, I think it’s a continuous scale. If you look at human brains, they’re developing and continuing to add new connections and stuff, at least into the mid-20s. Now, there could be even more plasticity after that. Unlike chimps, we’re still not totally myelinized for our whole lives. So there’s definitely room. It’s just there’s less room.

Dwarkesh - 00:28:20
The kind of social learning for which our learning biases are fit, right? Like where you pay attention to elders because they survived somehow and they’ve accumulated the knowledge from past generations, how much do you think that actually basically applies to the way in which the modern knowledge work economy accumulates knowledge? And how much do you think is just an artifact of the kind of environment our ancestors found themselves in? Right now we’re in Silicon Valley, or in San Francisco, and it’s very common here for a 20-something to make a new product or service which, without that much- they don’t have that much context of how the world works, but they can make big, big innovations and big changes. But at the same time, people on average maybe are more productive in their 40s and 50s in terms of wages or something. So basically, this sort of social learning you’re describing, how much does it actually describe the world as it exists today?

Joseph Henrich - 00:29:17
Yeah. And so, super important, so we’ll take the patent database and if your same-sex parent- has to be same sex- patents in a particular domain, you’re nine times more likely to patent in that same domain.

Dwarkesh - 00:29:30
And how do you separate the genetic versus cultural effect there?

Joseph Henrich - 00:29:32
So these are very fine domains, so we’re talking about, like, natural adhesives versus synthetic adhesives. So unless you think there’s genes for that. And same thing with transistors and electronics, so very fine domains. And if you grow up in Silicon Valley, you’re much more likely to patent, in general, but you’re probably gonna patent in computers. But if you grow up in Boston, it’s gonna be biotech. So having a father or mother who did biotech and then growing up in Boston, you’re even more likely to patent in biotech. And this is true even if you move to New York. So if you look at people in New York who are from Silicon Valley and from Boston, the Bostoners are more likely to patent in biotech, and the Silicon Valley kids are more likely to patent in computing. So all the same rules apply.

Dwarkesh - 00:30:20
Well, I’m not sure because maybe the location, you learn a lot from the environment. But the fact that it’s just kids, and so, if you think these old rules of cultural accumulation and collective brains apply, you would think, “Ah, you’re 50 by the time you’re writing your first useful line of Python code or something.” When in fact, a lot of these big innovations come from people who are much, much younger, before they should have accumulated much of the know-how that this theory implies.

Joseph Henrich - 00:30:48
So the model here is what to focus your efforts on, so it’s the throwing model. So if you grow up in Silicon Valley, you focus your efforts on learning how to code or whatever. And so that means that’s where you’re likely to make the innovation.

Dwarkesh - 00:31:03
Right. But in terms of, you know, when there’s 2% growth a year and a lot of the technologies that are dominant in the economy today, or a lot of the industries, didn’t even exist many generations ago. Basically, does that suggest this model of- should we just mistrust our instincts of who to give prestige to and what kinds of people we want to pay attention to, whether it’s professors or elders or whatever?

Joseph Henrich - 00:31:28
Yeah, so that’s definitely something I’ve thought about and I wrote about in Secret, which is that, as the rate of cultural change gets faster- and we sort of talked about this with environmental change- the value of older and older members of the previous generation declines because the world that they grew up in, and that they honed their skills to, is quite different from the current world. So you would expect the degree to which, I mean, optimally, you would look less far back or you would look to relatively younger individuals to get your inspiration from because the world they adapted to is closer to the world you’re gonna need to adapt to.

Dwarkesh - 00:32:00
Yeah. And speaking of Silicon Valley, let me ask you a little bit about AI. So one of the reasons to suspect some incredibly sharp discontinuity from the world as it exists today to a world with AIs. And by AI, I don’t mean just, like, GPT-4, I mean, like, replacement at least for anything you can do on a computer screen, the AI can do.

One of the reasons we expect this hard discontinuity is that they have potentially the step function increase in social learning and the ability to accumulate knowledge that maybe humans had, or the magnitude of which maybe humans had when, between them and non-human primates. And in particular, the fact that you can just copy everything you know. Like you don’t have to teach a young person, right? The constraints of biology mean you can’t just replicate brains.

You have much more efficient communication. You don’t have to communicate through words, you can just shoot your brain state across, the population size can grow arbitrarily large. So to the extent of, the collective brain is the size of the population and how interconnected it is, do you just expect we wouldn’t even recognize the kind of world these AIs could make as a result of their cultural skills?

Joseph Henrich - 00:33:15
Yeah. No, I mean, I, I definitely think it’s pretty interesting, and holds great potential for expanding the collective brain. There are little things in there which might make one worry. So if you study the history of innovation, you find out that, for example, serendipitous meetings are super important. There’s a great paper on Silicon Valley showing that companies will cross-reference each other’s patents more likely when the people at those companies tend to frequent the same coffee shops, and they track people on their cell phones and stuff to figure this out. So serendipitous meetings are important and improper copying. So a huge number of innovations are mistakes where somebody copied incorrectly and then got something better.

Dwarkesh - 00:33:55
I read this interesting theory that maybe evolution designed transcription and translation to have more errors than it could otherwise have, just so that you can have this sparse reward when you’re close to the right sequence.

Joseph Henrich - 00:34:08
Yeah, yeah. I think that’s a very interesting area of research, and it makes good sense to me.

I’m not sure of the current state of the evidence, but there are… Different parts of the genome are more or less susceptible to mutation, which is kind of interesting.

Dwarkesh - 00:34:21
Yeah. So going back to AI, maybe then another way to phrase it is, look, you’re talking about these serendipitous meetings where you can learn something another person knows. And the great advantage these AIs have is that they can sort of meet with everybody at once. Future versions of AIs, you could really imagine holding the whole internet in context, right? I mean, we’ll be the equivalent of those people isolated in Tasmania according to the AIs, right? Because they just have everything in context. You can get a PhD in every single field, and you can amortize that across all your copies.

Yeah, so are you sort of banking on, in the next 10 years, you’re just gonna be living in the singularity or something because of your belief in the value of cultural knowledge?

Joseph Henrich - 00:35:03
Yeah, I see there’s various potentials, and I’m particularly interested in using AIs to augment problem-solving in human groups. So you can imagine getting the humans together because the humans have the big advantage of having stuff they care about, right? There’s stuff they want to invent. So the AI is still a tool at some point. So I’m interested in that, but I’m interested in how these things like running out of training data is gonna be dealt with, the value of making mistakes and serendipity, if you get rid of all those things or how you’re gonna reintroduce them, those kinds of things.

Dwarkesh - 00:35:39
Yeah. I mean, it’s quite funny because until recently, people were saying the big issue with LLMs is that they hallucinate and make mistakes. And at some level, hallucination is no different than creativity.

Joseph Henrich - 00:35:50
Well, may- maybe there’s a way to harness that, right? But maybe we just didn’t know how to harness it.

Dwarkesh - 00:35:53
That’s right. What do you make of the idea that you could have AI firms or AI tribes, whatever way to think about it- is if the effective population size that can communicate with each other is such an important contributor to how much progress a group in history was able to make. If you could just literally run billions of AIs and they have this immediate ability to communicate with each other. And again, I’m not talking about current models, I’m talking about future, human equivalent. Basically I guess I’m just throwing out a bunch of different intuition pumps and I’m curious which one you find most promising or most interesting, or did you just find all of them not as convincing?

Joseph Henrich - 00:36:31
Yeah, no, I don’t have strong opinions about any of this, but I do think that one thing in order to make all that work is, humans are constantly getting hit with shocks, right? So there’s economic shocks, there’s weather shocks, there’s conflict with other groups, there’s pandemics. And the shocks have big effects on how things go. They inject new information into the system. So I’d worry about a system that is too homogeneous and doesn’t have enough noise adding shocks and new challenges and things like that.

One of the things missing from our conversation is the kind of creativity that cultural evolution has figured out. So to give you an example, at some point in human history, religions invented big, powerful, moralizing gods. And this may have increased the people’s ability to cooperate. So forget about the technological element. Some of the most important features of innovation over human history have been institutional and things that get people to cooperate.

So in my work, I argue that one branch of Christianity resulted in the transformation of the families in European societies into small monogamous nuclear families. I’m not sure an AI would have thought of that. Cultural evolution thought of it because of how it affected how the societies operated, so it sorted this out over historical time.

Dwarkesh - 00:37:48
Yeah. Can I play with that idea?

Joseph Henrich - 00:37:49
Yeah.

Dwarkesh - 00:37:50
Because that’s quite interesting. So again, I think this might even reinforce the advantages of AIs in this sense. So, to the extent that this sort of random fluke by the church led to this modernity and the great divergence or whatever, and it was a result of, there’s a bunch of variation, and then you can select over some group that’s doing the right thing.

The advantage you have with AIs is that you can have much more high fidelity replication of culture and you can explore a wider range of potential cultures. Sorry, and it sounds really vague when you talk about AIs in this way. So what do I mean? One problem companies have today is that, suppose a company’s working really well. Maybe it’s early Apple, or early Tesla, or SpaceX or something. And suppose it has to do with the culture, the people, something, it’s not clear how you replicate that culture, not only across different institutions, right? If I’m running a company, I don’t know how to copy SpaceX. Not only can you not replicate it across institutions, you can’t even replicate it across time.

So over time, many institutions tend to decay, their culture fades as people leave or die or something. And then imagine if all of SpaceX, at least maybe at the time you thought it was the most effective, are just AIs where you know every single byte, and when you can make a copy of it 1,000 times, and you can put it against every single hardware problem we have in the world. And then you can, if you think there’s another team that might have some different culture which is better, you can do the same thing with them. And culture here includes not only how they think about their work, but also even how does the board make decisions, right? Do they do this Monte Carlo tree simulation? And there’s so many different things you could do here. So because of the wider range of possibilities you can explore and the fact that you can have higher fidelity transmission of cultural information, maybe the ability to do this random evolution also increases. But I’m curious to get your take on that.

Joseph Henrich - 00:39:57
Yeah, as a kind of broad sketch, I think that is pretty interesting. I worry about too high a fidelity replication just because it’s important to take the details of historical context and time into account. So that same thing, if you fix it, it might not really work a decade later when everything else around it has changed, right? So it’s kind of a moving target. But you could fix that just by having lots of different variants that are different versions of that. So yeah, that would be cool.

Dwarkesh - 00:40:26
Sorry, that gives us a good opportunity to go into WEIRD. But before we get there, one more question about before the agricultural revolution. You know, obviously many groups around the world were different and many of those differences were probably inspired by the fact that they were living in different environments, so they needed to come up with different technologies and adaptations to survive best there. How much just random differences that had nothing to do with their environment do you expect there to have been, in terms of… I don’t even know what the right category would be, but in the modern world, you might imagine something like, does a society have slavery or not? And maybe pre-modern societies all had slavery, so that’s not the right way to think about variation, but that kind of…

Joseph Henrich - 00:41:07
Yep. So I think there’s lots of reason to think there was lots of variation. And one of the ways that researchers study this is they look at ethological variables and they compare it to phylogenetic variables. And here they mean cultural phylogeny.

So if your ancestral populations had matrilineal, matrilocal social organization, how likely are you to have it, controlling for ecology? And it turns out both of these things matter. People adapt to their ecology, but you can still see the signal of past society. So some degree of fidelity of transmission of social institutions, how we make our baskets, those kinds of things. There’s lots of examples in the modern world, and we see this all over the place. So if you look at gender inequality in the modern world, you find that a history of the plow leads to greater gender inequality. So males had a particular advantage in using the plow because it requires upper body strength.

This meant males became the dominant force in economic production at the household level. And that even today in populations where most people aren’t farmers, this persists. So this is cultural persistence that has to do with whether you had the plow. So this is work done by Alberto Alesina and Nathan Nunn. A former postdoc of mine now at the Harvard Business School, Anke Becker, has done a similar thing with pastoralism. So if your ancestors were pastoralists, and pastoralists, for various reasons, have quite strong gender norms that persists into the modern world and still shapes, for example, female entrepreneurship.

Dwarkesh - 00:42:34
Interesting. Yeah, and the reason I’m especially curious about this is because it informs the following question. If the Industrial Revolution didn’t happen in Europe but started somewhere else, how different would the world look like today? So obviously you discussed the fact that breaking apart these kin ties was necessary for the Industrial Revolution, but to the extent that was necessary, whichever place had the Industrial Revolution first would have had that, right? But, I mean, separate from the technologies or cultural practices which were necessary for the Industrial Revolution in the first place, still how different would the world look? How much variation was possible given our level of technology?

Joseph Henrich - 00:43:13
Yeah, I have a lot of trouble trying to answer that because, I mean, China or some place in the Middle East might have been the obvious alternative place for the Industrial Revolution to happen. But I feel like you have to give those places a lot of stuff that developed only in Europe.

So for example, universities begin spreading in the High Middle Ages. So you’d want them to have universities. You’d want them to have universal schooling, which began to spread and wasn’t present in these places prior to that. So that begins to spread in the 16th, 17th century. And so by the time you add all this stuff, it basically starts to look a lot like Europe.

And these are all things that are global now, right? So the universal schooling that we find around the world today, you know, begins with the Protestant Reformation in Germany and then later England. University’s models are the European university, and globalized.

Dwarkesh - 00:44:07
I guess one obviously very salient example of variance which might not have been replicated, but I’m curious if you think it might have been, is, it seems like the British Empire was the first major institution which decided that slavery is just morally wrong and we’re gonna throw our weight around to eliminate it. And I don’t know if you think that sort of naturally follows the development of social technologies that the Industrial Revolution would have brought about, but that seems super contingent. But I’m curious if you disagree.

Joseph Henrich - 00:44:40
Yeah. Well, so my story is that the rise of the Industrial Revolution in Europe has to do with the consolidation of Europe’s collective brain. And one of the things that requires is trust in strangers, and at least the beginnings of moral universalism.

And it’s that moral universalism that eventually causes the British to say, “We’ve got to stop with the slave trade thing.” It’s a moral decision that they made because it’s no longer consistent with the changing moral values over time.

Dwarkesh - 00:45:07
Right, right. Okay, so we’ve been dancing around the thesis of your book following The Secret, which is The Weirdest People in the World. And before we really jump into it, maybe you can give me a summary of what the thesis there is.

Joseph Henrich - 00:45:20
Yeah. So the first observation is that there’s a great deal of global psychological variation around the world. So European, American, Australian populations tend to be highly individualistic. They’re inclined towards analytic thinking over holistic thinking. They have a lot of impersonal prosociality, so trust in strangers. They’re against conformity, willingness to cooperate with strangers. So the question is, how can we explain the global variation in these features of psychology?

And towards the end of the book, I actually connect these features of the psychology to economic differences, including the Industrial Revolution that happened in Europe, which reshapes the world. And the story is that the key event is the spread of a particular form of Christianity into Europe, where the Catholic Church- what becomes the Catholic Church- systematically dismantles the intensive kinship systems in Europe, leading to small monogamous nuclear families.

And this transformation leads to the creation of new institutions. So by the high Middle Ages, you get the rise of guilds, which are voluntary groups of craftsmen and self-help societies, because people don’t have their families to rely on. People begin moving around. There’s occupational sorting into different occupations. You get urbanization rising. Charter towns are on the rise. Universities pop up. New kinds of monasteries pop up. And then Europe begins to urbanize, and you get new kinds of law that are based around the individual. Contract law. And then eventually this leads to a lot more innovation because ideas are flowing around Europe and then eventually the Industrial Revolution. So that’s the argument in a nutshell.

Dwarkesh - 00:46:53
Can you explain again what exactly the Church did which led to the kin-based existing system breaking down?

Joseph Henrich - 00:46:59
Yeah. So the Church outlaws polygyny, and so that stops elite males from having multiple wives and concubines and whatnot, and creating kind of a giant family through that. It outlaws cousin marriage going all the way out to sixth cousins at some point, and that included spiritual kin and other kinds of non-genetic relatives as well as the cousins. It frees up inheritance, so it has inheritance by testimony rather than normal patrilineal inheritance. And a simple example here is in most societies, you inherit access to land corporately, which means you and your brothers and stuff all own the land. And it might be your uncle is actually in charge of the land, your father’s brother. And so you can’t sell it, you can’t move it around, and you’re also tied to it. It’s where your ancestors are buried, and so there’s this big importance of land. So the Church allows people to give land to the Church, the Church becomes the largest landowner in Europe, because you can do it by testament.

So those are just some of the examples of the ways it transforms the family structure, and eventually you get these monogamous nuclear families which are basically unheard of around the world, at least if you look at the anthropological record. And you can really see this when you can compare, individuals can move to European cities as individuals or nuclear families. In China when you move to the city, you maintain contacts with the folks back home in the village, and people flow back and forth and you get these little enclaves of different clans and stuff in the cities, the connections. And this is really important because your ancestors, and there’s rituals that have to be done back in the home community. But Christianity does away with all those ancestor rituals.

Dwarkesh - 00:48:35
Okay, great. Let’s jump into it by starting before- much before- even the church. I wonder how much of the things that make Europe weird existed even in antiquity. If you read about how Roman society worked, or how these Greek polities worked, already you have an emphasis on nuclear families. You have these sort of universal norms around, “Hey, we believe in republicanism,” or, “We believe in democracy.” And the structure of the system matters more necessarily than… I guess at least before the empire, right? So yeah, was Europe already weird before the things that its church did?

Joseph Henrich - 00:49:19
Well, there were certainly some interesting things going on in Greece and Rome. But I don’t think there’s good evidence that you had the kinds of monogamous nuclear families that you would find later. I mean, even European law is built around patrilineage. So if your father’s still alive, you are not a full citizen. You’re in the pater house of these large families. Definitely intensive kinship, for sure, patrilineal, patrilocal.

Women, of course, don’t have any rights. And republicanism, but the formation of Rome is built around a series of elite families. So it’s a clan operation, and they call it republican because the clans all have some say in what’s going on. So people sometimes see representative government where it’s not really individuals being represented, but families. And so this fools a lot of people into thinking there was a lot more individualistic voting and things like that, whereas there’s actually no voting. So in Greece, in Athens, Athens is unique because it does a bunch of things in the laws of Solon, that break down the intensive kinship. So for example, in Greece, you get monogamy for the first time, and Athens is considered unusual. So males can only marry one Athenian woman, which has potentially positive effects among the competition among Athenian men, but they can have as many slave concubines as they want. So when Christianity spreads, it ends the whole slave concubine thing, which was also common in Rome, and that’s another effect on this whole thing.

Dwarkesh - 00:50:48
Yeah. But I guess the extent to which these practices were already formed hundreds or thousands of years before the church’s intervention, maybe this suggests that Europe was already on this trajectory and the church didn’t necessarily move the needle that much. Or what do you make of that thesis?

Joseph Henrich - 00:50:48
Well we know that the… in the book, I talk about data where we can look at contemporary kinship practices, and we can look at the number of centuries that that part of Europe was under the church… so we have a database of the diffusion of bishoprics, and there’s a clear connection between the intensive kinship practices and the number of years under the church.

Dwarkesh - 00:51:25
Hmm. And then if you think about what the church basically did, I think your work is often used to justify this idea of Chesterton’s Fence, where if you don’t necessarily understand a cultural practice, you should keep it around because… the stuff we were talking about, The Secret, where you don’t understand why the 10 steps lead to the bean being edible, but you should trust the sort of wisdom of the ages.

If you think about what the church did here, right? Like, isn’t this sort of like a… not necessarily a justification, but at least an example of just breaking down Chesterton’s Fence? The church doesn’t really understand why these kin-based networks that have existed for thousands of years might have utility. It still just gets rid of all of them. And that results in this lottery ticket that leads to The Great Divergence. So is there evidence for anti-Chesterton’s fence from the Weirdest People?

Joseph Henrich - 00:52:15
Sure. Sure. So what you’re talking about is just the idea that culture has imbued institutions and various practices with a logic that we might not understand. So it’s not calling for never changing the institutions. It’s saying make sure you figure out the logic and the cost and benefits.

So I was an expert witness in Canada for the attorney general of British Columbia, asked the Supreme Court of British Columbia, whether the statutes against polygyny were legal. And so my job was just to inform the court that polygyny has this unfortunate effect of the elite and high-status males tend to get a disproportionate share of the wives, and that creates this pool of low-status unmarried men.

Now you can decide what you want to do with that. But you need to understand the kind of underlying social dynamics that you’re gonna unleash if you legalize or decriminalize polygyny.

Dwarkesh - 00:53:08
Right. But again, going back to the Church, the Church didn’t understand the positive or negative effects they’re gonna have, right? They just like, “Let’s rip the Band-Aid, let’s see what happens.”

Joseph Henrich - 00:53:17
That’s right. And it actually benefited the Church because it released people’s responsibilities to their families and allowed them to migrate in and join the Church, and more heavily invest in the Church. And these later prohibitions against priests marrying and stuff was all an effort to get greater investment in the Church, because you weren’t torn between helping your son and then investing in the Church.

Dwarkesh - 00:53:37
Hmm. And then that end result which benefited the Church, was that their conscious intention in breaking apart these kin-based ties? Or was that just the accidental byproduct?

Joseph Henrich - 00:53:50
Yeah, I mean, in the record, I’ve never been able to find much evidence that they were thinking about the destruction of kinship ties. There is a great quote from Saint Augustine where he talks about the benefits of forcing people to marry more distantly. But, you know, this is something that was debated repeatedly in church councils all across Europe. And it doesn’t seem to pop up very often.

Dwarkesh - 00:54:12
And the Church doing this, for obviously selfish reasons, was that in any way related to why the Church spread through Europe in the first place? Or did it end up spreading for separate reasons and it won out over the other potential religious competitors for separate reasons? And basically, was this part of the selective mechanism which allowed the Catholic Church to become dominant in Europe in the first place?

Joseph Henrich - 00:54:43
Yeah, I think so, although it’s hard to know. There was a lot of randomness going on in the diffusion of the Church. But also as the Church arrived, these places, over a period of time, became more successful. So you can see rising urbanization in the wake of the Church’s arrival. That would have meant more trade, their citizens’ rights began developing because they were trying to attract citizens to the towns. That would have given the Church more cachet, ability to move. There’s also the appearance of these voluntary associations which are new kinds of monasteries. And so the monasteries begin spreading throughout Europe. But they’re all linked in a network because, like the Cistercians, they’re all connected and they have big meetings. So they’re actually spreading a lot of knowledge around Europe as well. So that’s part of this kind of collective brain story.

Dwarkesh - 00:55:30
And if you compare what’s happening in Europe at this time to the rest of the world, so starting with China: in 1500, the population of England, where a couple centuries later the Industrial Revolution starts, is three million, and then in Ming dynasty China, it’s somewhere between 100 million to 160 million. And if we take your previous perspective of the size of the collective brain really matters a lot, the size of the collective brain in China just seems so much bigger. So what is the best way to understand what went wrong here? Why weren’t they able to use their…

Joseph Henrich - 00:56:02
Right. So when you’re thinking about China, the first thing to remember is that for a lot of the history, we can actually see that size difference mattering a lot.

So a lot of European invention, or a lot of stuff used in Europe, is flowing in; gunpowder, paper, printing press, stuff, is flowing into Europe. And so okay, so then what happens after about 1000 CE? Well, the argument is that the destruction of the kinship group opens the floodgates to people moving around. So a recent analysis that we’ve done is, after a bishop arrives at a 1.5 by 1.5 grid cell in Europe, people begin flowing in and out of that grid cell more. And what we do to calculate that is we have a big database of a few million famous people. And we have birth and death locations. So we find out that the Church arrives and suddenly people are free to move around.

So you have a flow of individuals around, and you have rising urbanization. So Europe passes China around 1200 in the percentage of the population that lives in cities. And cities are where a lot of the action is, cities and towns. And you have the diversification of occupations. So normally clans would specialize in different crafts, and you’d learn from your clan brothers and clan fathers how to do the occupation. In Europe you get guilds, and you get masters and apprenticeships developing where strangers will become an apprentice to a master, learn from him, move somewhere else as a journeyman, learn from that master, and then eventually set up a separate shop. Lots of opportunities for a flow of ideas.

So it has to do with how the kinship system transformed the movement of people, the rising of urbanization, the nature of guilds, and then eventually you get universities and things like that. So it greatly intensifies the interconnectedness of the collective brain, and the amount of cognitive diversity.

Dwarkesh - 00:57:44
What do we know about India during this period? Because from what I’ve read, from a perspective of how much written history we have, it’s kind of a black box. But we know that there was trade between India and other parts of the world, and we know it had a big population. Do we know why India before the Mughal period or before the British period wasn’t… yeah. What was the effect of not having maybe Abrahamic gods or.. with the kind of other cultural practices India had?

Joseph Henrich - 00:58:14
Yeah, definitely not an expert on this one, but some of David Reich’s evidence suggests that the caste system is quite old. Because you can actually see it in the genetic system. And the caste system is not good for innovation, because if you happen to be good at another skill but your caste doesn’t do that skill, you can’t switch over. So it’s gonna prevent the sort of available genetic diversity. Complex families, intensive kinship, there’s reason to think that those things were all important. Yeah, so, patchwork of polities.

But there was lots of interesting ideas, actually, that are developed in India, and they move into Central Asia, eventually end up in the Islamic world, and then get into Europe. So for example, Arabic numbers are actually Indian numbers. Zero was probably developed… I mean, Indians were huge with numbers. It’s kind of interesting. Perhaps related to the religion.

Dwarkesh - 00:59:04
Not that this would be a justification for the caste system, but is there any way in which the specialization it engendered would be good from the perspective of this collective brain perspective?

Joseph Henrich - 00:59:16
So specialization is good. And so the interesting thing about most human societies is specialization automatically- or not automatically- it tends to evolve along some kind of kinship line. So in Oceania, there were different clans. In each clan, there’d be like a canoe-building clan and a warrior clan. So that allows specialization, but it doesn’t allow you to harness the genetic diversity, because it’s not like the canoe-building clan had special canoe-building genes. It’s just that you would pass down this cultural knowledge.

Now, a better system, but it seems hard to evolve, is one in which we all opt, select, into occupations in which we think we’re good at. But for that world to exist, you need a world of voluntary associations with emphasis on the individual rather than on the group. So this is the world that evolves in Europe, once you demolish the intensive kinship units, because otherwise you’re gonna get castes and clans and all these things we see elsewhere, where we see a division of labor, but it’s transmitted through a kin network, the knowledge.

Dwarkesh - 01:00:14
And why wasn’t there another sort of stultifying… because we began the conversation by talking about, there does seem to be this cycle where over time, because of cultural differences, because of kin, whatever else, over time groups will tend to diverge. Languages, culture, other things you mentioned, tend to diverge over time. Why didn’t the same thing happen in Europe where, let’s say, in the third century or fifth century, the Church starts doing this stuff, and then by a thousand years later, if you’re part of this guild or whatever, now that’s become like the new kin, and now there’s actually less mobility again?

Joseph Henrich - 01:00:51
Well, so there was a lingua franca, Latin, and so intellectuals would all write and communicate in Latin for a long time. So even though they were speaking different dialects of French and German and whatnot, they were able to communicate in Latin. And Christendom basically formed an overarching network that helped. And one of the key things that this world religion does, and other world religions do it too, is you have to marry other Christians, but it dissolves the tribal line.

So Europe has tons of tribes in the pre-Christian world, but because of how you have to marry other Christians, you had, you know, Celtics marrying Franks. And in fact, the early arrival of Christianity into England is when a Frankish princess is marrying an Angle in Kent. And so they’re marrying back and forth. And this is gonna dissolve the tribes because intermarriage, “what’s the kid?”, you know? So that’s how you get rid of the tribes, but you need a world religion to do that.

Dwarkesh - 01:01:45
Right. And so it seems like there were two very important features that were relevant here. One, it seems like obviously the competition between different European groups which gave incentive for monasteries or universities or cities to attract the best people and make all these advancements. But secondly, the fact that they were descended from a common empire, from the Roman Empire. Basically, in a world where the Roman Empire never existed, do you still have this collective brain in Europe emerge?

Joseph Henrich - 01:02:22
So I think the Roman Empire plays a big role, because, for example, there were Roman roads in parts of Europe which allow people to flow. And you have communities which were part of Rome and relatively sophisticated. Now, they go into a bit of decline, but there’s still the memory of the empire. So, you know, Charlemagne wants to be crowned emperor of the Holy Roman Empire in 800 even though there’s not very much left of it.

But it was still an ideal, and that Carolingian Empire actually had real effects because they worked with the pope in order to enforce a lot of the marriage and family programs. And the Carolingians had their own agenda because they were trying to use these marriage and family practices to weaken some of the other aristocratic families. But they had the religious tools that they could put to work, and if you didn’t have those religious beliefs, then you couldn’t put them to work.

Dwarkesh - 01:03:10
Sometimes people defend ancient conquerors based on a similar idea of they spread knowledge around the world, and they set up these lines of communication. People say that about Alexander and then causing Hellenistic expansion and trade and so forth. They say that about Genghis Khan and the Silk Road. They say… I mean, obviously this led to the Black Plague, so… so it’s tough where that nets out.

They say that about Napoleon and the Napoleonic Codes and so forth. Where do you basically come out on that story of: have the great conquerors been good for the collective brain or bad for the collective brain?

Joseph Henrich - 01:03:55
Well, I’ve never actually focused on that. I mean, I guess my immediate reaction was there must be a better way to transmit the knowledge than the whole conquest.

Dwarkesh - 01:04:05
And then I guess there’s also the open question of how the disruptions that are caused by them… How much did they set back the-

Joseph Henrich - 01:04:11
And population decline, spreading disease and stuff, so certainly it could have been done better.

Dwarkesh - 01:04:15
Right. And going back to David Reich’s work, If your theory implies that there should have been a decrease in genetic similarity in Europe as of, after the fifth century or something, as a result of the culture, the Church’s practices, do we see that in the genetic record?

Joseph Henrich - 01:04:35
Yeah. Yeah, so Europeans are quite well mixed compared to other populations. And so we were talking about India. You know, if you do a principal components analysis of this two-dimensional plot of the genetic similarity you can actually see the class, and people are pretty spread out. If you put the Europeans on the same plot, they’re well mixed.

Dwarkesh - 01:04:52
Right. But do we see the period of greatest mixing starting in the fifth century?

Joseph Henrich - 01:04:59
Yeah, good question. I don’t know the answer to that question. So that would be… That’s a nice piece of evidence, so… You know, there’s a couple of groups working on medieval ancient DNA, so hopefully we’ll have more answers on that question. I mean, there is enough now of pre-Roman burials, so Bronze Age type stuff, showing that early Europeans definitely had complex, intensive kin groups, patrilineal, patrilocal resonance kinds of stuff, polygyny.

Dwarkesh - 01:05:30
And then stepping back, I’ve read there’s so many different theories of the Industrial Revolution and of modernity. And there’s Robert Allen, the economic historian who thinks it’s a result of higher wages because of the Black Death. There’s Gregory Clark, who thinks it’s because of this positive eugenics in England where the upper classes were having more babies. I mean, there’s like 20, 30, probably hundreds of different theories of why it happened, where it happened, and when it happened. And so each one of these, when I read them, sounds plausible. I don’t have a knockdown argument against any of them. But I’m not sure how to think about which one is correct, and yet, if we’re stacking all of these up against each other… why you think this story is any more accurate than any other story that people tell? Is it overdetermined?

Joseph Henrich - 01:06:24
Well, I mean, the first thing to think about is, those other authors- possibly Greg Clark is an exception, but they don’t think about the psychological variation that exists. So there’s good reason to think there’s all these psychological differences, and you have to believe they don’t cash out. And if you think they exist, then you have to explain them, and so you need a theory that explains them. So Bob Allen, he’s a blank on the psychological variation. So let me give you some evidence that I think is pretty good evidence. So we have a database of the diffusion of Roman bishoprics through Europe. And then what we do is we look at what happens when a bishopric arrives. And I mentioned before, you get a flow of people.

But another fact is, if you look at the production of famous people, these places where the bishopric has arrived start producing more creative individuals relative to non-creatives. So there’s an uptick, and it keeps going up for centuries, so the relative increase, and so that suggests you’re producing more authors and inventors and writers and whatnot. And then if you take each of those, and you correlate it with modern patents, places that spent more time under the church produced more patents between 1980 and 2014 based on European patent data. And this holds if you just compare regions within the same country. So you can see long-term effects of this, and you can see immediate effects in the historical record on the production of creatives.

Dwarkesh - 01:07:45
So what would you make of a theory that says, “Look, all of these theories are basically correct. All of these stories are basically correct.” And it’s them stacking up that leads to the Industrial Revolution rather than any one of them being the most important proximal cause. And maybe there’s reasons, like, let’s say there’s 10 of these stories and you had to stack them all up and China had five of them, but they also had maybe independently, like three other ones, but they also had two stories which go against them or something. And it’s like, no one of them is that proximately important, it’s their combination, right? So, I guess your theory seems compatible with Robert Allen’s that coal being cheap and wages being high was important.

Joseph Henrich - 01:08:30
Well, I guess one of the things that I think is a mistake is to focus too narrowly on England, because England was benefiting from ideas that were flowing in from France. There was a lot of great science being done in France. And, I don’t think Gregory Clark’s right. I can explain almost everything he explains with cultural evolution, and he doesn’t even really take that seriously.

Dwarkesh - 01:08:51
Oh, interesting.

Joseph Henrich - 01:08:52
Yeah. So patience, one of the things he argues is patience. In WEIRD, I use the exact same data- I actually get the data from him- to show that you have this increase in patience. But we know that people can culturally learn patience, and this can all be culturally transmitted, and in this world, it leads to more success. So if you’re the kind of person who doesn’t waste a lot of his money and we begin to value thrift and stuff, which Protestantism does, then we should expect there to be an increase in fewer murders and lower interest rates.

Dwarkesh - 01:09:20
What’s the reason to think that… I mean, his main explanation, I think is genetic, right? Is there some reason to think that that is less likely to be as important a factor as culture?

Joseph Henrich - 01:09:31
Well, we know that populations- migrants- into the US and Europe shift their psychology over a few generations of being there, and we know there’s been natural experiments done by economists like Chris Blattman where you actively try to teach people, train them essentially, to discount the future less. And that seems to be culturally transmittable. So this suggests that- I can’t rule out that there’s been any genetic evolution, but I can show that culture can operate on this quickly and powerfully.

Dwarkesh - 01:10:00
Right. I think one of his key pieces of evidence, I don’t remember the exact numbers on this or even the particular detail, but it was something like if you look at the inheritance of wealth or titles or- I forget what the exact thing is- the correlation across generations points to the kind of pattern you would see with genetic inheritance rather than cultural spreading.

Joseph Henrich - 01:10:28
Yeah. He doesn’t actually fit any cultural evolutionary models, and if you include that people are learning from parents and learning from their social milieu, your parents determine where you are in the social milieu, so you’re surrounded by people who are being more patient and behaving in certain economic ways, then that’s gonna have a huge effect. I just told you about how if you grow up in Silicon Valley, you’re gonna patent in computers. It’s not because you have genes for computers.

Dwarkesh - 01:10:52
Yeah. Given the persistence of these cultural effects, how should that change how we think about whether it makes sense to have more immigration from societies that are already WEIRD or whether that matters or not?

Joseph Henrich - 01:11:07
Right. So we have a paper under review right now in which we show that from the 1850 roughly to 1940, a big driver of US innovation both the quality and quantity of patents, is the cultural diversity of counties. And we actually use immigration as a kind of way of showing the causal effect of this. So what you want is a lot of cultural variation. Now, if you get people from societies that are more distant from whatever the current US culture is at the time, it’s gonna take more time until they’re able to fully integrate. So if they’re coming from [a] very distrusting society, they’re not gonna be able to latch into the collective brain immediately.

So you see this in around 1900. There’s data from people, mostly coming from Southern Italy and coming from Germany and Britain, and the Southern Italians are coming from a society that has a high distrust of strangers. So their immediate effect on the patenting system is low. But the strength of these cultural traits seems to go down by an order of magnitude each generation. So the second generation, you can still see where they came from, but they’re a lot more like the majority culture than their parents. And then it goes down another order of magnitude. So yeah, so you just have to do more work essentially to assimilate people, enculturate them. But the potential value of having diverse ways of thinking, different ideas, can be really valuable, I think.

Dwarkesh - 01:12:35
What do you make of, so Garett Jones is an economist who has this, he calls this spaghetti theory of assimilation. I don’t know if you’ve heard of this. The idea is basically, look, immigrants do assimilate, but they also assimilate us, and so whatever cultural practices they have will, in proportion to their size of the population, have an effect on wherever they migrate to.

And his example is when Italians came to America, spaghetti, which is traditionally an Italian dish, also became an American dish, right? So they assimilated American culture as well. And sub in traits like trust or whatever there. Basically, this sort of reciprocal assimilation has implications of whether we want immigration from non-WEIRD societies. What do you make of that idea?

Joseph Henrich - 01:13:26
I think that the spaghetti example or the pizza- pizza or spaghetti, both good examples- is a great thing, right? Because American food is a fusion of cuisine from lots of societies, and that leads to quite good food. Same thing you can see [in] music. Things like jazz come, have African rhythms in them. And so rock and roll is along that lineage of-

Dwarkesh - 01:13:54
But we don’t want, like, do we want the diversity in low trust and high trust?

Joseph Henrich - 01:13:58
Well, the question is, do social interactional traits operate the same way as these things? And if you have ethnic enclaves, then you’re gonna get low trust in those ethnic enclaves. You get things like the mafia, right?

So you don’t want that. But if you had an immigration policy that distributed people, and in a high-trust society, you benefit by being higher trust because you can do things like collaborate and start companies and stuff, which you have to be high-trust to do that. Otherwise, you don’t do it in a very effective way. So there is a pressure for low-trust people to become more high-trust. But if you’re in mostly high-trust people, there’s not an incentive for you to move down. And social interactional traits are different than things like food types.

Dwarkesh - 01:14:45
In psychology, there’s been this problem of the replication crisis where a lot of the main results are hard to replicate and it’s not clear how much of this is real science. Have you looked into how many of the WEIRD results are based on, of the differences in psychology between different populations and so forth? How much of that actually replicates?

Joseph Henrich - 01:15:05
Yeah, my sense is it replicates quite well. So we published our paper in 2010, and in, oh, late 20 odds, Armin Falk, who’s an economist, and a bunch of others, Ahnke Becker, Ben Enke, published a paper where they measured economic preferences in 80,000 people around the world. And that just showed big variation in things like patience, various kinds of reciprocity, altruism, so the kinds of things we would expect. And then other large-scale projects have similarly shown lots of variation.

Dwarkesh - 01:15:40
The groups we have today, which are not WEIRD, when you study them, do you think that the fact that they have resisted modernity for so long suggests that they actually are weird in their own way? Which is to say that this is not representative of the way that Europeans might’ve been thinking in the second century; rather it’s like the Hadza’s particular weirdness leads them to be averse to modernity in a way that is unique?

Joseph Henrich - 01:16:10
Well, that’s always a concern, and it’s especially a concern when you’re studying hunter-gatherers because a lot of the hunter-gatherers that are left in the world today are [in] places where agriculturalists couldn’t easily move to.

So that’s something we’re thinking about all the time. The ways around that are to go to places where agriculturalists didn’t go or couldn’t go. So we have a lot of good ethnographic history on people in Australia, and then we have the Arctic populations, which, you know, big sections of Paleolithic Europe were probably kinda like Northern Canada is today. So at least environmentally it’s not crazy, because, you know, Ice Age Europe. And places like the Aleutian Islands, the West Coast of California, we have lots of ethnographic evidence, and that there was no agriculture there.

So just putting together all these different lines of evidence help us develop a picture. I wouldn’t wanna bank everything on the Hadza. Of course, we can’t do experiments with some of those groups. But we can see whether the Hadza look like these other hunter-gatherer groups.

Dwarkesh - 01:17:10
Oh, speaking of the Ice Age: do you have a take on why before the Ice Age agriculture didn’t develop even though genetically there probably weren’t that big a difference between, like, 20,000 years ago versus 10,000 years ago?

Joseph Henrich - 01:17:20
Yeah, I don’t have a clear picture of that. It is the case that the Holocene was a particularly long stretch, a particularly long interglacial. There were some long interglacials, but the soonest one before that was 120,000 years ago.

Other than that, they were getting broken up. I sometimes wonder if we may someday figure out that there was actually a little bit of agriculture. And it would’ve all been destroyed by the Ice Age, right? So it’s not clear that there would be any heritage of it left. So we would have to find some remnants of some domesticated crops.

Dwarkesh - 01:17:52
Although I was asking David Reich about this, and I forget the reason he said, but he did say that if there were societies before the Ice Age, we would have evidence of them in the archeological record or their genetic record or something like that.

Joseph Henrich - 01:18:05
Yeah, if they had gotten to any scale. But we know from modern societies, there are groups, for example, in the Amazon that have root-based agriculture that wouldn’t leave any archeological record.

Dwarkesh - 01:18:17
This is a question I also asked to David Reich. There’s, you know, with modern LMs and just generally with newer techniques for processing information and understanding evidence, there’s a potential for answering questions that maybe we couldn’t have done before, especially from maybe a cultural perspective where you can actually consider lots of different cultural artifacts at once in the context of you can run cultural simulations or something. Maybe those are more fanciful ideas, maybe there’s more practical ideas. But if there’s one question you have which it’s sort of bottlenecked by having the right data or being able to process the data the right way and future LLMs or AIs could help us there, does something immediately come to mind?

Joseph Henrich - 01:19:05
Well, this gives me a chance to tell you something we’ve been working on. So we were talking about this kinship hypothesis, the idea that kinship intensity affects psychology. And when I presented the ideas that are in The Weirdest People in the World, I get hard times sometimes from historians who will have some kind of very European-specific story about why this happened that has to do with European royalty or coal or something like that.

And so my move is to not try to get into the weeds and start reading Latin texts, but to go test it somewhere else. So if it’s true that kinship intensity should work like this, then we should be able to test it somewhere else. So we have a large corpus, two different corpus from China. And so we have late-imperial China, something called the Gazetteers, which were produced across the prefectures, and they had a stylized content. So this is the same genre, thousands of books, and then we have about 7,000 books going back 2,000 years in Chinese history. And the techniques that the AI is allowing us to do is to get measurements of psychology from the texts. So we take a questionnaire. This is a technique developed by my post-doc, Mohamed Attari. We take a psychological questionnaire that’s been validated, it’s in English, and we wanna get an equivalent in ancient Chinese.

So we run it through a semantic similarity comparison, and we look for quotes from the ancient Chinese corpora that match each sentence in the English corpora. And then we rebuild the questionnaire in ancient Chinese. So that’s our psychological measure for something like individualism or collectivism or moral universalism. And then we take each book or each paragraph in each book, and we do a comparison, cosine similarity comparison, between the two sets of text there, and that allows us to stamp each paragraph with a measure of individualism for example. And then we do that for the entire corpora. And this allows us to track psychological change across space and time in China. And then we can correlate that with kinship intensity, and we get the same correlations that we do in Europe.

Dwarkesh - 01:21:09
Interesting. How worried are you about the fact that, if you think about modernity, was a result of finding this one cultural variant- at least according to your story- which then helps us develop the better technologies, all these new institutions and so forth. How worried are you about the fact that the modern world because of the spread of WEIRD, has just much less cultural variants and because of this monoculture, a potential variant which might be useful in the next transition just wouldn’t emerge?

Joseph Henrich - 01:21:43
Yeah, I am worried about that, because just the destruction of languages. So we’re losing languages left and right. English has a particular form that you don’t find in lots of languages, so that’s just an example of the kind of cultural diversity we’re losing. So that’s definitely a worry. I do think that we’re getting new variants. Japan adopts a lot of WEIRD institutions. But it’s really creating a new third unique thing. It’s got a bunch of Weird elements, but Japanese law, despite being the same as the US, operates very differently. So unlike Americans, Japanese don’t tend to sue each other and they tend to use mediation and things like that, even though the superficial law is pretty similar. So I do think there is creation of novelty which could be useful. But yeah, in general, I worry about the loss of novelty.

I also should say that I think that the sort of impersonal world of impersonal institutions in WEIRD psychology is quite fragile. For example, in the US in rural areas, you see moral universality declining since 2008. This is based on John Haidt’s YourMorals.org data. And so there’s an increasing difference in the morality of urban areas versus rural areas in the US.

Dwarkesh - 01:22:58
Hmm. I’m sure you get asked about this a lot, but fertility seems to be declining around the world, and there doesn’t seem to be any existing cultural variance, other than maybe the Amish, who can resist it. Do you have takes from a cultural anthropologist perspective of what’s happening here? And a follow-on question here is, should we really encourage these cultural enclaves like the Amish to the extent that this anti-fertility meme is so viral that they can’t be part of the common culture and still preserve fertility, you really need this closed-off societies in order to preserve fertility. How seriously do you take that idea?

Joseph Henrich - 01:23:38
Well, I definitely take declining fertility pretty seriously. And especially if you’re a collective brain guy like me, just having fewer people is gonna be a big problem. And I think that there will be spread of ideologies or religions or something which are pronatalist, and those groups will have a big advantage in cultural evolution because a community that is pronatalist tends to produce more pronatalist babies.
Christianity spread using that. Mormonism spread in the 19th century using that trick. So I just feel like cultural evolution is gonna find the combination and there’s gonna be some pronatalist groups spreading.

Dwarkesh - 01:24:19
There are a lot of different societies in the world today. Is there some explanation of why none of them have the existing variants necessary to keep fertility high?

Joseph Henrich - 01:24:29
Well, from a cultural evolutionary perspective, this is relatively new, right? So the demographic transition is only really late 19th century. Lots of the world is just getting hit after 1970. And we also have things like rising female labor force participation which is gonna stifle it, rising female education. And so once that stuff maxes out, then you’ll see variation among different groups in terms of the number of babies they produce, right?
And this thing unfolds in demographic time scales, so we’re not gonna see it for a few generations, but there’ll be some group somewhere that will be producing more babies than everybody else. And the reason why I think religion is the likely one is because people do things because they think God wants to do it. And if people come to think that God wants them to have more babies because it’s a way of worshiping him or getting to heaven or whatever their religious configuration is, then that’ll be a group that produces more babies. I mean, Catholics were defying the demographic transition for a while. They just seem to have stopped.

Dwarkesh - 01:25:33
Right. How worried are you about the fact that… I mean, you discuss in WEIRD that throughout this period in European history, European states are at war most of the time. How worried are you about the fact that, because of a decline in war between great powers, that this build-up in cultural mutational load that you were talking about or maybe where things just get less efficient over time and there’s no selective pressure to get rid of that inefficiency. How worried are you about that just making governments or nations less efficient over time because there’s no outer loop loss function that says if you mess up enough your country might not exist, or your group might not exist? Which we were talking about the waves of Yamnaya or the Anatolian hunter-gatherers just conquering everything. If that doesn’t happen, will we just see culture and nations and governments degrade over time?

Joseph Henrich - 01:26:31
I think the answer to that is yes. The one way that countries or whatever the political institution is could address that would be doing this thing that I’ve curiously called “domesticating the competition”. So sports teams constantly renew themselves by competing. Firms live and die and renew themselves over time. Like you said, companies will start off being super great and then they get too big and then they get kind of inefficient and then eventually they disappear. None of them last forever. And that seems to be true of political units. But in principle, you could have at least some renewal processes. So democracy potentially provides a renewal process, although there are things like the institutions that the government builds that are hard to renew. So one simple political idea is I think when you create a new department, it should have the same thing that cells have where at a certain time, they time out and you gotta make a new one.
And that’s because just the way human bureaucracies, institutions work is they kind of corrode from the interior, from the inside, just the way cancer spreads in a cell. So you just gotta kill it and make a new one. So we could institute that. But I guess the idea hasn’t quite caught on yet. I mean, there are bits and pieces of it around, so it’s not like it’s unused.

Dwarkesh - 01:27:49
Yeah, I guess maybe the thing that’s less understood or is not appreciated is that the reason that over time in history institutions have improved is because of the selective process that at least at certain levels of selection no longer exists.

Joseph Henrich - 01:28:04
Yeah. I mean, the Roman Empire didn’t say, “You know, we gotta redo everything.” “Why not adopt democracy and we could have a republic.” Nobody does that. Things fall apart.

Dwarkesh - 01:28:15
The researcher Stuart Armstrong has this idea called Chesterton’s metafence. And here’s what it states: In our current system, democratic market economies with large governments, the common practice of taking down Chesterton’s fences is a process which seems well established and has a decent track record and should not be unduly interfered with.

Joseph Henrich - 01:28:38
Can you say any more about what the author has in mind? I would like an example or something.

Dwarkesh - 01:28:43
Yes. So I think this is a rebuttal against this common idea that we shouldn’t be arbitrarily changing- like suppose a new technology comes about and we’re worried about the risk it might pose to society. Suppose the young kids are doing something different culturally and we’re worried about like what effect that might have on the culture and so forth.

He’s basically saying, “Look, this has been happening for five centuries and this process by which new cultural variants enter the mix and so forth has worked really well for us, even if it’s happening at a rapid clip and we shouldn’t interfere with it.”

Joseph Henrich - 01:29:22
Well I don’t think I have strong feelings on this. I’m always focused on trying to understand the process of cultural evolution, and it is true that people often resist cultural change that in retrospect we think is good. But of course when you begin to make cities, you lead to all kinds of epidemic diseases. So if you’d anticipated that, you might have worked on some of the public health procedures before you built the cities. So foresight can be good.

Dwarkesh - 01:29:48
Right. I guess the sort of bigger question here is, it makes sense in the kind of environment our ancestors faced why they’d have intuitions against progress, because if you already have technologies that are well fit to the environment, any change is likely to be deleterious. And whether we find ourselves in a world that’s different enough where we can just, very intentionally disregard our worries about changes in culture or technology, or whether it’s similar enough that we should actually… I mean, there’s two ways you can interpret The Secret, right? One is, “Oh, look, this is why we should care more about culture” and so forth. Another is just, “Oh, wow, they’re living in such a different world. I understand why my intuition against progress- where it comes from, and now I can totally disregard it.”

Joseph Henrich - 01:30:34
Right. So I would say that the lesson from Secret is not to disregard it, but just to be cautious because we could be dismantling things that are really important for the structure of society. And you shouldn’t just dismiss valuable cultural practices as the relics of a medieval age, or the relics of a pre-enlightened age or something like that.

So ritual is a good example because rituals seem to have real psychological benefits in binding people in community and helping develop self-regulation. But it’s easy to be an atheist like me and say, “Ah, rituals are stupid. We gotta stop doing them.” Turns out they’re doing a bunch of stuff and if you don’t wanna lose that stuff, you gotta figure out another solution.

Dwarkesh - 01:31:20
A question that arises in my head is, look, we began this conversation talking about all these societies in the past, even when they figured out something successful, they did not themselves know why it was successful and so they just had to ascribe it to custom. They just say like, “Oh, this is the way we always made bows” or hunted or something. Why think that we have an answer for why the Industrial Revolution happened? Why think it’s any different than the Inuit trying to explain why they make their bows a certain way?

Joseph Henrich - 01:31:49
Oh, you mean, how or what we’re up to is different?

Dwarkesh - 01:31:51
Yeah.

Joseph Henrich - 01:31:51
Well, I mean, we’re trying to apply science, which has been very successful in all these different areas. So, you know, evidence, and we all put out our arguments and we go through the process and some evidence is better than others and that’s just our epistemology.

And I’m interested in the cultural evolution of epistemologies. So societies have varied over time in what counts as good evidence and what counts as a good argument. So one of the psychological changes that we see emerging in Europe compared to other places is how important is what the ancients say.

So in lots of societies, if someone says something and you can say Aristotle or Confucius didn’t believe that, then it’s like, “Oh gosh, I guess I’m sunk.” Whereas at some point, Europeans decided, “Actually who cares what Aristotle thought. We know a lot more than he did.” So this is a big epistemological change. And then the emphasis on empirical evidence in science is not something you find in earlier traditions within Europe, and it’s quite variable across different societies.

So you should think about the very standards by which we count a good argument and good evidence is itself culturally evolved.

Dwarkesh - 01:32:56
Yeah. And how much do you worry about the lack of cultural variation in science in particular, where we have like one big institution, the NIH, which funds most science. And there’s similar processes of consensus and peer review and whatever, that educated whether you get a grant, whether work is accepted, whether it’s considered worthy scholarship and so forth. Should we think that maybe we should have more variation in those methods because maybe the scientific method isn’t complete, there’s epistemic tools could be improved or it could be different and so forth?

Joseph Henrich - 01:33:38
Yeah, absolutely. So polycentricity. There should be lots of difference, competition amongst these groups, no single funding sources… lots of different priorities. One thing I point out in this book I’m working on is that scientific papers tend to be more impactful when the authors are from more diverse societies.

But interestingly, people are biased to work with people from their own society. So what we actually do is different than what would maximize innovation.

Dwarkesh - 01:34:05
I’m sure you’re gonna address this in the book, but I’m already curious now. It seems like the internet hasn’t been as big a deal as somebody who’s thinking from this zoomed out collective brain picture might have anticipated in the ’90s. Maybe it’s boosted the rate of innovation somewhat, although it’s not clear just looking at growth rates or productivity growth rates or something that it has. Why didn’t the internet cause the equivalent of another round of urbanization in our ability to do research or economic productivity or so forth?

Joseph Henrich - 01:34:40
Yeah. And that’s something I’m really interested in figuring out. And one thing that I’m still- I don’t have a full answer to this question, but one thing that is clear is that there’s something special about face-to-face interactions.

So something like whether even in the 21st century, whether two cities have direct long haul flights between them or direct flights between them increases the flow of ideas between those places. But we know these places are connected by the internet, whereas that effect doesn’t seem as good. And part of this is that people have to build trust, probably, before we start sharing ideas. And the more different someone is from us, the more you need that trust. So some of the research suggests that this effect is even bigger when places are more culturally different. And this is interesting, because they’re probably more valuable to each other when they’re more culturally different. But that’s then the face time and being in the same place is even more valuable. So that’s the kind of direction I’m going. And also, I still think that there should be exactly what you said, which is that the internet should have caused more innovation. I’m just not satisfied that we fully determined the best way to figure that out.

Dwarkesh - 01:35:55
You had a very interesting chapter in Weird where you talk about… a lot of innovation through history is more serendipitous than we imagine, and it’s the result of the collective brain being big enough to discover these things. But if you look at the sort of track record in science and research, there do seem to be certain individuals who make independent discoveries across many different fields that are each quite important, your Newtons and Einsteins and John von Neumanns and so forth. And if it was just the result of the right person is in the right lab who sees the right observation at the right time, this repeated excellence by certain people seems less explainable. So I’m curious if the collective brains theory can explain what’s going on there.

Joseph Henrich - 01:36:44
Right. Sometimes I’m interpreted as saying that there are no genetic differences among individuals, [I] definitely think there are genetic differences among individuals that affect their likelihood. But I think when we take the individual, we often import into our thinking about them the person’s life history. So for example Einstein, when he was a patent clerk, so he wasn’t succeeding as an academic, he and his friends got together in something called the Olympia Academy, they called themselves, which was just a group of, like, five people, who would get together and they would read the interesting books at the time. And if you look at the books that they read, and historians have done a lot of work on this, all the major ideas that go into special relativity were read and processed by the Olympia Academy before they do it. So the idea that people think that time is relative; well, people were kinda talking about that, and they were talking about multiple dimensions. And like Henri Poincaré, at the same time, comes up with the same equation as Einstein, but he doesn’t give it as radical an interpretation as Einstein does. He was thinking of the equations as kind of fudge factors, trying to make the math work. So there was an almost simultaneous invention of special relativity, and the ideas were all circulating. And Einstein happened to be in a place which allowed him to put all those things together.

As a patent clerk, he was actually processing devices to synchronize trains in different parts of Europe. And so he had to think about the amount of time a signal coming from different parts of the world. And there were a bunch of these. And we know the patents that went through, and we know that he had to look at these, there were only two guys. So he happened to be in a particular place and time in terms of the ideas circulating and what he was doing on a day-to-day basis that really did give him an edge. Now, you could then say, “Well, what about general relativity?” Well, there were a couple other guys who were probably gonna get general relativity. And Einstein himself was worried that he was gonna get scooped, because once you get the one, it’s just a matter of- I mean, it’s not “just” a matter of figuring out, the math is really hard. But there were a number of other people who probably would’ve gotten the math.

Dwarkesh - 01:38:48
Yes. I guess then there’s also still the question of why, in 1905, Einstein himself discovers not only special relativity, but Brownian motion and the two other things, right? But I guess you would just say it’s the result of he had the right reading group?

Joseph Henrich - 01:39:05
I mean, obviously he’s a special individual. Although he did spend the whole second half of his career trying to show why quantum mechanics was wrong. So his Brownian motion paper is foundational in quantum mechanics. But then he decides God doesn’t play dice with the universe, and he doesn’t like the stochastic nature of quantum mechanics, which has more or less proved true, and it’s part of our modern technology and stuff like that. And he spends the whole second half of his career fighting it. So that’s a case where his intuitions didn’t serve him well. And he turned out to be wrong for decades.

Dwarkesh - 01:39:39
And, sorry, I think that’s also a case in which your point about having the right ideas available to you is important, because, obviously I don’t know much about this, but from what I understand, the people who believe in the many worlds interpretation believe that Einstein was on the right track, and he just didn’t have this idea of the ever-ending multiverse available to him, which would’ve explained his…

Joseph Henrich - 01:40:04
That there’s this branching thing, and just a matter of figuring out which branch you’re on.

Dwarkesh - 01:40:08
Right. And so there’s no probability.

Joseph Henrich - 01:40:09
Yeah, yeah.

Dwarkesh - 01:40:09
Yeah. Okay, so we were just discussing, look, Einstein was reading the right things and making sure he had enough context to come up with these big discoveries. You’re somebody who has connected ideas across many different fields. How would you describe the input function for the way in which you come up with new ideas, and how much has it been informed by… I mean, you were like, you started off as an aerospace engineer, and you’ve done anthropology across so many different societies, on fieldwork and so forth. How do you basically think about the Joseph Henrich production function?

Joseph Henrich - 01:40:45
Yeah, I mean I try to implement the ideas of the collective brain. So in my lab at Harvard, I have post-docs and graduate students with backgrounds in evolutionary biology, psychology, anthropology, economics. I mean, some of my favorite collaborators are economists. And we’re just bringing ideas from very different places, but across the social sciences and even the biological sciences, we’re often trying to explain the same kind of phenomena, economic decision-making, cooperation, things like ethnicity, why does it exist, stuff like that. So the way we silo science is a big problem, and especially in the social science, I think it’s even a bigger problem.

So I’ve been a professor of anthropology, psychology, economics, and evolutionary biology, and just even the standards of evidence and what constitutes good research and how you tackle a problem really varies. So the academic disciplines are like different cultural worlds, different tribes, and just pulling the best from them has kind of been my approach.

Dwarkesh - 01:41:45
And among the more polymathic scholars, do you guys have, I don’t know… do you and David and a couple of the other ones in this category, is there some shared group you guys have, or, is it…

Joseph Henrich - 01:42:02
No, it’s a series of different networks. So David Reich and I, for example, are always talking about how we wanna read culture from the genes. So if humans have had a long history of gene-culture co-evolution and we wanna understand what Paleolithic populations were doing, we might be able to see imprints of it in the genes.

So in, in my own work, for example, we’ve measured cousin marriage across different populations and shown it correlates with runs of homozygosity in the genes. Something like polygeny can be revealed by looking at the ratio, the variance in Y chromosomes to X chromosomes. Fire: we probably have some special genetic adaptations for dealing with the toxins in fire. So if you wanna know when humans got fire, and you see the gene that gives us immunity to toxins in fire, then we can infer that fire is older than that, right?

Dwarkesh - 01:42:53
Interesting. And, so obviously the input from different scientific fields in your work is obvious. But from a more philosophical perspective, people compare your work in the same tradition as Burke or Chesterton or something. Do you personally find that your philosophy has been informed by reading them, or is it just independently converging on some of the same themes?

Joseph Henrich - 01:43:18
Yeah, I mean, any convergence is completely independent, because I haven’t really spent much time at all. I mean, I’ve gone back. After people said, “Your work is kinda like Hayek,” I went back and read some Hayek. I did read The Fatal Conceit in graduate school, but aside from that, I didn’t read very much. And then same thing with Adam Smith. I mean, I’ve tackled The Wealth of Nations, and I eventually read The Theory of Moral Sentiments. But it was pretty far into the process. I had picked up on themes they developed before I knew they had developed them.

Dwarkesh - 01:43:49
I want to suggest that maybe you should add a couple of AI people in your rotation.

Joseph Henrich - 01:43:50
Ok.

Dwarkesh - 01:43:51
Just because I think this perspective is incredibly informative, what I think is maybe the most important question of our period, of what does this transition towards a society of AIs look like? And, sorry to bring it back to AI, but one other consideration here that your work has really informed me on, is: I was sort of maybe hyping up how big a deal I expect societies of AIs to be, but something your work informs is the idea of a single superintelligence is not the place in which to expect these big impacts of John von Neumann times 10.

But then again, there is a key advantage in the scope of social learning they can do as a society, but, like, the idea of a single superintelligence being super powerful is maybe less likely as a result of your work. And I, anyways, I don’t know what you think about that interpretation, but…

Joseph Henrich - 01:44:44
Yeah, I think that’s right because the assumption, the sort of model that people seem to carry around in their heads is that humans have done all these things because of our individual brain power when really, it’s been the power of cultural evolution and a network of minds gnawing away at problems over long periods of time and gradually accumulating not just the obvious tools, but also the cognitive tools for, for addressing these things.

Dwarkesh - 01:45:08
I buy the idea that individual IQ isn’t the most relevant factor to understanding why discoveries are made. It’s not about having the right genius or something. But what’s the reason for thinking that the average IQ of a society isn’t super important in how much progress a society’s gonna make, not just the population size?

Joseph Henrich - 01:45:28
Right. So the first thing that we need to do though is to zoom back out and think about what we mean by IQ. And so Michael Muthu Krishnan and I have made the case that IQ is just a set of cognitive abilities that leads to success in 20th century contemporary institutions that have come to dominate the world. So it’s a set, it’s a culturally evolved system, and we talked before about the Flynn effect which illustrates that, and the fact that IQ is associated with all these positive outcomes now, but certainly wouldn’t be in pastoralist societies.

So that would mean that if you raise the average IQ of your society, it might lead them to have more abstract thoughts and do better science. And so that’s, that certainly fits. But the interesting thing for me is that the world of the future is gonna be quite different than the one now. So the set of cognitive abilities which is gonna be favored in the new AI world is not gonna be the same set that was favored in 1900 when they designed the IQ test. So for example, in the IQ test they ask you to remember digits backwards. I’m not sure how useful remembering lists of digits backwards [is]. It was in a previous world, where we had to write everything down and remember a lot of stuff. But in some sense, we’re interfacing with our technology and we’ve got to figure out what are the set of cognitive abilities which is gonna make people best able to solve problems? And like we talked about, it’s even the case that the most creative people aren’t the highest IQ people.

I guess one of the things I’m trying to say is that the minds that might lead us into the new world might not be the ones that have the highest IQ because once you’re sort of augmented with AI and all these kind of technological aids we have, the specialized thing that leads someone to do something creative- probably not gonna be the same abilities that did it in 1910.

Dwarkesh - 01:47:21
Yeah. I guess my real opinion is that, if in the world where that’s true- and I think that’s probably gonna be true- then it really doesn’t matter at all because AIs will be doing everything. But in the world where, let’s say AI plateaus or something, then I would expect roughly the same kinds of skills that have mattered for the last couple of centuries to keep mattering for a modern, technological society, which is analytical thinking and the ability to understand science and technology and so forth.

Joseph Henrich - 01:47:54
But maybe the AI world you’re imagining is different than the one I’m imagining, but I still think that people are gonna figure out what problems we need to solve, unless we’re just gonna tell the AIs to figure out what the problems are and then solve them.

They might not be that good at that. I don’t know. We’ll see.

Dwarkesh - 01:48:10
You shared with me this draft of work about collective brains, where you show that ants have developed many of the technologies that humans have, like farming, and livestock, and division of labor and so forth. And so maybe there’s some amount of blind selection, and it doesn’t matter if it’s natural selection, if it’s cultural selection, it’s the size of the group which can go through this learning process that matters, and how many people are available to experience the learnings or get the selection process acted upon them.

But I guess the big difference in where maybe individual IQ comes back into the picture here is, I don’t expect there to be any amount of natural selection which will allow ants to land on the moon or make a computer chip or do, you know, make a nuclear fusion plant or something. And so is the kind of broad generalization we see with humans, and we might see to an even greater extent with AI, a product of you’re really bottlenecked by the IQ of the individual? There’s no amount of collective learning that can get you to the moon if you’re an ant?

Joseph Henrich - 01:49:33
Right. Well, one thing to keep in mind is that most human societies over most of human evolutionary history didn’t get to the moon. So this is one particular group of humans. And one of the things we talked about earlier in this conversation was the cultural evolution of epistemology.

So it was the improvement in our what constitutes evidence, what constitutes a good argument that allows us to get to science and accumulate this kind of knowledge to do these kinds of things. So I see that as part of a continuous trajectory. But it’s just that we have new cognitive tools.

Dwarkesh - 01:50:05
Right. I guess, but… so there’s a really interesting point that even the epistemic tools which let us get to the moon are themselves a product of cultural evolution. But again, that seems bottlenecked by the fact that if you’re just dealing with chimpanzees or something, there’s no amount of cultural evolution that will result in the scientific method, that-

Joseph Henrich - 01:50:30
Right. It needs some gene culture co-evolution.

Dwarkesh - 01:50:32
Right, right. So you mentioned in the Secret the tool use started something like two million years ago, and fire, we started domesticating fire closer to 800,000 to a million years ago. Intuitively, it doesn’t seem like tool use is that that much simpler than fire, but is that a misunderstanding? Why was fire so much [more] recent?

Joseph Henrich - 01:50:50
Well, so what we know from other species is that lots of animals use tools, and particularly chimpanzees use tools. So we can assume tools in the common ancestor. Now, what we see in the paleoanthropological record is the increasing use of stone tools.

And these are pretty simple stone tools. You can see a cutting edge there, but not very much fancier than that. And then fire is… a lot of animals are afraid of fire, and they have to run from wildfires and stuff like that. So whatever your story about humans is, you have to overcome the fear of the fire in order to tame it. Probably humans first found wild fire and somebody approached the fire instead of running away, which is the usual thing to do. And then got some of it and then put it to use, I guess. So I think it’s the innate fear that animals have of fire, which is we don’t hang around when things go off.

Dwarkesh - 01:51:41
Interesting. Okay, final question, what’s the next thing you’re working on?

Joseph Henrich - 01:51:45
Well, the big thing is this book on collective brains, so I’m gradually working through that. One of the ideas that I really am excited about in this book is that we evolved to think in collectives. There’s this assumption that psychologists have had to understand human decision-making and how our minds solve problems is that we should put people by themselves in experiments and see how they do. But real human societies, hunter-gatherers, they actually work in groups. And when we want to solve a problem, the first thing we want to do is check with our friends or ask other expert members of the community.

So it’s the idea that we think collectively and solve problems collectively in a kind of naturally distributed brain. And Hugo Mercier and Dan Sperber, for example, have argued and shown with various lines of evidence that many of the sort of irrational biases we have, mitigate or disappear entirely when we solve problems as groups. So it’s almost like we evolved to have that positive interaction and correct each other’s errors.

Dwarkesh - 01:52:43
Right. That’s quite interesting that you care more about the portfolio of intelligences than any one person being calibrated.

Joseph Henrich - 01:52:57
Right.

Dwarkesh - 01:52:58
Excellent. This is super fun. Thanks so much for coming on the podcast.

Joseph Henrich - 01:52:58
All right. Great to be with you.

I’m so excited with how this visualization of Notes on China turned out. Petr, thank you for such beautiful watercolor artwork. More to come!

Watch on YouTube.

(0:00:00) - Intro
(0:00:32) - Scale
(0:05:50) - Vibes
(0:11:14) - Youngsters
(0:14:27) - Tech & AI
(0:15:47) - Hearts & Minds
(0:17:07) - On Travel

Satya Nadella on:

• Why he doesn’t believe in AGI but does believe in 10% economic growth,

• Microsoft’s new topological qubit breakthrough and gaming world models,

• Whether Office commoditizes LLMs or the other way around,

Watch on Youtube; listen on Apple Podcasts or Spotify.

• Scale partners with major AI labs like Meta, Google Deepmind, and OpenAI. Through Scale’s Data Foundry, labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you’re an AI researcher or engineer, learn about how Scale’s Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkesh

• Linear's project management tools have become the default choice for product teams at companies like Ramp, CashApp, OpenAI, and Scale. These teams use Linear so they can stay close to their products and move fast. If you’re curious why so many companies are making the switch, visit linear.app/dwarkesh

To sponsor a future episode, visit dwarkeshpatel.com/p/advertise.

(0:00:00) - Intro

(0:05:04) - AI won't be winner-take-all

(0:15:18) - World economy growing by 10%

(0:21:39) - Decreasing price of intelligence

(0:30:19) - Quantum breakthrough

(0:42:51) - How Muse will change gaming

(0:49:51) - Legal barriers to AI

(0:55:46) - Getting AGI safety right

(1:04:59) - 34 years at Microsoft

(1:10:46) - Does Satya Nadella believe in AGI?

Dwarkesh Patel

Satya, thank you so much for coming on the podcast.

In a second, we're going to get to the two breakthroughs that Microsoft has just made, and congratulations, same day in Nature: the Majorana zero chip, which we have in front of us right here, and also the world human action models. But can we just continue the conversation we were having a second ago? You're describing the ways in which the things you were seeing in the 80s and 90s, you're seeing them happen again.

Satya Nadella

The thing that is exciting for me... Dwarkesh, first of all, it's fantastic to be on your podcast. I'm a big listener, and I love the way that you do these interviews and the broad topics that you explore.

The thing that is exciting for me… It reminds me a little bit of my, I'd say, first few years even in the tech industry, starting in the 90s, where there was real debate about whether it's going to be RISC or CISC, or, "Hey, are we really going to be able to build servers using x86?"

When I joined Microsoft, that was the beginning of what was Windows NT. So, everything from the core silicon platform to the operating system to the app tier- that full stack approach- the entire thing is being litigated.

You could say cloud did a bunch of that, and obviously distributed computing and cloud did change client-server. The web changed massively. But this does feel a little more like maybe more full-stack than even the past that at least I've been involved in.

Dwarkesh Patel

When you think about which decisions ended up being the long-term winners in the 80s and 90s, and which ones didn't, and especially when you think about- you were at Sun Microsystems, they had an interesting experience with the 90s dotcom bubble. People talk about this data center build-out as being a bubble, but at the same time, we have the Internet today as a result of what was built out then.

What are the lessons about what will stand the test of time? What is an inherent secular trend? What is just ephemeral?

Satya Nadella

If I go back, the four big transformations that I've been part of, the client and the client-server. So that's the birth of the graphical user interface and the x86 architecture, basically allowing us to build servers.

It was very clear to me. I remember going to what is PDC in '91, in fact I was at Sun at that time. In '91, I went to Moscone. That's when Microsoft first described the Win32 interface and it was pretty clear to me what was going to happen, where the server was also going to be an x86 thing. When you have the scale advantages accruing to something, that's the secular bet you have to place. What happened in the client was going to happen on the server side, and then you were able to actually build client-server applications. So, the app model became clear.

Then the web was the big thing for us, which we had to deal with in starting, in fact as soon as I joined Microsoft, the Netscape browser or the Mosaic browser came out what, I think, December or November of '93, right? I think is when Andreessen and crew had that.

So that was a big game-changer, in an interesting way, just as we were getting going on what was the client-server wave, and it was clear that we were going to win it as well. We had the browser moment, and so we had to adjust. And we did a pretty good job of adjusting to it because the browser was a new app model.

We were able to embrace it with everything we did, whether it was HTML in Word or building a new thing called the browser ourselves and competing for it, and then building a web server on our server stack and go after it. Except, of course, we missed what turned out to be the biggest business model on the web, because we all assumed the web is all about being distributed, who would have thought that search would be the biggest winner in organizing the web? And so that's where we obviously didn't see it, and Google saw it and executed super well.

So that's one lesson learned for me: you have to not only get the tech trend right, you also have to get where the value is going to be created with that trend. These business model shifts are probably tougher than even the tech trend changes.

Dwarkesh Patel

Where is the value going to be created in AI?

Satya Nadella

That's a great one. So I think there are two places where I can say with some confidence. One is the hyperscalers that do well, because the fundamental thing is if you sort of go back to even how Sam and others describe it, if intelligence is log of compute, whoever can do lots of compute is a big winner.

The other interesting thing is, if you look at underneath even any AI workload, like take ChatGPT, it's not like everybody's excited about what's happening on the GPU side, it's great. In fact, I think of my fleet even as a ratio of the AI accelerator to storage, to compute. And at scale, you've got to grow it.

Dwarkesh Patel

Yeah.

Satya Nadella

And so, that infrastructure need for the world is just going to be exponentially growing.

Dwarkesh Patel

Right.

Satya Nadella

So in fact it's manna from heaven to have these AI workloads because guess what? They're more hungry for more compute, not just for training, but we now know, for test time. When you think of an AI agent, it turns out the AI agent is going to exponentially increase compute usage because you're not even bound by just one human invoking a program. It's one human invoking programs that invoke lots more programs. That's going to create massive, massive demand and scale for compute infrastructure. So our hyperscale business, Azure business, and other hyperscalers, I think that’s a big thing.

Then after that, it becomes a little fuzzy. You could say, hey, there is a winner-take-all model- I just don't see it. This, by the way, is the other thing I’ve learned: being very good at understanding what are winner-take-all markets and what are not winner-take-all markets is, in some sense, everything. I remember even in the early days when I was getting into Azure, Amazon had a very significant lead and people would come to me, and investors would come to me, and say, "Oh, it's game over. You'll never make it. Amazon, it's winner-take-all."

Having competed against Oracle and IBM in client-server, I knew that the buyers will not tolerate winner-take-all. Structurally, hyperscale will never be a winner-take-all because buyers are smart.

Consumer markets sometimes can be winner-take-all, but anything where the buyer is a corporation, an enterprise, an IT department, they will want multiple suppliers. And so you got to be one of the multiple suppliers.

That, I think, is what will happen even on the model side. There will be open-source. There will be a governor. Just like on Windows, one of the big lessons learned for me was, if you have a closed-source operating system, there will be a complement to it, which will be open source.

And so to some degree that's a real check on what happens. I think in models there is one dimension of, maybe there will be a few closed source, but there will definitely be an open source alternative, and the open-source alternative will actually make sure that the closed-source, winner-take-all is mitigated.

That's my feeling on the model side. And by the way, let's not discount if this thing is really as powerful as people make it out to be, the state is not going to sit around and wait for private companies to go around and… all over the world. So, I don't see it as a winner-take-all.

Then above that, I think it's going to be the same old stuff, which is in consumer, in some categories, there may be some winner-take-all network effect. After all, ChatGPT is a great example.

It's an at-scale consumer property that has already got real escape velocity. I go to the App Store, and I see it's always there in the top five, and I say “wow, that's pretty unbelievable”.

So they were able to use that early advantage and parlay that into an app advantage. In consumer, that could happen. In the enterprise again, I think there will be, by category, different winners. That's sort of at least how I analyze it.

Dwarkesh Patel

I have so many follow-up questions. We have to get to quantum in just a second, but on the idea that maybe the models get commoditized: maybe somebody could have made a similar argument a couple of decades ago about the cloud – that fundamentally, it's just a chip and a box.

But in the end, of course, you and many others figured out how to get amazing profit margins in the cloud. You figured out ways to get economies of scale and add other value. Fundamentally, even forgetting the jargon, if you've got AGI and it's helping you make better AIs – right now, it's synthetic data and RL; maybe in the future, it's an automated AI researcher – that seems like a good way to entrench your advantage there. I'm curious what you make of that, just the idea that it really matters to be ahead there.

Satya Nadella

At scale, nothing is commodity. To your point about cloud, everybody would say, "Oh, cloud's a commodity." Except, when you scale... That's why the know-how of running a hyperscaler... You could say, "Oh, what the heck? I can just rack and stack servers."

Dwarkesh Patel

Right.

Satya Nadella

In fact, in the early days of hyperscale, most people thought “there are all these hosters, and those are not great businesses. Will there be anything? Is there a business even in hyperscale?” And it turns out there is a real business, just because of the know-how of running, in the case of Azure, the world's computing of 60-plus regions with all the compute. It's just a tough thing to duplicate.

So I was more making the point, is it one winner? Is it a winner-take-all or not? Because that you've got to get right. I like to enter categories which are big TAMs, where you don't have to have the risk of it all being winner-take-all. The best news to be in is a big market that can accommodate a couple of winners, and you're one of them.

That's what I meant by the hyperscale layer. In the model layer, one is models need ultimately to run on some hyperscale compute. So that nexus, I feel, is going to be there forever. It's not just the model; the model needs state, that means it needs storage, and it needs regular compute for running these agents and the agent environments.

And so that's how I think about why the limit of one person running away with one model and building it all may not happen.

Dwarkesh Patel

On the hyperscaler side, and by the way, it's also interesting the advantage you as a hyperscaler would have in the sense that, especially with inference time scaling and if that's involved in training future models, you can amortize your data centers and GPUs, not only for the training, but then use them again for inference.

I'm curious what kind of hyperscaler you consider Microsoft and Azure to be. Is it on the pre-training side? Is it on providing the O3-type inference? Or are you just, we’re going to host and deploy any single model that's out there in the market, and we are sort of agnostic about that?

Satya Nadella

It’s a good point. The way we want to build out the fleet is [to], in some sense ride Moore's law. I think this will be like what we've done with everything else in the past: every year keep refreshing the fleet, you depreciate it over whatever the lifetime value of these things are, and then get very very good at the placement of the fleet such that you can run different jobs at it with high utilization. Sometimes there are very big training jobs that need to have highly concentrated peak flops that are provisioned to it that also need to cohere. That's great. We should have enough data center footprint to be able to give that.

But at the end of the day, these are all becoming so big, even in terms of if you take pre-training scale, if it needs to keep going, even pre-training scale at some point has to cross data center boundaries. It's all more or less there.

So, great, once you start crossing pre-training data center boundaries, is it that different than anything else? The way I think about it is hey, distributed computing will remain distributed, so go build out your fleet such that it's ready for large training jobs, it's ready for test-time compute, it’s ready- in fact, if this RL thing that might happens, you build one large model, and then after that, there’s tons of RL going on. To me, it's kind of like more training flops, because you want to create these highly specialized, distilled models for different tasks.

So you want that fleet, and then the serving needs. At the end of the day, speed of light is speed of light, so you can't have one data center in Texas and say, "I'm going to serve the world from there."

You've got to serve the world based on having an inference fleet everywhere in the world. That's how I think of our build-out of a true hyperscale fleet.

Oh, and by the way, I want my storage and compute also close to all of these things, because it's not just AI accelerators that are stateless. My training data itself needs storage, and then I want to be able to multiplex multiple training jobs, I want to be able to then have memory, I want to be able to have these environments in which these agents can go execute programs. That's kind of how I think about it.

Dwarkesh Patel

You recently reported that your yearly revenue from AI is $13 billion. But if you look at your year-on-year growth on that, in like four years, it'll be 10x that. You'll have $130 billion in revenue from AI, if the trend continues. If it does, what do you anticipate doing with all that intelligence, this industrial scale use?

Is it going to be through Office? Is it going to be you deploying it for others to host? You've got to have the AGIs to have $130 billion in revenue? What does it look like?

Satya Nadella

The way I come at it, Dwarkesh, it's a great question because at some level, if you're going to have this explosion, abundance, whatever, commodity of intelligence available, the first thing we have to observe is GDP growth.

Before I get to what Microsoft's revenue will look like, there's only one governor in all of this. This is where we get a little bit ahead of ourselves with all this AGI hype. Remember the developed world, which is what? 2% growth and if you adjust for inflation it’s zero?

So in 2025, as we sit here, I'm not an economist, at least I look at it and say we have a real growth challenge. So, the first thing that we all have to do is, when we say this is like the Industrial Revolution, let's have that Industrial Revolution type of growth.

That means to me, 10%, 7%, developed world, inflation-adjusted, growing at 5%. That's the real marker. It can't just be supply-side.

In fact that’s the thing, a lot of people are writing about it, and I'm glad they are, which is the big winners here are not going to be tech companies. The winners are going to be the broader industry that uses this commodity that, by the way, is abundant. Suddenly productivity goes up and the economy is growing at a faster rate. When that happens, we'll be fine as an industry.

But that's to me the moment. Us self-claiming some AGI milestone, that's just nonsensical benchmark hacking to me. The real benchmark is: the world growing at 10%.

Dwarkesh Patel

Okay, so if the world grew at 10%, the world economy is $100 trillion or something, if the world grew at 10%, that's like an extra $10 trillion in value produced every single year. If that is the case, you as a hyperscaler... It seems like $80 billion is a lot of money. Shouldn't you be doing like $800 billion?

If you really think in a couple of years, we could be really growing the world economy at this rate, and the key bottleneck would be: do you have the compute necessary to deploy these AIs to do all this work?

Satya Nadella

That is correct. But by the way, the classic supply side is, "Hey, let me build it and they’ll come." That's an argument, and after all we've done that, we've taken enough risk to go do it.

But at some point, the supply and demand have to map. That's why I'm tracking both sides of it. You can go off the rails completely when you are hyping yourself with the supply-side, versus really understanding how to translate that into real value to customers.

That's why I look at my inference revenue. That's one of the reasons why even the disclosure on the inference revenue... It's interesting that not many people are talking about their real revenue, but to me, that is important as a governor for how you think about it.

You're not going to say they have to symmetrically meet at any given point in time, but you need to have existence proof that you are able to parlay yesterday's, let’s call it capital, into today's demand, so that then you can again invest, maybe exponentially even, knowing that you're not going to be completely rate mismatched.

Dwarkesh Patel

I wonder if there's a contradiction in these two different viewpoints, because one of the things you've done wonderfully is make these early bets. You invested in OpenAI in 2019, even before there was Copilot and any applications.

If you look at the Industrial Revolution, these 6%, 10% build-outs of railways and whatever things, many of those were not like, "We've got revenue from the tickets, and now we're going to..."

Satya Nadella

There was a lot of money lost.

Dwarkesh Patel

That's true. So, if you really think there's some potential here to 10x or 5x the growth rate of the world, and then you're like, "Well, what is the revenue from GPT-4?"

If you really think that's the possibility from the next level up, shouldn't you just, "Let's go crazy, let's do the hundreds of billions of dollars of compute?" I mean, there's some chance, right?

Satya Nadella

Here’s the interesting thing, right? That's why even that balanced approach to the fleet, at least, is very important to me. It's not about building compute. It's about building compute that can actually help me not only train the next big model but also serve the next big model. Until you do those two things, you're not going to be able to really be in a position to take advantage of even your investment.

So, that's kind of where it's not a race to just building a model, it's a race to creating a commodity that is getting used in the world to drive… You have to have a complete thought, not just one thing that you’re thinking about.

And by the way, one of the things is that there will be overbuild. To your point about what happened in the dotcom era, the memo has gone out that, hey, you know, you need more energy, and you need more compute. Thank God for it. So, everybody's going to race.

In fact, it's not just companies deploying, countries are going to deploy capital, and there will be clearly... I'm so excited to be a leaser, because, by the way; I build a lot, I lease a lot. I am thrilled that I'm going to be leasing a lot of capacity in '27, '28 because I look at the builds, and I'm saying, "This is fantastic." The only thing that's going to happen with all the compute builds is the prices are going to come down.

Dwarkesh Patel

Speaking of prices coming down, you recently tweeted after the DeepSeek model came out about Jevons’ Paradox. I'm curious if you can flesh that out. Jevons’ Paradox occurs when the demand for something is highly elastic. Is intelligence that bottlenecked on prices going down?

Because when I think about, at least my use cases as a consumer, intelligence is already so cheap. It's like two cents per million tokens. Do I really need it to go down to 0.02 cents? I'm just really bottlenecked on it becoming smarter. If you need to charge me 100x, do a 100x bigger training run. I'm happy for companies to take that.

But maybe you're seeing something different on the enterprise side or something. What is the key use case of intelligence that really requires it to get to 0.002 cents per million tokens?

Satya Nadella

I think the real thing is the utility of the tokens. Both need to happen: One is intelligence needs to get better and cheaper. And anytime there's a breakthrough, like even what DeepSeek did, with the efficient frontier of performance per token changes, the curve gets bent, and the frontier moves. That just brings more demand. That's what happened with cloud.

Here’s an interesting thing: We used to think “oh my God, we've sold all the servers in the client-server era”. Except once we started putting servers in the cloud, suddenly people started consuming more because they could buy it cheaper, and it was elastic, and they could buy it as a meter versus a license, and it completely expanded.

I remember going, let’s say, to a country like India and talking about “here is SQL Server”. We sold a little, but man, the cloud in India is so much bigger than anything that we were able to do in the server era. I think that's going to be true.

If you think about, if you want to really have, in the Global South, in a developing country, if you had these tokens that were available for healthcare that were really cheap, that would be the biggest change ever.

Dwarkesh Patel

I think it's quite reasonable for somebody to hear people like me in San Francisco and think “they're kind of silly; they don't know what it's actually like to deploy things in the real world”.

As somebody who works with these Fortune 500s and is working with them to deploy things for hundreds of millions, billions of people, what's your sense on how fast deployment of these capabilities will be?

Even when you have working agents, even when you have things that can do remote work for you, with all the compliance and with all the inherent bottlenecks, is that going to be a big bottleneck, or is that going to move past pretty fast?

Satya Nadella

It is going to be a real challenge because the real issue is change management or process change. Here's an interesting thing: one of the analogies I use is, just imagine how a multinational corporation like us did forecasts pre-PC, and email, and spreadsheets. Faxes went around. Somebody then got those faxes and did an interoffice memo that then went around, and people entered numbers, and then ultimately a forecast came, maybe just in time for the next quarter.

Then somebody said, "Hey, I'm just going to take an Excel spreadsheet, put it in email, send it around. People will go edit it, and I'll have a forecast." So, the entire forecasting business process changed because the work artifact and the workflow changed.

That is what needs to happen with AI being introduced into knowledge work. In fact, when we think about all these agents, the fundamental thing is there's a new work and workflow.

For example, even prepping for our podcast, I go to my copilot and I say, "Hey, I'm going to talk to Dwarkesh about our quantum announcement and this new model that we built for game generation. Give me a summary of all the stuff that I should read up on before going." It knew the two Nature papers, it took that. I even said, "Hey, go give it to me in a podcast format." And so, it even did a nice job of two of us chatting about it.

So that became—and in fact, then I shared it with my team. I took it and put it into Pages, which is our artifact, and then shared. So the new workflow for me is I think with AI and work with my colleagues.

That's a fundamental change management of everyone who's doing knowledge work, suddenly figuring out these new patterns of "How am I going to get my knowledge work done in new ways?" That is going to take time. It's going to be something like in sales, and in finance, and supply chain.

For an incumbent, I think that this is going to be one of those things where—you know, let's take one of the analogies I like to use is what manufacturers did with Lean. I love that because, in some sense, if you look at it, Lean became a methodology of how one could take an end-to-end process in manufacturing and become more efficient. It's that continuous improvement, which is reduce waste and increase value.

That's what's going to come to knowledge. This is like Lean for knowledge work, in particular. And that's going to be the hard work of management teams and individuals who are doing knowledge work, and that's going to take its time.

Dwarkesh Patel

Can I ask you just briefly about that analogy? One of the things Lean did is physically transform what a factory floor looks like. It revealed bottlenecks that people didn't realize until you're really paying attention to the processes and workflows.

You mentioned briefly what your own workflow—how your own workflow has changed as a result of AIs. I'm curious if we can add more color to what will it be like to run a big company when you have these AI agents that are getting smarter and smarter over time?

Satya Nadella

It's interesting you ask that. I was thinking, for example, today if I look at it, we are very email heavy. I get in in the morning, and I’m like, man my inbox is full, and I'm responding, and so I can't wait for some of these Copilot agents to automatically populate my drafts so that I can start reviewing and sending.

But I already have in Copilot at least ten agents, which I query them different things for different tasks. I feel like there's a new inbox that's going to get created, which is my millions of agents that I'm working with will have to invoke some exceptions to me, notifications to me, ask for instructions.

So at least what I'm thinking is that there's a new scaffolding, which is the agent manager. It's not just a chat interface. I need a smarter thing than a chat interface to manage all the agents and their dialogue.

That's why I think of this Copilot, as the UI for AI, is a big, big deal. Each of us is going to have it. So basically, think of it as: there is knowledge work, and there's a knowledge worker. The knowledge work may be done by many, many agents, but you still have a knowledge worker who is dealing with all the knowledge workers. And that, I think, is the interface that one has to build.

Dwarkesh Patel

You're one of the few people in the world who can say that you have access to 200,000… you have this swarm of intelligence around you in the form of Microsoft the company and all its employees. And you have to manage that, and you have to interface with that, how to make best use of that. Hopefully, more of the world will get to have that experience in the future.

I'd be curious about how your inbox, if that means everybody's inbox, will look like yours in the morning.

Okay, before we get to that, I want to keep asking you more about AI, but I really want to ask you about the big breakthrough in quantum that Microsoft Research has announced. So can you explain what's going on?

Satya Nadella

This has been another 30-year journey for us. It's unbelievable. I'm the third CEO of Microsoft who's been excited about quantum.

The fundamental breakthrough here, or the vision that we've always had is, you need a physics breakthrough in order to build a utility-scale quantum computer that works. We took the path of saying, the one way for having a less noisy or more reliable qubit is to bet on a physical property that by definition is more reliable and that's what led us to the Majorana zero modes, which was theorized in the 1930s. The question was, can we actually physically fabricate these things? Can we actually build them?

So the big breakthrough effectively, and I know you talked to Chetan, was that we now finally have existence proof and a physics breakthrough of Majorana zero modes in a new phase of matter effectively. This is why we like the analogy of thinking of this as the transistor moment of quantum computing, where we effectively have a new phase, which is the topological phase, which means we can even now reliably hide the quantum information, measure it, and we can fabricate it. And so now that we have it, we feel like with that core foundational fabrication technique out of the way, we can start building a Majorana chip.

That Majorana One which I think is going to basically be the first chip that will be capable of a million qubits, physical. And then on that, thousands of logical qubits, error-corrected. And then it's game on. You suddenly have the ability to build a real utility-scale quantum computer, and that to me is now so much more feasible. Without something like this, you will still be able to achieve milestones, but you'll never be able to build a utility-scale computer. That's why we're excited about it.

Dwarkesh Patel

Amazing. And by the way, I believe this is it right here.

Satya Nadella

That is it.

Dwarkesh Patel

Yes.

Satya Nadella

I forget now, are we calling it Majorana? Yes, that's right. Majorana One. I'm glad we named it after that.

To think that we are able to build something like a million-qubit quantum computer in a thing of this size is just unbelievable. That's the crux of it: unless and until we could do that, you can't dream of building a utility-scale quantum computer.

Dwarkesh Patel

And you're saying the eventual million qubits will go on a chip this size? Okay, amazing.

Other companies have announced 100 physical qubits, Google's, IBM's, others. When you say you've announced one, but you're saying that yours is way more scalable in the limit.

Satya Nadella

Yeah. The one thing we’ve also done is we've taken an approach where we've separated our software and our hardware. We're building out our software stack, and we now have, with the neutral atom folks, the ion trap folks, we're also working with others who even have pretty good approaches with photonics and what have you, that means there'll be different types of quantum computers. In fact, we have what, I think that the last thing that we announced was 24 logical qubits. So we have also got some fantastic breakthroughs on error correction and that's what is allowing us, even on neutral atom and ion trap quantum computers, to build these 20 plus, and I think that'll keep going even throughout the year; you'll see us improve that yardstick.

But we also then said, "Let's go to the first principles and build our own quantum computer that is betting on the topological qubit." And that's what this breakthrough is about.

Dwarkesh Patel

Amazing. The million topological qubits, thousands of logical qubits, what is the estimated timeline to scale up to that level? What does the Moore's law here, if you've got the first transistor, look like?

Satya Nadella

We've obviously been working on this for 30 years. I'm glad we now have the physics breakthrough and the fabrication breakthrough.

I wish we had a quantum computer because by the way, the first thing the quantum computer will allow us to do is build quantum computers, because it's going to be so much easier to simulate atom-by-atom construction of these new quantum gates.

But in any case, the next real thing is, now that we have the fabrication technique, let us go build that first fault-tolerant quantum computer. And that will be the logical thing.

So, I would say now I can say, "Oh, maybe '27, '28, '29, we will be able to actually build this." Now that we have this one gate, can I put the thing into an integrated circuit and then actually put these integrated circuits into a real computer? That is where the next logical step is.

Dwarkesh Patel

And what do you see as, in '27, '28, you've got it working? Is it a thing you access through the API? Is it something you're using internally for your own research in materials and chemistry?

Satya Nadella

It’s a great question. One thing that I've been excited about is, even in today's world… we had this quantum program, and we added some APIs to it. The breakthrough we had maybe two years ago was to think of this HPC stack, and AI stack, and quantum together.

In fact, if you think about it, AI is like an emulator of the simulator. Quantum is like a simulator of nature. What is quantum going to do? By the way, quantum is not going to replace classical. Quantum is great at what quantum can do, and classical will also...

Quantum is going to be fantastic for anything that is not data-heavy but is exploration-heavy in terms of the state space. It should be data-light but exponential states that you want to explore. Simulation is a great one: chemical physics, what have you, biology.

One of the things that we've started doing is really using AI as the emulation engine. But you can then train. So the way I think of it is, if you have AI plus quantum, maybe you'll use quantum to generate synthetic data that then gets used by AI to train better models that know how to model something like chemistry or physics or what have you. These two things will get used together.

So even today, that's effectively what we're doing with the combination of HPC and AI. I hope to replace some of the HPC pieces with quantum computers.

Dwarkesh Patel

Can you tell me a little bit about how you make these research decisions which, in 20 years time, 30 years time, will actually pay dividends, especially at a company of Microsoft's scale? Obviously, you're in great touch with the technical details in this project. Is it feasible for you to do that with all the things Microsoft Research does?

How do you know the current bet you're making will pay out in 20 years? Does it just have to emerge organically through the org, or how are you keeping track of all this?

Satya Nadella

The thing that I feel was fantastic is when Bill, when he started MSR back in '95 I guess. I think in the long history of these curiosity-driven research organizations, to just do a research org that is about fundamental research and MSR, over the years, has built up that institutional strength so when I think about capital allocation or budgets, we first put the chips in and say, "Here is MSR's budget." We gotta go at it each year knowing that most of these bets are not going to pay off in any finite time frame. Maybe the sixth CEO of Microsoft will benefit from it. And in tech that is I think a given.

The real thing that I think about is, when the time has come for something like quantum or a new model or what have you, can you capitalize? So as an incumbent, if you look at the history of tech, it's not that people didn't invest. It's that you need to have a culture that knows how to take an innovation and scale it.

That's the hard part, quite frankly, for CEOs and management teams. Which is kind of fascinating. It's as much about good judgment as it is about good culture. Sometimes we've gotten it right; sometimes we've gotten it wrong; I can tell you the thousand projects from MSR that we should have probably led with, but we didn't. And I always ask myself why. It's because we were not able to get enough conviction and that complete thought of how to not only take the innovation but make it into a useful product with a business model that we can then go to market with.

That's the job of CEOs and management teams: not to just be excited about any one thing, but to be able to actually execute on a complete thing. And that's easier said than done.

Dwarkesh Patel

When you mentioned the possibility of three subsequent CEOs of Microsoft, if each of them increases the market cap by an order of magnitude, by the time you've got the next breakthrough, you'll be like the world economy or something.

Satya Nadella

Or remember, the world is going to be growing at 10%, so we'll be fine.

Dwarkesh Patel

Let's dig into the other big breakthrough you've just made. It's amazing that you have both of them coming out the same day, in your gaming world models. I'd love if you can tell me a little bit about that.

Satya Nadella

We're going to call it Muse. It's going to be the model of this world action, or human action model.

This is very cool. One of the things is that obviously, Dall-E and Sora have been unbelievable in what they've been able to do in terms of generative models. One thing that we wanted to go after was using gameplay data. Can you actually generate games that are both consistent and then have the ability to generate the diversity of what that game represents, and then are persistent to user mods?

That's what this is. They were able to work with one of our game studios, and this is the other publication in Nature.

The cool thing is what I'm excited about is bringing--we're going to have a catalog of games soon that we will start using these models, or we're going to train these models to generate, and then start playing them.

In fact, when Phil Spencer first showed it to me, he had an Xbox controller and this model basically took the input and generated the output based on the input. And it was consistent with the game. That to me is a massive moment of “wow”. It's kind of like the first time we saw ChatGPT complete sentences, or Dall-E draw, or Sora. This is one such moment.

Dwarkesh Patel

I got a chance to see some of the videos in the real-time demo this morning with your lead researcher Katja on this. Only once I talked to her did it really hit me how incredible this is, in the sense that we've used AI in the past to model agents, and just using that same technique to model the world around the agent gives consistent real-time – we'll superimpose videos of what this looks like atop this podcast so people can get a chance to see it for themselves. I guess it'll be out by then, so they can also watch it there.

This in itself is incredible. You, through your span as CEO, have invested tens of hundreds of billions of dollars in building up Microsoft Gaming and acquiring IP.

In retrospect, if you can just merge all of this data into one big model that can give you this experience of visiting and going through multiple worlds at the same time, and if this is the direction gaming is headed, it seems like a pretty good investment to have made. Did you have any premonition about this?

Satya Nadella

I wouldn't say that we invested in gaming to build models. We invested, quite frankly, because- here's an interesting thing about our history: We built our first game before we built Windows. Flight Simulator was a Microsoft product long before we even built Windows.

So, gaming has got a long history at the company, and we want to be in gaming for gaming's sake. I always start by saying I hate to be in businesses where they're means to some other end. They have to be ends unto themselves.

And then, yes, we're not a conglomerate. We are a company where we have to bring all these assets together and be better owners by adding value. For example, cloud gaming is a natural thing for us to invest in because that will just expand the TAM and expand the ability for people to play games everywhere.

The same thing with AI and gaming: we definitely think that it can be helpful in maybe changing- it's kind of like the CGI moment, even for gaming long-term. And it's great. As the biggest, world's largest publisher, this will be helpful. But at the same time, we've got to produce great quality games. I mean, you can't be a gaming publisher without, sort of, first and foremost being focused on that.

But the fact that this data asset is going to be interesting, not just in a gaming context, but it's going to be a general action model and a world model, it's fantastic. I mean like, you know, I think about gaming data as perhaps, you know, what YouTube is perhaps to Google, gaming data is to Microsoft. And so therefore I'm excited about that.

Dwarkesh Patel

Yeah, and that's what I meant, just in the sense of like, you can have one unified experience across many different kinds of games. How does this fit into the other, separate from AI, the other things that Microsoft has worked on in the past, like mixed reality? Maybe giving smaller game studios a chance to build these AAA action games? Just like five, ten years from now, what kinds of ways could you imagine?

Satya Nadella

I've thought about these three things as the cornerstones of, in an interesting way, even five, six, seven years ago is when I said the three big bets that we want to place [are] AI, quantum, and mixed reality. And I still believe in them, because in some sense, what are the big problems to be solved?

Presence. That's the dream of mixed reality. Can you create real presence? Like you and I doing a podcast like this.

I think it’s still proving to be the harder one of those challenges, quite honestly. I thought it was going to be more solvable. It's tougher, perhaps, just because of the social side of it: wearing things and so on.

We're excited about, in fact, what we're going to do with Anduril and Palmer, now, with even how they'll take forward the IVAS program, because that's a fantastic use case. And so we'll continue on that front.

But also, the 2D surfaces. It turns out things like Teams, right, thanks to the pandemic, we've really gotten the ability to create essentially presence through even 2D. And that I think will continue. That's one secular piece.

Quantum we talked about, and AI is the other one. So these are the three things that I look at and say, how do you bring these things together? Ultimately, not as tech for tech's sake, but solving some of the fundamental things that we, as humans, want in our life, and more, we want them in our economy, driving our productivity. And so if we can somehow get that right, then I think we will have really made progress.

Dwarkesh Patel

When you write your next book, you've got to have some explanation of why those three pieces all came together around the same time, right? Like, there's no intrinsic reason you would think quantum and AI should happen in 2028 and 2025 and so forth.

Satya Nadella

That's right. At some level, I look at it and say: the simple model I have is, hey is there a systems breakthrough? To me, the systems breakthrough is the quantum thing.

Is there a business logic breakthrough? That's AI to me, which is: can the logic tier be fundamentally reasoned differently? Instead of imperatively writing code, can you have a learning system? That's the AI one.

And then the UI side of it is presence.

Dwarkesh Patel

Going back to AI for a second, in your 2017 book… 2019 you invest in OpenAI, very early, 2017 is even earlier, you say in your book, "One might also say that we're birthing a new species, one whose intelligence may have no upper limits."

Now, super-early, of course, to be talking about this in 2017. We've been talking in a granular fashion about agents, Office Copilot, capex, and so forth. But if you zoom out and consider this statement you've made, and you think about you as a hyperscaler, as the person doing research in these models as well, providing training, inference, and research for building a new species, how do you think about this in the grand scheme of things?

Do you think we're headed towards superhuman intelligence in your time as CEO?

Satya Nadella

I think even Mustafa uses that term. In fact he’s used that term more recently, this “new species”.

The way I come at it is, you definitely need trust. Before we claim it is something as big as a species, the fundamental thing that we've got to get right is that there is real trust, whether it's personal or societal level trust, that's baked in. That's the hard problem.

I think the one biggest rate limiter to the power here will be how does our legal… call it infrastructure, we’re talking about all the compute infrastructure, well how does the legal infrastructure evolve to deal with this? This entire world is constructed with things like humans owning property, having rights, and being liable. That’s the fundamental thing that one has to first say, okay what does that mean for anything that now humans are using as tools? And if humans are going to delegate more authority to these things, then how does that structure evolve? Until that really gets resolved, I don't think just talking about the tech capability is going to happen.

Dwarkesh Patel

As in, we won't be able to deploy these kinds of intelligences until we figure out how to…?

Satya Nadella

Absolutely. Because at the end of the day, there is no way. Today, you cannot deploy these intelligences unless and until there's someone indemnifying it as a human.

To your point, I think that's one of the reasons why I think about even the most powerful AI is essentially working with some delegated authority from some human. You can say, oh, that's all alignment and this, that, and the other. That's why I think you have to really get these alignments to work and be verifiable in some way, but I just don't think that you can deploy intelligences that are out of control. For example, this AI takeoff problem may be a real problem, but before it is a real problem, the real problem will be in the courts. No society is going to allow for some human to say, "AI did that."

Dwarkesh Patel

Yes. Well, there's a lot of societies in the world, and I wonder if any one of them might not have a legal system that might be more amenable. And if you can't have a takeoff, then you might worry. It doesn't have to happen in America, right?

Satya Nadella

We think that no society cares about it, right? There can be rogue actors, I'm not saying there won't be rogue actors; there are cyber criminals and rogue states; they're going to be there.

But to think that human society at large doesn't care about it is also not going to be true. I think we all will care. We know how to deal with rogue states and rogue actors today. The world doesn't sit around and say “we’ll tolerate that”. That's why I'm glad that we have a world order in which anyone who is a rogue actor in a rogue state has consequences.

Dwarkesh Patel

Right. But if you have this picture where you can have 10% economic growth, I think it really depends on getting something like AGI working, because tens of trillions of dollars of value, that sounds closer to the total of human wages, around $60 trillion of the economy. Getting that magnitude, you kind of have to automate labor or supplement labor in a very significant way.

If that is possible, and once we figure out the legal ramifications for it, it seems quite plausible, even within your tenure that we figure that out. Are you thinking about superhuman intelligence? Like, the biggest thing you do in your career is this?

Satya Nadella

You bring up another point. I know David Autor and others have talked a lot about this which is, 60% of labor- I think the other question that needs to happen, let’s at least talk about our democratic societies. I think that in order to have a stable social structure, and democracies function, you can’t just have a return on capital and no return on labor. We can talk about it, but that 60% has to be revalued.

In my own simple way, maybe you can call it naive, we'll start valuing different types of human labor. What is today considered high-value human labor may be a commodity. There may be new things that we will value.

Including that person who comes to me and helps me with my physical therapy or whatever, whatever is going to be the case that we value, but ultimately, if we don't have return on labor, and there's meaning in work and dignity in work and all of that, that's another rate limiter to any of these things being deployed.

Dwarkesh Patel

On the alignment side, two years ago, you guys released Sydney Bing. Just to be clear, I think given the level of capabilities at the time, it was a charming, endearing, kind of funny example of misalignment.

But that was because, at the time, it was like chatbots. They can go think for 30 seconds and give you some funny or inappropriate response. But if you think about that kind of system--that, I think to a New York Times reporter, tried to get him to leave his wife or something--if you think about that going forward, and you have these agents that are for hours, weeks, months going forward, just like autonomous swarms of AGIs, who could be in similar ways misaligned and screwing stuff up, maybe coordinating with each other, what's your plan going forward so that when you get the big one, you get it right?

Satya Nadella

That is correct. That's one of the reasons why when we usually allocate compute, let's allocate compute for what is that alignment challenge?

And then more importantly, what is the runtime environment in which you are really going to be able to monitor these things? The observability around it? We do deal with a lot of these things today in the classical side of things as well, like cyber. We don't just write software and then just let it go. You have software and then you monitor it. You monitor it for cyber attacks, you monitor it for fault injections, and what have you.

Therefore, I think we will have to build enough software engineering around the deployment side of these, and then inside the model itself, what's the alignment? These are all, some of them are real science problems. Some of them are real engineering problems, and then we will have to tackle it.

That also means taking our own liability in all of this. So that's why I'm more interested in deploying these things in where you can actually govern what the scope of these things is, and the scale of these things is. You just can't unleash something out there in the world that creates harm, because the social permission for that is not going to be there.

Dwarkesh Patel

When you get the agents that can really just do weeks worth of tasks for you, what is the minimum assurance you want before you can let it run a random Fortune 500?

Satya Nadella

I think when I use something like Deep Research, even, the minimum assurance I think we want is before we especially have physical embodiment of anything, that I think is kind of one of those thresholds, when you cross. That might be one place.

Then the other one is, for example, the permissions of the runtime environment in which this is operating. You may want guarantees that it's sandboxed, it is not going out of that sandbox.

Dwarkesh Patel

I mean, we already have web search and we already have it out of the sandbox.

Satya Nadella

But even what it does with web search and what it writes -- for example to your point, if it's just going to write a bunch of code in order to do some computation, where is that code deployed? And is that code ephemeral for just creating that output, versus just going and springing that code out into the world?

Those are things that you could, in the action space, actually go control.

Dwarkesh Patel

And separate from the safety issues, as you think about your own product suite, and you think about, if you do have AIs this powerful, at some point, it's not just like Copilot- an example you mentioned about how you were prepping for this podcast- it's more similar to how you actually delegate work to your colleagues.

What does it look like, given your current suite, to add that in? I mean, there's one question about whether LLMs get commodified by other things.

I wonder if these databases or canvases or Excel sheets or whatever -- if the LLM is your main gate point into accessing all these things, is it possible that the LLMs commodify Office?

Satya Nadella

It's an interesting one. The way I think about the first phase, at least, would be: Can the LLM help me do my knowledge work using all of these tools or canvases more effectively?

One of the best demos that I've seen is a doctor getting ready for a tumor board workflow. She's going into a tumor board meeting, and the first thing she uses Copilot for is to create an agenda for the meeting because the LLM helps reason about all the cases, which are in some SharePoint site. It says, "Hey, these cases -- obviously, a tumor board meeting is a high-stakes meeting where you want to be mindful of the differences in cases so that you can then allocate the right time."

Even that reasoning task of creating an agenda that knows how to split time- super. So, I use the LLM to do that. Then I go into the meeting, I'm in a Teams call with all my colleagues. I'm focused on the actual case versus taking notes, because you now have this AI copilot doing a full transcription of all of this. It's not just a transcript, but a database entry of what is in the meeting that is recallable for all time.

Then she comes out of the meeting, having discussed the case and not been distracted by note-taking. She's a teaching doctor; she wants to go and prep for her class. And so she goes into Copilot and says, "Take my tumor board meeting and create a PowerPoint slide deck out of it so that I can talk to my students about it."

So that’s the type. The UI and the scaffolding that I have are canvases that are now getting populated using LLMs. And the workflow itself is being reshaped; knowledge work is getting done.

Here's an interesting thing: If someone came to me in the late '80s and said, "You're going to have a million documents on your desk," I would say, "What the heck is that?" I would have literally thought there was going to be a million physical copies of things on my desk. Except, we do have a million spreadsheets and a million documents.

Dwarkesh Patel

I don’t, you do.

Satya Nadella

They're all there. And so, that's what's going to happen with even agents. There will be a UI layer. To me, Office is not just about the office of today; it's the UI layer for knowledge work. It'll evolve as the workflows evolve. That's what we want to build.

I do think the SaaS applications that exist today, these CRUD applications, are going to fundamentally be changed because the business logic will go more into this agentic tier. In fact, one of the other cool things today in my Copilot experience is when I say, "Hey, I'm getting ready for a meeting with a customer," I just go and say, "Give me all the notes for it that I should know." It pulls from my CRM database, it pulls from my Microsoft Graph, creates a composite, essentially artifact, and then it applies even logic on it. That, to me, is going to transform the SaaS applications as we know of it today.

Dwarkesh Patel

SaaS as an industry might be worth hundreds of billions to trillions of dollars a year, depending on how you count. If really that can just get collapsed by AI, is the next step up in your next decade 10X-ing the market cap of Microsoft again? Because you're talking about trillions of dollars...

Satya Nadella

It would also create a lot of value in the SaaS. One thing we don't pay as much attention to perhaps is the amount of IT backlog there is in the world.

These code gen things, plus the fact that I can interrogate all of your SaaS applications using agents and get more utility will be the greatest explosion of apps, they'll be called agents, so that for every vertical, in every industry, in every category, we're suddenly going to have the ability to be serviced.

So there's going to be a lot of value. You can't stay still. You can't just say the old thing of, "Oh, I schematized some narrow business process, and I have a UI in the browser, and that's my thing." That's ain’t going to be the case. You have to go up-stack and say, "What's the task that I have to participate in?"

You will want to be able to take your SaaS application and make it a fantastic agent that participates in a multi-agent world. As long as you can do that, then I think you can even increase the value.

Dwarkesh Patel

Can I ask you some questions about your time at Microsoft?

Satya Nadella

Yeah.

Dwarkesh Patel

Is being a company man underrated? So you've spent most of your career at Microsoft, and you could say that one of the reasons you've been able to add so much value is you've seen the culture, the history, and the technology. You have all this context by rising up through the ranks. Should more companies be run by people who have this level of context?

Satya Nadella

That's a great question. I've not thought about it that way.

Through my 34 years now of Microsoft, each year I felt more excited about being at Microsoft versus thinking that, oh, I'm a company person or what have you. I take that seriously, even for anybody joining Microsoft. It's not like they're joining Microsoft as long as they feel that they can use this as a platform for their both economic return, but also a sense of purpose and a sense of mission that they can accomplish by using us as a platform. That's the contract.

So I think yes, companies have to create a culture that allows people to come in and become company people like me. Microsoft got it more right than wrong, at least in my case, and I hope that remains the case.

Dwarkesh Patel

The sixth CEO that you’re talking about, who’ll get to use the research you’re starting now, what are you doing to retain the future Satya Nadellas so that they're in a position to become future leaders?

Satya Nadella

It's fascinating. This is our 50th year, and I think a lot about it. The way to think about it is, longevity is not a goal; relevance is.

The thing that I have to do and all 200,000 of us have to do every day is: Are we doing things that are useful and relevant for the world as we see it evolving, not just today, but tomorrow?

We live in an industry where there's no franchise value, so that’s the other hard part. If you take the R&D budget that we will spend this year, it’s all speculation on what's going to happen five years from now. You have to basically go in with that attitude, saying, "We are doing things that we think are going to be relevant."

So that's what you have to focus on. Then know that there's a batting average, and you're not going to get- you have to have a high tolerance for failure. You have to take enough shots on goal to be able to say, "Okay, we will make it to the other side as a company." That's what makes it tricky in this industry.

Dwarkesh Patel

Speaking of- you just mentioned that you're two months away from your 50th anniversary of Microsoft’s founding. If you look at the top 10 companies by market cap, or top 5, basically, everybody else but Microsoft is younger than Microsoft. It's an interesting observation about why the most successful companies often are quite young. The average Fortune 500 company will last 10 to 15 years.

What has Microsoft done to remain relevant for this many years? How do you keep refounding?

Satya Nadella

I love that, Reed Hoffman uses that term, "refounding." That's the mindset. People talk about founder mode, but for us mere mortal CEOs, it's more like refounder mode.

To be able to see things again in a fresh way is the key. To your question: can we culturally create an environment where refounding becomes a habit thing? Every day we come in and say, "We feel we have a stake in this place to be able to change the core assumptions of what we do and how we relate to the world around us. Do we give ourselves permission?” I think many times, companies feel over-constrained by either business model or whatever. You just have to unconstrain yourself.

Dwarkesh Patel

If you did leave Microsoft, what company would you start?

Satya Nadella

Company I would start? Man. That’s where the company man and me sort of says, “I'll never leave Microsoft.”

If I were thinking of doing something, I think picking a domain that has... When I look at the dream of tech, we've always said technology is about the biggest, greatest democratizing force.

I feel like finally, we have that ability. If you say those tokens per dollar per watt is what we can generate, I would love to find some domain in which that can be applied, where it is so underserved.

That's where healthcare, education... Public sector would be another place. If you take those domains, which are the underserved places, where my life as a citizen of this country or a member of this society or anywhere, would I be better off if somehow all this abundance translated into better healthcare, better education, and better public sector institutions serving me as a citizen? That would be a place.

Dwarkesh Patel

One thing I'm not sure about, hearing your answers on different questions, is whether you think AGI is a thing. Will there be a thing which automates all cognitive labor, like anything anybody can do on a computer?

Satya Nadella

This is where I have a problem with the definitions of how people talk about it. Cognitive labor is not a static thing. There is cognitive labor today. If I have an inbox that is managing all my agents, is that new cognitive labor?

Today's cognitive labor may be automated. What about the new cognitive labor that gets created? Both of those things have to be thought of, which is the shifting…

That's why I make this distinction, at least in my head: Don't conflate knowledge worker with knowledge work. The knowledge work of today could probably be automated. Who said my life's goal is to triage my email? Let an AI agent triage my email.

But after having triaged my email, give me a higher-level cognitive labor task of, "Hey, these are the three drafts I really want you to review." That's a different abstraction.

Dwarkesh Patel

But will AI ever get to the second thing?

Satya Nadella

It may, but as soon as it gets to that second thing, there will be a third thing. Why are we thinking that somehow, when we have dealt with tools that have changed what cognitive labor is in history, why are we worried that all cognitive labor will go away?

Dwarkesh Patel

I'm sure you've heard these examples before, but the idea that horses can still be good for certain things, there are certain terrains you can't take a car on. But the idea that you're going to see horses around the street, they’re going to employ millions of horses, it’s just not happening.

And then the idea is, could a similar thing happen with humans?

Satya Nadella

But in one very narrow dimension? It's only 200 years of history of humans where we have valued some narrow sort of things called "cognitive labor" as we understand it.

Let's take something like chemistry. If this thing, quantum plus AI really helped us do a lot of novel material science and so on, that's fantastic to have novel material science being done by it. Does that take away from all the other things that humans can do?

Why can't we exist in a world where there are powerful cognitive machines, knowing that our cognitive agency has not been taken away?

Dwarkesh Patel

I'll ask this question, not about you, but in a different scenario, so maybe you can answer it without embarrassment. Suppose on the Microsoft board, could you ever see adding an AI to the board? Could it ever have the judgment, context, and holistic understanding to be a useful advisor?

Satya Nadella

It's a great example. One of the things we added was a facilitator agent in Teams. The goal there, it's in the early stages, is can that facilitator agent use long-term memory, not just on the context of the meeting, but with the context of projects I'm working on, and the team, and what have you, be a great facilitator?

I would love it even in a board meeting, where it's easy to get distracted. After all, board members come once a quarter, and they're trying to digest what is happening with a complex company like Microsoft. A facilitator agent that actually helped human beings all stay on topic and focus on the issues that matter, that's fantastic.

That's kind of literally having, to your point about even going back to your previous question, having something that has infinite memory that can even help us. You know, after all, what is that Herbert Simon thing? We are all bounded rationality. So if the bounded rationality of humans can actually be dealt with because there is a cognitive amplifier outside, that's great.

Dwarkesh Patel

Speaking of materials and chemistry stuff, I think you said recently that you want the next 250 years of progress in those fields to happen in the next 25 years. Now, when I think about what's going to be possible in the next 250 years, I'm thinking like space travel, and space elevators, and immortality, and curing all diseases. Next 25 years, you think?

Satya Nadella

One of the reasons why I brought that up was, I love that thing of, the industrial revolution was the 250 years. We have to take this entire change from a carbon-based system to something different.

That means you have to fundamentally reinvent all of what has happened with chemistry over the last 250 years. That's where I hope we have this quantum computer, this quantum computer helps us get to new materials, and then we can fabricate those new materials that help us with all of the challenges we have on this planet. And then I'm all for interplanetary travel.

Dwarkesh Patel

Amazing. Satya, thank you so much for your time.

Satya Nadella

Thank you so much. It's wonderful. Thanks.

Dwarkesh Patel

Great, thank you.

This week I welcome on the show two of the most important technologists ever, in any field.

Jeff Dean is Google's Chief Scientist, and through 25 years at the company, has worked on basically the most transformative systems in modern computing: from MapReduce, BigTable, Tensorflow, AlphaChip, to Gemini.

Noam Shazeer invented or co-invented all the main architectures and techniques that are used for modern LLMs: from the Transformer itself, to Mixture of Experts, to Mesh Tensorflow, to Gemini and many other things.

We talk about their 25 years at Google, going from PageRank to MapReduce to the Transformer to MoEs to AlphaChip – and maybe soon to ASI.

My favorite part was Jeff's vision for Pathways, Google’s grand plan for a mutually-reinforcing loop of hardware and algorithmic design and for going past autoregression. That culminates in us imagining *all* of Google-the-company, going through one huge MoE model.

And Noam just bites every bullet: 100x world GDP soon; let’s get a million automated researchers running in the Google datacenter; living to see the year 3000.

Watch on Youtube; listen on Apple Podcasts or Spotify.

• Scale partners with major AI labs like Meta, Google Deepmind, and OpenAI. Through Scale’s Data Foundry, labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you’re an AI researcher or engineer, learn about how Scale’s Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkesh.
  

• Curious how Jane Street teaches their new traders? They use Figgie, a rapid-fire card game that simulates the most exciting parts of markets and trading. It’s become so popular that Jane Street hosts an inter-office Figgie championship every year. Download from the app store or play on your desktop at figgie.com.
  

• Meter wants to radically improve the digital world we take for granted. They’re developing a foundation model that automates network management end-to-end. To do this, they just announced a long-term partnership with Microsoft for tens of thousands of GPUs, and they’re recruiting a world class AI research team. To learn more, go to meter.com/dwarkesh.

To sponsor a future episode, visit dwarkeshpatel.com/p/advertise.

0:00:00 - Intro

0:03:29 - Joining Google in 1999

0:06:20 - Future of Moore's Law

0:11:04 - Future TPUs

0:13:56 - Jeff’s undergrad thesis: parallel backprop

0:15:54 - LLMs in 2007

0:25:09 - “Holy shit” moments

0:27:28 - AI fulfills Google’s original mission

0:32:00 - Doing Search in-context

0:36:12 - The internal coding model

0:37:29 - What will 2027 models do?

0:43:20 - A new architecture every day?

0:49:10 - Automated chips and intelligence explosion

0:53:07 - Future of inference scaling

1:02:38 - Already doing multi-datacenter runs

1:08:15 - Debugging at scale

1:12:41 - Fast takeoff and superalignment

1:20:51 - A million evil Jeff Deans

1:24:22 - Fun times at Google

1:27:51 - World compute demand in 2030

1:34:37 - Getting back to modularity

1:44:48 - Keeping a giga-MoE in-memory

1:49:35 - All of Google in one model

1:57:59 - What’s missing from distillation

2:03:10 - Open research, pros and cons

2:09:58 - Going the distance

Dwarkesh Patel 00:00:00

Today I have the honor of chatting with Jeff Dean and Noam Shazeer. Jeff is Google's Chief Scientist, and through his 25 years at the company, he has worked on basically the most transformative systems in modern computing: from MapReduce, BigTable, Tensorflow, AlphaChip – genuinely, the list doesn't end – Gemini now.

And Noam is the single person most responsible for the current AI revolution. He has been the inventor or co-inventor of all the main architectures and techniques that are used for modern LLMs: from the Transformer itself, to Mixture of Experts, to Mesh Tensorflow, to many other things. And they are two of the three co-leads of Gemini at Google DeepMind. Awesome. Thanks so much for coming on.

Jeff Dean 00:00:50

Thank you. Super excited to be here.

Dwarkesh Patel 00:00:52

Okay, first question. Both of you have been at Google for 25, or close to 25, years. At some point early on in the company, you probably understood how everything worked. When did that stop being the case? Do you feel like there was a clear moment that happened?

Noam Shazeer 00:01:08

I joined, this was like, end of 2000, and they had this thing: everybody gets a mentor. I knew nothing. I would just ask my mentor everything, and my mentor knew everything. It turned out my mentor was Jeff.

It was not the case that everyone at Google knew everything. It was just the case that Jeff knew everything because he had basically written everything.

Jeff Dean 00:01:33

You're very kind. I think as companies grow, you kind of go through these phases. When I joined, we were 25 people, 26 people, something like that. So you eventually you learned everyone's name, and even though we were growing, you kept track of all the people who were joining.

At some point, you lose track of everyone's name in the company, but you still know everyone working on software engineering things. Then you lose track of all the names of people in the software engineering group, but you at least know all the different projects that everyone's working on. Then at some point, the company gets big enough that you get an email that Project Platypus is launching on Friday, and you're like, "What the heck is Project Platypus?"

Noam Shazeer 00:02:15

Usually it's a very good surprise. You're like, "Wow, Project Platypus!" I had no idea we were doing that.

Jeff Dean 00:02:23

But I think it is good to keep track of what's going on in the company, even at a very high level, even if you don't know every last detail. And it's good to know lots of people throughout the company so that you can go ask someone for more details or figure out who to talk to. With one level of indirection, you can usually find the right person in the company if you have a good network of people that you've built up over time.

Dwarkesh Patel 00:02:44

How did Google recruit you, by the way?

Jeff Dean 00:02:46

I kind of reached out to them, actually.

Dwarkesh Patel 00:02:50

And Noam, how did you get recruited?

Noam Shazeer 00:02:53

I actually saw Google at a job fair in 1999, and I assumed that it was already this huge company, that there was no point in joining, because everyone I knew used Google. I guess that was because I was a grad student at Berkeley at the time. I guess I've dropped out of grad programs a few times.

It turns out that actually it wasn't really that large. It turns out that I did not apply in 1999, but just kind of sent them a resume on a whim in 2000, because I figured it was my favorite search engine, and figured I should apply to multiple places for a job. But then it turned out to be really fun, it looked like a bunch of smart people doing good stuff. They had this really nice crayon chart on the wall of the daily number of search queries that somebody had just been maintaining. It looked very exponential. I thought, "These guys are going to be very successful, and it looks like they have a lot of good problems to work on." So I was like, "Okay, maybe I'll go work there for a little while and then have enough money to just go work on AI for as long as I want after that."

Dwarkesh Patel 00:04:08

Yeah, yeah. In a way you did that, right?

Noam Shazeer 00:04:10

Yeah, it totally worked out exactly according to plan.

Dwarkesh Patel 00:04:15

You were thinking about AI in 1999?

Noam Shazeer 00:04:17

Yeah, this was like 2000. Yeah, I remember in grad school, a friend of mine at the time had told me that his New Year's resolution for 2000 was to live to see the year 3000, and that he was going to achieve this by inventing AI. I was like, "Oh, that sounds like a good idea."

I didn't get the idea at the time that you could go do it at a big company. But I figured, "Hey, a bunch of people seem to be making a ton of money at startups. Maybe I'll just make some money, and then I'll have enough to live on and just work on AI research for a long time." But yeah, it actually turned out that Google was a terrific place to work on AI.

Jeff Dean 00:05:07

One of the things I like about Google is our ambition has always been sort of something that would require pretty advanced AI. Because I think organizing the world's information and making it universally accessible and useful, actually there is a really broad mandate in there. It's not like the company was going to do this one little thing and stay doing that. And also you could see that what we were doing initially was in that direction, but you could do so much more in that direction.

Dwarkesh Patel 00:05:36

How has Moore's Law over the last two or three decades changed the kinds of considerations you have to take on board when you design new systems, when you figure out what projects are feasible? What are still the limitations? What are things you can now do that you obviously couldn't do before?

Jeff Dean 00:05:51

I think of it as actually changing quite a bit in the last couple of decades. Two decades ago to one decade ago, it was awesome because you just wait, and like 18 months later, you get much faster hardware, and you don't have to do anything. And then more recently, I feel like the general-purpose CPU-based machine scaling has not been as good, like the fabrication process improvements are now taking three years instead of every two years. The architectural improvements in multi-core processors and so on are not giving you the same boost that we were getting 20 to 10 years ago. But I think at the same time, we're seeing much more specialized computational devices, like machine learning accelerators, TPUs, and very ML-focused GPUs, more recently, are making it so that we can actually get really high performance and good efficiency out of the more modern kinds of computations we want to run that are different than a twisty pile of C++ code trying to run Microsoft Office or something.

Noam Shazeer 00:07:02

It feels like the algorithms are following the hardware. Basically, what's happened is that at this point, arithmetic is very, very cheap, and moving data around is comparatively much more expensive. So pretty much all of deep learning has taken off roughly because of that. You can build it out of matrix multiplications that are N cubed operations and N squared bytes of data communication basically.

Jeff Dean 00:07:39

Well, I would say that the pivot to hardware oriented around that was an important transition, because before that, we had CPUs and GPUs that were not especially well-suited for deep learning. And then we started to build TPUs at Google that were really just reduced-precision linear algebra machines, and then once you have that then you want to exploit it.

Noam Shazeer 00:08:02

It seems like it's all about identifying opportunity costs. Like, okay, this is something like Larry Page, I think, used to always say: "Our second biggest cost is taxes, and our biggest cost is opportunity costs." If he didn't say that, then I've been misquoting him for years.

But basically it’s like, what is the opportunity that you have that you're missing out on? In this case, I guess it was that you've got all of this chip area, and you're putting a very small number of arithmetic units on it. Fill the thing up with arithmetic units! You could have orders of magnitude more arithmetic getting done.

Now, what else has to change? Okay, the algorithms and the data flow and everything else.

Jeff Dean 00:08:51

And, oh, by the way, the arithmetic can be really low precision, so then you can squeeze even more multiplier units in.

Dwarkesh Patel 00:08:58

Noam, I want to follow up on what you said, that the algorithms have been following the hardware. If you imagine a counterfactual world where, suppose that the cost of memory had declined more than arithmetic, or just invert the dynamic you saw.

Noam Shazeer 00:09:12

Okay, data flow is extremely cheap, and arithmetic is not.

Dwarkesh Patel 00:09:18

What would AI look like today?

Jeff Dean 00:09:20

You'd have a lot more lookups into very large memories.

Noam Shazeer 00:09:25

Yeah, it might look more like AI looked like 20 years ago but in the opposite direction. I'm not sure. I guess I joined Google Brain in 2012. I left Google for a few years, happened to go back for lunch to visit my wife, and we happened to sit down next to Jeff and the early Google Brain team. I thought, "Wow, that's a smart group of people."

Jeff Dean 00:09:55

I think I said, "You should think about deep neural nets. We're making some pretty good progress there."

Noam Shazeer 00:09:59

"That sounds fun." Okay, so I jumped back in…

Jeff Dean 00:10:02

I wooed him back, it was great.

Noam Shazeer 00:10:05

..to join Jeff, that was like 2012. I seem to join Google every 12 years: I rejoined Google in 2000, 2012, and 2024.

Dwarkesh Patel 00:10:14

What's going to happen in 2036?

Noam Shazeer 00:10:16

I don't know. I guess we shall see.

Dwarkesh Patel 00:10:21

What are the trade-offs that you're considering changing for future versions of TPU to integrate how you're thinking about algorithms?

Jeff Dean 00:10:29

I think one general trend is we're getting better at quantizing or having much more reduced precision models. We started with TPUv1, and we weren't even quite sure we could quantize and model for serving with eight-bit integers. But we sort of had some early evidence that seemed like it might be possible. So we're like, "Great, let's build the whole chip around that."

And then over time, I think you've seen people able to use much lower precision for training as well. But also the inference precision has gone. People are now using INT4 or FP4, which sounded like, if you said to someone like we're going to use FP4, like a supercomputing floating point person 20 years ago, they'd be like, "What? That's crazy. We like 64 bits in our floats."

Or even below that, some people are quantizing models to two bits or one bit, and I think that's a trend that definitely –

Dwarkesh Patel 00:11:25

One bit? Just like a zero-or-one?

Jeff Dean 00:11:27

Yeah, just a 0-1. And then you have a sign bit for a group of bits or something.

Noam Shazeer 00:11:33

It really has to be a co-design thing because, if the algorithm designer doesn't realize that you can get greatly improved performance, throughput, with the lower precision, of course, the algorithm designer is going to say, "Of course, I don't want low precision. That introduces risk." And then it adds irritation.

Then if you ask the chip designer, "Okay, what do you want to build?" And then they'll ask the person who's writing the algorithms today, who's going to say, "No, I don't like quantization. It's irritating." So you actually need to basically see the whole picture and figure out, "Oh, wait a minute, we can increase our throughput-to-cost ratio by a lot by quantizing."

Jeff Dean 00:12:27

Then you're like, yes, quantization is irritating, but your model is going to be three times faster, so you're going to have to deal.

Dwarkesh Patel 00:12:33

Through your careers, at various times, you’ve worked on things that have an uncanny resemblance to what we're actually using now for generative AI. In 1990, Jeff, your senior thesis was about backpropagation. And in 2007- this is the thing that I didn’t realise until I was prepping for this episode – in 2007 you guys trained a two trillion token N-gram model for language modeling.

Just walk me through when you were developing that model. Was this kind of thing in your head? What did you think you guys were doing at the time?

Jeff Dean 00:13:13

Let me start with the undergrad thesis. I got introduced to neural nets in one section of one class on parallel computing that I was taking in my senior year. I needed to do a thesis to graduate, an honors thesis. So I approached the professor and I said, "Oh, it'd be really fun to do something around neural nets."

So, he and I decided I would implement a couple of different ways of parallelizing backpropagation training for neural nets in 1990. I called them something funny in my thesis, like "pattern partitioning" or something. But really, I implemented a model parallelism and data parallelism on a 32-processor Hypercube machine.

In one, you split all the examples into different batches, and every CPU has a copy of the model. In the other one, you pipeline a bunch of examples along to processors that have different parts of the model. I compared and contrasted them, and it was interesting.

I was really excited about the abstraction because it felt like neural nets were the right abstraction. They could solve tiny toy problems that no other approach could solve at the time. I thought, naive me, that 32 processors would be able to train really awesome neural nets.

But it turned out we needed about a million times more compute before they really started to work for real problems, but then starting in the late 2008, 2009, 2010 timeframe, we started to have enough compute, thanks to Moore's law, to actually make neural nets work for real things. That was kind of when I re-entered, looking at neural nets.

But prior to that, in 2007...

Dwarkesh Patel 00:14:55

Sorry, actually could I ask about this?

Jeff Dean 00:14:57

Oh yeah, sure.

Dwarkesh Patel 00:14:58

First of all, unlike other artifacts of academia, it's actually like four pages, and you can just read it.

Jeff Dean 00:15:07

It was four pages and then 30 pages of C code.

Dwarkesh Patel 00:15:10

But it's just a well-produced artifact. Tell me about how the 2007 paper came together.

Jeff Dean 00:15:15

Oh yeah, so that, we had a machine translation research team at Google led by Franz Och, who had joined Google maybe a year before, and a bunch of other people. Every year they competed in a DARPA contest on translating a couple of different languages to English, I think, Chinese to English and Arabic to English.

The Google team had submitted an entry, and the way this works is you get 500 sentences on Monday, and you have to submit the answer on Friday. I saw the results of this, and we'd won the contest by a pretty substantial margin measured in Bleu score, which is a measure of translation quality.

So I reached out to Franz, the head of this winning team. I'm like, "This is great, when are we going to launch it?" And he's like, "Oh, well, we can't launch this. It's not really very practical because it takes 12 hours to translate a sentence." I'm like, "Well, that seems like a long time. How could we fix that?"

It turned out they'd not really designed it for high throughput, obviously. It was doing 100,000 disk seeks in a large language model that they sort of computed statistics over – I wouldn't say "trained" really – for each word that it wanted to translate.

Obviously, doing 100,000 disk seeks is not super speedy. But I said, "Okay, well, let's dive into this." So I spent about two or three months with them, designing an in-memory compressed representation of N-gram data.

We were using- an N-gram is basically statistics for how often every N-word sequence occurs in a large corpus, so you basically have, in this case, we had 2 trillion words. Most N-gram models of the day were using two-grams or maybe three-grams, but we decided we would use five-grams.

So, how often every five-word sequence occurs in basically as much of the web as we could process in that day. Then you have a data structure that says, "Okay, 'I really like this restaurant' occurs 17 times in the web, or something.

And so I built a data structure that would let you store all those in memory on 200 machines and then have sort of a batched API where you could say, "Here are the 100,000 things I need to look up in this round for this word," and we'd give you them all back in parallel. That enabled us to go from taking a night to translate a sentence to basically doing something in 100 milliseconds or something.

Dwarkesh Patel 00:18:03

There's this list of Jeff Dean facts, like Chuck Norris facts. For example, that “for Jeff Dean, NP equals "no problemo."” One of them, it's funny because now that I hear you say it, actually, it's kind of true. One of them is, "The speed of light was 35 miles an hour until Jeff Dean decided to optimize it over a weekend." Just going from 12 hours to 100 milliseconds, I got to do the orders of magnitude there.

Jeff Dean 00:18:36

All of these are very flattering. They're pretty funny. They're like an April Fool's joke gone awry by my colleagues.

Dwarkesh Patel 00:18:45

Obviously, in retrospect, this idea that you can develop a latent representation of the entire internet through just considering the relationships between words is like: yeah, this is large language models. This is Gemini. At the time, was it just a translation idea, or did you see that as being the beginning of a different kind of paradigm?

Jeff Dean 00:19:11

I think once we built that for translation, the serving of large language models started to be used for other things, like completion... you start to type, and it suggests what completions make sense.

So it was definitely the start of a lot of uses of language models in Google. And Noam has worked on a number of other things at Google, like spelling correction systems that use language models.

Noam Shazeer 00:19:36

That was like 2000, 2001, and I think it was all in-memory on one machine.

Jeff Dean 00:19:44

Yeah, I think it was one machine. His spelling correction system he built in 2001 was amazing. He sent out this demo link to the whole company.

I just tried every butchered spelling of every few-word query I could get, like “scrumbled uggs Bundict"—

Noam Shazeer 00:19:59

I remember that one, yeah yeah.

Jeff Dean 00:20:00

—instead of “scrambled eggs benedict”, and it just nailed it every time.

Noam Shazeer 00:20:04

Yeah, I guess that was language modeling.

Dwarkesh Patel 00:20:07

But at the time, when you were developing these systems, did you have this sense of, “look, you make these things more and more sophisticated, don't consider five words, consider 100 words, 1,000 words, then the latent representation is intelligence”. Basically when did that insight hit?

Noam Shazeer 00:20:24

Not really. I don't think I ever felt like, okay, N-gram models are going to–

Jeff Dean 00:20:32

–sweep the world–

Noam Shazeer 00:20:33

–yeah: “be” artificial intelligence. I think at the time, a lot of people were excited about Bayesian networks. That seemed exciting.

Definitely seeing those early neural language models, both the magic in that, “okay, this is doing something extremely cool” and also, it just struck me as the best problem in the world in that for one, it is very, very simple to state: give me a probability distribution over the next word. Also, there's roughly infinite training data out there. There's the text of the web; you have trillions of training examples of unsupervised data.

Jeff Dean 00:21:20

Yeah, or self-supervised.

Noam Shazeer 00:21:22

Self-supervised, yeah.

Jeff Dean 00:21:23

It's nice because you then have the right answer, and then you can train on all but the current word and try to predict the current word. It's this amazing ability to just learn from observations of the world.

Noam Shazeer 00:21:36

And then it's AI complete. If you can do a great job of that, then you can pretty much do anything.

Dwarkesh Patel 00:23:00

There's this interesting discussion in the history of science about whether ideas are just in the air and there's a sort of inevitability to big ideas, or whether they're sort of plucked out of some tangential direction. In this case, this way in which we're laying it out very logically, does that imply basically, how inevitable does this...

Noam Shazeer 00:22:05

It does feel like it's in the air. There were definitely some, there was like the neural Turing machine, a bunch of ideas around attention, like having these key-value stores that could be useful in neural networks to focus on things. I think in some sense, it was in the air, and in some sense, you need some group to go do it.

Jeff Dean 00:22:36

I like to think of a lot of ideas as being partially in the air, where there are a few different, maybe separate research ideas that one is squinting at when you’re trying to solve a new problem. You draw on those for some inspiration, and then there's some aspect that is not solved, and you need to figure out how to solve that. The combination of some morphing of the things that already exist and some new things lead to some new breakthrough or new research result that didn't exist before.

Dwarkesh Patel 00:22:57

Are there key moments that stand out to you where you're looking at a research area, you come up with this idea, and you have this feeling of, "Holy shit, I can't believe that worked?"

Jeff Dean 00:23:06

One thing I remember was in the early days of the Brain team. We were focused on “let’s see if we could build some infrastructure that lets us train really, really big neural nets”. At that time, we didn't have GPUs in our data centers; we just had CPUs. But we know how to make lots of CPUs work together.

So we built a system that enabled us to train pretty large neural nets through both model and data parallelism. We had a system for unsupervised learning on 10 million randomly selected YouTube frames. It was a spatially local representation, so it would build up unsupervised representations based on trying to reconstruct the thing from the high-level representations.

We got that working and training on 2,000 computers using 16,000 cores. After a little while, that model was actually able to build a representation at the highest level where one neuron would get excited by images of cats. It had never been told what a cat was, but it had seen enough examples of them in the training data of head-on facial views of cats that that neuron would turn on for that and not for much else.

Similarly, you'd have other ones for human faces and backs of pedestrians, and this kind of thing. That was kind of cool because it's from unsupervised learning principles, building up these really high-level representations. Then we were able to get very good results on the supervised ImageNet 20,000 category challenge that advanced the state of the art by 60% relative improvement, which was quite good at the time.

That neural net was probably 50x bigger than one that had been trained previously, and it got good results. So that sort of said to me, "Hey, actually scaling up neural nets seems like, I thought it would be a good idea and it seems to be, so we should keep pushing on that."

Dwarkesh Patel 00:25:14

These examples illustrate how these AI systems fit into what you were just mentioning: that Google is fundamentally a company that organizes information. AI, in this context, is finding relationships between information, between concepts, to help get ideas to you faster, information you want to you faster.

Now we're moving with current AI models. Obviously, you can use BERT in Google Search and you can ask these questions. They are still good at information retrieval, but more fundamentally, they can write your entire code base for you and do actual work, which goes beyond just information retrieval.

So how are you thinking about that? Is Google still an information retrieval company if you're building an AGI? An AGI can do information retrieval, but it can do many other things as well.

Jeff Dean 00:26:14

I think we're an "organize the world's information" company, and that's broader than information retrieval. Maybe: “organizing and creating new information from some guidance you give it”.

"Can you help me write a letter to my veterinarian about my dog? It's got these symptoms," and it'll draft that. Or, "Can you feed in this video, and can you produce a summary of what's happening in the video every few minutes?"

I think our multimodal capabilities are showing that it's more than just text. It's about understanding the world in all the different modalities that information exists in, both human ones but also non-human-oriented ones, like weird lidar sensors on autonomous vehicles, or genomic information, or health information.

And then, how do you extract and transform that into useful insights for people and make use of that in helping them do all kinds of things they want to do? Sometimes it's, "I want to be entertained by chatting with a chatbot." Sometimes it's, "I want answers to this really complicated question, there is no single source to retrieve from." You need to pull information from 100 web pages, figure out what's going on, and make an organized, synthesized version of that data.

Then dealing with multimodal things or coding-related problems. I think it's super exciting what these models are capable of, and they're improving fast, so I'm excited to see where we go.

Noam Shazeer 00:28:42

I am also excited to see where we go. I think definitely organizing information is clearly a trillion-dollar opportunity, but a trillion dollars is not cool anymore. What's cool is a quadrillion dollars.

Obviously the idea is not to just pile up some giant pile of money, but it's to create value in the world, and so much more value can be created when these systems can actually go and do something for you, write your code, or figure out problems that you wouldn't have been able to figure out yourself.

To do that at scale, we're going to have to be very, very flexible and dynamic as we improve the capabilities of these models.

Jeff Dean 00:29:22

Yeah, I'm pretty excited about a lot of fundamental research questions that come about because you see something could be substantially improved if we tried this approach or things in this rough direction. Maybe that'll work, maybe it won't.

But I also think there's value in seeing what we could achieve for end-users and then how can we work backwards from that to actually build systems that are able to do that. As one example: organizing information, that should mean any information in the world should be usable by anyone, regardless of what language they speak.

And that I think we've done some amount of, but it's not nearly the full vision of, "No matter what language you speak, out of thousands of languages, we can make any piece of content available to you and make it usable by you. Any video could be watched in any language." I think that would be pretty awesome. We're not quite there yet, but that's definitely things I see on the horizon that should be possible.

Dwarkesh Patel 00:30:26

Speaking of different architectures you might try, I know one thing you're working on right now is longer context. If you think of Google Search, it's got the entire index of the internet in its context, but it's a very shallow search. And then obviously language models have limited context right now, but they can really think. It's like dark magic, in-context learning. It can really think about what it’s seeing.

How do you think about what it would be like to merge something like Google Search and something like in-context learning?

Jeff Dean 00:30:51

Yeah, I'll take a first stab at it because – I've thought about this for a bit. One of the things you see with these models is they're quite good, but they do hallucinate and have factuality issues sometimes. Part of that is you've trained on, say, tens of trillions of tokens, and you've stirred all that together in your tens or hundreds of billions of parameters.

But it's all a bit squishy because you've churned all these tokens together. The model has a reasonably clear view of that data, but it sometimes gets confused and will give the wrong date for something.

Whereas information in the context window, in the input of the model, is really sharp and clear because we have this really nice attention mechanism in transformers. The model can pay attention to things, and it knows the exact text or the exact frames of the video or audio or whatever that it's processing.

Right now, we have models that can deal with millions of tokens of context, which is quite a lot. It's hundreds of pages of PDF, or 50 research papers, or hours of video, or tens of hours of audio, or some combination of those things, which is pretty cool. But it would be really nice if the model could attend to trillions of tokens.

Could it attend to the entire internet and find the right stuff for you? Could it attend to all your personal information for you? I would love a model that has access to all my emails, all my documents, and all my photos.

When I ask it to do something, it can sort of make use of that, with my permission, to help solve what it is I'm wanting it to do. But that's going to be a big computational challenge because the naive attention algorithm is quadratic. You can barely make it work on a fair bit of hardware for millions of tokens, but there's no hope of making that just naively go to trillions of tokens.

So, we need a whole bunch of interesting algorithmic approximations to what you would really want: a way for the model to attend conceptually to lots and lots more tokens, trillions of tokens. Maybe we can put all of the Google code base in context for every Google developer, all the world's source code in context for any open-source developer. That would be amazing.

Noam Shazeer 00:33:20

It would be incredible. The beautiful thing about model parameters is they are quite memory-efficient at memorizing facts. You can probably memorize on the order of one fact or something per model parameter.

Whereas if you have some token in context, there are lots of keys and values at every layer. It could be a kilobyte, a megabyte of memory per token.

Jeff Dean 00:33:56

You take a word and you blow it up to 10 kilobytes or something.

Noam Shazeer 00:33:59

Yes. There's actually a lot of innovation going on around, okay, A, how do you minimize that? And B, what words do you need to have there? Are there better ways of accessing bits of that information?

Jeff seems like the right person to figure this out. Okay, what does our memory hierarchy look like from the SRAM all the way up to data center worldwide level?

Dwarkesh Patel 00:34:32

I want to talk more about the thing you mentioned: look, Google is a company with lots of code and lots of examples. If you just think about that one use case and what that implies, so you've got the Google monorepo. Maybe you figure out the long context thing, you can put the whole thing in context, or you fine-tune on it. Why hasn't this been already done?

You can imagine the amount of code that Google has proprietary access to, even if you're just using it internally to make your developers more efficient and productive.

Jeff Dean 00:35:09

To be clear, we have actually already done further training on a Gemini model on our internal code base for our internal developers. But that's different than attending to all of it because it sort of stirs together the code base into a bunch of parameters. Having it in context makes things clearer.

Even the further trained model internally is incredibly useful. Sundar, I think, has said that 25% of the characters that we're checking into our code base these days are generated by our AI-based coding models with kind of human oversight.

Dwarkesh Patel 00:35:49

How do you imagine, in the next year or two, based on the capabilities you see around the horizon, your own personal work? What will it be like to be a researcher at Google? You have a new idea or something. With the way in which you're interacting with these models in a year, what does that look like?

Noam Shazeer 00:36:04

Well, I assume we will have these models a lot better and hopefully be able to be much, much more productive.

Jeff Dean 00:36:15

Yeah, in addition to kind of research-y context, anytime you're seeing these models used, I think they're able to make software developers more productive because they can kind of take a high-level spec or sentence description of what you want done and give a pretty reasonable first cut at that. From a research perspective, maybe you can say, "I'd really like you to explore this kind of idea similar to the one in this paper, but maybe let's try making it convolutional or something."

If you could do that and have the system automatically generate a bunch of experimental code, and maybe you look at it and you're like, "Yeah, that looks good, run that." That seems like a nice dream direction to go in.

It seems plausible in the next year or two years that you might make a lot of progress on that.

Dwarkesh Patel 00:38:08

It seems under-hyped because you could have literally millions of extra employees, and you can immediately check their output, the employees can check each other's output, hey immediately stream tokens.

Jeff Dean 00:38:21

Sorry, I didn't mean to underhype it. I think it's super exciting. I just don't like to hype things that aren't done yet.

Dwarkesh Patel 00:38:34

I do want to play with this idea more because it seems like a big deal if you have something kind of like an autonomous software engineer, especially from the perspective of a researcher who's like, "I want to build the system." Okay, so let's just play with this idea. As somebody who has worked on developing transformative systems through your careers, the idea that instead of having to code something like whatever today's equivalent of MapReduce is or Tensorflow is, just like, "Here's how I want a distributed AI library to look. Write it up for me."

Do you imagine you could be 10x more productive? 100x more productive?

Jeff Dean 00:39:13

I was pretty impressed. I think it was on Reddit that I saw we have a new experimental coding model that's much better at coding and math and so on. Someone external tried it, and they basically prompted it and said, "I'd like you to implement a SQL processing database system with no external dependencies, and please do that in C."

From what the person said, it actually did a quite good job. It generated a SQL parser and a tokenizer and a query planning system and some storage format for the data on disk and actually was able to handle simple queries. From that prompt, which is like a paragraph of text or something, to get even an initial cut at that seems like a big boost in productivity for software developers.

I think you might end up with other kinds of systems that maybe don't try to do that in a single semi-interactive, "respond in 40 seconds" kind of thing but might go off for 10 minutes and might interrupt you after five minutes saying, "I've done a lot of this, but now I need to get some input. Do you care about handling video or just images or something?" That seems like you'll need ways of managing the workflow if you have a lot of these background activities happening.

Dwarkesh Patel 00:40:44

Can you talk more about that? What interface do you imagine we might need if you could literally have millions of employees you could spin up, hundreds of thousands of employees you could spin up on command, who are able to type incredibly fast, and who- It's almost like you go from 1930s trading of tickets or something to now modern Jane Street or something. You need some interface to keep track of all this that's going on, for the AIs to integrate into this big monorepo and leverage their own strengths, for humans to keep track of what's happening. Basically what is it like to be Jeff or Noam in three years working day-to-day?

Noam Shazeer 00:42:26

It might be kind of similar to what we have now because we already have sort of parallelization as a major issue. We have lots and lots of really, really brilliant machine learning researchers, and we want them to all work together and build AI.

So actually, the parallelization among people might be similar to parallelization among machines. I think definitely it should be good for things that require a lot of exploration, like, "Come up with the next breakthrough."

If you have a brilliant idea that is just certain to work in the ML domain, then it has a 2% chance of working if you're brilliant. Mostly these things fail, but if you try 100 things or 1,000 things or a million things, then you might hit on something amazing. We have plenty of compute. Like modern top labs these days have probably a million times as much compute as it took to train Transformer.

Dwarkesh Patel 00:43:41

Yeah, actually, so that's a really interesting idea. Suppose in the world today there are on the order of 10,000 AI researchers in this community coming up with a breakthrough-

Jeff Dean 00:43:52

Probably more than that. There were 15,000 at NeurIPS last week.

Noam Shazeer 00:43:55

Wow.

Dwarkesh Patel 00:43:57

100,000, I don't know.

Jeff Dean 00:43:58

Yeah, maybe. Sorry.

Dwarkesh Patel 00:44:00

No, no, it's good to have the correct order of magnitude. The odds that this community every year comes up with a breakthrough on the scale of a Transformer is, let's say, 10%. Now suppose this community is a thousand times bigger, and it is, in some sense, like this sort of parallel search of better architectures, better techniques.

Do we just get like-

Jeff Dean 00:44:22

A breakthrough a day?

Dwarkesh Patel 00:44:23

-breakthroughs every year or every day?

Noam Shazeer 00:44:25

Maybe. Sounds potentially good.

Dwarkesh Patel 00:44:30

But does that feel like what ML research is like? If you are able to try all these experiments…

Noam Shazeer 00:44:37

It's a good question, because I don't know that folks haven't been doing that as much. We definitely have lots of great ideas coming along. Everyone seems to want to run their experiment at maximum scale, but I think that's a human problem.

Jeff Dean 00:44:55

It's very helpful to have a 1/1000th scale problem and then vet 100,000 ideas on that, and then scale up the ones that seem promising.

Dwarkesh Patel 00:44:06

So, one thing the world might not be taking seriously: people are aware that it's exponentially harder to make a model that's 100x bigger. It's 100x more compute, right? So people are worried that it's an exponentially harder problem to go from Gemini 2 to 3, or so forth.

But maybe people aren't aware of this other trend where Gemini 3 is coming up with all these different architectural ideas, trying them out, and you see what works, and you're constantly coming up with algorithmic progress that makes training the next one easier and easier. How far could you take that feedback loop?

Jeff Dean 00:45:43

I think one thing people should be aware of is that the improvements from generation to generation of these models often are partially driven by hardware and larger scale, but equally and perhaps even more so driven by major algorithmic improvements and major changes in the model architecture, the training data mix, and so on, that really makes the model better per flop that is applied to the model, so I think that's a good realization. Then I think if we have automated exploration of ideas, we'll be able to vet a lot more ideas and bring them into the actual production training for next generations of these models.

That's going to be really helpful because that's sort of what we're currently doing with a lot of brilliant machine learning researchers: looking at lots of ideas, winnowing ones that seem to work well at small scale, seeing if they work well at medium scale, bringing them into larger scale experiments, and then settling on adding a whole bunch of new and interesting things to the final model recipe. If we can do that 100 times faster through those machine learning researchers just gently steering a more automated search process, rather than hand-babysitting lots of experiments themselves, that's going to be really, really good.

Noam Shazeer 00:47:03

The one thing that doesn't speed up is experiments at the largest scale. You still end up doing these N = 1 experiments. Really, you just try to put a bunch of brilliant people in the room, have them stare at the thing, and figure out why this is working, why this is not working.

Jeff Dean 00:47:21

For that, more hardware is a good solution. And better hardware.

Noam Shazeer 00:47:25

Yes, we're counting on you.

Dwarkesh Patel 00:47:28

So, naively, there's this software, there's this algorithmic side improvement that future AI can make. There's also the stuff you're working on. I'll let you describe it.

But if you get into a situation where just from a software level, you can be making better and better chips in a matter of weeks and months, and better AIs can presumably do that better, how does this feedback loop not just end up in, Gemini 3 taking two years, then Gemini 4 is- or the equivalent level jump is now six months, then level five is three months, then one month? You get to superhuman intelligence much more rapidly than you might naively think, because of this software, both on the hardware side and from the algorithmic side improvements.

Jeff Dean 00:48:26

I've been pretty excited lately about how we could dramatically speed up the chip design process. As we were talking earlier, the current way in which you design a chip takes you roughly 18 months to go from "we should build a chip" to something that you then hand over to TSMC and then TSMC takes four months to fab it, and then you get it back and you put it in your data centers.

So that's a pretty lengthy cycle, and the fab time in there is a pretty small portion of it today. But if you could make that the dominant portion, so that instead of taking 12 to 18 months to design the chip with 150 people, you could shrink that to a few people with a much more automated search process, exploring the whole design space of chips and getting feedback from all aspects of the chip design process for the kind of choices that the system is trying to explore at the high level, then I think you could get perhaps much more exploration and more rapid design of something that you actually want to give to a fab.

That would be great because you can shrink fab time, you can shrink the deployment time by designing the hardware in the right way, so that you just get the chips back and you just plug them into some system. And that will then enable a lot more specialization, it will enable a shorter timeframe for the hardware design so that you don't have to look out quite as far into what kind of ML algorithms would be interesting. Instead, it's like you're looking at six to nine months from now, what should it be? Rather than two, two and a half years.

That would be pretty cool. I do think that fabrication time, if that's in your inner loop of improvement, you're going to like...

Dwarkesh Patel 00:50:19

How long is it?

Jeff Dean 00:50:20

The leading edge nodes, unfortunately, are taking longer and longer because they have more metal layers than previous, older nodes. So that tends to make it take anywhere from three to five months.

Dwarkesh Patel 00:50:32

Okay, but that's how long training runs take anyways, right? So you could potentially do both at the same time.

Jeff Dean 00:50:38

Potentially.

Dwarkesh Patel 00:50:39

Okay, so I guess you can't get sooner than three to five months. But the idea that you could get- but also, yeah, you're rapidly developing new algorithmic ideas.

Noam Shazeer 00:50:47

That can move fast.

Jeff Dean 00:50:48

That can move fast, that can run on existing chips and explore lots of cool ideas.

Dwarkesh Patel 00:50:54

So, isn't that a situation in which you're... I think people sort of expect like, ah, there's going to be a sigmoid. Again, this is not a sure thing. But just like, is this a possibility? The idea that you have sort of an explosion of capabilities very rapidly towards the tail end of human intelligence that gets smarter and smarter at a more and more rapid rate?

Noam Shazeer 00:51:17

Quite possibly.

Jeff Dean 00:51:19

Yeah. I like to think of it like this. Right now, we have models that can take a pretty complicated problem and can break it down internally in the model into a bunch of steps, can sort of puzzle together the solutions for those steps, and can often give you a solution to the entire problem that you're asking.

But it isn't super reliable, and it's good at breaking things down into five to ten steps, not 100 to 1,000 steps. So if you could go from, yeah, 80% of the time it can give you a perfect answer to something that's ten steps long to something that 90% of the time can give you a perfect answer to something that's 100 to 1,000 steps of sub-problem long, that would be an amazing improvement in the capability of these models. We're not there yet, but I think that's what we're aspirationally trying to get to.

Noam Shazeer 00:52:14

We don't need new hardware for that, but we'll take it.

Jeff Dean 00:52:20

Never look new hardware in the mouth.

Noam Shazeer 00:52:23

One of the big areas of improvement in the near future is inference time compute, applying more compute at inference time. I guess the way I like to describe it is that even a giant language model, even if you’re doing a trillion operations per token, which is more than most people are doing these days, operations cost something like 10 to the negative $18. And so you're getting a million tokens to the dollar.

I mean compare that to a relatively cheap pastime: you go out and buy a paper book and read it, you're paying 10,000 tokens to the dollar. Talking to a language model is like 100 times cheaper than reading a paperback.

So there is a huge amount of headroom there to say, okay, if we can make this thing more expensive but smarter, because we're 100x cheaper than reading a paperback, we're 10,000 times cheaper than talking to a customer support agent, or a million times or more cheaper than hiring a software engineer or talking to your doctor or lawyer. Can we add computation and make it smarter?

I think a lot of the takeoff that we're going to see in the very near future is of this form. We've been exploiting and improving pre-training a lot in the past, and post-training, and those things will continue to improve. But taking advantage of "think harder" at inference time is just going to be an explosion.

Jeff Dean 00:54:21

Yeah, and an aspect of inference time is I think you want the system to be actively exploring a bunch of different potential solutions. Maybe it does some searches on its own, gets some information back, consumes that information, and figures out, oh, now I would really like to know more about this thing. So now it iteratively explores how to best solve the high-level problem you pose to this system.

And I think having a dial where you can make the model give you better answers with more inference time compute seems like we have a bunch of techniques now that can kind of do that. The more you crank up the dial, the more it costs you in terms of compute, but the better the answers get.

That seems like a nice trade-off to have, because sometimes you want to think really hard because it's a super important problem. Sometimes you probably don't want to spend enormous amounts of compute to compute “what's the answer to one plus one”. Maybe the system –

Dwarkesh Patel 00:55:22

Shouldn’t decide to come up with new axioms of set theory or whatever!

Jeff Dean 00:55:25

– should decide to use a calculator tool instead of a very large language model.

Dwarkesh Patel 00:55:31

Interesting. So are there any impediments to taking inference time, like having some way in which you can just linearly scale up inference time compute? Or is this basically a problem that's sort of solved, and we know how to throw 100x compute, 1000x compute, and get correspondingly better results?

Noam Shazeer 00:55:50

We're working out the algorithms as we speak. So I believe we'll see better and better solutions to this as these many more than 10,000 researchers are hacking at it, many of them at Google.

Jeff Dean 00:56:06

I think we do see some examples in our own experimental work of things where if you apply more inference time compute, the answers are better than if you just apply 10x, you can get better answers than x amount of computed inference time. And that seems useful and important.

But I think what we would like is when you apply 10x to get even a bigger improvement in the quality of the answers than we're getting today. And so that's about designing new algorithms, trying new approaches, figuring out how best to spend that 10x instead of x to improve things.

Dwarkesh Patel 00:56:44

Does it look more like search, or does it look more like just keeping going in the linear direction for a longer time?

Jeff Dean 00:56:49

I really like Rich Sutton's paper that he wrote about the Bitter Lesson and the Bitter Lesson effectively is this nice one-page paper but the essence of it is you can try lots of approaches, but the two techniques that are incredibly effective are learning and search.

You can apply and scale those algorithmically or computationally, and you often will then get better results than any other kind of approach you can apply it to a pretty broad variety of problems.

Search has got to be part of the solution to spending more inference time. Maybe you explore a few different ways of solving this problem, and that one didn't work, but this one worked better. I'm going to explore that a bit more.

Dwarkesh Patel 00:57:36

How does this change your plans for future data center planning and so forth? Where can this kind of search be done asynchronously? Does it have to be online, offline? How does that change how big of a campus you need and those kinds of considerations?

Jeff Dean 00:57:55

One general trend is it's clear that inference time compute, you have a model that's pretty much already trained and you want to do inference on, it is going to be a growing and important class of computation. Maybe you want to specialize hardware more around that.

Actually, the first TPU was specialized for inference and wasn't really designed for training. Then subsequent TPUs were really designed more around training and also for inference.

But it may be that when you have something where you really want to crank up the amount of compute you use at inference time, that even more specialized solutions will make a lot of sense.

Dwarkesh Patel 00:58:38

Does that mean you can accommodate more asynchronous training?

Jeff Dean 00:58:41

Training? Or inference?

Dwarkesh Patel 00:58:42

Or just you can have the different data centers don't need to talk to each other, you can just have them do a bunch of...

Jeff Dean 00:58:52

I like to think of it as, is the inference that you're trying to do latency-sensitive? Like a user is actively waiting for it, or is it a background thing? Maybe I have some inference tasks that I'm trying to run over a whole batch of data, but it's not for a particular user. It's just I want to run inference on it and extract some information.

There's probably a bunch of things that we don't really have very much of right now, but you're seeing inklings of it in our deep research tool that we just released, like a week ago. You can give it a pretty complicated, high-level task like, "Hey, can you go off and research the history of renewable energy and all the trends in costs for wind and solar and other kinds of techniques, and put it in a table and give me a full eight-page report?" And it will come back with an eight-page report with like 50 entries in the bibliography.

It's pretty remarkable. But you're not actively waiting for that for one second. It takes like a minute or two to go do that.

And I think there's going to be a fair bit of that kind of compute, and that's the kind of thing where you have some UI questions around. Okay, if you're going to have a user with 20 of these kind of asynchronous tasks in the background happening, and maybe each one of them needs to get more information from the user, like, "I found your flights to Berlin, but there's no non-stop ones. Are you okay with a non-stop one?" How does that flow work when you kind of need a bit more information, and then you want to put it back in the background for it to continue doing, you know, finding the hotels in Berlin or whatever? I think it's going to be pretty interesting, and inference will be useful.

Noam Shazeer 01:00:33

Inference will be useful. There's also a compute efficiency in inference that you don't have in training. In general, transformers can use the sequence length as a batch during training, but they can't really in inference, because when you're generating one token at a time, so there may be different hardware and inference algorithms that we design for the purposes of being efficient at inference.

Jeff Dean 01:01:02

Yeah, as a good example of an algorithmic improvement is the use of drafter models. So you have a really small language model that you do one token at a time when you're decoding, and it predicts four tokens. Then you give that to the big model and you say, "Okay, here are the four tokens the little model came up with. Check which ones you agree with."

If you agree with the first three, then you just advance. Then you've basically been able to do a four-token width parallel computation instead of a one-token width computation in the big model. Those are the kinds of things that people are looking at to improve inference efficiency, so you don't have this single-token decode bottleneck.

Noam Shazeer 01:01:46

Right, basically the big model's being used as a verifier.

Jeff Dean 01:01:48

Right, “can you verify”, yeah.

Noam Shazeer 01:01:50

[inaudible] generator and verification you can do.

Jeff Dean 01:01:52

Right. "Hello, how are you?" That sounds great to me. I'm going to advance past that.

Dwarkesh Patel 01:01:56

So, a big discussion has been about how we're already tapping out nuclear power plants in terms of delivering power into one single campus. Do we have to have just two gigawatts in one place, five gigawatts in one place, or can it be more distributed and still be able to train a model? Does this new regime of inference scaling make different considerations there plausible? How are you thinking about multi-data center training now?

Jeff Dean 01:02:31

We're already doing it. We're pro multi-data center training. I think in the Gemini 1.5 tech report, we said we used multiple metro areas and trained with some of the compute in each place. And then a pretty long latency but high bandwidth connection between those data centers, and that works fine.

Training is kind of interesting because each step in a training process is usually, for a large model, is usually a few seconds or something, at least. So, the latency of it being 50 milliseconds away doesn't matter that much.

Noam Shazeer 01:03:06

Just the bandwidth.

Jeff Dean 01:03:08

Yeah, just bandwidth.

Noam Shazeer 01:03:10

As long as you can sync all of the parameters of the model across the different data centers and then accumulate all the gradients, in the time it takes to do one step, you're pretty good.

Jeff Dean 01:03:21

And then we have a bunch of work, even from early Brain days, when we were using CPU machines and they were really slow. We needed to do asynchronous training to help scale, where each copy of the model would do some local computation, send gradient updates to a centralized system, and then apply them asynchronously. Another copy of the model would be doing the same thing.

It makes your model parameters wiggle around a bit, and it makes people uncomfortable with the theoretical guarantees, but it actually seems to work in practice.

Noam Shazeer 01:03:56

It was so pleasant to go from asynchronous to synchronous because your experiments are now replicable, rather than your results depend on whether there was a web crawler running on the same machine. So, I am so much happier running on TPU pods.

Jeff Dean 01:04:20

I love asynchrony. It just lets you scale so much more.

Noam Shazeer 01:04:22

With these two iPhones and an Xbox or whatever.

Jeff Dean 01:04:25

Yeah, what if we could give you asynchronous but replicable results?

Noam Shazeer 01:04:29

Ooh.

Jeff Dean 01:04:31

So, one way to do that is you effectively record the sequence of operations, like which gradient update happened and when and on which batch of data. You don't necessarily record the actual gradient update in a log or something, but you could replay that log of operations so that you get repeatability. Then I think you'd be happy.

Noam Shazeer 01:04:53

Possibly. At least you could debug what happened, but you wouldn't be able to necessarily compare two training runs. Because, okay, I made one change in the hyperparameter, but also I had a-

Jeff Dean 01:05:08

Web crawler.

Noam Shazeer 01:05:09

-web crawler messing up, and there were a lot of people streaming the Super Bowl at the same time.

Jeff Dean 01:05:19

The thing that led us to go from asynchronous training on CPUs to fully synchronous training is the fact that we have these super fast TPU hardware chips and pods, which have incredible amounts of bandwidth between the chips in a pod. Then, scaling beyond that, we have really good data center networks and even cross-metro area networks that enable us to scale to many, many pods in multiple metro areas for our largest training runs. We can do that fully synchronously.

As Noam said, as long as the gradient accumulation and communication of the parameters across metro areas happens fast enough relative to the step time, you're golden. You don't really care. But I think as you scale up, there may be a push to have a bit more asynchrony in our systems than we have now because we can make it work, our ML researchers have been really happy how far we've been able to push synchronous training because it is an easier mental model to understand. You just have your algorithm sort of fighting you, rather than the asynchrony and the algorithm kind of battling you.

Noam Shazeer 01:06:28

As you scale up, there are more things fighting you. That's the problem with scaling, that you don't always know what it is that's fighting you. Is it the fact that you've pushed quantization a little too far in some place or another? Or is it your data?

Jeff Dean 01:06:46

Maybe it's your adversarial machine MUQQ17 that is setting the seventh bit of your exponent and all your gradients or something.

Noam Shazeer 01:06:56

Right. And all of these things just make the model slightly worse, so you don't even know that the thing is going on.

Jeff Dean 01:07:04

That's actually a bit of a problem with neural nets, is they're so tolerant of noise. You can have things set up kind of wrong in a lot of ways, and they just figure out ways to work around that or learn.

Noam Shazeer 01:07:15

You could have bugs in your code. Most of the time that does nothing. Some of the time it makes your model worse. Some of the time it makes your model better. Then you discover something new because you never tried this bug at scale before because you didn't have the budget for it.

Dwarkesh Patel 01:07:33

What practically does it look like to debug or decode? You've got these things, some of which are making the model better, some of which are making it worse. When you go into work tomorrow, how do you figure out what the most salient inputs are?

Noam Shazeer 01:07:50

At small scale, you do lots of experiments. There's one part of the research that involves, okay, I want to invent these improvements or breakthroughs in isolation. In which case you want a nice simple code base that you can fork and hack, and some baselines.

My dream is I wake up in the morning, come up with an idea, hack it up in a day, run some experiments, get some initial results in a day. Like okay this looks promising, these things worked, and these things didn't work.

I think that is very achievable because-

Jeff Dean 01:08:34

At small scale.

Noam Shazeer 01:08:35

At small scale, as long as you keep a nice experimental code base.

Jeff Dean 01:08:41

Maybe an experiment takes an hour to run or two hours, not two weeks.

Noam Shazeer 01:08:45

It’s great. So there's that part of the research, and then there's some amount of scaling up. Then you have the part which is integrating, where you want to stack all the improvements on top of each other and see if they work at large scale, and see if they work all in conjunction.

Jeff Dean 01:09:02

Right, how do they interact? Right, you think maybe they're independent, but actually maybe there's some funny interaction between improving the way in which we handle video data input and the way in which we update the model parameters. Maybe that interacts more for video data than some other thing.

There are all kinds of interactions that can happen that you maybe don't anticipate. So you want to run these experiments where you're then putting a bunch of things together and then periodically making sure that all the things you think are good are good together. If not, understanding why they're not playing nicely.

Dwarkesh Patel 01:09:41

Two questions. One, how often does it end up being the case that things don't stack up well together? Is it like a rare thing or does it happen all the time?

Noam Shazeer 01:09:52

It happens 50% of the time.

Jeff Dean 01:09:55

Yeah, I mean, I think most things you don't even try to stack because the initial experiment didn't work that well, or it showed results that aren't that promising relative to the baseline. Then you sort of take those things and you try to scale them up individually.

Then you're like, "Oh yeah, these ones seem really promising." So I'm going to now include them in something that I'm going to now bundle together and try to advance and combine with other things that seem promising. Then you run the experiments and then you're like, "Oh, well, they didn't really work that well. Let's try to debug why."

Noam Shazeer 01:10:28

And then there are trade offs, because you want to keep your integrated system as clean as you can, because complexity –

Jeff Dean 01:10:38

Codebase-wise.

Noam Shazeer 01:10:39

– yeah codebase and algorithmically. Complexity hurts, complexity makes things slower, introduces more risk.

And then at the same time you want it to be as good as possible. And of course, every individual researcher wants his inventions to go into it. So there are definitely challenges there, but we've been working together quite well.

Dwarkesh Patel 01:11:05

Okay, so then going back to the whole dynamic “you find better and better algorithmic improvements and the models get better and better over time”, even if you take the hardware part out of it. Should the world be thinking more about, and should you guys be thinking more about this?

There's one world where AI is a thing that takes two decades to slowly get better over time and you can sort of refine things over. If you've kind of messed something up, you fix it, and it's not that big a deal, right? It's like not that much better than the previous version you released.

There's another world where you have this big feedback loop, which means that the two years between Gemini 4 and Gemini 5 are the most important years in human history. Because you go from a pretty good ML researcher to superhuman intelligence because of this feedback loop. To the extent that you think that the second world is plausible, how does that change how you sort of approach these greater and greater levels of intelligence?

Noam Shazeer 01:12:14

I've stopped cleaning my garage because I'm waiting for the robots. So probably I'm more in the second camp of what we're going to see, a lot of acceleration.

Jeff Dean 01:12:24

Yeah, I mean, I think it's super important to understand what's going on and what the trends are. And I think right now the trends are the models are getting substantially better generation over generation. I don't see that slowing down in the next few generations probably.

So that means the models say two to three generations from now are going to be capable of... Let's go back to the example of breaking down a simple task into 10 sub pieces and doing it 80% of the time, to something that can break down a task, a very high level task, into 100 or 1,000 pieces and get that right 90% of the time. That's a major, major step up in what the models are capable of.

So I think it's important for people to understand what is happening in the progress in the field. And then those models are going to be applied in a bunch of different domains. I think it's really good to make sure that we, as a society, get the maximal benefits from what these models can do to improve things. I'm super excited about areas like education and healthcare, making information accessible to all people.

But we also realize that they could be used for misinformation, they could be used for automated hacking of computer systems, and we want to put as many safeguards and mitigations and understand the capabilities of the models in place as we can. I think Google as a whole has a really good view to how we should approach this. Our Responsible AI principles actually are a pretty nice framework for how to think about trade offs of making better and better AI systems available in different contexts and settings, while also sort of making sure that we're doing the right thing in terms of making sure they're safe and not saying toxic things and things like that.

Dwarkesh Patel 01:14:21

I guess the thing that stands out to me, if you were zooming out and looking at this period of human history, if we're in the world where, look, if you do post-training on Gemini 3 badly, it can do some misinformation – but then you fix the post training. It's a bad mistake, but it's a fixable mistake, right?

Noam Shazeer 01:14:40

Right.

Dwarkesh Patel 01:14:40

Whereas if you have this feedback loop dynamic, which is a possibility, then the mistake of the thing that catapults this intelligence explosion is misaligned, is not trying to write the code you think it's trying to write, and [instead] optimizing for some other objective.

And on the other end of this very rapid process that lasts a couple of years, maybe less, you have things that are approaching Jeff Dean or beyond level, or Noam Shazeer or beyond level. And then you have millions of copies of Jeff Dean level programmers, and- anyways, that seems like a harder to recover mistake.

Noam Shazeer 01:15:29

As these systems do get more powerful, you have to be more and more careful.

Jeff Dean 01:15:37

One thing I would say is, there are extreme views on either end. There's, "Oh my goodness, these systems are going to be so much better than humans at all things, and we're going to be kind of overwhelmed." And then there's, "These systems are going to be amazing, and we don't have to worry about them at all."

I think I'm somewhere in the middle. I've been a co-author on a paper called "Shaping AI," which is, you know, those two extreme views often kind of view our role as kind of laissez-faire, like we're just going to have the AI develop in the path that it takes.

And I think there's actually a really good argument to be made that what we're going to do is try to shape and steer the way in which AI is deployed in the world so that it is, you know, maximally beneficial in the areas that we want to capture and benefit from, in education, some of the areas I mentioned, healthcare.

And steer it as much as we can away- maybe with policy-related things, maybe with technical measures and safeguards- away from, you know, the computer will take over and have unlimited control of what it can do. So I think that's an engineering problem: how do you engineer safe systems?

I think it's kind of the modern equivalent of what we've done in older-style software development. Like if you look at, you know, airplane software development, that has a pretty good record of how do you rigorously develop safe and secure systems for doing a pretty risky task?

Dwarkesh Patel 01:17:18

The difficulty there is that there's not some feedback loop where the 737, you put it in a box with a bunch of compute for a couple of years, and it comes out with the version 1000.

Noam Shazeer 01:17:27

I think the good news is that analyzing text seems to be easier than generating text. So I believe that the ability of language models to actually analyze language model output and figure out what is problematic or dangerous will actually be the solution to a lot of these control issues.

We are definitely working on this stuff. We've got a bunch of brilliant folks at Google working on this now. And I think it's just going to be more and more important, both from a “do something good for people” standpoint, but also from a business standpoint, that you are, a lot of the time, limited in what you can deploy based on keeping things safe.

And so it becomes very, very important to be really, really good at that.

Dwarkesh Patel 01:18:48

Yeah, obviously, I know you guys take the potential benefits and costs here seriously, and it's truly remarkable. I know you guys get credit for it, but not enough. I think there's just, there are so many different applications that you have put out for using these models to make the different areas you talked about better.

Um, but I do think that… again, if you have a situation where plausibly there's some feedback loop process, on the other end, you have a model that is as good as Noam Shazeer, as good as Jeff Dean.

If there's an evil version of you running around, and suppose there's a million of them, I think that's really, really bad. That could be much, much worse than any other risk, maybe short of nuclear war or something. Just think about it, like a million evil Jeff Deans or something.

Jeff Dean 01:19:47

Where do we get the training data?

Dwarkesh Patel 01:20:20

But, to the extent that you think that's a plausible output of some quick feedback loop process, what is your plan of okay, we've got Gemini 3 or Gemini 4, and we think it's helping us do a better job of training future versions, it's writing a bunch of the training code for us. From this point forward, we just kind of look over it, verify it.

Even the verifiers you talked about of looking at the output of these models will eventually be trained by, or a lot of the code will be written by the AIs you make. What do you want to know for sure before we have the Gemini 4 help us with the AI research? We really want to make sure, we want to run this test on it before we let it write our AI code for us.

Jeff Dean 01:21:34

I mean, I think having the system explore algorithmic research ideas seems like something where there's still a human in charge of that. Like, it's exploring the space, and then it's going to, like, get a bunch of results, and we're going to make a decision, like, are we going to incorporate this particular, you know, learning algorithm or change to the system into kind of the core code base?

And so I think you can put in safeguards like that that enable us to get the benefits of the system that can sort of improve or kind of self-improve with human oversight, uh, without necessarily letting the system go full-on self-improving without any any notion of a person looking at what it's doing, right? That's the kind of engineering safeguards I'm talking about, where you want to be kind of looking at the characteristics of the systems you're deploying, not deploy ones that are harmful by some measures and some ways, and you have an understanding of what its capabilities are and what it's likely to do in certain scenarios. So, you know, I think it's not an easy problem by any means, but I do think it is possible to make these these systems safe.

Noam Shazeer 01:36:56

Yeah. I mean, I think we are also going to use these systems a lot to check themselves, check other systems. Even as a human, it is easier to recognize something than to generate it.

Jeff Dean 01:37:14

One thing I would say is if you expose the model's capabilities through an API or through a user interface that people interact with, I think then you have a level of control to understand how is it being used and put some boundaries on what it can do. And that I think is one of the tools in the arsenal of how do you make sure that what it's going to do is sort of acceptable by some set of standards you've set out in your mind?

Noam Shazeer 01:37:44

Yeah. I mean, I think the goal is to to empower people, but for the most part we should be mostly letting people do things with these systems that make sense and closing off as few parts of the space as we can. But yeah, if you let somebody take your thing and create a million evil software engineers, then that doesn't empower people because they're going to hurt others with a million evil software engineers.

So I'm against that.

Jeff Dean 01:38:14

Me too. I'll go on.

Dwarkesh Patel 01:38:16

All right, let's talk about a few more fun topics. Make it a little lighter. Over the last 25 years, what was the most fun time? What period of time do you have the most nostalgia over?

Jeff Dean 01:38:30

At work, you mean?

Noam Shazeer 01:38:31

Yeah. At work. Yeah.

Jeff Dean 01:38:32

I think the early sort of four or five years at Google when I was one of a handful of people working on search and crawling and indexing systems, our traffic was growing tremendously fast. We were trying to expand our index size and make it so we updated it every minute instead of every month, or two months if something went wrong.

Seeing the growth in usage of our systems was really just personally satisfying. Building something that is used by two billion people a day is pretty incredible.

But I would also say equally exciting is working with people on the Gemini team today. I think the progress we've been making in what these models can do over the last year and a half is really fun. People are really dedicated, really excited about what we're doing.

I think the models are getting better and better at pretty complex tasks. Like if you showed someone using a computer 20 years ago what these models are capable of, they wouldn't believe it. And even five years ago, they might not believe it. And that's pretty satisfying.

I think we'll see a similar growth in usage of these models and impact in the world.

Noam Shazeer 01:39:48

Yeah, I'm with you. Early days were super fun. Part of that is just knowing everybody and the social aspect, and the fact that you're just building something that millions and millions of people are using.

Same thing today. We got that whole nice micro kitchen area where you get lots of people hanging out. I love being in person, working with a bunch of great people, and building something that's helping millions to billions of people. What could be better?

Dwarkesh Patel 01:40:21

What was this micro kitchen?

Jeff Dean 01:40:23

Oh, we have a micro kitchen area in the building we both sit in. It's the new, so-named Gradient Canopy. It used to be named Charleston East, and we decided we needed a more exciting name because it's a lot of machine learning researchers and AI research happening in there.

There's a micro kitchen area that we've set up with, normally it's just like an espresso machine and a bunch of snacks, but this particular one has a bunch of space in it. So we've set up maybe 50 desks in there, and so people are just hanging out in there. It's a little noisy because people are always grinding beans and brewing espresso, but you also get a lot of face-to-face ideas of connections, like, "Oh, I've tried that. Did you think about trying this in your idea?" Or, "Oh, we're going to launch this thing next week. How's the load test looking?" There's just lots of feedback that happens.

And then we have our Gemini chat room for people who are not in that micro kitchen. We have a team all over the world, and there's probably 120 chat rooms I'm in related to Gemini things. In this particular very focused topic, we have seven people working on this, and there are exciting results being shared by the London colleagues.

When you wake up, you see what's happening in there, or it's a big group of people focused on data, and there are all kinds of issues happening in there. It's just fun.

Dwarkesh Patel 01:41:50

What I find remarkable about some of the calls you guys have made is you're anticipating a level of demand for compute, which at the time wasn't obvious or evident. TPUs being a famous example of this, or the first TPU being an example of this.

That thinking you had in, I guess, 2013 or earlier, if you think about it that way today and you do an estimate of, look, we're going to have these models that are going to be a backbone of our services, and we're going to be doing constant inference for them. We're going to be training future versions. And you think about the amount of compute we'll need by 2030 to accommodate all these use cases, where does the Fermi estimate get you?

Jeff Dean 01:42:30

Yeah, I mean, I think you're going to want a lot of inference. Compute is the rough, highest-level view of these capable models because if one of the techniques for improving their quality is scaling up the amount of inference compute you use, then all of a sudden what's currently like one request to generate some tokens now becomes 50 or 100 or 1000 times as computationally intensive, even though it's producing the same amount of output.

And you're also going to then see tremendous scaling up of the uses of these services as not everyone in the world has discovered these chat-based conversational interfaces where you can get them to do all kinds of amazing things. Probably 10% of the computer users in the world have discovered that today, or 20%. As that pushes towards 100% and people make heavier use of it, that's going to be another order of magnitude or two of scaling.

And so you're now going to have two orders of magnitude from that, two orders of magnitude from that. The models are probably going to be bigger, you'll get another order of magnitude or two from that. And there's a lot of inference compute you want. So you want extremely efficient hardware for inference for models you care about.

Dwarkesh Patel 01:43:52

In flops, global total global inference in 2030?

Noam Shazeer 01:43:58

I think just more is always going to be better. If you just kind of think about, okay, what fraction of world GDP will people decide to spend on AI at that point? And then, like, okay, what do the AI systems look like?

Well, maybe it's some sort of personal assistant-like thing that is in your glasses and can see everything around you and has access to all your digital information and the world's digital information. And maybe it's like you're Joe Biden, and you have the earpiece in the cabinet that can advise you about anything in real-time and solve problems for you and give you helpful pointers. Or you could talk to it, and it wants to analyze anything that it sees around you for any potential useful impact that it has on you.

So I mean, I can imagine, okay, and then say it's like your personal assistant or your personal cabinet or something, and that every time you spend 2x as much money on compute, the thing gets like 5, 10 IQ points smarter or something like that. And, okay, would you rather spend $10 a day and have an assistant or $20 a day and have a smarter assistant? And not only is it an assistant in life but an assistant in getting your job done better because now it makes you from a 10x engineer to a 100x or 10 millionx engineer?

Okay, so let's see: from first principles, right? So people are going to want to spend some fraction of world GDP on this thing. The world GDP is almost certainly going to go way, way up, two orders of magnitude higher than it is today, due to the fact that we have all of these artificial engineers working on improving things.

Probably we'll have solved unlimited energy and carbon issues by that point. So we should be able to have lots of energy. We should be able to have millions to billions of robots building us data centers. Let's see, the sun is what, 10 to the 26 watts or something like that?

I'm guessing that the amount of compute being used for AI to help each person will be astronomical.

Jeff Dean 01:46:48

I would add on to that. I'm not sure I agree completely, but it's a pretty interesting thought experiment to go in that direction. And even if you get partway there, it's definitely going to be a lot of compute.

And this is why it's super important to have as cheap a hardware platform for using these models and applying them to problems that Noam described, so that you can then make it accessible to everyone in some form and have as low a cost for access to these capabilities as you possibly can.

And I think that's achievable by focusing on hardware and model co-design kinds of things, we should be able to make these things much, much more efficient than they are today.

Dwarkesh Patel 01:47:36

Is Google's data center build-out plan over the next few years aggressive enough given this increase in demand you're expecting?

Jeff Dean 01:47:46

I'm not going to comment on our future capital spending because our CEO and CFO would prefer I probably not. But I will say, you can look at our past capital expenditures over the last few years and see that we're definitely investing in this area because we think it's important.

We are continuing to build new and interesting, innovative hardware that we think really helps us have an edge in deploying these systems to more and more people, both training them and also, how do we make them usable by people for inference?

Dwarkesh Patel 01:48:21

One thing I've heard you talk a lot about is continual learning, the idea that you could just have a model which improves over time rather than having to start from scratch. Is there any fundamental impediment to that? Because theoretically, you should just be able to keep fine-tuning a model. What does that future look like to you?

Jeff Dean 01:48:40

Yeah, I've been thinking about this more and more. I've been a big fan of models that are sparse because I think you want different parts of the model to be good at different things. We have our Gemini 1.5 Pro model, and other models are mixture-of-experts style models where you now have parts of the model that are activated for some token and parts that are not activated at all because you've decided this is a math-oriented thing, and this part's good at math, and this part's good at understanding cat images. So, that gives you this ability to have a much more capable model that's still quite efficient at inference time because it has very large capacity, but you activate a small part of it.

But I think the current problem, well, one limitation of what we're doing today is it's still a very regular structure where each of the experts is the same size. The paths merge back together very fast. They don't go off and have lots of different branches for mathy things that don't merge back together with the kind of cat-image thing.

I think we should probably have a more organic structure in these things. I also would like it if the pieces of those model of the model could be developed a little bit independently. Like right now, I think we have this issue where we're going to train a model. So, we do a bunch of preparation work on deciding the most awesome algorithms we can come up with and the most awesome data mix we can come up with.

But there's always trade-offs there, like we'd love to include more multilingual data, but that might come at the expense of including less coding data, and so, the model's less good at coding but better at multilingual, or vice versa. I think it would be really great if we could have a small set of people who care about a particular subset of languages go off and create really good training data, train a modular piece of a model that we can then hook up to a larger model that improves its capability in, say, Southeast Asian languages or in reasoning about Haskell code or something.

Then, you also have a nice software engineering benefit where you've decomposed the problem a bit compared to what we do today, which is we have this kind of a whole bunch of people working. But then, we have this kind of monolithic process of starting to do pre-training on this model.

If we could do that, you could have 100 teams around Google. You could have people all around the world working to improve languages they care about or particular problems they care about and all collectively work on improving the model. And that's kind of a form of continual learning.

Noam Shazeer 01:51:27

That would be so nice. You could just glue models together or rip out pieces of models and shove them into other...

Jeff Dean 01:51:33

Upgrade this piece without throwing out the thing...

Noam Shazeer 01:51:36

...or you just attach a fire hose, and you suck all the information out of this model, shove it into another model. There is, I mean, the countervailing interest there is sort of science, in terms of, okay, we're still in the period of rapid progress, so, if you want to do sort of controlled experiments, and okay, I want to compare this thing to that thing because that then is helping us figure out what to build. In that interest, it's often best to just start from scratch so you can compare one complete training run to another complete training run at the practical level because it helps us figure out what to build in the future. It's less exciting but does lead to rapid progress.

Jeff Dean 01:52:32

Yeah, I think there may be ways to get a lot of the benefits of that with a version system of modularity. I have a frozen version of my model, and then I include a different variant of some particular module, and I want to compare its performance or train it a bit more. Then, I compare it to the baseline of this thing with now version N prime of this particular module that does Haskell interpretation.

Noam Shazeer 01:52:58

Actually, that could lead to faster research progress, right? You've got some system, and you do something to improve it. And if that thing you're doing to improve it is relatively cheap compared to training the system from scratch, then it could actually make research much, much cheaper and faster.

Jeff Dean 01:53:16

Yeah, and also more parallelizable, I think, across people.

Noam Shazeer 01:53:24

Okay, let's figure it out and do that next.

Dwarkesh Patel 01:53:29

So, this idea that is sort of casually laid out there would actually be a big regime shift compared to how things are done today. If you think the way things are headed, this is a sort of very interesting prediction about... You just have this blob where things are getting pipelined back and forth – and if you want to make something better, you can do like a sort of surgical incision almost.

Jeff Dean 01:53:55

Right, or grow the model, add another little bit of it here. Yeah, I've been sort of sketching out this vision for a while in Pathways...

Noam Shazeer 01:54:04

Yeah, you've been building the...

Jeff Dean 01:54:05

...and we've been building the infrastructure for it. So, a lot of what Pathways, the system, can support is this kind of twisty, weird model with asynchronous updates to different pieces. And we're using Pathways to train our Gemini models, but we're not making use of some of its capabilities yet. But maybe we should.

Noam Shazeer 01:54:24

Ooh maybe.

Dwarkesh Patel 01:54:27

This is so interesting, and I don't want to lose this thread, but give me one moment.

Noam Shazeer 01:54:33

There have been times, like the way the TPU pods were set up. I don't know who did that, but they did a pretty brilliant job. The low-level software stack and the hardware stack, okay, you've got your nice regular high-performance hardware, you've got these great torus-shaped interconnects, and then you've got the right low-level collectives, the all-reduces, et cetera, which I guess came from supercomputing, but it turned out to be kind of just the right thing to build distributed deep learning on top of.

Dwarkesh Patel 01:39:15

Okay, so a couple of questions. One, suppose Noam makes another breakthrough, and now we've got a better architecture. Would you just take each compartment and distill it into this better architecture? And that's how it keeps improving over time?

Jeff Dean 01:39:33

I do think distillation is a really useful tool because it enables you to transform a model in its current model architecture form into a different form. Often, you use it to take a really capable but large and unwieldy model and distill it into a smaller one that maybe you want to serve with really good, fast latency inference characteristics.

But I think you can also view this as something that's happening at the module level. Maybe there'd be a continual process where you have each module, and it has a few different representations of itself. It has a really big one. It's got a much smaller one that is continually distilling into the small version.

And then the small version, once that's finished, you sort of delete the big one and you add a bunch more parameter capacity. Now, start to learn all the things that the distilled small one doesn't know by training it on more data, and then you kind of repeat that process. If you have that kind of running a thousand different places in your modular model in the background, that seems like it would work reasonably well.

Dwarkesh Patel 01:40:42

This could be a way of doing inference scaling, like the router decides how much do you want the big one.

Jeff Dean 01:40:47

Yeah, you can have multiple versions. Oh, this is an easy math problem, so I'm going to route it to the really tiny math distilled thing. Oh, this one's really hard, so...

Dwarkesh Patel 01:40:56

One, at least from public research, it seems like it's often hard to decode what each expert is doing in mixture of expert type models. If you have something like this, how would you enforce the kind of modularity that would be visible and understandable to us?

Noam Shazeer 01:41:13

Actually, in the past, I found experts to be relatively easy to understand. I mean, the first Mixture of Experts paper, you could just look at the experts.

Dwarkesh Patel 01:41:24

“I don’t know, I'm only the inventor of Mixture of Experts.”

Noam Shazeer 01:57:25

Yeah – oh, the what?

Jeff Dean 01:41:28

Yeah, yeah.

Noam Shazeer 01:41:30

Yeah, you could just see, okay, this expert, like we did, you know, a thousand, two thousand experts. Okay, and this expert, was getting words referring to cylindrical objects.

Jeff Dean 01:41:42

This one's super good at dates.

Noam Shazeer 01:41:44

Yeah.

Jeff Dean 01:41:45

Talking about times.

Noam Shazeer 01:41:46

Yeah, pretty easy to do.

Not that you would need that human understanding to figure out how to work the thing at runtime because you just have some sort of learned router that's looking at the example.

Jeff Dean 01:42:04