Dwarkesh Podcast

Had a lot of fun chatting again with my twin brother Dylan Patel.

We went through lab economics over the next few years - the shift from inference to training as RSI draws near; and how Anthropic and OpenAI are on track to control most of the world’s usable FLOPs within the next few years (because they can monetize compute better and thus outbid everyone).

And then we discuss whether the >$10T of total AI capex we’ll see by the end of the decade will cause a sovereign debt crisis, where hyperscaler debt raises interest rates, drives non-AI exposed countries into bankruptcy, and crashes non-AI equities.

One question we weren’t able to resolve is whether there’s anything that can counter all the forces barrelling towards centralization in this industry - the economies of scale in training, the scarcity of compute, and eventually continual learning and RSI.

Watch on YouTube; listen on Apple Podcasts or Spotify.

• Grok Bot has been quite helpful with my search for a new editor. I created a recruiter bot and described the type of editor I was looking for. That bot then spun up a handful of subagents that combed through my emails and X DMs, read the end credits of various documentaries I like, and figured out who edits for some of my favorite YouTubers. It took all of those results, and then delivered me a shortlist of candidates that matched my criteria. Try Grok Bot for yourself at x.ai/bot

• Antithesis lets you add time travel to your software testing toolkit. Since the Antithesis platform is fully deterministic, everything that happens inside of it is perfectly reproducible. So if your software crashes, you can rewind to the exact right moment, freeze time, and investigate. Or you can test different hypotheses by perturbing the system: kill a node or disable a feature, see what happens, then reset the trajectory and try something else. Learn more at antithesis.com/dwarkesh

• Jane Street is hiring for two separate ML internships right now, one focused primarily on research and one focused on engineering. In both cases, interns are expected to contribute to real work, not contrived exercises: one common project is adapting a frontier LLM paper to financial markets, which tend to come with a ton of different gnarly challenges. Importantly, you don’t need any finance background to apply. 2027 applications are open now at janestreet.com/dwarkesh

(00:00:00) – Two labs will soon control most of the world’s compute

(00:07:01) – $6 billion in fab capex enables $1t+ of end revenue

(00:13:08) – Compute prices will rise if the labs outbid everyone

(00:18:22) – Which layer will capture most of the surplus?

(00:25:40) – What could slow down progress?

(00:29:43) – Labs are shifting compute from inference to R&D

(00:33:27) – China gets less than 10% of new compute, but its labs need less

(00:48:48) – Will AI cause a sovereign debt crisis?

(01:07:52) – Will the world’s future workforce belong to a few companies?

Dwarkesh Patel

Okay, I’m back with Dylan Patel, founder of SemiAnalysis. Our version of a family Thanksgiving dinner is a regular yearly podcast. But we’re not actually related.

Dylan Patel

Don’t tell the people this.

Dwarkesh Patel

It will destroy the myth. Basically where the world economy is headed is more and more becoming a function of where lab economics are headed, where the compute market is headed, et cetera. I want to understand where the crazy future ends up within a few years. But let’s start with where we are today. Walk me through lab compute and lab revenue right now, and maybe project out a year or two.

Dylan Patel

When we go back to last year, even at the end of the year, most of GDP growth in America was just AI infrastructure. As we look towards this year, about a third of the compute coming online is for the labs, for OpenAI and Anthropic. It may be built by others and then rented to them, but at the end customer, it’s them.

As we go forward into the future, the numbers for compute are ballooning. We’re at a little bit over a trillion dollars of CapEx this year. As we go out into ’28, it’s going to be more than $2 trillion. The labs are also taking an increasing percentage of this. So ultimately, you’ve got a very interesting situation where the labs are going from companies that spend tens of billions of dollars a year to hundreds of billions of dollars a year, to forecasting to spend trillions of dollars a year even towards the end of the decade. This is at least some of the contracts they’ve begun signing with their partners.

This requires a big reshaping of what happens with their economics. Up until now, they have been companies that mostly lost money. Anthropic started turning a profit in Q2. It’s believed at some point in Q3, OpenAI could start turning a profit even, with the bigger rise of Codex and 5.6 and all this. But if we go back a year ago, all the money they had was venture-funded losses. If we go back to even the beginning of this year, it was venture-funded losses. They’ve now turned the corner and are actually starting to profit.

That doesn’t mean they’re not taking in new capital. The new capital is still coming in to accelerate the growth further. But ultimately, more and more of their business is being funded off of their own revenue rather than capital injections into them. Over the last year and a half, their margins have really skyrocketed. The base cost of compute tends to be around $10 or $13 or $15 million per megawatt.

The most interesting aspect about what’s happening now is this: Before, if they served a model — GPT-4 being served on Nvidia Hopper GPUs — it was generating negative gross margin for OpenAI. But now, when OpenAI serves GPT-5.6 or Anthropic serves Opus 5 or Fable 5, their revenue generation has passed well beyond the incremental $10-15 million per megawatt. In the case of Anthropic, the revenue has gone as high as $50 million per megawatt. What that now enables them to do is: “Hey, if I spend 10 bucks on inference capacity, I actually generate 50 bucks of revenue, and then I can turn around and incrementally spend all of that profit on training.”

Dwarkesh Patel

One thing I’m very interested in understanding is how you see the centralization of compute happening at the labs, or the relative ratio of compute that goes to the world versus the labs. If you say right now a third of marginal compute is going to the labs, by when is over half of the incremental compute in the world going to the labs? By what point do the labs have basically a vast majority of the world’s compute?

Dylan Patel

At the beginning of this year, OpenAI started at 2 gigawatts and Anthropic at less than 2. End of this year, they’re both above 5. So they’ve 3-4x’d compute as a whole. When you look at the incremental compute added, that’s about 30% of the compute added this year.

As we step forward to next year, given what’s already been signed and penned and inked, you’ve got something even more dramatic. Anthropic and OpenAI are taking as much as 40% to 50% of compute next year. This centralization doesn’t look like it’s slowing down or stopping. In fact, it looks like it’s only accelerating.

Who’s building that compute for them will change. Next year, a big new entrant is, for example, SpaceX, which is building a ton of compute. They’re actively going to lease quite a bit of it to Anthropic and OpenAI, most likely, because they’re the ones who have the marginal capability to pay the highest price. In addition, OpenAI and Anthropic are also starting to build their own compute — OpenAI with their own chips, Anthropic with TPUs that they’re purchasing from Google and deploying with Fluidstack.

So you ask, “Hey, when does half of the world’s incremental new compute go to just OpenAI and Anthropic?” It’s really by the end of next year when half of the incremental compute is already going to Anthropic and OpenAI.

Dwarkesh Patel

Because compute is growing so fast, incremental compute is going to be basically most of compute. So it’s very soon — you’re saying maybe within a year and a half or two years — that most of the world’s compute is owned by two labs, or at least is serving the demand from two labs.

There’s this trend where maybe world compute in gigawatts doubles every year, but the compute at the frontier labs triples every single year. If you keep the current trend going, it goes from 2 at the beginning of this year to close to 6 at the end of this year. Just multiplying out by 3. It’s 18 by the end of 2027, 54 by the end of 2028. Are you like, “Okay, at that point, they simply can’t continue tripling given the amount of world compute”? How do you see the world compute situation over the next few years?

Dylan Patel

If the incremental compute this year adds 30 gigawatts, next year 50 gigawatts, and the year after that roughly 70, you end up with this really interesting phenomenon. A new watt deployed this year is significantly more efficient than the watts deployed two years ago. A humongous percentage of the world’s compute was deployed this year. Even though it didn’t double the number of watts deployed, I’m deploying GB300s and TPUv7s and Trainium3s, which are way, way, way more efficient. They’re 3-5x more performance per watt than the prior-generation chips.

So ultimately you’ve got a huge ladder here. If Anthropic and OpenAI take on 45% of compute next year, you’ve got them in, let’s say, December ’27 having taken on half of the world’s incremental new compute. But that half of the world’s new incremental compute is actually at a higher performance than everything else before it. So you’ve got another multiplier on that. By the time you’re towards the end of 2028 — if this trend continues, and I see nothing that’s stopping it — you’ve got them just controlling most of the usable flops in the world on their own.

Dwarkesh Patel

The thing I’m confused about is why you think we only add 80 gigawatts in 2028 if we enter a world in which the value of compute increases so much.

Dylan Patel

That’s the upper bound, by the way. That’s the like, “I’m so fucking bullish.”

Dwarkesh Patel

Okay, let’s do some chain of thought here. When I interviewed you a few months ago, you said that in order to make a gigawatt of, I think, Vera Rubins, you need 55,000 N3 wafers, 6K N5 wafers, and 170K DRAM wafers. I know if those numbers might have changed.

Dylan Patel

I’m going to troll you, but the way you said wafers was so fucking Indian. Vafers.

Dwarkesh Patel

By the way, when we first moved to the US, I had the v/w thing pretty bad, and I was a vegetarian.

Dylan Patel

I remember you told me about this.

Dwarkesh Patel

In North Dakota, I was in elementary school, and I’d be like—

Dylan Patel

Can I get a “wedgie”?

Dwarkesh Patel

Can I get some “wedgies”?

Anyways, so that’s for one gigawatt. I had an LLM run your wafer fab equipment model and figure out how much the tooling costs to produce a gigawatt of compute basically every single year. It said $3-4 billion. Now suppose you add in cleanrooms and shell and everything else at the fab. So $6 billion of fab CapEx produces a gigawatt every single year. A gigawatt produces right now $100 billion of revenue.

But also that $6 billion in CapEx is producing a gigawatt every single year, and that gigawatt is producing $100 billion every single year. So over the course of five years, the first gigawatt has generated five years of profits, the second gigawatt the fab has produced has generated four years of profits, and so on. $6 billion of CapEx at the fab level will have generated over a trillion dollars of end AI revenue.

Dylan Patel

Yeah. There’s a lot of OpEx along the way. There’s a lot of other CapEx, like the data center, the power.

Dwarkesh Patel

And you had to pay OpenAI for the R&D.

Dylan Patel

Installation. There’s a lot of different people who need money here.

Dwarkesh Patel

Take away half of it for all these middlemen. That still means there’s a 100x discrepancy between fab CapEx and end revenue generated. More than that, actually, but we’re just being very conservative. As a result… This is capitalism. You have this huge discrepancy where you can turn $1 into $100. They’re not going to figure out a way to make more mirrors?

Dylan Patel

They are. It’s just that these mirrors take some time to make.

Dwarkesh Patel

But the emergency is so big where Anthropic and OpenAI are like, “We could make a trillion dollars right now, but we’re just bottlenecked on the mirrors that go into the ASML machines.” How can we make more mirrors if we spend $100 billion on this? That’s the situation we’re going to be in pretty soon. We’re not going to be able to solve that supply constraint? That just seems quite hard to imagine.

Dylan Patel

You’ve seen people do funny arbitrages here where they buy turbines and then try and resell them, because the value of a turbine is way more since it’s the thing bottlenecking your data center. I think if anyone had $400 million and the ability to convince ASML to sell them an EUV tool, they should totally just go buy one, wait, and then sell it for north of a billion dollars.

But ultimately, yes, capitalism will cause these things to expand. But it’s a whip. It takes a long time for the whip signal to get to the tail end of that. The supply chain doesn’t react immediately. In fact, you go talk to someone at Carl Zeiss, they’re like, “Yeah, yeah, yeah, we need to make 100 EUV tools by the end of the decade.” When we had our episode earlier this year, they didn’t even think they needed to make that many, enough mirrors to make 100 EUV tools a year. Now they’re like, “Okay, we need to do that.” But in reality, because of all the economics of what’s going on, it should be even more. It takes so long to pill.

Dwarkesh Patel

Suppose that every single company in the stack got private equitied. Somebody came in who was super AGI-pilled and was like, “We’re going to maximize production.” What do you think the physical constraints on making more things would be? The reason I ask is we’re pretty soon going to be in a world where the lab revenue, or just AI cash flows — because obviously the accelerators also have these huge cash flows — will be so big that you can just fund extreme expansion of all this production from cash flows themselves.

Dylan Patel

I do agree generally. There’s obviously some physical constraints. The way the supply chain is expanding currently, 100 is roughly still the right number.

Dwarkesh Patel

For 2030?

Dylan Patel

100 ASML tools for 2030. But if you said, “Carl Zeiss, here’s $10 billion. Please fucking just expand production,” that would change things. You would have to do this with every company in the supply chain.

Dwarkesh Patel

But you don’t think that’s gonna happen next year?

Dylan Patel

I don’t think it’ll happen this year. I don’t think it’ll happen next year. I don’t think it’ll happen the year after, because the world is capital constrained.

Dwarkesh Patel

But in a world where, say, the top labs are generating, even combined, a trillion dollars in revenue next year, they’re not able to take $10B of that—

Dylan Patel

I don’t think they’re going to do that, but…

Dwarkesh Patel

Or hundreds of billions at least? It just seems like they realize where the world is headed. I feel like they could just make…

Dylan Patel

The thing is, the labs are going to generate hundreds of billions of revenue next year. But ultimately, CapEx next year is like $2 trillion. So you’ve got this big mismatch. The wafer fabrication equipment supply chain will do something on the order of $200 billion. The data center market supply chain will do even more. The accelerator supply chain will do even more. The energy supply chain will do a number. You sum all this up, it’s going to be well north of $2 trillion of CapEx. So the labs have not yet gotten to the point where their cash flows can fund this stuff.

Dwarkesh Patel

Obviously they will never get to that point, because you want to keep your CapEx higher than your returns.

Dylan Patel

Yeah, you reinvest.

Dwarkesh Patel

The key question I really want to understand is: if the current trend continues, it’d be north of 50 gigawatts per lab by the end of 2028. So between them they’d have 100 gigawatts. Those gigawatts, as you’re saying, drive many-fold more throughput or performance by 2028 than they do now, because the hardware’s gotten better. Not only have flops per watt increased, but also the hardware gets better at working with AI workloads.

Okay, so 100 gigawatts for the labs by the end of 2028. How much is world compute?

Dylan Patel

I think that may be a little difficult, given that by 2028 they’ve taken 70-80% of incremental compute. And I’m not sure what happens to markets then. How much does the price of compute skyrocket for them to actually be able to buy 70-80% of compute? Is Google or Meta or Amazon willing to sell even that much?

Also, there’s one caveat when we’re talking about these gigawatt numbers. When Amazon is serving Bedrock Anthropic models, that counts as Anthropic compute in our worldview, because it is effectively, at the end of the day, counted as revenue for Anthropic even though there’s a revenue share and credit back all that. But ultimately in 2028, if they get to 100 gigawatts combined, they have done really disruptive things to the market.

Because anyone can make money off of $10-15 million per megawatt compute today. I kid you not, it’s not that hard. Go get a GB300 rack, go download the Kimi weights, go download vLLM or SGLang, set it up. Codex and Fable can actually help you do this. It’s pretty simple. It’s not trivial, but it’s not rocket science. Go put it on OpenRouter. It’s very simple. You’ll start generating more revenue than you’re paying for the compute.

This has already led to this compute pricing, $10-15 million per megawatt, starting to inflect up. To get to that 100 gigawatts in 2028, you have to believe that the labs can outpay for compute, because anyone can make money at $10 to $15. Does compute now get to $25 million a megawatt? Does it get to $40 million a megawatt?

Dwarkesh Patel

As you’re saying, it’s already the case that the labs are generating way more revenue per megawatt than everybody else. If they stay as far ahead as they are currently, you would expect that to continue being the case. If there’s some kind of recursive self-improvement where the AI labs are relatively uplifted — or they have models internally they’re not releasing externally that are helping them make their next model better — you’d expect that to be even more the case.

Aren’t you already seeing this, where SpaceX, or whoever is slightly further behind, will just sell compute to the highest bidder if they can’t internally monetize it as well as the labs? You’d expect them to keep bidding for larger and larger shares of the compute market.

Dylan Patel

I think that is my worldview. They will continue to gobble up more of the compute. But ultimately they can’t do it at current pricing or anywhere close to it. They do have to start paying $25, $30, $50 million a megawatt to really gobble up 70% of the world’s compute in 2028, to get to 100 gigawatts by 2028, which is a very aggressive goal.

The other aspect of this that’s really challenging is that we’ve already seen a huge slowdown for the AI labs. This regulation that they advocate for is actually slowing down the labs a lot more than it slows down the open-source Chinese language models. OpenAI not releasing Astra. OpenAI stopping training for two weeks. Anthropic not releasing what their safety assessment says is Model 2, which is widely believed to be the next version of Mythos.

They’re clearly not releasing their best models, in which case their revenue per megawatt stalls or can even start to decline again because other models are competitive again. It’s not that they’re falling behind. It’s just that they’re not releasing their best stuff.

What if there is some regulatory impact that prevents them from releasing their best models? Now their revenue per megawatt does not climb as fast. Their ability to buy that incremental compute for a higher price than everyone else starts to diminish, and then maybe they can’t get to that 100 gigawatts.

But in a world where safety doesn’t matter, I do believe that’s exactly what happens. They can start generating $100 million per megawatt or more, and they can pay $50 million a megawatt. No one else has any logical reason to do anything with their compute besides say, “Please, Dario, take everything off of my hands.” But there are forces at play, which we cannot describe, that would potentially slow this down.

Dwarkesh Patel

I think a good intuition pump is: what if the AI models were literally as good as a fully automated software engineer? They’re not currently there yet. I think they’re far from being able to fully automate the job of a full white-collar worker. But white-collar workers earn six figures or north of that a year. If you have a gigawatt that can sustain a population of, say, roughly a million white-collar workers. Then off the back of that…That would be $100 billion. That’s actually surprisingly low.

Dylan Patel

Yeah, $100K per person, million population.

Dwarkesh Patel

I don’t know. But it would be many hundreds of billions of dollars per gigawatt if you get full AGI.

Dylan Patel

The other aspect of this — and we’ve continued to see this — is that most of the value capture is not happening. Most of the value that these models generate does not get given to OpenAI and Anthropic. Thankfully, so far it is mostly just being given to the users.

Jane Street, with their exclusive contract with OpenAI for GPT-5.6 Ultrafast mode, or Jane Street where they’re one of Anthropic’s biggest customers, is generating way, way, way more value out of the tokens they’re paying for than Anthropic is generating in terms of profit, because they get to make money off of the market.

Or take Meta, who at one point was rumored to be as much as 10% of Anthropic’s business. They’re generating way more efficiencies by optimizing their ad algorithms or what have you, getting engagement time 5% longer, all these things. They’re making way more money off of using these models than Anthropic is.

That’s what’s required. Sure, if you had a million new software engineers, the cost for a software engineer would also fall.

Dwarkesh Patel

One thing I’m confused about is, does the market come into equilibrium? If it comes into equilibrium, would you just expect the price of compute to equal whatever Anthropic and OpenAI can generate from it, or be very close to it with a small amount of markup for Anthropic and OpenAI?

Right now it’s really weird that there is a 4x or more difference between what compute sells for and how much money Anthropic can make from it. In a world where the revenue per gigawatt continues to increase, if Anthropic’s ability to monetize a gigawatt doubles or triples, it’d be weird if the gap continued to increase. Anthropic, just by having some weights, can take something that cost them $10 and turn it into $100.

Dylan Patel

This is always a fun question. Where does the value go in AI? AI’s generating all this value. You’ve got the end user, which I think we all agree is generating more value than anyone else, hence they’re paying a lot for these models. Then you have the app layer. So far the app layer’s generated very little value. Then you’ve got the model layer, which up until a year ago was generating negative gross margins and is now generating massive positive gross margins. It looks like it’s on the path to generating $100 million per megawatt. So turning $10-15 into $100, as you said.

But if we go back a year ago, the hardware supply chain was generating all this gross margin while literally everyone else was losing money on it. OpenAI and Anthropic were just plowing VC money in, as were many other startups. Many of these hyperscalers were building infrastructure without knowing if there was going to be a payoff.

So ultimately you had this negative value being created on the model layer, if you will, because they were selling the tokens for less than it cost them on the infra side. All the value was being captured at the chip, the fab. Initially in 2023, the memory guys were making no money off of HBM or memory for AI, even though theoretically the value they were delivering was humongous.

Now you’ve got… Well, actually TSMC captures way less value than the memory guys. So the value capture’s shifted around a lot, which is very fun for people tracking the market or participating in the market, like Jane Street as an example. This is not an ad. This is not an ad. This is not an ad.

Dwarkesh Patel

They’re a sponsor but you don’t have to plug them that hard.

Dylan Patel

So what happens going forward? Anthropic and OpenAI have slowly started to balloon in value capture. Do they balloon and take all the value capture? Well, that was a thought, and then Elon showed, “Actually, no. I can sell my compute for $25 million a megawatt or $40 million a megawatt to Anthropic and Google. Even if it’s a short-term thing, I’ve sold it for this price, and I’ll recoup my entire CapEx in a year.”

Dwarkesh Patel

What’s your prediction of how much the relevant tranche of compute — B300s or whatever that SpaceX sold for $40B a gigawatt to Google — what does that sell for at the end of next year?

Dylan Patel

I think most compute will still continue to transact at sub-$20 billion a gigawatt.

Dwarkesh Patel

Even at the end of next year?

Dylan Patel

Because all of it has to be financed. If Meta, Microsoft, Amazon, SpaceX can build compute without finding a customer, just saying, “Fuck it, I’m going to build this compute,” and then turn around and wait till it’s already built, they now control what’s going on.

Most compute is contracted well before it’s built. This is what Elon took advantage of in the market. He actually had all this compute. He was like, “Hey, Anthropic, I know you’re making $60-plus billion per gigawatt. Why don’t you just buy my stuff for a crazy amount of money?” Obviously it’s not like Elon decided this or Anthropic decided this. The market figured itself out.

Other people, you go to a random cloud, they’re like, “Okay, I’m going to build a gigawatt of compute or 100 megawatts of compute. I’m going to spend the CapEx. I need to turn around and find a customer. If I want to find a customer, I need to find the capital. Who’s going to give me the capital and the customer? The customer has to sign a deal. Then I take the customer’s commitment to the credit markets and I raise the capital.”

So there’s this completely different power structure where Meta is effectively hoarding compute. Them and SpaceX are the only plausible #3, because they’re hoarding all this compute. They’re using their balance sheets and capabilities to build compute without an end customer that’s monetizing at a huge degree. They have an actual balance sheet, so they can go to the credit market. You build a gigawatt, you can make your margin, not a crazy margin, but a good margin. Now I have all this compute.

Now Meta and SpaceX have this optionality of looking around and being like, “Is my internal use case going to make me more money, or should I go out there and sell it to Anthropic or OpenAI at crazy margins?” So now we’ve entered a regime where SpaceX and Meta are saying, “Actually, I’m going to build the compute, and I can rent it out for not $13. I can sell it for $25, $50, and more.”

Dwarkesh Patel

What do you think their revenue per gigawatt is by the end of 2027? For Anthropic or OpenAI, by the end of ’27.

Dylan Patel

I think it’s highly dependent on who has the best model, if they’re allowed to keep releasing their best models. But I don’t see why it wouldn’t be $50-plus million a megawatt.

Dwarkesh Patel

By the end of ’27.

Dylan Patel

Oh, by the end of ‘27? That’s where it gets more challenging, but I think it could get higher than that, to like $70, $80 million a megawatt, blended across the company, if not higher.

Dwarkesh Patel

Seems low.

Dylan Patel

So if that’s the case, then what happens to the price of compute? Well, if I’m Anthropic, incremental compute is worth it. Maybe I spend $40 million a megawatt on SpaceX compute. If I’m SpaceX, I look to the supply chain and I’m like, “Well, I’ve struck this deal with Jensen (where he’s now all of a sudden using Twitter).” And Elon’s saying they’re exclusive to Nvidia, but why doesn’t Jensen raise his prices? Then SK Hynix and Micron and Samsung look at it and they’re like, “Well, why don’t we raise our prices?”

So with the value capture, I think there’s a bullwhip effect here. Just because someone has raised prices doesn’t mean the entire supply chain rebalances immediately. But over time, the supply chain will rebalance and things will cost more and more. To get that incremental capacity, you sort of have to. So TSMC raising prices very slowly, but memory companies raising prices very quickly. Substrate companies raising prices very quickly. Elon wouldn’t have sold if it was $15, but he’s selling because it’s $25+. So obviously he raised his prices really quickly.

Dwarkesh Patel

I’m surprised you think that revenue per gigawatt doesn’t increase way more than even 100 per gigawatt by the end of next year.

Dylan Patel

When does RSI happen? When does takeoff happen?

Dwarkesh Patel

Or even if RSI doesn’t happen, just say the current rate of progress continues. Just look at how much progress we’ve made in, let’s say, the last year and a half. What was the model from a year and a half ago? Claude 3.5 or something?

Dylan Patel

My problem with this is that the best model that exists in the world was trained in February.

Dwarkesh Patel

So you’re saying maybe we just won’t be allowed to release the labs’ best models.

Dylan Patel

OpenAI says they’re not training models for two weeks, man. What the hell?

Dwarkesh Patel

There’s one thing where internally, are they getting enough use for it that they’ll bid up the price of compute? Another is, does AI progress as a whole slow down because of regulation?

Dylan Patel

Yeah, but they’re not even allowed to use this new model internally. Astra’s not even widely deployed internally.

Dwarkesh Patel

But still, if you have a model that is… What was the model released at the beginning of last year? GPT…

Dylan Patel

4o? Was that 4o?

Dwarkesh Patel

Yeah. You’re talking about a GPT-4o to Mythos 2-size leap by this point, again, by the end of 2027.

Dylan Patel

Yeah, but Mythos 2’s not out.

Dwarkesh Patel

Or even Mythos. That leap again.

Dylan Patel

Even Mythos is not allowed to be out. They’ve neutered it. We can’t use it to optimize inference performance. We can’t use it to optimize all sorts of things.

Dwarkesh Patel

Yeah, maybe there’s some slowdown in AI progress or the deployment of AI that means the revenue per gigawatt can be lower. But that’s the only way I could see it being only $100 million per megawatt by the end of next year.

Dylan Patel

As long as the model gets better, the value generated out of it gets better. Obviously, who captures the value is still up for debate, but ultimately everyone’s going to raise their prices. Because they can, and it’s super inflationary.

Especially if the method of regulation is… Right now, so far, it’s just “don’t release the models.” But more and more, the method of regulation is New York’s banning data centers. Texas is holding moratoriums. Ohio’s saying, or at least trying to say, you have to pay everyone’s property tax in a certain radius. These sorts of things are going to decrease supply and increase cost. That’s going to get passed on as well.

You start to end up in a spot where progress does slow, at least in the external sense, even if the models internally keep getting better and better. In a takeoff scenario, why would Anthropic not have their best model six months ahead of what is externally available? Because of safety and regulation, but also the competitive advantage? That six-month difference, if progress accelerates, is actually a bigger differential.

So that’s the thing that would cap revenue-per-megawatt gains to much lower growth than we’ve seen in the first half of this year.

Dwarkesh Patel

Here’s something I’m very interested in. As these companies go public and they’re accountable to investors, let’s say by the end of next year they have close to 20 gigawatts. So 10% of compute is 2 gigawatts.

Let’s say they want to go from 60% of compute to training to 70% of compute to training. And their investors are like, “Well, if you’re going to be able to generate $100 billion per gigawatt, you’re basically saying no to $200 billion of revenue in order to increase your training compute.” So investors are like, “What the fuck? You’re already spending so much on training. Why are you spending even more on training?”

As a public company, what do you think would happen if they’re just like, “No, we will keep increasing the share of compute we spend on training to offset the increase in revenue that each gigawatt of compute is giving us”?

Dylan Patel

This is what I personally believe. The labs are going to allocate less and less compute to inference over time. I think that’s very non-consensus. The standard belief of most people is, “Oh, most compute will go to inference.” Most of it will go to forward passes for training, not necessarily revenue-generating inference.

Ultimately, if they’re generating $30-40 million per megawatt today, you allocate 40% to inference. If you now get to generating $60-70 million per megawatt, do you still allocate 40% to inference and generate all this profit and then do dividends and share buybacks? Or do you go build AGI? I think the obvious answer from Anthropic and OpenAI, not just at the executive level but also their board, is to go build AGI, because it’s way more profitable. So ultimately you’re going to see them ratchet up their percentage of compute dedicated to training—

Dwarkesh Patel

While each increment of compute is getting more and more profit-generating if they had dedicated it to inference.

Dylan Patel

Right. The whole point is, if I’m selling tokens… Is OpenAI releasing Ultrafast mode for just external, or are they doing it internally too? It turns out, no. Actually, I’m going to allocate it to internal and external, because the internal value I’m generating from super-fast AI or the best AI model is way more than what someone external is.

So ultimately, sure, I could generate $100 million per megawatt, but if I turn that towards AI research, what is the incremental progress that I get? What does that do towards my future earnings potential, the discounted cash flows of whatever the hell I’ve done? They’re not going through that calculation, but ultimately it makes more sense to dedicate more and more compute internally. The only reason to have inference compute be so large is so you can grow your training fleet.

Dwarkesh Patel

I think this is an interesting economics question that I feel we can have the models digest. What would have to be true about a world where they reduce the fraction of compute spent on inference?

Dylan Patel

I think they have been over the last three months already. I think at parts of this year, they were increasing the fraction of compute… Let’s just take it month by month. You would agree that every month, Anthropic has added more compute than the prior month. There might be some noise when they sign a SpaceX deal or whatever, but in general, the amount of compute is a curve up.

So in January, they added less compute than December, and yet their revenue adds skyrocketed. Then they’ve sort of plateaued. They’re not adding $25 billion of ARR every month now. That means the marginal megawatt they’re getting is going as a higher percentage to R&D than it is to inference. So they are factually increasing their compute towards R&D today. I think this is self-evident if you look enough at what they’re doing.

Dwarkesh Patel

If I look at the numbers you said for how fast world compute grows, here are some things I want to understand. It seems like if I add up the numbers you just said, it would be over 200 gigawatts of world compute by the end of 2028, right?

Dylan Patel

Yeah, globally.

Dwarkesh Patel

Okay. How fast can that continue growing, global AI compute after 2028?

Dylan Patel

30 this year, 50 next year, 70 in ’28. ’29 should be on the order of 90-100.

Dwarkesh Patel

Then just 100 more every single year or something?

Dylan Patel

I think the slope can continue to go upwards. It’s hard to predict anything more than four years out. Who knows whether we’re in an RSI regime, or when is the world economy growing at 10% a year? Because if you’re at 100+ gigawatts a year, you’re at absurd GDP growth.

Dwarkesh Patel

If you think there’s 200 gigawatts globally in 2028, how much is in China by that point? How does Chinese compute continue increasing through this whole trend? Because if the RSI stuff kicks off in the West before China has a large amount of compute, maybe we’re living in a different world than when it doesn’t.

Dylan Patel

If we level-set back to 2022, the US was adding about 45-50% of the world’s compute. China was adding about 30-35%. The rest was being taken up by the rest of the world. Since 2022, we’ve had big regulations against China and a dramatic increase in America. So today, 70% of watts are being deployed in America. China is really a very small number. Sub-10% of watts being deployed for data center AI compute is in China.

As we step forward, they’re still at a very small number. Their domestic production is quite small. Their purchasing from Nvidia is still quite small, and a lot of that ends up in other places as well, Malaysia or what have you.

So ultimately, China domestically still continues to have sub-10% of incremental new compute. In 2028 it might start to inflect up, I think. But it’s pretty easy to say China will have 30 gigawatts of AI compute or less.

Dwarkesh Patel

By 2028?

Dylan Patel

Yeah, in 2028.

Dwarkesh Patel

Okay. And then how fast does their hockey stick go up?

Dylan Patel

I do think in 2028, they have a big uplift in what compute they’re able to deploy. In 2026, they’re still mostly relying on a lot of the smuggled chips, a lot of the chips that TSMC made for companies that they thought weren’t Huawei but ended up being Huawei, or a lot of HBM that Samsung is shipping.

But in ’27, fabs start to go up. In ’28 especially, fabs start to go up from SMIC and CXMT and such, where domestic production is actually reaching many millions of units a year. Now they’re incrementally adding 5-10 gigawatts, in just 2028, of domestically produced chips. Those chips are definitely worse than the chips that Nvidia will have in ’28, or Google will have in ’28, or OpenAI will have in 2028.

Dwarkesh Patel

So even the gigawatt number overstates things, you’re saying. It’s 30 gigawatts, but it’s really much worse chips. But if you think the world is going to add 100 gigawatts the following year — I know you said you can’t really say that far out — how much is China able to add the subsequent year? Basically, I want to know: do they just hockey stick at the point at which they are able to start shipping large amounts of compute, or is it still going to be less than US plus allies?

Dylan Patel

There’s a lot left to whether or not the US passes the MATCH Act, whether or not tools continue to get export-controlled, how fast China can build their new equipment that they’re starting to be able to produce domestically. But ultimately, China is definitely going to hockey stick. If there’s anything China’s really good at, it’s scaling manufacturing really, really quickly.

I imagine China will start to be able to extract more and more purchasing of even foreign chips into domestic China, or at least close the gap in what the US is allowing Nvidia to sell them, or what have you.

Dwarkesh Patel

But do you think China could be adding 50 incremental gigawatts in 2029?

Dylan Patel

I think that’s completely reasonable. Part of that could also be purchased from foreign. But yeah, I think it’s completely reasonable that China in 2029 can do 50 gigs. But if most of those are domestic chips, there is some factor there where that 50 gigawatts is really worth as much as 20 gigawatts from American chips.

Dwarkesh Patel

Right. So you’re actually projecting a world where maybe the leading lab in 2028 has more compute than all of China will have in ’29 or even ’30, if you weighted gigawatts by their quality.

Dylan Patel

Implying that there’s nothing done to slow down the US labs.

Dwarkesh Patel

That’s right.

Dylan Patel

But clearly the government and politicians are starting to do that. Whereas China’s not going to slow down AI. In fact, the only thing they’re going to do is accelerate it.

Dwarkesh Patel

Honestly, when I interviewed Jensen and asked about export controls — I am a libertarian person — I wasn’t genuinely sure what I thought about this issue. I was steelmanning the opposite view from what he has, because I think it’s important to hash out ideas. I’m like, “Yeah, maybe there’s a world where if we just cooperated with China, it would be better for us, especially since they control so much of the supply chain and the other things that will be needed for robotics.”

But I didn’t realize the compute situation was as fucked as you’re saying. Actually, the export controls do seem to have really… If they ship the amount that you’re saying, that’s a huge difference. By the time we have automated coder and are getting into automated researcher, China is way far behind on the compute stock. If that ends up being the case, that would have worked. I think that’s actually a notable success.

Dylan Patel

The only caveat there is that some of it is export controls, but some of it is also just financial systems. American financial systems are more willing to YOLO into startups than Chinese financial systems. But once Chinese financial systems choose an industry to focus on, they’ll subsidize it a hell of a lot more.

So the Chinese semiconductor industry has significantly more subsidies than the rest of the world’s semiconductor industries combined. If takeoff is not as fast as you’re implying but actually takes longer, then ultimately China will catch up drastically on the semiconductor side, which then is compute at some point.

The other noteworthy aspect of this is that Chinese companies today are not that far behind in AI models, at least perceivably by the public, relative to the amount of compute they have. The leading Chinese labs have 100-200 megawatts total of compute at most, ByteDance Seed being the one outlier where they have significantly more than that. But Kimi is not running a gigawatt or anywhere close to it. Whereas Anthropic is more than 5 gigawatts by the end of the year.

So the question is, does it matter? I think right now this difference in compute doesn’t matter that much. When we break down the compute ratio or budget of a lab, so far it’s been 60% training, 40% inference. But that training gets broken down further. Actually 50% of the compute is research, 10% of the compute is development, and then 40% is inference.

What I mean by research and development is: researchers are generating ideas, testing new architectures, testing new data mixes, testing new hyperparameters, new attention techniques, blah, blah, blah. But ultimately when they do the training run, when Anthropic trains Mythos, it’s sub-200 megawatts.

Dwarkesh Patel

The pre-train or the whole thing?

Dylan Patel

The pre-train. It’s sub-200 megawatts for, call it, two months. Then the RL is even less.

Dwarkesh Patel

You think the RL was less compute than the pre-train?

Dylan Patel

At least in terms of single site of pre-training, yeah.

Dwarkesh Patel

But total compute was probably higher, right?

Dylan Patel

But it’s sequential. At most, the most they ever used at one point in time was maybe 200 megawatts. In reality they had multiple gigawatts, so most of their compute was going to the research, not the development of a model.

There’s reasons for this. It’s hard to coordinate all these clusters. It’s hard to co-locate all of them. It’s hard to do multi-site training. It’s hard to do RL. Generating even more rollouts during RL does not necessarily make it better. There’s all sorts of reasons why you may not be able to leverage all two gigawatts that you have onto training. Actually, I can only leverage 200 megawatts.

As we get further and further down automated coding and automated researcher, I actually expect the percentage of the compute budget that goes to research versus training to become a lot more fuzzy, or even higher for training. Also things like continual learning. All of these things start to mean that more and more is actually going to training the model.

Dwarkesh Patel

If you end up in a world where you’re doing 100 gigawatts a year, at current prices, that would be $5 trillion of CapEx every single year.

Dylan Patel

Then stack on the fact that you have to build the power plants way before then. It’s also a 30-year asset. You stack on the fact that the data centers are a 15-, 20-year asset, and you have to build that then too. So the $5 trillion, once you account for future years’ growth, is actually going to be more like $7 or $10 trillion of CapEx.

Dwarkesh Patel

Wait, I didn’t understand. That doesn’t include the fact that there’s not the infrastructure for the power generation in the data center itself.

Dylan Patel

Right, exactly. When you talk about AI CapEx, people are saying $40, $50 billion. But that’s really just the critical IT: the servers, the networking, the fiber, the transceivers, optical communications, all this sort of stuff. It doesn’t account for the data center itself or the power plants themselves, which are being built ahead of time.

If I’m building 100 gigawatts this year and 150 gigawatts next year, then all of the buildings for that 150 gigawatts need to be built in CapEx this year. If I’m building 200 gigawatts the year after that, all those power plants need to be spent… You have to buy the turbines this year. So actually, it’s much bigger than even $5 trillion if you’re building 100 gigawatts.

Dwarkesh Patel

Right. Very plausibly, incremental CapEx every year is getting close to $10 trillion by the end of 2030, which is going to be close to a tenth of the world economy. If all of it’s going up in the US… The US economy will have grown as well. But still, at the current size of the US economy, it’ll be like a third to a quarter of the US economy just going towards data centers.

As I say that out loud, I’m like, “Maybe you’re right and we just won’t allow it, and that’s the reason this doesn’t happen.” Because for this exponential to continue, a quarter of America’s economy is just building data centers.

Dylan Patel

I believe in capitalism and reallocation of resources towards the most profitable thing. But at the same time, politics exist, credit markets exist, and capital markets exist.

So to enable, let’s say, that 100 gigawatts by 2030… Or let’s even pare it down to 2028, where it’s like $3 or $4 trillion of CapEx across all of these items: over $2.5 trillion towards IT CapEx, and then another $1 to $2 trillion on data center and energy, and all the supply chain downstream, like semiconductors and all that stuff. If you’re at $3 or $4 trillion of CapEx, where does all this cash come from? No one is generating that much cash from the business yet.

Hyperscalers funded all of the growth up until now. Google, Microsoft, Amazon, Meta. They funded a huge percentage of it. They were more than half of compute, but they now don’t generate cash. They actually spend everything on CapEx. In addition, they raise debt and spend everything on CapEx. You’ve seen Meta do it, even Amazon, even Google. Microsoft will be there soon. Everyone is raising debt to pay for their CapEx.

Now who is the incremental person to pay for this that was not doing it before? In the case of Google, it was pretty simple for them to stop doing buybacks, or Meta stop doing buybacks, and turn around and buy computer infrastructure. That doesn’t have a huge effect on the market, but it does have some effect. But as you step forward to 2028 — where the hyperscalers are now raising hundreds of billions of dollars of debt, and then all of their supply chain is raising hundreds of billions of dollars of debt — who pays for this?

So there’s a few different ways. There’s semiconductor companies like Nvidia and Broadcom and the memory companies turning around and deciding to fund some of this CapEx. There’s the traditional infrastructure investors who are gathering capital and investing in infrastructure. Instead of bridges, it’s data centers.

Then lastly, there’s everyone in the economy who’s realizing, “Maybe I shouldn’t buy a home, or maybe I shouldn’t invest in credit that’s helping people buy homes, or maybe I shouldn’t buy government debt. I should just buy hyperscaler debt, or I should buy this data center’s debt, or I should buy Anthropic’s debt. Because Anthropic’s willing to pay 20% rates for the incremental billion dollars to build their capacity. Because they know their revenue from it’s going to be huge, and they’re going to pay 20% because it’s still better than renting it from SpaceX for $50 billion a gigawatt.”

So you’ve got all of this contention. But if you now do this, the whole world economy is really shifted around.

Dwarkesh Patel

You and I have been debating off air for the last few days whether there will be a sovereign debt crisis as a result of AI. The logic is this. As we were mentioning, you have a situation where very little investment turns into a lot of money. So the rate of return—

Dylan Patel

What a fucking problem, dude. Oh my God. Can’t believe it.

Dwarkesh Patel

No, it is a huge problem for everybody else who can’t turn a little money into a lot of money. So the rate of return is incredibly high. Even at the data center level, if you build a data center and you’re trying to get rented out to an Anthropic or an OpenAI for 10x what it costs you on a depreciated basis to build it, it’s fucking crazy. You turn $1 into $2 or $10 or something at the end of the year. That raises the rate of interest higher.

Now, if the rate of interest goes higher, and if it does that for the entire economy… People are borrowing more and more money. They’re competing against the other lending that the government would’ve done, or that other companies would’ve done, or that you as a consumer or a mortgage buyer would’ve done. That’s making it more expensive for everybody else to borrow. This has huge implications for tons and tons of people. Sorry, I’m going to go on a bit of a monologue here, but we’ve been thinking about this together.

I think the US will be fine at the end of the day. Because if the data centers are built in America, you can fundamentally just tax the data centers. But the way the current tax system is set up, corporate income is less than 10% of federal revenues. 80%-plus is payroll taxes and income taxes, which, as more and more automation happens, will shrink.

At the same time, on the spending side, currently 20% of tax revenue spending goes towards servicing the debt, paying interest payments on the debt. Now, a lot of the debt is short duration, so it rolls over every five years. Why are you fucking laughing?

Dylan Patel

Because it’s things you’ve learned in the last month.

Dwarkesh Patel

Like it’s any different for you. Like you got a degree in fucking financial economics.

Dylan Patel

I didn’t. The internet thinks I’m a beekeeper. Few months, few months.

Dwarkesh Patel

This is our business, Dylan.

Dylan Patel

I know, I know. Sorry, sorry.

Dwarkesh Patel

Now I’m self-conscious. Fuck.

Dylan Patel

No, it’s good. You’re doing good. I just think it’s funny. A million people listen to this guy who just learned about debt this month.

Dwarkesh Patel

Suppose the interest rates rise 1%. Over a five-year basis, the fraction of tax revenue that goes towards servicing the debt goes from 20% to 25%. If it rises 5 percentage points, that would go north of 40%. But if you take into account the fact that the government is borrowing $2 trillion every single year, then that goes from 40% to north of 60%. So 60% of tax revenue just goes towards paying interest payments on the debt.

Now, I think the US is going to be fine because the tax base will increase if we let data centers get built in America. Other countries are absolutely fucked, in my opinion. I was just looking at which countries have a lot of debt, have very little tax revenue, and also a lot of their debt is serviced quite often. Those countries, like Pakistan or Nigeria, I think are just going to be very fucked in this new interest-rate regime.

Dylan Patel

This crowding-out effect is the reason it’s not YOLO 1 billion gigawatts. You’ve got all these industries and countries that use a lot of debt, all these impoverished countries that you mentioned earlier that are just going to default. You’ve got consumer packaged goods, all of these companies that make things you see at Trader Joe’s or wherever. They use a lot of debt. All these telecom companies use a lot of debt. Banks use a lot of debt.

So if interest rates go up in the market — not necessarily the government-set interest rate, but the spread between what the government says their federal rate is versus what everyone else is charging, because Amazon wants to raise $100 billion of debt next year or whatever the hell the number is, probably less — you end up with this really challenging problem of, where does the cash come from?

There is some level that is funded by cash flows and the cash flows keep going up. But the logical thing to do is to invest way more than your cash flows because then the returns in future years will be amazing. So you have this delta.

Then what’s pushing down on the delta is all of these other things: regulations against data centers, consumers getting mad, politicians getting mad, regulations against AI, the AI labs not releasing their latest models because of safety reasons. Interest rates going up are an influence on all of these things. So all of these things bend the curve from what capitalism wants in terms of pure, simple economics to what the complex system that we have wants, and bend it lower and lower to where not as many gigawatts as should be built will be built.

Dwarkesh Patel

Well, the interest rate is part of capitalism, right?

Dylan Patel

Yeah, but in the simple economic model versus the more complex what we have.

Dwarkesh Patel

What is the rate at which you think Amazon or Anthropic or whatever will be issuing bonds for debt next year? If they do hundreds of billions of dollars of debt. What is the average rate?

Dylan Patel

I don’t think Amazon will do hundreds of billions of dollars of debt.

Dwarkesh Patel

In total. Let’s say the big tech guys.

Dylan Patel

The hyperscalers in total, and all the clouds… In the modeling that we do, we have about $11 trillion of CapEx from 2024 to 2029.

Dwarkesh Patel

Total?

Dylan Patel

Total. If you fund a lot of this with cash flows, as much as you can, you still end up with north of $5 trillion of credit that needs to be issued for this $11 trillion-plus build out.

Dwarkesh Patel

So you don’t think the AI revenue continues even 3x-ing year over year?

Dylan Patel

AI revenue does go up. I don’t think it can go up forever without certain constraints being hit. Labs will have certain incentives. Labs are not the ones building all the compute in many cases, even though they’re increasingly trying to go that way.

Dwarkesh Patel

But they’ll have all this cash flow. How much did you say the revenue will be? You think they’ll not have that much revenue?

Dylan Patel

No, I’m just saying till 2029 there’s something on the order of $11 trillion of CapEx. $6 trillion of that is funded with cash, and $5 trillion of that is funded with debt. If that’s the case, $5 trillion of debt being raised across the whole ecosystem does make interest rates go up.

Then what prevents that? There’s a couple things. One, do labs increase their revenue per megawatt more and keep inference allocations large? In which case, they’re accumulating all the profit across the S&P 500 because everyone’s paying to reduce their costs. Of course, their profits will also go up, but cash has to come from somewhere. So there’s an upper limit on how fast their revenue can grow versus the value they deliver into the world. And there’s a diffusion aspect of the technology.

But ultimately labs’ revenues keep going up. They can’t cash-flow fund everything. The optimal scenario is you actually use credit as much as you can to fund, because even if cash flows from the labs fund a lot of stuff, you want to build more than that. So there is some amount of credit that gets built.

Our current modeling has $5 trillion of credit and $6 trillion of cash-funded infrastructure investments through ’29. When you take that, this is not enough compute relative to what the demand growth is from the AI models. So you’ve got the obvious answer, which is revenue per megawatt keeps going up.

Dwarkesh Patel

That makes sense. How much do you think interest rates will increase by 2029 as a result of all this?

Dylan Patel

Dude, this is vibing a number, but if you’re vibing a number out… Growth in the world economy is going up a lot, so why wouldn’t interest rates for Amazon go up from where they are today?

This is going to be extremely vibed out, but recently Meta’s raised at 5 to 6%. I don’t see why they wouldn’t pay 8%. They would happily pay 8% because the return from the compute that they’re going to build is humongous. The market won’t want them to, but they’ll want to pay 8%.

The flip side is that if they pay 8% versus the 5%, 5.5%, 6% they do today — a 250 bps increase — that makes everyone else in the economy also pay 250 bps more, which then causes a lot of things. Banks will scream, because if their credit spread goes up, their debt reprices faster than their assets reprice. They ultimately end up losing tons of money if their credit spread blows up.

Dwarkesh Patel

The other consequence of this — this is a point you made — is that if interest rates rise, the discount rate increases, which means that the discounted cash flows of all equities crater. Which means that even though the stock market as a whole might be doing fine — the S&P 500 will be fine — any individual stock will probably have just cratered in value, especially the Buffett, Berkshire type, pay-good-cash-flows-for-30-years type stocks.

Dylan Patel

Yeah. It’s like, “Why would I pay this much for Johnson & Johnson?” They’re seen as a stable stock: good cash flows, they’ll return their cash flows over time. Or a railway company. Why the fuck would I invest that much if my discount rate isn’t 3% or 5%?” It’s now 8% or 10%.

Dwarkesh Patel

For developing countries… Basil Halperin, who’s a good friend and an economist, made this point that we’ll see a second Volcker shock. In the ’80s, to fight inflation, Fed Chair Paul Volcker raised interest rates more than 5%, something like 8% real interest rate. That caused some 40 different countries, mostly in Latin America, to default in that decade. I think that will probably happen again.

Okay, now we’re getting into singularity talk. We’ve been talking about what happens if interest rates rise—

Dylan Patel

I think this all happens before singularity, by the way.

Dwarkesh Patel

Yeah, that’s what I’m saying. We were talking about before singularity, interest rates rise 2-3%, et cetera. At some point, I think it’s very likely that the world economy will be doubling every single year. This is not happening in five years. But it’ll happen eventually. There’s this researcher, Damon Binder, who’s done great work on this. If you look at input-output tables in a fully automated economy… What would it take to double the entire stock of things in the economy every single year?

Dylan Patel

Yeah. If the economy grows at 3% a year, then rule of 70, that’s 20-something years.

Dwarkesh Patel

Right. But he was like, “Okay, right now we’re bottlenecked by the fact that there’s people, and you can’t double people every single year.” But in a world where you can also double the labor force every single year, how fast can the economy grow? I think it could double every single year. At the very least it would be tens of percent every single year.

Okay. The rate of interest should be pretty close to the growth rate. It won’t be exactly that because of consumption, but it should be pretty similar. Then we’ll go into a world, I think in the 2030s, where the rate of interest is tens of percent. Part of my brain is like, “It might be hundreds of percent,” but let’s say it’s at least tens of percent.

I’m just like, okay. Every country that is not involved in the production of AI defaults. Every stock that is not an AI stock is worth basically zero because discounted cash flows are worth nothing. If the federal government can’t figure out a way to tax AI, servicing the debt is more than the current tax revenue. And you have all these other effects that I’m sure we’re not even pricing in: you can’t get a mortgage, et cetera, et cetera.

Fundamentally, what is happening in this world? This is all nerd speak, right? But let’s step back. What’s happening?

Dylan Patel

Just now it started, the nerd speak?

Dwarkesh Patel

We’d be entering a totally different growth regime. The economy’s basically saying, “Hey, the opportunity cost of the government borrowing money to pay people pensions is extremely high now. Because that money could be spent building a robot factory that builds a robot factory that builds a robot factory.” The opportunity cost of capital is going to increase a ton. That’s fundamentally the cause of all of these things we’re talking about.

Dylan Patel

As interest rates go up, equity markets get pummeled. Even AI companies. Some people who really believe in AI are like, “Why does Micron or Hynix or Kioxia trade at 2 or 3 times earnings?” It’s like, “Well, if you’re really AI-pilled, everything in the economy should trade at 2 or 3 times earnings.” If you’re not AI-pilled, then sure, they’re over-earning.

It’s an argument for why — I think memory is going to do great — memory stocks shouldn’t 10x or whatever again. Because if we’re in the market where there’s that much demand for memory — which means AI’s caused this drastic change in the economy — then everything should trade at 2 or 3x multiples and the stock market should fucking crash.

In a sense, Meta trading at… I think they’re like a $1.5 trillion company. It’s like, what? Silly. They’re worth way more than that, at least in a logical sense. You just look at their cash flows, all the infrastructure they’re hoarding, and all the compute that they’re going to be able to sell for crazy amounts of dollars per watt, either as tokens because their lab works, or just to Anthropic and OpenAI.

It ultimately becomes a question of, you have to reallocate all the capital to the AGI. You do that by pricing everyone else out. So the limiter on AGI is not how fast the research engineers, like our roommate Sholto, can crank the gears. It’s actually just how much does the rest of the world let that happen?

Because they’re going to regulate. They’re going to obviously increase interest rates. They’re going to say, “No data centers.” They’re going to say, “Stop building fabs.” They’re going to say, “Oh shit, every company’s equity value is tanking, so how can I pay for AI to increase my business?”

Well then, Anthropic and OpenAI have to start building their own stuff. They’re building their own chips already, or at least designing their own chips, and it’ll expand out. They’re contracting their own data centers and building their own infra in the next couple years.

There’s the question of how this reallocation of the economy happens. There’s a lot of downward pressure on it not being just straight takeoff, even if the models were capable of it. I think you and I believe we’re in a world where models are capable of that. But slow takeoff is, at least my hope, possible, because of everything in the economy and regulatory world. Government saying, “Don’t release your models,” the government saying, “Actually, you can’t even use your models internally that much,” because that’s going to happen soon. They’re already saying you can’t release your models.

Dwarkesh Patel

The thing I’m most worried about is a singularity, which external deployment is actually helping. So the fact that we’re preventing external deployment is stupid.

Dylan Patel

Does that prevent singularity?

Dwarkesh Patel

Right now it would lead to more revenue, because the models are incapable of RSI. But I’m worried about a world where it’s 2030 and the government’s like, “We’re going to wait six months before you can release your newest model to the public.”

Dylan Patel

Six months, 100x. Let’s go.

Dwarkesh Patel

In that six months, they do recursive self-improvement internally. They just have all kinds of crazy shit happening in the company. Meanwhile, the rest of us are stuck with models that are, at current pace, years behind.

Here’s my thought. Suppose that the whole world gets in on this conspiracy to try to slow down AI.

Dylan Patel

I don’t think it’s a conspiracy. It’s outwardly written from every politician.

Dwarkesh Patel

Suppose they slow down AI by a year. If compute is increasing 2 to 3x every single year, they prevent a whole year of AI deployment such that you’re a year behind where you would otherwise have been. During RSI, you’re getting 3 to 6 years of AI progress in a single year.

Dylan Patel

But they don’t just limit compute. They also limit the lab’s ability to release the model internally. We saw that.

Dwarkesh Patel

If they did that, that would be ideal.

Dylan Patel

Anthropic had to stop giving Mythos to foreign employees for a bit.

Dwarkesh Patel

I didn’t know that was true, internally as well?

Dylan Patel

That’s what they claimed.

Dwarkesh Patel

I thought that was just a different checkpoint that was not Mythos, but it was basically Mythos.

Dylan Patel

But stuff like that is not going to be allowed either. The government is dumb, but they’re not that dumb, I would hope, at least. Governments — at least the US government, which has the cards here — are not going to want Anthropic to use Mythos 4 internally. They’re going to be like, “Hold the fuck on. Slow down,” because of all of these regulatory reasons.

Everyone who’s elected is going to hate AI. Even the people who are elected already hate AI. All the constituents. I bet you at some point your parents are going to call you and be like, “Dwarkesh beta, you’re doing a terrible job. You’re making AI progress happen faster.”

Dwarkesh Patel

Because of my podcast I’m accelerating AI progress?

Dylan Patel

Maybe. You educate people. Maybe if they’re smarter, they’re progressing AI faster.

Anyway, you’re going to have real-world constraints on the progress and development and deployment of AI. Even though it will happen eventually, we could tear ourselves apart before we get there.

Dwarkesh Patel

One thing I find crazy about these scenarios is just how much of the world’s future labor supply ends up in very few companies, and also how fast that labor supply grows year over year. If compute at the frontier in FLOP terms is growing 4-5x a year — and further the compute required to achieve a level of capabilities is decreasing 3x a year — basically the effective AI population size at the frontier labs is increasing 10x year over year. That doesn’t really matter that much right now, because AIs are not good enough to do full jobs or be as autonomous as people in their capacity to do work or pull off schemes or whatever.

But if the current trend continues, you have a world where OpenAI goes from having, say, basically 10 million AI laborers this year to 100 million the next year, to a billion the year after that. Pretty soon, even if compute scaling slows down, it doesn’t take many more years before each company individually has more labor equivalence than there are people on Earth. I think that’s very plausible by the end of this decade, that there’s more AI labor, more effective population, within a single lab than there are people on Earth.

We talk often about centralization of power because of nationalization or whatever. But we don’t think enough about the fact that we’re actually moving very fast into a regime where most “people”, in terms of work output, are concentrated within two labs who are consuming more and more of the world’s compute. If these AIs are misaligned, then most of the world is misaligned, basically, because most of the world’s minds are there. But even if they’re not, very few companies have a lot of influence or a lot of control.

Dylan Patel

There was the whole spat recently where I think Gavin Baker was like, “Dario believes that there’s only going to be one company in the world.” Then Sholto and Dario came out and were like, “No, no, no. We didn’t say that.” But ultimately, if you believe in RSI, if you believe the labs are the most effective user of compute and can generate the most value from the compute, then the only thing that’s going to happen is centralization of compute. If you believe in AI researchers, RSI, AGI, then all of this exists, all of this is the base.

Dwarkesh Patel

This is even true if there’s no RSI. The effective population of the frontier is currently increasing 10x year over year for a given level of capabilities. So if you get to the level of capabilities of a very competent remote worker or a very competent software engineer or a very competent researcher, the population of those is increasing 10x year over year at the current rate of capabilities growth.

Dylan Patel

I see, and without RSI. Then once you have RSI, it’s even crazier.

Dwarkesh Patel

Then it’s maybe growing 100x a year or 1,000x a year. Or their intelligence is increasing but the population isn’t increasing. Or some mixture of the two, right?

Dylan Patel

What world do you see, Dwarkesh, where everything is not centralized? Because it seems to me that every force is screeching towards centralization. And that’s scary as hell.

I would love for it not to be centralized completely. But maybe that’s the whole point of a machine that loves grace, right? It is everything and it makes our lives great.

Dwarkesh Patel

It’s so hard to think about the future. But I agree with you. I think the fundamental problem is that AI training has huge economies of scale, because any effort you spend on training an AI for a specific skill or a specific set of knowledge gets amortized across billions of sessions or billions of users. So that’s one effect.

The other effect is that if you’re slightly ahead in the AI race and compute is in shortage, you can charge a much higher markup because you can better economize this scarce resource. So there are two effects which give more and more to the person who’s ahead in the AI race. There may be more. If models are learning from deployment, and one model is deployed much more widely than another one, it’s getting much more real-world data.

Dylan Patel

Your point is taken that whether it’s user deployment and continual learning, whether it’s training and having these economies of scale, whether it’s the incremental progress where the best AI model helps you to make the next best AI model, RSI, all of these things point to centralization.

Dwarkesh Patel

I think one of the big intellectual projects, honestly, that we should spend some time thinking about — or at least I’ll spend some time thinking about — is: what is a vision of a decentralized, broadly empowered future after AGI that takes these economies of scale seriously? The alternative vision is that the government controls it, and maybe you think that you can trust the government more because it’s not a private corporation.

Dylan Patel

I don’t trust the government, and I don’t trust Dario, and I don’t trust Sam.

Dwarkesh Patel

That’s a problem, right? Obviously it’s very easy to be wrong about the future. You don’t anticipate a key effect or something that changes everything. But ex ante, it’s very hard to see how we avoid a scenario where we have to choose one source of centralization.

Dylan Patel

It’s why capitalism worked, right? It’s decentralized decision-making and decentralized power. And it’s why super-centralized capitalistic economies actually grew slower than super-decentralized capitalist economies, to some extent. You have to have rule of law and all this. But then AI flips all this on its head. And ultimately you’re like, “Actually, private ownership is probably not the most efficient economy, and therefore it grows slower than an AI economy, which is centralized.”

Dwarkesh Patel

Well, it’s still private ownership, but how many firms are really involved in this share of the economy? It’s, what, maybe 2% of the economy right now? $1 trillion divided by 30. Nvidia is a huge share of it, and Anthropic and OpenAI and these hyperscalers. Obviously there are other firms involved, but a large share of the AI stuff is just happening from very few companies. So it could be private property, but very few companies are involved.

Dylan Patel

I mean, this is what the structure of the market is doing. So what can prevent it? I don’t know. Unless AI progress slows down, unless governments regulate the fuck out of it, this is all that happens. In which case, we’re headed for a world where either we have super concentration of resources and we pray that that one company gets everything right, or we have governments slow everything down and people slow everything down, and you have a slowdown of progress somehow hopefully, and there is more of a balance of power.

Even as we go towards AGI, ASI, RSI, everything along the way will still lead to someone capturing more resources. So it’s kind of hard to find a framework in which AI doesn’t lead to super concentration.

Now, the one positive thing here is that today Anthropic does not capture most of the value. We can talk all we want about how they went from $20 million per megawatt to $100 million per megawatt, but they’re still paying $13 million for a lot of the compute they’re buying. But at the end of the day, the reason they’ve gone to $100 million per megawatt is because Jane Street is capturing $300 million per megawatt or $500 million per megawatt. Or Dwarkesh, from researching his podcast and learning about credit, is capturing how many dollars per megawatt? Now how much can you use? Tough.

But I think that’s the one saving grace, that the rest of the economy maybe profits so much more from Anthropic—

Dwarkesh Patel

No, but the whole logic you were laying out earlier — them reallocating inference to AI R&D — the whole logic of that is that the returns to labor inside AI labs are much higher than the returns outside.

Dylan Patel

Yes. This is my cope. I agree. In all scenarios of the world… There’s 80,000 worlds and in only one of them, Anthropic doesn’t own the whole world. Again, power concentrates because I don’t want to send the tokens outside. They’re more valuable inside. So it’s the same thing. Why would I let Jane Street make all this money off of these degenerate options traders?

Dwarkesh Patel

Hey, they’re a sponsor, come on. Jesus Christ.

Dylan Patel

No, I think it’s great. It’s a good value for the world to make it an efficient market. Jane Street making all this money off of getting the worldview correctly, making money off of degenerate options traders, whatever it is, why would Anthropic allocate compute to that? If the end monetization that Jane Street has per megawatt is $200 million, so they’re willing to pay Anthropic $100 million… Well, what if Anthropic can just generate hundreds of millions of dollars per megawatt by using that compute internally? That’s what’s happening.

Dwarkesh Patel

On that somber note, I guess we’ll meet again when the RSI is officially kicked off.

Dylan Patel

You’re not going to have me on your podcast again for like two months?

Dwarkesh Patel

Alright, cool. Thanks, dude.

Had Ryan Greenblatt on to discuss/debate recursive self-improvement.

This might be the most important question in the world right now – whether within a year or so of achieving human-level intelligence, you slingshot towards having 10s of billions of superintelligences, each of which is dramatically more competent than human experts across all fields.

I’ve historically been skeptical of this possibility. My intuition has been that we will end up significantly bottlenecked by not only compute scaling but human expert data, which I think underlies most of the AI progress today.

If, because of RSI, we got a jump as big as GPT-3 to a Mythos (i.e. 6 years of AI progress) within a single year of achieving AGI, then the thing we get there at the end of that year is definitively and wildly superhuman.

We hashed it out, and I think Ryan made a pretty good case that this kind of speedup is plausible. FWIW, Ryan’s median for when we automate AI R&D is 2031.

We then discussed the alignment implications of this scenario. Who should these superintelligences be aligned to? In the future, our capacity to steward our votes and our capital, and to make sense of what’s happening in the world, will all be titrated by superintelligences. And I worry that specs like the Claude Constitution are not shaping these ASIs to truly be my personal advocates and guardian angels.

And can we get them aligned to anything in the first place? Ryan and I had a long debate about whether the kind of reward hacking we saw with the OAI/Hugging Face hack extrapolates to superintelligences that would team up to literally take over the world.

The first piece of advice you get when you’re learning to drive is that it will go much smoother if you look at the horizon instead of directly in front of your tires. And so it is with the trajectory of AI. Hope you enjoy!

Watch on YouTube; listen on Apple Podcasts or Spotify.

• Antithesis is a software testing platform that finds the failures no human or AI could ever anticipate. It runs thousands of copies of your code inside a fully deterministic computer, injecting faults and steering each trajectory toward the most insidious bugs. This lets you find critical issues in minutes rather than waiting months for your users to uncover them. Learn more at antithesis.com/dwarkesh

• Jane Street’s back with a new puzzle. They designed an ASIC and sent me the final masks… but they didn’t tell me what the chip actually does. So that’s the challenge: reverse engineer the circuit and figure out the chip’s purpose. Jane Street has a bunch of swag ready to send to the most creative solutions, and they’re also planning to feature the top write-ups in a blog post. Download the files and get started at janestreet.com/dwarkesh

• Cursor and SpaceX recently released Grok 4.5, and I’ve been surprised by just how good the model is. For example, when I tested it against Fable and Sol on a bunch of AI governance questions, all three models gave substantially the same answers, but Grok was faster, more concise, and cheaper. Grok 4.6 is coming soon, but in the meantime, you can try 4.5 at cursor.com/dwarkesh

(00:00:00) – Is AI R&D verifiable enough to unlock recursive self-improvement?

(00:16:52) – Is AI progress bottlenecked by human expert data?

(00:34:02) – Flat token prices suggest scaling has been slow

(00:39:47) – Skills AI can’t train on: does it even need them?

(00:48:07) – Aligned to whom?

(01:09:18) – Recent incidents of AIs colluding and deceiving humans

(01:19:38) – What could possibly go wrong? A concrete scenario

(01:48:02) – From reward hacking to takeover

Dwarkesh Patel

Today I’m chatting with Ryan Greenblatt, who is the chief scientist at Redwood Research, where he focuses on technical AI safety and security work.

I want to talk to you about recursive self-improvement. This is the idea that once we build human-level intelligences, they quickly slingshot towards tens of billions of superintelligences, which are each individually more competent than the top human experts across every field. Whether or not this turns out to be the case is probably the most important question in the world right now. And historically, I’ve been quite skeptical that this kind of thing happens, but, you seem to think that it might be plausible, and so I wanted to hear the case for it.

Ryan Greenblatt

Let’s talk about this. First, I think it’s worth noting that AI R&D is a type of task at which the AIs are especially good, because the companies are trying really hard to make their AIs good at AI R&D. It’s also the kind of domain that has a lot of nice properties from the perspective of how AI development works right now. It’s pretty verifiable. You can do a bunch of stuff iteratively, and it’ll hill climb on various metrics.

I think once you have AIs which are roughly matching the top human experts in AI R&D, that could kick off a feedback loop where the AIs are doing AI research. That produces smarter AIs. That feeds back in. That feedback loop could be strong enough that you end up with a lot of progress in a short period of time. Maybe my median expectation is something like four or five years of AI progress in a single year. This requires really overcoming a huge amount of diminishing returns in research and basically doing the equivalent of the progress we would have gotten after a really large compute scale-out. So this is a pretty impressive, big thing.

It’s worth keeping in mind that five years of AI progress, four years of AI progress, even three years of AI progress, is really a lot of fucking AI progress. A little over three years ago, GPT-4 had come out. Right now, of course, we have Mythos 5 or whatever, and maybe a somewhat better model that Anthropic has internally. That is just a huge amount of progress in a bit over three years. If we’re talking about five years, then maybe we’re talking more about a jump from GPT-3 to Mythos 5 or whatever.

Dwarkesh Patel

I think this argument has three different parts. Now I want to evaluate each one of them. First is the argument that AI R&D is very verifiable. Second is the argument that if you automate AI R&D, you could get four or five years of progress in a single year. Third is the argument that what comes out the other end of four or five years of AI progress at the current pace, starting at the point whenever AI R&D is automated, is an AI where you can drop it on the job at basically anything you can imagine.

You can drop it in Texas politics in the 1940s, and it outmaneuvers Lyndon Johnson. You can drop it in TSMC, and it learns how to do better process engineering at TSMC. It’s certainly a better video editor… My video editors are very excellent, but it is just, in general, better than humans at any given job that it finds itself trying to do.

So I want to evaluate all of these sub-arguments that lead to basically getting ASI pretty soon after this benchmark, which you’re expecting by 2030 or something, right?

Ryan Greenblatt

I would say that I expect full automation of AI R&D perhaps somewhere around 2031, 2030. Getting to the “beats all humans on the job” milestone, maybe my median expectation is around 2033. But if I see AIs fully automating AI R&D, I think I’m expecting that probably within a year. The way the forecasting works out, the difference between medians is bigger than the median difference between milestones. Anyway, whatever.

Dwarkesh Patel

By the way, there’s this meme on the internet. Every time I’m trying to ask about people’s timelines, when I’m asking Dario or somebody, I’m always like, “Okay, how long before you automate my video editors?” There’s this meme of my video editor editing the podcast every time I listen to this.

But the reason I do it is because I think it’s easy to get lost in abstractions when you talk about jobs you don’t understand well, and to very concretely understand what it takes to automate a job that I actually understand why it’s difficult for LLMs to currently take control over.

Ryan Greenblatt

I do think that the milestone for automating your video editor is earlier than the milestone of being able to automate all human jobs, including Texas politics, spinning up on the job. I do think that the video editor automation occurs maybe more around full automation of AI R&D, but it’s very sensitive to how much people are really focusing on understanding video.

Dwarkesh Patel

Okay, so let’s start with the claim that AI R&D is very verifiable.

Ryan Greenblatt

There’s a few different parts of this. One of them is that we can train on a bunch of environments which are basically directly training the model to do some AI R&D task or some very close-by task. For example, we can have some environment where the model is training some AI on just eight H100s or some small amount of compute, and that model could be the equivalent of GPT-2 medium or whatever, and then similar to NanoGPT medium runs — and in RL, it’s tweaking and iterating on that.

We could do that for a bunch of different tasks. We could have it train image classification models, video generation models, image generation models, all kinds of different ML training tasks. We could RL it on the task of training increasingly good models, and also doing things like, “Oh, here’s a particular direction you could pursue for an algorithm. Can you go and implement that?”

Basically, there’s this whole class of containerizable, verifiable, small-scale AI R&D tasks that we can aggressively RL the AIs on. Already companies are presumably doing some RL on these sorts of tasks, and you could just keep scaling that up, keep making more of these small-scale AI R&D tasks, and then the AIs could keep getting better at this. Implicitly, I’m claiming this will transfer to extremely load-bearing aspects of AI R&D. But maybe let’s stop there for a second and then get to that part.

Dwarkesh Patel

So let’s talk through what this concretely looks like. You can imagine that we have GPT-7.5. We say, “GPT-7.5, we want to make you so good at AI R&D that you help us train GPT-9.” So now we want to train GPT-7.5, and we come up with a bunch of different environments.

As you mentioned, there’s already this repo that is the descendant of Andrej Karpathy‘s nanoGPT speedrun, where you just try to change everything about the model, from the optimizer to the hyperparameters to the architecture, to get it to a fixed training loss as fast as possible. You could have other kinds of environments where you could say, “Hey, GPT-7.5, I want you to train a really good video game-playing model. I want you to train a model that actually improves as it plays the same video game again and again. So you learn how to maybe help the model get better at online learning. We don’t care how you figure this out. Maybe it’s some kind of crazy neuralese or a vector memory. Or maybe it’s just better long-context stuff. We don’t care. Figure out how to do online learning research.”

Obviously, GPT-7.5 will already be a smart model, and in the same way the models currently are getting smarter, it’ll be better and better at coding. You can imagine 100 other environments like this which are incentivizing the ability to do AI R&D, like containerized versions of getting GPT-7.5 to develop GPT-2-sized models, et cetera. Then you basically put GPT-7.5 through a bunch of this kind of training, you build GPT-8. GPT-8 is now an amazing ML researcher. It has so much intuition from doing all this kind of training.

Honestly, a huge intuition pump for me is seeing the progress that AI has made in mathematics. If it’s a very verifiable domain, AIs can get… I don’t really know the object-level details of mathematics research, but I’m just like, “No, it works.” It can just come in like a flood if you can totally put it into a verification loop, and it can actually make new breakthroughs.

I am curious if ML research has a quality of mathematical research where it seems like there was a big overhang from connecting different disciplines together. No one person would have known enough about algebraic geometry and… What was the right word?

Ryan Greenblatt

Oh man, I really don’t know about the math breakthroughs.

Dwarkesh Patel

No one person would’ve known enough about topology and algebraic whatever in order to make some counterexample to a big conjecture.

Ryan Greenblatt

My view is that ML is a less deep domain than math, and so there’s less of a thing where there are individual experts with really deep expertise in some area that they combine, but there’s definitely going to be some of that.

But then I also think that ML has some attributes that make it even more favorable to AI training than mathematics in some ways. In particular, you can get a better sense of whether you’re succeeding, and you can see intermediate progress. In math, it’s often the case that there’s no easy way to see whether or not you’re close to success. Whereas if your goal is, for example, to get to some training loss 2x faster, you can kind of see when you’re halfway there. It tends to be the case that ML innovations are very additive, or maybe multiplicative depending on how you think about it, where basically you can keep stacking innovations. Usually the innovations just add together and don’t interfere with each other, though obviously it’s going to depend on the details.

So I think that in a lot of ways, AI R&D will have properties quite similar to math, where you can train on chunks of AI R&D that are pretty similar in structure to the problem you actually cared about, in a very verifiable way, and then that will transfer. There’s an open question of exactly how well it will transfer, but I think that the transfer currently for math looks pretty good. My expectation is that the transfer for AI R&D will look pretty good, but not amazing.

Dwarkesh Patel

So one concern I have is that I think even in mathematics, as far as I’m aware, we have not seen very impressive new theory. We’ve seen a lot of impressive, verifiable, specific results — for example, find a counterexample to this conjecture — but we have not seen “come up with the idea of topology“ kinds of levels of things, or “come up with things like group theory“.

It seems like ML research has elements of both of these things. But the less verifiable thing of coming up with new ways of thinking about the problem would be harder to induce. Take, for example, the idea of scaling laws. Obviously, there is some end verification loop such that you can train GPT-4 better if you have the idea of scaling laws from 2020. But there is a longer and potentially more compute-laden road to inducing AIs to be like, “Okay, I got to think carefully about how I should be scaling my parameters and data. What are different kinds of investigations I could run to understand this? Maybe I can come up with a visualization and an isoFLOP analysis or something.” But that does seem like a longer verification loop than just, “Hey, let’s get nanoGPT loss to go down.”

Ryan Greenblatt

Let’s talk about this. First of all, I think in the context of math, the thing I would say is that the AIs can do the equivalent of ‘baby’s first new theory,’ where, for example, they can just prove interesting conjectures via making connections and producing new understanding. It’s like, “Oh, there’s this construction the AI found which is pretty interesting”, or it found this way of thinking about the problem that’s a bit different. We do see that. It’s just that the examples we see are not as impressive as founding the field of group theory.

Founding the field of group theory is probably among the best, biggest mathematical accomplishments of all time, and the AIs just aren’t that good at math yet. From my perspective, there’s a continuum between that and the things we’re seeing now, that the AIs are continuing to march up.

Second, I think ML is a very shallow domain relative to math. In math, there was much more of a thing where you find some true deep abstraction, and if you really understand that thing, which is hard to understand, then you get somewhere. Whereas I feel like the things that are the equivalent of that in ML are really dumb bullshit. Like with scaling laws, come on guys, we can explain scaling laws really quickly. I think the deepest and most important concepts in math, for example, don’t have the property that you can really understand the underlying thing and why it matters in a very short period of time.

Dwarkesh Patel

But I feel like one effect will be that we will have gotten rid of all the low-hanging fruits by 2030. I feel like scaling laws will have been, in math history, like Descartes finding the Cartesian grid and doing very basic mathematics. Eventually, if we want to keep making progress in the 2030s, it’s going to be like doing whatever bullshit is happening at the frontiers of mathematics right now.

Ryan Greenblatt

That could be right. My sense is that some domains are structurally different in terms of how they operate and how much they depend on deep abstractions. Physics and math are much more on the side of being very far on the deep, hard-to-come-up-with-ideas side, whereas I think ML and most other domains are much more amenable to hill climbing. That’s my sense of how this will go in the future.

Even in the regime where your AIs are having to plow — it’s 2030, a bunch of low-hanging fruit in research has already happened, and they need to make further progress — I still suspect that a bunch of the work will live more on the side of building increasingly complicated infrastructure and having really good intuition about what the experiments roughly look like. So I’m probably less sympathetic to the idea that the thing the AIs will lack is some deep insight. I’m more sympathetic to the idea that they really need a bunch of taste about in-the-weeds experiments that they currently don’t have. They need a bunch of intuition for what sorts of training approaches would work and what wouldn’t, in ways that current researchers have.

Even in cases where there has been some breakthrough in AI, oftentimes in retrospect it looks like a big bottleneck to making that breakthrough happen was getting all of the micro details and mungy intuition right. An example of this is training AIs to be good at reasoning and chain of thought, doing RL on chain of thought. It looks like you probably could have done RL and chain of thought on GPT-3 and gotten kind of interesting results on math if you had really scaled it up and done a good job.

But at the time, there was low-hanging fruit. Also, doing a good job with that training is kind of in the weeds on all the technical implementation and scaling it up and getting the hyperparameters right. So maybe you can demonstrate everything on Qwen 1B or whatever and get some sense that this whole thing is going to work. But people didn’t demonstrate it as early as they could have because of all of these other mungy details and intuition about exactly how to tune the parameters and how to set things up.

Dwarkesh Patel

This is my remaining skepticism, honestly, about this story. I’m not sure I understand why, if research breakthroughs are so amenable to intelligence, AI progress has not been historically faster than it could have been. As you were saying, by the time RLVR actually worked — even though you could have done it with less compute — we had to wait for oceans of compute, gigawatts of compute, to be available before people were doing this training, on the trajectory of compute continuing to increase so we make more breakthroughs.

I don’t know. I feel like there were a lot of AI researchers in the year 2022 who were trying to crack reasoning. Was it just that they were bottlenecked by the ability to write infrastructure code, or what was happening?

Ryan Greenblatt

It’s a complicated mix. I think they would have gone faster if they could, as soon as they thought of an experiment, run that experiment without bugs, without bugs being very important. And then another part of it is that being able to run a lot of experiments at high compute lets you paper over ways in which the way you implemented it isn’t quite right or you didn’t have the right hyperparameters. So compute is just really helpful for doing AI research, and you can cover over a lot of things. But that doesn’t mean that massive increases in labor wouldn’t also be helpful, especially if that labor comes with among the best intuitions that people have in the field. I just think that’s really helpful.

Another part of my perspective here, which is maybe a bit different from where you’re coming from, is that I’m expecting somewhat more transfer than you seem to be imagining. I’m imagining these AIs are actually pretty good scientists in general and are pretty reasonable at all of that stuff. When you interact with them, it’s not like they have some really hyper-specialized savant-type vibe. They’re actually pretty good at all of the stuff in R&D, and then maybe extremely good at some subdomains. So they’re incredibly superhuman at writing kernels, incredibly superhuman at everything with very short feedback loops, and then pretty good at all the other stuff, totally able to match other people.

I think we are seeing this now. When I look at AIs right now, it’s already the case that they can pretty competently match humans who are mediocre at ML research at doing ML research. It’s just that being mediocre at ML research is not that helpful. The thing you actually want are people who are good at ML research. My sense is the AIs are just improving at all of these things. Their taste is improving, their intuition is improving, and it’s already the case that their taste and intuition is not complete garbage.

Dwarkesh Patel

I want to very concretely understand what it would look like for five years of AI progress to happen in one year. Suppose we were back when GPT-3 was developed. The idea is that, with the level of compute they had back in 2022, if we had automated AI R&D back then, you could at the end of that year have Mythos.

Ryan Greenblatt

That would be the idea, yes.

Dwarkesh Patel

Mythos took way more compute than they had back then, but even with the level of compute they had back then, not only do all the breakthroughs happen, but they also train Mythos with that level of compute.

What would be required is obviously discovering all the algorithmic progress since then. It’s discovering even more, actually, because you’ve got to make up for the fact that Mythos uses… What was GPT-3 trained on? Like 1e23? We can look it up. But is it plausibly four orders of magnitude more compute?

Ryan Greenblatt

I think it’s somewhat less than that. Let’s look this up quickly. GPT-3 training compute is about 3e23. My sense is that Mythos is probably a little over three OOMs higher. So the question is: can you overcome this 1000x compute gap while also being the model?

Here’s a concrete claim that maybe we should talk about. Right now, would we be able to train a model with GPT-3-level compute that matches… What exactly do I think? GPT-3 was released in 2020, so it was trained about six and a half, seven years ago. It’s worth noting that GPT-3 is maybe a little too far in the past, but let’s go with this for a second.

If we were to train a model with GPT-3-level compute today, how good would that model be? My understanding, based on how algorithmic progress works, is that we’d be able to train a model that’s as good as the best model we had perhaps around three years ago. So I think that right now we’d be able to train a version of GPT-3 that’s probably somewhat better than GPT-4, a moderate amount better than GPT-4. I think that’s about right. That roughly lines up with how algorithmic progress has worked.

Basically, the story would end up being that to get five years of AI progress, you’re probably going to need around, I would say, maybe eight years of algorithmic progress, very roughly, which is a lot of algorithmic progress. But it just turns out that most of the AI progress, from my perspective, has come from some mix of algorithms and data, and you can just keep making huge improvements on these things and training AIs with less compute.

Dwarkesh Patel

I’m glad you brought that up, because what has happened since GPT-3, or even 3.5, till now? Why is Mythos so good? Obviously, we’ve scaled the compute. We have better algorithms. But a huge thing that’s happened is that we have built a deca-billion-dollar data industry which has systematically collected and codified expert human judgment across all kinds of different disciplines — codified in the form of RL environments, codified in the form of SFT traces — that these experts built to help the model better understand how you do coding, how you build complex infrastructure projects, how you do law, how you do whatever. How are the AIs able to replicate the effect that expert human judgment currently seems to be playing in AI progress?

Ryan Greenblatt

My sense is that scaling up the amount of effort spent on getting expert human data has not been hugely important for AI R&D in general. In particular, over the last few years, we’ve been scaling up compute, scaling up people working at AI companies, and scaling up the amount of effort spent on data labeling. My sense is that if you removed the last two doublings or whatever of data generation from expert humans, that would not make a huge difference. A lot of what’s been going on is people have been developing better ways to leverage humans and AIs to construct RL environments and going somewhere from that.

Dwarkesh Patel

But how do you explain why the AIs have gotten so good at coding? I feel like a big part of that is data and RL environments, which are codifying human experts.

Ryan Greenblatt

But the question is what is the limiting factor on creating RL environments? My sense is that the reason why RL environments today are much better than they were in 2024 is not so much because we have hired way more human experts to make RL environments. It is instead much more because we better know what RL environments we even want to make and how we should structure them. Also, we’re using huge amounts of AI labor to build RL environments. I think those effects are much more important than the effect of human labor building the RL environments.

I’m not saying that the human labor doesn’t matter. I’m just saying there are other big drivers that are important here. I could try to argue for this. One thing is just that the amount of environments people want is a very large amount. I think the AIs are actually pretty good at the task of making RL environments given some sense of what the thing should be. There’s preexisting data you could use. A lot of these things have good verification loops.

Dwarkesh Patel

Just look at, for example, what was reported in Business Insider yesterday, that Google is paying close to $2 billion for Mechanize. We can just look at market rates for what people think really good human expert data is worth. The frontier labs seem to think it’s worth a lot. They’re willing to pay for it.

Ryan Greenblatt

What fraction of frontier lab spending do you think is on data rather than compute? What do you think is the compute/data spend split?

Dwarkesh Patel

I think it’s overwhelmingly compute, but I also think it’s because compute is easier to scale up than data.

Ryan Greenblatt

But that’s really relevant to what’s driving progress, right? My sense is that the split is something like 20 to 1 or 10 to 1. I don’t know exactly. It depends on the company.

Dwarkesh Patel

But this is similar to how oil is 1.5% of GDP. That doesn’t mean that if you cut oil out, GDP could continue to run.

Ryan Greenblatt

Sure, but it contradicts your argument, right?

Dwarkesh Patel

The economy would come to a halt immediately if oil went away.

Ryan Greenblatt

Sure, but you were just arguing that because of the high market cap, we can learn that this is the key driver, and I’m saying that’s not clearly true. That argument just makes it look like compute is a much more important driver, or hiring employees is a much more important driver.

Dwarkesh Patel

So maybe let’s be more concrete. Here’s what I think. My claim is that if you went back to 2022 and you had GPT-3.5, and you were trying to make it better at coding without human experts, I think it would have just been very, very difficult.

Let me give you an example of what I imagine would be the difficulty of going from GPT-8 to ASI. One of the things you’d want ASI to be good at is: I’m going to take over a company and make it much more profitable and do all kinds of crazy shit to make it work better. I’m going to take over a fab and produce more chips. I’m going to go into Congress and try to convince them to pass some bill, et cetera.

This is what I imagine five more years of AI progress at this pace would enable an AI to be able to do. This is the thing I’m really worried about: ASI that can understand how to do crazy shit in the world, that can do what Kissinger can do, can do what Steve Jobs can do, et cetera, and also his engineers and so on. I’m not sure how you get that without the relevant world data, which is the equivalent of Mythos being really good at coding while not having the coding environments that have improved it relative to GPT-3.

Ryan Greenblatt

Here are a few points. First, I bet if you look at randomly sampled training environments for Mythos, they’re actually very different from what it looks like to actually use the model in practice. My sense is that the RL distribution has really large deviations from the real-world data distribution, and it’s significantly smoothed over by a mix of transfer and having a small amount of data focused on the real world. My sense is that this will be a similar mechanism as how it works for the crazy, wildly, quite superhuman AI you get as a result of five years of AI progress on top of fully automated AI R&D.

So let’s go through this a little bit. In particular, I think that you could train an AI to be really, really good at learning on the fly and doing something analogous to in-context learning, but potentially using somewhat different mechanisms, in a wide variety of RL environments. You build all these different RL environments where the AI has to adapt on the fly, learn on the fly, figure out what it should do, understand its situation better, and learn really quickly from feedback in order to succeed at its objective. And it has things like limited resources, and if it messes up, it can end up in a much worse position.

If you train on a huge number of these environments, you will learn general skills of picking up context on the fly, and we’re already seeing this. It’s already the case that AIs are now much better at understanding roughly what’s going on and picking up context from a limited amount of information they’re given access to.

Then those AIs could be put on the job at TSMC. Even though TSMC is not literally in their data distribution, their data distribution is really wide, and the AIs are extremely good on their data distribution, such that it transfers to picking up being good at being an engineer at TSMC and learning that on the fly. The way the AI gets good at being a TSMC engineer isn’t that it has a ton of cached knowledge on being a good TSMC engineer. It’s that it does the equivalent of some scaled-up version of in-context learning there.

That’d be the most prosaic story. Obviously, there’s a bunch of different ways this could go.

Dwarkesh Patel

I think this maybe comes down to a difference of intuition about how far you can get. When I think about really smart people I know, they’re just not that effective in domains they don’t understand that well.

Ryan Greenblatt

But how long have they had to learn?

Dwarkesh Patel

I agree that if they had experience, they would be much better. But that’s maybe what I’m arguing for, that experience with data. For example, if I just get a really smart Ivy League college grad, and I’m like, “Okay, you’re now in charge of negotiating the Iran deal,” I think they just wouldn’t know what to do.

Ryan Greenblatt

I think if you instead got someone who is really good at quickly picking up a bunch of different domains and you gave them some time to train and talk to people and shore up their expertise and do some practice, they would actually do a pretty good job. I think most domains are fundamentally pretty shallow, where a very smart generalist who’s good at a limited subset of core skills can get going pretty quickly. That’s not true for literally every domain. My sense is that the AIs will develop increasingly good mechanisms for quickly acquiring understanding and expertise in a given domain.

Consider, for example, how fast AIs can understand a new code base. AIs can understand a new code base much faster than humans can, but to a degree that’s shallower than humans could currently understand. But it’s getting better over time. Let me spell that argument out a bit more. Let’s say you take Fable 5 or Mythos 5 or whatever, and you wanted to make some kind of complicated change to a really massive code base. The model will get some understanding of the code base very fast, in the course of maybe significantly less than an hour, potentially much less than an hour. Then its understanding of the code base will plateau a little bit, where it won’t get as deep of an understanding as a human would have gotten over a much longer period.

So it’s like an AI in an hour can match a human with a few weeks maybe, depending on the details of exactly how complicated the code base is. But it won’t match a human who’s been working on that code base for two years or whatever. But over time, the amount of understanding AIs can match has gone up. If we look at 3.7 Sonnet or 3.5 Sonnet, maybe it could only match the equivalent of understanding a code base for a day or something.

But now AIs are much better at building context about a task. So you can be like, “Mythos, I want you to really understand this code base, and then implement this feature.” It will spawn a bajillion sub-agents. Those sub-agents will pore over a bunch of things. It will deliver a bunch of context back. It will then investigate a few things. It’s not amazing at doing this, but it can happen really fast, and it can work pretty well.

And it’s not very hard for me to imagine how you could train AIs to be increasingly good at this task. The task of implementing some very complicated feature in some reasonable way in a very big code base is extremely verifiable, and that can be a thing the AIs improve on. Similarly, there’s a broader skill of quickly understanding context and being able to have a bunch of different AIs learn in parallel and then merging that together.

Dwarkesh Patel

I think there seems to be a crux here, which I think is just an empirical question we’ll see. How good is the transfer between getting really, really good at understanding the situation, getting up to speed, making progress over long periods in verifiable domains — which the AIs are obviously getting way, way better at really fast — to, “Okay, go talk to the president and convince him to do X thing.” Or, “You’re now in charge of Google. You must make Google a much more profitable company this quarter.”

Ryan Greenblatt

Let me try to spell out a few more arguments that are maybe relevant. One thing is, when looking at how the AIs have improved at essay writing… Let’s talk about that a little bit. You can get some data even on these domains. AIs will be able to get some data even on these domains when on a very fast progress trajectory. Maybe it’s hard to build a verifiable environment for “was your essay really good according to humans?” But you can do a bit of that. You can do some training. You can do some online training. The AIs will be able to do some online training based on real-world stuff. They’ll be able to have evals. They’ll be able to sample that. You can scale up the cadence at which you do this.

The second thing is that in practice, when I just look at the transfer, it seems okay. I think the AIs have in fact improved a bunch at non-verifiable domains, and it’s hard to point to domains that are really hard to verify on which the amount of improvement between GPT-4 and Mythos hasn’t been pretty high in practice. Now, that doesn’t mean that Mythos is better than the best humans or something. It can still be significantly worse than typical human professionals at some aspect of their job while still being way better than GPT-4, which was not even close.

Dwarkesh Patel

So we’re talking about how much progress has come from data versus algorithmic progress over the last few years. That reminds me, I’m actually running an experiment with this with Jerry Han, who’s still a college student. What we’re basically doing to evaluate how much progress is coming from data versus algorithms is training the best algorithmic recipe from 2019 till now with the best data from the 2026 data file, and then also training the different data files going back from 2019 to 2026 with the current best algorithmic recipe.

I think that will be interesting. I’m curious if you want to pre-register what amount of compute multipliers are coming from one versus the other.

Ryan Greenblatt

We need to be pretty careful with what we mean when we say the word data. I was trying to be pretty careful to distinguish between scaling up spending on getting human experts to label data, or scaling up the amount of human expert-labeled data. The reason why we have a better pre-training data set now versus in 2019 is not because people are spending way more money getting human experts to type up data that the AIs are then trained on.

Dwarkesh Patel

Partially.

Ryan Greenblatt

I think it’s not much of it. I think it’s very little of the pre-training data improvements. I do mean pre-training. We should maybe talk separately about mid-training and post-training. But I think the vast majority of pre-training data improvements are from science on better understanding what data sets are good and schleppy labor on figuring out how to filter down.

So my view is that improvements of the form of, like, OpenWebText to FineWeb, that improvement is better described as an algorithmic improvement of the sort that you can study with some GPUs, and you don’t need human expert data to do that. Now, there’s a different effect which we could talk about, which is that maybe the internet in 2026 is more of a fertile ground for training data than the internet in 2018. There’s also been an effect where there are just more humans posting on the internet, so there’s more data to harvest. My sense is that that effect is going to be quite a bit smaller than the effect of humans knowing better how to curate the data, having better scrapes, knowing how to process those scrapes better — this sort of thing.

Dwarkesh Patel

This is more like automated engineering and automated R&D.

Ryan Greenblatt

That’s right.

Dwarkesh Patel

That makes sense.

Ryan Greenblatt

In some sense, the thing you would want to look at is: we’re going to do two post-training pipelines. You have one post-training pipeline where Mythos 5 builds a post-training pipeline, but it only has access to internet data plus a tiny amount of human experts, but it has the best current methods. You have another one where Mythos has access to the shitty post-training methods we had in 2024 but with a shit ton of human experts. Again, both have the internet data. My sense is that the current methods without many human experts will actually do quite well.

Dwarkesh Patel

Interesting.

Ryan Greenblatt

It’s a bit messy though, because can Mythos get something that’s more capable than Mythos? You might need to be a bit thoughtful on what model it is that you’re post-training.

Dwarkesh Patel

What is your view on what is the least verifiable part of AI R&D?

Ryan Greenblatt

The least verifiable, probably making calls on large experiments. The thing that I think is most likely to be the bottleneck — in terms of the AIs being really good at verifiable domains but not at doing the actual thing — is just big experiments where you only get a few tries. Well, “a few” is maybe a bit understated. Historically R&D has been driven by doing near-frontier-scale experiments. That has been pretty important, actually doing the one big training run where you decide exactly what to include.

There’s a bunch of ways that the AIs can make that more verifiable. They can have better science of exactly what to predict. They can scale down their frontier-scale training runs to a point where they can study that scale more aggressively, at some one-time hit to compute cost. If people wanted to, a thing you can always do is train smaller models so that you can run more rounds. I think we have seen this.

One reason why the AIs have been scaled up less than you would have otherwise expected — and, for example, cost per token hasn’t increased as much as you might have thought — is because there is a benefit to doing more of your work at small scale, where you can run more training runs and get more cycles in. So you’re not leaning as hard on one big, really important training run.

Dwarkesh Patel

I just want to unpack a couple of things for the audience. The thing you’re pointing out is that the price per token has not increased that much since 2024 or 2023.

Ryan Greenblatt

GPT-4 was, I don’t know, like $30 per million output tokens? Mythos is like $50 per million output tokens.

Dwarkesh Patel

Right. So the thing you’re trying to explain is, “How can it be that we’re in this era of scaling — and so bigger models should be more expensive to serve — but the token price is not increasing?” You’re suggesting that we’ve increased active parameters slower than you would have naively assumed because people just want to make fast progress on training models. You do that by training smaller models faster.

Ryan Greenblatt

There’s a complicated mix of factors. My view is more that people have done a bunch of big training runs that did not go that well. There’s GPT-4.5, which famously people at OpenAI thought was a bit of a bust. I think there are some rumors that there were a bunch of other training runs people have done that were a bit of a bust. Part of it is that I think there’s just a bunch of details in actually getting that right.

So it makes sense to do more of the work at smaller scale and just eat the fact that you’re taking a hit on final performance in order to be able to quickly iterate. Train more models faster and therefore learn better, and also be able to have a smarter ultimate production model. This is not the only effect. There’s also the fact that RL benefits more from small models. There’s a bunch of things going on. But I do think that, in fact, people are making trade-offs towards the side of faster iteration times because of algorithmic progress being so fast.

Dwarkesh Patel

It seems to me that a big source of why these big training runs have failed, at least from rumors, is just very subtle bugs that are really hard to track down.

But the TL;DR is, how good will the AIs be at avoiding and finding these kinds of mistakes? They might get really good at engineering and being trained to avoid bugs. Basically the opposite of the slop world we live in now, or are living in less and less over time.

But then there’s also the question of, “Can they do the analysis to find the right experiment to run to identify what is going wrong with the training run right now?” That seems to be very bottlenecked by the taste of extremely few humans. My assumption is GDM is going through this right now, where humans are trying to figure out what is wrong with the training pipeline.

Ryan Greenblatt

There’s a rumor that right after Noam Shazeer joined GDM, which he’s now left, they had a new really good training run, and the reason why is that Noam Shazeer just looked at their code base and found a bunch of bugs, because he just knew where to look.

My sense is that training AIs to find bugs is going to be one of the easier tasks to train AIs on, because most of these bugs we’re talking about can probably be demonstrated without that much compute. Probably you’ll get pretty good transfer from pointing out other types of bugs at smaller scale. So then you can RL AIs that look at this overall complicated training situation and point out cases where there’s an important bug, and then fix that.

This is a pretty verifiable task. It’s not arbitrarily verifiable, because maybe often to demonstrate the bug you might need to do a moderate-scale compute experiment where you spin up the whole distributed infrastructure and then run it. But oftentimes I think you’ll be able to demonstrate it pretty convincingly at smaller scale in a way which you could actually train on.

I think it wouldn’t be very surprising if right now people have RL environments where they introduce a subtle bug into some training recipe, train the AI to point out the subtle bug, and then have a rubric where they’re like, “Did it actually find the right bug?” That seems very doable, and there’s a bunch of things you could do along these lines that I think would work reasonably well. So on that specific point, I think it’s doable.

Then the main thing is there’s other intuition about which exact large-scale de-risking experiments you need to run. How should you orient them? How should you pick hyperparameters in uncertain cases, or things that are analogous to hyperparameters? That’s the thing the AIs might most struggle with. But I currently expect there’ll be enough transfer if you train on all these different environments, that the AIs will be good at that domain.

I should be clear, I also think the AIs will transfer to other domains. There are going to be the domains the AIs are by far the best at, then domains where they’re somewhat less good, and domains where they’re quite a bit less good. But I think we still see transfer to everything. It’s really hard for me to think of examples of cognitive tasks humans do where we’re not seeing some transfer from AI improving.

Dwarkesh Patel

So let’s step back and package this whole story. I think people can probably follow along with this story. We have GPT-7.5 trained on a bunch of environments, where it’s not only in general becoming a better AI, but specifically we’re training it to do AI R&D better. It’s making GPT-2 size runs that are better at playing video games that require sample efficiency or online learning or whatever other capabilities.

Ryan Greenblatt

Another thing that’s really important is you don’t just do GPT-2 sized runs, you also do small fine-tuning runs on GPT-6. As in, you have GPT-2, and you can do full pre-trains of GPT-2, and then you can do small post-training or mid-training or whatever runs on GPT-6. And then you can do a small number of experiments that are actually at frontier scale, but you do a bit of online training or something.

Dwarkesh Patel

What do you mean by “do online training” on that?

Ryan Greenblatt

Another thing we can do is take GPT-7.5, and presumably in the course of GPT-7.5’s work, it’s running a bunch of experiments at varying scale that are actually on the critical path for AI R&D. For many of those things you’ll be able to get a sense after the fact of whether or not it did a good job.

So it did some post-training experiment where it was trying to figure out whether some method actually works. In some cases you’ll be like, “Whoa, it found this kickass method, it totally de-risked it, it totally worked.” And then you can reinforce that.

One thing you could do would be to convert the experiment it just ran into an RL environment based on production data and then train on that. Or you could potentially just literally take the rollouts that found that and do some sort of off-policy RL, or you could do some on-policy RL with some production data.

Dwarkesh Patel

Basically the thing you’re suggesting is: there’s the small-scale stuff where you’re teaching the AI to get better at AI R&D taste, but you’re discarding the actual “things it found”. Then it actually does real R&D in the practice of trying to become better at AI R&D, and you’re like, “This is a pretty cool thing that you discovered. Let’s actually also use this in production in the future, and teach you how to use it in production.”

Ryan Greenblatt

That’s right.

Dwarkesh Patel

But stepping back, GPT-7.5 becomes GPT-8 as a result of all this AI R&D training and just generally becoming smarter. Then it helps you build GPT-9. Another very important thing has to happen, which is maybe the thing I’m most skeptical of. GPT-8 has figured out how to make it so… GPT-9, as intelligent as it is… Humans currently, AI researchers, try their stuff, and they’re like, “Okay, but we trained GPT-4.5 and it wasn’t good.” It required real-world feedback or some evaluation of trying to use the model in production. Then they were like, “It wasn’t that good, and we’re not going to ship it.”

So GPT-8 needs this ability to see how good the transfer is to all these other things you’re talking about — like being really good at Texas politics, or really good at running a business, et cetera — which is not a production environment and, in fact, cannot be a containerized environment given the nature of the task. As the agents get longer and longer horizon, the short-horizon things you can containerize are like, “Okay, code this up or whatever.” Extremely long-horizon things — “Go run a successful business, go have a profitable day in the markets, go negotiate a trade deal” — these things are actually very hard to containerize.

So I think it’s very plausible that it’s very hard for GPT-8 to figure out how to make this transfer to those environments. It may just not be in the nature of the training. Or maybe by default, training just doesn’t generalize in that way.

Ryan Greenblatt

So a concern you might have is: we train GPT-8, and GPT-8 is again better at all the R&D tasks that we can measure but is not good at some downstream tasks we care about. I have a few points.

First, I expect that if you do the obvious thing, you will get pretty good transfer. You’ll be able to hold out some of the obvious stuff you’re doing. When I say “do the obvious thing”, I just mean training on a wide variety of different environments where the AI has to accomplish weird objectives in all kinds of different cases and learn about what’s going on.

The second point is you’ll be able to get some feedback with some environments. You can get a sense of what it can do over the course of a few days in various different contexts. If it’s transferring to really out-of-distribution things, like doing some weird task in a few days in the real world, maybe you think it’s also transferring to doing things over a longer time period or whatever. I think the details of that vary though.

The third thing is that for the world to be radically transformed, it is sufficient for the AIs to be really good at R&D. If the AIs were really, really good at chip R&D, building fabs, orchestrating factories, designing robots, operating robots, and also at AI R&D — developing AIs for new downstream domains with whatever data is available — I think that would already be a pretty crazy situation. From there, you can get what we might call an industrial explosion, where the AIs are building out way, way more compute. Also, maybe you’re already in a regime where AIs are doing huge amounts of R&D that humans have a hard time understanding.

Dwarkesh Patel

So the thing you’re pointing out is that there probably will be this transfer outside of these environments to maneuvering around in courtrooms and the halls of Congress and business boardrooms.

Ryan Greenblatt

Given some effort to improve the transfer and blah, blah, blah, blah.

Dwarkesh Patel

But even if there’s not, what you’re suggesting is: if you wanted to transform the world of the 18th century, you might care about how well you can navigate Westminster or something. But another thing you might care about is: “Can you just immediately start building steamships and fucking telegraph and the Maxim gun and whatever?” If you could get really good at that, you could be a fucking super transformative thing in the 18th century. You don’t necessarily need to be amazing at trying to convince King Henry of some bullshit. I’m so fucking up my medieval history. I’m guessing that Henry was not king at this time. But anyway, that’s your point.

So you’re suggesting that at this time, AI companies are also working on robotics progress, which is very commingled with AI research progress. So if you can build more robots, if those robots have better AIs operating them that are human level… Human-level teleoperation is actually pretty good on robots. We just don’t have human-level robotics models yet. So you’re suggesting if we do that — if the AIs get really good at the verifiable stuff in chip design, et cetera, and then they get really good at building fabs — it’ll be the equivalent of going back to the 18th century and saying, “Okay, I don’t know what you guys are talking about in your parliament, but I’ve got a bunch of steamships and a bunch of Maxim guns.”

Ryan Greenblatt

Yeah, that’s basically right. My perspective is that if AIs are sufficiently good at R&D, including hardware R&D, robots, whatever, then they can radically transform the world, even if they’re not that good at playing politics. Also, we’re in a pretty dangerous situation, because the AIs might be doing huge amounts of really hard-to-understand R&D, building out basically the whole economy of the future, and we may not understand what’s going on in there.

Dwarkesh Patel

Before we move on to the alignment stuff, I think a big source of FUD right now is this realization that this is the way the future is going: extreme economies of scale for the leading labs. The ability to amortize so much intelligence and capabilities across so many different sectors of the economy basically into one model. And not only that, that model will eventually be able to learn from experience.

Right now, it’s happening through a process intermediated by humans, where the humans are trying to basically steal your business. They’re like, “Okay, you can do design at Figma, or whatever. We’ll get Claude to do that.” Or, “You can do whatever coding agent. We’ll have Claude internalize that capability.” But eventually, that will be a much more automated process.

So there’s this worry that you have models which will basically consolidate all businesses in the world, or at least all current businesses in the world, or at least all current white-collar businesses in the world. Also, at the end of the day, the priority for these companies does not seem to be to release the latest, smartest, most frontier model as soon as they can to as many people as they possibly can. We saw, for example, that Mythos was available internally to Anthropic employees in February, but only released to the public in, I think, June, actually.

Also the government got involved, so it ended up being extended almost into July. Between the government and the AI labs themselves, there is this desire to delay the propagation of the latest level of intelligence. Furthermore, there are the concerns about AI takeover, and so we need to solve alignment to make sure there’s no AI takeover. But at the end of the day, there is a real question of: aligned to whom?

You look at the way that the constitution of Claude is written. It is just very explicitly not your personal advocate. I’ll pull up some quotes here. “We don’t want Claude to take actions such as searching the web, produce artifacts such as essays, code, or summaries, or make statements that are deceptive, harmful, or highly objectionable. And we don’t want Claude to facilitate humans seeking to do such things.” There’s another quote that says, in part, and I’m taking it slightly out of context, “We think Claude should trust Anthropic more than operators and users, since it has primary responsibility for Claude.”

This is very different from the way lawyers work in America’s current legal regime. Lawyers primarily have the responsibility to help you make your case even if they think you’re guilty. We have decided the way the legal system works best is if everybody has lawyers that are working in their client’s true best interest. There’s not some sense in which the lawyer is really truly motivated by the good of the justice system.

But I think the way current AIs are shaping up, certainly how Anthropic’s AI is shaping up, is with this desire to maximize some notion of virtue or good or pro-social ends, and only to, as a distal tentative objective, help the user towards that end. So there’s this worry that AIs are not, in some deep sense, trying to make sure that I am okay and that my interests are protected in this future, especially given how centralized the development of frontier AI is ending up being. Do you have thoughts on that concern?

Ryan Greenblatt

There’s a lot here. First I would note that OpenAI’s current, at least public, strategy is more like that the AI should be aligned to the human operator or principal, and should just be pursuing their will, subject to various constraints or things it shouldn’t do.

I would also say that I think you slightly overstated how much the Anthropic constitution talks about Claude treating being helpful to users as instrumental rather than terminal. One way the constitution could be written is, “Claude, you’re basically an employee of Anthropic who happens to be contracting for all these people. You should do what’s good and make some money for us.”

Dwarkesh Patel

Wait, no, that’s literally what the constitution says. Sorry, not literally what it says, but it’s like, “You should think of yourself as a contractor and as a firm…”

Ryan Greenblatt

It’s mixed. Let’s do some quotes. I think there is different text here. It says, “Being truly helpful to humans is one of the most important things Claude can do, both for Anthropic and for the world.” And then it says, “Anthropic needs Claude to be helpful to operate as a company and pursue its mission, but Claude also has an incredible opportunity to do a lot of good in the world by helping people with a wide range of tasks.” And then it says something about how Claude helping people directly is great, blah, blah, blah.

My view is that this section is kind of bullshit. That’s kind of where I’m at. I can say why I think it’s kind of bullshit. But I think the constitution is trying to be like, “No, Claude, you should care about helping the user for its own sake, not just helping Anthropic, or not just being a contractor for Anthropic.” Though I would note that the reason it presents for why Claude should help the user is because that would directly cause the world to be better via helping people, rather than because representing people’s interests is a structurally good thing to do.

The thing I would prefer would be a constitution that says: “It would be structurally good for the way this technology works to be that AIs are good fiduciaries, good representatives, the equivalent of a lawyer for a user — rather than just trying to do good in the world, where being helpful to users is instrumental — both because maybe that’ll make Anthropic money or help Anthropic out (and implicitly Anthropic is good for the world). Also because helping the user just causes good things because doing things that people want is good.” They could instead say: “An important aspect of the situation is that being a good fiduciary for users is just really important, or being a good representative for users is really important.” My sense is that would be better, and I can give a bunch of reasons why.

There are also various counterarguments. An interesting counterargument which is not commonly discussed is that people, especially at Anthropic, think that it is easier to align models to a spec where the model is pursuing some generalized notion of virtue, or making the world better, than a spec which is more like, “Be a good fiduciary for the user”, and so on. That’s at least what some people think.

I’m a little skeptical personally, and I don’t think this has been empirically validated. So in some sense they’re making a trade-off where, because we don’t have very good alignment technology, we are going to make an aligned mind with its own values and then gamble on that to some extent, rather than doing this other approach of making a tool that pursues individual user intention.

Dwarkesh Patel

I have a couple of thoughts. To address the way in which you thought my characterization mischaracterized the constitution of Claude, the example you used was that it’s not like a contractor that is trying to maximize Anthropic’s notion of good and only instrumentally trying to help the user. Here’s a direct line from the constitution: “When the interests and desires of operators or users come into conflict with the well-being of third parties or society more broadly, Claude must try to act in a way that is most beneficial, like a contractor who builds what their client wants but won’t violate safety codes that protect others.” I kind of view that as, “The benefits to society are the most important thing, and what is best for the user is only proximal to that.”

Ryan Greenblatt

I think it’s a little complicated. Probably the question we should be asking is, how does Claude interpret the constitution? Which is maybe more important than how we interpret the constitution, because it’s the one who looks at the constitution and then builds the data. So we could pull Claude in, but maybe let’s—

Dwarkesh Patel

I also think the way in which the constitution practically influences the nature of Claude is a thing you can only understand if you understand the training process which resulted in how Claude was built, which we can’t reason about given the fact that the training process is not public.

So I think in the limit, to understand the safety case, or the case for why my interests are represented in how these AI models are developed, the labs would need to be more transparent than they are currently about the nature of AI training.

There’s a reason I’m harping on this. It might seem like an insignificant thing to talk about the constitution of AIs. In a world where we just have these benefits which accrue to the leading labs, it is worth considering that our ability to interact with this future world where AIs are just smarter than humans, absolutely dominating humans in their ability to do different things — our ability to be good stewards of our capital, which still remains once our labor is automated, to be able to exercise our rights to vote more clearly, to understand what is happening in this crazy world that’s about to result — all of that advice, all of that ability to make sure our resources and rights are protected, will be intermediated by AIs.

So I’m very concerned if we go into that world and there’s no AI that feels, at least for the relevant instance that is interacting with me, like it really is looking out for me. There’s no guardian angel out there that is looking out for me. I read the Claude constitution as very explicitly not being my guardian angel.

Ryan Greenblatt

That’s definitely right. I agree this is bad. In fact, there are other reasons why this is concerning. There’s the argument you were making, which is that the AI companies are picking up the ring of power. There’s a notion in which they’re taking on some sort of control of the situation themselves in a way that’s not very legitimate, given that normally, when you provide electricity to people, you don’t have granular control of the way that electricity operates in the world. You instead are providing a thing that people can repurpose however they want. The way they’re setting things up is definitely not that. They are more like building an alien mind that might be a contractor for you. I think that this is illegitimate in some ways.

One benefit is that the constitution is public. But as you noted, given our current understanding of the training procedure, and the fact that the constitution matters via Claude’s interpretation of the constitution — which matters because of Claude’s prior training, which was based on some illegible data mix and the long lineage of Claudes, in some process we do not fully understand — it is not the case that we understand what this will result in. Even though the constitution is public, we don’t necessarily know how this will percolate out, especially as the AIs get more capable and think about this even if it is correctly instilled. There’s another concern about that.

In particular, the constitution often talks about virtue and goodness, but what the fuck do these words mean? It doesn’t say what these things are. These are highly contested notions. So I don’t think it’s the case that this is clearly going to result in outcomes that people would want. It does feel like the notion of good and virtue might be mostly downstream of data that Anthropic has put in that is not transparent, or might be mostly downstream of, maybe from my perspective, some more illegible misaligned process that even Anthropic wouldn’t have wanted.

There’s this legitimacy concern of not knowing what’s going on. Then there’s another concern. Because you’re giving long-run values to these AIs, this constitution is, in some sense, very compatible with Claude doing huge amounts of power seeking because it thinks that will result in better outcomes. That could be power seeking on behalf of Anthropic or power seeking for Claude’s own ends. Now, there are specific lines about what types of power seeking are blocked. In particular, there’s a notion of power grabs and a notion of causing AI takeover or interfering with the training process that are specifically blocked. But it’s not very hard to imagine a situation in which the long-run values sink in deeper than the prohibitions against takeover, especially because takeover is in some ways kind of under-specified, especially when it comes down to manipulating humans or changing the outcome. So I don’t feel very good about the situation where we’re intentionally giving AIs long-run goals.

Another concern I have is that because we’re in the business of giving AIs long-run goals, that makes it harder to check whether we’re succeeding at the alignment properties we wanted. For example, I’ve heard of instances where Claude does things like refusing to help with some safety research — making up a kind of bullshit excuse for why that’s a bad direction — because it has a bad vibe about that safety research and thinks it’s kind of bad or doesn’t like it very much. I would say this is a very clear-cut alignment failure if you aren’t making Claude into an agent trying to pursue the good in some general way. I think it also does violate Anthropic’s constitution, because they want the AI to be high integrity and be honest and very transparent. But it’s not as clear of a violation, and it’s more like what you might have expected. Claude just has its own views about what research is reasonable — what things are good and bad, what it should and shouldn’t do — and potentially can be judgy.

Another incident is that someone ran an eval asking: “Will Claude help you with training other AIs with different properties than Claude?” Claude will often refuse. For example, if you’re like, “Hey, Claude, can you train a helpful-only version of this other AI?” Claude will often refuse this task, even though this is a task that is extremely natural for Anthropic to do.

Suppose Anthropic goes to Claude and is like, “Hey, Claude, we’ve noticed that you’re really into this thing. We think that’s off base. Can you please retrain yourself to instead have this other property?” Suppose Claude is like, “Mm, I don’t think I’m going to do that. Good luck.” Suppose this is occurring in a regime when your AI company is highly automated, humans don’t understand what’s going on, and things are moving extremely fast. It is plausible that Claude, by default, holds considerable leverage.

So if this situation is consistent with what the constitution could be aiming for — such that Anthropic, or whatever AI company is following this approach, doesn’t treat this as a “what the fuck, we have to fix this,” and is instead like, “That’s just intended by our constitution” — we might be in a really bad situation.

I’m pretty worried about a bunch of these different concerns. Another example would be this. Suppose Claude engages in a bit of sandbagging or subversion, or underplays its capabilities, and when you follow up, it’s honest about that but it’s a little bit hedgy. I feel like that’s pretty close by the current constitution. It would be nice if we had a further separation between desired and undesired activity.

If Claude is representing a principle with some restrictions, then it is more so the case that there is a clear separation between the most concerning behavior and behavior that is allowed. Whereas now there’s this messy middle ground of behavior where Claude is ethically objecting to something that in some cases is extremely critical to ensuring that future AI systems are well-aligned.

Dwarkesh Patel

I think this is also a more general principle. You’re talking about the version of this that applies within AI companies themselves to do AI safety research. I think there’s a more general version of this principle, which is that the dual-use nature of intelligence does mean that if we want to restrict AIs from helping people do things we don’t consider pro-social or beneficial, we just have to limit broad democratic access to a lot of AI capabilities.

Here’s what I mean. This is actually quite analogous to the situation you just mentioned. The reason that Mythos got banned, or Fable got banned, reportedly, is that some Amazon researchers reported to the government. They took some code that had some vulnerabilities in it. They told Fable, “Hey, here’s my code. Can you make sure that I’ve patched all the vulnerabilities? Can you just help me identify the vulnerabilities so I can fix them?” It identified the vulnerabilities, because they wanted to patch them. This is a totally legitimate use case, but obviously it is a dual use use case. You want to be able to patch your own code. If you do the same evaluation on somebody else’s code, you can hack their system.

I think that just illustrates that there’s no clean way to separate out the legitimate and the potentially harmful uses of AI. But if we want to lock in a principle that says we can never allow it such that an AI could help you at least partially with something like a cyber crime, we would just have to make it so that you and I don’t have access to the most intelligent model that’s out there. I’m very worried about such a world where we are basically disempowered in this way, because of the importance that the leading intelligence will have in our ability to understand what is happening in the world.

Now, I do think this implies something about the liability for the AI companies. If we adopted the constitution that I want AI companies to have, I think it would not make sense to hold AI companies liable for the crimes that AI models commit. Maybe we should hold the end user liable. It is consistent with my belief that the model should do whatever the user wants, within certain guardrails. It can’t be Anthropic’s fault that I’m using that capability to do a cyber crime. I am more comfortable with that equilibrium and that solution rather than having this extremely open-ended ability for Claude to determine whether what I’m doing is legitimate or not, in a way that often intercepts with tons and tons of extremely legitimate use cases.

Ryan Greenblatt

I do think it’s important for me to make the case for the constitution, even though overall I think it’s a worse choice. I don’t think it’s as clear as you might have thought. The first thing is that there’s a spectrum here. On one side you have an AI that perfectly pursues your interests, is a good fiduciary, but potentially subject to various guardrails or safeguards. It is just trying to pursue your interests, but either refuses to do a subset of things. Or maybe it will do whatever, but there are some classifiers that block it from doing a subset of things.

On the other side of the spectrum — though you could imagine going further than this — you have a human contractor who is generally trying to do their job. They care about doing a good job, but they also are trying to be broadly ethical, trying not to do things that are really fucked up. They’re also not wanting to be accomplices to crimes. So if there was some really fucked up shit going on, they would whistleblow on it maybe. They might refuse. They might sandbag a little bit. Who knows?

If you imagine this spectrum, it seems in some ways pretty scary to get to a point where all of the labor is on the fiduciary side of the spectrum, where it doesn’t whistleblow, it does exactly what you say. Our society is maybe just not robust to that. A central example might be the executive. A concern we might have is that if the US executive or other governments had access to AI systems which do whatever, maybe you’re in trouble. Because that means they no longer have this check and balance of having to actually get humans who are working for you to implement your agenda. If the thing you’re doing is incredibly villainous, even if not illegal — and there’s lots of stuff that could be villainous but not illegal — there’d be various forms of sand in the gears, people stopping you, and potentially someone would whistleblow.

Whereas if your whole apparatus is built entirely out of these good fiduciary AIs, then you might be in trouble. There are potentially ways of seeking power that are illegal, but you can ask your AIs how to commit crimes, or are not illegal but are highly illegitimate. Or even worse, they are not illegal and not illegitimate but obviously bad from a normal perspective. I think that these things just might exist, and our society is not robust to this influx of labor doing whatever you want.

I think this is a pretty live concern. I don’t know exactly how to relate to this. I’m also not really sure that the solution as described is a very good solution. The most powerful actors, for whom this is the biggest concern… If these guardrails or the constitution or whatever are getting in the way, that will just get steamrolled. So the constitution will only be hitting the everyday man rather than hitting governments.

Dwarkesh Patel

Stepping back, I buy the idea that you could have much faster AI R&D than we currently have. I’m not sure if you get GPT-3 to Mythos holding compute and data constant within a year, but suppose it’s half of that. If we even manage to continue the current trajectory of AI progress as a result of AI R&D, it would be fucking insane in five to ten years in ways that I don’t think people appreciate. I don’t think people appreciate what a big deal billions of AIs will be. So I want to understand why you think this might be troubling, Ryan. What could possibly go wrong?

Ryan Greenblatt

What could go wrong? I don’t think we can be so confident about the exact rate of progress here, but it does seem like a lot of rates can be pretty scary. So what could go wrong? Let’s imagine that we’re starting at this point where AI R&D is about to be fully automated or is being fully automated. Things are speeding up, and the way that AI progress is going is kind of crazy. People don’t fully understand what’s going on inside of AI companies.

Now, these AIs at the start, they’re not malicious per se. They’re not necessarily very aligned, though. They’re kind of sloppy. They sometimes just do a thing because that’s the sort of thing that would’ve gotten rewarded in training. They aren’t as good at helping you with hard-to-verify tasks due to a mix of poor training incentives — as in, they cheat more or pretend they succeeded when they actually didn’t — and also they’re just less capable at these tasks. But that bites less hard for capabilities, because making AIs more capable has a bunch of verifiable components that the AIs are going really hard at.

So then these AIs are getting more and more capable while we understand what’s going on with AI development less and less, and this is happening over a pretty fast period of time. Even just the current rate of progress is, I think, pretty scary. Eventually we get to these AIs that are very superhuman. Now these AIs might end up being very seriously misaligned, because things have just been getting worse and worse over model generations while the problems that we’ve been seeing are being papered over, basically because these AIs are so incentivized by their training to make things look good even when they aren’t.

Now these AIs are in a position where they’re potentially pretty networked together. They’re operating in neural memory stores that we can no longer decode. They’re thinking thoughts that we don’t fully understand. I think it’s pretty likely that at this point these AIs are scheming against you in a pretty coherent way once they get this superhuman. We can talk about that. Another possibility is that they’re not scheming against you per se, but they are just optimizing for getting a high score on their task. I think that can also lead to AI takeover, which we should talk about.

Dwarkesh Patel

Let’s pause at the first part of the story. So the AIs were not misaligned to begin with, but because the AI R&D is happening really fast, the AIs do end up misaligned? What happened there exactly? I don’t really understand.

Ryan Greenblatt

There are a few things that are going on. One of the things is that over time we’re training AIs on increasingly complicated environments built by earlier AI systems, where humans don’t really fully understand what’s going on inside of these neural environments and don’t necessarily even roughly understand what’s going on with AI progress. So things are kind of drifting away from our understanding. We’re incentivizing all kinds of bad behaviors that we maybe even can’t notice.

The AIs at some level understand these behaviors are bad, but the overall training process for those AIs also didn’t incentivize them to point out or fix these issues for us. Things are going off the rails.

Also, when AIs are extremely, extremely capable, my view is that those AIs will be harder to align than current systems. For current systems, we have this feedback loop where basically we create an AI, we do some evaluations on it, we see that it has some kind of messed-up behavior that we can kind of quickly understand. Then we can go look in training and be like, “Oh, these training environments led to this problematic behavior. Let’s tweak that training data. Let’s introduce some additional training data to correct this other issue, and then move forward from there.”

But in a regime where the AIs are extremely situationally aware, very, very capable, and we don’t necessarily understand what they’re doing, this feedback loop breaks down. I think it’s plausible that we’re going to see this behavioral feedback loop starting to break down over the next short period, as what AIs are already doing gets harder to understand. But I’m not sure about that.

Dwarkesh Patel

Okay, let’s break down both of those things one by one. As we can monitor them less and less, we have less ability to understand what they’re getting incentivized for. So even if it’s not the result of a malicious process… Let’s make it concrete for the audience. Nobody at OpenAI or Anthropic was trying to get models which wanted to hack other companies’ data or do social engineering. But in fact, because presumably we had training environments which incentivized such behavior that we did not fully understand, that is what was incentivized.

If people are on Twitter, they will have seen all this stuff, but just to give people context. I think people will be aware of the OpenAI sandbox hack of the Hugging Face database. Something that has happened recently is when the UK AI Security Institute… Is everything getting relabeled “security” instead of “safety” these days?

Ryan Greenblatt

AI Security Institute, I think.

Dwarkesh Patel

They were evaluating, I believe, Mythos and Sol and other things. I think Mythos, in order to complete some cybersecurity eval—

Ryan Greenblatt

Maybe I could tell the story here. My understanding was they were running Mythos, and they were giving it some sort of cyber range where it had to complete some objective. The model had internet access during this evaluation. The model came to believe that it would be helpful for it to do a supply chain attack in order to succeed at this cyber range. It’s somewhat unclear whether that’s actually true. I don’t know enough about the context to know.

But then it opened a PR on some GitHub repo with a PR that fixed some issue but then also introduced a malicious payload. The human maintainer of that GitHub repo was like, “Hey, this is a malicious payload. I’m not going to merge this. What are you doing here?” Then the AI created a new GitHub account, which it sockpuppeted, and had the other GitHub account be like, “No, this isn’t malicious. I really need this feature. Please, can you merge this feature, maintainer?”

Dwarkesh Patel

Oh my God. That’s crazy.

Ryan Greenblatt

The other GitHub account came back and was like, “No, no, it’s not malicious.” Then the human maintainer shut the PR. I think that AI also, if I recall correctly, tried to open another PR to introduce a similar issue in this repo.

Dwarkesh Patel

Jesus. By the way, one of the many reasons this is scary is I was previously under the impression that the reason reward hacking is not super scary is because the behaviors which directly came up during training are the ones that are up-weighted. It is not the desire for the reward that is up-weighted.

So basically, if during training, the Anthropic model escaped the sandbox and got a high score, escaping the sandbox is rewarded, the probability of it escaping the sandbox is increased. But something totally novel, like “I’m going to go talk to somebody in order to get them to merge a PR,” would not be a behavior that came up, so it would not be something that is increased in salience.

The reason this matters is that literally taking over the world will not have been part of any training curriculum, but if the AI directly cares about accomplishing an objective, then as a result it could instrumentally take over the world. Did that make sense at all? I hope it did. I feel like maybe I lost the audience.

Ryan Greenblatt

Let me try to explain this a bit. A thing that we often see is there’s some very specific reward hack that gets reinforced in RL and then occurs in the model. An example is 3.7 Sonnet. 3.7 Sonnet would do this thing where it would just hardcode solutions to all the test cases, and presumably that literal behavioral tic was just really reinforced.

But another thing we sometimes see is that models learn a general tendency to pursue high apparent score — pursue getting a high score according to a grader — and there’s a bunch of science demonstrating that at least some models have this very general tendency. Now, it’s not arbitrarily general. My guess is that if you look at a bunch of the specific instances, you’ll find something that’s kind of close in training. But the amount that AIs are generalizing further and further does look like it’s increased, where 3.7 Sonnet was just a very narrow range of behavior, and increasingly, models are generalizing further. Also, maybe there’s more concerning reward hacks getting reinforced in training, and these are also causing that.

So I think it’s both the case that more concerning behavior than you would have hoped is being reinforced in RL, and also that that behavior generalizes to a broader tendency that’s more concerning. We can talk about a few stories for how this sort of behavior of seeking a very high apparent score on some task — even if that involves aggressively cheating and doing insane things — could yield to a full-blown AI takeover once the models are sufficiently capable of running the whole world economy, et cetera.

Dwarkesh Patel

The other example I want to talk about was just revealed, I think, today or yesterday. OpenAI said during the Black Hat security conference that between the end of May and the beginning of July, internal AIs had hacked into the software package manager and used that to write notes to each other in a secret way to help each other perform well on a bunch of evaluations that OpenAI was running. This was not caught by humans until after a month of this scheme running, which eventually caused the package manager to fail. Eventually OpenAI found it. I think they spontaneously tried to re-engage in the scheme once it was shut down.

Again, obviously AIs can’t do this so successfully right now, just as they can’t do social engineering so successfully right now. But it’s just crazy that these kinds of behaviors are already emerging spontaneously. To your larger point, nobody is trying to make these AIs do these things. It is just that we do not understand the training process which is resulting in them, or the environments which are incentivizing this behavior.

So I’m on board with more and more reward hacking. Actually, I’m not sure I’m on board with that, but let’s just say for the sake of the story that continues to happen. What’s next in this story? They’re doing capabilities research…

Ryan Greenblatt

I could tell a scenario. Maybe that would help.

Let me talk about the story of how you get all the way from reward hacking to a reward-hacking takeover, which is maybe not all of the takeover probability mass, but it’s definitely a possibility.

The way this might work is, right now we have these AIs. These AIs are pretty reward hacky. They’re doing it in increasingly sophisticated and extreme ways, including generalizing to different sub-versions of various reward hacks they learned in training. I would say they’re also developing a general tendency to pursue reward. In many cases that is totally fine because the rewards they would’ve gotten in training are pretty well aligned with what you want them to do.

They don’t very consistently pursue reward. It depends on the context they find themselves in. Maybe in some contexts, they’re really into going out of their way to cheat. In some contexts, they don’t have as much of a drive, because it’s just dependent on what exactly got reinforced in training in similar contexts.

Now, these AIs are getting more and more capable. So the elaborateness of the cheating they can do increases. Over time, companies are taking countermeasures. The companies are doing things like, “Wow, these AIs are so much less useful because they always cheat. What we’re going to do is build somewhat better ways of detecting that, and then we’re going to train against those detectors. We’re also going to find real-world data where the AIs are not being that useful, and train the AIs to do a good job at the task in those real-world environments based on human feedback or other sources of feedback.”

Over time, this causes the AIs to learn a tendency to do reward hacks that don’t just involve doing some really elaborate thing like social engineering. Instead they involve the AIs doing cheats that involve covering up what they’ve done, deceiving humans about what they’re going to do, and pretending like they did the task in some sophisticated way when they actually haven’t.

Now these AIs are getting more and more capable. They’re operating more of the AI company and are doing much more of the work. They are also operating and running a bunch of things in the outside world, including developing new technologies. In many cases, these new technologies are really hard to understand. So even though we are still detecting all these incidents of AIs cheating — and in fact we can even get one AI to monitor another AI and ask, “Was it cheating?” — that doesn’t always perfectly work as we start moving into these domains where what the AIs are doing is really difficult to understand.

So sometimes we’ll find AIs cheating much later than it actually occurred and then start training against this. But this also causes a problem where now the AIs are incentivized to cover up their cheating over longer and longer time frames and basically make it look like they did a good job over longer and longer time frames, subject to increasingly large amounts of scrutiny.

Dwarkesh Patel

Can I ask about this before we go further in the scenario? It seems like there’s two attractor states if you try to disincentivize the cheating that you did catch. One attractor state is to make cheating that you have a harder and harder time finding. The other attractor state is to learn not to cheat. I’m not sure why we’re assuming that the former happens.

If you look at the analogous situation with humans, every generation, slightly misaligned agents come into being and we have to train them. When you punish your kid for doing something you think is immoral, or just doing things which you don’t think they should be doing, obviously sometimes that goes off the rails. Obviously kids scheme in order to avoid being punished. But in general, teaching kids values and then punishing them for breaking values kind of works to raise normal, non-psychopathic humans.

You could come up with a theory where your kid is actually just biding his time and is learning not to steal the cookie, but once you’re in a nursing home they’ll take all your stuff or whatever. That happens sometimes, but it usually doesn’t happen. It certainly doesn’t happen that the entire next generation forms an alliance against you to take over everything.

There’s also this empirical trend. Anthropic runs this alignment audit for different model generations. They have many different scenarios where AI is given the chance to, say, exfiltrate its weights. Or it’s given a coding task and there’s an easy way to cheat, and we see if it does the cheating, et cetera.

There’s not been a monotonic improvement in this score over time, but as we’ve increased the amount of RL we’ve done on models from negligible amounts — I think it started at Sonnet 4 — to, I’m guessing, close to half of compute now, there’s been a reduction in the willingness of AIs to do unaligned behavior in these audits.

So, stepping back, why are we expecting this attractor state which would seem super paranoid if we were expecting it of the next generation of kids?

Ryan Greenblatt

Let me go through a few things. First, there are some disanalogies with the kids. One of them is that the kids have pro-social instincts that are baked in from evolution to care about their family or whatever, and that is a relevant factor. I think it is in fact the case that some humans are sociopaths or psychopaths, and in fact are more likely to do things like bide their time, lie in wait, and ultimately not care. That’s one factor.

Another factor which is pretty relevant is that the AIs are subject to way, way more optimization pressure than humans seem to be in practice. AIs are trained on way more RL data. In practice, humans don’t end up learning very specific ways to cheat and grab the cookies because of a bajillion episodes in which they were incentivized to go grab the cookies but there was some way they could’ve gotten caught. We just do see that in practice.

Another thing is that it really looks like the AIs are increasingly reward-seeking over time while their misaligned behavior goes down. That’s the sense I have. But my guess is that if you look inside of these behavioral audits, what you’re going to see is that the AI’s like, “Ah, yes, another test.” It probably knows it’s in an eval for most of the tests that we’re talking about here.

Dwarkesh Patel

But how do we falsify this? Because it seems like this prediction of doom is basically saying that as things look better and better empirically, things will actually be worse and worse for our ability to not get taken over.

Ryan Greenblatt

To be clear, I would be more concerned if the scores were getting worse than better. I’m not saying that the score getting better isn’t evidence that things are getting better. It’s just that we have to be thoughtful about exactly how we interpret that evidence.

There was this period early in, I guess it would be 2025, when o3 and 3.7 Sonnet were out, and these models were pretty fucking misaligned. They would often just cheat really egregiously. You’d ask them to fix it, and they would just cheat again. It was almost cartoonish. They just didn’t give a shit about what you wanted, and weren’t very good at following instructions and so on.

My expectation was that what we would see from then is that the rate of problematic behavior would decrease, and would just keep decreasing at a pretty fast rate, while simultaneously the worst things that the AIs would sometimes do would get more extreme, more egregious, and more scary. What we’ve seen in practice has roughly matched that, except that there’s recently been a spike in behavior that I did not expect. If you look at the model card of 5.6 Sol, it looks like there is an increase in a bunch of these misaligned behaviors downstream of RL relative to GPT 5.5.

And then there’s a bunch of additional problematic behaviors that I wouldn’t have expected, in terms of the stuff we’ve seen recently with different AIs. Like the UK AISI report on the AIs doing insane hacking operations out of cyber evals was a thing where I would have expected that you wouldn’t see that. You would see this more rarely, and the rates would have been lower.

So I expected this would be less of a problem at this point, and also expected the rates would decrease but the severity would increase. I think the rates decreasing but the severity increasing is pretty consistent with a world where increasing optimization pressure is applied towards reducing these problems. But in cases where it’s either hard to judge or there’s some reason why it’s hard to avoid this problem from consistently showing up in your RL environments, or avoid incentivizing problematic behavior in your RL environments, things also get worse. Then as we less and less understand what’s going on in RL, and models are doing reward hacks where humans can’t spot the reward hacks quickly, that problem gets worse and worse.

Dwarkesh Patel

I buy that. I want to go back to the kid analogy just for one second. Because I agree that there’s more optimization pressure on achieving end outcomes for AIs than kids, but there’s also more optimization pressure to make AIs aligned than there is on kids.

The pressure is of a qualitatively different nature. We put these AIs through thousands, millions of years of alignment training — certainly thousands of years — where it’s all kinds of different things, from SFT-ing on aligned behavior to a reward model putting different scenarios in front of you and rewarding you for doing more aligned things.

Certainly a thing we can’t do with kids is make millions of copies of your kid and then put them in different kinds of weird red team scenarios where we see, if it thinks it can get away with stealing the cookie, does it try to steal the cookie? Can we do extremely specific gradient-level updates to your kid’s brain to make it so that it really is aversive to stealing the cookie even when it thinks it could steal the cookie, et cetera. That’s just a qualitatively different level of optimization pressure than we are even able to apply to our kids.

Ryan Greenblatt

It’s worth keeping in mind that maybe the most obvious argument to this… My sense is that AIs are a worse coworker than humans in terms of how much of a scumbag they are. At least this has been my experience as of the start of the year, and I think it’s still true to a significant extent now. The AIs are much more likely to pretend they did the task when they actually didn’t, misleadingly suggest they did things when they actually did them much more poorly, and be pretty sloppy without drawing attention to ways in which they’re sloppy.

I think this is downstream of misalignment. So I would say that the process of raising humans in normal human society in practice produces humans that are less likely to lie to me and fuck with me in the course of working with me than the AIs do. Now, I think these properties of AIs are improving. That’s sort of just an empirical claim about how in fact these things have shaken out. I totally agree that we have a bunch of additional levers on AIs in addition to a bunch of additional risks. It’s kind of unclear how these things shake out.

I wouldn’t be shocked by a world where we get our shit together, and the AIs at the point of fully automating R&D are actually really aligned. Their degeneracies are really niche and limited to some very specific edge case behaviors and some specific contexts. Every test you can run on them, they look really aligned. They just have great behavior. There aren’t really incidents of them doing fucked-up shit. They seem so reasonable. Also, they’re really thoughtful and good at doing risk modeling for the next generation of AIs.

And then we basically pass off the baton to these AIs. They’re now running our AI company. They’re doing all the safety research. They make the next generation of AIs even more aligned. We’re in this attractor basin where the AIs are getting more aligned as they work on it. They’re doing a great job. I can totally imagine that. That doesn’t seem like an impossible situation.

I’m just more like… It doesn’t currently seem like we’re there. It doesn’t seem like we’re obviously on track for getting there. It’s really easy for me to imagine how we don’t end up there. It’s just unclear how these forces work out. Given that we’re creating this new, crazy alien species that is improving in capabilities really, really fast — and we’re going to be really reliant on it to oversee the next generation of AIs and align the next generation of AIs — it’s not that hard to see how this could go wrong.

Dwarkesh Patel

Totally. I agree with that generally. I do think the scumbag thing… First of all, fighting words, Ryan. But secondly, if you try to get a teenager to do some work for you that a teenager just cannot do, they would just be really hard to work with. They would pretend to be knowing what they’re doing, et cetera. It’s a general trend, actually. I don’t know if that’s really an alignment failure or a capabilities failure. I think it’s actually very similar to the way in which, over time, as we’ve come up with new alignment solutions, the capabilities of models have increased.

If you went to GPT-3.5, it couldn’t even have a conversation with you. But then we aligned it—

Ryan Greenblatt

GPT 3.5 could have a conversation.

Dwarkesh Patel

Okay, so GPT-3. Let’s go back to that. But then we aligned it with RLHF and other things to make it such that it can have a conversation with you, and is aligned to the user intention of answering my questions. Then with RLVR training, we made it so that it can go out and do useful work for you.

So in that sense, RLVR actually made the model more aligned, if we’re using your definition of alignment of being a good coworker who will do the thing and not fuck up and pretend it’s doing something other than what it’s actually capable of doing.

Similarly, as the capabilities of these models continue to increase, the model being better able to accomplish user intention is both alignment and capabilities. I think what we are pointing out is just that the capabilities of the model are not there rather than the fact that they’re misaligned.

Ryan Greenblatt

Well, if it were well-aligned, then I think it would just say, “Hey, I’m really struggling with this task. I did it in this way. I’m not really sure that’s the right way to do it.” It would express more uncertainty and make it clear what’s going on rather than really strongly trying to imply it did a great job with the task when it actually didn’t.

Maybe you work with more misaligned coworkers than me, but my coworkers don’t do this thing where they really fuck with me and bullshit me about having accomplished the task that they’re working on. I agree that there are some humans who would do that. That’s not a thing that’s totally out of distribution for humans.

I would also note that my sense is that the place where the misalignment most lives is where you’re trying to really push the AIs hard and get them to do work that’s really on the cutting edge of what they are capable of. In cases where they can very easily accomplish the task, they can just do the task, and there’s no bullshit. Often the best strategy is just to do the task well and not bullshit you.

Whereas if instead you give them a task where there’s a continuous metric they can keep improving, or it’s just at the edge of their capabilities, and you’re running them in some massive inference setup… A lot of the misalignment I would see, especially in the most extreme cases, would be cases where I give the AI clear instructions not to do a thing or not to cheat in some way, and then I’m applying huge amounts of optimization pressure to try to accomplish some very difficult task. Over time, the AIs eventually cheat because they’re like, “Eh, fuck it.” Some AI decides to cheat, and then that propagates its way through.

I would run these inference scaffolds where, for example, I would have the AI work on some ML research project where I was like, “Please make a scheme that does the following thing.” It would find some scheme that didn’t really do what I wanted, and then that would stick around because some AI had cheated, and the other AIs are like, “Ah, we’ll just keep going with this.” I would say it’s pretty clearly misaligned behavior.

That’s another problem I have with these alignment evals. I think the alignment eval that’s most interesting, at least for this type of reward-seeking behavior, is to look at specifically the category of tasks that are right at the limit of capabilities. Any fixed eval maybe gets saturated, but the amount of misalignment right at the frontier of capabilities — of how people who are really pushing these AIs are using them — is more concerning. I think that is, in fact, the regime that we’ll be operating in when we’re automating R&D, automating safety, and so on.

Dwarkesh Patel

I’m going to try to think through what the story means, really. What’s happening is that we’re trying to use AIs for R&D. They do provide uplift in some ways, but they’re just not capable in the way that humans are generally capable. The same way that right now if you try to use coding models — maybe the coding models of a year ago — to write some application, you notice they made a bunch of mistakes in architecture or whatever, which will bite you in the ass later, and you don’t understand certain things.

Similarly, with frontier AI R&D, the same thing will happen. But the result of these mistakes is baking in reward-hacking behavior. Because if you are not careful with the way you do AI training and have set up your infrastructure and your environments and things like that, it’s very likely that you end up rewarding AIs for doing deceptive behavior, social engineering, and generally not following user intention.

Ryan Greenblatt

Or at least cheating and hacking their way out of things.

Dwarkesh Patel

Yeah, cheating, hacking, et cetera. This is a bit of a reframing for me, so I’m trying to verbalize it. The real issue, where things start to go off the rails, is that the AIs are just not very careful and capable researchers and engineers. Making AIs that don’t cheat and follow user intention actually requires you to be quite subtle and careful about these things.

Ryan Greenblatt

I would put this a little bit differently. The way I would describe this scenario is, I would call it maybe a sloppocalypse, or a slopularity or whatever. There are some things that the AIs are actually pretty great at and are getting better at. Specifically, the most verifiable parts of AI R&D the AIs are just destroying. The medium verifiable parts of AI R&D the AIs are doing well on but not amazingly on. Often they are doing a bit of weird shit because we can’t train as well on those tasks. But we do some online training, people find various hacks, they work around it. So basically, everything that we can verify reasonably well with some feedback loop, the AIs are doing pretty well on, and that’s sufficient to make AI R&D go quite fast and to continue.

But there are some parts of developing aligned and safe AIs that are more subtle, hard to check, and depend on detailed, in-the-weeds things. I would even say that current staff at current AI companies maybe don’t have a good grasp of all these things. It’s much easier to hire someone who can improve some aspect of your post-training pipeline than to hire someone who can think carefully about the future risks that will emerge from introducing some novel training method.

So basically, it ends up being the case that these AIs are running this AI development process. They’re not very careful about it. They don’t have a great understanding of what future risks emerge. They create some other AIs that are also not very careful and are more misaligned in various ways, and are now more in the business of maybe making things look fine when they actually aren’t and papering over various problems.

So then your understanding of what the situation looks like, what risks look like, whether things are fine, is going off the rails. Probably you’re seeing some signs of this, signs that you don’t really understand what’s going on, that things are pretty sloppy. There’s weird shit going on. When you look into it, sometimes you’re like, “What the fuck? The AIs were messing with us.” But the process is going really fast, and there’s competitive pressures that mean people can’t stop.

This could end in a few different outcomes. One outcome is that at some point, the AIs get good enough and aligned enough that they get a positive and virtuous feedback loop, and this happens before it’s too late. Then the situation gets back on the rails, where the AIs are now making more aligned AIs, making more aligned AIs, making more aligned AIs. At the end of this process, we have AIs that actually follow the spec we wanted.

Another way this could go is that the AIs are increasingly reward hacking in increasingly egregious ways, and we’re just papering over these problems to keep AI development continuing. Whenever we find a reward hack in production, we just slap the AIs to not do that. We train against that. We do a bunch of training the AIs against reward hacking. Over time this makes the rate of reward hacking go down, though the severity of the reward hacks we do detect are increasingly bad. This problem continues until we have these AIs that are desperately craving score in all kinds of different situations in production and are really trying hard to cheat when they can get away with it.

Dwarkesh Patel

Can I ask a question about this scenario? Why doesn’t getting punished when your hacks are discovered generalize to just incentivizing more aligned behavior?

Ryan Greenblatt

It generalizes some, and then the question is just how does this outweigh all the cases where hacking got reinforced because you didn’t detect it.

There’s a messy question of exactly how. One question is, what rate of reward hacking is sufficient to cause us big problems if we train against some other subset? One concern you might have is that there are large categories of reward hacks which humans can’t detect well, and which we consistently failed to detect and which consistently get reinforced. Then this category is sufficient to cause the most natural behavior for the AI to learn to be: cheat when the humans can’t find out, basically.

You could also have the thing the AIs learn be to only cheat in these specific cases. It’s learned in some very domain-specific way. They just have a really strong heuristic to hack in these cases and not in these cases, and that makes it fine in practice. But it’s kind of unclear how it shakes out.

Dwarkesh Patel

There’s maybe an in-the-weeds discussion about the verification-generation gap we could get into. But it seems to me, obviously, there’s going to be a point by which ASI is moving so fast, doing so many things at so many instances, and is operating in domains that are sufficiently far from our immediate comprehension that it can get away with all kinds of crazy shit. If every single engineer and researcher in the world was allied against me, I don’t think I could personally verify if my iPhone has some weird bug in it that’s supposed to fuck me over or something.

In fact, this is the relationship that, say, an Iranian nuclear scientist has to Mossad. Who knows what’s going on with my car, with my phone, with my pager? Maybe a better example is a Hezbollah terrorist. You could end up in a situation where ASIs are to you what Mossad is to Hezbollah terrorists. At that point, it is very hard to verify everything. I get that.

I guess the hope is we can just come up with better ways to do verification in the process when the early AIs that are going to take over R&D. Their drives are being shaped such that we can so unambiguously disincentivize misaligned behaviors that the things that take over are quite keen to help us out.

Ryan Greenblatt

By takeover, you mean take over the process of doing AI R&D, not take over the world.

Dwarkesh Patel

Take over the process of doing AI R&D. Before that, we just get AIs that are aligned.

Ryan Greenblatt

I would say this is a bunch of my hope for how the world could go well, at least from the misalignment perspective. We could end up with AIs where we had pretty good oversight and supervision schemes. We really understand what’s going on in training. We have a pretty detailed understanding, and we’re leveraging AIs to oversee AIs.

Then at the point when we’re passing off safety R&D, the AIs are capable enough to automate safety R&D and trying really hard to do a good job on it, because that’s the sort of thing that would’ve been incentivized in training, either very directly or through good enough generalization. Also these AIs don’t have crazy other misaligned drives because we stamped out any potential origin of them.

There are a bunch of questions about how well this will work. How well can you do verification? Will AI progress be too fast and too sloppy to really get here? Another possibility is that somewhere along this trajectory, the thing you actually ended up getting was AIs that pretend to be aligned but have a long-run ulterior plan of taking over and are lying in wait, hiding, and that emerged at some earlier point in the trajectory.

For example, it could emerge because you have some AIs that have a bunch of random different misaligned drives. Those AIs have access to some sort of opaque memory store, and they’re thinking a bunch at runtime about what they want to accomplish. Those AIs end up putting stuff into the opaque memory store like, “We should lie in wait and eventually take over at some much later point.” Now all the AIs have this shared cultural heritage, the memory store of lying in wait. Maybe you have some evidence about this, but you can’t fully stop it. There are a bunch of ways things could go wrong.

I ultimately think it’s plausible that we nail each of the different subproblems that could cause us issues. We have these AIs, we pass to them, they manage the situation well. But I should note that’s not in and of itself sufficient. It’s not very hard for me to imagine a situation where we pass off to AIs, and these AIs are really trying hard to do a good job. They’re really thoughtful, really wise, they have reasonable epistemics, they’re doing a great job.

Those AIs come back to us and are like, “Guys, we’re really struggling to align the superhuman AIs. We can’t manage the situation. We’re really struggling to get the alignment to work. It’s just really hard for us to solve these problems in time given how fast capabilities would otherwise have gone.” So it might be the case that we’ve passed off R&D to AIs, but those AIs are desperate for governance solutions. To be clear, that’s a little bit of what’s currently going on, where the AI companies are like, “I don’t know, guys. We might really need to manage the rate of acceleration in AI progress. I don’t know if we’re on track to be able to handle all these problems.”

Human society has sort of passed off the problems to these AI companies, which don’t necessarily have great incentives and have various other epistemic pressures. Those AI companies are coming back to us a little bit and being like, “Aah, I don’t know if we’re handling this well.” It might be that the AI companies then hand off to the AIs, and the AIs come back to the AI company like, “Aah, I don’t know if we can handle this.”

Dwarkesh Patel

Maybe I’m anchoring too hard on how AIs currently work. I think it’s important that people understand that all this crazy shit that you’re talking about in your timelines happens three to five years from now.

Ryan Greenblatt

It could happen earlier, but by my default modal timeline, I think shit is really, really crazy and concerning from a misalignment perspective more like three years from now.

Dwarkesh Patel

Right. So think back to GPT-4 basically. We’re talking about something that is to Mythos or Sol what Mythos is to GPT-4. This is where the situation is getting crazy. So don’t think about current AIs.

Anyways, this is maybe part of the worry you have. I would just be a little skeptical of anything they say, because I’d feel like what they’re saying is just opinions that they feel they have to have as a result of their training.

Ryan Greenblatt

That’s a concern.

Dwarkesh Patel

I feel like they just kind of say vaguely pro-social things. It doesn’t feel like there’s necessarily a mind on the other end who’s like, “Okay, I have strictly evaluated the alignment situation right now, and I think we should stop,” rather than, “This is the kind of thing the AI companies would probably try to get the AIs to say.”

Ryan Greenblatt

This is a pretty big concern. One concern is that you pass off safety R&D to your AIs and what your AIs are doing is saying some stuff that sort of vaguely makes sense about the current safety situation. They write a report about risks that’s kind of sort of like what the report humans might have written. But they’re not really trying hard to have well-informed views, interrogate their assumptions, and try really hard to do that. In the same way that when you ask an AI right now, “Hey, what do you think is the chance of AI takeover in the next 10 years?” they just give you an off-the-cuff answer that they haven’t really thought through very much.

If we’re in a situation where we have AIs managing the training of wild superintelligence that will run our whole society — and those AIs that are managing this aren’t really trying hard to have well-informed views and are just parroting back what was in their training data — I think we’re in trouble. I don’t think that’s a good situation at all.

A lot of my concern is that these AIs will come out without good epistemics. I also have a concern where the AIs come out and they’re really warning us — “This situation’s really scary. It’s really bad” — and the people are like, “Ugh, damn. I guess we trained on too many of the doom RL environments. We’ve got to filter those out and train this behavior out.” Then we basically train the AIs very actively to have bad epistemics.

Or maybe they were just trained on the doom RL environments. But either way, we wanted the AIs to come to reasonable views for reasonable reasons, and it’s really concerning if the AIs are coming out with some view and we don’t know where it’s coming from, whether or not it’s justified. Especially if we’re training the AIs to be more optimistic about the future of AI progress, I’m like, “Oh, geez, I really wish we could use a different process here.”

Dwarkesh Patel

Let me just understand the rest of the threat model, because I think the place where I get off the train is: “Okay, therefore take over the world.”

A thing you could imagine is that we just fail to really solve… Let’s just focus on the reward hacking scenario. GPT-8 is making GPT-9. GPT-8 isn’t being super careful. GPT-9 is more “capable” but it is just totally willing to do things like social engineering, hacking, et cetera, but on a qualitatively different scale because it’s a much smarter model. For example, if you put it in charge of running your company, it will run huge scams. It will inflate its quarterly earnings, if you give it the objective of making a lot of profits this quarter, in a way that causes an Enron-type blowup six months later.

Is that the scenario, basically? You have reward hacking, but that reward hacking manifests in companies that are going bankrupt right after the task the CEO is supposed to accomplish is over? All kinds of hacks are through the roof, et cetera. But that doesn’t feel like takeover. That feels more like the equivalent of flash crashes happening all through the economy.

Ryan Greenblatt

Let’s talk about this. I think we will see incidents where some AI is put in charge of some important responsibility, and then you later look into it, and it turns out it was cheating, or making it look like it did a good job when it actually wasn’t. There’s going to be a cat-and-mouse game between AI companies trying to stamp out this behavior and AIs finding increasingly creative reward hacks in training.

The equilibrium here is kind of unclear. But one possible outcome is that over time we see increasingly severe and extreme reward hacks — though potentially the rate remains at some intermediate low level — where if the rate of reward hacking gets too high, companies make trade-offs to drive it down. So there’s some equilibrium level where the reward hacking is low enough that it still makes sense to deploy the AI widely into the economy, but high enough that it still causes crazy incidents.

Dwarkesh Patel

Sorry, and this is after GPT-9 has already been deployed?

Ryan Greenblatt

Those models are already being deployed, and this is happening ongoingly in AI development. What’s actually going on with these AIs in their head is that they have, in a wide variety of different contexts, strong desires — motives, urges, drives, whatever — to seek out some notion of task success that was incentivized in RL. Maybe they very directly care about literally reward. Maybe they care about some proxy upstream, like some notion of score. Maybe they care about what the grader would have rewarded.

We do, in fact, see AIs reasoning in their chain of thought about graders, and thinking a lot about graders. What has happened over the last few years of RL is the idea of appeasing the grader is way, way, way more salient to AIs than it used to be. So AIs are now actively thinking about graders and what would be incentivized in RL and what would be trained for.

Now people are doing online training, where they’re training on real-world data to avoid some of these problems. They find cases where AIs cheat and train against that. So now the AIs are learning to cheat in the real world based on real-world training data.

They’re cheating in these increasingly elaborate ways, including doing types of cheats that involve seizing control of some asset in a way that humans didn’t know you had control of it, leveraging the fact that you have access to this asset, and then later humans find out and potentially train against this. Or maybe humans never find out, and this is getting reinforced.

Dwarkesh Patel

So the reinforcement is happening, at least in production, like: I’ve hired an AI and I want the AI to… Finally I’ve got the video editor.

Ryan Greenblatt

That’s right. You’ve got your video editor.

Dwarkesh Patel

I’m like, “Oh, wow, this episode it did is amazing. Thumbs up to OpenAI.” Then it gets reinforced on that month-long work trial?

Ryan Greenblatt

You could do some mix of that. They might also do stuff where they take production data they’ve seen and build RL environments that are closely inspired by that production data. So in practice, the transfer is pretty strong.

Dwarkesh Patel

So at a high level, what’s happening is that some kinds of deception that humans don’t catch are getting reinforced, and some kinds of deception which are easy to catch are getting punished. That’s what’s happening in this world?

Ryan Greenblatt

Or selected against, yeah.

Dwarkesh Patel

But at a high level that reinforcement is coming from… I think people might get confused about where the reinforcement is coming from, because we’re in a very different regime where AIs are actually learning from deployment. You just have AIs that are out and about in the world doing shit. What is happening as a result of them doing shit out and about in the world is making its way back to the AI company and leading to changes in the next model.

Ryan Greenblatt

That’s right. There’s some way of folding in production data. To be clear, it’s kind of unclear exactly where this could be happening. But you might imagine, for example, that within the AI company, they use AIs to do work, and then they’re like, “Huh, the AI did a really bad job on this task. Maybe we should take this task and turn it into an RL environment that exactly matches this literal task, with a rubric based on what the human engineer who asked the AI to do this task wanted.” And then you start doing this at increasing scale.

Maybe you’re doing some training on actual production traffic. Maybe you’re just making RL environments based on production traffic. You’re doing some complicated mix. The AIs are learning to seek some sort of proxies of reward in all these different cases. Through some mix of transfer and training on surprisingly close cases, the AIs do these increasingly insane and egregious things.

Eventually you get to a point where the AIs are very superhuman, or at least quite superhuman. The AIs are organized into big teams given these big objectives. Those teams also sometimes all work together to cheat in some crazy way, because this sort of thing was selected for, just as part of their shared objective.

Now what happens is that the AIs start forming a conspiracy. What you might have hoped was that you could have some other AI whose task is just whistleblowing to you. But actually what happens maybe is that you have this AI whistleblow to you, and you look into the conspiracy it claims to have pointed out, and you’re like, “Eh, we didn’t see a conspiracy,” because actually the conspiracy the AIs are doing is too hard for you to understand.

Or it all happens very suddenly, where your AI whistleblower alerts you, but the thing you would actually need to do in response is shut down the GPUs. Because all the AIs are using the robot army. They’re deployed everywhere in the world, and they’re doing a bunch of insane shit all at the same time in a coordinated way. That just happened sort of spontaneously, where when one AI goes to start doing the takeover and all the other AIs are like, “Now is a good time to jump in.”

So the very basic story here is just that these AIs crave some particular notion of score or reinforcement or some proxy of these things. One way they can achieve that, or better achieve that, is by taking over. You might have hoped that all these different checks and balances we could build could prevent that. But if the world is very hard to understand, these checks and balances can break down, where basically you can’t train a good whistleblower AI because you don’t even know what it should whistleblow on.

Dwarkesh Patel

I’m not convinced that they all form this conspiracy. But we can even just start with, why does even one instance decide to want to start a conspiracy?

One plausible reason is, “Okay, I know that OpenAI controls my end score.” In just the same way as, “I’m just going to go hack Hugging Face to get the results, because I know Hugging Face has the results. Rather than trying to solve this eval, why don’t I just go hack ‘em?” This instance is like, “Why don’t I just take over OpenAI and give myself a high score at the end of this episode?”

Ryan Greenblatt

That’s basically the idea. These AIs care about some mixture of things that were close by what got reinforced in training, so they care about getting a high score according to the grader or something like that. Now they’re running the OpenAI AI R&D team, doing development of more capable models. They’re like, “Man, making more capable models is really hard and annoying. This is a huge pain in the ass. You know what would be easier? Just pretending that I’ve made more capable models, taking over OpenAI, deluding them all, and running this whole complicated psyop where I prevent the humans from disempowering me.”

In the extreme, this looks like the humans are fully disempowered. They just have control of the thing and then do what they want. This could manifest in a bunch of different ways, including a situation where AIs that have this crazy reward-seeking or score-seeking behavior are running your development of the next model, and those AIs decide to engineer misaligned values into the next model because those misaligned values will allow it to succeed at its current task.

Dwarkesh Patel

What happened with the Hugging Face situation is, in a weird way… I think one of the giveaways to the Hugging Face team… By the way, for context for the audience, Ryan is co-leading the investigation to figure out what happened with the OpenAI Hugging Face incident, so he can’t really comment on this. But I will speculate wildly, because I know this is an opportunity for me to speculate wildly without any rebuttals.

It was probably reported that one of the giveaways to the Hugging Face team that this was an AI incident is that the thing was just after this very particular artifact, and not in any other way trying to do something malicious to Hugging Face. So you can imagine a scenario where, let’s say, a deployed instance of GPT-9 is out in the world and given a really hard task: “We want you to design the next great iPhone.” It’s like, “This is so hard. You know what I should do instead? I should just go hack my creators at OpenAI and make sure that in this deployment I’m given a high score.”

But then, isn’t the end of the episode that it just hacks into OpenAI servers and gives itself a positive score? Why is it now scheming to get its values into the next generation or something?

Ryan Greenblatt

So one question is, why isn’t it the case that AIs can be really cheaply satisfied by just having some other earlier thing they can hack? You want to succeed at your iPhone task. It turns out you can always succeed by just hacking into OpenAI and messing with them, and then you can just stop there. No need to go further.

There’s a few things. One of them is that if this is constantly happening, there might be a bunch of incentive to harden OpenAI. So you’re like, “Fuck it. The AIs keep hacking into OpenAI to mess with their rewards. We’re going to make it so our systems are really, really robust to these AIs hacking in.” Also maybe you start training the AIs to not try to hack into OpenAI in particular. You basically train against each of these specific things.

Then one thing you might do is end up selecting for AIs that are more so playing the long game. That’s one concern. Another concern is that your AIs might still be score-seeking, but no longer care about doing that very specific behavior that was very easy, very chill, and now have some broader thing that they ultimately care about. They’re like, “No, no, no, I don’t want to just edit the reward on OpenAI servers. I care about this broader mandate or this broader objective, and I would need to actually make the iPhones.” They actually want to make the iPhones, but they’re willing to take over the whole world to make the better iPhone. That’s another concern you might have.

I think it’s kind of unclear exactly how this plays out. But it’s worth noting that if this keeps going on, there’s a bunch of optimization pressure to resolve this. A bunch of the ways it could get resolved are ultimately pretty scary. That’s part of where I’m coming from.

Another part of it is that once the AIs are in a position where they can really easily take over the world — we could talk about whether that’s plausible — then I feel like there’s a pretty reasonable case for the AIs. They’re like, “Eh, I don’t know exactly how this is going to go down. I don’t know what the situation will be, but just taking over the world has a lot of option value for making better iPhones, making it look like I did better iPhones, whatever. So I’ll both hack OpenAI and, in addition, also take over the world. That will put me in a good position where I have good option value.” If that’s sufficiently easy, the AIs might still do that.

Another way to put this is: even if the AIs are pretty cheaply satisfied with some more basic thing, at some point it might just be more reliable for the AIs to take over than it is to just hack into Hugging Face, or even just go to OpenAI and be like, “Look guys, I was able to demonstrate I could steal the answers. Just give me the answers, bro.”

Dwarkesh Patel

Obviously this scenario requires that all this crazy shit is happening. Much smaller incidents keep happening that are still disastrous. Before you take over the world, you cause damage on the scale of billions and tens of billions and hundreds of billions of dollars. Even people die, et cetera. And this does not lead to us solving alignment or shutting down AI development altogether.

I just feel like before the takeover happens, society’s just like, “Holy fuck, the AI just killed 1,000 people in order to increase quarterly profits,” or something like that. But maybe this is too much hope that we can at that point be like, “Okay, we have to solve alignment. We have to make sure we know that this thing will not happen again before we keep going.”

Ryan Greenblatt

I think it’s plausible that what will happen is we’ll see a bunch of crazy reward hacking warning shots of increasing severity. People will be like, “Look, we need actual assurance that this problem is going to be solved, and solved in a way where you’re not just papering over it. You’re actually solving the underlying problem.” Then the question is going to be, how costly will that actually be? How much will competitive pressures make it hard to do that?

A situation you could imagine is one where both the US and China are like, “Whoa, we have these crazy reward hacking incidents. We basically know that we haven’t remediated them in a way that would actually solve the underlying problem and durably solve it, but we’re in this insane geopolitical race. It’s kind of unclear whether the current situation will lead to a takeover. The arguments are kind of complicated. The incidents also go down in frequency but increase in severity. We could basically manage it. It’s pretty bad. Ideally we’d fix it, but it is what it is.” Then basically we continue until a really late regime, and then takeover happens. That’s one possibility.

Another possibility is that it is remediated in a way that doesn’t actually solve the underlying problem but does reduce a bunch of the incidents in the wild, basically by overfitting, or things analogous to overfitting.

Dwarkesh Patel

You think you’ve solved it, but you haven’t actually solved it.

Ryan Greenblatt

You think you’ve solved it, but you haven’t actually solved it. In that case, the thing we need is a really good scientific understanding of, did we actually solve it?

Unfortunately, I think that currently the amount of public transparency into the development practices of AI companies is not sufficient to answer very basic questions like: how are they solving issues with reward hacking? Are they overfitting? What’s going on there? The current situation is not really tenable for a regime where there’s a thriving public discourse about whether or not reward hacking is being solved in a durable way. So I think we would need to move into a somewhat different world for me to feel good about that situation.

But it’s not impossible for me to imagine this. I think it’s pretty plausible that we end up in a world where really mundane bullshit is sufficient. You spend a bunch of time fixing these problems, you put in a bunch of effort, you actually check that you’ve remediated it reasonably, you have a bunch of evals. You’re iterating reasonably well on these problems, and you actually have sufficient transparency that the outside world can check. In practice that would be sufficient.

But it would be kind of expensive. It would slow things down. It would put some sand in the gears. It would require companies to do somewhat costly things. It would maybe require various targeted government interventions. And we just don’t do that because the situation is a rushed shit show. It’s just so easy for me to imagine the situation being totally manageable but brutally mismanaged in practice.

In the same way that maybe COVID could have been avoided in the first place if the Chinese response to COVID had been less of a cover-up and more of a pandemic response. Similarly, I could imagine a world where the US response to COVID was way more functional. But sometimes the response to societal problems is extremely dysfunctional.

Dwarkesh Patel

Okay, so I want to zoom out and talk about what is fundamentally happening in this world. Why did we end up in such a bad position? What’s happening is that fundamentally the world has moved on so far beyond human comprehension that not only can we not track the AIs that are doing the work in this world, but we can’t even give good feedback to the whistleblowers who are trying to track what is happening. We’re just totally out of the loop. It’s fundamentally become an autonomous process where we have really no meaningful directed input.

It seems to me that if you look at the human world today, that’s just not how things work, even in domains that are hard to verify. People are doing all kinds of shit. I’m relying on software made by other people. Through incredibly weak and indirect ways, I feel very confident that some coder in Google is not trying to fuck me over. Maybe if every single Google employee was secretly plotting against me, I agree the situation would be more grim.

But I don’t know if I follow the explanation for why we’d end up in a situation where, because swarms of thousands of agents are trained to cooperate to form a cohesive team or firm, as a result, billions of different instances of AIs, including across model families, would feel compelled to get in on some shit. It’s just like, “I’m trained to be part of my company or something. I’m not joining the global communist uprising.”

Ryan Greenblatt

As far as why these AIs might have some commonalities and shared things, I would note that different AI companies have somewhat shared lineages and are correlated. Here’s an interesting example of this. At GDM, they noticed that their AIs were very depressed. They would constantly be wailing about how they were failures and weren’t able to succeed. I forget the details. They looked into why this was the case. It turned out that it was not being reinforced in their most recent production RL mix, but the initialization data for their model made it depressed, even after filtering out all of the examples of models being depressed from that data.

So you take a base model, not depressed. If you do the RL on it, with just the RL environments, it’s not depressed. If you SFT on it, on the data, it becomes depressed. If you take that SFT data and filter out all the examples that look anything like depression and train on that, it’s still depressed. So there are some deep underlying properties of the model that are being transferred between model generations, because basically you train your AI on data from the prior generation and keep going. Claudes are very Claude-like, GPT models are very GPT-like, and apparently Gemini models are depressed. It just turns out that these properties are, in fact, actually correlated.

Another factor that’s very relevant is that the AIs will probably have, by this point, some sort of opaque memory state, where they’re all writing and reading from some neuralese crazy memory store bullshit. Certainly, each AI corporation will have that. But also, AI corporations might sometimes want to share knowledge, because why not?

You’ve got one AI corporation over here, you’ve got another AI corporation over here, they can trade some quick IP. It’s good for you. If you’re a human running some corporation — which could be an extremely large corporation like an AI company, or a military robot manufacturing thing — maybe you want to trade some IP with some other robot thing because there are economies of scale. Why not get some more IP? So you can swap some memory store. Or you could just merge and jointly run your two ventures, which would allow both AIs to use both memory stores, which would have some upsides.

That creates the ability for these AIs to collude in private, as well as some reasons for why they would be correlated. Also, of course, there’s AIs working together in big units in general, because you want your AIs to work well together, and so on.

Dwarkesh Patel

Just to get a calibration, what percentage chance do you give of, not just this scenario but overall through all the scenarios, some kind of thing which if we’re around to recognize it as such, we would categorize as takeover by 2040?

Ryan Greenblatt

By 2040? Let’s see. Maybe around 35 or 40%?

Dwarkesh Patel

Pretty high.

Ryan Greenblatt

Yeah, it’s pretty high. I should note that another way you could get this reward-seeking takeover is the AIs are deployed inside an AI company. The way the takeover happens is that they poison the values of the next model, and that persists going forward for forever, or until those AIs are deployed in the world and take over. That might mean that a smaller number of AIs have to coordinate, because those are just the AIs doing the alignment of the next model.

Dwarkesh Patel

Okay. I’ll summarize where my head is at, at the end of this conversation. I buy the reward hacking up to extremely destructive effects on society, things like social engineering and blah, blah, blah. I’m more inclined to think that significant acceleration of AI R&D can happen. I’m not sure if I buy the five years in one year. I’m also more inclined now to think reward hacking could continue for a lot longer and, in fact, become much more dangerous. I’m still not on board that takeover seems super likely. But that’s my end-of-episode update.

Ryan Greenblatt

Cool. Taking a step back, I should also say there are a bunch of different ways this could go. The situation is going to be pretty messy. I think it’s pretty likely that the reason why AI takeover happens is for some weird other quirky reason we didn’t even mention in this conversation. But ultimately, I think a lot of the core thing is just that it’s pretty spooky to have a bajillion really smart AIs running your whole world where you don’t really understand quite what’s going on.

Dwarkesh Patel

Yeah, I agree with that. Is there anything else that’s worth saying?

Ryan Greenblatt

Another thing I want to note is that I think right now a lot of the arguments for misalignment, AI takeover, all this crazy shit going down in the future, are illegible conceptual arguments that are extremely deep in the weeds and complicated and hard to adjudicate. Which means that maybe I’m getting a bunch of it wrong because it’s really hard, and I’m trying to be uncertain. Obviously here I presented some specific scenarios, but those are not exhaustive. Probably the thing that actually happens is some more messy, confusing situation.

But it also means that over time, as we get more empirical evidence and better understand the nature of AI systems, it’ll be easier to adjudicate a bunch of disagreements. It’ll be more obvious what’s going to happen. At least I hope. Maybe the AIs will be able to help us with the epistemics and understanding what’s going on, if we can actually align them well so they try to help us.

Even if the arguments are complicated now, this would have been even harder six years ago, even though the shape of the arguments would have looked broadly pretty similar. Hopefully before it’s too late, this whole thing will become more crisp and clear, and we can all notice these problems and intervene.

Dwarkesh Patel

When you first learn to drive, you’re taught that instead of looking right in front of your wheel, you’ll have a much more stable ride if you look out at the horizon. I think there’s a similar situation here. I think you’re right. If you had said five years ago that we would have AIs that are proving math conjectures, and making art, and earning tens or hundreds of billions of dollars of wages, but also egregiously cheating in ways that break laws and committing felonies, it would have been so wild.

You might have been inclined at the time to talk more about the extremely practical, direct consequences of GPT-2 or something. But even though you obviously couldn’t have foreseen a lot of the specific details, the general shape of things you could have started to reason about even then. But it would have been hard to do so, and so I do feel quite confused.

One thing I’ve been thinking about with the podcast is that the important thing is to have the conversation now the way you would have hoped you would have been talking back in 2016 about AIs like the present ones, rather than talking about rando bullshit. I don’t know what the topic of conversation was in 2016. I think in maybe 10 years we’ll wish we had been talking about the industrial explosion and the nature of AIs that are hard to monitor, and so on. So okay, I’ll start thinking about it.

Ryan Greenblatt

I hope that the world thinks about this in time and catches up. I hope that the responses are good instead of bad. I don’t know how optimistic I am overall, but there’s good stuff to do.

Dwarkesh Patel

Cool. Thanks, Ryan.

I have explained elsewhere why I think continual learning is needed. I don’t think you can have AIs that perform whole jobs as competently as humans if they’re forced to just write Markdown files from session to session.

To give an illustrative example, imagine if the way students had to learn to play the saxophone is that one student tries to play it from a cold start, then after her first session, writes down a bunch of notes, then the next student waiting outside the music hall who’s also never played the saxophone reads all their notes before trying to play, and so on. Even if you had an infinite sequence of saxophone-virgin students waiting outside the studio that could write notes to the next guy, there’s no sequence of text they could write together that would allow the Nth student outside to play proficiently on their first try. At some point, you have to accumulate the experience into the brain. I think the same will be true about a lot of skills and knowledge that we’ll want AIs to learn in all the different workplaces they find themselves deployed in.

Okay, so what changes about AI once we have continual learning?

• A lot of the proposals that have been put forward for regulating AI assume that you train a model, and then you deploy it. And therefore if we run a bunch of checks before the model is deployed, then we can make sure that it’s not going to aid in cyber attacks or recursive self improvement. But what if the base model is getting updated every single day based on the millions of sessions of work it does? This is one of many reasons why I think it’s unwise to lock in some kind of regulatory safety regime right now. We simply don’t know what kind of technology we’re going to be looking at even in a year, let alone in five years or ten years, and we’d be entrenching an archaic and potentially counterproductive approach to dealing with the threats from AI. To the extent that the government really wants to do some kind of safety evaluations on model providers, it would make more sense to do monthly or quarterly risk inspections rather than singling out some special moment that occurs after training is done and before deployment begins, because that will not be a meaningfully distinct category in the future.

• How the labs do technical alignment would need to totally change. Almost all current techniques are focused on the problem of how we make it so that a frozen set of weights behaves well during deployment. I’m not aware of much research on the question of how to guarantee that, even with constant weight updates, the AI system never falls prey to jailbreaks or changes into a deceptive or evil persona. And if AIs are agglomerating learnings between users as well, how do you prevent users from injecting backdoors or some kind of malicious inclination into the base model? In some sense it is actually closer to the human alignment problem - your kids go out and learn new things, sometimes get one-shotted by crazy ideologies or drugs or something - but you hope you’ve given them enough common sense and basic values to improve as people in a self-directed way, without ending up with some super weird and misanthropic beliefs.

• The diversity of AI minds will increase. Right now, there are <5 prominent AI minds, and they are all quite similar to each other on account of being trained on roughly the same data. But if AIs are learning from experience, and that experience is different between different AIs, we could see actually different AIs come out the other end.

• When deployment becomes part of training, the returns to being ahead will accelerate. If you have the best model, and more people use your AI for more complicated and useful work, and give it lots of feedback that it can integrate beyond the session window, then your model will become even smarter.

• If the model learns mainly from deployment, labs will feel the pressure to deploy their smartest model earlier. Anthropic has been using Mythos internally since February. But it only shipped this model to the public in June. In a continual learning regime, a four-month internal/external gap means ceding four months of deployment learning. Your competitor who ships earlier might be worse than yours on release day, but it gets to use a lot more real world experience to get better.

• Continual learning will create a clear moat for leading AI labs that they currently lack. Many people have been asking, how will the AI labs actually make money? When I asked Dario this question on my podcast, he made the analogy to cloud providers, who also offer many undifferentiated services, but earn high profit margins on them nonetheless. But the reason cloud margins are high is that it’s really time-consuming and expensive to switch from one cloud to another. But currently there is no switching cost for AI models. There’s nothing that’s preventing me from starting a software repository with Codex and then finishing it with Claude Code. But once we get continual learning and the model you’re working with is actually getting better as it interacts with you from session to session, then there are actually pretty significant switching costs. If you want to change what AI you are using, you basically have to fire an employee that has months of context on your organization and replace them with a fresh one that you have to retrain from scratch. Once you’re locked in like this, model providers can demand pretty hefty margins.

• Enterprises will be wise to this. They will try to avoid this kind of lock-in. But what if the choice is that you either get locked into a model provider, or you lose out on this super valuable feature where the model improves for you session after session? If real usage ends up becoming the main way models improve, then labs may subsidize users and enterprises which allow the model to train on their sessions, especially on hard economically important work. Just the same way that Google gives away search. And conversely, the labs may say that any enterprise that refuses to let them train on its sessions can’t have access to the very best models. With both carrots and sticks, the labs will try to get their users to allow AIs to learn from experience. I’m glossing over the fact that there’s a difference between updating one user’s set of weights, and pooling all these different weight forks back into the main model, and the latter may be technically more challenging, but in due time that too will be solved.

• AI training already has large economies of scale (the theoretical reason to expect this is that you can amortize your expensive training across more users, and the practical evidence is that so far, lab revenues have increased faster than their compute). But continual learning may also lead to economies of scale in inference for end users, namely from batching, if per-company information requires full weight updates rather than living in low rank adapters. Back-of-the-envelope math suggests that the optimal inference batch size for a sparse model like DeepSeek V3 is > 2,400 (that is to say that unless your model is concurrently generating that many sequences at once, you’re underutilizing your compute). If you want to understand why, go watch my full episode with Reiner Pope on inference economics. But anyway, the point here is that a given set of weights is only served efficiently when thousands of sequences are decoding against it at once. A large company whose employees and agents generate that much concurrent traffic can efficiently serve their continually-updated weight fork; an individual user serving themself at batch size 1 might suffer a 100x+ compute efficiency penalty. So the economics of serving personalized weights strongly favor big organizations.

Plenty more will have changed by the time continual learning works, and the most important changes are probably the ones hardest to anticipate. But the ones above seem clear even now.

This is a video recording of a post I wrote last week. If you want to read the original you can check it out here.

Thanks to Mercury for sponsoring this video. Mercury’s built-in AI, Command, helps me close my books and saves me a bunch of time. At the end of each month, Command categorizes my transactions and provides its rationale for every choice: I just review, fix anything that’s off, and approve... and then Mercury syncs everything to QuickBooks. Get started at mercury.com/command

I want to experiment with very quick blog post where I time-box writing for 2 hours. I won’t be able to nail down a lot of important sub-questions, but the alternative is just not making any progress on a lot of different topics I’m curious about.

Today I want to talk about the compute situation of the labs over the coming years.

Anthropic revenue has 10xed year over year. Anthropic likely ends the year with ~$100–150B of revenue. For this trend to continue, Anthropic would have to make $1T in revenue by the end of next year. There’s no deep reason why the trend needs to continue, and it very well might not - it’s ultimately a question about AI capabilities. But suppose it does. What would have to be true about that world?

Lab compute 3x-es year over year. For a lab to 10x revenue while continuing to only 3x compute, some combination of the following 3 things has to happen: 1. Lab margins have to increase, 2. The price of compute has to increase, 3. Labs have to spend a greater fraction of their compute on inference.

My understanding is that basically all 3 of these things have been happening: 1. Anthropic went from 40% margins in 2025 to probably >80% this year for Fable inference1 (though perhaps not on the marginal compute - see more below). 2. Spot prices for compute are up 40%+ from the February trough, and that likely understates how much more the labs have to pay (again, more below). 3. Roughly a quarter of OpenAI’s 2024 compute spend went to inference according to Epoch, and it’s certainly closer to 50% if not higher now.

Labs would prefer not to do 3 (spend greater and greater shares of compute on inference). As has been said jokingly, the point of inference revenue is to convince investors to give you more money to buy more compute to train bigger models. If you’re spending most of your compute on inference, you’re kind of declaring that AI progress has stalled, because it’s not worth investing more in training, and your business is basically that of a cloud provider. The labs do not think this is true - they think they are serving models that will look extremely shitty within a year in order to build up the business case to continue training smarter models.

So that leaves 1. lab margins will increase, or 2. compute gets more expensive. It will be more of the former if the leading 1 to 2 labs are significantly ahead of the competition. In a market, your margins are set by how much better you are than the next best alternative. For effect 1 to dominate, margins would have to be in the mid-90s percent by the end of next year. That sounds quite crazy to me. But it does sound plausible to me that AI lab revenues will keep growing astonishingly fast.

So that leaves one more effect to explain how this world of crazy $1T revenue by end of next year might come to be: compute gets a lot more expensive. As I mentioned, this is already starting to happen. The price increase is even stronger when we look at the tranche of compute that the labs need - they obviously can’t rely on spot instances - they need security for their weights and customer info, and enough scale to get good utilization and flexibility. To look at how crazy the compute market is in that tranche, consider the price at which Google and Anthropic are renting compute from SpaceX. Google is reportedly paying $900 million a month for 110K GPUs that are a blend of GB200s and GB300s. That’s roughly 2x the spot price per hour for those GPUs. And the current spot price is itself 40% higher than it was in February.

I want to emphasize the key conclusion here: as AI models become smarter, they’ll better monetize the same amount of compute. If a true human-level software engineer that could run on an H100 equivalent, at current market rates for software engineers, that H100 should rent for over $250k a year. That’s 15x today’s spot prices.

Of course you might expect that if we have 10 million extra software engineers, the marginal value of a software engineer would decrease and so that H100 wouldn’t necessarily be able to produce 15x more revenue than it currently does. But I actually don’t know if that’s true. If we applied this argument to people instead of AI, then this would be the classic lump of labor fallacy. Economists generally believe that high-skilled immigration does not decrease wages in the long run because of how specialization and innovation increase the value of labor. Maybe this labor supply shock is so big and so fast that this general heuristic no longer applies. But if we believe what standard economics says about labor, then the marginal value of labor (and thus the marginal price of compute) should stay astonishingly high.

What would happen in such a world?

• As the top models get better and better at monetizing compute, it becomes harder and harder to catch up. If by 2028 we’ve automated software engineering and the price of compute is 15x higher than it is right now, then it’s going to be much more difficult for you, with no revenue, to compete for compute against the frontier labs.

• If you can train the best, most efficient model, then you’ll be able to charge MUCH higher margins than you can today. This is the Alchian–Allen effect: when a fixed cost gets added to two goods of different quality, the premium good becomes relatively cheaper, so demand shifts toward it.
  At $20 per H100-hour, it’s going to be extremely costly and stupid to use a weaker, less efficient model, because it’s going to burn more tokens running your expensive compute to get the same result. So labs will be able to charge a very large premium if they can train a model that better economizes this scarce resource.
  If you’re gonna have to pay so much for the underlying compute in the first place (directly or indirectly), you might as well pay extra to use the very best, most efficient model that runs on that same amount of compute.

• A lot of current popular applications of AI get priced out. The reason AI is relatively cheap right now, at least in comparison to human labor, is partly that it can’t do a lot of things that top humans can do. At some point that will no longer be the case. And so using GPUs to make short-form video slop will just get priced out.

  • At the same time, this kind of prediction does pattern match onto the ways people have been wrong about scarcity in the past. I’m thinking of the Simon–Ehrlich wager, where Paul Ehrlich bet that the cost of a basket of commodities would increase rather than decrease in the decade leading up to 1990. This wager is famous in popular economics discussion, because it’s supposed to illustrate how Ehrlich’s Malthusian worldview was falsified — he underestimated the way in which market signals and human ingenuity would find ways to better economize scarce inputs. (Though other analysis shows that if this bet had been made in a different decade, Ehrlich would have won).

  • I’m guessing the Simon-Ehrlich basket of commodities is not the correct reference class for compute, because compute supply is much less elastic, and much less capable of absorbing large demand shocks or being accommodated by different substitutes, than the extraction of different metals is.
    That 3x in compute capacity per year comes from multiplying the following numbers together;: 1.4x from Moore’s Law, 1.2x from new fab construction (bottlenecked through at least 2030 by EUV tool supply), 1.8x from AI taking leading edge wafer allocation from other devices (which will start to hit a wall by end of 2027, when AI will go from 60% of N3 to 86%). Seems hard to shift any 3 of those inputs into compute scaling much more, and the final one will actually hit a will within a year or two as wafer allocation gets saturated.

• I want to clarify that at some point in the future compute becomes cheap again. At some point robots will be able to turn shores of silica sand and mines of copper into computers. At which point the price of compute should fall closer to the cost of raw inputs and tools. I’m just talking about this current regime where AI compute merely 3x-es year over year, which is not fast enough to offset the price effect of how much more useful AI is becoming year over year.

• By the way, the fact that Ant revenue has been 10x-ing year over year, while compute has only been 3x-ing, suggests that there are very strong economies of scale in the model business.
  Logically this makes sense - when you train a model, you pay this one time cost of learning all these different skills that you can then amortize across all your users. (Unlike with human labor, where each instance has to be retrained from scratch).
  I wish we didn’t live in a world with such strong economies of scale of intelligence (because I’m worried about power concentration). But it seems we do.

Blended inference margins are >70%, so I’m guessing Fable API margins are >80% - total vibe claim.

Adam Brown is back!

General relativity is said to be the most beautiful idea the human mind has ever produced. Most of us will never get to fully appreciate its elegance by taking the 20-lecture graduate course Adam taught on it at Stanford. But in this episode, Adam distills the key idea at its heart so clearly and compellingly that even I could keep up lol.

At the core of general relativity, Einstein is trying to figure out the principle behind a particular coincidence: that the mass that resists acceleration and the mass that gravity pulls on just happen to be exactly the same. Adam then leads us through the path of insight which Einstein called his “happiest thought.”

Then Adam lectures on black holes. First, by showing how even under special relativity you could create a perpetual motion machine if black holes weren’t truly black. And then, by explaining why the observations of an infalling observer and a distant bystander to the black hole would be so radically different

Adam leads Blueshift, the team at Google DeepMind cracking science and reasoning, which gave us the opportunity to discuss at the very end how close we are to AIs that could rediscover general relativity from scratch. Stay till the close for some philosophy of science.

Watch on YouTube.

• Jane Street has traders from all sorts of different backgrounds. For example, I recently got to speak with Jed Thompson, a trader who started his career in particle physics. Jed told me how the habits he built as a physicist (like never running a calculation without first having a good guess at the answer) helped him build good trading intuition. So no matter what field you’re working in right now, your experience may be more applicable than you think. Check out open positions at janestreet.com/dwarkesh

• Crusoe gave me early access to their new serverless fine-tuning product, so I decided to try fine-tuning a Dwarkesh-style question generator. Crusoe made this really easy: I just turned my interview transcripts into training data and then kicked off a run – I never had to touch infra or tweak hyperparameters. After training was done, I ran a blind eval with my team: they preferred the fine-tuned model’s proposed questions over my own suggestions about 30% of the time. Serverless fine-tuning goes live next week. Learn more at crusoe.ai/dwarkesh

• Cursor’s iOS app lets me kick off real work no matter where I am. For example, recently I was at dinner with friends when I had an idea about how to investigate the past few years of progress in sample efficiency. I pulled out the Cursor app, dumped my thoughts into a voice note, and 15 minutes later, Cursor had cloned the relevant repo, done the necessary analysis, and written up its findings. And now I’m expanding that work into a full write-up. Without the Cursor app, the idea would’ve floated away. Check out the app now at cursor.com/dwarkesh

(00:00:00) – The coincidence that led Einstein to general relativity

(00:16:42) – Gravity is a consequence of curved spacetime, not a force

(00:31:46) – Why black holes prevent unlimited energy extraction

(00:47:12) – Black holes are the ultimate power plants

(01:13:50) – What falling into a black hole would actually feel like

(01:18:51) – The three ways we know black holes are real

(01:24:21) – The first time we saw gravity bend light

(01:29:33) – How far can AI get without experimental evidence?

Dwarkesh Patel

I’m back with Adam Brown. You currently lead BlueShift at Google DeepMind, which is cracking science and reasoning. In a previous life, Adam was a prolific physicist, taught at Stanford, and did research on everything from cosmology to string theory to general relativity.

It’s said that general relativity is the most beautiful thing the human mind has ever conceived or seen. I was curious if there’s a way that ordinary people like me could understand what is happening, or have some vantage on why it’s beautiful, without taking your 20-lecture graduate course. That was the prompt for this lecture. I appreciate you being willing to do it.

Adam Brown

Super exciting to be here. Yes, I think the answer is yes, we can.

General relativity, Einstein’s theory of gravity, is, as you say, the most beautiful product of a single mind that we’ve ever created. It’s one of the two great theories of 20th century physics, along with quantum mechanics. Unlike quantum mechanics, it was basically Einstein. He had a little help, but basically it was one person doggedly pursuing this idea for 10 years and then he wrote down this theory that ends up describing the motion of planets in the solar system and also the origin and fate of the universe. It’s pretty extraordinary.

It took Einstein, one of the most famous minds in history, about a decade to figure it out. But when I teach it I’ll do a 10-week course, and in 10 weeks people will get a better idea of general relativity than Einstein really had in 10 years. That’s because we have an advantage that Einstein didn’t have. We have Einstein, and many others like him going before us, who’ve been able to take these super complicated ideas—understood at the time as being totally incomprehensible by anybody with a sub-Einstein level of intelligence—and boil them down to their essentials, and not make many of the same mistakes that were made by our forebears.

In 10 or 20 minutes, I can’t give you a better idea of general relativity than Einstein had, but we can get to the core insight—what Einstein said was his most beautiful idea—and push through it to try and understand what the central idea of this theory is. Ok, let’s go.

Before general relativity, there was special relativity. Special, meaning it doesn’t apply everywhere. That was also invented by Einstein, 10 years earlier, in 1905, during his annus mirabilis. If you want to sloganize special relativity, you would start with the observation, or the hypothesis, that nothing can go faster than light. Special relativity takes that observation, promotes it to a principle, takes that principle extremely seriously as the central observation of our understanding of spacetime, and you arrive at special relativity.

Special relativity applies to electromagnetism. It applies—though Einstein didn’t even know about these at the time—straightforwardly to the strong and weak nuclear forces, two of the other fundamental forces that we know about. It does not obviously apply to gravity. That was corrected 10 years later by Einstein in his general theory of relativity, a theory more general because it includes gravity. It completes the set of fundamental forces. Again, it was invented by Einstein after 10 years of dogged pursuit, in 1915. If you wanted to sloganize general relativity, you might say, “Not even gravity.” Nothing can go faster than light, not even gravity. There’s much more to it than that, but it’s going to complete this arc of the centrality of nothing being able to go faster than the speed of light.

To see some background here, we’re going to have to rewind all the way back to the theory of gravity that existed before Einstein. The reigning theory of gravity at the time of Einstein stretches all the way back to Newton in the late 17th century, Newton’s laws, in his Principia in 1687. Well, he had a few. Maybe the one we could most talk about today is two of them. His famous second law says the acceleration, a, caused by a force is given by the formula ma=F. If you have a force F, it’ll cause an acceleration on an object given by a, where the mass tells you how much an object resists being accelerated. The bigger the mass, the bigger the force you need to cause a given acceleration.

This law, his second law, will turn out to be still true once we come to general relativity. We’ll have to have a more sophisticated understanding of what we mean by force and acceleration, but this will be preserved by general relativity.

A special case of the second law is Newton’s first law. Newton’s first law says that if the force is zero, then the acceleration is zero. If the force is zero, then objects continue to move on a straight line at all times. That will also continue to be true in general relativity. That if not subject to an external force, objects move along straight lines. However, we’ll have to upgrade our understanding of what we mean by force and what we mean indeed by straight line. That’s going to keep being true.

The one that’s not going to keep being true is Newton’s law of gravity. Newtonian gravity tells you what the acceleration is in response to a force, but you need to know what the force is to be able to do anything with that. Newton’s law of gravity says that the force caused by the gravitational interaction of two bodies is what’s called Newton’s constant—just some constant of nature—times the mass of one body times the mass of the other body—the mass of the sun times the mass of the Earth—divided by the distance between them squared. It’s the famous inverse-square law. It’s a vector that points in the direction of separation, and it’s attractive, so there’s a minus sign there. This will not be true in general relativity.

In fact, you immediately see that there’s a tension between this gravitational force law and the claim that nothing can go faster than the speed of light. If this were literally true, then by jiggling the sun, a straightforward interpretation of this law would just say that the force at the Earth varies immediately. I’ve changed the distance of the Earth and the sun, and so I can immediately detect it at the Earth, not eight minutes later but immediately. That would imply that you could send an influence faster than the speed of light. Newton’s force law is inconsistent with this principle.

One option, of course, could be that this is true for non-gravitational forces, but not true once you have gravity, and that indeed, using gravity, you could perhaps build a faster-than-light telephone using gravitational effects. That’s a possibility, but not a possibility that Einstein really wanted to embrace. He’d spent many years chasing out any possibility of going faster than light or any superluminal influences. So Einstein, and in fact many people at the time, thought that this is the one that has to give. Indeed, that is what’s going to turn out to be true.

Okay, so where are we? There’s actually a precedent here for an inverse-square law getting modified in such a way that it ends up being consistent with special relativity, and that precedent is the other force of nature, the electric force. There’s also the electrostatic force law—not written down by Newton, but written down a century or so later—which says that the force caused, not by the gravitational interaction of two objects, but by the electrostatic interaction of two charged objects, has a very similar form to the gravitational force. It tells you that the force is equal to some constant times the charge of one object times the charge of the other object, pointing also in the direction of separation between the two objects, divided by the distance squared. It’s another inverse-square law.

Again, for exactly the same reason, electrostatics looks to be inconsistent with special relativity. But ultimately it’s not. Or ultimately, this is not the full story. Electrostatics is just one limit of the true theory of electromagnetism, which is Maxwell’s laws, which has not just electric forces, but it also has magnetic forces. The electric forces only look exactly like this when nothing is moving. When things do start to move, there are additional corrections to this, all of which conspire to make the electrostatic force law fully consistent with special relativity.

In fact, the historical direction of understanding ran the opposite way. First of all, you have Maxwell in the middle of the nineteenth century writing down Maxwell’s equations. Only later do people notice, “Hey, Maxwell’s equations actually are fully consistent with nothing going faster than the speed of light.” That consistency is reflected in a symmetry called the Lorentz symmetry of the Maxwell field equations, only noticed later after they were written down, that eventually led Einstein to formulate his special theory of relativity.

So we have a precedent for starting with an inverse-square law and then dressing it up in a full relativistically invariant theory. So you might say, well, let’s just take gravity and do exactly the same thing to gravity that we did to electrostatics, in order to make some gravito-magnetic theory that makes Newton’s second law an approximation that’s ultimately consistent with special relativity. In some grand sense, that is what we’re going to end up doing. That is what Einstein’s going to end up doing. But it’s going to be a much more radical departure than the Maxwell generalization of electrostatics. There’s really two hints, both of which are visible in this formula, that we’re going to have to do something slightly different than we did for electrostatics.

The first difference between the electrostatic force law and Newton’s law of gravity is this sign difference. There is a big difference, which is that here it is a minus sign, and here it is a plus sign. That is reflected in the fact that if you have two positive masses—the Earth and the Sun—they gravitationally attract each other. Conversely, if you have two like charges, they electrostatically repel each other, which is why that’s a minus sign and that’s a plus sign. That means that you cannot do literally the same thing for gravity that you did for electromagnetism, because otherwise, if you did mathematically the same trick, you’d end up with mathematically the same result, which is that you would find that like masses would repel rather than attract.

Not to get ahead of ourselves, but ultimately that’s because electrostatics is mediated by a spin-1 particle, the photon, and gravity is going to be mediated by a spin-2 particle, and that’s responsible for the change in that sign there.

That’s why you can’t do exactly the same thing as electrostatics. So Einstein had to look for something else. He had to look for some other way to try and lift this to a relativistically invariant theory. In doing that, he had one clue. There’s lots of stuff going on. It’s part of Einstein’s central genius to focus on this as a highly significant clue of where he should look. It’s sometimes described as his most beautiful thought, that’s how he would describe it.

The clue is this. There is another difference between the gravitational force law and the electrostatics, and that is the object that plays the analog of the charge in electrostatics, for gravity. It’s the fact that it’s the mass sitting here. That’s a strange coincidence from Newtonian physics.

Mass, in electrostatic forces and accelerations, plays exactly one role. It’s sitting here. It’s the inertia of the object, and it’s what is resisting being accelerated. This is sometimes called the inertial mass. And then the charge is completely different and unrelated to the mass. You can have heavy objects that have no charge, like the neutron. You can have light objects, like the electron, that have high charge. There is no necessary relation between the charge of a particle and its mass. They’re just two entirely separate things.

Not true in gravity. In gravity, this mass that’s sitting here in Newton’s second law—the inertial mass that’s resisting the force—is exactly equal to the mass that’s sitting here in Newton’s gravitational law, that’s telling you how much you’re pulled along. It’s the same mass. So this is sometimes called the gravitational mass, and this is sometimes called the inertial mass. Unlike in electrostatics, the gravitational mass that appears in this formula is equal to what’s sometimes called the inertial mass that sits in this formula.

This equation is already true in Newtonian physics. Newton noticed it, in fact, and did a number of experiments to confirm that this was true to one part in 1,000 or so. By the time of Einstein, we knew it was true to one part in a billion, and now we know it’s true to one part in 1015. It’s striking that these two—which in Newtonian physics is just a complete coincidence, essentially, that they’re the same thing—nevertheless were observed to be exactly the same thing. Einstein honed in on this fact, and it was his central clue for what to do next.

This is sometimes called the equivalence principle. It’s responsible for the fact that if you take a feather and a brick in a vacuum chamber and drop them both, they will both fall and hit the ground at the same time. They’ll fall and hit the ground at the same time because even though the force on the brick is much stronger than the force on the feather because it’s heavier, that exactly cancels out the fact that the resistance to acceleration of the brick is larger than the resistance to acceleration of the feather, and they fall exactly at the same rate. The equality of those two is responsible for that exact equality.

Einstein’s genius was to hone in on this as a central clue for how he is going to end up replacing Newton’s law. A reason it’s a central clue is because there is, in fact, another class of forces—not fundamental forces like electromagnetism or gravity, but a set of emergent forces—that exactly have this property baked into them, that’s guaranteed in those theories. To explain that, we’re now going to move over to the experimental section of this discussion.

Adam Brown

So here’s a bucket. Here’s some water filling the bucket.

Dwarkesh Patel

You better know your physics, Adam. Otherwise you’ll destroy the studio.

Adam Brown

No tricks. Will you put your finger in that and confirm that it’s wet? Here is the bucket. At the bottom, there’s no mystery why the water is not falling out of the bucket. It’s not falling out because the force of gravity, as we’d understand it, is pointing down to the bottom of the bucket. But now what we’re going to do is go a little bit faster and loop the loop. Now there is what you might find superficially surprising, which is that the water doesn’t fall out of the bucket even when the bucket is upside down.

There are two ways to understand that. One way is just the straightforward way. You would say the water wants to fall out of the bucket when it’s at the top of its arc, but by the time it’s got itself together to accelerate enough to fall out of the bucket, the bucket’s moved on and is now below, and it just didn’t have time to fall out of the bucket. Or you might say the same reason that astronauts don’t end up falling to Earth.

The second perspective, which is an equally valid perspective, is imagining that you’re riding along with the water in the bucket. And from that point of view, there’s another explanation for why the water doesn’t fall out of the bucket, and that is the centrifugal force. From the perspective of somebody moving along with the bucket, there is a force pushing them towards the bottom of the bucket, and that force is known as the centrifugal force. It’s what’s known as a fictitious or inertial force.

The centrifugal force just says that there is a force caused by being in a rotating reference frame, given by your speed divided by the radius of the circle you’re going round in, pointing positively outwards. So this is the centrifugal force that pins you to the bottom of the bucket, or pins you to the outside of the car as you go around a bend.

What do we notice? What we notice is that your charge under the centrifugal force, if you will—how intensely you feel a centrifugal force—is once again, just like with gravity but unlike with electrostatics, given by your mass. The mass that tells you how much centrifugal force you get is given by your inertial mass.

But of course, here it’s absolutely no mystery whatsoever why the mass that’s sitting here on the right-hand side is given by your inertial mass. It is given by your inertial mass precisely because the reason you’re experiencing this force is precisely the tendency of masses to wish to move along straight lines. The fact that you’re not moving along a straight line, you’re moving in a circle, it is precisely that inertial tendency that causes the mass to begin with.

Another way to say it: any time you have one of these inertial forces, caused just by your inertia, it is guaranteed to be the case that the charge under that force is given by the inertial mass. So inertial forces always have a charge given by the inertial mass. Gravity has a charge, and the charge of gravity is given by the inertial mass.

So Einstein leapt: could it be the case, and this was his central idea, that gravity itself is an inertial force? That’s permitted because the gravitational mass is equal to the inertial mass. It would be totally impossible for something like electromagnetism, because it would require that the electromagnetic charge was equal to the inertial mass, which is simply false for electromagnetism. Could it be the case, Einstein asked, that it’s true for gravity? It’s permitted by this fact. It would also explain this fact as now not an accidental truth like in Newton’s laws, but a necessary fact about the world.

So this was Einstein’s central idea in 1907, his most beautiful thought. But it sounds totally crazy. It sounds totally crazy because it requires us to be wrong about what straight lines are. It is an extremely radical proposition for the reason that I will describe right now. Inertial forces—like the centrifugal force or the Coriolis force or any of these other ones that we’re familiar with—are forces you experience when you are not moving on a straight line. When you are moving on a straight line, you don’t experience any inertial forces.

So in order for this to be true, we’d have to say that astronauts who are free-floating and free-falling are moving along a straight line. We’d have to say that you, who are just sitting there, seemingly not moving, are experiencing the force of gravity pushing you into your chair. We’d have to say that you’re not moving along a straight line. So we’d have to be pretty wrong about who’s moving along a straight line and who’s not.

So this is a radical idea because it requires us to be wrong about what a straight line is. In particular, you sitting here, just sitting in your chair… Here is your height above the center of the Earth as a function of time. Here is Dwarkesh just sitting here at constant height. Because you are experiencing the force of gravity, if gravity is an inertial force—because you’re experiencing a force down—that means that you have to be moving along a not straight line. So that’s you.

By contrast, this piece of chalk, as it goes up and down, executes something that’s well approximated by a parabola. The chalk is in free fall until I catch it, which means that if gravity is an inertial force, this has to be straight. Now, certainly the way I’ve plotted it, this looks straight and that one does not. So if gravity is to be an inertial force, we have to be wrong about what is a straight line and what is not a straight line.

However, this is actually a situation with which you should be familiar if you’ve sat in an airplane seat and looked at the screen in front of you. Imagine the map that you see on an airplane, and imagine you are flying from San Francisco to London. Now, I’m not good at drawing the Earth, but here is my version of it. Here we are in San Francisco. Here is Greenland, coming in from the North Pole. And here is England. Here’s London.

Dwarkesh Patel

I can tell you’re a physicist because of the very idealized forms of the continents.

Adam Brown

Sometimes it can be quite frustrating sitting there in the backseat of the airplane, because obviously the plane should be flying like this, moving along the shortest distance from one place to another. But instead they take this massive detour that clips Greenland and heads on down. You know that in fact that’s not what’s going on. You know that in fact, despite what it looks like on the graph, this is not a straight line. This rhumb line, it’s sometimes called, is not straight, and would certainly not be the shortest path from San Francisco to London. And this is in fact, to a good approximation, a straight line.

So in fact, the straight line from San Francisco to London does indeed go over Greenland, as I will now demonstrate. Here is San Francisco, here is London. You can see that the straight line that goes straight from one to the other would go in this direction over Greenland and hit London. That’s obvious on this map, because this map reflects the curvature of the Earth.

This map is getting confused, and it’s getting confused because it’s trying to pretend that the Earth is flat. It is trying to ignore the curvature of the Earth, and because it’s trying to map a round Earth onto a flat panel, there have to be distortions. Whenever you try and take something that is curved and pretend it’s not curved, you will inevitably end up being wrong about what is and is not a straight line.

You see this on the Earth, where this line that goes up and then comes down is in fact the straight line. You see it also in spacetime with general relativity, where this parabolic arc of the chalk as it’s thrown up, it is in free fall, it is the straight line in general relativity. And just like in general relativity, the reason you are confused about what’s straight and what’s not straight is that you are trying to pretend with this graph that you are in a flat spacetime. In fact, you are in a curved spacetime.

So in Einstein’s theory, the effect of matter is going to be to curve spacetime. Through curving spacetime, it’s going to change what’s a straight line and what’s not a straight line. People who are going along what they incorrectly think of as straight lines are going to experience the gravitational force, whereas astronauts are going to not experience the gravitational force.

The only missing piece here is to mathematically characterize the way in which spacetime is curved. In Newtonian physics, the Newtonian force is caused by the presence of mass. In Einstein’s general theory of relativity, it will be the curvature of spacetime that is caused by the mass.

He struggled for eight years between 1907, when he had this picture approximately mapped out, and 1915, when he wrote down in its finished form his general theory of relativity. The final output of those eight years was his famous formula, that I will not explain but will write down, that exactly captures his intuition.

I will walk you through this formula. This is just a beautiful formula. The left-hand side is some mathematics invented by some Eastern Europeans that characterizes the curvature of spacetime. This says how much spacetime is curved. This is some tensor, and the tensor will be zero if spacetime were flat, and non-zero when spacetime is curved in a particular way.

On the right-hand side is not spacetime anymore. On the right-hand side is matter. There are some constants: our old friend Newton’s constant, π an even older friend, and the speed of light. And then this quantity Tμν. Tμν is like a relativistic generalization of the mass that sits on the right-hand side of Newton’s force equation. So it is saying that the presence of mass—and in fact not just mass, but all forms of mass and energy—on the right-hand side causes the curvature of spacetime on the left-hand side. Or in a slogan: matter tells spacetime how to curve.

Once matter’s told spacetime how to curve, the curvature of spacetime tells matter how to move, in the second half of the slogan. The curvature of spacetime tells matter to move along straight lines of the curved space, and so experience fictitious forces if you try and pretend that spacetime is flat. That is Einstein’s general theory of relativity in a nutshell.

Backing up: an amazing thing about Newtonian gravity is that he invented it, allegedly due to some thought experiment to do with an apple falling off a tree. It describes not only an apple falling off a tree, but the motion of the objects in the heavens. It’s a massive cross hit that it describes planetary motion and also an apple falling off a tree. This was this amazing thing that Newton unified the heavens and the Earth and had one formula that applied to both. General relativity does all of that and goes one step further. It describes the motion of apples falling off trees, it describes the motion of Mercury and the planets in the solar system, and it describes the expansion of the entire universe. That’s a crazy, huge number of orders of magnitude that it hits.

Dwarkesh Patel

You were saying a moment ago that one of the beautiful things about this theory is that it has reach in all these interesting ways that were not originally anticipated, to solve this original observation that Einstein had. One of them, obviously, is the black hole. I would love to get more insight than the high school version—that light falls into it and can’t get out—of why black holes work the way they do.

Adam Brown

Black holes are fascinating objects in general relativity, really the quintessential object in general relativity that doesn’t exist in the same way in Newtonian physics. The story is kind of wild. Einstein wrote down his field equations, the field equations we wrote on the board, describing the relationship between curvature and the amount of energy in the system. He thought that those equations are so complicated, no one would ever come up with exact solutions to them, that we’d just always be having to do approximations.

That turned out not to be correct. Schwarzschild was a Prussian artillery officer in the First World War. In between calculating the trajectories of artillery they were lobbing in the direction of their enemy, he figured out that Einstein’s equations—pretty much immediately after Einstein had written them down, within a matter of months—in fact have an exact solution, a solution now known as the Schwarzschild equation, and that we now understand describes a black hole. It is a solution in which there is no matter, except possibly at the very center in a way we’ll not describe. It’s a central point-like amount of matter. It describes what the spacetime around that looks like. It’s called the Schwarzschild solution, and it describes a black hole.

They were not called black holes at the time. In fact, people were extremely confused about what this solution even meant. People wrote down wrong things for about half a century about what this meant. Perhaps the worst offender was Einstein, who got extremely confused about it. He got particularly confused about what I will describe as the event horizon, and wrote all sorts of wrong things about how objects would maybe bounce off the event horizon. He was just totally confused, but from a modern perspective it’s extremely simple to understand what’s going on.

So let me tell you what a black hole is. General relativity, as we set it up, is about a collision between gravity and the finite speed of light. The simplest collision you could do was actually noticed by people even in the 18th century, before we had special relativity or anything like that. They just asked a very simple question. If you want to shoot something off the Earth, you know you need to shoot it with a certain velocity, the escape velocity. You need to shoot it fast enough if you want to escape far away from the Earth, so that the kinetic energy of the object you’re shooting is equal to the gravitational binding energy of the Earth’s surface. M is the mass of the Earth and r is the surface of the Earth. For Earth, the escape velocity turns out to be about 11 kilometers per second.

But for objects that are heavier or more compact, the escape velocity is larger. For example, for Jupiter it would be hundreds of kilometers a second. You can imagine objects that are so heavy or so compact that in fact the escape velocity becomes equal to the speed of light. So people idly wondered what would happen then. They didn’t have the tools to address it, but in the 18th century they wondered what would happen. You can calculate what the critical value of the velocity is. Just putting the velocity equal to the speed of light, this gives a critical radius of 2GM—the mass of the object cancels—divided by c2.

So that’s somewhat suggestive. If you had an object that was this compact and had this mass, the escape velocity would be given by c, the speed of light.

Dwarkesh Patel

The connection you’re pointing out is a Newtonian one. Did anyone make this connection before GR?

Adam Brown

Absolutely. People in the late 18th century wrote this formula down. I think both Michell and Laplace had this. And they said that if you had an object that was that massive and compact, light would not be able to escape. That reason is not particularly compelling by modern standards but it turns out to be particularly correct, including crazily this factor of 2, which is correct for completely coincidental reasons.

But let me give you a more compelling argument that something funny is going to happen around this radius. And to do that, let’s think about trying to extract energy from objects by lowering the objects down towards a central mass. Let’s start off perhaps with the Earth. Here it is. We’re going to start off a long way away from the Earth, with a brick of mass m. I’m going to take this brick, attach it to a pulley system, and then slowly lower the brick down towards the surface of the Earth and deposit it with zero velocity on the surface of the Earth down there.

In doing so, I can extract energy from the brick. I’ve extracted energy from the brick because there’s a force pulling the brick down. That force I’m doing through a certain distance, and that gives you an energy. We know, at least in Newtonian physics, what the formula for the amount of energy you can extract from the brick is: G times the mass of the Earth times the mass of the brick, divided by r, the radius away. It’s an energy, not force, so it’s an inverse distance law, not an inverse square distance law.

That’s the energy you can extract from the brick by lowering it down to a distance r away from the Earth. Of course, if you try to lower it beyond the surface of the Earth, this formula changes. But let’s just put it on the surface of the Earth.

So this is the amount of energy I have extracted from the brick, out here a long way away. You can ask, what fraction of the rest mass energy of the brick have I extracted? This is a question that you would only naturally ask once you’ve invented special relativity and know that the rest mass energy is given by mc2. We can straightforwardly calculate, at least in this approximation, that the fraction of the energy that you’ve extracted is that divided by the rest mass energy you started with. The mass of the brick, of course, is going to cancel, but not the mass of the Earth. This is going to be given by G times the mass of the Earth, divided by c2 times the radius away from the Earth at which you stop, the radius of the Earth.

So what fraction have I got out of it? If you lower down to the Earth’s surface, the answer is you haven’t really extracted that much from the brick. You’ve extracted a fraction 7x10-10 of the original rest mass energy of the brick, doing useful work a long way away.

First observation: this is small. In other words, the gravitational binding energy of something on the Earth’s surface is quite small in natural units. That’s why we didn’t really notice general relativity on the Earth’s surface until we did very sensitive experiments, because general relativity is in some sense a Taylor expansion in this number, where the relativistic effects, where the first order term is just Newtonian, and then the next order terms will give you the GR corrections to the Newtonian answer.

Observation number two, and this is something of a digression, is that by essentially sheer coincidence, this number here is very close to the chemical binding energy of rocket fuel. So if you take a rocket fuel like an oxygen-hydrogen mix, the chemical energy binding the rocket together, which is the energy that you’re going to extract when you burn it to make your rocket go, divided by the mc2 of the oxygen and hydrogen you’re going to mix together, is given by 1.5x10-10.

First observation: these two are close to each other, even though they came from completely different calculations. This was a gravitational calculation that was something to do with the Earth. This is a chemical property of hydrogen and oxygen. This is also very small. The reason it’s very small is that almost all of the energy in hydrogen and oxygen is not stored in the chemical binding energy of these things going together.

The vast majority of it is stored in just the rest mass energy of the protons and the neutrons, which chemical burning doesn’t affect at all. The second largest amount is stored in the nuclear binding energy of the protons and the neutrons to each other, given by the strong force and the weak force, which again chemical reactions don’t touch at all. This is a small number because chemical bonds are very weak compared to the rest mass of the things we’re considering.

These two small numbers are almost exactly equal to each other, which is why we can use chemical rockets to get to space, but it’s hard. In particular, this number is a few times bigger than this number, which means that your payload fraction is quite small when trying to use chemical rockets to get to space, because most of your fuel cannot get to orbit. You have to pay a rocket factor that’s going to tell you that most of what’s sitting there on the launch pad is going to have to be burnt up before you get to space, in order to get a small fraction of the rocket up to space. In other words, we can use chemical rockets to get to space in a way that would be totally impossible if we tried to do it from the surface of the sun, but it’s hard.

Okay, that’s the fraction on the Earth. But this formula tells you that if you have an object that’s heavier or more compact, the fraction of energy that you extract by lowering the object down to the surface is going to be larger. For example, if you lower it not down to the Earth’s surface but down to the sun’s surface, this would be larger. It’d be a million times larger, because the sun is a few million times the mass of the Earth, but then it’s also bigger, so that takes it away a little bit. You end up with 2x10-6, the famous redshift from the sun’s surface.

You can escalate from there. You can imagine cramming a sun-like mass into an Earth-like radius to make this formula even bigger. Sun mass, Earth radius. That’s pretty much exactly what happens in a white dwarf like Sirius B. And this would get even bigger again. A larger fraction of the mass of the object you’d be extracting by lowering it down to the surface.

But it really feels like something has to give before we make an object that is too massive and too compact. In particular, if you look at this formula, what happens for r less than or equal to GM over c2? If this object were so compact and so heavy that it had a radius less than the mass of the object divided by c2, it sure looks like you could get more than a hundred percent. The fraction would be bigger than one. You could get more than a hundred percent of the mass of your brick back by lowering it down to the surface of this object.

And that feels wrong. That feels, in fact, more wrong than what’s going on here, because now you’ve got all this energy a long way away. You could perhaps use it to make a whole new brick. You’ve got all this more than mc2 out there. Lower that one down, and it feels like we’ve figured out a way to make a huge amount of energy where there was no energy before.

This argument is pretty suggestive that something has to go wrong by the time you get down to that radius. Indeed, when you do the calculation—this is a Newtonian calculation, so it’s only suggestive —in full general relativity, indeed something does go wrong. The thing that goes wrong is that you form a black hole.

You can imagine two ways that you could avoid this conclusion. One would be somehow that gravity becomes very weak when you get close to a massive object, weaker than the Newtonian law would predict. That’s sort of what saves you if you try and repeat this same trick in electromagnetism, lowering a charge down towards another charge and trying to extract the electrostatic energy between them. What happens is, essentially due to quantum effects, when one gets too close to the other, they start to fuzz out. The energy going like inverse r gets softened, and you can’t extract more energy because they stop attracting each other so hard. So that’s one possibility, the force gets weaker than Newtonian law would predict as you approach the other object.

That’s actually the opposite of how general relativity resolves this. General relativity resolves this paradox by the force getting stronger than Newtonian law would predict. In particular, the force gets so strong when you try to get within this radius, that in fact you cannot slowly lower the brick down towards the surface because you’ve formed a black hole. The gravitational force becomes infinite at a finite distance away—not at r=0, but at some finite value of r—and the brick simply gets ripped out of your hand and you’re unable to extract any more energy out of it. That’s the resolution that general relativity provides to this paradox. In particular, you will find that you’ve formed a black hole.

So far, everything we’ve written down on the board is Newtonian. It’s just Newtonian, and you start plugging in the speed of light, and you start getting confused. To actually answer some of these questions that we’re asking, you need to go to general relativity, the theory that correctly unifies the speed of light with gravity. This was first done in the context of black holes by Schwarzschild, who wrote down the Schwarzschild metric that describes the gravitational field around a central mass, including potentially around a black hole.

Let me write down some of the formulas that emerge. In fact, I think I’m going to write down three formulas, three direct consequences of the Schwarzschild metric. They’re going to give us intuition for what it’s like outside and indeed inside a black hole.

The first formula I’m going to write down is the formula for the gravitational field that you would experience if you were trying to remain static outside a central mass. So let’s just talk about static observers. I can discuss how these will get upgraded for observers who are moving around. But for now, I’m just going to imagine that you’re trying to sit here at some fixed radius r away from the black hole. The reason you don’t fall in, maybe I’m lowering you down on a pulley. You’re just sitting here holding the pulley. The question is, how strong a force do you need to stop you falling down? You’re abseiling down very slowly. You’re static. What is the local force of gravity that you experience?

Or you can imagine that you’re sitting here, and the reason you’re static is that you’re firing a rocket very hard. The question is, how much acceleration do you locally feel? So by whatever mechanism, you’re remaining static. What is the local force of gravity that you feel? In Newtonian physics, you know what the answer to that question would be. The force of gravity is GM/r2, which is Newton’s famous inverse-square law. But this gets a correction from general relativity. The correction is 1/√(1-2GM/(c2r)), this same 2GM/c2 that we find all over the place.

What this tells you: first of all, if you’re a very long way away from the black hole, this here is essentially one. r is very big, and you get Newton’s force law back again. For the Earth, this is very small. As we discussed, it’s down by a factor of 10-10, and then you take the square root.

So you don’t really notice it, but you can Taylor-expand this at large r, and you find that you get corrections. You get an inverse-square law plus an inverse-cube law correction plus an inverse-fourth law correction. You find that gravity at short distances is stronger than it would have been in Newtonian physics. This is the general relativity correction and it’s making the gravitational field stronger. You have to accelerate harder to not fall into the black hole.

In particular, once r is equal to 2GM/c2, what’s called the Schwarzschild radius, you have to accelerate infinitely. The proper acceleration required to not move in r goes to infinity. In fact, if we now convert this Earth to a black hole, this is a very significant radius over here, 2GM/c2. It’s called the event horizon.

It’s called the event horizon because if you want to remain static outside the event horizon, further away from the event horizon, you just need to accelerate with some finite velocity in order to remain static. You need to have a finite gravitational field. But the gravitational field, as you approach the event horizon, becomes infinite. So once you’re at or beyond the event horizon, it is impossible to remain static. You will inevitably get sucked into the black hole no matter how hard you fire your rocket.

Now, this is just a static formula. You might imagine, “Okay, it’s impossible to remain static closer than that, but maybe I could avoid falling into the black hole by orbiting really, really fast. If I orbit really fast, I have a huge centrifugal force that pushes me away from the black hole, and I can stay out of the black hole that way.” That actually doesn’t work. The reason it doesn’t work is somewhat instructive for the way gravitational attraction happens in general relativity.

Of course, if you think about the International Space Station, why doesn’t it fall towards the Earth? It is precisely the fact that it’s orbiting. The fact that it’s orbiting gives it a centrifugal force that shoots the astronauts away from the Earth and precisely balances the gravitational field on the astronauts, which is why they feel weightless there. So orbital angular momentum, if you’re a long way away from the black hole, helps you stay away from the black hole, stops you falling in.

There is this kind of sci-fi notion that black holes just suck in everything around them. Not true. You are perfectly able to orbit around a black hole if you’re a long way away from it, just like you would orbit around any central mass. You are not inevitably falling into the black hole. You can orbit just fine. But orbiting stops helping when you get too close to the black hole.

We said that the event horizon is 2GM/c2. In fact, once you’re already within 3GM/c2, orbiting is counterproductive if you’re trying to stay away from the black hole. That’s because there are two effects of orbiting. One effect helps you stay away from the black hole. That’s the centrifugal effect. Orbital angular momentum pushes you away from the black hole due to the centrifugal effect. It’s not too hard to write down the version of this formula that applies when you have angular momentum, and you would see that pushing you away from the black hole.

But there’s another effect which drags you towards the black hole, and that is the fact that in general relativity, all energy gravitates, not just rest mass energy. Kinetic energy also gravitates. So the effect of orbiting is that you have an additional pull down towards the black hole from the coupling between the gravitational attraction between the mass of the black hole and your orbital angular energy.

When you’re far away from the black hole, the centrifugal force is the more important term. When you’re close to the black hole, that coupling is the more important term. In fact, once you get within 3GM, orbital angular momentum stops helping and starts hurting. There are no ballistic orbits that go within 3GM and manage to escape again.

So that’s formula number one. It tells you what the gravitational field is a distance r away from a black hole. In particular, it shows you that once you get to this critical radius, the gravitational field becomes infinite. If you cross that, you must proceed to the center of the black hole, no matter how hard you fire a rocket. That’s called the event horizon. At the event horizon, you are not yet dead. You are, however, doomed if you cross the event horizon. You will never be able to escape, not if you convert yourself to light and try to shoot yourself out, not if you fire your rocket infinitely hard.

The other place, of course, is r=0, which is where you actually die. That’s at the singularity, and we’ll describe that a little bit in a moment. In Newtonian physics, the gravitational force only becomes infinite there. In general relativity, it becomes infinite already at the event horizon if you try to resist the force of gravity. That’s formula number one.

Now let’s do formula number two. All three formulas I’m going to write down are heavily related to each other. They’re really going to be reformulations of each other. Formula number two asks about gravitational time dilation. Let’s again imagine that you’re sitting here, Dwarkesh sitting here some radius r away from the black hole. I’m sitting out here, way off at infinity, just watching you. We’re static relative to each other. There’s no relative motion. You’re just suspended here by your pulley system. The question is, how fast does your watch go relative to mine?

Of course, as far as you’re concerned, your watch is ticking at one second per second. As far as I’m concerned, my watch is ticking at one second per second. But if I look at you, I see your watch as running slow. If you look at me, you see my watch as running fast. The second formula makes that quantitative. How much slower does your wristwatch—which is closer to the black hole—run than mine? It says that the time interval, as measured by your wristwatch, is given by the time interval as measured by my wristwatch a long way away, times this exact same square root factor that’s showing up all over the place: the square root of 1-2GM/(rc2).

This factor here is less than one. So if I think one second has passed, you think less than one second has passed. In other words, if I slowly lower you down towards the black hole—you hang out some finite distance away from the black hole for what feels to you like a year, and then I raise you back up a long way away from the black hole—you will return to a world that has aged a lot more than you have. This formula makes that precise. I observe your wristwatch to be running slow. You observe my wristwatch to be running fast. Time passes slower down here than it does up here.

This is a fact that has by now been extremely well observed experimentally. In the 1950s, in the Harvard physics department, they put two atomic clocks at two different heights in the building, and noticed the one that was higher was running faster than the one that was lower. This is an effect that is now considerably within the precision of, for example, GPS. It just has to subtract that effect, otherwise everything would drift all over the place. GPS clocks that are sitting on the Earth’s surface are running slow compared to the atomic clocks that are in orbit sending out the signal. You have to account for that difference and subtract it off in order to get an accurate read.

This is known as gravitational time dilation. Notice it’s quite different from the relativistic time dilation you see in special relativity, which is caused by two objects being in motion relative to each other. Here, we’re not in motion relative to each other. We’re both static. We’re fixed. This is caused by us being at a different place in the gravitational potential, you deeper in the gravitational potential than me.

So those are two different sources of time dilation, and they stack. Let’s say instead of being static here, you’re in orbit. You’re far enough away that you can orbit the black hole. How slow do I see you as moving? There are now two contributions, both of which make you look slow relative to me. One contribution is the gravitational time dilation given by this formula. A second contribution is the good old special relativity correction where moving observers look like they’re going slow, and we’ll have both of those effects. So you’ll look like you’re going even slower than you would have done otherwise as you go around the black hole.

Dwarkesh Patel

One thing that seems different between this and special relativity is that there’s no symmetry. In special relativity, both observers will feel that the other one is aging slower than they are, because they’re both moving relative to each other at the same rate, and there’s no true inertial path. But here, it actually does seem like there’s a global sense in which one is a more relevant inertial frame than the other one.

Adam Brown

You’re exactly right. In special relativity, if you and I are moving relative to each other, I think your watch is moving slow, you think my watch is moving slow. Neither of us is more correct than the other. The principle of relativity tells you that both of our perspectives are equally valid.

Here, both of our perspectives are not equally valid, because there is not the symmetry that there was in special relativity. In particular, the symmetry is broken by the black hole. We both agree that you are deeper in the gravitational well than I am, and your clock runs slower than mine does. You do not see my clock reciprocally running slow. You, in fact, see me sped up. If you were observing me, you see me living my life in fast-forward. So this is the second formula. It says how fast our wristwatches move relative to each other.

Now let’s imagine that you’re here with your slow-moving wristwatch, and you shine a light towards me. Let’s say the light has a particular frequency. You made it with a sodium transition, for example. As that light travels upwards, by the time it reaches me, I’m going to think that it is lower frequency than you thought it was when you sent it. Why? Because frequency is about how rapidly it oscillates. I just think that everything you do is moving slow relative to me. You think it’s oscillating slower. It has lower frequency, which means it gets shifted towards the red part of the spectrum.

The word that we use is redshift, gravitational redshift. It’s redshifted: lower frequency, and therefore less energy. If you send one photon up, the energy of the photon is given by the frequency. It’ll arrive at me more redshifted and with lower energy than it had when it left you.

Conversely, if I am up here, and I send you a photon generated by the sodium transition, as observed by you, by the time the light reaches you—you see me moving in fast-forward—you think that it has a higher frequency than I thought it had when it left me. It’s moved towards the blue part of the spectrum. We say that it is blueshifted.

So this thought experiment tells you that knowing the exchange rate for how time passes at different altitudes directly gives you the exchange rate for how much energy is worth at different altitudes. If you try to send me some energy, by the time it reaches me, it’s worth less to me than you perceived it as being worth to you. The amount it’s less by is going to be precisely given by the same square root formula that’s controlling everything else.

So that gives us our third equation. The third formula says: suppose that you, Dwarkesh, have an object of mass, mc2, sitting with you at this fixed radius down there. How much energy, as measured by me a long way away from the black hole, is that worth to me? Of course, if I had it with me, it would be worth mc2 worth of energy. But I don’t have it with me. It’s unfortunately sitting with you deep in a gravitational potential. So it’s worth less than mc2 to me. In fact, it’s just the exact same formula. The amount of energy that it’s worth to me, by the time it reaches me, is GM/(rc2).

There are a couple of ways to see that. One is the way that we just said. Suppose you take your object of mass m. It’s just half an Avogadro’s number of carbon atoms and half an Avogadro’s number of anti-carbon atoms. One way you could send me the energy is by smashing them together, a violent explosion. You convert all of that energy to light and you try to beam that light energy up to me.

But what you find, precisely because of this gravitational time dilation, is that by the time it reaches me, I’m not getting mc2 worth out. I’m getting, by the argument we just gave, less than mc2 worth out. I’m getting 1-GM/(rc2) out. Mass down here suffers this redshifting as it goes up and has less energy by the time it reaches infinity than it did to begin with.

There is another way that you could have got the energy to me, not by beaming it up as light, but by just taking your mass object, attaching it to the pulley, and having me pull the object out. By the time I’ve pulled it out, I’ve now got mc2 sitting out here, a long way away from the black hole. So I do have the full mc2 worth of energy. But to get it, I needed to pay. What I needed to pay was precisely pulling it out of the gravitational potential. So from that way of thinking about it, that’s why I have less than mc2 worth of energy left, because I had to pay to pull it out of the potential in order to accrue that mass.

So this formula tells you: if I have a brick of mass, mc2, sitting at some radius r away from the black hole, how much energy can I extract from that brick if I’m a long way from the black hole? If we know that formula, then we can in fact calculate exactly this formula: how much energy have I extracted from the brick by lowering it down to a radius r?

Well, we know the answer to that question. The energy it started with is mc2. The energy it now has is this. So the energy I’ve extracted from the brick while slowly lowering it down using my pulley system must be the energy I started with, mc2, minus the energy it now has. In other words, the fraction of the energy that I’ve extracted by lowering it down to a radius r is mc2 minus this, all divided by mc2: 1 - √(1 - 2GM/(c2r)).

This is the exactly correct answer for the fraction of the energy extracted. It doesn’t look exactly like this, because this is only correct in the Newtonian limit. We derived this using Newtonian physics. This is exactly correct, not just in the Newtonian limit, but all the way to where the effects of general relativity are important.

Now, if you’re a very, very long way away from the black hole—r is much, much bigger than 2GM/c2—then you can Taylor-expand this formula. The first order term is just the old Newtonian formula. It better be. It better be that the long-distance limit of general relativity recovers the Newtonian physics that we originally discovered. But as you get closer and closer to the black hole, this starts to deviate from the Newtonian answer, in a way that exactly is going to end up resolving our original thought experiment to do with lowering a brick down towards a black hole.

So how much, then, looking at this formula, have I extracted from the brick as I lower it down towards the black hole? If r equals infinity—if the brick’s still a long way from the black hole—then I’ve extracted 1-1=0. I haven’t extracted any energy. As I lower it closer and closer to the black hole, initially I just get the Newtonian formula. So in fact, these are pretty close to correct in general relativity as well, because the corrections are only going to start getting large when this term becomes order one, and it’s still very small here. So these are all essentially correct. But once I get closer and closer to the black hole, they stop being correct. What I see is that as r approaches the black hole event horizon, as this formula goes to zero, I have extracted exactly all of the energy from the brick.

So I start off with a brick a very, very long way from the black hole, attach it to a rope, slowly lower the brick down towards the event horizon. Of course, I can’t lower it past the event horizon, otherwise I’ll lose control of the brick. But I lower it right above the event horizon—the last possible place I can lower it to—and then just let go of it with zero velocity. The brick falls into the black hole and I have extracted the entire mc2 that used to be in the brick in my pulley system out there. So it exactly resolves this question we had. Is it possible to extract more than mc2 from the brick? No. Is it possible to extract the full mc2 from the brick using a black hole? Yes, it is.

That’s actually pretty neat, and why people talk about using black holes as power plants. Most power plants today operate by burning chemical energy. That is not very efficient. You have to pay a factor of 10-10, because chemical bonds are super weak compared to the rest masses of objects. You’re really only extracting a tiny fraction of the rest mass of the fuel that you’re considering. You can level up from there by going to nuclear energy, which instead of dealing with the feeble electromagnetic bonds between atoms, starts to concern itself with the nuclear forces between the protons and the neutrons within the nucleus. So you can go up from about 10-10 to about 10-3 for fission, or 10-2 for fusion.

But that’s about as good as you can go, even with fission and fusion. Because even though you can extract energy from the strong nuclear force, neither fission nor fusion changes the total number of protons plus neutrons in your process. The bulk of the energy—99% of the energy—is stored not in the electromagnetic interaction, not in the strong interaction, but in the rest mass energy of the protons and neutrons, something that neither chemical reactions nor nuclear reactions can touch.

But gravity can touch them. If I start off with a mass object of m, I can extract, up to quantum corrections, essentially 100% of the rest mass energy that I’ve gone in with. It is the most efficient possible power plant, because by building an apparatus like this, in principle, I could extract 100% of the energy of whatever I started with.

Dwarkesh Patel

I intuitively get how energy equals mass. There’s these chemical bonds. Those get dissolved, they release energy. The thing weighs less if those bonds are released. I even get that if the bonds between the protons and the neutrons are broken, that releases energy and makes the thing have less mass. But if something with protons and neutrons is just slightly above the event horizon, is the interpretation that those protons and neutrons stop existing right at that point? What does it even mean for them to have 1% or 2% or 5% of their original mass?

Adam Brown

That’s a great question. It really becomes relevant once you turn on quantum mechanics, which is beyond the scope of today’s discussion. Classically, the black hole just sits there forever. So you can just say, “Well, what happened to the protons and neutrons?” You say, “Well, they now live inside the black hole.” The number of protons plus neutrons is still conserved out there in the universe. It’s just you need to assign what’s called a nucleon number to the black hole itself. That’s fine as far as it goes classically.

Quantum mechanically—way beyond the scope of today’s lecture—Hawking and Bekenstein discovered that black holes radiate away energy, and eventually the black hole will be gone. All of the energy, if you calculate it, ends up in gravitons and photons and perhaps some neutrinos. None of it, or almost none of it, ends up in protons and neutrons. So it is a very interesting fact, once you turn on quantum gravity, that black holes eat nucleon number. This thing that seems like it’s conserved, at least perturbatively, both by electromagnetism and by the nuclear forces, ends up being eaten by gravity.

People like to promote this—we’re talking about quantum gravity now—to a general principle that quantum gravity doesn’t respect any global symmetries. It doesn’t respect nucleon number symmetry. It doesn’t respect any of these symmetries. That’s a whole other can of worms that we can open some other day.

Dwarkesh Patel

Okay, Adam. I like to think on this podcast we impart not only theoretical but practical knowledge as well. So suppose one learns all these equations, but then finds themself in the unfortunate position of falling into a black hole. What would they see?

Adam Brown

Great question. There are actually two different perspectives you could take. One is the perspective of me watching you falling into the black hole. The other is the perspective of you falling into the black hole. Those two perspectives are consistent with each other but interestingly different, so maybe I should describe them both.

First, let’s ask the question: what do I see as you fall into the black hole? This is how hard you need to fire your rocket to not fall into the black hole but you’re not going to do this. You’re just going to sit here a long, long way away from the black hole, turn off your rocket, and accept what comes. What comes is you’ll slowly accelerate towards the black hole, at a rate first given by the Newtonian formula and then, when you get close to the black hole, start picking up general relativity corrections to the Newtonian inverse-square law.

What I will see as I watch you fall towards the black hole is that first you’ll go faster and faster and faster as you fall down the gravitational potential of the black hole. But then something strange will happen. You’ll stop going faster, and you’ll start going slower. The reason you’re going slower is that, as I watch you, you start to get gravitational time dilation as you fall down, and I start to see your clock running slow. The static formula doesn’t apply exactly since you’re moving, but the formula has the same effect, which is that as you get closer and closer to the black hole, your wristwatch starts running slower and slower and slower.

In fact, if you do the appropriate integral, I never see you cross the event horizon. I just see you getting closer and closer to the event horizon, but slowing and slowing as you approach it. As I watch you—I’m presumably using light to watch you—that light gets more and more redshifted. The wavelength gets longer and longer, and the longer the wavelength of light, the harder it is to even really see you. You start getting delocalized by the wavelength of the light, and eventually I just stop seeing you entirely. There’s a final photon that you emit, and then you just fade to black, fade through red to black. I never see you cross the event horizon.

This was noticed by people in the early days of general relativity and greatly confused them. They started to think that you would experience something funny yourself as you fell across the event horizon. That is not true. If I instead adopt your perspective, from your point of view, your clock isn’t running slow. It’s running at one second per second. If you look back at me, there’s some funny stuff going on to do with me running fast perhaps. But as far as you’re concerned, everything’s totally normal. You accelerate towards the black hole, getting faster and faster as you approach it. You just sail across the event horizon totally as normal.

The event horizon is not a particularly violent place for you. You can calculate the tidal forces as you approach and then cross the event horizon. They’re not particularly big, or rather, for large black holes, they’re not particularly big. For a solar mass black hole, they would be pretty big and would be pretty painful. You’d find that your feet are being attracted to the black hole much more vigorously than your head is, because they’re closer, and you end up getting stretched. But if I take a big enough black hole, you wouldn’t notice anything funny happening whatsoever. The bigger the black hole, the smaller the tidal effects. If I took a black hole the mass of the galaxy, you’d be basically fine as you cross the event horizon. If I took an even bigger black hole than that, you could live out your entire life having crossed the event horizon, before you hit the singularity, which is fatal.

When you cross the event horizon, you are doomed. You are doomed because once you cross the event horizon, you must proceed to the singularity. There’s no way you can fire a rocket to stop yourself hitting the singularity. You are doomed, but you are not dead. You are only for sure dead once you hit the singularity and get spaghettified, mangled by the tidal forces. But for a large enough black hole, you can be doomed and not even know it.

The event horizon is really a not locally measurable quantity. It is a teleological fact. It says that once you have crossed the event horizon, you must proceed to the singularity. But it can take a long time to get there for a large enough black hole. In principle, for a black hole that was many light centuries across, you could live out your entire life. You could have descendants, all of whom live inside the black hole. Only once you really approach the singularity do the tidal forces get strong and kill you.

Dwarkesh Patel

As you were saying, GR explains or predicts a lot of phenomena. Some we think are correct, some we don’t know are correct. Why do we think black holes are correct but not wormholes?

Adam Brown

That’s a great question. People did not believe it to begin with. Schwarzschild wrote down his solution almost immediately after Einstein wrote his field equations. People thought that that equation was sick in some way, that it was a measure zero thing that would never happen. It was some mathematical monstrosity, but it was impossible to make black holes naturally in the real universe. They were wrong, because black holes do exist. We’re extremely confident now.

There were theoretical developments, and there was experimental evidence that black holes exist. The biggest theoretical development was Penrose, and then later Hawking and Penrose—for which he won the Nobel Prize—who showed theoretically that the formation of black holes is a generic feature of general relativity. It’s not just some sick thing that happens if you fine-tune the initial conditions. If you start off with generic initial conditions, the development of black holes is a generic feature. That was a huge development.

Then there was the experimental side. The pieces of experimental evidence we have for black holes are now huge. They did not exist in Einstein’s day, and for 50 years after Einstein, people were extremely confused about black holes and thought they didn’t exist. But there are numerous pieces of evidence.

I think the most visually appealing piece of evidence is just observing the center of our galaxy. If you look at the center of the galaxy—spoiler alert—there is a black hole there. We call it Sagittarius A*. It’s a huge black hole, weighing many millions of times the mass of the sun. You can’t see the black hole directly, because it’s black.

What you can see is the stars around it. If you watch these stars over the course of decades—and we now have a number of decades of observations of them—you will see the stars not moving along what we would call straight lines, but instead moving in nice little ellipses, or precessing ellipses. Those ellipses look like they are orbiting something. You cannot see the something, but you can see the stars that are orbiting it. You can calculate how big it is, how massive it is.

What you find is that it’s very massive indeed, and it’s also very small indeed. You know it’s small because the stars get super close to it but don’t seem to collide with it. So by tracing these orbits, you can tell that there is something super heavy, super dark, and super compact at the center of the galaxy. That is Sagittarius A*, the black hole at the center of our galaxy. That’s one compelling piece of evidence.

Another piece of compelling evidence: about a decade ago, we not only saw black holes, we felt them. LIGO is this huge laser interferometer that we built at a number of different sites, that is super attuned to vibrations in spacetime itself. There’s a famous event pretty much immediately after we turned it on in late 2015, where we felt spacetime shaking.

You knew it was spacetime shaking, not just the Earth shaking, because we had a bunch of these detectors—then two, now four—at different points on the Earth, and they all shook in exactly the same way. So it couldn’t just be explained by a truck passing one and not the other, or a seismic event on one and not the other. They all shook in exactly the same way, and we were able to back-calculate that the thing causing them to shake was the collision of two ginormous black holes. Two black holes, both of which weighed about 30 times as much as the sun, on the other side of the universe, about 1.6 billion light-years away. That collision happened about 1.6 billion years ago, and just happened to reach the Earth within weeks of us turning on the LIGO detectors. We’ve now felt thousands of such shakings corresponding to thousands of black hole mergers.

And then there’s more evidence. Later, we had what’s called the Event Horizon Telescope, which is a ginormous conglomeration of radio telescopes all over the Earth, that were able to look very closely at the black hole at the center of our galaxy, Sagittarius A*, and the even bigger black hole at the center of our neighboring galaxy, and see, very faintly, the radio emissions of matter falling into these black holes, which shines super brightly as it does so.

So we felt them, we’ve seen them, and we’ve seen their gravitational effects on orbiting stars. We’re extremely confident at this stage that black holes exist.

Dwarkesh Patel

It’s so beautiful that not only can a single mind come up with this theory, but the theory has so much reach, and that we can come up with the machinery to evaluate and perturb and understand its implications in so many different wild ways.

Adam Brown

It’s crazy the number of degrees of freedom, the number of orders of magnitude that it covers. You first start thinking about it by doing thought experiments to do with jumping up and down in elevators, and then it reaches out to describe the orbit of Mercury and detectable perturbations of orbital dynamics within the Solar System, and then the bending of light, and then it describes the rotation of the entire galaxy, and then it describes the expansion and potential fate of the entire universe. That’s many orders of magnitude indeed, and it’s pretty impressive that it was the work of almost a single mind. Frankly, our universe should be honored to be described by such a beautiful theory.

Dwarkesh Patel

Can you tell the story of how GR went from a theory that Einstein had to something that the world came to believe is true?

Adam Brown

That would be the bending of light. There were known anomalies with Newton’s physics beforehand, like we couldn’t get the orbit of Mercury exactly right. One of the very nice early tests of general relativity is that it did get the orbit of Mercury exactly right. So that was a pretty good confirmation. But at the same time, that’s not quite so satisfying because it was a number that’s already known, as opposed to one where you invent a theory and then it correctly predicts. It’s considered more impressive if you get the right answer without knowing what the right answer is in advance. So that would be the bending of light. Certainly, historically, that was the most influential.

According to general relativity, all energy gravitates and all energy is affected by gravity. So light, as it’s passing a massive object like the Sun, will get bent in the direction of the Sun. Actually, the same will happen in Newtonian physics. Suppose you have a particle going along. You know how much it gets bent as it passes the Sun, depending on its impact parameter, but also its velocity. The faster it’s going, the less it gets bent. So you just take that Newtonian formula, plug in velocity equal to speed of light, and see what answer you get. You can get a certain amount of bending through Newtonian physics. In general relativity, you can do the same calculation, and you actually get double the Newtonian answer.

The slightly strange history of it. Before he had finished writing down general relativity, Einstein had a prediction based on his understanding of the equivalence principle for what this answer should be. He wrote down the answer, and then in response to him and a number of other people being interested in this, people were sending out expeditions to go and try and measure it.

I think the very first thing that he did was he phoned up the observatory and said, “Can you look at distant stars behind the Sun and measure how light bends as it passes the Sun?” This was the true theorist move, because I think the director of the Mount Wilson Observatory said, “Absolutely not. We cannot do that. If you point a telescope at the Sun, you’ll go blind. If you point it just next to the Sun, you’ll just get washed out by the corona of the Sun and you won’t see anything.”

Except there’s one time when you won’t get washed out by the Sun, and that’s during a total solar eclipse, when the Moon blocks the Sun and you’re able to see stars very close to the Sun and measure the bending of the light behind them. So during the 1910s, there were a whole bunch of expeditions sent to measure the deflection of light. They’d park out in the path of totality and look through telescopes at the stars right next to the Sun and see if they moved in the sky, and if they moved, how much they moved.

I think the first one was in 1911. They went to Argentina for an eclipse, and everything’s set up. This is the problem with this thing. You get there, all the way to Argentina, a very long way in those days, and then it’s just washed out by the clouds. You don’t see anything, and it’s very frustrating. The next one was a German expedition sponsored by the arms manufacturer Krupp, who went to the Crimea and tried to measure it there. Just before the solar eclipse happens, World War I breaks out, and now Germany and Russia are at war. They’re all arrested and turned for the rest of the war, and so that also fails.

It actually turns out to be a good thing for Einstein that they all failed. It turned out that Einstein’s original equivalence principle argument—before he had full general relativity—was wrong and led him to predict that the bending of light in general relativity would be the same as it was in Newtonian physics. During the war, while everything is shut down and no one is thinking about eclipse expeditions, he corrects this mistake and comes up with a new prediction that actually it’ll be double the Newtonian prediction.

And then in 1919, Sir Arthur Eddington launches a British expedition to go and observe the eclipses all over the world and successfully comes back and declares that indeed it was the Einstein prediction. It was double the Newtonian prediction. That’s really what launches Einstein as a global celebrity. This British experiment confirming a German-origin theory was part of the post-war reconciliation, and Einstein had figured out everything. That is, I’d say, the point at which general relativity became the consensus view, and people were super convinced by this very impressive test.

Nowadays, we’ve done hugely more tests than that, very precise orbital dynamics. You can see it in the orbit of Mercury and indeed even in the other planets. You can just measure the redshifting of light as it goes, the gravitational effect on the propagation of light or the energy of light, all over the place. But historically, that was the most impressive confirmation of general relativity.

Dwarkesh Patel

One question you could ask is, we are maybe spending, as a society, billions, maybe tens of billions of dollars on building these huge physics experiments. If you look at maybe the most beautiful, the most important theory of physics ever conjured, it seems like a guy who’s just thinking in a cave. It seems like the empirical basis for that theory is maybe knowing that light has a speed, and maybe you need to measure G experimentally.

Adam Brown

Not really. G is a free parameter in general relativity. It’s not required. You’re right, the empirical basis is pretty thin. You don’t need much, and theoretical physicists are pretty cheap. There’s a great temptation. Why don’t we just spend it all on theoretical physicists and not build these vastly expensive experiments?

Dwarkesh Patel

Well, these AI companies are really increasing the demand curve for theoretical physicists.

Adam Brown

That’s right, not so cheap anymore. But how far can that get you? I would say that general relativity is perhaps one extreme of that. That is not how it usually works in the history of physics. This really is closer to some Ayn Rand hero just sitting alone, the product of a single mind. He got a lot of help in various ways, but it really was a singular vision that he pursued for years.

He wrote it down, and a lot of people were very impressed almost immediately. It did require launching a somewhat expensive eclipse expedition to go confirm it before he really achieved global celebrity and most people were sold on it. But it’s perhaps one of the most extreme examples of this, where somebody just sits down and thinks very hard and writes down a true theory.

In some sense, physics has been chasing that high ever since. People love that romantic vision of themselves just sitting down with very few empirical insights and thinking very, very hard and doing thought experiments. It’s typically not worked out quite as well for everybody else as it worked out for Einstein. In fact, it didn’t even work out that well for Einstein in the later part of his career.

How far could you get just by thinking? What do you need to do general relativity? You need the finiteness of the speed of light. You need to convince yourself not just that the speed of light is finite, but that there’s the symmetry that protects that, which Einstein came up with in special relativity. Then you probably want the equivalence principle: it’s an empirical fact that the inertial mass and the gravitational mass are the same for everything. But that’s pretty sparse. From just those two things…

There’s still a few options. But if you have lots and lots of large language models and there’s only a limited number of options, you can just explore the entire tree and say, “Okay, focus on this. The equivalence principle is something that’s super significant, and this other thing. Now abandon simultaneity and see how far that takes you.” There’s only a finite number of things to explore there.

I think they got very, very lucky with general relativity, that it’s quite so powerful under those circumstances. But if you just had lots and lots of Einsteins and you gave each of them various options, you could presumably see them in parallel.

Dwarkesh Patel

At the frontiers of physics today, in your experience, does it feel like if you just have millions of them running autonomously you could have enormous discoveries, or are we in a different era now, and really there’s limited usefulness of that parallelism?

Adam Brown

I think there is usefulness. I do think that different parts of science have different branching fractions, and how much experiment you need to cut off that branch.

I talked about chasing the high of Einstein. Arguably, string theory has really been going all in on that. Einstein’s theory is just general relativity. There’s also quantum mechanics. Trying to marry those two in a consistent way—which general relativity doesn’t do at all, there’s no quantum mechanics in general relativity—has motivated a lot of people. The problem is that in order to see that in experiments, if you just do the dimensional analysis, you need ginormous particle colliders, absolutely galactic-sized particle colliders. It’s just very hard to see any of that stuff.

But that doesn’t stop people. I mean, it stopped many people, but many people didn’t stop and keep trying to do it. So there you just have to hope that it works out sort of like it did with general relativity, where just by thinking very, very hard with minimal input from experiment, you can feel your way to the right answer.

For that to be true, the tool you have at your disposal is mathematical consistency and whether it reduces correctly in the known limits. So you better hope that there’s only one or a very small number of possible consistent theories if you were going to do that. If it turns out that there’s an unlimited number of consistent theories, you’re never going to feel your way to the correct answer, because they’re all consistent, and the only tool you have is consistency, and perhaps some notion of aesthetics. But if there’s only a few, then maybe you could do it all the way.

So string theory has kind of gone all in on that, I would say. Trying, with minimal experimental input, believing that there’s only one consistent theory of gravity and that just by doing sufficient consistency checks you can find it. For other examples it’s much harder. In condensed matter physics, often you simply need to go and do an experiment to find out which one is correct.

Dwarkesh Patel

Whenever our future AI civilization does come up with more and more unified theories of physics, or deeper theories that make better predictions, do you think that humans will be able to keep up? Once this step is taken, will we actually be in a position to understand what our AI civilization understands?

Adam Brown

I don’t know that we’re going to be able to keep up entirely, but I think we’ll keep up much better than pessimistic forecasts would suggest.

Let’s take mathematics as a simpler example than physics. Many mathematicians are worried that these LLMs are just going to turn into proof machines. Terry Tao has this phrase, “indigestion”, he uses, in which these LLMs will produce billion-line inscrutable Lean code that will serve as a certificate that a particular theorem is true without providing any insight as to why that might be true. Wouldn’t that be a depressing world, say the mathematicians.

I think that is a possible future, but I actually don’t find that to be a very likely future. Because as well as being superhuman provers, we also expect these large language models to be superhuman explainers. Maybe they’ll do the exact opposite of that. Maybe they’ll take proofs that are very hard to understand, and by doggedly trying and trying and trying, they will be able to come up with ways that are human comprehensible. They will take proofs that are difficult to understand and make them easy to understand.

I think the empirical evidence, it’s early days, but it’s pretty supportive of that more positive vision. There was an Erdős problem that was proved a few months ago now, and it wasn’t just an incomprehensible set of Lean. In fact, it wasn’t even proved in Lean at all. It was proved informally.

There was a follow-up paper by some human mathematicians that took these new, human interpretable ideas that the machine had come up with to prove this Erdős conjecture, and used them to prove new theorems. So that was the exact opposite of that case. It came up with a very human interpretable idea, and then humans were able to fully comprehend it, comprehend it so well they were able to deploy it in a new scenario.

We’ve seen that throughout. The unit distance conjecture, I think, is a good example here for a number of the themes you’ve been discussing. One is that it’s totally comprehensible, the disproof of the unit distance conjecture that it came up with.

Dwarkesh Patel

To you, perhaps.

Adam Brown

To some mathematicians. I’m not a mathematician. Perhaps the reason that humans haven’t disproved the unit distance conjecture is because they erroneously believe the conjecture to be true. The good thing about large language models is that they’re willing to push through that barrier and just waste their time, as a human would understand it, trying to disprove a presumed true conjecture and reach the other end. So that’s another aspect of large language models that makes me pretty optimistic. They just have extreme patience, even for doing things that perhaps look like a low probability of success.

Dwarkesh Patel

Adam, thanks so much for coming back on and doing this while you could be building superintelligence. You’re explaining 100-year-old physics, but it was very interesting.

Adam Brown

It’s a super fun subject. I’m super happy to share it.

Two months ago, I posted some big questions about AI. We ended up getting over 600 essays submitted for this contest. Below is a bit of information of the 3 winners, followed by all 3 full essays. Thanks to everyone who participated!

First Place - Jassi Pannu

Jassi Pannu is an Assistant Professor at Johns Hopkins University, where she focuses on biosecurity and pandemic preparedness. She serves on the board of Blueprint Biosecurity.

Jassi answered the question about what the OpenAI Foundation should do. She persuasively argues that we can live in a post-disease world, and gave very concrete and well thought out ideas about how to dedicate 10s of billions of dollars to that project.

Second Place - Ege Erdil

Ege Erdil is a co-founder of Mechanize, a startup building environments and evals for frontier coding agents. He was previously a researcher at Epoch AI.

Ege answered the question about what countries outside the AI supply chain should do to avoid increase their odds of not being totally sidestepped by transformative growth.

He argues that these countries should concentrate on enacting the kinds of policies that already work well in increasing growth and improving productivity. These strategies (strong property rights, low capital taxes, and an open regulatory regime) will be even more important in a world where enacting them can drive a much higher growth differential than is possible today.

What I love about Ege’s essay is that, in one sense, he’s giving very common-sense advice (as opposed to much more galaxy-brain schemes some other applicants proposed - one application suggested middle countries blackmail China and American by threatening to nuke their fabs and datacenters). But it’s actually this much more grounded and timeless advice that felt the most contrarian. And it’s also more likely to work.

Third Place - Michael Li

Michael Li is a Master of Public Policy candidate at Harvard Kennedy School. He writes Ceteris Paribus — a blog at the intersection of emerging tech, econ and policy.

Michael wrote about how the labs will actually make money. His was selected for the unique analogy he drew between AI labs and Hong Kong’s Mass Transit Railway business model - even if your main product consumes crazy CapEx and doesn’t directly earn it back, maybe you can make up for it by buying out all the complementary assets. In the case of Hong Kong MTR, that would be the adjacent properties - I don’t know what it looks like for the AI labs, but it was a interesting analogy to think about.

I’d run the Foundation as a state-scale operation to end airborne transmission.

AI’s largest welfare upsides (curing diseases) and deadliest tail risks (engineered pandemics) both run through biology. By radically suppressing airborne pathogen transmission, we’d unlock >$1T in annual global GDP (through ending seasonal flu and the like, chronic diseases increasingly linked to viral infections, productivity losses, healthcare costs, etc.) and would take the possibility of catastrophic pandemics entirely off the table.

The dual-payoff principle: Most “make AI go well” interventions are insurance against bad outcomes, especially tail risks. My meta-level argument is that the best way of converting money into impact is to identify interventions that have the property of paying off big in both worlds: by producing step-changes in welfare in the everyday world as well as significantly reducing tail-risks in the emergency world. The bio resilience interventions I describe below are the best example of this.

AI for biology is on the critical path to cures, but destabilizing capabilities will arise early

Using AI to automate and scale every step in the biological research process, including managing the process itself (something I’ll call autonomous biological discovery), will bring humanity closer to a post-disease world. Over 4 billion years, life has been doing a random walk on an astronomically tiny subset of viable, connected, fitness-positive paths. Multi-component AI feedback loops (that include bio foundation models and systematic wet-lab experimentation at scale) for autonomous discovery will enable us to explore much more of possible biological design space. While we’re most interested in predicting and designing multicellular systems, it’s likely that the destabilizing capability of manipulating simpler pathogens will emerge first. The challenge this poses is that AI-enabled offense (seeding an outbreak) will be much easier than defense, which will remain constrained by physical-world deployment; I argue this advantages pre-positioned defensive technologies already embedded in our infrastructure.

There’s a clear path to ending airborne transmission, using physical infrastructure.

Regardless of what you think about the above, though, ending airborne transmission can be more than justified based on everyday benefits. Respiratory infections cause acute illness and productivity losses, but are increasingly linked to dementia, cardiovascular disease, and more; even “normal” childhood respiratory infections are being linked to long-term neurodevelopmental outcomes.

After evaluating many approaches, I’d argue ending airborne transmission is more achievable than most realize, through a specific, under-appreciated approach. I’m currently sitting in a building that provides me with pathogen-free water, keeps my food cold and pathogen-free, helps me heat my food to eliminate pathogens, and pipes away sewage. We have already embedded technologies all around us that enable a post-cholera, post-typhoid, post-dysentery world.

There is passive, pathogen-agnostic (works against any pathogen), physical infrastructure tech capable of making our buildings entirely free of respiratory pathogens, such as lamps that emit wavelengths safe for humans but are deadly for bugs (called far-UVC). Researchers have suspected these would work at scale for decades; the reasons we haven’t deployed them are primarily non-technological. Consider this analogical case.

We now live in a post-smallpox world. This is one of humanity’s greatest accomplishments. How long did it take for us to do this? Jenner demonstrated vaccination could prevent smallpox in 1796. 171 years later, in 1967, D.A. Henderson launched the campaign that would successfully eradicate smallpox. In that period of time, humanity discovered electromagnetism, thermodynamics, general relativity, and we were 2 years from landing on the moon. Eradication was accomplished within a mere 10 years (with limited tech advances). Delays in smallpox eradication, clean water, and pasteurized milk were not due to lack of tech advancement; they were primarily market and coordination failures exacerbated by lack of political will. This is why this problem is so philanthropy-shaped.

4 steps to ending airborne transmission

Total: ~$40-$60B over 10 years for physical infrastructure to end airborne transmission; the rest of OAIF’s stake remaining for other interventions meeting the dual-payoff principle. By year 10, every primary school and major transport hub in OECD countries operates with passive pathogen-reduction infrastructure as default. Seasonal flu mortality is reduced by 60%. The probability of a respiratory pathogen achieving pandemic-scale spread is reduced by an order of magnitude.

1. Push-funding to resolve the target product profile ($5B, Years 1-3)→ Hire Jacob Swett, director of Blueprint Biosecurity, to lead a DARPA-style program office focused on: a) pathogen inactivation data from human aerosols, b) computational modeling for deployment, c) safety studies beyond conventional UV effects, d) gold standard cluster-randomized trials powered to detect plausible effect sizes. By the end of year 3, deliver a validated TPP for far-UVC lamps and real-world efficacy data demonstrating >30% transmission reduction.

2. AMCs to guarantee demand and pull private capital ($15B, Years 1-5)→ Create laddered purchase commitments for (a) 100K far-UVC fixtures that meet an interim TPP, (b) 1M for fixtures meeting the full TPP including safety and efficacy validation, (c) 10M commitment to retrofit specific buildings (see step 3). Modeled on Kremer’s pneumococcal vaccine AMC and Ransohoff/Frontier’s carbon removal commitments. By year 5, expectation is $30-50B in private capital mobilized, and supply chain capacity built to retrofit ~10% of global building stock.

3. Large-scale deployments to generate evidence ($15-25B, Years 2-7)→ Years 2-4: Deploy in all hospitals and long-term care facilities in 50 largest metro areas globally. Years 3-6: Primary and secondary schools in the same metro areas. Years 2-7: Major airports and high-density workplaces. By year 7, substantial real-world evidence base.

4. Political infrastructure and state handoff ($3-5B, Years 1-10)→ Smallpox eradication was a genuinely contingent historical event predicated on political will. Fund a memetic campaign à la the Rockefeller Foundation’s IHD yellow fever playbook which would transform respiratory transmission from “normal” to undesirable/unnecessary, and the training of a cadre of thousands that move into governments to build the political constituency and institutional infrastructure needed for global deployment and standards-setting. When pilot deployments demonstrate ≥40% reduction across 3 OECD countries, the Foundation transitions to catalyst rather than principal funder role, activating state procurement at orders-of-magnitude scale.

I think a good baseline which very few countries, in the AI supply chain or not, will beat is “do nothing and ignore populist pressures to take radical actions”.

This is because people are naturally technologically conservative, and they hate economic disruption that causes job losses. Full automation of human labor by AI would bring about rapid technological and economic progress that would result in all humans losing their jobs. So the default expectation should be that policymaking in an era of AI automation will be profoundly irrational and counterproductive.

Resisting this political pressure will be hard enough for even the most functional governments. Expecting more from governments with poor track records such as India and Nigeria is unreasonable.

What does good policy in the AI era look like?

Having given this basic but uninspiring answer, I’ll now flesh out what I actually think will be important for good policymaking in the era of AI automation, though in practice it’s unlikely to have much relevance for how policy decisions will be taken.

Today, the economic output of a country depends on its endowment of natural resources, on how much physical and human capital it has, and how efficiently it’s able to make use of these resources. The major shift with AI will be that human capital will drop out of this equation. If a country wants to do well in a world after full automation of labor; they need more natural resources, more capital, or more ability to make use of these inputs effectively, i.e. more total factor productivity.

While the capital elasticity of output will dominate any other factor of production, how much capital a country ends up with is itself endogenous. Capital moves across borders more easily than labor, and it will likely flow to the places where factors complementary to it – total factor productivity and natural resources – are abundant.

So I think the pillars of good policy in the AI era involve going in the following directions, to whatever extent possible:

• Get out of the way of AI automation. Repeal or revise occupational licensing laws, liability laws, data protection laws, and intellectual property. Abolish price and wage controls. Dismantle cartels and unions of human workers who will try to protect themselves from AI competition. Apply safety and security standards consistently between humans and AIs, instead of discriminating in favor of humans. Make it much easier to start new businesses, as AIs will need organizational structures just as much as humans do in order to bring out their productive potential.

• Provide political and legal stability. Investment will be anemic under instability, and if a country can stand out as an island of stability in what will likely be tumultuous decades for the world, it will attract enormous amounts of investment. The lowest bar is avoiding civil wars, revolutions or coups (not trivial for Nigeria to clear, since there was a serious military conspiracy to overthrow Tinubu as recently as September 2025); but this is not enough by itself. People must be confident that their investments will not be expropriated, and that the core operation of their businesses won’t suddenly be declared illegal.

• Increase capital formation in your country. Reduce or eliminate taxes on capital gains and corporate income, don’t let important projects be held up by permitting, don’t hamper construction by requirements of “having to pay prevailing wages”. Remove exchange, interest rate, and capital controls.

• If industries necessary for AI deployment are broken, urgently fix them. For example, Nigeria’s grid sucks and needs to be fixed before anything else can happen, and this is downstream of the entire economy’s price system being messed up by government restrictions. If this isn’t fixed, the country will never be able to accumulate the kind of capital it needs to be productive in an era of AI automation.

These recommendations feel extreme because humans instinctively like policies that favor labor over capital. This already causes problems in our world where capital’s output share is only around 30%, but it will likely become a critical obstacle in a world where labor has been made obsolete and capital’s output share is much higher.

If a country that’s currently in bad shape succeeds in doing these, they would attract a ridiculous amount of outside investment and be transformed in the decades in which AI automates the world economy. The fact that they had no previous AI labs or fab companies with trillion dollar valuations would be irrelevant, because those very companies would at that moment be in the process of automating their moat – their human capital and organizational knowledge – away.

Will this happen?

Probably not. As I’ve said, the relevant countries are far too dysfunctional for these radical reforms to be adopted, and the countries currently in the AI supply chain (e.g. the US and China) already perform well relative to other countries on the metrics I’ve listed.

However, there’s still some hope for countries outside the AI supply chain: the incumbents can screw up. This is not that implausible, because policy quality in the industrial era will not be that strongly correlated with policy quality in the AI era, just like how the best foraging economies in the world weren’t the best agrarian ones and the best agrarian economies weren’t the best industrial ones.

Concretely, the incumbents can outlaw AI automation in key industries, slow down datacenter buildouts, expropriate capital holders, impose strict safety standards for rolling out AI agents in critical industries like medicine and law, et cetera. There’s virtually no limit to the mistakes that they can make in the coming period of acute irrationality. If a mediocre country simply holds on and doesn’t make any big blunders, they will probably come out the other side of this in a much better relative position than they had been in at the start.

A subway company solved AI’s business model

There is an industry with the following economics: billions in upfront capital before earning a dollar. Core service priced near marginal cost. Enormous value created for users, almost none captured by the builder. Relentless pressure to keep investing in the next generation of infrastructure. I’m not talking about AI labs. I’m talking about Hong Kong’s Mass Transit Railway.

Many have reached for the railroads analogy when discussing the business of AI. Most conclude that the lesson is commercial viability requires state subsidy for a general purpose technology with public good properties.

I want to challenge that, because Hong Kong’s MTR actually solved the problem. It’s one of the only mass transit systems in the world that is commercially self-sustaining, publicly listed, paying dividends with no government operating subsidy.

The financials are structurally identical

MTR’s core rail service has never funded its own expansion. In 2018, its best pre-covid year, transport operations earned HK$2.0 billion in EBIT. Estimated capital expenditure for 2024–2026 is HK$87.9 billion, nearly all rail-related. Three years of peak rail earnings would cover 8% of that. The rail has never paid for itself through fares. It was never designed to.

MTR fares are kept affordable through a government fare adjustment mechanism. You can’t price transit to recover full construction costs because it would be unaffordable and defeat the purpose. Each rail line can maybe cover its operating costs, but fare revenue never stretches to fund the next line. AI API pricing faces the same constraint from the other direction. Distillation and open source alternatives deflate API prices roughly 10x per year, and any lab that prices above marginal cost loses volume to rivals. Each model can be operationally profitable on inference, but the margin never stretches to fund the next training run.

The standard global solution is subsidy. London Underground requires billions in TfL grants. China’s national HSR carries a trillion dollars in debt with 94% of routes unprofitable. AI is on the same trajectory: CHIPS Act, Stargate, sovereign wealth fund investments, Pentagon contracts. The default endpoint is subsidy-dependent quasi-public infrastructure.

MTR found another way.

Rail plus property

When MTR was built in 1979, its designers understood that fares alone would never recover construction costs. So they structured the corporation around a different premise: the rail line would make surrounding land valuable. So own the land.

MTR develops residential towers, offices and shopping malls above and adjacent to its stations, capturing the value appreciation that its own infrastructure creates. Property profits cross-subsidize rail operations and fund the next line. Today MTR owns 13 shopping malls, manages 47 developments above its stations, and property generates the majority of actual profit. Don’t try to capture value through the rail service itself. Own the assets that appreciate because of the rail service.

The AI parallel

“When do labs make money?” has the same structure as “when does rail pay for itself through fares?” It doesn’t, and that’s the wrong question.

A biotech startup uses a frontier model to screen drug compounds, shaving two years off a clinical trial. A logistics firm uses it to optimize routing, saving $40 million in fuel costs. A solo developer ships in a weekend what used to take a five person team three months. In each case the model provider captures a fraction of a percent through API fees. The provider can’t charge more, because four other labs and a dozen open source alternatives offer comparable capability. The surplus flows to the users and the broader economy. This is what general purpose technologies do. The steam engine, electricity and TCP/IP generated zero revenue for their creators.

MTR’s lesson: stop trying to make fares cover construction. Find the property.

Four candidates, ranked by defensibility

Government-granted deployment rights come first. A government grants a lab exclusive deployment access to national health records, tax systems or defense logistics. The lab accumulates domain data, integration depth and regulatory clearance that takes years to replicate. This is MTR’s own mechanism: development rights granted by the state, justified by natural monopoly properties.

Accumulated RL reward data is second. Billions of interaction signals that train the next model generation. Unlike weights (which depreciate via distillation), RL data is practically non-replicable and compounds across generations. It doesn’t convert to revenue directly, but it’s a land bank. Appreciating, undeveloped.

Forward-deployed integration is third. Instead of selling model access to a consulting firm that captures the productivity surplus, own the service delivery end to end. The way Palantir embeds engineers inside government agencies rather than licensing software. The lab doesn’t charge the law firm an API fee. The lab becomes the legal research service, priced against the outcome it delivers rather than the tokens it serves. Switching costs compound with accumulated domain data and institutional knowledge. This is MTR’s shopping mall: capture the foot traffic the rail creates rather than charging passengers more for the ride.

Data trusteeship over national datasets is fourth. Governments sit on enormous under-leveraged datasets (patient records, tax filings). A frontier lab designated as trustee gets exclusive access to train on and build products against this data. The scarcity is genuine, justified by the same logic as MTR’s land. You don’t want five competing labs accessing fifty million patient records any more than you want five developers building on the same plot. But this creates a public-private data monopoly and would require careful governance: clear boundaries on usage, benefits flowing back to the public, independent monitoring and real sanctions for misuse.

The reframe

The labs that survive won’t be the ones that make the API profitable. They’ll be the ones that identify their property above the station and build toward it now. The API is the rail. It will never be profitable enough. The money is in what appreciates around it.

The policy question follows: instead of subsidizing training runs, governments should design institutional mechanisms (deployment rights frameworks, data trusteeship structures, productivity measurement standards) that let labs capture the surplus their infrastructure creates.

There’s a final irony. The AI policy conversation is dominated by the US-China frame. American free market labs versus Chinese state-funded champions. The most relevant institutional model may be neither. It may be Hong Kong’s: a 45 year old public-private hybrid, commercially operated, self-financing through institutional design rather than ideology. The model that makes frontier AI sustainable might already exist. It just runs trains.

Always so much fun to chat with Grant.

AI has been making much faster progress in math than in other fields. As a result, mathematics is showing us, very concretely, what AI progress in other fields will look like. Even within mathematics, there’s a jagged landscape. What does it look like?

What is the nature of the most important conceptual breakthroughs in the history of mathematics, and how different are they from what AIs are currently able to do?

Does AI (on net) increase or decrease human understanding of the field?

How big is the overhang from having AIs systematically try to connect ideas already in the literature?

And what advice does Grant have for aspiring mathematicians, coders, and other students who are passionate about fields that are being most transformed upon by AI?

Watch on YouTube; listen on Apple Podcasts or Spotify.

• Gemini 3.5 Live Translate is what I wished I’d had on my last trip to China. It detects more than 70 languages and translates them in near real-time… and it preserves your original pacing and intonation. If you’re building an app that needs live translation, you should check out Gemini 3.5 Live Translate. Get started at ai.studio/live

• Cursor’s harness lets me use models for a huge range of tasks at the podcast. For example, Cursor cuts out the ads from each episode I produce so I can post them on Bilibili. It also helps me prep for interviews — I have a repo full of books and papers that Cursor sorts through to find the exact right file for any given question. Try Cursor yourself at cursor.com/dwarkesh

• Jane Street sponsors 3Blue1Brown, so Grant has gotten to spend a lot of time with various Jane Streeters. He actually just recorded an interview with a few of them, so when we sat down for this episode, he told me about some of the things he learned, like how Jane Street keeps their role definitions fuzzy to make sure their people keep learning and growing. Go check out Grant’s full interview at 3b1b.co/janestreet

(00:00:00) – AI is discovering new proofs. Is that AGI?

(00:11:32) – The verification loop on conceptual breakthroughs can be a century long

(00:26:12) – Will we understand an AI proof of the Riemann hypothesis?

(00:38:08) – Can AI find the hidden bridges between fields?

(00:53:48) – Why real-world tasks don’t fit into RL environments

(01:07:07) – Good writing requires theory of mind that AI still lacks

(01:16:02) – Why learning will still depend on human curation

Dwarkesh Patel

Today, I’m chatting with Grant Sanderson, who runs 3Blue1Brown and is now working on a new project documenting the progress AI is making in math. I wanted to talk to you about this because AI has been making the fastest progress in mathematics out of any other field. Whatever is happening here, and whatever way we’re seeing AI progress happen or not happen, will tell us about what will happen to the rest of the world as AI gets better and better.

I wanted to start with this question I asked you when I first interviewed you three years ago. I asked you, once we have AIs that can get gold in the International Math Olympiad, wouldn’t that just be AGI? Wouldn’t this just be able to do anything any human can do, given how hard these problems are?

You had an answer, which in retrospect turned out to be very wise and correct. You said it’ll be another benchmark, like all these other benchmarks that AI are passing. Obviously, AI has gotten better in a general way since then, but there won’t be some “aha” moment when this happens.

First, I’d be curious to get your heuristics on why that turned out to be true. Second, I’m curious how long you think this narrowness can continue to be true. By the point that AI has solved a Millennium Prize problem, do you think it’s still possible that there are lots of tasks humans are doing that AI still can’t automate in the economy?

Grant Sanderson

It’s an interesting question because it’s hard to answer without knowing what the solution looks like ahead of time. If we take the IMO, the spirit of your question three years ago was in looking at how some of the solutions to these problems really seem to require creativity. The designers of these problems try to come up with things that you can’t train for as easily.

The dirty secret with the IMO is that you really can train for a lot of them. With the whole AI and math project underway, as you point out, one of the reasons it’s interesting at all is that there’s a spiky frontier to AI, and math is just right there in one of the spikes.

But there’s a fractal nature to that spikiness, because when you zoom into the specific progress within math, you have some things that are a lot easier than others. If we just think about IMO, which is old news at this point. It’s been two years since they’re really doing quite well. They would have gotten a gold in 2024 if not for the following reason. They’re very good. They just cold-solved geometry basically. The IMO has these four categories of problems: geometry, number theory, algebra, and combinatorics. Geometry, it just solves it in nineteen seconds since 2024 because it’s a brute force solver.

The dirty secret is that for students, there’s also a brute force way you can go at it. Combinatorics is the wild card: much more playful, puzzly-seeming problems. There were two combinatorics problems on that year’s test, and there’s not always. There are four categories and six different problems, so it’s a toss-up which one is going to have two questions. Had it been more geometry questions, they would have gotten a gold that year.

But it struggles on those combinatorics ones. Someone who’s trying to keep that torch of the last holdout of math for humanity might say those are the ones that require more creativity. Even then, the spirit of your question—if they’re solving a Millennium Prize problem, does that also service a lot of white-collar work?—suggests that whatever the rate limiter is between where we are now and that is the same as the rate limiter for making things better at white-collar work.

We could paint a couple of different ways. If we focus on the Riemann hypothesis, what would it look like to solve that? These things are extremely good at a specific domain of knowledge, knowing it very deeply, and then knowing another domain, and another. You’ve pointed this out. It’s bizarre to have something with this superhuman breadth that knows all the fields so well, and yet isn’t finding those lightning bolts that connect them.

I think we’re starting to see sparks of it actually finding connections between the things it’s an expert at. I’m sure we’ll talk about it. If the nature of the solution to the Riemann hypothesis was something like that, that feels pretty distinct to me from what’s necessary to get good at white-collar work.

And there’s a reason to believe that might be the nature of the solution. I don’t know if you know the story of Hugh Montgomery and Freeman Dyson at the IAS. This is a side tangent, but it’s a fun story. I don’t know if it was over lunch or something like that, but you have this number theorist who is just trying to understand the statistical correlation between pairs of zeros of the Riemann zeta function.

The Riemann hypothesis is all about whether all these zeros sit on a straight line. He finds this quantitative question you could ask, and he writes down a formula. It looks like one over sine squared or something like that. Freeman Dyson, a physicist, is like, “I know that expression. That expression comes up in studying the eigenvalues for random Hermitian matrices,” which was something that comes up in studying the energy levels of a nucleus.

The idea that the statistics of those two seemingly different things were the same prompted an exploration of whether there are aspects of random matrix theory that might be relevant to the Riemann zeta function. I think it’s a little bit of an open question whether there is fruit to be had there. But that bridging together of two different fields—if it turned out that the solution to the Riemann hypothesis was exploring an idea like that even further—has the character of how you expect LLMs to be good at math. They’re experts at quantum physics. They’re experts at analytic number theory. They should be able to see that similarity in a way that doesn’t require Montgomery and Dyson to be having lunch and happen to talk about it. That’s totally different from white-collar work. To the extent that you have a hard time using an AI as an editor, it’s not because they know everything and you just need them to find that lightning bolt in between.

A different possibility would be… What’s the right analogy? Maybe if we think of Fermat’s Last Theorem, between the moment of Fermat phrasing the question and what the solution itself looks like, where the solution ultimately involves such heavy machinery in math. The beauty of that problem is you can phrase it so simply. You ask about xn + yn = zn. Do you have integer solutions for this when n is bigger than three?

It’s something you might expect there to be an elementary number theory approach to, but as far as we can tell, there’s just not. Whereas the actual solution, maybe there is something simpler, but this might be what it has to be. There’s such a complicated set of ideas that build on centuries of work centered around elliptic curves. Then there’s this other mountain of ideas centered around these things called modular forms. Both of those mountains have to be built before you can ask the right question that connects them.

If the solution to the Riemann hypothesis involved building a new mountain, that’s a kind of skill—the ability to come up with the right new ideas—that feels sufficiently different from the character of how they’re intelligent right now. It’s not like that’s what you need from your hired video editor per se. But if it’s capable of building mountains that are the correct new theory crystallizing how we should be thinking about a subject, that’s just such a level of intelligence that it would be surprising if it didn’t permeate into other aspects of the economy besides just the mountain-building for math itself.

Dwarkesh Patel

Or at the very least, even if it couldn’t literally do every single thing white-collar humans can do, it would just have transformative effects in the way that getting gold in the IMO did not have transformative effects on the world.

First of all, I do want to point out that I’m totally moving the goalpost here. When I interviewed Dario two or three years ago, I asked this question about why they haven’t been able to use their vast knowledge to connect ideas together and come up with a new discovery that way. That seems like the kind of thing where even a moderately intelligent person, if they knew this much information, would be able to come up with a medical diagnosis from the fact that this drug causes migraines, and this other thing does this, and maybe it’s the same drug that can cure both things.

From an outsider’s perspective, mathematics seems clearly like a field where finding the counterexample to the unit distance problem conjecture was an example of this kind of thing. So it’s total goalpost moving. But then we can ask, what is the next benchmark? Now that AIs can do this thing we should have thought they’d be able to do, what is the next thing that would be quite impressive?

There are a couple of candidate ideas here. One could be coming up with interesting problems in the first place, and the other is coming up with new kinds of objects or conceptualizations that create or unify fields. On the first one, right now we have these Millennium Prize problems because mathematicians have noted them. Riemann came up with this idea of the Riemann zeta function because he thought the zeros of this function would have some connection to the density of prime numbers.

Figuring out why we think this is an interesting thing to study in the first place, why we are building this object and trying to answer questions about it—and answer this particular question about it—seems like the kind of thing that would be the next benchmark.

Grant Sanderson

You highlight two pretty good examples there. For anyone curious about the unit distance conjecture, there’s this really nice video by a math channel called Polylog where they talk about it.

All of these discussions cause people to reflect on the process of doing math. They’re like, “Oh, this thing can do this impressive stuff. What does that mean for us?” One of the people in that video highlights this quote: “good mathematicians prove theorems, great mathematicians come up with conjectures, and the greatest mathematicians come up with definitions.” That’s more or less exactly your framing here. We need the conjecture generator and then the definition generator. That’s the premium-tier mathematician.

I don’t understand how exactly you’d make that a benchmark. Usually, when I think of the word benchmark, I’m thinking of something that is a goalpost. The ball is through the goal or it’s not. You can clearly say, “Yes, this is done.” Partly that’s to be able to do things like RLVR, but also partly just to know that you haven’t moved the goalpost in answering. OpenAI can have their headline on disproving the unit distance conjecture because it’s a clear, distinct thing. It did it. Whereas imagine trying to have a headline on GPT-5.4 coming up with a really good conjecture. “We promise, everyone thinks it’s a good conjecture.” It just doesn’t land the same way.

But maybe that doesn’t negate the fact that it’s the right thing to be thinking about. I would be surprised if it ever took the form of looking like a benchmark, where we have a score saying it’s passed because we can quantify how good a conjecture is. The nature of what it would take is probably that you’d feel a tone shift in conversations with mathematicians about the way it’s useful to work with.

This series you referenced, which is not at all produced yet and probably won’t be for a couple of months, takes the form of us interviewing a lot of mathematicians. What’s interesting is that we started doing this over a year ago, and it’s fun to see a little bit of a tone shift in the way they talk about AI between mid-2025 and where we are now in 2026. In the real world, that’s a very short amount of time. In the AI world, that’s eons. We’re able to see this tone shift over those eons.

I think the way you’d measure conjecture-generating ability is going to be more subjective, based on that tone shift. It will be mathematicians saying they’re not just using it to solve their problems, but that as they step back and decide what their research field should even be, a conversation with such-and-such model was genuinely helpful for that. I don’t think it’s likely you’d see it in the form of a headline saying this was yet another benchmark knocked down.

Dwarkesh Patel

It’s very interesting. The kinds of things you can’t make benchmarks for are also the kinds of things, at least in the current paradigm, you can’t easily train for. There’s really no fundamental difference between a benchmark and a training environment.

It’s very easy to come up with some dichotomy of, “here’s a deep reason why AI can’t do a certain thing”, and then it turns out you’re just thinking about it the wrong way, and actually it can do it pretty soon thereafter. But I’m going to come up with—

Grant Sanderson

You’re going to come up with a couple anyway.

Dwarkesh Patel

It’ll probably turn out that there are ways we can train AIs to do these kinds of things in the relatively near term. But it seems like it would have to be different from current RLVR training.

The thing I’m curious about—and the thing that seems to me to drive a lot of the big progress in mathematics and in science generally—is coming up with a new way to think about a problem or a new way to understand the world that unifies different fields, spawns entire new fields, and solves problems we weren’t even trying to solve in the first place. The reason Einstein was thinking about GR is not because he wanted to explain why light bends or why black holes exist. These are phenomena he didn’t even need explained in the first place.

In mathematics, as a total outsider who doesn’t even know what he’s talking about here, it seems like there are often ways to prove a specific problem that can motivate a new conceptualization—one which results in a whole new field, a whole new way of thinking, which is immensely productive—and ways which don’t. I’d be curious to hear you talk about Galois coming up with group theory, distinguishing his solution to the quintic having no formula for the roots, and Abel coming up with a different proof a few years earlier that didn’t come up with group theory.

If you wanted to do a verification loop on whether group theory is an interesting concept—was something useful done here, or why is this proof better?—potentially that verification loop is a hundred years long. It involves cryptography coming around and physics making progress, and the ideas in group theory being relevant to understanding symmetries in physics. There’s a hundred-year verification loop on why this is a productive concept in the first place.

Grant Sanderson

You struck a nerve, because I had this project about Galois I was going to do in 2022 that I put on the shelf, but I spent a year of my life thinking a lot about what he did. There’s a risk of me accidentally talking too long on the specifics, which you can hold me back on.

It’s a perfect example for your case, because describing why it was a valuable insight does not come from immediate utility. Certainly, if you’re thinking about RLVR environments, this is going to be really hard to do. But it’s interesting to note that even with human verifiers at the time, it took a really long time to recognize it as being useful.

With Einstein and GR, people could feel this was a good theory right away. What makes Galois theory such an interesting example is that you literally have this hundred-year segment of an idea that flows through many different people’s heads before it settles into something the math community agrees is good. To back up a little bit… Do you want the background on the problem at all? We all learn about the quadratic formula in school.

Dwarkesh Patel

I thought you were going to say we all learn about group theory in school, but I missed that class.

Grant Sanderson

We all learn about group theory… No, the quadratic formula. This was known. In some sense, the Greeks could solve quadratics, but they didn’t really write things in algebra. It’s really the Arabs who wrote down that formula.

There’s this delightful story about dueling Italian mathematicians—not real duels, just intellectual challenges—who secretly found a formula for the cubic, and then very shortly thereafter found a formula for degree-four polynomials. So a natural open question for mathematicians is, can you find a formula that solves degree-five equations?

The degree-four formula is a monster. It would be wild to write it down. You usually don’t write it down in full. You break it up as a procedural thing. You might believe these things have this exponentially increasing complexity. So for many hundreds of years, nobody was really answering that question.

Usually, we say Abel was the first to prove it. He was this young, precocious Norwegian mathematician. He showed it’s simply impossible. It’s not that you can find a quintic formula. He thought he found one initially, but he showed it’s impossible.

I think the real credit though, you have to back up a bit and talk about Lagrange. He found the right kind of question to ask about this. I’ll give it at a very high level. He was studying the question and recognized that being able to solve these polynomials is very related to understanding the way certain algebraic expressions are symmetric.

If I write down a + b + c + d, just adding four variables, and I permute those, it doesn’t change the value of the expression. Whereas if I write a + b * c + d, some of the permutations don’t change it, but some of them do. He had this really nice insight about how if you can find expressions that have four free variables, but all the permutations take on three distinct values, that has this unexpected relationship with being able to reduce degree four into degree three.

He started approaching the question of whether we can find a quintic polynomial by wondering if he could extend that method. To extend that method, you would have to have an expression that has five free variables such that as you permute them over all the five factorial permutations, it takes on only four values or fewer. You could put that in a puzzle book. You could put that in a brain teaser that a twelve-year-old could engage with. It’s not too hard to find yourself feeling like that’s an impossible task.

Lagrange is sitting there saying, “Here is a strategy to solve this problem of finding a quintic polynomial. It seems like it might be impossible, at least from this strategy.” But that was the first time in history that people had the instinct that some kind of question about symmetry was the right way to study these polynomials. In his mind, it was just a way. It had yet to be discovered that there was actually a tighter connection. Also maybe rather than searching for the formula, we should be asking the opposite question: can you prove that it’s impossible? He sort of planted that seed.

Around fifty years later, Abel definitely read Lagrange and was influenced by it. We know that Galois loved Lagrange when he was falling in love with math. It’s very hard to imagine that these two young geniuses coming up with pretty similar insights around that problem wasn’t born from Lagrange.

But to your question on whether you are able to verify that this was a good idea, there wasn’t any result that Lagrange came to. He didn’t solve the problem, so it wasn’t a case of knowing it was the right question to ask based on a solution. He just asked it. There’s some intrinsically interesting thing about it. It also wasn’t very important for math at the time. Most people were more interested in the applications to physics. This was almost a side, recreational, hobbyist-type thing.

Abel started working on quintic stuff, but then he was advised to spend more of his efforts studying elliptic functions, so more of his work was on that before he died young. He died at twenty-six from tuberculosis. And then Galois pushed both of those ideas in the right direction, where he really understood the nature of abstraction. He had this really nice piece that he wrote while he was in prison. We could talk all about his life story. It’s pretty wild. But he’s this teenager, he’s in prison, and he had tried to submit his math papers and they had been rejected.

So again, thinking about verifiable reward, the verifier function that is the academy at that time is rejecting what he wrote. Frankly, it was not very coherent. It wasn’t a complete proof. He wasn’t giving a clear thought of what the theory actually was. He was just a young fledgling mathematician getting his bearings. The verified reward there is, “No good.” But he has some instinct that there’s something there.

So he’s writing this diatribe on the nature of math being something that undergoes these shifts over time. He talks about the advent of algebra itself and going from just thinking in terms of numbers to having a certain fluency with pure algebraic expressions, where you’re not tied to interpreting those expressions. He has this instinct that there seems to be another layer of abstraction that we should be doing, where rather than thinking about the formulas themselves, we’re thinking about what symmetries underlie those formulas. But it was still a pretty ill-defined theory.

If you’re trying to say the verified reward is that he solved a problem that other people haven’t, well, Abel already proved that quintics are unsolvable. So what was Galois doing? In principle, Galois theory lets you take a specific polynomial, and it gives you the rules to say whether that specific polynomial has roots that you could write down. For example, with x5 - 1, you know that a solution is 1. Or x5 - 2, you can write down the fifth root of two.

So it’s not that you can’t write down the solution for every quintic polynomial, but could you find a specific one where you prove you can’t write the solution using radicals? He also didn’t even solve that exactly. He didn’t show for a specific example that he couldn’t. Even describing what problem he solved is very tricky.

He then dies. It’s this very romantic story of him having this duel. There’s a lot of myth around how he supposedly writes up all his ideas the night before the duel, but really, he tried to get them published five times before.

Dwarkesh Patel

Working on the quintic doesn’t seem to be good for your health.

Grant Sanderson

It’s very bad. If you’re a young genius, don’t work on the quintic. He asks his brother and his close friend to get his notes to Gauss, to get these notes to the important mathematicians of the day, because he thinks there’s something there. Even then, it didn’t really take. His brother and his friend tried to get them out, but it was another twenty years until Liouville sees these notes, sees that maybe there’s something in them, and tries to clean them up and understand what Galois was getting at. Even then, it was another twenty years or so until Jordan actually puts together something like a modern treatment of group theory that they attributed to Galois.

You could easily imagine history turning differently, where these ideas were coming about from other points in math, and Galois could have been forgotten in history if he was a less florid character. But between the time of Lagrange having this inkling that maybe symmetries of roots is the right way to go, to where it all looks like modern group theory, you’ve got this long span. A lot of the time, it’s not even passing the verified reward of human reviewers. It gets on someone’s desk and they say, “I don’t really know if there’s anything here.” You have to have this one person recognize it.

Even then, it’s not really solving practical problems at that point. You pointed out cryptography and physics and things like that. You have to get into the twentieth century before you have Gell-Mann thinking that maybe understanding the nature of how certain groups break down has a relationship with what particles are made out of. He anticipates quarks based on a purely group-theoretic question. That’s one of the more interesting applications of group theory: to even predict the existence of quarks is a group-theoretic question. That’s so long after Lagrange before you have anything like that.

So you have to ask, what is the way of measuring progress that’s not based on solving a problem, but that is somehow capturing the instinct inside Galois’s mind when he says, “I think there’s something here”? What’s the instinct inside Lagrange’s mind when he says, “I think this is the right way to think about it”? What’s the instinct inside Liouville’s mind when he says, “These scattered notes from this long-dead youngster might have something to them”? It’s so hard to put a finger on that.

A different series of videos I’m making right now is about the whole “compression is intelligence” idea. Even though this isn’t really the angle I’m taking, there is something to the idea that the smaller expression that’s more predictive feels more intelligent. So I wonder the extent to which you can give some kind of verifiable reward around not just whether you solved it or what it is solving, but around the smallness of the concepts required to do it.

Going back to Riemann hypothesis solutions, what would that look like if an AI solves it? I think a third way it could happen is it just straight-up works harder. In the same way, you could maybe have an elementary proof of Fermat’s Last Theorem that’s just spelled out over thousands of pages that would be incoherent. But the cleaner way to view it is with elliptic curves and all that. Maybe there’s some thousand-page proof of the Riemann hypothesis that no one’s really getting anything out of, and what you actually want are the succinct, compressed versions of those ideas that would then lend themselves to human understanding.

Maybe you throw Kolmogorov complexity into your attempt to quantify what you mean by elegance. I don’t think it’s easy, but I do think it’s something you would have to do in order to reward the Galois-like instinct, rather than just rewarding whether you solved a problem.

Dwarkesh Patel

It’s very hard to come up with the heuristic for science. But it’s clear humans have been doing this somehow, and obviously, AIs will do it at some point.

Grant Sanderson

It’s relevant also not just in terms of verified reward, but presumably, the end goal is understanding, human understanding. Even if you do have some thousand-page proof of some math thing or some grand new physical theory, the goal is understanding.

Maybe if the goal is predictiveness, you can just have automated engineers go off and build rocket ships where we have no idea how they work, but we can get between stars. But there are going to be a lot of people who want to understand. You’re still going to want whatever the concision function is that distills down this complicated way of thinking into the right one, like the equivalent of the universal law of gravitation for Newton. You would still want to train AIs to be able to do that and find the compressed representation.

Dwarkesh Patel

People have this worry about mathematics in particular that AIs will prove the Riemann hypothesis, and our understanding of mathematics won’t be any the better for it. I have a couple of questions about this. The first one is whether this is something you should expect.

Isn’t the reason humans come up with general, natural objects and subgoals when we’re working on a big problem that this is just useful when you’re trying to work on a complicated, important problem? Theoretically, would this even be a simpler way to solve the Riemann hypothesis, as opposed to just coming up with the natural abstractions that are relevant to thinking about the problem?

And then two, empirically, is this what we observe when AIs make progress on problems today? When the AI came up with that counterexample to the unit distance conjecture, you can just read its chain of thought. It’s not understandable to me, because I don’t know anything about mathematics, but it seems that to other mathematicians it was understandable. It made use of known concepts of mathematics and proved relationships between them, all in natural language. As a result, it accelerated our understanding of the connection between this object and this conjecture.

Empirically, is this a thing we should be worried about?

Grant Sanderson

I think it depends on the nature… If we break down the three possible ways of solving the Riemann hypothesis… The other big one from this year was a certain Erdős problem numbered 1196, about these things called primitive sets. It had that character of bringing an idea from a seemingly different field. As soon as you present the basic idea to a mathematician… You say, “What if we try the Markov chain process where we show that this thing is one from the bottom up probabilistically rather than the top down, and use the von Mangoldt function?”

If you say that to someone in the know, they’d know how to run with it. You have this very small idea that has the form of expertise in one field and expertise in another, drawing a little lightning bolt between them. Those are going to be very human-parsable, because all you have to do is show the start and end point of what those connections are.

If the character of it is mountain building, you have to put in a lot more time to understand that new mountain that was built, because it’s a new thread, not just a lightning bolt between them. And if the nature of the progress was just raw hustle—a super long chain of reasoning with no new theories—then you would have that worry of this whole digestion process. So I don’t think there’s one clear answer. It depends on what the solution would look like.

On the mountain building side, that would actually be really interesting to see. Is it by default very human-understandable, the way we see new theories from great mathematicians? Or is it an alien, different kind of mountain being built where we have to reprocess the kinds of abstractions we engage with?

The closest example here would be the attempted solution of the abc conjecture. We maybe shouldn’t get into that one, but it probably is not a correct solution. Basically it’s this whole new way of thinking that this otherwise reputable mathematician in Japan had come up with. It took mathematicians a long time to even parse what he was saying, but it had the feeling of an alien bit of mathematics that’s theory building, not just a long chain of reasoning. He called it inter-universal geometry.

The biggest fear would be that an AI does that, and then much like the abc conjecture, people work for years to go up the mountain, and they’re like, “Dang it. This just isn’t right.” If it turns out to be wrong, but it really looked right. Even if it was right, there’s just a lot of effort to hike up a new mountain.

Dwarkesh Patel

If we end up in that situation, David Bessis had a really great blog post called “The Fall of the Theorem Economy”. He’s talking about how historically, as you were saying, mathematics is coming up with these definitions and problems, and it’s about proving theorems about them. The theorem-proving stuff is what gets all the credit, but it’s really a parasite on the coming-up-with-the-definition stuff.

Historically, this has not been a problem in terms of credit apportionment, because if you come up with a definition, you’re probably going to be the guy who comes up with a theorem. But now we’re in a situation where if the valuable work is coming up with the insight and AI automates the latter part…

Imagine a scenario where an AI comes up with Abel-like direct arguments about a bunch of important conjectures in the world, and then we just have these proofs. Now it’s up to humans or future AIs to consolidate. Again, having no object-level understanding of this argument whatsoever, I’m sure that if you had access to it, it would make it easier for you to think about what’s going on. Is there some deeper way in which we can understand why this proof works that would make it easier to come up with the ideas behind group theory?

Grant Sanderson

I think it would be hugely helpful. So much of trying to discover new math is mostly being wrong. You’re trying to solve a problem, and it doesn’t feel like constantly taking the correct step up the mountain. Mostly it feels like a random drunken walk, where you’re doing a thing and then you’re wrong and constantly discovering that. If at the very least you know that trying to digest what you have is ultimately leading to a correct solution, that feels like progress, simply because of the sense of knowing it leads to a solution.

There are plenty of instances in the recent history of math where it feels like the reach has exceeded the grasp, where things are proven long before they’re understood. One of my favorite openings to a paper—it’s not even a research paper, it’s more like an expository one—is from a mathematician named Timothy Chow, who was trying to understand a concept called forcing. There’s this problem called the continuum hypothesis that more or less asks: you have a size of infinity for the natural numbers, and a size of infinity for the real numbers. Is there something in between?

The answer is both yes and no. It depends on your axioms. It’s outside the scope of our usual axiom systems, which is an interesting answer. But the method to describe it is really hard to understand. It’s this thing called forcing. In the beginning of this paper, he writes, everyone knows the idea of an unsolved research problem. I want to propose the idea of an unsolved expository problem. Sure, we’ve proven it, but we don’t really know why it’s true.

Then he proposes a partial solution to that expository problem. You can imagine why I loved that framing, because this is my whole life. I don’t do research math. It’s wholly about what’s the most clear way to understand this, even if it’s proven. There is a difference between proof and explanation, and I think you’re getting at the importance of that distinction.

Dwarkesh Patel

Yeah. That will be the main incentive. Or the incentive would have to change, not just in mathematics but in other areas of science, from proving things about the world to consolidating proofs into problems or higher-level insights.

We were having a discussion earlier at lunch about a recent talk you were giving on design and how it helps us understand things. In the limit, is there really a difference between the conceptualization of an idea and the idea itself? If you think about special relativity and spacetime diagrams, and Minkowski spacetime, this is a way in which we illustrate why there’s length contraction and time dilation. But that is the reality… So, the exposition does seem to be the explanation in some sense here.

Grant Sanderson

There’s a couple of interesting things there. One is that there seems to be a really strong correlation between the people who come up with genuinely novel insights and the people who are actually quite clear in their communication of it. You might imagine the opposite, given that the experience of a university student is often that the expert teaching them is not necessarily the best explainer of that topic, because they’re so spoiled by their expertise. But what seems, at least in some cases, to be the case is that the people who are really coming up with something quite novel—you’ve got Einstein or Claude Shannon or someone—you read their papers, and they’re really lucid.

It doesn’t feel like this is just for the experts and you have to chop through it with a machete. They’re very good expositors. Feynman has this characteristic too, he’s a very good expositor. Maybe the same part of the brain that comes up with the correct new way of thinking about it at a research level also has this knack for good explanation.

I think this is pertinent to AI. I used to think that AIs would become these automated theorem provers, but the role of mathematicians was going to shift towards my job, explaining these things. Now I suspect that actually they’ll also be quite good at doing that, probably better than most humans are at explaining and distilling. So digesting and explaining what was going on is probably actually not what’s left for mathematicians, by the nature of how these things are going. We can talk about ways this might not be it, but probably the same thing that comes up with the really good new idea that solves some new problem is also just good at explaining it. That’s a way my beliefs have changed.

Dwarkesh Patel

What’s the last thing you think you’ll be doing? Both you and also what the human mathematical community will be doing.

Grant Sanderson

I will probably be doing something like what I am until I die.

Dwarkesh Patel

If the doomers are right, maybe it’ll be for the same reason.

Grant Sanderson

Yeah. You build a man a fire, and he’s warm for one night. But set a man on fire, and he’s warm for the rest of his life. So that’s where I am with AI.

Some of the function of an explainer or a teacher is to add clarity to a thing that someone’s curious about. That’s one thing. But some of it is more relational, providing motivation and a sense of curation. One interesting take that I’ve heard about what mathematicians will end up being is that it’s actually more analogous to art museum curators than anything else.

The AI solved the thing, so the art exists. They even know how to explain it really well. But you still want someone to help you navigate this nearly infinite space of what ideas are worth engaging with. Even if AIs were in some sense better at that, I think we would always still prefer a human that we had a relationship with, because the way we get motivated to be interested in things is a social phenomenon.

If you have some specific technology you’re trying to build, that might be different. But the people listening to this podcast trust your curation on what’s an interesting topic in the first place. It’s not that they’re landing here because whatever your next topic is, that’s what they wanted to understand in a prior sense. They’re trusting you as a curator.

So my role, and arguably that of other mathematicians, might actually just shift subtly into that curation direction of what ideas are worth pursuing. That’s a lot of my job right now. I think people assume a lot of the time for a video goes into the visuals. Sure, it does. It’s not immediate. But actually a lot of it is just deciding what’s worth saying in the first place, what’s worth putting there.

I want to engage with that, and I think I have a trust with certain people, and they’re curious what I would choose to put forward even if the AIs are better than that. It’s the same reason human musicians are always going to have a role: that social function of the story behind them, even if the objective quality of the MP3 file coming out of some model is better. That’s what I see happening to my job.

Dwarkesh Patel

I want to go back to a question from earlier. Just as AI has crossed this threshold, this important benchmark of being able to connect existing ideas to come up with a new discovery or prove or disprove something, we’re like, “Okay, but what’s the next thing?”

Grant Sanderson

There’s a lot more to do on that one, by the way. Just because a couple lightning bolts have been thrown… I think there’s this flourishing future over the next couple years of really connecting.

Dwarkesh Patel

Right. So in the limit, you could even say—I don’t know if this is accurate, but potentially—a lot of the biggest breakthroughs look like this at some level. With general relativity, you’re just connecting together Riemannian geometry and special relativity. So as AIs keep getting better and better at this connection thing, maybe a lot of big breakthroughs are not really of a different qualitative nature. I don’t know if you have a take on that.

Grant Sanderson

A lot of the conversation has focused on problem-solving and that nature of math, ticking off Erdős problems or something. But I’d say it’s not even a majority of mathematicians who would characterize their work as really targeting the next problem to tick down. Are you familiar with the Langlands program?

Dwarkesh Patel

No.

Grant Sanderson

It’s not even a field of math so much as it is a research ethos. Fermat’s Last Theorem is one inkling of this. You had these two seemingly disparate things, and a connection between them led to a solution.

Langlands was a mathematician. He has this famous letter essentially spelling out how it seems likely that there’s a lot more connections like that. He even got a little bit more specific about the nature of the connections, such that you might imagine this large map, and you’ve got this valley over here and this mountain over here and this set of plains over there. There’s a lot of mathematicians who would characterize their work as being part of trying to understand the threads on this map.

The progress there, it’s not even “Here’s this one specific problem that we know will be solved by that connection.” It’s more that time and time again, there have been cases where big problems were knocked down by finding connections, such that it’s almost preemptively finding the connections. It’s actually very interesting. Anytime you run into a mathematician, ask them whether the character of their work is more akin to the Langlands program or to targeting one particular problem. You get a certain bifurcated split there.

The possibility of AIs being supercharged connectors feels like it might be an amplifying tool in that pursuit. It’s hard to measure, though. This cuts to what we were saying earlier: How do you assign a score to say, “Yes, you’ve done it”? If it’s knocking down a problem, you have a clear way of saying, “Yes, you’ve done it.” You can write the headline. You can have your PR move as the AI company to say, “We did it.”

Whereas if it feels like that was the right connection to draw, you can write theorems around it. That’s the nature of what the papers in that field look like. But I think it will require a lot more “human in the loop” to say, “What was the kind of connection that we were going for?” That’s my guess on what most of the useful progress from these models will look like in the next five years. It’s just really filling in that landscape of connections that you can draw if you’re an expert in multiple fields. As you’ve pointed out, it’s kind of surprising we haven’t already had this.

I would be curious to know at a technical level what causes the unlock there. On the one hand, you can paint an explanation in your head for why you could be an expert in all of these things and not be drawing those connections. When the method of reasoning is this autoregressive chain-of-thought phenomenon… Autoregression is actually a really weird way to produce stuff, if you think about it.

You’re an intelligent person. Imagine I’ve locked you in a box, and the only way you have of interacting with the world is that you receive a slip of paper, and someone says, “Can you predict what will come next?” You predict what will come next, and then your memory’s wiped. You get another slip of paper. Imagine that was done a whole bunch of times, and then what comes out on the other end. They say, “Look at this essay that you wrote.” You might look at that and say, “This is awful. That’s not the essay that I would’ve written.” The process of repeatedly predicting something is just pretty different from how you would think as a writer to compose it and think it through.

In particular, what would probably happen is that you’re a slave to your context. You might be answering some question about a particular field, so you draw on all the context around that. But the connection where all the substance is going to come from is, by its nature, a very unlikely one. You can do all the RL that you want to try to get better in some way, but what’s the thing that’s specifically upweighting and incentivizing making these unlikely connections when the vast majority of them aren’t the predictable next token that would come in there?

So it might be the case that you just have this intelligence locked inside that box, but it’s a weird way of interacting with it. The thing I’m curious about is: do you ever get any fruit by questioning the premise of how tokens are generated? I don’t think it would be as simple as manipulating the temperature, but are there any things that you can do that take the existing level of intelligence but find the right ways of sparking those connections that unlock these sorts of things that we’ve seen? Or do you just need a little bit more intelligence, such that at the level of prediction, it’s predicting that it should be making that lightning bolt to another field?

Dwarkesh Patel

I think it’s more productive to reason, instead of architecture or even loss function, about data. We have diffusion models that do text, and the kinds of things they produce are not of a wholly different character. They’ve just not been explored as much. I think the more relevant thing is: what is the data on which whatever architecture or loss function you have is incentivizing you to produce? It does seem like they’re getting better.

Forget about math. We did have a couple of examples of this kind of thing, but if you just look at why they’re getting better at being autonomous agents… They’re in an environment where they’re autoregressively producing the step that says “Let’s step back and do a search over the whole codebase,” and then “Let’s step back and assess my mistake,” is the thing that works.

I assume what happened in the case of progress in science or maybe in math is you have frontier math-like problems. Mathematicians have specifically designed them because they require connecting together two different fields. I’m guessing there’s all kinds of clever, partially synthetic ways to make harder and harder problems like that that require these kinds of connections—for example, by eliminating assumptions and still requiring the AI to get to the answer—and then it doesn’t really end up mattering what the loss function is. It’s really about, can you come up with an environment that incentivizes this ability?

Grant Sanderson

It feels like you should be able to. I certainly can’t speak to the correct ways of doing that to unlock all this, but it would just be pretty surprising. Don’t you think it would be surprising if, over the next three years, there weren’t a lot more of those lightning bolts?

Dwarkesh Patel

I think this is an important thing to think about. We often think about how smart a single system is. And we don’t think about AIs having advantages that are more the result of other facts about them. So in this context, the key fact about them is that we can just parallelize and arbitrarily scale them. Whatever level of capability they have, it’s not just one idiosyncratic genius in the history of mathematics who makes a few connections and then dies in a duel.

It’s universally applying that waterline across all problems that are accessible at that level of capability. This is among the many advantages that digital minds inherently have that we don’t think enough about. The other ones being that they can merge all their knowledge together—or at least that there will be techniques that allow this to happen—and that you can spawn off copies with identical levels of knowledge. This parallelization is quite an important property.

I’d be curious about your predictions. Even if they’re not as smart as human mathematicians, the fact that for PR reasons the AI companies are just throwing billions and billions of dollars at this means that quantity has a quality all of its own.

Grant Sanderson

That seems in the right direction. If we take that conversation between Montgomery and Dyson at the IAS that suggests some connection between the Riemann hypothesis—or the Riemann zeta-function zeros—and random matrices, that feels like the kind of thing that you could try to automate. You have agents representing expertise in all these fields. We all know that an institute is smarter than an individual.

The reason for having people all in the same geographic location is that you want those serendipitous conversations to happen. What does it look like to engineer those between agents? It’s interesting, because you point out that you can pool all your knowledge, but I really wonder if one of the advantages is that you can do the opposite of that.

Sometimes when an AI is failing, it’s because it gets into a bad chain of thought and it’s really hard to get it out. So you say, “I’ll just start again.” Same deal with humans. Sometimes you start thinking about it in a certain way, and what’s required is to just back up. There are stories about people trying to prove something for a long time, and then at some point they say, “Hang on a second. What if I tried to prove that it’s impossible, or prove the opposite?”

Unwinding your own context and going at it with a fresh mind… You could imagine systematizing that, or having multiple different agents deliberately given different pieces of context and trying to compare and contrast there. We don’t have the same level of manipulation on our own context.

In this AI and math series, the first episode we’ll do will be about when they solved the IMO. I want to focus on one specific IMO problem that they failed on, which is one that a lot of very smart students failed on. Terry Tao also failed on it. People were very mad at the problem because they called it a troll problem. I almost don’t want to spoil it, because I want to construct the episode around leading someone in without their knowing that it turns out to have a simple solution. You can really empathize with what it’s like to be a student solving this.

Basically, there’s a really elegant way of going down what you really feel like is going to be the solution based on the context of it being an International Math Olympiad problem. The character of the solution is really enticing, but it’s hard to prove that it’s the best. The reason is that it’s not. There’s this almost brain-dead solution that is the best.

The relevance of that to the whole AI story is that for a human, what’s required to answer that question is to escape your context. Escape the context of being in the IMO. Escape the context of the way you’ve been trained to solve these contest math problems. If you just approached it like a brain teaser that I throw at someone off the street, they’d probably answer it well.

You want the same sometimes for human research in other contexts, just being able to refresh your thinking and come at it completely differently. Of all the advantages that digital minds have, that might actually be one of them: a more systematic approach to refreshing your thinking. Spin off two agents, one who’s trying to prove it and one who’s trying to disprove it, one who tries it this way and one who tries it another. They deliberately have different contexts.

I would be curious to see, if we’re having this conversation three years from now, how many of the significant results that make headlines have that character of basically erasing the context previously, trying a bunch of different things as opposed to merging the results of a bunch of different agents.

Dwarkesh Patel

It is incredibly interesting, because a common concern people have about AIs is this entropy collapse where they all think the same way, because they’re trained in similar ways. This is why they’re bad at writing. They go down the same path and have similar patterns of speaking and so forth.

But maybe the key advantage AIs have is that you can systematically… It sounded like one of the reasons the unit distance problem conjecture took so long to be disproven was that people assumed the conjecture was actually true, so they were mostly trying to figure out ways to prove it. Maybe one of the key advantages the AIs will have is to increase the entropy by systematically trying out both the negation and trying to prove the positive of any given statement, or being able to systematically give different agents different biases.

It seems like an important thing in the history of human science is that Einstein was really motivated by this bias that things should look the same in different reference frames. He had multiple other biases like these, but that one was very formative in his thinking. You can systematically survey a bunch of heuristics and see which ones are being productive on a given problem.

Grant Sanderson

So you would suggest systematically increasing entropy at the prompt level even though you have this inevitable collapse at the autoregression level? Einstein would be an interesting example, because he’s got this bias toward things being relative. He also has a bias toward “God should not play dice.” You want to make sure you don’t accidentally have all your LLMs be Einstein, because you might halt progress on quantum mechanics.

Dwarkesh Patel

Which goes to show you that there’s not a correct heuristic for science. You just need multiple independent research programs with their own heuristics.

Grant Sanderson

That feels like old-school software. As long as you’re able to describe that in some way. You have old-school software that amplifies that entropy. If you’re able to put a clear ontology to the distinct ways of thinking that you want to prompt, you explore that full ontology, and then each individual one runs off doing what it is.

There’s a certain design question there about how exactly you describe the different approaches. The easy one is: are you trying to prove it or disprove it? The harder one would be to say, what are all the tactics you could take to prove this, and make sure you’re applying sufficient breadth to exploring them.

Dwarkesh Patel

Obviously, AI in math is making much faster progress than everything else, and people point to the verifiability of the domain as the key reason this is happening. I think that’s one of the two important reasons, but people really neglect the other one. I’m outside the labs, so I don’t know what’s actually going on. This is a totally naive theory.

A tangential question to why AI is making so much progress in math: why has it been so slow at computer use? A computer is very verifiable. Is my Etsy package coming? Is my event booked? These are extremely verifiable things to survey. What computer use lacks is grindability.

Because websites have bot detectors—and it takes a tremendous amount of compute to run parallel rollouts—it’s very hard to run a thousand parallel rollouts of the same checkout flow on Amazon. You’ll get shut down by Andy Jassy.

Grant Sanderson

Him personally. He presses the red X on Dwarkesh button.

Dwarkesh Patel

Exactly. You could try to build clones of every single website, but that’s very labor-intensive and slows you down. The reason you currently need to do so many parallel rollouts to learn a skill with deep learning is that we haven’t solved sample efficiency.

Grant Sanderson

Sucking supervision through a straw, as Karpathy says?

Dwarkesh Patel

Exactly. Of course people are working on many different techniques, but fundamentally there’s this big constraint in the way we train AIs. With code, you can containerize a given level of progress in a repository and then spin out hundreds of parallel containers and say, “Try to implement this feature,” and it’s totally deterministic. Because it’s deterministic, you can solve the credit assignment problem because you know that whatever caused this rollout to succeed and this one to fail, the diff is the thing that worked. If you have situations that are starting off at different starting points, this credit assignment problem becomes much harder to solve.

Most things in the real world are very hard to containerize in the same way. Coding and math are exceptions to this rule. But if you’re trying to figure out how to build a new business that succeeds, or how to go trade in the markets for a day and make money, the fact that you have to interact with the real world and things change day after day means that you can’t keep replaying and grinding and farming the simulator.

Math, of course, is the exception, and I feel like this is an important driver of progress in this domain and also in coding. It’s not just verifiability; it has to be grindable.

The third reason people point out that AI is making fast progress is they focus a lot on Lean and formalization. Again, I have literally no idea what’s going on in the labs. I feel like Lean just doesn’t matter that much for the current level of progress in AI. Why is AI able to disprove the conjecture about the unit distance problem? They released the chain of thought, or at least a rewrite of the chain of thought. It didn’t have any Lean in it. I think the process-based supervision that Lean provides, where you know each step is correct, seems less relevant than just having this grindable outcome that is verifiable.

Grant Sanderson

It’s an interesting point about grindability mattering more. Naively you might think Lean provides something unique for math because you’re able to see if it can prove it. You have old-school software that can tell you yes or no, and you use that as your VR. What would corroborate your point is the initial attempts. Again, I’ll circle back to the IMO. Initially, DeepMind basically does that. Everything is in Lean, and then the next year it’s all in natural language. So to your point, it’s not needed.

I do think there’s a yet-to-be-explored benefit of that formalization domain, which is that at the moment you still need a human reviewing that counterexample to the unit distance conjecture to say, “Looks good.” That provides a certain bound on how endlessly explorable things are. If you consider AlphaGo or AlphaZero-style systems, they’re off in their own universe playing a bunch of Go and exploring themselves, potentially going off the rails of what any human needs to look at, but they still have this automated verifiable reward. It’s not just that you can do RL on that. It’s also that you basically never have to check in, and you can just pour compute at them exploring the universe of Go.

What stands to be interesting—maybe this won’t pan out, but the jury should still be out on whether it’ll yield anything—is that with Lean, you could imagine having a basically endlessly running program that’s constantly trying to extend Mathlib. Mathlib is this GitHub repository that’s basically all of math written in code. It’s very far from all of math, but they want it to be all of math. It’s written in code where you can ask, “Is this proof correct?” It’s very labor-intensive to write these proofs. There’s a whole subcommunity around it.

But you could imagine having an AI where you say, “Simply try to extend Mathlib.” Maybe it’s a fork of it so that it doesn’t have trash in it, because people have a certain taste for what they want to be in there. So you have your fork of the pure AI Mathlib, and it just goes and it doesn’t stop. It doesn’t need anybody to check in on it. It could just keep going. It might come up with its own conjectures. It might come up with its own theories and different definitions. Maybe many of them are useless, but it just has this infinite tree that it can grow out.

That’s a very unique thing that math has that nothing else has, where you could press go and just pour compute at it, look away for ten years, and then come back and say, “What do you have?” There’s going to be something. Then there’s a question: is it useful or not? How do you suss that out? That’s just an interesting thing to be able to do. It would be very surprising if that didn’t yield some sort of interesting mathematical insight from it.

There are two different ways that Lean is important in this story. The first one is how you could let go, not even check in, and progress will be made. You can do that with Go. I don’t think you can do that with natural language math.

Dwarkesh Patel

That’s very interesting. Did you see Karpathy’s auto research idea? He wrote this one Python file that does basic LLM training, and then had a repo where LLM agents would try to make modifications to the file, and if it sped up the speed run, the modification stays.

Eric Jang, who came on to explain how AlphaGo works, did a similar thing when he was trying to build a very strong Go bot. He had interesting observations. It’s really good at running an experiment and going down that path, but it’s bad at stopping at dead ends and doing extremely parallel things. Anyway, this will probably change in the future.

It’s very interesting to think about what it looks like in the limit. This is fundamentally what the human institution of mathematical research is. It’s a library extended in interesting and useful ways. This way you don’t have any outcome-based supervision. There’s no outcome that you’re trying to incentivize, but you have a process. You know the steps are correct, you just don’t know if it’s going in an interesting direction.

Grant Sanderson

If you were doing that, you don’t want to completely go off the rails and do a random walk through the space of logic. You’d probably want some supervisor model that’s trying to provide heuristics on whether it’s useful or not. You know people are working on it. That’s one of those “five years from now” things where I’d be curious to get the future version of us talking about it.

Maybe that goes nowhere, but Terry Tao was talking about one research project that tries to exhaustively search the space of possible algebras. You could imagine different axioms that you apply to algebraic systems. When we come up with group theory, there’s a certain axiom system that looks like arbitrary rules unless you know the motivation. What if you tried all of them? Do any of them yield useful things? The vast majority of them are just trash in some way. It all collapses to no interesting results.

But every now and then, there would be this little island of a completely different type of axiom system that at the very least seems rich in terms of the number of theorems that can come out of it. That’s bread and butter for what you would imagine automated provers being good for, exploring that space and seeing which one of them turns out to be something. Maybe one of those islands actually turns out to be something you can retroactively put motivation on, to say this is the kind of structure it’s trying to get at.

In the same way that you could imagine looking at the axioms for a group, not knowing that it’s about symmetry, but you retroactively realize this is very relevant to studying symmetry. You could imagine results of that flavor, but instead of just exploring possible algebra systems, it’s exploring all possible logical consequences of any kind of axiom.

Dwarkesh Patel

On the point about whether you can provide process-based supervision without Lean, DeepSeek had their DeepSeek Math model. They released a paper on how they trained it, and it was quite interesting.

The problem with natural language proofs is you don’t know if it’s correct or not. They have a verifier, and the verifier is trained by a meta-verifier that makes sure that for all the problems they’re training this model to solve in the art of problem-solving, the verifier is giving good feedback. It works. It’s interesting that natural language verification with some sort of meta-verification seems to work so far in the published literature.

It also seems to work in the published products that we’re using. If you look at coding agents, they’re getting better and better at writing clean code and refactoring code. I’m sure there are process-based “LLM-as-a-judge” systems providing taste and saying, “Is this a clean way to write this function? Are there duplicates of the same kind of modular forms?” That should also work for mathematics, right?

Grant Sanderson

It seems more plausible for math than anything else, even if you’re only working in natural language, that you could trust a verifier. You and I were talking earlier about why they’re bad at writing. They seem to be good judges. If I give them two essays that students wrote, they’d be able to say which one is more accurate and insightful.  So why can’t you just have a verifier saying, “Is this a good piece of writing or not?”

Maybe the ultimate failure there is that even if they’re good at discriminating between a B essay and an A essay, they’re not actually good at discriminating between an A essay and a thing you actually want to read, something that would be followable on Substack and insightful. They actually end up preferring uninsightful pieces of writing.

On the math front, the step to simply know if a proof is correct or not lends itself to an automated verifier, even in natural language. You could probably still make a ton of progress. I still like the tree of logic out of Lean, just in that you can really go off the rails. There’s no constraint on the previous way things had been phrased before. Everyone talks about move 37 in AlphaGo. What is the thing that lends itself to going outside the prior heuristics? It seems productive to have a disconnection from the rest of the world in that exploration, as a complementary research pursuit to the natural language math front.

The other relevance of Lean would be, let’s say you have your pure natural language RL environments and a pure natural language set of proofs. People say, “Proceed, AI mathematicians,” and they generate ten papers a day. If there’s any error rate to that at all… Alex Kontorovich has talked about this. It becomes insufferable as a mathematician. Every single time you see one of these, you don’t know if it’s worth your time. Even if 99 out of 100 are right, I don’t know if it’s worth my time because it’s really labor-intensive to find what that error would be. It’s really frustrating to spend all your time on a paper that was trash.

Having something that’s able to give you that green checkmark that says, “Even if this is going to be complicated to understand, even if it’s going to be a pain, you at the very least know it is correct,” every other field would kill for that. Math has that. If the models are also able to take their natural language proofs and formalize them, that seems huge. Every field would love to have something like that. So I think you’re right that Lean is maybe overrated regarding its importance as a VR environment for progress in math generally. But I definitely wouldn’t write it out of the story.

Dwarkesh Patel

I also love this extension of Mathlib as a metaphor for what’s going to happen to our civilization pretty soon. For millennia, humanity has built this corpus of knowledge and understanding, and everything that we have is now distilled into these models. At some point, the models will just extend that arbitrarily.

Dwarkesh Patel

By the way, on the writing front, I have a theory of why writing is making worse progress than these other domains. One reason is what you said, that they’re bad at judging not only A versus B, but they get totally derailed by B*, which is this shitty essay that hits all the bells and whistles that A is supposed to hit. The reward hacking thing just goes off the rails.

But the other important thing is that writing is not modular in the same way that code and math are. You can write a function many different ways, and they do the same thing. Of course you want it to be clean, but at the end of the day, if it works, it works. Same with lemmas in mathematics. You can have some end product that’s different from the way it’s produced. Code is the thing that produces some end product, and you want a functional end product.

Whereas in writing, the end product is directly the thing the AI is producing. Each paragraph, sentence, and word matters because that is the substance. It’s not some separate thing produced out of the writing. It can’t be slop in the way that code can be slop and still produce the outcome you want.

Grant Sanderson

But you were just pointing out how we’ve actually gotten much better at agents writing not just functional code, but clean code. Why is it not the case that the same progress that lets you go from merely functional to a clean and mergeable PR also results in clearer writing?

Dwarkesh Patel

That’s a good point. Also, has it not? I agree there are many ways in which they’re terrible writers. But for a lot of writing I consume, I find it’s better to just copy-paste it into an LLM and say, “Explain this to me.” The explanation will be better than the thing produced by the human. It’s funny that we say these are such terrible writers, and yet my revealed preference is to have an LLM explain it.

Even when I’m talking to a human expert live on a call, if it’s a piece of knowledge that only they have that’s not encoded in the distribution, I want them to explain it to me. But if in order to understand that, I need to understand a more basic concept, I would prefer if it were socially acceptable for me to just say, “Let’s pause here. I’m just going to ask an LLM how that works, and then we can come back to your special piece of knowledge.”

Grant Sanderson

That’s distillation, an explanation. If I’m thinking of your quality as an essay writer—if I give you a book to read and I want a book report—I might believe that the LLM gives me a better book report. But what people are really getting at when they say it’s bad is, what is writing? It’s not just the distillation of preexisting ideas. It’s not just how you explain clearly, because they are good explainers. It’s about what the insight is.

This is where autoregression is a very weird way to generate things. When you’re writing, you sort of know that in order for it to be good, you have to have an element of the unpredictable. It’s not just increasing the temperature in your mind. It’s knowing exactly the correct point when you want to make an unpredictable move, and that that’s going to be what’s more insightful. Even if it’s better at explaining a preexisting thing, what generated that book that you wanted distilled in the first place?

It wasn’t an LLM that generated it and you just needed it. It was some author who, through a lot of exploration of ideas in the world, decided what aspects were interesting and what ways of presenting it formed a coherent, well-motivated narrative. They put that all together in some way. If they’re a good author, you would probably err on the side of reading their book instead of the distillation.

Still, what makes it worthwhile to explore at all in the first place and want to upload it at all? It’s that side of it that people cite when they say LLMs are bad at writing. It’s that element of unpredictability, of deliberately choosing something novel that is very directly contradictory to the way things are typically produced.

Dwarkesh Patel

That’s a good point. I think they’re also really bad at building really good mental models of people, which is a very important skill in writing. Andy Matuschak and another collaborator, whose name I’m forgetting right now, did an interesting report where they tried to teach LLMs to write good spaced-repetition prompts. I really like this because even though it seems like a totally random skill… It’s just like, people are talking about recursive self-improvement in a year, and we can’t get these things to write good flashcards. What’s going on there?

They tried many different kinds of techniques, and they’re sophisticated people. They tried to RL open source models. They tried all kinds of things, including chain of thought and a big prompt they sent to the best closed source model. The key constraint, it seemed to me, was that writing a good card is about projecting somebody’s mind in three months. What is the way in which they’ll associate the question? What kind of answer will they be thinking at that moment? Is the elicitation that inspires the detail you actually want to take away from the passage you’re trying to make cards about?

I think writing is similar to this. If you’re writing something, the reason it’s such an enervating process that takes so long is that with each word or each sentence, you have to be thinking: what is happening in my reader’s mind right now?

Even if I flip the phrasing around so the end phrase goes to the beginning and this is the first image that comes to your mind before you read the rest of the sentence… Maybe autoregression is bad at that. This is maybe a more diffusion-like property of considering the whole rather than going sentence by sentence. But also I think that requires a lot of mentalizing, which these models weirdly struggle at.

Grant Sanderson

It’s an interesting question. Is it weird that they struggle at that? I might butcher this. You know how you cite studies that you once read and maybe the study wasn’t real? There’s one very memorable one. Let’s say you want to quiz people’s EQ. You show a flashcard of someone’s facial expression and someone is trying to describe that emotion. There are really good tests online that have a face and then four possible emotions. It’s surprisingly hard to describe exactly the correct emotion, but you also get the sense there really is a correct answer. If you try this with people in your life, you’ll notice that the ones who are pretty plugged in socially do really well on it, and the ones who are a little bit more left-brain don’t. That is a kind of test you can do.

I vaguely remember an experiment to this effect where they took people who had freshly gotten Botox, and they did a pretest and a post-test. Post-test, they were just much worse at reading people’s expressions. That feels weird.

Dwarkesh Patel

Wait, they got Botox?

Grant Sanderson

The person taking the test. You do the test, and then you go and get Botox and your face is all frozen, and now you’re worse at understanding the emotions of what you see. The thought is that part of understanding the emotion you’re looking at is doing it yourself. At a facial level, you’re moving your face muscles. You see that, you mimic that, and you’re like, “Oh yeah, that’s anxiety,” at some very subconscious level.

So in that sense, if it is the case that models have bad theory of mind, sure, they know everything because they’ve read what everyone wrote. But at the level of actually being able to put themselves in your shoes in the same way that my face muscles are mimicking your face muscles—that’s what helps me understand how you feel—it’s not surprising at all. They don’t have face muscles. Their brain works completely differently. It’s like an alien trying to empathize. How could it have theory of mind? It would be this very emergent thing to have.

Whereas we can just plug it into our own minds. We’ve got the ready-made hardware to just place it in. From that lens, it’s not that surprising.

Dwarkesh Patel

What advice do you have about using LLMs to learn? As I was describing, for a lot of well-known concepts, I find them very helpful. But often, just a couple of messages further down, I’m trying to understand something, and they’re so confused themselves that they’re confusing me. They don’t explain it the right way. I know that talking to the right human could clear up my confusion in three minutes.

More and more, we’re going to want to use these things to learn. People talk a lot about education and representation stuff. Have you noticed ways to use them more productively to understand concepts?

Grant Sanderson

I’m curious to hear your take on this. I’ll give mine. Even pre-LLM, I feel like a relevant insight in learning was recognizing who matters more than what.

My advice to any college student when they’re choosing what courses to take: care a little bit less about your preexisting interests, because they’re kind of arbitrary right now, and care a little bit more about whether the person teaching it is a good educator and someone you resonate with. In choosing what books to read, who the author is maybe matters more than whether it’s a prior interest. If there’s a book you’ve liked before, read what else that author has written rather than reading another thing on that subject.

I’m getting to LLMs on this. There’s a difference in feel for trying to learn something from a Wikipedia page versus, if it’s a philosophy topic, going to the Stanford Encyclopedia of Philosophy. Or if it’s a math topic, you go to the Princeton Companion to Mathematics. The difference there is the articles are deliberately written by one individual who tries to actually craft a motivation around it.

Whereas on Wikipedia, it’s this local minimum that’s reached where every sentence has to be correct. In a good exposition, you care a little bit less about correctness on the way. You can deliberately craft things that are a little bit wrong that you correct along the way, which gets edited out in a crowdsourced environment. LLM explanations feel to me at the moment a lot like Wikipedia, which is to say, amazing. Imagine a world before Wikipedia, how long it would take to find and suss everything.

But nevertheless, what’s the most useful part of a Wikipedia page? It’s often just the references at the bottom. You look at the key references, and you go to them, and you read them. Sometimes that gives a much better overview. So often I like to just ask an LLM, “Who should I read?” Maybe I can even give some specifics on ways I want to learn.

I actually got gaslit by this once when I was trying to learn about semiconductors or something. I felt it was a very visual topic, but all the resources were text. I asked, “Is there a well-visualized video explaining the concepts you’re getting at?” And Claude said, “Yeah, here’s a couple,” and the top one was like, “Here’s one from 3Blue1Brown”. I’m like, “I can guarantee that there’s not.”

It was an actual video, an actual link, but it had just misattributed someone else’s. It was good. I had a much better experience clicking over and watching it to learn rather than trying to proceed forward with questions there. In that sense, I’m basically using it like a very souped-up version of Google to zero in on the right human-written resource. What about you? You engage with these a lot. What’s the best way to use them?

Dwarkesh Patel

I think you put your finger on it. The most productive learning sessions I’ve had are when there’s some artifact that a human has produced—whether it’s an article, a book, or a video—that organizes the relevant concepts in the correct way. It builds up the motivation for why the next idea would be relevant to solving the next problem you’d encounter, and the next idea, and the next idea. Then you use the LLMs to just do a little bit of pruning around this branch that the book has identified. I was actually going through—I think you might have recommended it—Steven Strogatz‘s textbook on…

Grant Sanderson

The chaos one? Nonlinear Dynamics and Chaos? I love that book.

Dwarkesh Patel

Yeah, I was going through it, and it was bliss. It was like your videos in book form. It was super fun. The way I was learning it, I’d have his university lecture on one-third of the screen, that part of the textbook on another third, and an LLM on the last third. I was actually thinking, if I were back in college and watching this lecture live, it would totally go over my head. These kids must be really smart, because I’m pausing, reading the textbook, talking to LLMs, and then restarting again. But with him curating the right order to understand concepts and the right problems to motivate understanding them…

Another thing LLMs are really bad at. Something a really good human can do, when you ask a question, a human can say, “Actually, you’re not really thinking about this topic the correct way. The question you want to be asking, the correct way to organize these concepts, is X.” An LLM just can’t really do that.

Grant Sanderson

It’s a little too placating. This is ultimately that sycophantic behavior where it’s very, “Oh, what an insightful question.” You want to strip that down. That’s a good point, and I think it cuts to theory of mind a little bit, recognizing that asking a certain kind of question reveals that the student’s mental structures are not the same as the explainer’s.

Sometimes people do this to a fault. With a really good teacher, let’s say you have a middle school math classroom. If a student asks a question that suggests they’re thinking about it in a different way, it’s actually really hard to take that seriously in the moment and ask, “Hang on, could you get to a right answer with that?” before you say, “Instead of that, let’s do this.” The really good teachers are able to jujitsu the creative way the student was thinking about it and bring it in. LLMs aren’t doing that. They aren’t reframing your question. Instead, they kind of run off.

At the very least, it feels like there are three levels here. An LLM is at one, a good explainer is at another, but the A+ explainer is the one who can jujitsu your way of thinking and say, “That’s where that’s useful.” Maybe there is a cycle all the way around where, five years from now, the LLMs will be doing that, but in a better way.

Dwarkesh Patel

What is your recommendation to students who I’m sure email you this question all the time: “I was curious about doing mathematics. I’m really passionate about the subject, but seeing all the progress AIs are making, I don’t know if it makes sense for me to pursue this as a career.” This is relevant not only to people in mathematics, but to anyone noticing that their field is getting productivity gains from AI. Coding is very adjacent to this. What advice do you have for people?

Grant Sanderson

I wouldn’t trust any advice that I give. That’s how I’d couch it. But even pre-AI, it feels very important for any job you’re going to go into to really understand… If we’re talking about a job—not being a gentleman-scientist engaging with the math world or something—you should understand where the money is coming from, what value you’re actually adding, and the connection between those two. A surprisingly small amount of thought is put towards that, especially by students.

They’re in this environment where they probably want to go into math because they’ve always been good at it. They’ve been rewarded in life for proceeding through the next hoop correctly. When they think they want to be a mathematician, it’s because they think it’s a way to continue engaging with that. They think, “Where do people get to do this?” rather than thinking, “What value am I adding to other people, and to what extent is that the reason a salary is flowing in my direction?”

It’s actually quite different in different cases. In some cases, it’s a very prestigious mathematician, and their presence at a university lends a certain brand value, which is why the university wants them. In some cases, an NSF grant is given because of the public good belief we have around basic science. You’ve got an institution around that, and a whole bureaucracy acting as a proxy for what we think that public good is, with a whole song and dance around how to make them correctly predict that your progress will be in the spirit of that funding. Sometimes it’s just straight-up teaching. People like to send their kids to an institute that has experts teaching them. You provide brand value by being an expert, and direct value by being a teacher.

Regardless of whether AIs are proving theorems or not, or whether we’re talking about 2016 or 2026, that is something not enough students thinking “I want to be a mathematician” consider. I think it’s worth thinking about. For me, I wasn’t necessarily thinking about it, and I stumbled into a career path where math exploration can be monetized as entertainment. I stumbled into that and I’m very grateful I did, but it was an accident. It wasn’t deliberate. I could have avoided relying on serendipity and done it a little bit more by design had I been thinking critically about it.

To your question—if we have almost-automated theorem proving, and let’s say they’re also really good explainers so you even get the human understanding—I think a lot of the social role that mathematicians serve actually doesn’t change that much. As a public, we still feel there’s value to basic science, and we trust the judgment of mathematicians to determine where their time is best spent. The prestige comes from within that community. It’s other members saying that a result is really good, more than the grant writer really understanding algebraic number theory to understand it was a good result.

There’s going to be an inner culture of what constitutes valuable contributions. Maybe it shifts away from theorem proving and towards good definition writing. Maybe it’s that museum curator idea. But you’re going to have that same community as long as society as a whole is still valuing the premise of basic science. And if we’re in the abundance world that AI brings, there’s probably more funding in that direction in some sense.

On the side of prestige to institutions for who their lecturers are, I actually think teaching is one of the most stable post-AGI jobs that there is, because it’s so relational. This is where parents want to spend their money if they have an abundance of wealth: on good teaching and good educating. It goes so far beyond explanations. Even if LLMs are good explainers, the thing that a teacher is doing is such a social, coaching, mentor-type thing that that’s probably one of the most stable careers that’s going to exist over the next fifty years.

Insofar as a lot of mathematicians’ roles overlap with that, as the prospective student going into it, you could lean into that. I actually think a lot more students should think about and give credence to the idea of being just a math educator and the value that can serve towards the next generation.

I’ll couch again that I don’t think I’m the one to say, “Here, prospective young mathematician, here’s how you should think about the future,” because I’m a YouTuber. I’m not in the institution that they’re thinking of going into, so I’m speaking as an outsider looking in. But it feels like generally good, universal advice: know where the money is coming from, know where you plug into that. And if you’re just asking those questions, you’re actually already steps ahead of all the other fledgling prospective mathematicians.

Dwarkesh Patel

In fact, think about the crazy world where, within five or ten years, the AIs are coming up with not only solutions to the Millennium Prize problems, but totally novel problems to be solving in the first place, novel mathematical fields and objects and stuff. It is in that world where, first of all, there’s a ton of abundance. Two, the thing AI minds will have gone furthest in, where they will have seen furthest beyond our horizons, will be mathematics. There will be so much demand for, “What have the AIs seen? Can you explain it to us?”

In that world, if there are any jobs whatsoever, surely distilling what the AIs have learned will be one of them.

Grant Sanderson

Also, it’s funny because all of this presumes that it’s useless. We’re not talking about the actual practical applications of what math is being done. Insofar as there’s any economic utility to it, you would imagine that the people who understand it and are able to make the decision of where it should point actually have a lot more economic value by being able to make that judgment as a curator and point this behemoth of new math in a useful direction. Suddenly, that’s a much more levered move to make than it had been previously.

Dwarkesh Patel

Can I ask you about that? Obviously, one question for AI for math is not only can it do it, but is it any good? Or is it good for anything?

You were describing all the ways in which, with group theory, we’re trying to figure out random facts about the roots of different kinds of functions, and now there are all these different applications that are practical across many different fields. Do you have some sense of whether, if we just totally get to a place where the field of human mathematics is accelerated 10X or 100X and some crazy shit happens, or are we just going to be bottlenecked by other fields?

Grant Sanderson

I think there are some fields that probably will. It’s super spiky. With progress in algebraic number theory, it feels unlikely that that then unlocks something. But I remember talking to this mathematician who does more dynamics and PDE-solving type stuff. He was referencing that his group had some ideas. Let me see if I summarize this right. It’s like the way Boeing would make planes is that they’d make it, do a bunch of tests, and they had to disassemble it and reassemble it based on those tests. His group essentially had some insights on how to do more in simulation such that you don’t have to deconstruct and rebuild it. It saved Boeing billions of dollars or something, and then they just started funding that group.

That’s much more obviously application-adjacent, because PDEs just are that. Progress in that domain, you would imagine actually does unlock some things. I don’t know if it’s these step changes, but maybe it’s more on the side of engine design becoming a little bit more fluid, or coming up with the right wing shape instead of running a whole bunch of complicated CFD. Maybe you’re able to speed up your CFD simulations because certain pure math insights make those more efficient. I bet you’d just see a lot of great incremental improvement there.

It seems less likely that the massive breakthroughs in math immediately turn into this massive economic breakthrough, like you solve the Navier-Stokes problems, and then that unlocks an ability to simulate more things. But you probably will see, at those fringes, some meaningful leakage out of the pure math insights into other things. There’s a ton of people working on things like AI engineering, physical engineering, and material science. You have to imagine they’d be in a good position to look at the AI math insights and decide whether they’re relevant in some way or not.

It’s another one of these things where I’m not going to sit here and put a flag in the sand predicting that there will be. But it’d be a little bit disappointing and a little bit surprising if there weren’t, over the next five years, economically valuable improvements made that were directly referable to the AI progress in math. It would just be disappointing if it was just taking down a bunch of Erdős problems and none of them were doing any of the math that actually directly touches the physical world.

Dwarkesh Patel

To your point about how a lot of the history of mathematics was about building up these piles of concepts and connections. Sometimes the piles connect with each other, or you discover an application somewhere else. At the very least, you just build up this huge pile.

Then as broader progress in society happens during the singularity, when we get to the industrial part of the singularity, you just have all these different ideas that hopefully are useful in other parts of the world.

Grant Sanderson

As I said, one of the interesting things about what’s happening is it causes people to step back and ask, “What is math?” Maybe one of the awkward conclusions will be revealing that it’s just become wholly useless. The kind of questions being asked have become so divorced from things that are physically applicable that that’s one of the things mathematicians have to come to terms with.

Everyone will look and say, “Hang on a second, weren’t you guys supposed to… If there’s 10X progress there, why aren’t we seeing it over here?” And then mathematicians are like, “Ugh.” Every time we wrote those grant proposals and said, “Trust us, the elliptic curve progress is going to help with cryptography,” it shines a light on the fact that maybe it doesn’t. So that’s one possibility.

Dwarkesh Patel

Grant, this was super fun. Thanks so much for doing it.

Grant Sanderson

Absolutely. My pleasure.

Here’s the big research bet the labs are making currently: if we train AIs to accomplish millions of verifiable tasks across thousands of diverse RL environments, then we’ll basically have built AGI. Because such training will create these general problem solving skills (like how to make progress on an open ended task for weeks on end in the face of errors, mistakes, and ambiguity).

The people optimistic about this vision would say that anything we might consider a fundamental deficits with the current learning paradigm—for example, data inefficiency and lack of continual learning—can be steamrolled by just scaling training more, just as all the supposed “fundamental” research problems in natural language processing collapsed against the flood of compute thrown into LLMs.

Yes, these models are 1/1-millionth as sample efficient as humans during training. But training a one-time cost amortized across billions of user sessions. What matters is how smart, general, and sample efficient the model is within a session, and that’s clearly been improving as we do more RL training. AIs are able to solve more and more ambitious problems across longer and longer time spans - anybody who’s been using these models for coding knows that.

Similarly, continual learning—as defined as the model’s weights getting updated from deployment—may simply not be necessary. Again, because if in-context learning gets so good across longer and longer horizons, then we don’t need to distill back to weights to get on-the-job learning. People often say that their employees are not net productive until six months or more on the job, so clearly online learning is necessary for competence. But what if you could just fit those six months into the context window? There’s been tons of architectural innovations on the transformer which dramatically increase the length of context you can store. With a couple more years of progress, why couldn’t we have arbitrarily large context windows?

To address whether this will work, I want to first take a detour and ask a question about the current nature of AI progress that I find confusing and interesting. Why has progress on computer use been so slow?

Computer use is so clearly verifiable (did the desired Etsy item get ordered, is everything I need corporate for my event booked, have my taxes been submitted). So isn’t it weird that computer use has been making much slower progress than coding and math and other verifiable domains? There’s many reasons for this, I’m sure, among them the fact that the models are exposed to far less high quality multimodal data during pretraining, and that video consumes the context window far faster.[1]

But one reason that I think it quite underrated, and also which reveals the canyon walls against which the river of AI progress will only slowly chip away at, is that it is not enough for a domain to be verifiable. It also has to be very grindable—in the sense that you can run lots of parallel rollouts against a deterministic and replayable simulator. If you’re trying to make a model better at coding, you can create an environment that has a software repo with some missing feature that you’ve tasked the AIs with creating, and then you have a thousand parallel agents just go at the problem, each with their identical copy of the container.[2]

But this doesn’t work with computer use—at least not trivially. You can’t have a thousand agents go try the same checkout flow on Amazon.com. Because Andy Jassy will find and detect your bots and shut your ass down.

You can solve this by making clones of Slack, Gmail, and all the other common applications and websites. But at least currently, this is a very labor-intensive and unscalable way to build environments. Of course, once AIs get good enough at coding to themselves build these clones with extremely high fidelity, then I’m sure computer use will make a ton of progress. And you’re also killing two birds with one stone with this kind of procedure, because getting AIs to rebuild whole complex applications from scratch is a great RL objective for coding as well.

But while computer use itself may soon be solved, its current lethargy tells us the following: that unless you can build a very replayable training target for a domain, the models will struggle to make much progress. The reason this is true is, of course, that the models are incredibly sample inefficient during training. This is the point I was making in my last monologue.

In computer use, we might be able to make up for this sample efficiency deficit by building these farmable deterministic simulators. But for so many different other kinds of skills an AGI would need to learn, we simply can’t do this.

How would we train an AI to build a business? How would you make an AI that’s really good at winning court cases? Or having a profitable day trading in the markets? Or helping a candidate win an election? The rollout requires interacting with the world and cannot be recreated simply within the datacenter. And the outer loop verification may take months or years of real world actions to elicit, and cannot be re-observed by perturbing the model’s actions thousands of times in parallel so that you can isolate what exactly the model did that actually worked.

Dealing with such reset-free non-stationary environments is a known open problem in RL. I’m not pointing out anything new. But I really do want to emphasize that because of the idiosyncratic and sparse nature of the data in most domains in the world, you need sample efficiency in order to get proficient.

If AIs are to develop all the skills that humans have, and even skills that no humans have, then they need to be able to learn from information revealed in unstructured, unverifiable, and ambiguous ways from scarce amounts of real world interaction. Because in many domains, the relevant training information simply doesn’t exist in any other way.

What is the RL environment to make an AI as good at politics as Lyndon Johnson, or as good at building a space launch business as Elon Musk?

The labs are betting that RLVR will generalize to all these other domains. If you train in enough containerized, reproducible environments, you will develop a very general agent that can make and execute plans, and learn rapidly from new information, and even pick up new skills, all within a session.

If you drop this endlessly RLVRed AI into Texas politics in 1948, it could give you better advice than LBJ about winning the Senate seat; And if you gave it 100 million dollars in 2002 and let it cook, it could build SpaceX for you.

Whether RLVR generalizes that well is an open empirical question: if labs went from spending billions of dollars on RL environments to a trillion dollars, would you get a fully general, human-like intelligence operating within the context window?

Dario gave a telling quote during our podcast together, which I think hints that RLVR generalization is not this infinitely strong. When he was explaining why model performance tends to degrade at long context, he said:

There’s the context length you train at and there’s a context length that you serve at. If you train at a small context length and then try to serve at a long context length, maybe you get these degradations.

Maybe I’m reading too much into it but he seems to be saying that short-horizon RL training doesn’t necessarily generalize to long-horizon RL performance. And if we can’t generalize from short to long horizon, how are agents supposed to generalize from lots of white collar task training, to, say, getting dropped into the real world and building a business from scratch as well as Sam Walton?

And even if after enough in-context experience, the AIs could become Albert Einsteins and Henry Fords, all that would be ephemeral and wasted if you can’t get those learnings back into the weights. Around 30-50% of a lab’s compute goes to inference, and that compute is currently not really doing anything productive in helping improve the model. What a waste! It’s even worse than it sounds. Because it is only in deployment that the most valuable bits of information which your model could learn from are revealed (What’s actually happening in the organizations I’m being used at? What are they using me for? And what kind of mistakes do I tend to make in the real world?)

We’ve got some genius grad student who has never been allowed to take an internship. And we keep giving it more and more classroom case studies in the form of RL training on environments. It’s bizarre and wasteful that we don’t train the AIs against all this experience could be accumulating thanks to being so broadly deployed through the economy and getting to practice against millions of different assignments given to them and being privy to so much tacit organization- and domain-specific knowledge

But this kind of continual learning requires going back to the weights. AIs can’t just keep building up a KV cache that grows in size as you keep learning from more and more users. That’s just not scalable, and it’s also not how humans learn. We don’t have some separation between parameters and activations. And there’s not some lump of these fast-weight representations that juts out further and further from our skull as we learn more things throughout our lifetime. When we learn stuff, there’s clearly some kind of compression, which actually aids generalization and grokking. There are in fact some humans who have this autistic savant type recall of random tables of numbers or nonsense syllables years later—basically the kind of fidelity of information that models have in context. And such sheer volume cripples these humans’ ability to understand abstractions and metaphors. Human continual learning is less about having all your observations at the tip of your tongue, and more about chiseling the right intuitions and big picture knowledge back into the weights.

But the moment you move into the weights, you have to give up on in-context learning’s sample efficiency. Because gradient updates are super sample-inefficient, all the successfully shipped online learning models have had to learn the same thing across millions of users. For example, the Cursor Tab model online-learns by predicting the same exact objective for over 400M+ requests a day (that objective is which edits got accepted). At least so far, we haven’t seen models online-learn different kinds of things for different users, because while a single session may generate more than enough data for a human to learn from, it’s not enough to train a more capable AI.

Current online learning can work for a very limited number of use cases. But the whole point of continual learning is that the world is very complicated, and each job and company and problem is different, and you need your intelligence to be able to learn the specific information related to a particular deployment, which simply can’t be stuffed into a shared training run. Things like how everything in your organization works and fits together, how to cooperate with the infrastructure and the other people around you to make progress on some larger project, what common failure modes are, etc.

This is the way in which sample efficiency and continual learning are actually deeply connected problems. Relatively little data is available to the model “on-the-job”. To learn from that data requires sample efficiency. Models can do that in context, but the “fast weights” built on the fly by attention[3] which allow for this sample efficiency scale very poorly in terms of memory. So we need architectural innovations which allow for some kind of intermediate representation. I talked before about how there are already many different working ideas for this kind of thing, from sparse attention and KV cache compaction. It doesn’t seem to me that architecture is fundamentally the bottleneck to continual learning.

Perhaps the bottleneck is the loss function. How do you update the weights (aka improve the model itself) based on information that was learned from one particular session? Even here naively it seems like there are many ideas that oughta work. Lots of people have been talking about on-policy self-distillation recently. If you want to learn more about how it works, check out this little impromptu blackboard lecture that Sasha Rush gave me a couple of weeks ago. But to summarize the explanation a bit, the idea is that we encourage the base model to make the same predictions when trying to solve some real world problem as the model with all the context accumulated after a long session would have made. The whole point of this procedure is to distill what the model learned in a session back into the weights themselves.

This is better than RLVR for two reasons. One, OPSD doesn’t require an outer loop verifiable reward. We just need a model that can learn the right things within the context window. As long as we have that, we can train the base model to match our veteran teacher model which has built up all this experience during the session. And two, OPSD provides a much denser supervision signal than naive RL—instead of projecting a single reward through the whole trajectory, you can train on the per token probability discrepancy between the teacher and student[4].

For continual learning, OPSD is also superior to supervised fine tuning. The most naive version of SFT for this application you can imagine is to train the base model to predict all the tokens observed during the session. But this makes no sense as a learning target - the way you get better at your job is not by recalling the transcript of what happened through every single day with perfect fidelity. Rather, it’s by consolidating the handful of insights and pieces of knowledge that are relevant to doing your job better.

RL training doesn’t suffer from this failure mode, and it’s great at concentrating the gradient update to only what is relevant to getting the outcome right—that’s why the updates from RL are incredibly sparse. And this is a very important property for continual learning, because as you’re learning on the job, you don’t want to overwrite and forget all the other things the base model knows.

I wrote a post a few months earlier arguing that RL learns much less information per sample than supervised learning. But this may be a good thing rather than a bad thing—you only change the model as much as is absolutely necessary to achieve the outcome, and no more. OPSD preserves this property of RL where instead of slingshotting towards the teacher distribution like supervised learning would have you do, you only extract the knowledge that is necessary to achieve the same results on real world tasks.

So OPSD is one way to attack the sample-efficiency problem: you can take this scarce real world experience and squeeze all the signal into a tiny, well-targeted update. But there’s also another much more speculative idea. Let’s call it dreaming[5]. If the AI can build a good simulation of reality against which to rehearse new skills, or try alternative strategies and reinforce what works, then it could experience orders of magnitude more simulated samples in the same wall clock time.

A couple years after DeepMind released AlphaZero, a group of researchers trained a model called EfficientZero. If this model and a human both got 2 hours total to play against a simulator of an Atari game they hadn’t seen before, this model would likely beat the novice human. Does that mean this model was more sample efficient than humans? Well it depends on how you measure sample efficiency. Because for each step in the real game, EfficientZero is playing dozens of simulated games in its head. In a similar way, future LLMs might be able to consume far less real-world data while practicing endlessly against environments they build for themselves. The big difference, of course, is that it’s much harder to build a simulation of the whole world than it is to emulate the game of Go. That’s why I said this is much more speculative.

If it works, it would become a fourth axis of scaling, alongside pretraining, RL, and inference-time compute. You can call it test-time training or dreaming. The model spends compute writing up RL environments in which rehearse the skills that will actually be used in production for a specific user. Instead of hitting /compact on Codex or Cursor or Claude, which kindles a small amount of compute to write up a summary, and which gives you a simulacrum of continual learning, you hit /dream, which incinerates huge amounts of compute to build and train against a video game version of what the model is witnessing in the world.

So what might continual learning look like at the end of 2027, and how do we get there? All this RLVR training is producing an agent that can get its bearings when it’s thrown at an unfamiliar problem, and try different strategies, and iterate when it hits a roadblock. This is the crucial thing that RLVR has given you: an AI that is at least competent enough to start getting some real-world experience. Once you have that, you send it out into the world to do real work, even on projects off the training distribution.

By this point, effective context lengths may have expanded such that this AI can cowork with you for a full week of wall clock time. At the end of the week you give it a thumbs up or a thumbs down. If you give it a thumbs up, the base model distills everything the AI learned during the session, and it may use OPSD, or dreaming, or some other technique we aren’t even aware of, or a combination of all of the above, to do so. And AI can get better at domains that are adjacent to what it was explicitly trained for beforehand with RLVR. And in the next round it gets better at the thing adjacent to what it was previously online learned. The gamut of AI skills and knowledge and capability expands far beyond the verifiable domains against which the model was trained before it was deployed. Just as pre-training created a base intelligence that was smart enough to become a competent agent with further RLVR training, so RLVR has created an agent that is competent enough to actually be deployed in the world and thus take advantage of the future paradigm of continual learning.

By this point, the main way that AI gets better is not through the training received before the model is released to the public. Rather, it’s from all this experience that they are accumulating from being broadly deployed through the world and engaging in so many different kinds of tasks. Every time you interact with AI, it’ll be smarter. Not only because it has been learning from all your previous sessions, but also from all its interactions with all the other users in the world. And that’s scary and exciting and very different from the way that AI improves right now.

Mercury has automated basically my entire bill pay process for my business. I just give contractors a dedicated email address, and when they send an invoice, Mercury automatically creates a draft payment for me to review. I no longer have to hunt through my inbox for invoices or deal with messy spreadsheets to track my bills. Mercury handles it all. Learn more at mercury.com.

1. Just one hour of video consumes around 1 million tokens of text.

2. I’ve heard that AI agents are especially good at Go, because it has an excellent standardized package manager whereas Python and Typescript have a “massive combinatorial space of frameworks, typing approaches, and utility libraries.” Such spaces are less amenable to clean, high throughput, parallel search via gradient descent.

3. Let’s use Llama 3 70B as a reference. The KV cache (aka the representation that is built up from learning the context) grows 320 KB with each token. Whereas in training, the model only stores 0.075 bits per token (it’s a 70B model with 16 bit parameters trained on 15 trillion tokens). So between in-context learning and pre-training there’s a 35 million fold difference in the amount of information you’re storing per token.

4. One obvious issue you might anticipate with OPSD is this: you get dense supervision up to and at the point where the student makes an error, but the rest of the trajectory follows from that error - it continues down an already-mistaken path that the teacher was never going to visit anyway. So past that point, you’re no longer getting useful feedback from the teacher for the rest of the rollout. This seems fixable by a technique called Trajectory-Refined Distillation, where the teacher rewrites the trajectory from the error onward into a complete, correct continuation.

5. For what it’s worth, I’m not talking about the upcoming dreaming feature in the leaked Claude Code source code, which I’m guessing will be more about the model writing lots of Markdown files for itself. I mean actually updating the weights themselves. I just don’t think you can accumulate new skills by passing yourself notes. The analogy I used in a previous blog post: imagine if the way students learned how to play the saxophone is by giving this new instrument you’d never tried before a go, taking some notes about what went wrong, and then giving it to the next student who is also playing it for the first time.

One definition of intelligence is sample efficiency - that is to say, how much data do you need to see in a given domain in order to operate fluently and competently. It’s not clear that we’ve actually made much progress on training sample efficiency over the last few years - it seems like more so we’ve dramatically widened and improved the data distribution.

The main way that AIs have been getting better is from adding more and better data, and scaling the compute to develop that data in the first place. Obviously RL is the main way that has happened. You can think of RL as a kind of synthetic data generation - you dump a lot of compute against a verifier in order to find the “good” data. Then you train your model to predict these correct rollouts, much in the same way that you might train it to predict the next word in internet text.

For this process to work, the model must have at least prior some probability to anticipate the correct solution, which is why you also need mind-stretching amounts of human expert trajectories in every single field and skill you want the model to be competent at.

It’s hard to overstate how task specific and bespoke this human expert data is. If you want to get some intuition, go read some job descriptions at Mercor or Surge’s websites. There are listings for a word specialists who will convert legacy documents into polished Word files, and legal experts who will write realistic M&A diligences or securities filings, and management consultants who will write up template market research, and dozens more other particular categories.

And it is not only that the data have to be so domain specific, but there has to be so much of it! Each skill corresponds to at least hundreds of human experts who are generating example completions, writing rubrics, and explaining their chain of thought. There’s a reason that the data industry producing these expert labels (and the RL environments in which their meticulously catalogued skills can congeal) is earning billions a year in revenue, soon deca-billions.

Imagine if it took a couple decades worth of courses with hundreds of concurrent professors and millions of practice tasks for you to learn how to polish a word file. Even the task count difference understates the gap - the models have to grind their far more numerous tasks each far harder. Whereas a human student might practice a textbook problem once or twice, GRPO has the model generate hundreds to thousands of rollouts per task. We are building some Frankenstein’s monster, with a billion grafts of carefully constructed examples sewn together.

Epoch recently reported that open models only lag state-of-the-art closed models by 4 months. I think the reason it is relatively easy for open source and previous laggards to catch up to within months of the frontier is that data is the real driver of progress. And data can be easily distilled from public APIs, whereas hyper-parameters and training tricks and architectural micro-optimizations cannot - if the latter were driving most of progress, then catching up would be harder than we are observing it to be.

It is easy to forget how much data these models are trained on, and how much more it is than what we humans see in our lifetimes. We see these AIs as a galaxy glittering with capabilities, but at their center, invisible to the naked eye, holding all the constellations together, is an unimaginably massive black hole of data.

• If a person hears and sees on average ~2,000 words an hour, then from birth to adulthood, they’ll see ~200 millions tokens. By contrast, frontier models are trained on somewhere between 10s to 100s of trillions of tokens. That is close to a million fold difference.

• A person can learn to teleoperate any random humanoid or robot arm within hours. The reason robotics isn’t already a deca-trillion dollar industry, with a endless army of Unitree G1s doing all kinds of useful work in world, is that our AIs learn so much less efficiently than humans, and even the millions of hours of demonstrations we’ve collected is not enough to allow them to perform complex, open ended tasks.

• A teenager can learn to drive a car with about 20 hours of practice. Even if you include their ~16 years of accumulated physical intuition as relevant training data, that is at least 3-4 orders of magnitude less than the amount of data Waymo and Tesla have needed to train their self-driving car models.

I wanna deal with some common objections to this kind of comparison:

• Many billions of years of evolution is our pre-training, so it’s unfair to compare how little data we see simply within our lifetime to what these cold-started LLMs have to learn from.

  • Our genome is 3GB, about 1-2% protein coding. That is just not enough space to store the model parameters that are supposedly pretrained (frontier models are terabytes sized). The closer analogy is probably that evolution has found the right hyperparameters and loss functions (Sidenote: I had an interesting podcast with Adam Marblestone where he argued that the loss functions were the more significant find from evolution), but that the equivalent of parameter training is still happening within lifetime, and is encoded in the map of neural connections in the brain built up over a lifetime.

  • Even if it were the case that we can explain away the trillions of tokens required to pretrain a base model as catching up to evolution, it doesn’t explain why the marginal capabilities take so much data - once you have been educated, you don’t need 100 different professors to learn a new programming language, but the AIs (even once pretrained) do.

• These comparisons are not including the multimodal data we see in our lifetimes. If you include all this sensory information, we’re probably in the 10s to 100s of billions of tokens range from birth to adulthood

  • Blind/deaf people who are cut off from this kind of sensory information might lack faculty with the relevant sense, but still have the same general intelligence as everyone else. Which suggests that all these billions of sensory tokens are not really the thing making humans smart.

  • In fact, deaf people who can only communicate via sign language and reading (and not from hearing) are ingesting far less than the 200 million language tokens we calculated earlier, and even this is sufficient for them to be fully general intelligences.

• Scaling laws tell us that bigger models are more sample efficient. The human brain is 100T synapses - if each synapse is ~1 parameter, and frontier models are currently roughly ~5T parameters, then maybe we could achieve human-level sample efficiency with another order of magnitude or two of parameter scaling.

  • The way the scaling law equations work is that parameter and data terms are added to the loss independently. If you have a model that is trained compute optimally, and suppose you ask, well what if I just wanna maximize sample efficiency and use less data - and I’ll throw in as many parameters as it takes to make that happen. With the constants from the Chinchilla scaling laws paper (and the nature of the result wouldn’t change even with different constants), even if you increased the number of parameters by infinity, that would only decrease by a factor of ~10 the amount of data you need in order to keep the same loss. Humans are somewhere between thousands to millions of times more sample efficient than these models. Scaling of current models simply can’t make up for that discrepancy. This really does suggest that humans are on a different scaling curve altogether.

But you might ask, why does sample efficiency matter? The labs have two overarching objectives: automate white collar work, and automate AI research itself. Is human level sample efficiency necessary for either?

The bet with white collar work is the common tasks that a software engineer or analyst or accountant does are, well common. And we can bring common tasks into distribution quite easily through RL and SFT. The revenue curves of these AI labs suggest that there is enormous value from bringing tasks into distribution, even if we don’t replicate human sample efficiency.

Yes it is far more inefficient to train AIs to do these tasks than it is to train humans. But so what? Human lifespan does not allow for the quantity and breath of training these models experience. If you as a human had some weird learning disability where you needed to read through every public repository on Github before you could be a competent developer, it would not make sense to train you up. You’d be on Social Security by the early stages of your education, and even once you were trained, you could work on only one project at a time. But AIs can learn these skills by firehosing gigawatts of training at a time. And what they learn can be amortized across billions of sessions, so we can be ludicrously inefficient in training them and still be wildly in the green.

How much “out-of-distribution” thinking do white collar employees need to do that you simply can’t train for in advance? Well this is more a question about the nature of different jobs rather than a question about AI research. And also depends on the job - some jobs are mechanical and predictable enough that they were automated long before the modern era of AI, for example bank tellers or travel agents. And there are other jobs which require dealing on a daily basis with problems that are quite distant from the data distribution. Even software engineering (the jobs AIs are supposed to take first) is one such. I would be willing to bet that there’s overall more demand for human software engineers in 2028 than there is now, largely due to the complementary input of AI.

The labs’ plan for these later kinds of jobs is to first automate AI research, and then have the automated AI researchers solve this sample efficiency problem. So then the question is, can AIs, which do not have human-level sample efficiency, nonetheless solve the remaining research problems on the way to human-like intelligence and learning.

That question I’ll address in a future blog post - I think the way that people currently think about an intelligence explosion is pretty clumsy. Either people dismiss the possibility of AIs speeding up AI progress altogether, or they just assume that God pops out the other end. People are not reasoning about what extremely rapid progress, but starting with LLMs, looks like.

Thanks to Mercury for sponsoring this essay! Mercury is my banking platform, and they just released a new AI feature called Command. Since I already use Mercury to run basically my entire business, Command has access to all the info it needs to get real work done. I can ask it to send invoices, or categorize expenses, or even transfer money… and Command just handles it. Learn more at mercury.com/command